Instruction Tuning

Instruction Tuning

Instruction tuning fine-tunes language models on (instruction, output) pairs, enabling them to follow natural language instructions and generalize to unseen tasks.

Why Instruction Tuning?

Pre-trained language models possess knowledge but lack the ability to follow instructions directly. Instruction tuning bridges this gap.

Instruction Tuning Pipeline

FLAN (Fine-tuned LAnguage Net)

FLAN scales instruction tuning across diverse tasks, demonstrating emergent cross-task generalization.

FLAN Data Construction

from datasets import Dataset
import random

def format_flan_example(dataset_name, task_type, input_text, output_text):
    templates = {
        "SINGLE_LABEL_TASK": [
            "Classify the following: {input}\nAnswer: {output}",
            "What is the label for: {input}?\nAnswer: {output}",
        ],
        "GENERATION_TASK": [
            "Given the following input, generate a response.\nInput: {input}\nResponse: {output}",
        ],
        "SUMMARIZATION_TASK": [
            "Summarize the following: {input}\nSummary: {output}",
        ],
    }
    template = random.choice(templates.get(task_type, templates["SINGLE_LABEL_TASK"]))
    return {
        "input": template.format(input=input_text),
        "output": output_text,
        "dataset": dataset_name,
        "task_type": task_type,
    }

examples = [
    format_flan_example("sst2", "SINGLE_LABEL_TASK", "This movie is great!", "Positive"),
    format_flan_example("ag_news", "SINGLE_LABEL_TASK", "Stock market hits new high.", "Business"),
    format_flan_example("xsum", "SUMMARIZATION_TASK", "Long article about climate policy...", "New climate legislation passed."),
]

dataset = Dataset.from_list(examples)
print(dataset[0])

FLAN-T5 Training

from transformers import T5ForConditionalGeneration, T5Tokenizer, Seq2SeqTrainer, Seq2SeqTrainingArguments

model_name = "t5-base"
tokenizer = T5Tokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name)

def preprocess_flan(examples):
    inputs = ["instruction: " + inp for inp in examples["input"]]
    targets = examples["output"]

    model_inputs = tokenizer(inputs, max_length=512, truncation=True, padding="max_length")
    labels = tokenizer(targets, max_length=128, truncation=True, padding="max_length")

    labels["input_ids"] = [
        [(l if l != tokenizer.pad_token_id else -101) for l in label]
        for label in labels["input_ids"]
    ]

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

training_args = Seq2SeqTrainingArguments(
    output_dir="./flan-t5",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    learning_rate=3e-5,
    weight_decay=0.01,
    warmup_steps=1000,
    evaluation_strategy="steps",
    eval_steps=500,
    predict_with_generate=True,
    fp16=True,
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
)

trainer.train()

InstructGPT (SFT + RLHF)

InstructGPT pioneered the two-stage approach: supervised fine-tuning followed by reinforcement learning from human feedback.

import torch
import torch.nn.functional as F

class InstructGPTTrainer:
    def __init__(self, sft_model, ref_model, reward_model, tokenizer, beta=0.1):
        self.sft_model = sft_model
        self.ref_model = ref_model
        self.reward_model = reward_model
        self.tokenizer = tokenizer
        self.beta = beta

    def compute_sft_loss(self, batch):
        outputs = self.sft_model(
            input_ids=batch["input_ids"],
            labels=batch["labels"]
        )
        return outputs.loss

    def compute_reward(self, prompts, responses):
        inputs = self.tokenizer(
            [f"{p}\n{r}" for p, r in zip(prompts, responses)],
            padding=True, truncation=True, return_tensors="pt"
        )
        with torch.no_grad():
            rewards = self.reward_model(**inputs).logits
        return rewards

    def compute_rlhf_loss(self, batch):
        prompts = batch["prompts"]
        responses = batch["responses"]

        policy_logprobs = self._get_logprobs(self.sft_model, prompts, responses)

        with torch.no_grad():
            ref_logprobs = self._get_logprobs(self.ref_model, prompts, responses)

        kl = policy_logprobs - ref_logprobs
        rewards = self.compute_reward(prompts, responses)
        loss = -(rewards - self.beta * kl).mean()
        return loss

Multi-Task Instruction Formatting

Scaling Laws for Instruction Tuning

Evaluation Frameworks

Best Practices

1. Data quality over quantity - 10K high-quality examples outperform 100K noisy ones
2. Diverse task coverage - Include classification, generation, reasoning, and extraction
3. Consistent formatting - Standardize instruction templates across all examples
4. Appropriate model size - Larger models benefit more from instruction tuning
5. Curated learning rate - 2e-5 to 5e-5 works well for most base models

Key Takeaways

• FLAN demonstrates that multi-task instruction tuning enables emergent cross-task generalization
• InstructGPT established the SFT then RLHF paradigm used by most modern assistants
• Data quality matters more than data volume for instruction tuning
• Format consistency across instruction templates improves model learning
• Evaluation requires both automatic metrics and human judgment for reliable assessment