Skip to main content

Chapter 1: Dataset Formats

Structuring Data for Fine-Tuning​

Different formats serve different purposes. Understanding them helps you choose and prepare data effectively.

Common Formats​

1. Alpaca Format​

Single-turn instruction following:

{
  "instruction": "Summarize the following text",
  "input": "The quick brown fox jumps over the lazy dog...",
  "output": "A fox jumps over a dog."
}
# Alpaca template
template = """### Instruction:
{instruction}

### Input:
{input}

### Response:
{output}"""

2. ShareGPT Format​

Multi-turn conversations:

{
  "conversations": [
    {"from": "system", "value": "You are a helpful assistant."},
    {"from": "human", "value": "What is Python?"},
    {"from": "gpt", "value": "Python is a programming language..."},
    {"from": "human", "value": "How do I install it?"},
    {"from": "gpt", "value": "You can download it from python.org..."}
  ]
}

3. Simple Completion​

Plain text for continued pre-training:

{"text": "Full document or paragraph of text..."}

Chat Templates​

Models expect specific formats. Use the tokenizer's template:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

messages = [
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi! How can I help?"},
]

text = tokenizer.apply_chat_template(messages, tokenize=False)
print(text)

Format Conversion​

def alpaca_to_messages(example):
    """Convert Alpaca format to messages."""
    messages = []
    
    if example.get("input"):
        user_content = f"{example['instruction']}\n\n{example['input']}"
    else:
        user_content = example["instruction"]
    
    messages.append({"role": "user", "content": user_content})
    messages.append({"role": "assistant", "content": example["output"]})
    
    return {"messages": messages}

# Apply to dataset
dataset = dataset.map(alpaca_to_messages)

Key Takeaways​

✅ Alpaca: Single-turn instructions ✅ ShareGPT: Multi-turn conversations ✅ Always use the model's chat template ✅ Consistency in formatting is critical

Continue to Chapter 2 →