Fine-tuning (LoRA, QLoRA, PEFT)

🎯 Goal

After reading this chapter:

You know the difference between fine-tuning and RAG and when to choose which
You can work with LoRA, QLoRA, PEFT (Parameter-Efficient Fine-Tuning)
You can fine-tune small LLMs with HuggingFace SFTTrainer
You use OpenAI/Anthropic fine-tuning APIs
You know how to prepare custom datasets and do synthetic data generation

What to learn

Full fine-tuning vs LoRA vs QLoRA vs Prompt tuning
PEFT library — HuggingFace
LoRA — Low-Rank Adaptation, mathematical intuition
QLoRA — 4-bit quantization + LoRA
Datasets — formats (chat, instruction, completion)
SFTTrainer — HuggingFace
Unsloth — 2-5x faster fine-tuning
Evaluation — perplexity, ROUGE, custom benchmarks
Cloud platforms — RunPod, Lambda Labs, Vast.ai
OpenAI / Anthropic fine-tuning APIs

Libraries

pip install transformers peft trl bitsandbytes accelerate datasets
pip install unsloth  # 2-5x faster

RAG vs Fine-tuning — when which?

Use case	RAG	Fine-tuning
Adding new knowledge	✅	❌
Teaching style/tone	❌	✅
Citation needed	✅	❌
Format consistency	Medium	✅
Latency optimization	❌	✅ (small model)
Domain-specific terms	✅	✅ (better)
Cost	Per-query	One-time + cheaper inference

**Rule:**RAG first; if insufficient — fine-tuning. In most cases RAG is enough.

Fine-tuning types

1. Full Fine-tuning

All parametersof the model are updated
Memory: ~40GB GPU for a 7B model
Speed: slow (days/weeks)
Quality: best (but overfitting risk)

2. LoRA (Low-Rank Adaptation)

Trains only small adapter matrices(≤1% parameters)
Memory: ~14GB GPU for a 7B model
Speed: fast (hours)
Quality: very close to full fine-tuning (95-99%)

Original matrix W (d × k)
↓
W ← W + ΔW
ΔW = A × B
A: d × r   (r << d)
B: r × k

Faqat A va B o'rganiladi. r=8, 16, 32, 64 odatda

3. QLoRA (Quantized LoRA)

LoRA + 4-bit quantization
Memory: ~6GB GPU for a 7B model (consumer GPU!)
Quality: equal to LoRA
The most recommended approach

4. Prompt Tuning / P-Tuning

Only soft prompt embeddings are learned
Smallest (<<1% params)
Quality: medium

5. Adapter Tuning

Adapter layers are added
Pre-LoRA approach

Code examples

Dataset preparation — Instruction format

# Format: instruction-following
data = [
    {
        "instruction": "Classify the following text by sentiment",
        "input": "This product is excellent!",
        "output": "positive",
    },
    {
        "instruction": "Find the error in this code",
        "input": "def foo(): print('hi'",
        "output": "Missing closing parenthesis on print() call",
    },
    # ... 1000+ examples
]

# Chat format (modern)
chat_data = [
    {
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "What is a list in Python?"},
            {"role": "assistant", "content": "A list in Python is ..."},
        ],
    },
    # ...
]

Fine-tuning with LoRA — HuggingFace

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
from datasets import load_dataset

# 1. Base model
model_name = "meta-llama/Llama-3.2-1B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# 2. LoRA config
lora_config = LoraConfig(
    r=16,                          # rank
    lora_alpha=32,                 # scaling factor
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 4M (0.4% of 1B) — very few!

# 3. Dataset
dataset = load_dataset("json", data_files="my_data.jsonl")

def format_prompt(example):
    return {
        "text": f"### Instruction:{example['instruction']}\n"
                f"### Input:{example['input']}\n"
                f"### Response:{example['output']}"
    }

dataset = dataset.map(format_prompt)

# 4. Training
training_args = TrainingArguments(
    output_dir="./llama-lora",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    warmup_steps=100,
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    dataset_text_field="text",
    max_seq_length=512,
    tokenizer=tokenizer,
)

trainer.train()

# 5. Save (adapter weights only)
model.save_pretrained("./llama-lora-adapter")

QLoRA — most efficient

from transformers import BitsAndBytesConfig

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

# Prepare for training
from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model)

# LoRA config (same as before)
model = get_peft_model(model, lora_config)

# Rest is same as LoRA

Unsloth — 2-5x faster

from unsloth import FastLanguageModel

# Auto: 4-bit quantization + LoRA + optimizations
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3.1-8b-bnb-4bit",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
)

# Training (same TRL API)
trainer = SFTTrainer(model=model, ...)
trainer.train()

# Inference
FastLanguageModel.for_inference(model)

Inference with LoRA adapter

from peft import PeftModel

# Base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Load adapter
model = PeftModel.from_pretrained(base_model, "./llama-lora-adapter")

# Generate
inputs = tokenizer("### Instruction: Hello\n### Response:", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))

Synthetic data generation (creating a dataset with an LLM)

from openai import AsyncOpenAI

async def generate_training_pair(topic: str) -> dict:
    """Create (instruction, response) pairs using an LLM."""
    
    prompt = f"""Create: 1 (question, answer) pair for teaching Python. Topic:{topic}JSON format:
{{
  "instruction": "...",
  "response": "..."
}}
"""
    
    response = await openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

# Generate 1000 synthetic examples
import asyncio

topics = ["list comprehension", "decorators", "async/await", ...] * 50
tasks = [generate_training_pair(t) for t in topics]
dataset = await asyncio.gather(*tasks)

# Save
with open("synthetic_data.jsonl", "w") as f:
    for item in dataset:
        f.write(json.dumps(item) + "\n")

OpenAI Fine-tuning API

from openai import OpenAI
client = OpenAI()

# 1. Upload file
file = client.files.create(
    file=open("data.jsonl", "rb"),
    purpose="fine-tune",
)

# 2. Start fine-tuning
job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-4o-mini-2024-07-18",
    hyperparameters={
        "n_epochs": 3,
        "batch_size": 4,
        "learning_rate_multiplier": 0.1,
    },
)

# 3. Monitor
job = client.fine_tuning.jobs.retrieve(job.id)
print(job.status)  # running → succeeded

# 4. Use fine-tuned model
response = client.chat.completions.create(
    model=f"ft:gpt-4o-mini-2024-07-18:my-org::{job.fine_tuned_model}",
    messages=[{"role": "user", "content": "Test"}],
)

Backend integration

Fine-tuned model serving (vLLM)

# vLLM — fastest LLM inference server
pip install vllm

# Start server
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.2-1B-Instruct \
    --enable-lora \
    --lora-modules my-adapter=./llama-lora-adapter \
    --port 8000

# OpenAI-compatible API
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy",
)

response = client.chat.completions.create(
    model="my-adapter",
    messages=[{"role": "user", "content": "Hello"}],
)

Training as a service (Celery + FastAPI)

@celery_app.task(bind=True)
def fine_tune_task(self, dataset_path: str, base_model: str, config: dict):
    # 1. Load dataset
    dataset = load_dataset("json", data_files=dataset_path)
    
    # 2. Setup model (QLoRA)
    model = setup_model_with_qlora(base_model)
    
    # 3. Training with progress updates
    trainer = SFTTrainer(...)
    
    class ProgressCallback(TrainerCallback):
        def on_log(self, args, state, control, logs=None, **kwargs):
            if logs:
                self.update_state(
                    state="PROGRESS",
                    meta={"step": state.global_step, "loss": logs.get("loss")}
                )
    
    trainer.add_callback(ProgressCallback())
    trainer.train()
    
    # 4. Save adapter
    output_path = f"models/{self.request.id}"
    model.save_pretrained(output_path)
    
    return {"model_path": output_path}

@app.post("/finetune")
async def start_finetuning(dataset_url: str, base_model: str = "llama-3.2-1b"):
    # Download dataset
    path = await download_dataset(dataset_url)
    
    # Queue task
    task = fine_tune_task.delay(path, base_model, {})
    return {"task_id": task.id}

Resources

HuggingFace PEFT docs — huggingface.co/docs/peft
TRL docs — huggingface.co/docs/trl
Unsloth GitHub — github.com/unslothai/unsloth
“QLoRA: Efficient Finetuning” — paper (Dettmers et al., 2023)
“The Novice’s LLM Training Guide” — Alpin Dale
OpenAI fine-tuning docs — platform.openai.com/docs/guides/fine-tuning
Maxime Labonne — LLM Course(github.com/mlabonne/llm-course)

🏋️ Exercises

🟢 Easy

Load pretrained Llama 3.2 1B in Colab GPU.
Create 50 synthetic instruction pairs (with GPT-4o-mini).
Read LoRA config syntax and explain the parameters.

🟡 Medium

TinyLlama fine-tuning: 100 examples, QLoRA, Colab T4 GPU.
OpenAI fine-tuning: Fine-tune GPT-4o-mini with a custom dataset (cost: ~$1).
Unsloth speedrun: Fine-tune Mistral-7B in 1 hour (Kaggle GPU).

🔴 Hard

Llama in Uzbek: 1000+ Uzbek instruction pairs, Llama 3.1 8B QLoRA — compare result to baseline.
DPO (Direct Preference Optimization): Preference tuning after SFT.
Production training pipeline: dataset versioning + training + evaluation + deployment.

Capstone

notebooks/month-05/08_finetuning.ipynb:

**Project:**Customer support bot in Uzbek
200+ (question, answer) pairs
Llama 3.2 1B or TinyLlama
QLoRA + Colab/Kaggle GPU
Inference deployment (FastAPI + vLLM)
Baseline vs fine-tuned comparison

✅ Checklist

RAG vs Fine-tuning when which
LoRA mathematical intuition
QLoRA — the most recommended approach
Working with PEFT library
Instruction dataset format
Training with SFTTrainer
Adapter weights save/load
Serving with vLLM
OpenAI fine-tuning API

Month 5 complete! Review the Exercises and proceed to Month 6 — MLOps and Production — the last and most important month.

Keyboard shortcuts

Backend to ML: 6 Oylik Roadmap