LLM fundamentals

🎯 Goal

After reading this chapter:

• You will understand what an LLM is and how it works
• You will know the difference between model types (proprietary vs open source)
• You will correctly use terms like token, context window, temperature
• You will know models’ strengths/weaknesses and choose correctly

What to learn

• LLM architecture — Transformer decoder
• Training stages — pretraining, SFT, RLHF
• Token and tokenization — how to count, how to price
• Context window — evolution from 4K → 1M+
• Temperature, top_p, top_k — sampling parameters
• Hallucination — what it is and how to reduce it
• Model families — GPT (OpenAI), Claude (Anthropic), Gemini (Google), Llama (Meta), Mistral, Qwen, DeepSeek

Important topics

How LLMs work (simple)

Input:  "Today the weather is very"
              ↓
        LLM (50B parameters)
              ↓
Output: probability distribution next token
        "nice" 0.45
        "cold" 0.20
        "warm" 0.15
        ...
              ↓
        Sampling (temperature)
              ↓
        "nice"

Then "Today the weather is very nice" → next token, and so on.

An LLM is a next-token predictor. It predicts one next token at a time.

What is a token?

"Hello world!" → ["Hel", "lo", " wor", "ld", "!"]
                  ~5 tokens (approximate, depending on model)

GPT-4 pricing:
- Input: $2.50 / 1M tokens
- Output: $10 / 1M tokens

Our chatbot $0.001 / message (approximate, with GPT-4o-mini)

Token cost by language:

• English — cheapest (1 word ≈ 1.3 tokens)
• Uzbek/Russian — more expensive (1 word ≈ 2-3 tokens)
• Chinese — several tokens per character

Context Window

How many tokens the model can “see” at once:

Goes into context window:

• System prompt
• Historical messages
• User input
• LLM response (output)

Together, input + output must be smaller than the context window.

Training stages

1. Pretraining (foundation)
   - Trillions of tokens (internet, books)
   - Next-token prediction
   - Result: "base model" — can do completion

2. SFT (Supervised Fine-Tuning)
   - High-quality (prompt, response) pairs
   - Instruction following
   - Result: "instruct model"

3. RLHF (Reinforcement Learning from Human Feedback)
   - Based on human preferences
   - Better, more helpful, safer
   - Result: "chat model" (production-ready)

Temperature and sampling

# temperature=0.0 — deterministic (same prompt → same answer) # temperature=1.0 — default, balanced # temperature=2.0 — chaotic, creative

# top_p (nucleus sampling) # top_p=1.0 — choose from all # top_p=0.9 — choose from top 90% cumulative probability

# top_k # top_k=50 — only choose from top 50 tokens

When which?

Hallucination

An LLM can give incorrect answers in a confident-looking way:

• “Toshkent metro has 24 stations” (actually 30+)
• “Python has a dict.merge() method” (no, dict | dict or .update())

Reasons:

1. Training data is outdated or incorrect
2. Internal knowledge is limited
3. The model doesn’t admit “not knowing”

Solutions:

1. RAG — providing context from real data
2. Tool use — calculator, search, DB query
3. Prompt engineering — “If you don’t know, say ‘I don’t know’”
4. Citation — show where the answer came from

Proprietary vs Open Source

Model families (2024-2026)

OpenAI

• GPT-4o — multimodal (image, audio, text)
• GPT-4o-mini — cheapest flagship
• o1, o3 — reasoning models (math, code)

Anthropic

• Claude Opus 4.7 — most powerful, 1M context (extended)
• Claude Sonnet 4.6 — balanced (speed/cost/quality)
• Claude Haiku 4.5 — fastest and cheapest

Google

• Gemini 2.5 Pro — 1M context
• Gemini 2.5 Flash — fast and cheap
• Gemma 2 — open weights

Meta (open)

• Llama 3.1 — 8B, 70B, 405B
• Llama 3.2 — multimodal versions

Others (open)

• Mistral / Mixtral — Europe (MoE architecture)
• Qwen 2.5 — Alibaba (strong multilingual)
• DeepSeek V3 — strong reasoning model

Code examples (intro)

Token counting

import tiktoken

# For OpenAI
enc = tiktoken.encoding_for_model("gpt-4o")
text = "Hello world, machine learning is interesting"
tokens = enc.encode(text)
print(f"Token count:{len(tokens)}")
print(f"Tokens:{tokens}")

# Approximate (for other models) # 1 token ≈ 4 chars (English), ≈ 2 chars (uzbek/Russian)
def estimate_tokens(text: str) -> int:
    return len(text) // 3  # rough

Context window monitoring

class ConversationManager:
    def __init__(self, max_tokens: int = 100_000):
        self.max_tokens = max_tokens
        self.messages = []
        self.enc = tiktoken.encoding_for_model("gpt-4o")
    
    def add_message(self, role: str, content: str):
        self.messages.append({"role": role, "content": content})
        self._truncate_if_needed()
    
    def _count_tokens(self) -> int:
        return sum(len(self.enc.encode(m["content"])) for m in self.messages)
    
    def _truncate_if_needed(self):
        """Keep the system message, remove old ones."""
        while self._count_tokens() > self.max_tokens and len(self.messages) > 2:
            # Keep the system message (index 0)
            self.messages.pop(1)

Cost calculator

PRICES = {
    "gpt-4o": {"input": 2.50, "output": 10.00},          # $ per 1M tokens "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "claude-opus-4-7": {"input": 15.00, "output": 75.00},
    "claude-sonnet-4-6": {"input": 3.00, "output": 15.00},
    "claude-haiku-4-5": {"input": 0.80, "output": 4.00},
}

def calculate_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    p = PRICES[model]
    return (input_tokens * p["input"] + output_tokens * p["output"]) / 1_000_000

Backend integration (preview)

Details in the next chapters. Mental model:

User → FastAPI → LLM API → Response
              ↓
           PostgreSQL (history)
              ↓
           Redis (caching)
              ↓
           Sentry / Datadog (observability)

An LLM API call is an HTTP request, just to the AI side. For the backend, you can already:

• Work with retry logic
• Timeout and circuit breaker
• Rate limiting
• Async (async def in FastAPI)
• Streaming responses (SSE or WebSocket)

Resources

• Andrej Karpathy — “Intro to LLMs”(YouTube, 1 hour) — MUST WATCH
• 3Blue1Brown — “But what is GPT?”(visual explanation)
• Anthropic Cookbook — github.com/anthropics/anthropic-cookbook
• OpenAI Cookbook — cookbook.openai.com
• “Hands-On Large Language Models” — Jay Alammar and Maarten Grootendorst (O’Reilly, 2024)
• Hugging Face NLP Course (LLM section)
• Latent Space Podcast — industry trends

🏋️ Exercises

🟢 Easy

1. With tiktoken, compare token counts in multiple English and Uzbek texts.
2. List context windows of different models.
3. Send a question to a chosen LLM with 3 different temperatures (0, 0.5, 1.5), compare the answers.

🟡 Medium

1. Conversation manager: A class that stores historical messages and ensures they don’t exceed the context window.
2. Cost tracker: Log every LLM call, output daily/monthly cost analysis.
3. Model comparison: Send the same 20 questions to GPT-4o-mini, Claude Haiku, Llama 3.1 8B, compare in terms of quality and time.

🔴 Hard

1. LLM Router: A service that automatically chooses the cheapest and best-quality model based on input (simple question → Haiku, complex → Sonnet, code → Opus).
2. Token budget manager: Monthly quota system for users (FastAPI + Redis + Postgres).

Capstone

notebooks/month-05/01_llm_fundamentals.ipynb:

• 5 different LLMs (GPT-4o-mini, Claude Haiku, Gemini Flash, Llama 3.1, Mistral) — same 10 questions
• For each: answer, time, token count, cost
• Create a Markdown report

✅ Checklist

• I understand LLM next-token prediction
• What is a token and context window
• I know the Pretraining → SFT → RLHF process
• I know the difference between Temperature, top_p, top_k
• What is hallucination and how to reduce it (RAG, tools)
• I know the difference between Proprietary and Open Source LLMs
• I know the main model families (GPT, Claude, Gemini, Llama)
• I can write a cost calculator

Moving on to Prompt Engineering.