RNN, LSTM, GRU

🎯 Goal

After reading this chapter:

• You will know NN architectures for working with sequence data (text, time series, audio)
• You will know the difference between RNN, LSTM, GRU and when to use each
• You will understand the vanishing gradient problem and the LSTM solution
• You will write time series forecasting and text classification
• You will be ready to move on to Transformers (in Month 4)

**Note:**The current era is the Transformers(BERT, GPT, T5) era. RNN/LSTM are becoming obsolete in many cases. But still useful for time series and important for NN history/intuition.

What to learn

• RNN — Recurrent Neural Network basics
• Vanishing/Exploding Gradientproblem
• LSTM — Long Short-Term Memory
• GRU — Gated Recurrent Unit
• Bidirectional RNN/LSTM
• Seq2Seq — encoder-decoder
• Attention mechanism(bridge to Transformers)
• Time series forecasting — sliding window approach
• Text classification with LSTM

Libraries

pip install torch torchtext pandas

Important topics

RNN — Recurrent Neural Network

Sequence: [x₁, x₂, x₃, ...]

   x₁              x₂              x₃
    │               │               │
    ▼               ▼               ▼
  [RNN] ──h₁──> [RNN] ──h₂──> [RNN] ──h₃──>
                                              
h_t = tanh(W_h · h_{t-1} + W_x · x_t + b)

The main idea: previous hidden state (h_{t-1}) together with current input produces a new state.

Vanishing Gradient problem

In long sequences, gradients pass through tanh repeatedly and approach zero — the model can’t learn long dependencies.

**Solution — LSTM:**store/forget information with special “gates”.

LSTM — full structure

                    cell state (C)
                    ──────────────►
                       ↑    ↑    ↑
                       │    │    │
                    [forget] [input] [output]
                       gate    gate    gate
                       │    │    │
                       └────┴────┘
                          ↑
                       h_t-1, x_t

3 gates:

• **Forget gate (f):**what to forget from cell state
• **Input gate (i):**what new to add
• **Output gate (o):**what the next hidden state will be

GRU — simplified LSTM

• 2 gates (reset, update)
• Faster than LSTM, fewer parameters
• Accuracy equal or close to LSTM

Which one when?

**Rule:**In new projects, start with transformers. Use RNN/LSTM only with a real reason (small dataset, real-time inference, simple time series).

Code examples

Simple RNN

import torch
import torch.nn as nn

class SimpleRNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x):
        # x shape: (batch, seq_len, input_size)
        out, hidden = self.rnn(x)
        # out shape: (batch, seq_len, hidden_size) # get the last timestep
        last_output = out[:, -1, :]
        logits = self.fc(last_output)
        return logits

model = SimpleRNN(input_size=10, hidden_size=64, num_classes=5)
x = torch.randn(32, 20, 10)  # batch=32, seq_len=20, features=10
print(model(x).shape)  # (32, 5)

LSTM — Time Series Forecasting

class LSTMForecaster(nn.Module):
    def __init__(self, input_size=1, hidden_size=64, num_layers=2, output_size=1):
        super().__init__()
        self.lstm = nn.LSTM(
            input_size, hidden_size,
            num_layers=num_layers,
            batch_first=True,
            dropout=0.2 if num_layers > 1 else 0,
        )
        self.fc = nn.Linear(hidden_size, output_size)
    
    def forward(self, x):
        # x shape: (batch, seq_len, input_size)
        out, (h_n, c_n) = self.lstm(x)
        # Last timestep
        last_output = out[:, -1, :]
        return self.fc(last_output)

Sliding window approach

def create_sequences(data, seq_length):
    """1D time series → (X, y) pairs."""
    X, y = [], []
    for i in range(len(data) - seq_length):
        X.append(data[i:i + seq_length])
        y.append(data[i + seq_length])
    return torch.tensor(X, dtype=torch.float32).unsqueeze(-1), torch.tensor(y, dtype=torch.float32)

# Example — sin function forecasting
import numpy as np
data = np.sin(np.linspace(0, 100, 1000))
X, y = create_sequences(data, seq_length=20)
# X shape: (980, 20, 1), y shape: (980,)

Text classification with LSTM

class TextClassifierLSTM(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes, n_layers=2):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(
            embed_dim, hidden_dim,
            num_layers=n_layers,
            batch_first=True,
            bidirectional=True,
            dropout=0.3,
        )
        # Bidirectional → hidden_dim * 2
        self.fc = nn.Linear(hidden_dim * 2, num_classes)
        self.dropout = nn.Dropout(0.5)
    
    def forward(self, x, lengths=None):
        # x shape: (batch, seq_len) — token IDs
        embedded = self.embedding(x)
        
        if lengths is not None:
            # For variable length sequences
            packed = nn.utils.rnn.pack_padded_sequence(
                embedded, lengths.cpu(), batch_first=True, enforce_sorted=False
            )
            _, (hidden, _) = self.lstm(packed)
        else:
            _, (hidden, _) = self.lstm(embedded)
        
        # Bidirectional final hidden: forward + backward
        hidden = torch.cat([hidden[-2], hidden[-1]], dim=1)
        hidden = self.dropout(hidden)
        return self.fc(hidden)

Training loop (for sequence data)

def train_sequence_model(model, train_loader, val_loader, epochs=20, lr=1e-3):
    device = next(model.parameters()).device
    optimizer = torch.optim.Adam(model.parameters(), lr=lr)
    criterion = nn.MSELoss()  # for forecasting; CrossEntropy for classification
    
    for epoch in range(epochs):
        model.train()
        train_loss = 0
        for X, y in train_loader:
            X, y = X.to(device), y.to(device)
            optimizer.zero_grad()
            
            pred = model(X)
            loss = criterion(pred.squeeze(), y)
            loss.backward()
            
            # IMPORTANT: gradient clipping for RNN
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
            
            optimizer.step()
            train_loss += loss.item()
        
        # Eval
        model.eval()
        val_loss = 0
        with torch.no_grad():
            for X, y in val_loader:
                X, y = X.to(device), y.to(device)
                pred = model(X)
                val_loss += criterion(pred.squeeze(), y).item()
        
        print(f"Epoch{epoch+1}: train={train_loss/len(train_loader):.4f}"
              f"val={val_loss/len(val_loader):.4f}")

Encoder-Decoder (Seq2Seq) preview

class Encoder(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
    
    def forward(self, x):
        _, (h, c) = self.lstm(x)
        return h, c  # context

class Decoder(nn.Module):
    def __init__(self, output_size, hidden_size):
        super().__init__()
        self.lstm = nn.LSTM(output_size, hidden_size, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)
    
    def forward(self, x, h, c):
        out, (h, c) = self.lstm(x, (h, c))
        return self.fc(out), h, c

# Seq2Seq: # encoder(input) → context # decoder(<START>, context) → output_1 # decoder(output_1, context) → output_2 # ...

Backend integration

Time series forecasting API

from fastapi import FastAPI
from pydantic import BaseModel
import torch
import numpy as np

app = FastAPI()
model = LSTMForecaster()
model.load_state_dict(torch.load("forecaster.pt"))
model.eval()

class ForecastInput(BaseModel):
    historical_values: list[float]
    forecast_steps: int = 7

class ForecastOutput(BaseModel):
    predictions: list[float]

@app.post("/forecast", response_model=ForecastOutput)
@torch.no_grad()
def forecast(data: ForecastInput):
    # Last 20 values as input
    history = torch.tensor(data.historical_values[-20:], dtype=torch.float32)
    history = history.unsqueeze(0).unsqueeze(-1)  # (1, 20, 1)
    
    predictions = []
    for _ in range(data.forecast_steps):
        pred = model(history).item()
        predictions.append(pred)
        # Slide window: drop first, append prediction
        history = torch.cat([history[:, 1:, :], torch.tensor([[[pred]]])], dim=1)
    
    return ForecastOutput(predictions=predictions)

Text sentiment API (LSTM)

@app.post("/sentiment")
@torch.no_grad()
def predict_sentiment(text: str):
    tokens = tokenizer(text, max_length=200, padding="max_length", truncation=True)
    X = torch.tensor([tokens]).long()
    
    logits = model(X)
    probs = torch.softmax(logits, dim=1).squeeze()
    
    labels = ["negative", "neutral", "positive"]
    return {
        "sentiment": labels[probs.argmax().item()],
        "scores": {label: float(p) for label, p in zip(labels, probs)},
    }

**Note:**For production sentiment, using HuggingFace BERTgives much better results. LSTM here is just an example.

Resources

• Andrej Karpathy — “The Unreasonable Effectiveness of RNNs”(blog)
• Colah’s blog — Understanding LSTMs(colah.github.io/posts/2015-08-Understanding-LSTMs)
• PyTorch Sequence tutorials
• “Deep Learning for Time Series Forecasting” — Jason Brownlee
• fast.ai NLP course(RNN and beyond)

🏋️ Exercises

🟢 Easy

1. Compare the number of parameters of nn.RNN, nn.LSTM, nn.GRU.
2. Next-step forecasting on sinusoidal data with LSTM.
3. Compare results of Bidirectional LSTM and unidirectional.

🟡 Medium

1. Time series: 30-day forecasting with real stock price (yfinance) data.
2. Text classification: Binary sentiment on the IMDB reviews dataset with LSTM (80%+).
3. Char-level RNN: Character-level text generation Karpathy-style.

🔴 Hard

1. Seq2Seq translation — launch on a small language (English ↔ German small dataset).
2. Attention mechanism — add attention on top of LSTM (intro to transformers).
3. Time series API — compare Prophet vs LSTM, deploy the best with FastAPI.

Capstone

notebooks/month-03/06_rnn_timeseries.ipynb:

• Load a stock price via yfinance(5-year)
• Classical baseline: Prophet, ARIMA
• Your LSTM model
• Compare forecasting accuracy on the test set
• FastAPI endpoint

✅ Checklist

• I know the difference between RNN, LSTM, GRU
• I understand the vanishing gradient problem
• I know the role of LSTM gates
• Difference between Bidirectional and unidirectional
• I can prepare data for time series with sliding window approach
• I know why gradient clipping is important in RNN
• Text classification with LSTM
• I know that Transformers (Month 4) surpass RNN and why

Month 3 complete! Review the Exercises and proceed to Month 4 — CV + NLP.