Neural Network foundations

🎯 Goal

After reading this chapter:

• You will understand what a neural network is and how it’s built
• You will know perceptron, MLP, activation functions, and loss functions
• You will understand Forward pass and Backpropagation algorithms
• You will know Gradient Descent and its variants
• You will be able to write a simple NN with pure NumPy

What to learn

• Perceptron — the simplest “neuron”
• Multi-Layer Perceptron (MLP) — deeper
• Activation functions — ReLU, Sigmoid, Tanh, Softmax
• Loss functions — MSE, CrossEntropy, Binary CrossEntropy
• Forward pass — input → output path
• Backpropagation — computing gradients
• Gradient Descentvariants — SGD, Momentum, Adam, RMSprop
• Universal Approximation Theorem — why NNs work

Libraries

pip install numpy matplotlib torch torchvision

Important topics

Perceptron — the simplest neuron

input  ─x₁──┐
input  ─x₂──┤── (weighted sum) ── activation ── output
input  ─x₃──┘
            ↑
           bias

z = w₁x₁ + w₂x₂ + w₃x₃ + b
a = activation(z)

Activation functions — why are they needed?

Without activation, the entire NN is one big linear regression. Activation adds nonlinearity.

MLP architecture

Input Layer       Hidden Layer 1     Hidden Layer 2    Output Layer
    [x₁]                [h₁₁]               [h₂₁]
    [x₂]    ──W₁,b₁──>  [h₁₂]   ──W₂,b₂──> [h₂₂]   ──W₃,b₃──> [y]
    [x₃]                [h₁₃]               [h₂₃]
    [x₄]                [h₁₄]

input shape: (n,)
W₁ shape: (hidden_1, n)
W₂ shape: (hidden_2, hidden_1)
W₃ shape: (1, hidden_2)

Loss functions

Backpropagation — propagating gradients “backwards”

Forward:  input → ... → output → loss
                                  │
Backward: ∂loss/∂w ← ... ← ∂loss/∂a ← ─┘

Chain rule:
∂L/∂w = ∂L/∂a × ∂a/∂z × ∂z/∂w

In PyTorch/TensorFlow this is automatic(autograd). But the intuition is important to know.

Optimizers

**Recommendation:**Start with Adam or AdamW. Try others when it’s time to tune.

Code examples

MLP with Pure NumPy (for intuition)

import numpy as np

class SimpleMLP:
    def __init__(self, input_size, hidden_size, output_size):
        # Xavier initialization
        self.W1 = np.random.randn(hidden_size, input_size) * np.sqrt(2 / input_size)
        self.b1 = np.zeros(hidden_size)
        self.W2 = np.random.randn(output_size, hidden_size) * np.sqrt(2 / hidden_size)
        self.b2 = np.zeros(output_size)
    
    def relu(self, x):
        return np.maximum(0, x)
    
    def relu_derivative(self, x):
        return (x > 0).astype(float)
    
    def softmax(self, x):
        # Numerical stability
        exp_x = np.exp(x - np.max(x, axis=1, keepdims=True))
        return exp_x / exp_x.sum(axis=1, keepdims=True)
    
    def forward(self, X):
        # X shape: (batch, input_size)
        self.z1 = X @ self.W1.T + self.b1
        self.a1 = self.relu(self.z1)
        self.z2 = self.a1 @ self.W2.T + self.b2
        self.a2 = self.softmax(self.z2)
        return self.a2
    
    def backward(self, X, y_true, learning_rate=0.01):
        # y_true: one-hot encoded
        batch_size = X.shape[0]
        
        # Output layer gradient
        dz2 = (self.a2 - y_true) / batch_size
        dW2 = dz2.T @ self.a1
        db2 = dz2.sum(axis=0)
        
        # Hidden layer gradient
        da1 = dz2 @ self.W2
        dz1 = da1 * self.relu_derivative(self.z1)
        dW1 = dz1.T @ X
        db1 = dz1.sum(axis=0)
        
        # Update weights
        self.W1 -= learning_rate * dW1
        self.b1 -= learning_rate * db1
        self.W2 -= learning_rate * dW2
        self.b2 -= learning_rate * db2
    
    def train(self, X, y, epochs=100, batch_size=32, learning_rate=0.01):
        for epoch in range(epochs):
            # Mini-batch
            indices = np.random.permutation(len(X))
            for start in range(0, len(X), batch_size):
                batch_idx = indices[start:start + batch_size]
                X_batch = X[batch_idx]
                y_batch = y[batch_idx]
                
                self.forward(X_batch)
                self.backward(X_batch, y_batch, learning_rate)
            
            # Track loss
            y_pred = self.forward(X)
            loss = -np.mean(np.sum(y * np.log(y_pred + 1e-9), axis=1))
            if epoch % 10 == 0:
                print(f"Epoch{epoch}: Loss ={loss:.4f}")

# Example — Iris (3-class)
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)
X = StandardScaler().fit_transform(X)
y_onehot = np.eye(3)[y]

model = SimpleMLP(input_size=4, hidden_size=16, output_size=3)
model.train(X, y_onehot, epochs=200, batch_size=16, learning_rate=0.05)

The same thing in PyTorch — DRAMATICALLY simpler

import torch
import torch.nn as nn
import torch.optim as optim

class SimpleMLP(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, output_size)
    
    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.fc2(x)
        return x  # logits (softmax inside loss)

model = SimpleMLP(4, 16, 3)
optimizer = optim.Adam(model.parameters(), lr=0.05)
loss_fn = nn.CrossEntropyLoss()

X_t = torch.tensor(X, dtype=torch.float32)
y_t = torch.tensor(y, dtype=torch.long)

for epoch in range(200):
    optimizer.zero_grad()
    logits = model(X_t)
    loss = loss_fn(logits, y_t)
    loss.backward()           # backprop automatic
    optimizer.step()
    
    if epoch % 20 == 0:
        accuracy = (logits.argmax(dim=1) == y_t).float().mean()
        print(f"Epoch{epoch}: Loss={loss.item():.4f}, Acc={accuracy.item():.4f}")

**Note:**Same result — pure NumPy 60 lines, PyTorch 20 lines. Productivity = framework.

Backend integration

For now (basics) — this chapter is theoretical. Production deployment is detailed in the PyTorch chapter and Month 6 MLOps.

But the mental model: A NN is a mathematical function. As a backend dev, you always:

• input → output (a REST API is the same way)
• Stateless (weights are function parameters)
• Versioning is needed (model_v1.pt, model_v2.pt)
• Monitoring (latency, prediction distribution)

Resources

• 3Blue1Brown — Neural Networks playlist(YouTube) — visual, MUST WATCH
• Andrew Ng — Deep Learning Specialization (Course 1) — theoretical foundations
• “Neural Networks and Deep Learning” — Michael Nielsen (free: neuralnetworksanddeeplearning.com)
• Andrej Karpathy — “Neural Networks: Zero to Hero”(YouTube) — strong hands-on course
• fast.ai — Practical Deep Learning(free course)

🏋️ Exercises

🟢 Easy

1. Plot Sigmoid, ReLU, Tanh functions with Matplotlib.
2. For the 2x + 1 linear function, find w and b that minimize MSE loss using gradient descent.
3. Create nn.Linear(10, 1) in PyTorch and try a forward pass on a random tensor.

🟡 Medium

1. NumPy MLP: Tune the above code to 90%+ accuracy on the Iris dataset.
2. XOR problem: Solve the XOR problem with a 2-layer MLP.
3. PyTorch vs Numpy speed: Compare training time for a 1M parameter model.

🔴 Hard

1. From-scratch backprop — 3 hidden layer MLP with dropout, batchnorm — all in pure NumPy.
2. Visualize: Plot the PyTorch model’s loss landscape (3D plot over 2 weights).

Capstone

notebooks/month-03/01_neural_network_scratch.ipynb:

• Write a 2-layer MLP with Numpy
• Train on a small sample of MNIST (1000 samples, 10 classes)
• Write the same thing in PyTorch
• Compare accuracy and training time

✅ Checklist

• I know the difference between Perceptron and MLP
• I know when to use ReLU, Sigmoid, Softmax
• I understand Forward pass and Backprop intuition
• I know the difference between Gradient Descent, SGD, Adam
• I know when to use CrossEntropy vs MSE
• I write a simple nn.Module in PyTorch
• I know how to build a simple MLP with pure NumPy

Moving on to PyTorch basics.