Regression

🎯 Goal

After reading this chapter:

• You will know what a Regression task is and when to use it
• You will understand the difference between Linear, Polynomial, Ridge, Lasso, and ElasticNet
• You will correctly interpret regression metrics (RMSE, MAE, R²)
• You will build a regression model on a real dataset and serve it with FastAPI

What to learn

• Linear Regression — the most fundamental algorithm, every ML engineer knows it
• Polynomial Regression — subtle curves
• Regularization — Ridge (L2), Lasso (L1), ElasticNet (L1+L2)
• Effect of feature scalingon regression
• Multicollinearity — when features are correlated with each other
• Assumptions — linearity, normality, homoscedasticity (at a basic level)
• Metrics — MSE, RMSE, MAE, R², MAPE
• Robust regression — when outliers are present

Libraries

pip install scikit-learn statsmodels

• scikit-learn — main
• statsmodels — if statistical details (p-value, confidence interval) are needed

Important topics

Linear Regression intuition

Goal — find a line of the form y = w₀ + w₁x₁ + w₂x₂ +... + wₙxₙ:

• As close as possible to the actual values (y_true)
• “Closeness” measure — typically MSE(Mean Squared Error)

Optimization: **Ordinary Least Squares (OLS)**or Gradient Descent.

Why is regularization needed?

If there are many features (often more than observations) or they are correlated with each other, the model overfits. Solution — regularization:

• Ridge (L2):loss + λ * Σwᵢ² — shrinks features toward zero
• Lasso (L1):loss + λ * Σ|wᵢ| — makes some features exactly zero(feature selection)
• **ElasticNet:**a mixture of both

λ kichik (0)         λ o'rta              λ katta
Overfitting          Optimal               Underfitting
(model murakkab)                          (model oddiy)

Linear assumptions

1. Linearity — is the relationship between y and X actually linear?
2. Independence — observations are independent (this is violated for time series)
3. Homoscedasticity — error variance is constant (check with residual plot)
4. Normality — errors are normally distributed (Q-Q plot)
5. No multicollinearity — features are not too correlated with each other (VIF)

**Backend dev tip:**We don’t always need to check these assumptions for business use — random forest or XGBoost work without them. But useful for getting nice results with Linear Regression.

Metrics — which one when?

Code examples

Linear Regression — California Housing

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
import numpy as np

# 1. Data
data = fetch_california_housing(as_frame=True)
X, y = data.data, data.target  # y = median house price ($100k)

# 2. Split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# 3. Pipeline
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("lr", LinearRegression()),
])
pipeline.fit(X_train, y_train)

# 4. Predict and metrics
y_pred = pipeline.predict(X_test)

rmse = np.sqrt(mean_squared_error(y_test, y_pred))
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f"RMSE:{rmse:.3f}")  # 0.745
print(f"MAE:{mae:.3f}")   # 0.533
print(f"R²:{r2:.3f}")    # 0.576

# 5. Coefficients
coefs = dict(zip(X.columns, pipeline.named_steps["lr"].coef_))
for name, c in sorted(coefs.items(), key=lambda x: abs(x[1]), reverse=True):
    print(f"  {name}:{c:+.3f}")

Ridge, Lasso, ElasticNet

from sklearn.linear_model import Ridge, Lasso, ElasticNet

models = {
    "LinearRegression": LinearRegression(),
    "Ridge (L2)":       Ridge(alpha=1.0, random_state=42),
    "Lasso (L1)":       Lasso(alpha=0.01, random_state=42),
    "ElasticNet":       ElasticNet(alpha=0.01, l1_ratio=0.5, random_state=42),
}

for name, model in models.items():
    pipe = Pipeline([("scaler", StandardScaler()), ("model", model)])
    pipe.fit(X_train, y_train)
    score = pipe.score(X_test, y_test)  # R²
    print(f"{name:20s}R² ={score:.4f}")

Polynomial Regression

from sklearn.preprocessing import PolynomialFeatures

poly_pipeline = Pipeline([
    ("poly", PolynomialFeatures(degree=2, include_bias=False)),
    ("scaler", StandardScaler()),
    ("lr", LinearRegression()),
])
poly_pipeline.fit(X_train, y_train)
print(f"Polynomial R²:{poly_pipeline.score(X_test, y_test):.4f}")

Hyperparameter tuning — GridSearchCV

from sklearn.model_selection import GridSearchCV

ridge_pipe = Pipeline([("scaler", StandardScaler()), ("ridge", Ridge())])

param_grid = {"ridge__alpha": [0.01, 0.1, 1.0, 10.0, 100.0]}

gs = GridSearchCV(ridge_pipe, param_grid, cv=5, scoring="neg_root_mean_squared_error")
gs.fit(X_train, y_train)

print(f"Best alpha:{gs.best_params_['ridge__alpha']}")
print(f"Best CV RMSE:{-gs.best_score_:.3f}")

Backend integration

Price prediction API (FastAPI)

from fastapi import FastAPI
from pydantic import BaseModel, Field
import joblib
import numpy as np
from contextlib import asynccontextmanager

@asynccontextmanager
async def lifespan(app):
    app.state.model = joblib.load("models/california_housing_v1.joblib")
    yield

app = FastAPI(lifespan=lifespan, title="California Housing Price Predictor")

class HouseFeatures(BaseModel):
    MedInc: float = Field(..., gt=0, description="Median income (10k USD)")
    HouseAge: float = Field(..., ge=0, le=100)
    AveRooms: float = Field(..., gt=0)
    AveBedrms: float = Field(..., gt=0)
    Population: float = Field(..., gt=0)
    AveOccup: float = Field(..., gt=0)
    Latitude: float
    Longitude: float

class PricePrediction(BaseModel):
    predicted_price_100k: float
    predicted_price_usd: float

@app.post("/predict/", response_model=PricePrediction)
def predict_price(features: HouseFeatures):
    X = np.array([[
        features.MedInc, features.HouseAge, features.AveRooms,
        features.AveBedrms, features.Population, features.AveOccup,
        features.Latitude, features.Longitude,
    ]])
    pred = float(app.state.model.predict(X)[0])
    return PricePrediction(
        predicted_price_100k=pred,
        predicted_price_usd=pred * 100_000,
    )

Logging and monitoring (basic)

import logging
from datetime import datetime

logger = logging.getLogger("ml_service")

@app.post("/predict/", response_model=PricePrediction)
def predict_price(features: HouseFeatures):
    start = datetime.now()
    X = np.array([list(features.dict().values())])
    pred = float(app.state.model.predict(X)[0])
    duration_ms = (datetime.now() - start).total_seconds() * 1000
    
    logger.info(
        "prediction",
        extra={
            "input": features.dict(),
            "prediction": pred,
            "duration_ms": duration_ms,
            "model_version": "v1",
        },
    )
    return PricePrediction(predicted_price_100k=pred, predicted_price_usd=pred * 100_000)

Resources

• Scikit-learn Regression — scikit-learn.org/stable/supervised_learning.html#regression
• StatQuest — Linear Regression(YouTube)
• StatQuest — Ridge, Lasso, ElasticNet(3 separate videos)
• “Introduction to Statistical Learning”(ISLR) — free PDF, deep on regression
• Andrew Ng — ML Specialization Course 1(Linear Regression module)

🏋️ Exercises

🟢 Easy

1. Train Linear Regression on sklearn.datasets.load_diabetes(), output R².
2. Try alpha = [0.001, 0.01, 0.1, 1, 10, 100] in Ridge and plot how R² changes.
3. Compare train and test R² — when is overfitting happening?

🟡 Medium

1. California Housing: Compare all — Linear, Ridge, Lasso, ElasticNet, Polynomial — in table form.
2. Manual gradient descent: Write Linear Regression yourself with numpy (without using sklearn).
3. Residual analysis: Visualize y_test - y_pred. Is there a pattern? (homoscedasticity check)

🔴 Hard

1. Production service: California Housing model with Docker + FastAPI + Postgres (for predictions log). Healthcheck, Prometheus metrics (request_count, prediction_duration).
2. A/B test infra: serve two models at the same time (v1 and v2), split traffic 50/50, collect separate metrics for each.

Capstone

notebooks/month-02/01_regression.ipynb:

• Kaggle — House Prices: Advanced Regression Techniquescompetition
• Submit for the first time
• Goal: top 50% (RMSE log <= 0.16)
• Steps: EDA → preprocessing → baseline with Ridge → feature engineering → feature selection with Lasso → submission

✅ Checklist

• I understand the mathematical formula of Linear Regression (y = wx + b)
• I know the difference between Ridge and Lasso (L1 vs L2)
• I know when to use RMSE vs MAE
• I can explain the meaning of R² to business
• I can create a Pipeline (scaler + model)
• I tune hyperparameters with GridSearchCV
• I served a regression model with FastAPI
• I made my first Kaggle submission

Moving on to Classification.