Classification

🎯 Goal

After reading this chapter:

• You can distinguish Classification from Regression tasks
• You know Logistic Regression, KNN, SVM, and Decision Tree algorithms
• You recognize the imbalanced data problem and know its solutions
• You correctly interpret Confusion matrix, Precision, Recall, F1, ROC-AUC
• You understand the difference between Binary and multi-class classification

What to learn

• Logistic Regression — named “regression” but used for classification
• K-Nearest Neighbors (KNN) — lazy learning
• Support Vector Machines (SVM) — kernel trick
• Decision Trees — tree of rules
• Naive Bayes — classic for text classification
• Imbalanced classes — SMOTE, class_weight, undersampling
• Multi-class strategies — OvR (One-vs-Rest), OvO (One-vs-One)
• Probability calibration — to make predict_proba reliable
• Threshold tuning — 0.5 is not always the best

Libraries

pip install scikit-learn imbalanced-learn

• scikit-learn — main models
• imbalanced-learn — SMOTE and other imbalance strategies

Important topics

Algorithm selection cheatsheet

Logistic Regression — how does it work?

1. Linear combination: z = w₀ + w₁x₁ +... + wₙxₙ
2. Sigmoid function: p = 1 / (1 + e^(-z)) → result in the (0, 1) range
3. Threshold: if p > 0.5, class 1, otherwise class 0

sigmoid(z):
   1 |        ___________
     |       /
   0.5|------/
     |     /
   0 |____/_____________
       -∞    0    +∞

Confusion Matrix

                 Predicted
                  0     1
Actual    0     [TN]  [FP]
          1     [FN]  [TP]

• **TP (True Positive):**correctly identified as 1
• **TN (True Negative):**correctly identified as 0
• **FP (False Positive):**incorrectly said 1 (Type I error)
• **FN (False Negative):**incorrectly said 0 (Type II error)

Metrics — which one when?

Real example — Precision vs Recall tradeoff

Cancer detectionmodel:

• Recall = 99% → 99% of sick patients are found
• Precision = 60% → 60% of those labeled “sick” are actually sick
• This is acceptable — not missing the sick is more important

Spam filter:

• Precision = 99% → 99% of those labeled as spam are actually spam
• Recall = 80% → 20% of spam slips through
• This is acceptable — important emails must not be lost

Imbalanced data problem

If 95% of data is class 0 and 5% is class 1, a model that always predicts 0gets 95% accuracy! But this is useless.

Solutions:

1. class_weight='balanced'(in sklearn models)
2. SMOTE — create synthetic minority samples (imbalanced-learn)
3. Undersampling — remove some from the majority class
4. Stratified sampling — ratio is preserved in train/test split
5. Threshold tuning — threshold below 0.5 (recall increases)
6. Other metrics — F1, PR-AUC instead of accuracy

Code examples

Logistic Regression — Breast Cancer

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    roc_auc_score, confusion_matrix, classification_report,
)

# 1. Data
data = load_breast_cancer(as_frame=True)
X, y = data.data, data.target  # 0 = malignant, 1 = benign

# 2. Split (stratify IMPORTANT!)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# 3. Pipeline
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000, random_state=42)),
])
pipe.fit(X_train, y_train)

# 4. Evaluation
y_pred = pipe.predict(X_test)
y_proba = pipe.predict_proba(X_test)[:, 1]

print(classification_report(y_test, y_pred, target_names=["malignant", "benign"]))
print(f"\nROC-AUC:{roc_auc_score(y_test, y_proba):.4f}")
print(f"Confusion Matrix:\n{confusion_matrix(y_test, y_pred)}")

Imbalanced data + class_weight

import numpy as np
from sklearn.linear_model import LogisticRegression

# Synthetic imbalanced data
from sklearn.datasets import make_classification
X, y = make_classification(
    n_samples=10_000, n_features=20, n_informative=10,
    weights=[0.95, 0.05], random_state=42,
)
# 95% class 0, 5% class 1

# Variant 1: default (accuracy = high, recall = low)
m1 = LogisticRegression(max_iter=1000).fit(X, y)

# Variant 2: class_weight balanced
m2 = LogisticRegression(max_iter=1000, class_weight="balanced").fit(X, y)

# Variant 3: manual weights
m3 = LogisticRegression(max_iter=1000, class_weight={0: 1, 1: 19}).fit(X, y)

Oversampling with SMOTE

from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline

# imblearn Pipeline (SMOTE doesn't work inside sklearn Pipeline!)
pipe = ImbPipeline([
    ("scaler", StandardScaler()),
    ("smote", SMOTE(random_state=42)),
    ("clf", LogisticRegression(max_iter=1000)),
])

pipe.fit(X_train, y_train)

Threshold tuning

import numpy as np

y_proba = pipe.predict_proba(X_test)[:, 1]

# Default threshold 0.5
y_pred_default = (y_proba >= 0.5).astype(int)

# Custom threshold for higher recall
y_pred_recall = (y_proba >= 0.3).astype(int)

# Optimal threshold (F1 maximizing)
from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_test, y_proba)
f1_scores = 2 * precisions * recalls / (precisions + recalls + 1e-9)
best_threshold = thresholds[np.argmax(f1_scores)]
print(f"Best threshold for F1:{best_threshold:.3f}")

Multi-class classification

from sklearn.datasets import load_digits
from sklearn.svm import SVC

X, y = load_digits(return_X_y=True)  # 10 classes (0..9)

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("svm", SVC(kernel="rbf", probability=True, random_state=42)),
])
pipe.fit(X_train, y_train)

# Multi-class metric
from sklearn.metrics import classification_report
print(classification_report(y_test, pipe.predict(X_test)))

Backend integration

Churn prediction API

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
import joblib
import numpy as np

app = FastAPI(title="Customer Churn Predictor")
model = joblib.load("models/churn_v1.joblib")

class CustomerFeatures(BaseModel):
    tenure_months: int = Field(..., ge=0)
    monthly_charges: float = Field(..., gt=0)
    total_charges: float = Field(..., ge=0)
    contract_type: int = Field(..., ge=0, le=2)  # 0=monthly, 1=1yr, 2=2yr
    has_internet: bool
    payment_method: int = Field(..., ge=0, le=3)

class ChurnPrediction(BaseModel):
    will_churn: bool
    churn_probability: float
    risk_level: str  # low / medium / high
    recommended_action: str

@app.post("/predict/churn", response_model=ChurnPrediction)
def predict_churn(customer: CustomerFeatures):
    X = np.array([list(customer.dict().values())])
    proba = float(model.predict_proba(X)[0, 1])
    
    # Custom business threshold
    if proba > 0.7:
        risk, action = "high", "immediate_retention_call"
    elif proba > 0.4:
        risk, action = "medium", "send_discount_offer"
    else:
        risk, action = "low", "monitor"
    
    return ChurnPrediction(
        will_churn=proba > 0.5,
        churn_probability=proba,
        risk_level=risk,
        recommended_action=action,
    )

Batch prediction endpoint

class BatchInput(BaseModel):
    customers: list[CustomerFeatures]

@app.post("/predict/churn/batch")
def predict_batch(payload: BatchInput):
    X = np.array([list(c.dict().values()) for c in payload.customers])
    probas = model.predict_proba(X)[:, 1]
    return {
        "predictions": [
            {"index": i, "churn_proba": float(p), "will_churn": bool(p > 0.5)}
            for i, p in enumerate(probas)
        ],
        "summary": {
            "total": len(probas),
            "at_risk": int((probas > 0.5).sum()),
            "high_risk": int((probas > 0.7).sum()),
        },
    }

Resources

• Scikit-learn Classification — scikit-learn.org/stable/supervised_learning.html
• StatQuest — Logistic Regression(YouTube playlist)
• Imbalanced-learn docs — imbalanced-learn.org
• Andrew Ng — Course 2: Advanced Learning Algorithms
• Article:“Beyond Accuracy: Precision and Recall” — Towards Data Science

🏋️ Exercises

🟢 Easy

1. Compare 4 classifiers (LogReg, KNN, SVM, Tree) on load_iris().
2. Plot a Confusion Matrix on the breast cancer dataset (ConfusionMatrixDisplay).
3. Try k values of [1, 3, 5, 10, 50] in KNN.

🟡 Medium

1. Imbalanced demo: Create 95/5 imbalanced data with make_classification. Compare precision/recall for default vs class_weight='balanced' vs SMOTE.
2. ROC curve: Plot ROC curves of 3 models in a single chart.
3. Threshold tuning: Find the F1-maximizing threshold on the Telco Churn dataset.

🔴 Hard

1. Production churn service: Complete churn prediction service with Docker + FastAPI + Postgres. /predict, /feedback (for returning real outcomes), /metrics (Prometheus) endpoints.
2. Online learning: Use SGDClassifier and partial_fit the model on each new feedback — adapt to drift.

Capstone

notebooks/month-02/02_classification_models.ipynb:

• Kaggle — Telco Customer Churn
• EDA → preprocessing → compare 5 classifiers
• Working with class imbalance
• Plotting ROC, PR curves
• Deploy the best model with Docker

✅ Checklist

• I know the difference between Classification and Regression
• I can read a Confusion Matrix
• I can explain Precision, Recall, F1 to business
• I know the difference between ROC-AUC and PR-AUC
• I know 3 strategies for imbalanced data
• I can distinguish predict_proba from predict
• I can tune results with a custom threshold
• I served a classification model with FastAPI

Moving on to Clustering.