Ensemble Methods (XGBoost, LightGBM, CatBoost)

🎯 Цель

После прочтения этой главы:

• Поймёте разницу между ensemble methods (Bagging, Boosting, Stacking)
• Узнаете, когда использовать Random Forest, XGBoost, LightGBM, CatBoost
• Сможете получать хорошие результаты в Kaggle competition на tabular data
• Сможете делать deploy алгоритмов Gradient Boosting в production

Что нужно изучить

• Bagging: Random Forest, Extra Trees
• Boosting: AdaBoost, Gradient Boosting, XGBoost, LightGBM, CatBoost
• Stacking: meta-learner, blending
• Voting: hard vs soft voting
• Feature importance — Gini, permutation, SHAP
• Hyperparameter tuning — специально для XGBoost/LightGBM
• Early stopping — предотвращение overfitting
• Categorical handling — преимущество CatBoost

Библиотеки

pip install scikit-learn xgboost lightgbm catboost shap

Важные темы

Bagging vs Boosting

Bagging (Bootstrap Aggregating):
- Parallel: har model bir-biriga bog'liq emas
- Random Forest = Bagging + Decision Trees + random features
- Maqsad: variance kamaytirish (overfitting'ni)

Boosting:
- Sequential: har model oldingisining xatolarini tuzatishga harakat qiladi
- Gradient Boosting, XGBoost, LightGBM, CatBoost
- Maqsad: bias kamaytirish (underfitting'ni)

Алгоритм Gradient Boosting

1. F₀(x) = mean(y)  ← boshlang'ich bashorat
2. Repeat (M ta marta):
   a. r_i = y_i - F_{m-1}(x_i)   ← residual (xato)
   b. Yangi tree h_m(x) — r ni bashorat qilish uchun
   c. F_m(x) = F_{m-1}(x) + learning_rate * h_m(x)
3. Final: F_M(x)

Какой gradient boosting?

**Совет:**сначала LightGBM(быстрый), затем XGBoost(стабильный), наконец CatBoost(если много categorical).

Самые важные hyperparameters (LightGBM/XGBoost)

Early Stopping

# Training останавливается, если validation loss не улучшается
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    early_stopping_rounds=50,
)
# Сохраняет best_iteration

Примеры кода

Random Forest

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

rf = RandomForestClassifier(
    n_estimators=500,
    max_depth=None,
    min_samples_split=2,
    min_samples_leaf=1,
    max_features="sqrt",
    n_jobs=-1,
    random_state=42,
    class_weight="balanced",
)

scores = cross_val_score(rf, X, y, cv=5, scoring="f1")
print(f"F1: {scores.mean():.3f} ± {scores.std():.3f}")

# Feature importance
rf.fit(X, y)
importance_df = pd.DataFrame({
    "feature": X.columns,
    "importance": rf.feature_importances_,
}).sort_values("importance", ascending=False).head(10)
print(importance_df)

XGBoost

import xgboost as xgb
from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

model = xgb.XGBClassifier(
    n_estimators=1000,
    learning_rate=0.05,
    max_depth=6,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_alpha=0.1,
    reg_lambda=1.0,
    random_state=42,
    eval_metric="logloss",
    early_stopping_rounds=50,
)

model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    verbose=False,
)

print(f"Best iteration: {model.best_iteration}")
print(f"Best score: {model.best_score:.4f}")

LightGBM

import lightgbm as lgb

model = lgb.LGBMClassifier(
    n_estimators=2000,
    learning_rate=0.05,
    num_leaves=63,
    max_depth=-1,
    min_child_samples=20,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_alpha=0.1,
    reg_lambda=1.0,
    random_state=42,
    n_jobs=-1,
    class_weight="balanced",
)

model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    eval_metric="auc",
    callbacks=[lgb.early_stopping(50), lgb.log_evaluation(0)],
)

print(f"Best iter: {model.best_iteration_}")

CatBoost — Categorical Magic

from catboost import CatBoostClassifier

cat_features = ["city", "department", "education"]  # имена столбцов

model = CatBoostClassifier(
    iterations=1000,
    learning_rate=0.05,
    depth=6,
    l2_leaf_reg=3,
    cat_features=cat_features,  # automatic handling!
    early_stopping_rounds=50,
    random_seed=42,
    verbose=100,
)

model.fit(X_train, y_train, eval_set=(X_val, y_val))

Optuna tuning (для LightGBM)

import optuna
import lightgbm as lgb

def objective(trial):
    params = {
        "objective": "binary",
        "metric": "auc",
        "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
        "num_leaves": trial.suggest_int("num_leaves", 15, 256),
        "max_depth": trial.suggest_int("max_depth", 3, 15),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
        "subsample": trial.suggest_float("subsample", 0.5, 1.0),
        "colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 1.0),
        "reg_alpha": trial.suggest_float("reg_alpha", 1e-3, 10, log=True),
        "reg_lambda": trial.suggest_float("reg_lambda", 1e-3, 10, log=True),
        "n_estimators": 2000,
        "random_state": 42,
        "n_jobs": -1,
        "verbose": -1,
    }
    
    model = lgb.LGBMClassifier(**params)
    model.fit(
        X_train, y_train,
        eval_set=[(X_val, y_val)],
        callbacks=[lgb.early_stopping(50, verbose=False)],
    )
    
    return roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=100, show_progress_bar=True)

print(f"Best params: {study.best_params}")
print(f"Best AUC: {study.best_value:.4f}")

SHAP — интерпретация модели

import shap

# TreeExplainer для tree-based моделей (быстро)
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Global importance
shap.summary_plot(shap_values, X_test, plot_type="bar")
shap.summary_plot(shap_values, X_test)  # beeswarm plot

# Local explanation (для одного prediction)
shap.force_plot(explainer.expected_value, shap_values[0], X_test.iloc[0])

Stacking — объединение нескольких моделей

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression

estimators = [
    ("rf", RandomForestClassifier(n_estimators=200, random_state=42)),
    ("xgb", xgb.XGBClassifier(n_estimators=200, random_state=42)),
    ("lgb", lgb.LGBMClassifier(n_estimators=200, random_state=42, verbose=-1)),
]

stack = StackingClassifier(
    estimators=estimators,
    final_estimator=LogisticRegression(),
    cv=5,
    n_jobs=-1,
)

stack.fit(X_train, y_train)

Интеграция с backend

XGBoost FastAPI serving

from fastapi import FastAPI
from pydantic import BaseModel
import xgboost as xgb
import numpy as np

app = FastAPI()
model = xgb.XGBClassifier()
model.load_model("models/xgb_v1.json")  # XGBoost native format (быстрее)

class Features(BaseModel):
    feature_vector: list[float]

@app.post("/predict")
def predict(input_data: Features):
    X = np.array([input_data.feature_vector])
    proba = float(model.predict_proba(X)[0, 1])
    return {
        "prediction": int(proba > 0.5),
        "probability": proba,
    }

ONNX export — Cross-platform

# LightGBM/XGBoost → ONNX → работает везде (C++, Java, Go, .NET)
from onnxmltools import convert_lightgbm
from skl2onnx.common.data_types import FloatTensorType

initial_types = [("input", FloatTensorType([None, X_train.shape[1]]))]
onnx_model = convert_lightgbm(model, initial_types=initial_types)

with open("model.onnx", "wb") as f:
    f.write(onnx_model.SerializeToString())

# Serving (ONNX Runtime)
import onnxruntime as ort
session = ort.InferenceSession("model.onnx")
predictions = session.run(None, {"input": X_test.astype("float32")})[0]

Batch prediction with Celery

from celery import Celery

celery_app = Celery("ml_tasks", broker="redis://localhost:6379")

@celery_app.task
def batch_predict(file_path: str):
    df = pd.read_csv(file_path)
    model = joblib.load("model.joblib")
    
    predictions = model.predict_proba(df)[:, 1]
    df["churn_probability"] = predictions
    df["risk_segment"] = pd.cut(predictions, bins=[0, 0.3, 0.7, 1.0],
                                 labels=["low", "medium", "high"])
    
    output_path = file_path.replace(".csv", "_predictions.csv")
    df.to_csv(output_path, index=False)
    
    return {"file": output_path, "n_predictions": len(df)}

Ресурсы

• XGBoost docs — xgboost.readthedocs.io
• LightGBM docs — lightgbm.readthedocs.io
• CatBoost docs — catboost.ai
• “XGBoost: A Scalable Tree Boosting System” — Chen & Guestrin (paper)
• SHAP docs — shap.readthedocs.io
• Kaggle Learn — Intermediate ML(посвящён XGBoost)
• “Effective XGBoost” — Matt Harrison (книга)

🏋️ Упражнения

🟢 Easy

1. Сравните RandomForest и LogisticRegression (Titanic).
2. В XGBoost сравните n_estimators со значениями 100, 500, 1000.
3. Возьмите feature importance из 3 разных моделей.

🟡 Medium

1. 3-way comparison: на одинаковом dataset XGBoost vs LightGBM vs CatBoost — сравните accuracy + training time.
2. Optuna: tuning LightGBM 100 trial, сравните default vs tuned.
3. SHAP: SHAP summary plot для вашей обученной модели, top 10 features.

🔴 Hard

1. Stacking ensemble: 5+ base моделей + meta-learner. Submit в Kaggle.
2. ONNX serving: export XGBoost в ONNX, serving в Go или Node.js (глубже backend integration).
3. Custom objective: напишите business-aware objective function (например, asymmetric loss: FN $50, FP $10).

Capstone — Kaggle Competition

notebooks/month-02/06_kaggle_competition.ipynb:

• Kaggle — Titanic или Spaceship Titanic
• Полный pipeline: EDA → FE → 3+ модели → ensemble → submission
• Цель: top 30% (~0.80 accuracy на Titanic, ~0.81 на Spaceship Titanic)
• Загрузите notebook и на Kaggle (публичный notebook)
• Commit в GitHub, в README ссылка на Kaggle Profile

✅ Чек-лист

• Знаю разницу между Bagging и Boosting
• Знаю, когда применять Random Forest, XGBoost, LightGBM
• Умею работать с early stopping
• Делаю hyperparameter tuning через Optuna
• Знаю преимущество CatBoost в categorical handling
• Могу интерпретировать модель через SHAP
• Знаком с export в ONNX и loading
• Сделал submit в Kaggle competition (top 30%)

Месяц 2 завершён! Изучите Упражнения и переходите к Месяц 3 — Deep Learning.