Инженерия признаков

🎯 Цель

После прочтения этой главы:

Узнаете, что Feature Engineering занимает 60-80% времени ML-проекта
Сможете правильно готовить categorical, numerical и datetime features
Освоите искусство создания новых features (domain-based)
Снижаете dimensionality через техники feature selection
Используете PCA и другие техники dimensionality reduction

Что нужно изучить

Scaling: StandardScaler, MinMaxScaler, RobustScaler, Normalizer
Encoding: OneHot, Label, Ordinal, Target, Frequency, Binary
Missing data: SimpleImputer, KNNImputer, IterativeImputer
Feature creation: polynomial, interaction, binning, datetime extraction
Text features: BoW, TF-IDF, n-grams
Feature selection: Filter, Wrapper, Embedded methods
Dimensionality reduction: PCA, LDA, t-SNE, UMAP
Outliers: detection (IQR, z-score) и treatment

Библиотеки

pip install scikit-learn category_encoders feature-engine

scikit-learn — основное
category_encoders — расширенный encoding (Target, James-Stein и т.д.)
feature-engine — pipeline для feature engineering

Важные темы

Feature Engineering — айсберг ML

   ML algoritmi (10%)
   ───────────────────  ← ko'rinadigan qism
       Feature Engineering (60%)
       Data Quality (20%)
       Domain Knowledge (10%)

Andrew Ng:“Coming up with features is difficult, time-consuming, requires expert knowledge. Applied machine learning is basically feature engineering.”

Scaling — когда и какой?

Scaler	Формула	Когда
StandardScaler	`(x - μ) / σ`	Default, приводит к normal distribution
MinMaxScaler	`(x - min) / (max - min)`	Neural networks ([0,1] для цветов)
RobustScaler	`(x - median) / IQR`	При наличии outliers
Normalizer	`x /

Правила:

Distance-based алгоритмы (KNN, SVM, K-Means) — scaling обязательно
Tree-based (Random Forest, XGBoost) — scaling не обязательно
Linear models — scaling рекомендуется(для regularization)

Categorical Encoding

# Cat values: ['cat', 'dog', 'fish']

# 1. Label Encoding (только для ordinal data!)
[0, 1, 2]  # ['cat'=0, 'dog'=1, 'fish'=2] — ложный порядок!

# 2. OneHot Encoding (для nominal data)
cat: [1, 0, 0]
dog: [0, 1, 0]
fish:[0, 0, 1]

# 3. Target Encoding (для high cardinality)
# Среднее значение target для каждой category
cat: 0.5    # avg(y | category=cat)
dog: 0.3
fish: 0.7

Проблема High Cardinality

Если у feature 1000+ unique value (например, user_id, city), OneHot encoding создаст 1000 столбцов и приведёт к overfitting.

Решения:

Target encoding — замена на mean(y) (внутри CV!)
Frequency encoding — счётчик каждой category
Embedding — neural network (в более глубоком месяце)
Hashing — HashingEncoder
Grouping — объединение редких в “Other”

Datetime features

import pandas as pd

df["date"] = pd.to_datetime(df["date"])

df["year"] = df["date"].dt.year
df["month"] = df["date"].dt.month
df["day"] = df["date"].dt.day
df["weekday"] = df["date"].dt.dayofweek
df["weekend"] = df["weekday"].isin([5, 6]).astype(int)
df["hour"] = df["date"].dt.hour
df["quarter"] = df["date"].dt.quarter

# Cyclic encoding (поскольку время циклично)
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

Feature Selection — 3 подхода

Filter methods(без алгоритма)

Variance Threshold (удалить features с низкой variance)
Correlation-based (корреляция с target)
Chi-squared test (для categorical)
Mutual Information

Wrapper methods(с алгоритмом)

Recursive Feature Elimination (RFE)
Sequential Forward/Backward Selection

Embedded methods(внутри алгоритма)

Lasso (L1) → features с coefficient = 0 удаляются
Tree-based feature importance

Примеры кода

Полный ColumnTransformer pipeline

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, OrdinalEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

numeric_features = ["age", "income", "tenure"]
nominal_features = ["city", "department"]
ordinal_features = ["education"]
education_order = [["primary", "secondary", "bachelor", "master", "phd"]]

# Numeric pipeline
numeric_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])

# Nominal categorical pipeline
nominal_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="constant", fill_value="missing")),
    ("onehot", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])

# Ordinal pipeline
ordinal_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("ord", OrdinalEncoder(categories=education_order)),
])

preprocessor = ColumnTransformer([
    ("num", numeric_pipe, numeric_features),
    ("nom", nominal_pipe, nominal_features),
    ("ord", ordinal_pipe, ordinal_features),
])

full_pipeline = Pipeline([
    ("preprocess", preprocessor),
    ("model", LogisticRegression()),
])

Target Encoding (with cross-validation, leak-free)

from category_encoders import TargetEncoder
from sklearn.model_selection import cross_val_score

# Внимание: обычный TargetEncoder протекает (preprocessor видит target)
# Правильный путь — внутри Pipeline

pipeline = Pipeline([
    ("encoder", TargetEncoder(cols=["city", "category"])),
    ("scaler", StandardScaler()),
    ("model", LogisticRegression()),
])

# Внутри CV encoder заново fit для каждого fold
scores = cross_val_score(pipeline, X, y, cv=5)

PCA — Dimensionality Reduction

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# Scale (PCA чувствителен к scaling)
X_scaled = StandardScaler().fit_transform(X)

# Компоненты, сохраняющие 95% variance
pca = PCA(n_components=0.95)
X_pca = pca.fit_transform(X_scaled)

print(f"Original features: {X.shape[1]}")
print(f"PCA components:    {X_pca.shape[1]}")
print(f"Explained variance: {pca.explained_variance_ratio_.sum():.3f}")

# Scree plot
import matplotlib.pyplot as plt
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel("Number of components")
plt.ylabel("Cumulative explained variance")
plt.axhline(0.95, color="r", linestyle="--")
plt.show()

Пример Feature Selection

from sklearn.feature_selection import SelectKBest, f_classif, RFE
from sklearn.ensemble import RandomForestClassifier

# 1. SelectKBest (filter)
selector = SelectKBest(score_func=f_classif, k=10)
X_selected = selector.fit_transform(X, y)
selected_features = X.columns[selector.get_support()]

# 2. RFE (wrapper)
rfe = RFE(estimator=RandomForestClassifier(random_state=42), n_features_to_select=10)
rfe.fit(X, y)
selected_rfe = X.columns[rfe.support_]

# 3. Feature importance (embedded)
rf = RandomForestClassifier(random_state=42)
rf.fit(X, y)
importance_df = pd.DataFrame({
    "feature": X.columns,
    "importance": rf.feature_importances_,
}).sort_values("importance", ascending=False)

Domain-based feature creation

# Новые features в e-commerce dataset
df["price_per_item"] = df["total_price"] / df["quantity"]
df["discount_pct"] = (df["original_price"] - df["price"]) / df["original_price"]
df["is_weekend"] = df["order_date"].dt.dayofweek.isin([5, 6]).astype(int)
df["days_since_signup"] = (df["order_date"] - df["signup_date"]).dt.days
df["customer_lifetime_orders"] = df.groupby("customer_id")["order_id"].transform("count")
df["avg_order_value"] = df.groupby("customer_id")["total_price"].transform("mean")

Интеграция с backend

Feature Store pattern

# Feature engineering в backend — Django service
class FeatureService:
    def compute_user_features(self, user_id: int) -> dict:
        user = User.objects.get(id=user_id)
        orders = Order.objects.filter(user=user)
        
        return {
            "user_age_days": (timezone.now() - user.created_at).days,
            "total_orders": orders.count(),
            "avg_order_value": orders.aggregate(Avg("amount"))["amount__avg"] or 0,
            "days_since_last_order": (
                timezone.now() - orders.latest("created_at").created_at
            ).days if orders.exists() else 999,
            "preferred_category": orders.values("category")
                .annotate(n=Count("id")).order_by("-n").first()["category"]
                if orders.exists() else None,
        }

# В FastAPI
@app.post("/predict/")
def predict(user_id: int):
    features = feature_service.compute_user_features(user_id)
    pipeline = joblib.load("model.joblib")  # ColumnTransformer + model
    
    df = pd.DataFrame([features])
    prediction = pipeline.predict(df)[0]
    return {"prediction": float(prediction)}

Feature versioning

# config.py
FEATURE_SCHEMA_V1 = {
    "version": "1.0",
    "numeric": ["age", "income"],
    "categorical": ["city", "department"],
}

# Сохранение схемы вместе с моделью
joblib.dump({
    "pipeline": full_pipeline,
    "feature_schema": FEATURE_SCHEMA_V1,
    "trained_at": datetime.now().isoformat(),
}, "model_bundle.joblib")

Ресурсы

“Feature Engineering for Machine Learning” — Alice Zheng, Amanda Casari (O’Reilly)
scikit-learn Preprocessing docs — scikit-learn.org/stable/modules/preprocessing.html
category_encoders docs — contrib.scikit-learn.org/category_encoders
“Feature Engineering A-Z” — Kaggle tutorial
Feast (Feature Store) — feast.dev — production feature storage

🏋️ Упражнения

🟢 Easy

Сравните StandardScaler, MinMaxScaler, RobustScaler на dataset с outliers.
Покажите разницу между OneHotEncoder и OrdinalEncoder на примере.
Через pd.cut разделите continuous age на bins [child, teen, adult, senior].

🟡 Medium

Datetime FE: на NYC Taxi dataset создайте 10+ новых features из pickup_datetime (hour, weekday, season, holiday, rush_hour и т.д.).
Показать leak в Target Encoding: выполните target encoding внутри и снаружи CV, посмотрите разницу R².
PCA + classification: на высокоразмерном digit dataset снизьте dimensionality до 20 через PCA, посмотрите изменения accuracy и training time.

🔴 Hard

Production feature store: создайте в Django FeatureService class — вычисляет real-time features для каждого user, кэширует в Redis (TTL=1h), интегрируется с ML pipeline.
Auto FE: с одной из AutoML-библиотек (featuretools, tsfresh) сделайте automatic feature engineering и сравните с manual FE.

Capstone

notebooks/month-02/04_feature_engineering.ipynb:

Снова откройте Telco Churn dataset
Создайте 30+ новых features (datetime, ratios, aggregations, interactions)
Ranking feature importance
Эксперимент с PCA
Original vs модель после FE — покажите разницу accuracy

✅ Чек-лист

Понимаю, что Feature Engineering — самая важная часть ML
Знаю 4 типа scaler, знаю, когда какой применять
Знаю типы categorical encoding и проблему high cardinality
Могу создать минимум 10 features из datetime
Понимаю, зачем нужны PCA и dimensionality reduction
Знаю 3 метода feature selection
Пишу полный preprocessing через ColumnTransformer + Pipeline
Знаю проблему leak в target encoding

Переходим к Model Evaluation.

Keyboard shortcuts

Backend to ML: 6 Oylik Roadmap