Feature Engineering

🎯 Goal

After reading this chapter:

You will know that Feature Engineering takes 60-80% of ML project time
You will be able to properly prepare categorical, numerical, and datetime features
You will learn the art of creating new features (domain-based)
You will reduce dimensionality with feature selection techniques
You will use PCA and other dimensionality reduction techniques

What to learn

Scaling: StandardScaler, MinMaxScaler, RobustScaler, Normalizer
Encoding: OneHot, Label, Ordinal, Target, Frequency, Binary
Missing data: SimpleImputer, KNNImputer, IterativeImputer
Feature creation: polynomial, interaction, binning, datetime extraction
Text features: BoW, TF-IDF, n-grams
Feature selection: Filter, Wrapper, Embedded methods
Dimensionality reduction: PCA, LDA, t-SNE, UMAP
Outliers: detection (IQR, z-score) and treatment

Libraries

pip install scikit-learn category_encoders feature-engine

scikit-learn — main
category_encoders — extended encoding (Target, James-Stein, etc.)
feature-engine — feature engineering pipelines

Important topics

Feature Engineering — the iceberg of ML

   ML algoritmi (10%)
   ───────────────────  ← ko'rinadigan qism
       Feature Engineering (60%)
       Data Quality (20%)
       Domain Knowledge (10%)

Andrew Ng:“Coming up with features is difficult, time-consuming, requires expert knowledge. Applied machine learning is basically feature engineering.”

Scaling — when and which?

Scaler	Formula	When
StandardScaler	`(x - μ) / σ`	Default, adjusts to a normal distribution
MinMaxScaler	`(x - min) / (max - min)`	Neural networks (for colors [0,1])
RobustScaler	`(x - median) / IQR`	When outliers are present
Normalizer	`x /

Rules:

Distance-based algorithms (KNN, SVM, K-Means) — scaling is required
Tree-based (Random Forest, XGBoost) — scaling is not required
Linear models — scaling is recommended(for regularization)

Categorical Encoding

# Cat values: ['cat', 'dog', 'fish']

# 1. Label Encoding (only for ordinal data!)
[0, 1, 2]  # ['cat'=0, 'dog'=1, 'fish'=2] — false order!

# 2. OneHot Encoding (for nominal data)
cat: [1, 0, 0]
dog: [0, 1, 0]
fish:[0, 0, 1]

# 3. Target Encoding (for high cardinality) # Mean target value for each category
cat: 0.5    # avg(y | category=cat)
dog: 0.3
fish: 0.7

High Cardinality problem

If a feature has 1000+ unique values(for example, user_id, city), OneHot encoding creates 1000 columns and leads to overfitting.

Solutions:

Target encoding — replace with mean(y) (inside CV!)
Frequency encoding — count of each category
Embedding — neural network (in a deeper month)
Hashing — HashingEncoder
Grouping — group rare ones into “Other”

Datetime features

import pandas as pd

df["date"] = pd.to_datetime(df["date"])

df["year"] = df["date"].dt.year
df["month"] = df["date"].dt.month
df["day"] = df["date"].dt.day
df["weekday"] = df["date"].dt.dayofweek
df["weekend"] = df["weekday"].isin([5, 6]).astype(int)
df["hour"] = df["date"].dt.hour
df["quarter"] = df["date"].dt.quarter

# Cyclic encoding (since time is periodic)
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

Feature Selection — 3 approaches

Filter methods(without algorithm)

Variance Threshold (remove low-variance features)
Correlation-based (correlation with target)
Chi-squared test (for categorical)
Mutual Information

Wrapper methods(with algorithm)

Recursive Feature Elimination (RFE)
Sequential Forward/Backward Selection

Embedded methods(inside the algorithm)

Lasso (L1) → features with coefficient = 0 are removed
Tree-based feature importance

Code examples

Complete ColumnTransformer pipeline

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, OrdinalEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

numeric_features = ["age", "income", "tenure"]
nominal_features = ["city", "department"]
ordinal_features = ["education"]
education_order = [["primary", "secondary", "bachelor", "master", "phd"]]

# Numeric pipeline
numeric_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])

# Nominal categorical pipeline
nominal_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="constant", fill_value="missing")),
    ("onehot", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])

# Ordinal pipeline
ordinal_pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("ord", OrdinalEncoder(categories=education_order)),
])

preprocessor = ColumnTransformer([
    ("num", numeric_pipe, numeric_features),
    ("nom", nominal_pipe, nominal_features),
    ("ord", ordinal_pipe, ordinal_features),
])

full_pipeline = Pipeline([
    ("preprocess", preprocessor),
    ("model", LogisticRegression()),
])

Target Encoding (with cross-validation, leak-free)

from category_encoders import TargetEncoder
from sklearn.model_selection import cross_val_score

# Note: a simple TargetEncoder leaks (preprocessor sees target) # Correct way — inside Pipeline

pipeline = Pipeline([
    ("encoder", TargetEncoder(cols=["city", "category"])),
    ("scaler", StandardScaler()),
    ("model", LogisticRegression()),
])

# Inside CV, encoder is re-fit for each fold
scores = cross_val_score(pipeline, X, y, cv=5)

PCA — Dimensionality Reduction

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# Scale (PCA is sensitive to scaling)
X_scaled = StandardScaler().fit_transform(X)

# Components that preserve 95% variance
pca = PCA(n_components=0.95)
X_pca = pca.fit_transform(X_scaled)

print(f"Original features:{X.shape[1]}")
print(f"PCA components:{X_pca.shape[1]}")
print(f"Explained variance:{pca.explained_variance_ratio_.sum():.3f}")

# Scree plot
import matplotlib.pyplot as plt
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel("Number of components")
plt.ylabel("Cumulative explained variance")
plt.axhline(0.95, color="r", linestyle="--")
plt.show()

Feature Selection example

from sklearn.feature_selection import SelectKBest, f_classif, RFE
from sklearn.ensemble import RandomForestClassifier

# 1. SelectKBest (filter)
selector = SelectKBest(score_func=f_classif, k=10)
X_selected = selector.fit_transform(X, y)
selected_features = X.columns[selector.get_support()]

# 2. RFE (wrapper)
rfe = RFE(estimator=RandomForestClassifier(random_state=42), n_features_to_select=10)
rfe.fit(X, y)
selected_rfe = X.columns[rfe.support_]

# 3. Feature importance (embedded)
rf = RandomForestClassifier(random_state=42)
rf.fit(X, y)
importance_df = pd.DataFrame({
    "feature": X.columns,
    "importance": rf.feature_importances_,
}).sort_values("importance", ascending=False)

Domain-based feature creation

# New features on an e-commerce dataset
df["price_per_item"] = df["total_price"] / df["quantity"]
df["discount_pct"] = (df["original_price"] - df["price"]) / df["original_price"]
df["is_weekend"] = df["order_date"].dt.dayofweek.isin([5, 6]).astype(int)
df["days_since_signup"] = (df["order_date"] - df["signup_date"]).dt.days
df["customer_lifetime_orders"] = df.groupby("customer_id")["order_id"].transform("count")
df["avg_order_value"] = df.groupby("customer_id")["total_price"].transform("mean")

Backend integration

Feature Store pattern

# Feature engineering on the backend — Django service
class FeatureService:
    def compute_user_features(self, user_id: int) -> dict:
        user = User.objects.get(id=user_id)
        orders = Order.objects.filter(user=user)
        
        return {
            "user_age_days": (timezone.now() - user.created_at).days,
            "total_orders": orders.count(),
            "avg_order_value": orders.aggregate(Avg("amount"))["amount__avg"] or 0,
            "days_since_last_order": (
                timezone.now() - orders.latest("created_at").created_at
            ).days if orders.exists() else 999,
            "preferred_category": orders.values("category")
                .annotate(n=Count("id")).order_by("-n").first()["category"]
                if orders.exists() else None,
        }

# In FastAPI
@app.post("/predict/")
def predict(user_id: int):
    features = feature_service.compute_user_features(user_id)
    pipeline = joblib.load("model.joblib")  # ColumnTransformer + model
    
    df = pd.DataFrame([features])
    prediction = pipeline.predict(df)[0]
    return {"prediction": float(prediction)}

Feature versioning

# config.py
FEATURE_SCHEMA_V1 = {
    "version": "1.0",
    "numeric": ["age", "income"],
    "categorical": ["city", "department"],
}

# Save schema together with model
joblib.dump({
    "pipeline": full_pipeline,
    "feature_schema": FEATURE_SCHEMA_V1,
    "trained_at": datetime.now().isoformat(),
}, "model_bundle.joblib")

Resources

“Feature Engineering for Machine Learning” — Alice Zheng, Amanda Casari (O’Reilly)
scikit-learn Preprocessing docs — scikit-learn.org/stable/modules/preprocessing.html
category_encoders docs — contrib.scikit-learn.org/category_encoders
“Feature Engineering A-Z” — Kaggle tutorial
Feast (Feature Store) — feast.dev — production feature storage

🏋️ Exercises

🟢 Easy

Compare StandardScaler, MinMaxScaler, RobustScaler on a dataset with outliers.
Show the difference between OneHotEncoder and OrdinalEncoder with an example.
Use pd.cut to bin continuous age into [child, teen, adult, senior] bins.

🟡 Medium

Datetime FE: On the NYC Taxi dataset, create 10+ new features from pickup_datetime (hour, weekday, season, holiday, rush_hour, etc.).
Spot Target Encoding leak: Do target encoding inside and outside CV, and see the R² difference.
PCA + classification: On a high-dimensional digit dataset, reduce dimensionality to 20 with PCA, and observe the change in classifier accuracy and training time.

🔴 Hard

Production feature store: Create a FeatureService class in Django — computes real-time features for each user, caches in Redis (TTL=1h), integrates with the ML pipeline.
Auto FE: Use one of the AutoML libraries (featuretools, tsfresh) for automatic feature engineering, and compare with manual FE.

Capstone

notebooks/month-02/04_feature_engineering.ipynb:

Reopen the Telco Churn dataset
Create 30+ new features (datetime, ratios, aggregations, interactions)
Feature importance ranking
Experiment with PCA
Original vs FE-enhanced model — show the accuracy difference

✅ Checklist

I understand Feature Engineering is the most important part of ML
I know 4 types of scalers and when to use each
I know categorical encoding types and the high cardinality problem
I can create at least 10 features from datetime
I understand why PCA and dimensionality reduction are needed
I know 3 methods for feature selection
I write complete preprocessing with ColumnTransformer + Pipeline
I know the leak problem in target encoding

Moving on to Model Evaluation.

Keyboard shortcuts

Backend to ML: 6 Oylik Roadmap