DVC — Data Versioning

🎯 Goal

After reading this chapter:

• You know why data versioning is needed
• You know how to use DVC with Git
• You can create DVC pipelines (dvc.yaml)
• You connect to remote storage (S3, GCS)
• You’re familiar with DVC alternatives (LakeFS, Pachyderm)

What to learn

• DVC basics — dvc init, dvc add, dvc push, dvc pull
• Remote storage — S3, GCS, Azure, SSH
• DVC pipelines — dvc.yaml, stages
• dvc.lock — reproducibility
• dvc repro — automatic pipeline re-execution
• DVC + MLflowintegration
• DVC + CI/CD

Libraries

pip install dvc
pip install "dvc[s3]" # for S3
pip install "dvc[gs]" # for GCS
pip install "dvc[azure]" # for Azure

Why DVC?

Problem

Git can’t work with large files (datasets, models):

• git push 100GB CSV — no
• git diff is useless on binary files
• The repository grows quickly

Solution — DVC

Git tracks:        small files (code, configs, .dvc files)
DVC tracks:        large files (data, models, embeddings)
Storage:           S3, GCS, local NAS, SSH

my_project/
├── .git/                       # code versioning
├── .dvc/                       # DVC config
├── data/
│   └── train.csv               # .gitignore'da
├── data/train.csv.dvc          # ← bu kichik fayl Git'da
├── model.pkl                   # .gitignore'da
├── model.pkl.dvc               # ← bu kichik fayl Git'da
└── dvc.yaml                    # pipeline definition

Code examples

Initial setup

# 1. Initialize project
cd my_project
git init
dvc init
git commit -m "Initialize DVC"

# 2. Add remote storage (S3 example)
dvc remote add -d s3remote s3://my-bucket/dvc-storage
dvc remote modify s3remote endpointurl https://s3.amazonaws.com
dvc remote modify s3remote access_key_id "$AWS_ACCESS_KEY_ID"
dvc remote modify s3remote secret_access_key "$AWS_SECRET_ACCESS_KEY"

# or local storage (for testing)
dvc remote add -d localremote /Users/me/dvc-storage

# Commit configuration
git add .dvc/config
git commit -m "Configure DVC remote"

Data versioning

# 1. Add data file to DVC
dvc add data/train.csv

# What this does:
# - data/train.csv ni .dvc/cache ga ko# - removes data/train.csv from git # - creates data/train.csv.dvc (small metadata file) # - adds data/train.csv to .gitignoreshadi

# 2. Commit to Git
git add data/train.csv.dvc data/.gitignore
git commit -m "Add training data v1"

# 3. Push to remote
dvc push

# 4. Changes
# (data/train.csv ni o# (if you modify the data) dvc add data/train.csv git add data/train.csv.dvc git commit -m "Update training data to v2" dvc push # 5. Rollback to old version git checkout HEAD~1 data/train.csv.dvc dvc pull

On another machine (or in CI)

git clone https://github.com/me/my_project.git
cd my_project
dvc pull         # load data from remote storage

DVC Pipeline — dvc.yaml

# dvc.yaml stages:
  prepare:
    cmd: python src/data/make_dataset.py
    deps:
      - src/data/make_dataset.py
      - data/raw/data.csv
    outs:
      - data/processed/train.csv
      - data/processed/test.csv
    params:
      - prepare.test_size
      - prepare.random_state

  features:
    cmd: python src/features/build_features.py
    deps:
      - src/features/build_features.py
      - data/processed/train.csv
    outs:
      - data/features/train.parquet
    params:
      - features.feature_set

  train:
    cmd: python src/models/train.py
    deps:
      - src/models/train.py
      - data/features/train.parquet
    outs:
      - models/model.pkl
    metrics:
      - metrics/train_metrics.json:
          cache: false
    plots:
      - plots/learning_curve.png:
          cache: false
    params:
      - train.n_estimators
      - train.max_depth
      - train.learning_rate

params.yaml — hyperparameters

# params.yaml prepare:
  test_size: 0.2
  random_state: 42

features:
  feature_set: "v2"

train:
  n_estimators: 200
  max_depth: 10
  learning_rate: 0.1

Running the pipeline

# Reproduce the pipeline
dvc repro

# DVC is smart — only changed stages re-run!
# Masalan: faqat params.yaml o# If only train.n_estimators changes → only the train stage runs

# Force re-run
dvc repro -f

# Specific stage
dvc repro train

# View metricsrish
dvc metrics show

# View plots (HTML report)
dvc plots show

DVC + Git workflow

# 1. Experiment
git checkout -b experiment-1
# (modify params.yaml)
dvc repro
dvc push

# 2. Compare metrics
dvc metrics diff main
# Output:
# Path                            Metric    main    workspace    change
# metrics/train_metrics.json accuracy 0.85 0.89 0.04

# 3. If good — merge
git add params.yaml dvc.lock metrics/
git commit -m "Improved accuracy to 89%"
git checkout main
git merge experiment-1
dvc push

Experiments (DVC 2.0+)

# Quick experiments (no commit)
dvc exp run --set-param train.n_estimators=500
dvc exp run --set-param train.n_estimators=1000
dvc exp run --set-param train.max_depth=20

# Compare
dvc exp show

# Commit the best
dvc exp apply <exp-name>
git add .
git commit -m "Best params"

Python API

import dvc.api

# Read DVC tracked file
with dvc.api.open("data/processed/train.csv", repo=".") as f:
    df = pd.read_csv(f)

# Or with URL (from remote)
url = dvc.api.get_url(
    path="data/processed/train.csv",
    repo="https://github.com/me/my_project.git",
)
df = pd.read_csv(url)

# Params
import yaml
params = yaml.safe_load(open("params.yaml"))
n_estimators = params["train"]["n_estimators"]

Metrics tracking (DVC + MLflow integration)

# src/models/train.py
import json
import mlflow
from sklearn.ensemble import RandomForestClassifier

# DVC params
params = yaml.safe_load(open("params.yaml"))["train"]

mlflow.set_experiment("dvc_pipeline")
with mlflow.start_run():
    mlflow.log_params(params)
    
    model = RandomForestClassifier(**params)
    model.fit(X_train, y_train)
    
    metrics = {
        "accuracy": accuracy_score(y_test, model.predict(X_test)),
        "f1": f1_score(y_test, model.predict(X_test)),
    }
    
    mlflow.log_metrics(metrics)
    
    # MLflow log
    mlflow.sklearn.log_model(model, "model")
    
    # DVC metrics file
    os.makedirs("metrics", exist_ok=True)
    with open("metrics/train_metrics.json", "w") as f:
        json.dump(metrics, f)

Backend integration

CI/CD: GitHub Actions + DVC

# .github/workflows/dvc-train.yml name: Train Model

on:
  push:
    paths:
      - "src/**"
      - "data/**.dvc"
      - "params.yaml"

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
      
      - name: Configure DVC + AWS
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          dvc remote modify s3remote access_key_id "$AWS_ACCESS_KEY_ID"
          dvc remote modify s3remote secret_access_key "$AWS_SECRET_ACCESS_KEY"
      
      - name: Pull data
        run: dvc pull
      
      - name: Run pipeline
        run: dvc repro
      
      - name: Push artifacts
        run: dvc push
      
      - name: Comment metrics on PR
        if: github.event_name == 'pull_request'
        uses: iterative/cml-action@v1
        run: |
          dvc metrics diff main >> report.md
          cml comment create report.md

Production data pipeline

# scheduled retrain.py
import subprocess
from datetime import datetime

def retrain_pipeline():
    # 1. Pull latest data
    subprocess.run(["dvc", "pull", "data/raw/data.csv.dvc"], check=True)
    
    # 2. Update data (new day's data)
    update_raw_data()
    
    # 3. Track new version
    subprocess.run(["dvc", "add", "data/raw/data.csv"], check=True)
    
    # 4. Reproduce pipeline (auto train if changes)
    subprocess.run(["dvc", "repro"], check=True)
    
    # 5. Check metrics
    with open("metrics/train_metrics.json") as f:
        metrics = json.load(f)
    
    # 6. If improved, push + register
    if metrics["accuracy"] > THRESHOLD:
        subprocess.run(["dvc", "push"], check=True)
        subprocess.run(["git", "commit", "-am", f"Auto-retrain{datetime.now()}"], check=True)
        register_model_in_mlflow()

Resources

• DVC docs — dvc.org/doc
• DVC tutorials — dvc.org/doc/start
• CML (Continuous Machine Learning) — DVC team CI/CD: cml.dev
• “DVC: A New Tool for Versioning Data” — Towards Data Science
• Alternatives:
• LakeFS — lakefs.io — Git for data lakes
• Pachyderm — Kubernetes-native data versioning
• lakeFS — data lake versioning

🏋️ Exercises

🟢 Easy

1. Versioning for one CSV file with dvc init + dvc add.
2. Local DVC remote setup.
3. Create 2 versions, rollback to the old one.

🟡 Medium

1. Full pipeline: prepare → train → evaluate stages in dvc.yaml.
2. DVC + MLflow: use both together.
3. DVC experiments: 5 different hyperparam experiments.

🔴 Hard

1. Production DVC + S3: With AWS S3 or MinIO, GitHub Actions CI/CD.
2. Multi-stage pipeline: 5+ stages, parametrized, plots, metrics.
3. Distributed: Strategies for working with large datasets (100GB+).

Capstone

In notebooks/month-06/03_dvc.ipynb + dvc.yaml:

• ML project + DVC + MLflow + GitHub Actions
• Pipeline: prepare → features → train → evaluate
• Metrics, plots tracking
• S3 (or MinIO) remote

✅ Checklist

• I know why DVC is needed
• I use dvc add, dvc push, dvc pull
• I set up remote storage (S3 or similar)
• I write dvc.yaml pipelines
• Automatic retraining via dvc repro
• DVC + MLflow integration
• DVC pipeline in GitHub Actions
• Understanding of alternatives (LakeFS)

Moving on to FastAPI + ML Serving — your strong suit.