Text Preprocessing

🎯 Goal

After reading this chapter:

• You will know how to clean real, “dirty” text data
• You will be able to find complex patterns with regex
• You will be able to work with HuggingFace tokenizers
• You will be able to write a production text pipeline

What to learn

• Text cleaning — HTML, URL, emoji, punctuation
• Unicode normalization — NFC, NFD, NFKC
• Encoding issues — UTF-8, Windows-1251, latin1
• Regex — pattern matching, capture groups
• Subword tokenization — BPE, WordPiece, SentencePiece
• **HuggingFace tokenizers**library
• Truncation and padding strategies
• Multi-language handling

Libraries

pip install nltk spacy transformers tokenizers ftfy unidecode emoji
pip install beautifulsoup4 lxml                  # HTML parsing

Important topics

Text cleaning pipeline

Real text comes in like this:

"<p>Hello!!! 😊 My email: ali@gmail.com,&nbsp;phone: +99890-123-45-67. Marketing manager 🚀</p>"

Our task — make it “clean” for ML:

"hello my email phone marketing manager"

Subword Tokenization — what and why?

Problem with classical word-level tokenization:

• Vocabulary is too large (millions of words)
• “running”, “runs”, “runner” — handled separately
• Unknown words (OOV) — become [UNK]

Subword tokenization solution:

Example (BPE):"unfortunately" → ["un", "for", "tun", "ate", "ly"]

New words are also split into pieces, no OOV problem.

Code examples

Basic text cleaning

import re
from bs4 import BeautifulSoup
import emoji
import unicodedata

def clean_text(text: str) -> str:
    # 1. Remove HTML
    text = BeautifulSoup(text, "lxml").get_text()
    
    # 2. URLs
    text = re.sub(r"https?://\S+|www\.\S+", "", text)
    
    # 3. Emails
    text = re.sub(r"\S+@\S+", "", text)
    
    # 4. Phone numbers (simple)
    text = re.sub(r"\+?\d[\d\-\s\(\)]{7,}\d", "", text)
    
    # 5. Convert emojis to text or remove
    text = emoji.demojize(text, delimiters=("", ""))    # 😊 → smiling_face # or: text = emoji.replace_emoji(text, "")
    
    # 6. Unicode normalize
    text = unicodedata.normalize("NFKC", text)
    
    # 7. Special chars — keep only alphanumeric + space
    text = re.sub(r"[^\w\s]", " ", text, flags=re.UNICODE)
    
    # 8. Multiple spaces
    text = re.sub(r"\s+", " ", text).strip()
    
    # 9. Lowercase
    text = text.lower()
    
    return text

# Test
dirty = "<p>Salom!!! 😊 Mening email: ali@gmail.com.</p>"
print(clean_text(dirty))
# "salom mening email"

Encoding fix (ftfy)

from ftfy import fix_text

broken = "“Helloâ€\x9d" # incorrectly encoded
print(fix_text(broken))
# "Hello"

Regex patterns (useful)

import re

# Hashtags (#ai #machinelearning)
hashtags = re.findall(r"#(\w+)", text)

# Mentions (@username)
mentions = re.findall(r"@(\w+)", text)

# Dates (2024-05-28, 28/05/2024)
dates = re.findall(r"\b\d{4}[-/]\d{2}[-/]\d{2}\b|\b\d{2}[-/]\d{2}[-/]\d{4}\b", text)

# Phone numbers (UZ)
phones = re.findall(r"\+998[\s\-]?\d{2}[\s\-]?\d{3}[\s\-]?\d{2}[\s\-]?\d{2}", text)

# IP addresses
ips = re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", text)

# URLs
urls = re.findall(r"https?://[^\s<>\"'{}|\\^`\[\]]+", text)

HuggingFace Tokenizer

from transformers import AutoTokenizer

# BERT
tokenizer = AutoTokenizer.from_pretrained("bert-base-multilingual-cased")

text = "Salom dunyo! Bu mashina o'rganish."
tokens = tokenizer.tokenize(text)
# ['sal', '##om', 'duny', '##o', '!', 'bu', 'mash', '##ina', "'", 'ran', '##ish', '.']

# Token IDs
ids = tokenizer.encode(text, add_special_tokens=True)
# [101, ..., 102] ([CLS] and [SEP] added)

# Decode (back)
decoded = tokenizer.decode(ids)

# Batch processing (padding + truncation)
batch = ["Salom!", "Bu uzunroq matn. Bir necha gap bor."]
encoded = tokenizer(
    batch,
    padding=True,
    truncation=True,
    max_length=128,
    return_tensors="pt",
)
# {'input_ids': tensor(...), 'attention_mask': tensor(...), 'token_type_ids': tensor(...)}

Custom BPE tokenizer (HuggingFace tokenizers)

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

# 1. Train custom tokenizer
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()

trainer = BpeTrainer(
    vocab_size=30000,
    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],
)

files = ["data/uzbek_corpus.txt"]
tokenizer.train(files, trainer)

# 2. Save / load
tokenizer.save("uzbek_bpe.json")
tokenizer = Tokenizer.from_file("uzbek_bpe.json")

# 3. Encode
encoded = tokenizer.encode("Salom dunyo")
print(encoded.tokens)
print(encoded.ids)

Truncation and padding strategies

texts = [
    "Short text",
    "Medium length text with some more words",
    "Very long text " * 100,
]

# Truncation: shorten to max_length
encoded = tokenizer(
    texts,
    truncation=True,        # cut what exceeds max_length
    max_length=128,
    padding="max_length",   # pad to 128 with [PAD]
    return_tensors="pt",
)

# Other padding strategies: # padding="longest" — match the longest text (saves memory) # padding=False — no padding (for a single sample)

# Dynamic padding (longest in batch):
encoded = tokenizer(texts, padding=True, truncation=True, max_length=512)

Sliding window — for long texts

def chunk_text(text: str, tokenizer, max_length: int = 512, stride: int = 50):
    """Split a long text into overlapping chunks."""
    tokens = tokenizer.encode(text, add_special_tokens=False)
    chunks = []
    
    for i in range(0, len(tokens), max_length - stride):
        chunk_tokens = tokens[i:i + max_length]
        chunk_text = tokenizer.decode(chunk_tokens)
        chunks.append(chunk_text)
    
    return chunks

# Example: 10000-token text → 20 chunks of 512 tokens
long_text = "..." * 5000
chunks = chunk_text(long_text, tokenizer, max_length=512, stride=50)

Multi-language handling

from langdetect import detect

def preprocess_multilingual(text: str) -> dict:
    lang = detect(text)
    
    if lang == "en":
        cleaned = clean_text_english(text)
    elif lang == "uz":
        cleaned = clean_text_uzbek(text)
    elif lang == "ru":
        cleaned = clean_text_russian(text)
    else:
        cleaned = clean_text(text)
    
    return {"language": lang, "cleaned_text": cleaned}

Backend integration

Text preprocessing service

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class TextInput(BaseModel):
    text: str
    operations: list[str] = ["clean", "tokenize"]

class TextOutput(BaseModel):
    original: str
    cleaned: str
    tokens: list[str]
    language: str
    stats: dict

@app.post("/preprocess", response_model=TextOutput)
def preprocess(data: TextInput):
    original = data.text
    cleaned = clean_text(original) if "clean" in data.operations else original
    tokens = tokenizer.tokenize(cleaned) if "tokenize" in data.operations else []
    
    return TextOutput(
        original=original,
        cleaned=cleaned,
        tokens=tokens,
        language=detect(original) if original.strip() else "unknown",
        stats={
            "original_length": len(original),
            "cleaned_length": len(cleaned),
            "token_count": len(tokens),
        },
    )

Bulk processing (Celery)

@celery_app.task
def preprocess_dataset(csv_path: str, text_column: str):
    df = pd.read_csv(csv_path)
    df["cleaned"] = df[text_column].apply(clean_text)
    
    output_path = csv_path.replace(".csv", "_cleaned.csv")
    df.to_csv(output_path, index=False)
    
    return {"output": output_path, "n_rows": len(df)}

Resources

• HuggingFace Tokenizers docs — huggingface.co/docs/tokenizers
• “Natural Language Processing with Transformers” — Lewis Tunstall (O’Reilly)
• Regex101 — regex101.com (interactive regex tester)
• Unicode normalization — Unicode.org docs
• ftfy library — github.com/rspeer/python-ftfy

🏋️ Exercises

🟢 Easy

1. Try the clean_text function above on “dirty” Uzbek text.
2. Find phone numbers in text with regex.
3. BERT tokenizer with Uzbek text — how many tokens are produced?

🟡 Medium

1. Custom BPE: Train a BPE tokenizer on 100MB of Uzbek text, compare vocabulary with default bert-multilingual.
2. Sliding window: Split a 50,000-word book into 512-token chunks.
3. Multi-language preprocessor: Class that applies different preprocessing pipelines by language.

🔴 Hard

1. Production text pipeline: FastAPI service that cleans/tokenizes/embeds text streamed from Kafka in real-time.
2. Custom tokenizer service: Custom tokenizer training and inference via REST API.
3. NER + Anonymization: Find PII (personal info) in text and replace with [NAME], [EMAIL], [PHONE] placeholders (for GDPR).

Capstone

notebooks/month-04/05_text_preprocessing.ipynb:

• Collect 10,000 messages from Uzbek Telegram channel posts
• Build a complete cleaning pipeline
• Custom BPE tokenizer
• Compare with pretrained BERT tokenizer (vocab coverage, OOV rate)

✅ Checklist

• I know how to remove HTML, URLs, emails, phones
• What is Unicode normalization (NFKC)
• I can work with regex
• I understand BPE/WordPiece subword tokenization
• Work with HuggingFace tokenizers
• Truncation and padding strategies
• I can train a custom BPE tokenizer
• Multi-language preprocessing pipeline

Moving on to Introduction to Transformers.