NLP foundations

🎯 Goal

After reading this chapter:

• You will know the main task types of NLP (Natural Language Processing)
• You will be able to build a classical NLP pipeline with spaCy and NLTK
• You will know TF-IDF, Word2Vec, GloVe vector representations
• You will be ready for the HuggingFace ecosystem (deeper in Month 5)

What to learn

• NLP task types — classification, NER, POS, parsing, generation, translation
• Tokenization — word, subword (BPE, WordPiece, SentencePiece), char-level
• Stemming and Lemmatization
• Stop words
• **Bag of Words (BoW)**and TF-IDF
• n-grams
• Word embeddings — Word2Vec, GloVe, FastText
• POS tagging, dependency parsing
• Named Entity Recognition (NER)
• Language detection
• NLP for Uzbek

Libraries

pip install nltk spacy textblob
python -m spacy download en_core_web_sm    # English
python -m spacy download ru_core_news_sm   # Russian (closer to Uzbek)
python -m spacy download xx_ent_wiki_sm    # Multilingual

pip install gensim                          # Word2Vec, topic modeling
pip install langdetect polyglot            # Language detection

pip install scikit-learn                   # TF-IDF

NLP task types

Code examples

NLTK — classical NLP

import nltk
nltk.download(['punkt', 'stopwords', 'wordnet', 'averaged_perceptron_tagger'])

from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer

text = "Natural Language Processing is amazing! It allows computers to understand human language."

# Sentence tokenization
sents = sent_tokenize(text)
# ['Natural Language Processing is amazing!', 'It allows computers to understand human language.']

# Word tokenization
words = word_tokenize(text)
# ['Natural', 'Language', 'Processing', 'is', ...]

# Stop words removal
stop_words = set(stopwords.words('english'))
filtered = [w for w in words if w.lower() not in stop_words and w.isalpha()]

# Stemming
stemmer = PorterStemmer()
stems = [stemmer.stem(w) for w in filtered]
# 'amazing' → 'amaz', 'computers' → 'comput'

# Lemmatization (POS-aware, better)
lemmatizer = WordNetLemmatizer()
lemmas = [lemmatizer.lemmatize(w.lower()) for w in filtered]
# 'computers' → 'computer'

spaCy — modern NLP pipeline

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion in 2024.")

# Tokenization + POS + NER + DEP
for token in doc:
    print(f"{token.text:15s} {token.pos_:10s} {token.dep_:10s} {token.lemma_}")

# Named Entity Recognition
for ent in doc.ents:
    print(f"{ent.text:20s} {ent.label_}")
# Output: # Apple ORG # U.K. GPE # $1 billion MONEY # 2024 DATE

# Noun chunks
for chunk in doc.noun_chunks:
    print(chunk.text)

TF-IDF — Bag of Words

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "Natural language processing is fun",
    "Machine learning powers natural language processing",
    "Deep learning has revolutionized NLP",
    "Backend development requires understanding APIs",
]

vectorizer = TfidfVectorizer(
    max_features=100,
    ngram_range=(1, 2),       # unigrams + bigrams
    stop_words="english",
    min_df=1,
    max_df=0.95,
)

X = vectorizer.fit_transform(corpus)
print(X.shape)                                # (4, 100)
print(vectorizer.get_feature_names_out()[:10])

Text classification — Naive Bayes baseline

from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer

pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2), max_features=5000)),
    ("clf", LogisticRegression(max_iter=1000)),
])

pipeline.fit(train_texts, train_labels)
accuracy = pipeline.score(test_texts, test_labels)

# For new text
prediction = pipeline.predict(["This product is excellent!"])

Word2Vec — embeddings

from gensim.models import Word2Vec

sentences = [
    ["natural", "language", "processing"],
    ["machine", "learning", "models"],
    ["deep", "learning", "neural", "networks"],
    # ...
]

model = Word2Vec(
    sentences,
    vector_size=100,
    window=5,
    min_count=1,
    workers=4,
    epochs=10,
)

# Single word vector
vec = model.wv["natural"]                     # shape (100,)

# Most similar words
similar = model.wv.most_similar("natural", topn=5)

# Cosine similarity between words
sim = model.wv.similarity("language", "processing")

Pretrained embeddings (GloVe)

import gensim.downloader

# 100MB GloVe (Wikipedia 6B tokens)
model = gensim.downloader.load("glove-wiki-gigaword-100")

print(model["king"].shape)                    # (100,)
print(model.most_similar("king", topn=5))
print(model.most_similar(positive=["king", "woman"], negative=["man"]))
# → "queen" close result

Language detection

from langdetect import detect, detect_langs

print(detect("Salom! Mening ismim Ali."))     # uz (or close to uz, in many cases)
print(detect_langs("Hello, how are you?"))    # [en:0.99]

NLP for Uzbek

Current situation

• Resources are limited: few pretrained Uzbek models for nlp
• Good side: multilingual models(mBERT, XLM-R, mT5) support Uzbek partially
• Need to consider both Latin and Cyrillic

Useful resources

• Uzbek models on HuggingFace(search: uzbek)
• OpenAI/Anthropic — GPT-4 and Claude understand Uzbek well (Month 5)
• Whisper — can transcribe Uzbek speech
• Common Voice — Uzbek dataset(Mozilla)

Working with Uzbek text

import spacy

# Multilingual model (partial Uzbek)
nlp = spacy.load("xx_ent_wiki_sm")

text = "Toshkent shahri 2024 yilda yangi loyihalar boshladi."
doc = nlp(text)

for ent in doc.ents:
    print(ent.text, ent.label_)

# Better: HuggingFace XLM-R based (Month 5)

Latin ↔ Cyrillic converter (simple)

LATIN_TO_CYRILLIC = {
    "sh": "ш", "ch": "ч", "yo": "ё", "yu": "ю", "ya": "я", "o'": "ў", "g'": "ғ",
    "a": "а", "b": "б", "d": "д", "e": "е", "f": "ф", "g": "г", "h": "ҳ",
    "i": "и", "j": "ж", "k": "к", "l": "л", "m": "м", "n": "н", "o": "о",
    "p": "п", "q": "қ", "r": "р", "s": "с", "t": "т", "u": "у", "v": "в",
    "x": "х", "y": "й", "z": "з", "'": "ъ",
}

def latin_to_cyrillic(text: str) -> str:
    result = text.lower()
    # 2-character first
    for lat, cyr in sorted(LATIN_TO_CYRILLIC.items(), key=lambda x: -len(x[0])):
        result = result.replace(lat, cyr)
    return result

Backend integration

Text classification API

from fastapi import FastAPI
from pydantic import BaseModel
import joblib

app = FastAPI()
pipeline = joblib.load("text_classifier.joblib")  # TfidfVectorizer + Classifier

class TextInput(BaseModel):
    text: str
    language: str = "en"

@app.post("/classify")
def classify_text(data: TextInput):
    prediction = pipeline.predict([data.text])[0]
    proba = pipeline.predict_proba([data.text])[0]
    
    return {
        "predicted_class": str(prediction),
        "confidence": float(proba.max()),
        "all_probabilities": dict(zip(pipeline.classes_, proba.tolist())),
    }

Sentiment + NER endpoint

import spacy

nlp_en = spacy.load("en_core_web_sm")

@app.post("/analyze")
def analyze_text(data: TextInput):
    doc = nlp_en(data.text)
    
    entities = [
        {"text": ent.text, "type": ent.label_, "start": ent.start_char, "end": ent.end_char}
        for ent in doc.ents
    ]
    
    pos_counts = {}
    for token in doc:
        pos_counts[token.pos_] = pos_counts.get(token.pos_, 0) + 1
    
    return {
        "entities": entities,
        "pos_distribution": pos_counts,
        "tokens": len(doc),
        "sentences": len(list(doc.sents)),
    }

Text similarity service

import gensim.downloader as api
import numpy as np

model = api.load("glove-wiki-gigaword-100")

def text_to_vector(text: str) -> np.ndarray:
    words = text.lower().split()
    vectors = [model[w] for w in words if w in model]
    return np.mean(vectors, axis=0) if vectors else np.zeros(100)

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

@app.post("/similarity")
def similarity(text1: str, text2: str):
    v1 = text_to_vector(text1)
    v2 = text_to_vector(text2)
    return {"similarity": float(cosine_similarity(v1, v2))}

Resources

• NLTK Book — nltk.org/book
• spaCy docs — spacy.io
• “Speech and Language Processing” — Jurafsky & Martin (free PDF — the bible)
• HuggingFace NLP Course — free, preparation for Month 5
• Stanford NLP videos — Chris Manning
• gensim docs — Word2Vec, topic modeling

🏋️ Exercises

🟢 Easy

1. Tokenize a text, remove stop words, lemmatize.
2. POS tagging and NER with spaCy.
3. Compute similarity between 5 documents with TF-IDF.

🟡 Medium

1. News classification: 4-5 categories (BBC dataset), TF-IDF + Logistic Regression, 90%+ accuracy.
2. Spam classifier: SMS Spam dataset, compare Naive Bayes vs LogReg.
3. NER pipeline: Find named entities in text and group by type.

🔴 Hard

1. Uzbek text classifier: Collect your own dataset from Telegram channels (2-3 categories), TF-IDF + LR baseline.
2. NER service: FastAPI + spaCy + caching (Redis) — optimize for high RPS.
3. Topic modeling: Split 1000+ documents into topics with LDA or BERTopic, visualize.

Capstone

notebooks/month-04/04_nlp_basics.ipynb:

• **Project:**Dataset from Uzbek news (Daryo.uz, Day.uz) or Telegram channels
• Baseline classifier with TF-IDF + Logistic Regression
• NER with spaCy multilingual
• Train Word2Vec and find similar words
• FastAPI service

✅ Checklist

• I know the difference between tokenization, stemming, lemmatization
• I know how to use BoW and TF-IDF
• NER, POS, parsing with spaCy
• I use Word2Vec and GloVe embeddings
• Text classification baseline (TF-IDF + LR)
• I know NLP limitations for Uzbek
• I can create NLP endpoints in FastAPI

Moving on to Text Preprocessing.