Основы NLP

🎯 Цель

После прочтения этой главы:

• Знаете основные типы задач NLP (Natural Language Processing)
• Можете строить классический NLP pipeline через spaCy и NLTK
• Знаете векторные representations TF-IDF, Word2Vec, GloVe
• Готовы к экосистеме HuggingFace (глубже в Месяце 5)

Что нужно изучить

• Типы NLP задач — classification, NER, POS, parsing, generation, translation
• Tokenization — word, subword (BPE, WordPiece, SentencePiece), char-level
• Stemming и Lemmatization
• Stop words
• **Bag of Words (BoW)**и TF-IDF
• n-grams
• Word embeddings — Word2Vec, GloVe, FastText
• POS tagging, dependency parsing
• Named Entity Recognition (NER)
• Language detection
• NLP для узбекского языка

Библиотеки

pip install nltk spacy textblob
python -m spacy download en_core_web_sm    # English
python -m spacy download ru_core_news_sm   # Russian (ближе для узбекского)
python -m spacy download xx_ent_wiki_sm    # Multilingual

pip install gensim                          # Word2Vec, topic modeling
pip install langdetect polyglot            # Language detection

pip install scikit-learn                   # TF-IDF

Типы задач NLP

Примеры кода

NLTK — классический NLP

import nltk
nltk.download(['punkt', 'stopwords', 'wordnet', 'averaged_perceptron_tagger'])

from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer

text = "Natural Language Processing is amazing! It allows computers to understand human language."

# Sentence tokenization
sents = sent_tokenize(text)
# ['Natural Language Processing is amazing!', 'It allows computers to understand human language.']

# Word tokenization
words = word_tokenize(text)
# ['Natural', 'Language', 'Processing', 'is', ...]

# Stop words removal
stop_words = set(stopwords.words('english'))
filtered = [w for w in words if w.lower() not in stop_words and w.isalpha()]

# Stemming
stemmer = PorterStemmer()
stems = [stemmer.stem(w) for w in filtered]
# 'amazing' → 'amaz', 'computers' → 'comput'

# Lemmatization (POS-aware, лучше)
lemmatizer = WordNetLemmatizer()
lemmas = [lemmatizer.lemmatize(w.lower()) for w in filtered]
# 'computers' → 'computer'

spaCy — современный NLP pipeline

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion in 2024.")

# Tokenization + POS + NER + DEP
for token in doc:
    print(f"{token.text:15s} {token.pos_:10s} {token.dep_:10s} {token.lemma_}")

# Named Entity Recognition
for ent in doc.ents:
    print(f"{ent.text:20s} {ent.label_}")
# Output:
# Apple                ORG
# U.K.                 GPE
# $1 billion           MONEY
# 2024                 DATE

# Noun chunks
for chunk in doc.noun_chunks:
    print(chunk.text)

TF-IDF — Bag of Words

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "Natural language processing is fun",
    "Machine learning powers natural language processing",
    "Deep learning has revolutionized NLP",
    "Backend development requires understanding APIs",
]

vectorizer = TfidfVectorizer(
    max_features=100,
    ngram_range=(1, 2),       # unigrams + bigrams
    stop_words="english",
    min_df=1,
    max_df=0.95,
)

X = vectorizer.fit_transform(corpus)
print(X.shape)                                # (4, 100)
print(vectorizer.get_feature_names_out()[:10])

Text classification — Naive Bayes baseline

from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer

pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2), max_features=5000)),
    ("clf", LogisticRegression(max_iter=1000)),
])

pipeline.fit(train_texts, train_labels)
accuracy = pipeline.score(test_texts, test_labels)

# Для нового текста
prediction = pipeline.predict(["This product is excellent!"])

Word2Vec — embeddings

from gensim.models import Word2Vec

sentences = [
    ["natural", "language", "processing"],
    ["machine", "learning", "models"],
    ["deep", "learning", "neural", "networks"],
    # ...
]

model = Word2Vec(
    sentences,
    vector_size=100,
    window=5,
    min_count=1,
    workers=4,
    epochs=10,
)

# Вектор одного слова
vec = model.wv["natural"]                     # shape (100,)

# Наиболее похожие слова
similar = model.wv.most_similar("natural", topn=5)

# Cosine similarity между словами
sim = model.wv.similarity("language", "processing")

Pretrained embeddings (GloVe)

import gensim.downloader

# 100MB GloVe (Wikipedia 6B tokens)
model = gensim.downloader.load("glove-wiki-gigaword-100")

print(model["king"].shape)                    # (100,)
print(model.most_similar("king", topn=5))
print(model.most_similar(positive=["king", "woman"], negative=["man"]))
# → результат близкий к "queen"

Language detection

from langdetect import detect, detect_langs

print(detect("Salom! Mening ismim Ali."))     # uz (или uz hidoyat, в большинстве случаев)
print(detect_langs("Hello, how are you?"))    # [en:0.99]

NLP для узбекского языка

Текущая ситуация

• Ресурсов мало: pretrained узбекских моделей для nlp немного
• Плюсы: multilingual модели(mBERT, XLM-R, mT5) частично поддерживают узбекский язык
• Нужно учитывать и Latin, и Кириллицу

Полезные ресурсы

• Узбекские модели на HuggingFace(поиск: uzbek)
• OpenAI/Anthropic — GPT-4 и Claude хорошо понимают узбекский язык (Месяц 5)
• Whisper — может транскрибировать узбекскую речь
• Common Voice — Uzbek dataset(Mozilla)

Работа с узбекским текстом

import spacy

# Multilingual model (узбекский частично)
nlp = spacy.load("xx_ent_wiki_sm")

text = "Toshkent shahri 2024 yilda yangi loyihalar boshladi."
doc = nlp(text)

for ent in doc.ents:
    print(ent.text, ent.label_)

# Better: HuggingFace XLM-R based (Месяц 5)

Latin ↔ Кириллица конвертер (простой)

LATIN_TO_CYRILLIC = {
    "sh": "ш", "ch": "ч", "yo": "ё", "yu": "ю", "ya": "я", "o'": "ў", "g'": "ғ",
    "a": "а", "b": "б", "d": "д", "e": "е", "f": "ф", "g": "г", "h": "ҳ",
    "i": "и", "j": "ж", "k": "к", "l": "л", "m": "м", "n": "н", "o": "о",
    "p": "п", "q": "қ", "r": "р", "s": "с", "t": "т", "u": "у", "v": "в",
    "x": "х", "y": "й", "z": "з", "'": "ъ",
}

def latin_to_cyrillic(text: str) -> str:
    result = text.lower()
    # 2-character first
    for lat, cyr in sorted(LATIN_TO_CYRILLIC.items(), key=lambda x: -len(x[0])):
        result = result.replace(lat, cyr)
    return result

Интеграция с backend

Text classification API

from fastapi import FastAPI
from pydantic import BaseModel
import joblib

app = FastAPI()
pipeline = joblib.load("text_classifier.joblib")  # TfidfVectorizer + Classifier

class TextInput(BaseModel):
    text: str
    language: str = "en"

@app.post("/classify")
def classify_text(data: TextInput):
    prediction = pipeline.predict([data.text])[0]
    proba = pipeline.predict_proba([data.text])[0]
    
    return {
        "predicted_class": str(prediction),
        "confidence": float(proba.max()),
        "all_probabilities": dict(zip(pipeline.classes_, proba.tolist())),
    }

Sentiment + NER endpoint

import spacy

nlp_en = spacy.load("en_core_web_sm")

@app.post("/analyze")
def analyze_text(data: TextInput):
    doc = nlp_en(data.text)
    
    entities = [
        {"text": ent.text, "type": ent.label_, "start": ent.start_char, "end": ent.end_char}
        for ent in doc.ents
    ]
    
    pos_counts = {}
    for token in doc:
        pos_counts[token.pos_] = pos_counts.get(token.pos_, 0) + 1
    
    return {
        "entities": entities,
        "pos_distribution": pos_counts,
        "tokens": len(doc),
        "sentences": len(list(doc.sents)),
    }

Text similarity service

import gensim.downloader as api
import numpy as np

model = api.load("glove-wiki-gigaword-100")

def text_to_vector(text: str) -> np.ndarray:
    words = text.lower().split()
    vectors = [model[w] for w in words if w in model]
    return np.mean(vectors, axis=0) if vectors else np.zeros(100)

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

@app.post("/similarity")
def similarity(text1: str, text2: str):
    v1 = text_to_vector(text1)
    v2 = text_to_vector(text2)
    return {"similarity": float(cosine_similarity(v1, v2))}

Ресурсы

• NLTK Book — nltk.org/book
• spaCy docs — spacy.io
• “Speech and Language Processing” — Jurafsky & Martin (free PDF — библия)
• HuggingFace NLP Course — бесплатно, подготовка к Месяцу 5
• Stanford NLP videos — Chris Manning
• gensim docs — Word2Vec, topic modeling

🏋️ Упражнения

🟢 Easy

1. Tokenize текст, удалите stop words, lemmatize.
2. POS tagging и NER через spaCy.
3. Через TF-IDF рассчитайте похожесть между 5 документами.

🟡 Medium

1. News classification: 4-5 категорий (BBC dataset), TF-IDF + Logistic Regression, 90%+ accuracy.
2. Spam classifier: SMS Spam dataset, сравнение Naive Bayes vs LogReg.
3. NER pipeline: поиск именованных объектов в тексте, группировка по типам.

🔴 Hard

1. Uzbek text classifier: соберите свой dataset с Telegram-каналов (2-3 категории), TF-IDF + LR baseline.
2. NER service: FastAPI + spaCy + caching (Redis) — оптимизация для высокого RPS.
3. Topic modeling: разделите 1000+ документов на topic’и через LDA или BERTopic, визуализируйте.

Capstone

notebooks/month-04/04_nlp_basics.ipynb:

• **Проект:**Узбекоязычные новости (Daryo.uz, Kun.uz) или dataset с Telegram-каналов
• Baseline classifier через TF-IDF + Logistic Regression
• NER через spaCy multilingual
• Обучите Word2Vec и найдите похожие слова
• Сервис FastAPI

✅ Чек-лист

• Знаю разницу между tokenization, stemming, lemmatization
• Умею использовать BoW и TF-IDF
• NER, POS, parsing через spaCy
• Использую Word2Vec и GloVe embeddings
• Text classification baseline (TF-IDF + LR)
• Знаю ограничения NLP для узбекского языка
• Могу создавать NLP endpoint в FastAPI

Переходим к Text Preprocessing.