MODELCATALOG

Master the strongest classical baseline before moving to neural NLP.

NLP / Classical NLP / Basic → Intermediate

3. Bag of Words, N-grams and TF-IDF

Master the strongest classical baseline before moving to neural NLP.

01 Foundations Represent language 02 Classical NLP Build baselines 03 Neural NLP Learn sequence 04 Transformers Learn attention 05 Applications Apply models 06 Engineering Operate safely 07 Capstone Integrate

01 Learning objectives

By the end of this module, you should be able to explain the concept in plain language, connect it to a use case, and recognize when it is — and is not — appropriate.

  1. Objective 1

    Understand count vectors, n-grams and TF-IDF.

  2. Objective 2

    Know what information these representations preserve and lose.

  3. Objective 3

    Build a text classifier with scikit-learn.

02 Bag of Words

Bag of Words represents a document by vocabulary counts. It is simple, sparse and highly interpretable, but by itself it discards much of word order.

03 N-grams

N-grams partially restore local order.

  • unigram: unauthorized
  • bigram: unauthorized transaction
  • trigram: did not authorize

This is why classical models can remain extremely strong for domain classification.

04 TF-IDF

TF-IDF increases the influence of terms that are frequent in a document but relatively uncommon across the collection.

Conceptually:

TF-IDF(term, doc) = term frequency × inverse document frequency

05 Code: TF-IDF + Logistic Regression 

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

X_train, X_test, y_train, y_test = train_test_split(
    df["clean_text"], df["category"],
    test_size=0.20, random_state=42, stratify=df["category"]
)

model = Pipeline([
    ("tfidf", TfidfVectorizer(
        lowercase=True,
        ngram_range=(1, 2),
        min_df=2
    )),
    ("clf", LogisticRegression(max_iter=1000))
])

model.fit(X_train, y_train)
pred = model.predict(X_test)

06 Pitfalls and nuances

These are the mistakes most likely to create a wrong mental model or a weak implementation.

  1. Pitfall 1

    Data leakage by fitting the vectorizer on the full dataset before the train/test split.

  2. Pitfall 2

    Using accuracy alone on highly imbalanced categories.

  3. Pitfall 3

    Exploding vocabulary size with large n-gram ranges.

07 Checkpoint questions

Try answering these without looking back at the module. If the explanation is clear in your own words, you are ready to continue.

  1. Check 1

    Why must `fit_transform` be done on training text only?

  2. Check 2

    What does a bigram capture that a unigram cannot?