MODELCATALOG

Learn how raw text becomes model-ready units.

NLP / Foundations / Basic

2. Text Preprocessing and Tokenization

Learn how raw text becomes model-ready units.

01 Foundations Represent language 02 Classical NLP Build baselines 03 Neural NLP Learn sequence 04 Transformers Learn attention 05 Applications Apply models 06 Engineering Operate safely 07 Capstone Integrate

01 Learning objectives

By the end of this module, you should be able to explain the concept in plain language, connect it to a use case, and recognize when it is — and is not — appropriate.

  1. Objective 1

    Understand normalization, tokenization, stop words, stemming and lemmatization.

  2. Objective 2

    Know when preprocessing helps and when it destroys information.

  3. Objective 3

    Understand subword tokenization used by Transformers.

02 Pipeline

Traditional NLP often uses:

raw text → clean/normalize → tokenize → optional stemming/lemmatization → vectorize

Modern Transformer pipelines often use much lighter manual cleaning because punctuation, casing, and word pieces may contain useful signal.

03 Tokenization

A token is a unit presented to a model. Depending on the tokenizer, unbelievable may be one word token or several subword pieces.

Subword tokenization helps with:

  • out-of-vocabulary words,
  • rare words,
  • morphology,
  • manageable vocabulary size.

04 Code: simple preprocessing 

import re

def normalize_text(text):
    text = text.lower()
    text = re.sub(r"\s+", " ", text)
    return text.strip()

text = "My ACCOUNT   shows an unknown debit."
print(normalize_text(text))

05 Pitfalls and nuances

These are the mistakes most likely to create a wrong mental model or a weak implementation.

  1. Pitfall 1

    Removing negation words such as `not` as stop words.

  2. Pitfall 2

    Lowercasing when case conveys meaning (IDs, acronyms, entities).

  3. Pitfall 3

    Applying aggressive stemming before a pretrained Transformer.

06 Checkpoint questions

Try answering these without looking back at the module. If the explanation is clear in your own words, you are ready to continue.

  1. Check 1

    Why can removing `not` be harmful?

  2. Check 2

    Why do Transformer tokenizers use subwords?