MODELCATALOG

Learn why modern NLP rarely starts from random weights.

NLP / Transformers / Advanced

11. Pretraining, Fine-tuning and Transfer Learning

Learn why modern NLP rarely starts from random weights.

01 Foundations Represent language 02 Classical NLP Build baselines 03 Neural NLP Learn sequence 04 Transformers Learn attention 05 Applications Apply models 06 Engineering Operate safely 07 Capstone Integrate

01 Learning objectives

By the end of this module, you should be able to explain the concept in plain language, connect it to a use case, and recognize when it is — and is not — appropriate.

  1. Objective 1

    Understand pretraining objectives.

  2. Objective 2

    Understand task fine-tuning.

  3. Objective 3

    Know when to freeze, fine-tune or prompt.

02 Pretraining

Large models first learn broad language patterns from very large corpora using self-supervised objectives such as masked-token prediction or next-token prediction.

03 Fine-tuning

Fine-tuning adapts pretrained weights to a downstream task using a smaller labeled dataset.

04 Hugging Face example 

from transformers import AutoTokenizer, AutoModelForSequenceClassification

name = "distilbert-base-uncased"

tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(
    name,
    num_labels=6
)

batch = tokenizer(
    ["My account shows an unauthorized debit."],
    padding=True,
    truncation=True,
    return_tensors="pt"
)

logits = model(**batch).logits

05 Pitfalls and nuances

These are the mistakes most likely to create a wrong mental model or a weak implementation.

  1. Pitfall 1

    Fine-tuning with too high a learning rate.

  2. Pitfall 2

    Ignoring class imbalance and label quality.

  3. Pitfall 3

    Evaluating on data too similar to training examples.

06 Checkpoint questions

Try answering these without looking back at the module. If the explanation is clear in your own words, you are ready to continue.

  1. Check 1

    What knowledge is transferred during fine-tuning?

  2. Check 2

    Why is a tokenizer model-specific?