4. From Sparse Vectors to Embeddings
Understand the representation change that enables modern NLP.
01 Learning objectives
By the end of this module, you should be able to explain the concept in plain language, connect it to a use case, and recognize when it is — and is not — appropriate.
-
Objective 1
Contrast one-hot/TF-IDF vectors with dense embeddings.
-
Objective 2
Understand semantic geometry and cosine similarity.
-
Objective 3
Distinguish static and contextual embeddings.
02 Dense representation
A one-hot vector can be enormous and contains no learned semantic closeness. An embedding maps a token, sentence or document to a much smaller dense vector.
Example intuition:
king, queen, prince should occupy related regions of embedding space.
03 Static vs contextual
Static embeddings: a word has essentially one vector regardless of sentence context.
Contextual embeddings: the vector for a token changes with surrounding language.
bank in river bank should differ from bank account.
04 Cosine similarity
For semantic search, similarity is often computed using cosine similarity:
cos(A,B) = (A · B) / (||A|| ||B||)
Closer direction → greater semantic similarity.
05 Pitfalls and nuances
These are the mistakes most likely to create a wrong mental model or a weak implementation.
-
Pitfall 1
Assuming nearest vector always means semantically correct for the business task.
-
Pitfall 2
Comparing embeddings from incompatible models or dimensions.
-
Pitfall 3
Ignoring normalization/index configuration in vector search.
06 Checkpoint questions
Try answering these without looking back at the module. If the explanation is clear in your own words, you are ready to continue.
-
Check 1
Why are embeddings dense?
-
Check 2
How can contextual embeddings represent two meanings of `bank`?