MODELCATALOG

Fewer dimensions, most of the information.

sklearn.decomposition

decomposition

Fewer dimensions, most of the information.

1 Get data Something to practise on
2 Split & tune An honest test set
3 Prepare Into numbers a model accepts
4 Compose One object, no leakage
5 Model The algorithm itself
6 Unsupervised No labels available
7 Measure Is it any good, and why
8 Utilities Glue and extension

What it is for

Compressing many correlated features into a few. The one to know is that PCA needs dense input, so after TF-IDF you use TruncatedSVD instead — same idea, works on sparse matrices.

Contents

PCA common Dimensionality reduction on dense data
TruncatedSVD common The sparse equivalent. What you use after TF-IDF. Also called LSA
NMF Non-negative, parts-based. Topic modelling
LatentDirichletAllocation Classical topic modelling
FastICA Source separation
IncrementalPCA Data too large for memory

Worked examples 2

Runnable, with the arguments that change the outcome called out. Output shapes are shown as comments where they help.

PCA, and how many components to keep

Do not guess. Ask for a proportion of variance and let it choose the number.

from sklearn.decomposition import PCA
import numpy as np

# Pass a float < 1 and it keeps however many components reach that
# proportion of explained variance
pca = PCA(n_components=0.95, random_state=42).fit(X_scaled)

print(f"{pca.n_components_} components retain 95% of variance "
      f"(from {X_scaled.shape[1]})")
print(np.cumsum(pca.explained_variance_ratio_)[:10].round(3))

X_reduced = pca.transform(X_scaled)

# PCA REQUIRES scaling first. Without it, whichever feature has the
# largest units dominates every component, and you have effectively
# just selected that column.

The sparse equivalent, for text

PCA cannot take sparse input. After TF-IDF you want TruncatedSVD — same idea, and in a text context it is called latent semantic analysis.

from sklearn.decomposition import TruncatedSVD
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import Normalizer

# TF-IDF gives 200k sparse features. Reduce to 300 dense ones.
lsa = make_pipeline(
    TruncatedSVD(n_components=300, random_state=42),
    Normalizer(copy=False),      # standard after SVD for cosine work
)

X_dense = lsa.fit_transform(X_tfidf)
print(X_tfidf.shape, "->", X_dense.shape)

svd = lsa.named_steps["truncatedsvd"]
print(f"variance retained: {svd.explained_variance_ratio_.sum():.1%}")

# Now usable by models that refuse sparse input - KNeighbors,
# GaussianNB, most clustering.