sklearn.feature_selection

feature_selection

Which features actually earn their place.

1 Get data Something to practise on

2 Split & tune An honest test set

3 Prepare Into numbers a model accepts

4 Compose One object, no leakage

5 Model The algorithm itself

6 Unsupervised No labels available

7 Measure Is it any good, and why

8 Utilities Glue and extension

What it is for

Reducing the feature set. Mostly useful for speed, interpretability and regulatory defensibility rather than accuracy — modern models tolerate irrelevant features better than they used to.

`VarianceThreshold`	Drop near-constant columns
`SelectKBest, SelectPercentile`	Univariate scoring. Pair with chi2, f_classif or mutual_info_classif
`SelectFromModel` common	Keep what a fitted model considered important
`RFE, RFECV`	Recursive elimination. Thorough, slow
`SequentialFeatureSelector`	Forward or backward stepwise

Worked examples 2

Runnable, with the arguments that change the outcome called out. Output shapes are shown as comments where they help.

Select features inside the Pipeline, never before it

Selecting on the full dataset is leakage. Inside a Pipeline the selection is refit on each training fold, which is correct.

from sklearn.feature_selection import SelectKBest, chi2, SelectFromModel
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

# Univariate: score each feature against the target independently.
# chi2 needs non-negative input, so it suits TF-IDF.
pipe = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2))),
    ("select", SelectKBest(chi2, k=5000)),      # 200k features -> 5k
    ("clf", LogisticRegression(max_iter=1000)),
])

# Model-based: keep what a fitted model considered important.
# L1 regularisation drives coefficients to exactly zero, so it selects.
pipe = Pipeline([
    ("select", SelectFromModel(
        LogisticRegression(penalty="l1", solver="liblinear", C=0.1),
        threshold="median")),
    ("clf", LogisticRegression(max_iter=1000)),
])

pipe.fit(X_train, y_train)
mask = pipe.named_steps["select"].get_support()
print(f"kept {mask.sum()} of {len(mask)} features")

See which features survived

Selection is only useful if you can report what it kept. This is the part that goes into a model document.

from sklearn.feature_selection import SelectKBest, f_classif
import numpy as np

sel = SelectKBest(f_classif, k=10).fit(X_train, y_train)

names  = np.array(feature_names)
scores = sel.scores_
keep   = sel.get_support()

order = np.argsort(scores)[::-1]
for i in order[:15]:
    mark = "KEPT" if keep[i] else "    "
    print(f"  {mark}  {names[i]:28} F={scores[i]:8.1f}  p={sel.pvalues_[i]:.2e}")

# f_classif is a univariate ANOVA F-test: it cannot see interactions.
# A feature that only matters in combination with another scores low
# and gets dropped. That is the main limitation of univariate selection.

Worth knowing

Put selection INSIDE the Pipeline. Selecting on the full dataset before splitting is leakage.