sklearn.datasets

datasets

Practice data, real and synthetic.

1 Get data Something to practise on

2 Split & tune An honest test set

3 Prepare Into numbers a model accepts

4 Compose One object, no leakage

5 Model The algorithm itself

6 Unsupervised No labels available

7 Measure Is it any good, and why

8 Utilities Glue and extension

What it is for

Bundled toy sets for instant experiments, fetchers for real datasets, and generators that let you manufacture data with exactly the properties you want to test against.

Bundled — load instantly

`load_iris, load_wine, load_digits`	Tiny classification sets. Instant, no download
`load_breast_cancer`	Small binary classification
`load_diabetes`	Small regression

Fetched — downloads once, caches

`fetch_california_housing`	Real regression problem
`fetch_20newsgroups`	The classic text classification set
`fetch_openml`	Gateway to thousands of real datasets by name or ID

Generated — you control the properties

The most useful of the three for learning. You decide the class balance, the noise level and how many features carry signal, which lets you test whether your pipeline handles imbalance before pointing it at real data.

`make_classification`	Set weights= for imbalance, n_informative= for signal
`make_regression`	Continuous target
`make_blobs`	Clustering demos

Worked examples 3

Runnable, with the arguments that change the outcome called out. Output shapes are shown as comments where they help.

sklearn.datasets

from sklearn.datasets import make_classification

# Manufacture a fraud-shaped problem: 0.5% positives, mostly noise features
X, y = make_classification(
    n_samples=50_000, n_features=30,
    n_informative=6,          # only 6 of 30 carry signal
    weights=[0.995, 0.005],   # 0.5% positive - realistic for fraud
    flip_y=0.01,              # label noise
    random_state=42,
)
print(y.mean())   # 0.005

Manufacture data with the properties you want to test

The most underused function in the library. Before pointing a pipeline at real data, generate a problem with known imbalance and known signal so you can check the pipeline behaves.

from sklearn.datasets import make_classification
import numpy as np

X, y = make_classification(
    n_samples=50_000,
    n_features=30,
    n_informative=6,          # only 6 of 30 actually carry signal
    n_redundant=4,            # 4 are linear combinations of those
    weights=[0.995, 0.005],   # 0.5% positive - fraud-shaped
    flip_y=0.01,              # 1% label noise, like real annotation
    class_sep=0.8,            # lower = harder problem
    random_state=42,
)

print(X.shape, y.mean())
# (50000, 30) 0.0052

# Now you can verify your pipeline handles this BEFORE real data:
# does it still find the positives? does the metric you chose notice
# when it stops finding them?

Load a real text dataset for NLP practice

20 newsgroups is the standard text classification benchmark. Remove the metadata or you will get suspiciously good results.

from sklearn.datasets import fetch_20newsgroups

cats = ["rec.autos", "sci.med", "comp.graphics", "talk.politics.guns"]

train = fetch_20newsgroups(subset="train", categories=cats,
                           remove=("headers", "footers", "quotes"),
                           random_state=42)
test  = fetch_20newsgroups(subset="test", categories=cats,
                           remove=("headers", "footers", "quotes"),
                           random_state=42)

print(len(train.data), train.target_names)
print(train.data[0][:200])

# remove= matters enormously. Headers contain the newsgroup name, so
# without it a trivial model scores ~0.95 by reading the answer.
# This is a textbook leakage demonstration.