MODELCATALOG

Predict from whatever is nearby.

sklearn.neighbors

neighbors

Predict from whatever is nearby.

1 Get data Something to practise on
2 Split & tune An honest test set
3 Prepare Into numbers a model accepts
4 Compose One object, no leakage
5 Model The algorithm itself
6 Unsupervised No labels available
7 Measure Is it any good, and why
8 Utilities Glue and extension

What it is for

No training phase at all — the cost is entirely at predict time. NearestNeighbors is also the raw similarity search, which is useful for deduplication and retrieval independent of any prediction task.

Contents

KNeighborsClassifier / Regressor Prediction by nearest examples
NearestNeighbors common Raw similarity search — deduplication, entity matching
LocalOutlierFactor Anomaly detection by local density
KernelDensity Density estimation

Worked examples 1

Runnable, with the arguments that change the outcome called out. Output shapes are shown as comments where they help.

Similarity search for deduplication

NearestNeighbors without any prediction task attached. This is the core of complaint deduplication and entity matching.

from sklearn.neighbors import NearestNeighbors
from sklearn.feature_extraction.text import TfidfVectorizer

vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5))
X = vec.fit_transform(complaint_texts)

nn = NearestNeighbors(n_neighbors=5, metric="cosine").fit(X)
distances, indices = nn.kneighbors(X)

# Skip column 0 - every row is its own nearest neighbour
THRESHOLD = 0.25          # cosine distance; tune against human judgement
for i, (d, idx) in enumerate(zip(distances, indices)):
    dupes = [(j, dist) for j, dist in zip(idx[1:], d[1:]) if dist < THRESHOLD]
    if dupes:
        print(f"\nrow {i}: {complaint_texts[i][:60]}")
        for j, dist in dupes:
            print(f"   ~{dist:.3f}  {complaint_texts[j][:60]}")

# The threshold IS the job here. There is no principled way to set it
# without labelled pairs, so budget time for tuning it.