MODELCATALOG

Linear and logistic regression, and their many variants.

sklearn.linear_model

linear_model

Linear and logistic regression, and their many variants.

1 Get data Something to practise on
2 Split & tune An honest test set
3 Prepare Into numbers a model accepts
4 Compose One object, no leakage
5 Model The algorithm itself
6 Unsupervised No labels available
7 Measure Is it any good, and why
8 Utilities Glue and extension

What it is for

Prediction as a weighted sum of features. Unfashionable and still everywhere, because the weights are the explanation rather than an approximation of one.

In credit decisioning this is not a baseline you move on from. It is the incumbent, for regulatory reasons rather than technical ones.

Classification

LogisticRegression bank Despite the name, this is classification. The scorecard workhorse
LogisticRegressionCV Same with built-in cross-validated regularisation
SGDClassifier Huge datasets — learns incrementally via partial_fit

Regression

LinearRegression Plain least squares
Ridge common L2 penalty. The default when features are correlated
Lasso L1 penalty — drives coefficients to exactly zero, so it selects features
ElasticNet Both penalties
QuantileRegressor Predict a quantile rather than the mean
PoissonRegressor, GammaRegressor, TweedieRegressor Counts and insurance-style targets
HuberRegressor, RANSACRegressor, TheilSenRegressor Robust to outliers
BayesianRidge, ARDRegression Gives uncertainty estimates

Worked examples 4

Runnable, with the arguments that change the outcome called out. Output shapes are shown as comments where they help.

sklearn.linear_model

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(
    max_iter=1000,             # default 100 is too low for real data
    class_weight="balanced",   # first thing to try on imbalance
    C=1.0,                     # regularisation: SMALLER is STRONGER
    solver="lbfgs",            # 'saga' for L1 or very large data
)
clf.fit(X_train, y_train)

# The coefficients ARE the model - not a post-hoc estimate of it.
# This is the property that keeps scorecards in production.
import numpy as np
terms = vectorizer.get_feature_names_out()
for i, label in enumerate(clf.classes_):
    top = np.argsort(clf.coef_[i])[-10:][::-1]
    print(label, "<-", ", ".join(terms[top]))

Logistic regression, and reading the coefficients

The coefficients ARE the model. Not an approximation of it. This is the property that keeps scorecards in production.

from sklearn.linear_model import LogisticRegression
import numpy as np

clf = LogisticRegression(
    max_iter=1000,             # default 100 is too low for real data
    class_weight="balanced",   # first thing to try on imbalance
    C=1.0,                     # regularisation - SMALLER IS STRONGER
    solver="lbfgs",            # 'saga' for L1 or very large data
)
clf.fit(X_train, y_train)

# Odds ratios are the interpretable form: exp(coef) is the multiplicative
# effect on the odds per one-unit increase in the feature.
for name, coef in zip(feature_names, clf.coef_[0]):
    print(f"  {name:26} {coef:+.3f}   odds x{np.exp(coef):.2f}")

#   months_delinquent          +0.847   odds x2.33
#   income_scaled              -0.412   odds x0.66
#   n_products                 -0.203   odds x0.82
#
# "Each additional month delinquent multiplies the odds of default by
# 2.33" is a sentence you can put in a model document and defend.

Lasso for automatic feature selection

L1 regularisation drives coefficients to exactly zero rather than merely small, so the model selects features as a side effect of fitting.

from sklearn.linear_model import LassoCV
import numpy as np

# LassoCV picks the penalty strength by cross-validation for you
las = LassoCV(cv=5, random_state=42, n_alphas=100, max_iter=5000)
las.fit(X_train_scaled, y_train)

kept = np.array(feature_names)[las.coef_ != 0]
print(f"kept {len(kept)} of {len(feature_names)} features")
print(f"alpha chosen: {las.alpha_:.5f}")
for n, c in sorted(zip(kept, las.coef_[las.coef_ != 0]),
                   key=lambda x: -abs(x[1]))[:10]:
    print(f"  {n:26} {c:+.4f}")

# Lasso REQUIRES scaled features. Unscaled, the penalty falls unevenly
# and it will drop features purely because their units are large.

Predict a quantile rather than the mean

For liquidity and capital work the tail is the answer, not the central estimate.

from sklearn.linear_model import QuantileRegressor

# The 5th percentile: a conservative floor to plan a buffer against
low = QuantileRegressor(quantile=0.05, alpha=0.0, solver="highs")
mid = QuantileRegressor(quantile=0.50, alpha=0.0, solver="highs")
high = QuantileRegressor(quantile=0.95, alpha=0.0, solver="highs")

for name, m in [("p05", low), ("p50", mid), ("p95", high)]:
    m.fit(X_train, y_train)
    print(name, m.predict(X_test[:3]).round(0))

# p05 [ 82000.  91000.  77000.]
# p50 [104000. 115000.  98000.]
# p95 [131000. 144000. 126000.]
#
# Three separate models. Ordinary least squares gives you the middle
# row only, which cannot support a question about how much cash to hold.