Evaluate & tune
Evaluation
Separates model fitting from the evidence used to accept or reject a model.
immutable model candidate + untouched test population
evaluation.json + acceptance evidence
I treat evaluation as an explicit lifecycle gate rather than assuming a successful training job means a production-ready model.
Mental model
What this means before worrying about syntax.
Evaluate on validation or test data using metrics appropriate to the use case and persist those metrics for comparison and governance.
- Evaluation is an explicit acceptance gate after training.
- Aggregate metrics are not enough; inspect classes, slices and operating thresholds.
- The test set should remain untouched until candidate selection is substantially complete.
trained model + held-out data -> metrics -> acceptance decision
Implementation anatomy
The sequence you would actually build or configure.
- 1
Run predictions on held-out data.
- 2
Calculate task-appropriate metrics.
- 3
Review confusion matrix/per-class results.
- 4
Evaluate business slices.
- 5
Test thresholds and operational volume.
- 6
Persist metrics and acceptance result.
Concrete example
A real model-development or production flow.
Macro F1 highlights weak small categories
Confusion matrix shows fraud vs card-payment confusion
0.75 confidence threshold controls auto-route coverage
Track coverage and error rate among auto-routed complaints
Implementation lab
Concrete code, data manipulation and configuration for this lifecycle step.
Per-class metrics + confusion matrix
from sklearn.metrics import (
classification_report,
confusion_matrix,
f1_score,
)
pred = pipeline.predict(test_df["text"])
print("macro F1:", f1_score(
test_df["label"], pred, average="macro"
))
print(classification_report(test_df["label"], pred))
cm = confusion_matrix(
test_df["label"],
pred,
labels=pipeline.classes_,
)
print(cm)Threshold sweep for operational trade-off
This turns a vague '0.75 confidence' rule into evidence: what volume is automated and what error rate remains?
import numpy as np
import pandas as pd
proba = pipeline.predict_proba(test_df["text"])
best_idx = proba.argmax(axis=1)
pred = pipeline.classes_[best_idx]
confidence = proba.max(axis=1)
rows = []
for threshold in np.arange(0.50, 0.96, 0.05):
mask = confidence >= threshold
coverage = mask.mean()
accuracy = (
(pred[mask] == test_df.loc[mask, "label"]).mean()
if mask.any() else np.nan
)
rows.append({
"threshold": round(float(threshold), 2),
"coverage": coverage,
"accepted_accuracy": accuracy,
})
print(pd.DataFrame(rows))Write machine-readable evaluation.json
A pipeline condition or approval process can consume this artifact instead of scraping notebook output.
import json
from pathlib import Path
metrics = {
"macro_f1": float(macro_f1),
"auto_route_threshold": 0.75,
"auto_route_coverage": float(coverage),
"auto_route_accuracy": float(accepted_accuracy),
}
Path("/opt/ml/processing/evaluation").mkdir(
parents=True, exist_ok=True
)
Path("/opt/ml/processing/evaluation/evaluation.json").write_text(
json.dumps(metrics, indent=2)
)Design decisions that matter
- Define acceptance metrics from business loss.
- Set threshold before production.
- Champion/challenger comparison.
Common failure modes
- Repeated tuning on the test set turns it into validation data.
- Aggregate accuracy can hide high-risk segment failures.
- Offline results can be invalidated by serving skew.
Production readiness check
Items to close before treating this step as production-ready.