Monitor & operate
Monitoring
Production correctness includes data behavior, model behavior and operational health.
production requests, predictions, service metrics and delayed labels
drift/quality/SLO signals + alerts
I monitor both system health and model health; latency alone does not tell me whether the model is still valid.
Mental model
What this means before worrying about syntax.
Monitor data quality, prediction or model quality, drift, endpoint performance and business outcomes, with CloudWatch and SageMaker monitoring capabilities as appropriate.
- Operational health and model health are different.
- Ground truth often arrives later, so true quality monitoring lags input/prediction monitoring.
- Drift is a signal to investigate, not automatic proof of degradation.
production inputs/predictions -> monitoring -> alerts/investigation
Implementation anatomy
The sequence you would actually build or configure.
- 1
Capture representative inputs/outputs where policy allows.
- 2
Create accepted baseline statistics/constraints.
- 3
Monitor service metrics and data/prediction behavior.
- 4
Join delayed labels to predictions for actual model-quality metrics.
- 5
Alert through CloudWatch/ops tooling and follow a response playbook.
Concrete example
A real model-development or production flow.
Endpoint: p95 latency, 5xx, throttles
Input: complaint length/language/missing text
Prediction: category mix/confidence distribution
Delayed labels: per-category precision/recall/F1 and auto-route error rate
Implementation lab
Concrete code, data manipulation and configuration for this lifecycle step.
Operational metrics to emit/capture
operational = {
"endpoint_latency_ms": 84,
"feature_lookup_ms": 11,
"request_count": 1,
"prediction": "Fraud_Unauthorized_Transaction",
"confidence": 0.86,
"model_version": "17",
}Simple drift comparison outside any product-specific monitor
Statistical drift is a signal to investigate; it is not automatic evidence that predictive quality degraded.
from scipy.stats import ks_2samp
# Example numeric feature drift:
reference = baseline_df["complaint_length"]
current = current_df["complaint_length"]
stat, p_value = ks_2samp(reference, current)
print({
"feature": "complaint_length",
"ks_stat": float(stat),
"p_value": float(p_value),
})Join delayed ground truth back to predictions
Model quality requires labels. Prediction logs alone cannot tell you actual accuracy.
scored = predictions.merge(
resolved_labels[
["complaint_id", "final_category"]
],
on="complaint_id",
how="inner",
)
from sklearn.metrics import f1_score
quality = f1_score(
scored["final_category"],
scored["predicted_category"],
average="macro",
)
print("production macro F1:", quality)Design decisions that matter
- Which signals trigger investigation vs retraining.
- Custom monitoring for text/image where tabular drift is insufficient.
- AWS states Model Monitor and Clarify closed to new-customer access on 2026-07-30; confirm account eligibility and design custom alternatives where needed.
Common failure modes
- Model-quality measurement requires ground truth.
- Univariate drift may be benign while concept drift hurts performance.
- Captured data can contain sensitive content and needs retention/access controls.
Production readiness check
Items to close before treating this step as production-ready.