Monitor & operate
Retraining loop
Closes the lifecycle rather than treating deployment as the end.
sustained trigger + new labeled data
new candidate that still passes evaluation and approval
Deployment is one state in a continuous lifecycle, not the finish line.
Mental model
What this means before worrying about syntax.
Monitoring or business triggers can initiate new data preparation, retraining, evaluation and governed promotion.
- Retraining is a governed response to evidence, not an automatic response to every alert.
- A retrained model remains a candidate until it passes the same gates as any other model.
- Champion/challenger comparison reduces accidental regression.
monitor -> trigger -> retrain -> evaluate -> register -> redeploy
Implementation anatomy
The sequence you would actually build or configure.
- 1
Trigger from schedule/new labels/sustained drift/quality loss/business change.
- 2
Create new reproducible snapshot.
- 3
Run processing/training/evaluation pipeline.
- 4
Compare challenger to champion.
- 5
Register/approve/deploy only if gates pass.
- 6
Retain rollback.
Concrete example
A real model-development or production flow.
Monthly labeled complaints accumulate
Create time-aware train/holdout
Train challenger
Compare category recall and auto-route error
Register if gates pass
Approve/promote and watch post-cutover metrics
Implementation lab
Concrete code, data manipulation and configuration for this lifecycle step.
Champion / challenger acceptance function
def promote(champion, challenger):
return (
challenger["macro_f1"] >= champion["macro_f1"] + 0.005
and challenger["fraud_recall"] >= 0.92
and challenger["p95_latency_ms"] <= 150
and challenger["data_quality_passed"]
)Trigger only after evidence is sustained
Do not retrain automatically on one transient drift alarm. The trigger should be governed and explainable.
def should_retrain(signals):
return (
signals["new_labeled_rows"] >= 50_000
or signals["quality_below_floor_days"] >= 3
or signals["approved_policy_change"]
)Design decisions that matter
- Trigger frequency from label freshness/model decay.
- Automatic vs manual promotion.
- Warm-start vs full retrain.
Common failure modes
- Bad labels can be amplified by automated retraining.
- Retraining without fresh validation can hide concept drift.
- A new model can improve average metrics while hurting critical segments.
Production readiness check
Items to close before treating this step as production-ready.