Register & govern
Pipelines / MLOps orchestration
Turns manually repeated notebook steps into a reproducible workflow.
step definitions + parameters + artifact dependencies
repeatable model lifecycle execution graph
Pipelines operationalize the lifecycle; they do not replace the underlying training or serving concepts.
Mental model
What this means before worrying about syntax.
Orchestrate preparation, training, evaluation, conditional acceptance, registration and potentially deployment or retraining.
- Pipeline is a dependency graph that operationalizes repeatable lifecycle steps.
- Each step needs explicit input/output contracts.
- Automation should preserve quality and approval gates, not erase them.
process -> train -> evaluate -> condition -> register -> deploy
Implementation anatomy
The sequence you would actually build or configure.
- 1
Processing step prepares data.
- 2
Training step launches managed training.
- 3
Evaluation step emits metrics.
- 4
Condition step tests acceptance.
- 5
Register step creates candidate version.
- 6
Deployment can be automated or held for approval.
Concrete example
A real model-development or production flow.
New monthly data snapshot
Process/split
Train candidate
Evaluate macro F1/category recall
If gates pass register PendingManualApproval
Approved version promoted via controlled deployment
Implementation lab
Concrete code, data manipulation and configuration for this lifecycle step.
Think in step contracts
# PSEUDOCODE
#
# process_step.output("train") ----------+
# process_step.output("validation") -----+--> training_step
#
# training_step.model_artifact ----------+--> evaluation_step
# process_step.output("test") -----------+
#
# evaluation_step.evaluation_json
# |
# v
# condition_step
# / \
# PASS FAIL
# | |
# register_step stop / notifyAcceptance logic should be explicit
The exact Pipeline API may change; this acceptance logic is the business/engineering contract that must remain explicit.
def candidate_passes(metrics: dict) -> bool:
return (
metrics["macro_f1"] >= 0.90
and metrics["fraud_recall"] >= 0.92
and metrics["auto_route_accuracy"] >= 0.98
and metrics["p95_latency_ms"] <= 150
)Design decisions that matter
- Human gate vs automatic promotion.
- Caching/idempotency.
- Parameterize environment/account/buckets.
- Failure/retry behavior.
Common failure modes
- Automating a flawed metric gate makes the flaw faster.
- Do not hard-code secrets.
- Non-idempotent steps complicate retries.
Production readiness check
Items to close before treating this step as production-ready.