Evaluate & tune
Tuning & experiments
Makes model iteration traceable and comparable.
search space + objective metric + repeatable training job
ranked candidate runs + selected configuration
The goal is reproducibility and model selection, not tuning for its own sake.
Mental model
What this means before worrying about syntax.
Track configurations and metrics across runs; tune hyperparameters where the expected gain justifies the compute.
- Experiments make data/code/config/metrics comparable across runs.
- Tuning is repeated training over a bounded search space.
- Reproducibility requires more than hyperparameters.
candidate configs -> training runs -> metrics -> selected candidate
Implementation anatomy
The sequence you would actually build or configure.
- 1
Choose meaningful parameters.
- 2
Define objective metric/direction.
- 3
Bound search ranges.
- 4
Track image/code/data versions.
- 5
Select candidate on validation.
- 6
Confirm on independent test data.
Concrete example
A real model-development or production flow.
TF-IDF min_df {2,5,10}
ngram_range {(1,1),(1,2)}
LogisticRegression C on log scale
class_weight None vs balanced
objective = validation macro F1
Implementation lab
Concrete code, data manipulation and configuration for this lifecycle step.
Simple reproducible search before managed tuning
The search-space design matters more than the tuner API. Know what each hyperparameter changes.
from sklearn.model_selection import ParameterGrid
from sklearn.metrics import f1_score
grid = ParameterGrid({
"tfidf__min_df": [2, 5, 10],
"tfidf__ngram_range": [(1, 1), (1, 2)],
"clf__C": [0.1, 1.0, 10.0],
})
results = []
for params in grid:
pipeline.set_params(**params)
pipeline.fit(train_df["text"], train_df["label"])
pred = pipeline.predict(val_df["text"])
results.append({
**params,
"macro_f1": f1_score(
val_df["label"], pred, average="macro"
)
})
best = max(results, key=lambda x: x["macro_f1"])
print(best)Hyperparameters as ModelTrainer inputs
trainer = ModelTrainer(
training_image=SKLEARN_TRAINING_IMAGE_URI,
source_code=source_code,
role=SAGEMAKER_ROLE_ARN,
hyperparameters={
"C": 1.0,
"min_df": 5,
"ngram_max": 2,
"class_weight": "balanced",
},
)Design decisions that matter
- Tuning budget vs likely value.
- Early stopping for expensive models.
- One metric vs multi-constraint selection.
Common failure modes
- Tiny validation differences can be noise.
- A run is not reproducible if its data snapshot/image is unknown.
- Hyperparameter search cannot repair target leakage.
Production readiness check
Items to close before treating this step as production-ready.