Build & train
Training container anatomy
Explains where your code, input channels and model output physically appear.
image + code + mounted channels + hyperparameters
logs + files under /opt/ml/model
The container is the execution environment; channels are mounted inputs, and my script writes the trained artifact to the SageMaker model directory.
Mental model
What this means before worrying about syntax.
The training image provides the runtime/framework. SageMaker exposes job inputs and expects model output in SageMaker-defined locations.
- The container is the runtime contract for training.
- SageMaker supplies inputs/configuration; your code produces trained assets.
- Training and serving containers/contracts can differ.
framework image + your script + /opt/ml/input/data/* -> train -> /opt/ml/model
Implementation anatomy
The sequence you would actually build or configure.
- 1
Container starts on managed training compute.
- 2
Inputs appear under SageMaker-defined paths/environment.
- 3
Entry point parses hyperparameters and channel paths.
- 4
Logs flow to CloudWatch.
- 5
Assets written to /opt/ml/model are packaged/uploaded.
Concrete example
A real model-development or production flow.
/opt/ml/input/data/train -> train channel
/opt/ml/input/data/validation -> validation channel
/opt/ml/model/vectorizer.joblib
/opt/ml/model/model.joblib
SageMaker packages model directory to S3
Implementation lab
Concrete code, data manipulation and configuration for this lifecycle step.
Inspect the runtime contract in train.py
import json
import os
from pathlib import Path
print("SM_MODEL_DIR =", os.environ.get("SM_MODEL_DIR"))
print("SM_CHANNEL_TRAIN =", os.environ.get("SM_CHANNEL_TRAIN"))
print("SM_CHANNEL_VALIDATION =", os.environ.get("SM_CHANNEL_VALIDATION"))
model_dir = Path(os.environ["SM_MODEL_DIR"])
model_dir.mkdir(parents=True, exist_ok=True)Persist all inference-critical assets
Everything required to reconstruct inference behavior should be either in the artifact or in an explicitly versioned serving dependency.
import json
import joblib
from pathlib import Path
model_dir = Path(os.environ["SM_MODEL_DIR"])
joblib.dump(pipeline, model_dir / "pipeline.joblib")
(model_dir / "model_metadata.json").write_text(
json.dumps({
"model_family": "tfidf-logreg",
"model_version": "17",
"confidence_threshold": 0.75,
"label_order": list(pipeline.classes_),
}, indent=2)
)Design decisions that matter
- Managed framework image vs custom image.
- Dependency pinning.
- Network isolation and package download requirements.
Common failure modes
- A local notebook dependency may not exist in the image.
- Saving outside /opt/ml/model can omit the asset from the packaged model.
- Training image compatibility does not guarantee serving-image compatibility.
Production readiness check
Items to close before treating this step as production-ready.