MODELCATALOG

Build & train

Build & train

File, Pipe & FastFile

Controls how S3 training data reaches the training container.

Inputs

S3 channel objects

Outputs

File, Pipe or FastFile delivery into training

Core explanation

Input mode is separate from channel naming: the channel identifies the dataset, while File, Pipe or FastFile describes how SageMaker delivers it.

01

Mental model

What this means before worrying about syntax.

File downloads data before training; Pipe streams it; FastFile provides file-like access without the same full up-front download behavior.

  • Channel identifies the dataset; input mode controls how S3 data reaches the training process.
  • File favors simplicity, Pipe favors streaming access, FastFile offers file-like access without the same full up-front download pattern.
Lifecycle position
S3 -> [File | Pipe | FastFile] -> training container
02

Implementation anatomy

The sequence you would actually build or configure.

  1. 1

    File: stage channel data to local storage.

  2. 2

    Pipe: stream data through named pipes; code/data format must support it.

  3. 3

    FastFile: use normal file-style access against supported S3 inputs.

Objects / terms worth remembering
training_input_mode
03

Concrete example

A real model-development or production flow.

1

Small/medium repeatedly scanned CSV -> File

↓
2

Very large sequential record stream -> consider Pipe

↓
3

Large file collection needing file APIs -> consider FastFile

04

Implementation lab

Concrete code, data manipulation and configuration for this lifecycle step.

01

File mode

python

Best starting point: simple file semantics inside /opt/ml/input/data/<channel>.

trainer = ModelTrainer(
    training_image=TRAINING_IMAGE_URI,
    source_code=source_code,
    role=SAGEMAKER_ROLE_ARN,
    training_input_mode="File",
)
02

Pipe mode

python

Use only if your training reader/data format is designed to consume a stream. It is not a transparent replacement for normal file access.

trainer = ModelTrainer(
    training_image=TRAINING_IMAGE_URI,
    source_code=source_code,
    role=SAGEMAKER_ROLE_ARN,
    training_input_mode="Pipe",
)
03

FastFile mode

python

Useful for large S3 datasets where file-like access is desirable without the same full up-front download behavior.

trainer = ModelTrainer(
    training_image=TRAINING_IMAGE_URI,
    source_code=source_code,
    role=SAGEMAKER_ROLE_ARN,
    training_input_mode="FastFile",
)

Design decisions that matter

  • Dataset size and file count.
  • Local disk and startup time.
  • Sequential vs random access.
  • Framework compatibility.

Common failure modes

  • Pipe is not a transparent filesystem.
  • Many tiny objects can behave very differently from a few large ones.
  • Input mode changes delivery, not channel semantics.

Production readiness check

Items to close before treating this step as production-ready.

✓ data volume✓ file count✓ access pattern✓ local disk✓ framework support✓ benchmark