Deploy & inference
Serverless, async & batch inference
Inference architecture should match latency, payload size and traffic pattern.
workload SLA + payload + traffic profile
right-sized inference execution mode
I select serving mode from workload characteristics rather than assuming every model needs a real-time endpoint.
Mental model
What this means before worrying about syntax.
Serverless suits intermittent request/response workloads; async suits longer-running or larger requests; batch transform suits offline bulk scoring.
- Serving mode follows workload shape, not model type alone.
- Latency, traffic regularity, payload size, processing duration and idle cost drive the choice.
- Batch scoring avoids persistent endpoint infrastructure.
workload pattern -> choose serving mode -> inference
Implementation anatomy
The sequence you would actually build or configure.
- 1
Serverless for intermittent request/response where cold-start tradeoff is acceptable.
- 2
Async for queued longer-running/larger requests.
- 3
Batch Transform for finite offline scoring datasets.
Concrete example
A real model-development or production flow.
Occasional internal complaint API -> serverless candidate
Large document taking tens of seconds -> async candidate
Nightly score 20M accounts -> batch candidate
Implementation lab
Concrete code, data manipulation and configuration for this lifecycle step.
Decision table in code
def choose_inference_mode(
latency_required: bool,
bursty_or_infrequent: bool,
long_running: bool,
offline_bulk: bool,
):
if offline_bulk:
return "batch-transform"
if long_running:
return "asynchronous"
if bursty_or_infrequent and latency_required:
return "serverless"
return "real-time-endpoint"Batch scoring contract
# Input S3:
# s3://bank-ml/scoring/2026-08-17/accounts.csv
#
# Batch job:
# read records
# score records
# write result records
#
# Output S3:
# s3://bank-ml/scoring/2026-08-17/output/
#
# Always include a stable record ID in input and output so
# predictions can be reconciled back to the source population.Design decisions that matter
- Latency SLA.
- Traffic shape.
- Payload/processing duration.
- Idle cost vs provisioned capacity.
Common failure modes
- Serverless is not zero-latency or unlimited-concurrency.
- Batch results must map cleanly back to source record IDs.
- Async changes application workflow because result is not immediate.
Production readiness check
Items to close before treating this step as production-ready.