Managed deployment and serving with Amazon SageMaker

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: Amazon SageMaker · Step 21 of 35 in MLOps / ML platform engineer · version 1

Amazon SageMaker offers managed hosting in which AWS runs the instances, load balancing and health management while the team supplies a container and a model artifact. It provides several inference options that map onto online, asynchronous and batch demand. The platform work lies in choosing the right option, wiring autoscaling and making deployments safe and repeatable.

How it is done in practice

Teams define models, endpoint configurations and scaling policies in Terraform, CDK or pipeline code, never in the console. A release pipeline registers the model version, runs a batch transform against a holdout set as an evaluation gate, deploys to a staging endpoint, runs a load test, then updates production with canary traffic shifting and alarm-based rollback. Cost reviews look at instance utilisation per endpoint and move low-traffic models to serverless, asynchronous or shared multi-model hosting.

Technologies

Diagram

flowchart TD
  R[Model registry] --> M[SageMaker Model]
  M --> EC[Endpoint config with variants]
  EC --> EP[Real-time endpoint]
  EP --> AS[Application Auto Scaling]
  CW[CloudWatch metrics and alarms] --> AS
  CW --> RB[Automatic rollback]
  M --> BT[Batch transform job]
  S3in[(S3 input)] --> BT
  BT --> S3out[(S3 output)]

flowchart TD
  R[Model registry] --> M[SageMaker Model]
  M --> EC[Endpoint config with variants]
  EC --> EP[Real-time endpoint]
  EP --> AS[Application Auto Scaling]
  CW[CloudWatch metrics and alarms] --> AS
  CW --> RB[Automatic rollback]
  M --> BT[Batch transform job]
  S3in[(S3 input)] --> BT
  BT --> S3out[(S3 output)]

Code

import boto3

aas = boto3.client("application-autoscaling")
rid = "endpoint/churn-prod/variant/AllTraffic"
dim = "sagemaker:variant:DesiredInstanceCount"

aas.register_scalable_target(
    ServiceNamespace="sagemaker", ResourceId=rid,
    ScalableDimension=dim, MinCapacity=2, MaxCapacity=10)

aas.put_scaling_policy(
    PolicyName="invocations-target", ServiceNamespace="sagemaker",
    ResourceId=rid, ScalableDimension=dim,
    PolicyType="TargetTrackingScaling",
    TargetTrackingScalingPolicyConfiguration={
        "TargetValue": 70.0,
        "PredefinedMetricSpecification": {
            "PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"},
        "ScaleOutCooldown": 60, "ScaleInCooldown": 300})

Subtopics

The endpoint object model and container contract

Model, endpoint configuration, endpoint

Hosting is built from three resources. A Model pairs a container image with a model artifact in S3 and an execution role. An EndpointConfig lists one or more production variants, each naming a model, an instance type, an initial instance count and a traffic weight. An Endpoint is the running, addressable thing created from a configuration; updating it means pointing it at a new configuration. Clients call the runtime API with the endpoint name, and SageMaker routes to healthy instances across availability zones when more than one instance is configured.

What the container must do

A custom serving container listens on a fixed port and answers two routes: a health check that returns success once the model is loaded, and an invocations route that takes the request body and returns the prediction. The artifact is unpacked into a known directory before the server starts. AWS publishes prebuilt containers for common frameworks and for LLM runtimes, so many teams only write an inference handler. Keeping the same image runnable outside SageMaker makes local testing and later migration much easier.

Choosing among real-time, serverless, asynchronous and batch

Four options, four demand shapes

Real-time endpoints keep instances running and suit interactive, low-latency traffic with modest payloads. Serverless inference removes instance management and scales to zero, at the price of cold starts and no GPU support, so it fits spiky, low-volume CPU models. Asynchronous inference queues requests, reads large payloads from S3, writes results to S3 and can scale its instances to zero when the queue is empty; it fits long processing times. Batch transform has no persistent endpoint at all: a job starts instances, splits input files from S3 into mini-batches, writes outputs and shuts down.

Batch transform details that matter

A transform job is tuned with a few parameters: how records are split within a file, how many records go into each request, the maximum payload size and the number of concurrent requests per instance. Input and output can be joined so that predictions sit next to the source record, and filters select which fields reach the model. It is the usual choice for scoring a full dataset, for offline evaluation of a candidate model, and for precomputing predictions that are later served from a database.

Autoscaling endpoints

How scaling is configured

Endpoint scaling uses Application Auto Scaling. A production variant is registered as a scalable target with a minimum and maximum instance count, then a policy is attached. Target tracking is the default choice: pick a metric and a target value and the service adds or removes instances to hold it. The predefined metric is invocations per instance; custom metrics such as GPU utilisation or concurrent requests are often better for models whose request cost varies. Step scaling and scheduled scaling cover sharp ramps and known daily patterns.

Finding the target value

The target comes from a load test, not a guess: find the request rate at which a single instance still meets the latency objective, then set the target below it to leave headroom for the time a new instance takes to come into service. Scale-out cooldown should be short and scale-in cooldown longer, to avoid oscillation. Because new instances must download the image and the artifact, large models need more headroom or a higher minimum count.

Safe updates: variants, traffic shifting and rollback

Deployment guardrails

Updating an endpoint uses a blue/green approach by default: a new fleet is provisioned, traffic moves to it, and the old fleet is removed. Traffic can shift all at once, as a canary, or linearly in steps, with a baking period between steps. CloudWatch alarms attached to the update trigger automatic rollback to the old fleet if error rate or latency breaches a threshold. Multiple production variants on one endpoint support weighted A/B tests, and a shadow variant receives a copy of live traffic without returning its responses, which validates a new model or instance type under real load.

Sources

• docSageMaker Real-time inference (docs.aws.amazon.com)
• docSageMaker Batch transform (docs.aws.amazon.com)
• docAutomatic scaling of SageMaker models (docs.aws.amazon.com)

Related lessons

• AWS and EKS foundations for an ML platform
• Model serving architecture: inference APIs for batch and real-time demand
• Staged rollouts: shadow, canary, champion/challenger and rollback
• Autoscaling and cost efficiency: right-sizing, spot mix and GPU utilization
• Model registry, packaging and versioning
• Integrating hosted LLMs: AWS Bedrock, the Claude API and prompt engineering