Integrating hosted LLMs: AWS Bedrock, the Claude API and prompt engineering

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: AWS Bedrock · also covers Claude / Claude Code, Generative AI, Prompt engineering · Step 24 of 35 in MLOps / ML platform engineer · version 1

Hosted LLMs turn model serving into an API integration problem: the platform team does not run GPUs, but it owns access, quotas, cost, latency and the contract that application code uses. Amazon Bedrock exposes foundation models from several providers behind AWS identity and networking, and its Converse API gives one message format across them. Prompts become versioned, tested artifacts, not strings buried in code.

How it is done in practice

A team usually starts by enabling a small set of approved models in one or two regions and exposing them through an internal gateway or a thin client library. Access is granted per workload role with IAM, invocation logging is switched on with a retention and redaction policy, and every call carries team and application labels. Prompts live in the repository next to the code that uses them, with a version identifier in the logs. Dashboards show tokens, cost, time to first token and throttle rate per team; quota increases and provisioned capacity are planned from those numbers. Model upgrades are treated as releases: run the eval suite, shadow or canary the new model, then switch the logical model name.

Technologies

Diagram

flowchart LR
  A[Application service] --> G[LLM gateway]
  G --> P[Policy: auth, rate limit, budget]
  P --> R[Model router]
  R --> B[Bedrock runtime]
  R --> C[Other provider API]
  B --> M[Foundation model]
  G --> T[Traces and token metrics]
  G --> L[Invocation logs]

flowchart LR
  A[Application service] --> G[LLM gateway]
  G --> P[Policy: auth, rate limit, budget]
  P --> R[Model router]
  R --> B[Bedrock runtime]
  R --> C[Other provider API]
  B --> M[Foundation model]
  G --> T[Traces and token metrics]
  G --> L[Invocation logs]

Code

import boto3

client = boto3.client("bedrock-runtime", region_name="us-east-1")

resp = client.converse(
    modelId=MODEL_ID,
    system=[{"text": "You write concise release notes."}],
    messages=[{"role": "user", "content": [{"text": "Summarize this diff: ..."}]}],
    inferenceConfig={"maxTokens": 512, "temperature": 0.2},
)

print(resp["output"]["message"]["content"][0]["text"])
print(resp["stopReason"], resp["usage"], resp["metrics"]["latencyMs"])

Subtopics

How Bedrock exposes foundation models

A managed model catalog behind AWS controls

Amazon Bedrock is a fully managed service that offers foundation models from multiple providers through one AWS API surface. There is no cluster to operate: a caller names a model identifier and sends a request. For a platform team the value is that the usual AWS controls apply. IAM policies decide which roles may invoke which models, requests can stay on private networking through VPC endpoints, invocations can be logged, and usage lands on the AWS bill where it can be tagged and attributed.

Control plane and runtime

Bedrock separates management operations from inference. The control plane lists models and manages resources such as guardrails, knowledge bases and provisioned capacity. The runtime endpoint is the one applications call for inference. Model availability and quotas differ per region, so a platform must record which model is served from which region, request quota increases ahead of launches, and decide whether cross-region inference is acceptable for the data involved.

The Converse API: messages, tools and streaming

One request shape across models

The Converse API gives a consistent interface for message-based models. A request carries a model identifier, an optional system prompt, a list of messages with alternating user and assistant roles, and an inference configuration with fields such as maximum tokens, temperature and top-p. Each message holds content blocks, which can be text, images, documents or tool results. Because the shape is the same for every supporting model, swapping models becomes a configuration change, with provider-specific options passed in a separate field.

Tool use, stop reasons and streaming

Tools are declared with a name, a description and a JSON schema. When the model wants a tool, the response stops with a tool-use stop reason and a content block holding the arguments; the caller runs the tool and sends back a tool result message. The response also reports token usage and latency, which are the raw inputs for cost and performance dashboards. A streaming variant returns incremental events so user interfaces can render text as it is generated.

Prompt engineering as an engineering practice

Structure of a production prompt

A production prompt has stable parts and variable parts. The system prompt states role, constraints and output format; examples show the expected behavior; retrieved context and the user input are inserted into clearly delimited sections so the model can tell instructions from data. Asking for structured output against a schema, and validating it in code, makes downstream handling deterministic. Lower temperature suits extraction and classification; higher values suit drafting.

Versioning and release

Prompts change behavior as much as code does, so they are stored in version control or a prompt registry, reviewed, and released with an identifier that is logged on every call. A prompt change runs through the same evaluation suite as a model change, because a new model version can break a prompt that was tuned for the old one. Keeping the stable prefix of a prompt identical across calls also makes prompt caching effective where the provider supports it.

Managing cost, latency and quota at scale

What drives the bill and the latency

Cost is driven by input and output tokens, priced per model. Latency has two parts: time to first token, and generation time, which grows with output length. The practical levers are choosing a smaller model for simple tasks, capping output tokens, trimming context, caching repeated prefixes, and moving non-interactive work to batch inference. On-demand capacity is subject to request and token quotas per account and region; steady high-volume workloads can buy provisioned throughput instead.

Throttling and resilience

Quota exhaustion shows up as throttling errors, so clients need exponential backoff with jitter, timeouts sized for long generations, and a fallback path to another model or region. Per-team budgets and rate limits stop one tenant from consuming the shared quota. Token counts returned by each call should be emitted as metrics with team, application and model labels, so spend can be attributed and anomalies alerted on before the invoice arrives.

The LLM gateway as a platform component

One internal entry point for all model traffic

Most platform teams put a gateway between applications and model providers. Applications call one internal endpoint with a team credential; the gateway maps a logical model name to a provider and model version, applies rate limits and budgets, attaches guardrails, records traces and token usage, and handles retries and failover. This keeps provider keys out of application code and makes it possible to migrate a model, add a provider, or enforce a data policy in one place. The cost is an extra hop and a component that must be highly available, so it is deployed like any other critical proxy: stateless, horizontally scaled, with its own service level objectives.

Sources

• docWhat is Amazon Bedrock? (docs.aws.amazon.com)
• docAmazon Bedrock: Converse API (docs.aws.amazon.com)

Related lessons

• AWS and EKS foundations for an ML platform
• Serving open-weight LLMs with vLLM and SGLang
• Retrieval-augmented generation infrastructure and vector search
• Agent architectures, agent frameworks and the Model Context Protocol
• LLMOps: evals, tracing, guardrails and AI-assisted development with Claude Code and Cursor
• Autoscaling and cost efficiency: right-sizing, spot mix and GPU utilization
• Security, identity and access, and compliance for ML platforms