Design exercise: LLM gateway

Generated by Claude (Fable 5.1) by hand, Scout learning center M4a on 2026-10-12, reviewed by a second reader (Claude Fable 5.1), Scout learning center M4b on 2026-10-12.

Concept: System design and architecture · also covers API design, Scaling, Cost efficiency · Part C (Design exercises) of the system design module · design exercise · version 2

An AI-era design exercise: one internal API in front of hosted model providers and a self-hosted fleet, with per-team keys, token-based limits, budgets, safe failover, streaming and redacted logging. The reference answer sizes tokens per minute against per-minute provider limits, uses prompt caching deliberately with its documented pricing, retries only before the first byte, bounds budget overshoot to one aggregation interval, and treats the gateway as a rate limiter, a cost allocator and a privacy boundary.

Attempt the design from the prompt, the constraints and the questions before reading the reference answer.

The prompt

Design the company's gateway to large language models: every internal application calls one API that fronts several hosted providers and a self-hosted model fleet. Teams get keys, quotas and budgets; the gateway retries and fails over safely, streams tokens, keeps spend visible, and logs enough to debug without leaking what users typed.

Constraints and numbers

Functional

One request format for chat completions and embeddings with streaming; per-team keys, token-based rate limits and monthly budgets; routing of a model alias to one or more deployments with fallback; retries that never duplicate a side effect; prompt caching where the provider supports it; request logging with redaction; spend attribution per team and per key.

Scale (assumption)

200 teams; 5 million requests per day, peak 3x the average; an average request of 3,000 input tokens; 20 percent of requests are embeddings, which return a vector and no output tokens; a chat request averages 400 output tokens; two hosted providers and one self-hosted fleet of open-weight models. Chosen for the exercise; derive the request rate, the token rate and the daily token volume yourself.

Targets (assumption)

Gateway overhead under 10 ms at p99 on top of the provider; first token streamed within 1 s at p50 for chat; 99.9 percent availability of the gateway itself (providers have their own); a team can never exceed its budget by more than one minute of traffic; no raw prompt is stored unless the team opted in.

Non-goals

Model quality evaluation itself, prompt authoring tools, fine-tuning, agent orchestration, the self-hosted model servers' internals (treated as deployments with a known capacity).

Clarifying questions worth asking

• Whose rate limits matter? Both: the provider's limits per model class, expressed in requests per minute, input tokens per minute and output tokens per minute and signalled by a 429 with a retry-after header, and the gateway's own per-team limits, which must sum to less than the provider's.
• What is a safe retry? A retry of a request that produced no output; a chat completion that failed mid-stream is not retried transparently (the client would see duplicated text), it is surfaced as an error with a resumable marker.
• What does the cache cache? Two things: the provider's prompt cache for a shared prefix (system prompt, tools, documents), and an opt-in exact-match response cache at the gateway for deterministic, repeated requests (embeddings, classification with temperature zero).
• Can a request choose the provider? It names a model alias; policy maps the alias to deployments in order. A team may pin a deployment for a test, never for production traffic.
• What is logged? Metadata always (team, key, model, route, tokens in and out, cached tokens, latency, status); prompt and completion text only for teams that opted in, after redaction of recognisable personal data, with a retention of 30 days (assumption).
• Budgets: hard or soft? Hard at the monthly budget with a 90 percent warning; a team may request a raise, which is a policy change, not a code change.

Reference answer

Requirements and estimates

Derived numbers: requests and tokens

5 million requests per day is 58 per second average and 174 at peak. Tokens: 5 million times 3,000 input tokens is 15 billion input tokens per day; the 4 million chat requests (80 percent) times 400 output tokens is 1.6 billion output tokens per day, since the 1 million embedding requests produce none; at peak that is about 520,000 input tokens per second, or 31 million per minute, which is the number to compare against provider limits that are expressed per minute per model class. A provider's limit is a hard ceiling the gateway must stay under across all teams, so the sum of team limits is capped below it with headroom (assumption: 80 percent).

Derived numbers: what caching buys

With a shared prefix of 2,000 of the 3,000 input tokens (system prompt, tool definitions, policy text, assumption), prompt caching changes the economics: the provider documentation prices a cache write at a small premium over the base input price (1.25x for the default five-minute cache) and a cache read at a fraction of it (10 percent of the base input price for most models), with a one-hour cache available at a higher write price; and for most models only uncached input tokens count toward the input-tokens-per-minute limit. The prefix applies to chat requests only (an embedding request carries the text to embed and no system prompt), so 8 billion of the 15 billion daily input tokens (2,000 of each chat request's 3,000; the 3 billion embedding tokens are not cacheable) can become cache reads, cutting both the bill and the limit pressure by a large factor; the gateway's job is to keep the cacheable prefix first and byte-identical across requests so the cache hits.

The latency shape

The provider's time to first token is of the order of hundreds of milliseconds (assumption) and dominates; the gateway adds one authentication lookup, one limit check and one routing decision, all local or in-memory, so the 10 ms overhead target is met if nothing on the hot path crosses the network except the provider call itself.

High-level design

Container view

Stateless gateway nodes authenticate the key, check the team's token buckets and budget, pick a deployment through the router (alias to an ordered list of hosted or self-hosted deployments, with health and limit state per deployment), and proxy the request with streaming. Keys, policies and budgets live in a small store cached on every node; token counters live in a shared counter store with the same local-first correction loop as the rate limiter exercise. Every request writes a redacted log record; an aggregator turns the log into spend per team and per key, which feeds the budget check and a review of routing decisions. Self-hosted deployments are model servers with continuous batching behind an OpenAI-compatible API (the vLLM documentation describes that server shape); the open-source LiteLLM proxy documents the same feature set (virtual keys with budgets per key, team and user, spend tracking, fallbacks across deployments, guardrails) and is the reference for the shape.

flowchart LR
  A[Internal apps and agents] -->|one API, streaming| GW[Gateway nodes: auth, limits, routing]
  GW --> KEY[(Keys, teams, policies, budgets)]
  GW --> CNT[(Token counters: per team per minute)]
  GW --> RT[Router: model alias to deployments, fallback order]
  RT --> P1{{Hosted provider A}}
  RT --> P2{{Hosted provider B}}
  RT --> SH[Self-hosted model servers: continuous batching]
  GW --> CACHE[(Response cache: exact-match, opt-in)]
  GW --> LOG[[request log: redacted prompts, tokens, latency, route]]
  LOG --> SPEND[Spend and usage aggregator]
  SPEND --> KEY
  LOG --> EVAL[Evaluation and routing review]

flowchart LR
  A[Internal apps and agents] -->|one API, streaming| GW[Gateway nodes: auth, limits, routing]
  GW --> KEY[(Keys, teams, policies, budgets)]
  GW --> CNT[(Token counters: per team per minute)]
  GW --> RT[Router: model alias to deployments, fallback order]
  RT --> P1{{Hosted provider A}}
  RT --> P2{{Hosted provider B}}
  RT --> SH[Self-hosted model servers: continuous batching]
  GW --> CACHE[(Response cache: exact-match, opt-in)]
  GW --> LOG[[request log: redacted prompts, tokens, latency, route]]
  LOG --> SPEND[Spend and usage aggregator]
  SPEND --> KEY
  LOG --> EVAL[Evaluation and routing review]

Boxes are containers; cylinders are stores; double brackets are logs; hexagons are external providers.

The data model

Entities

Team {team_id, monthly_budget, limits (requests per minute, input and output tokens per minute), allowed_aliases[], logging_policy}. Key {key_id, team_id, hash, scopes, created_at, revoked_at}. ModelAlias {alias, deployments[] of (provider, model, region, weight, max_tokens_per_minute), fallback_policy}. DeploymentState {deployment, healthy, open_breaker, observed_limit_remaining, p50_first_token}. RequestLog {request_id, team, key, alias, deployment_used, attempts[], tokens_in, tokens_cached, tokens_out, first_token_ms, total_ms, status, prompt_hash, prompt_text?, completion_text?}. Spend {team, day, deployment, tokens_in, tokens_cached, tokens_out, cost}.

API

POST /v1/chat/completions {model: alias, messages[], stream, cache_prefix_marker?, idempotency_key?} returns a token stream then a usage block; POST /v1/embeddings {model, input[]}; GET /v1/usage?team=&from=&to=; admin endpoints for keys, aliases and budgets. Errors carry the provider's class (rate limited, overloaded, invalid) and the deployment tried, so clients can distinguish a quota problem from an outage.

The core flow, step by step

One streamed chat completion with a cacheable prefix.

1. Authenticate the key (hash lookup in the node's cache of the key store), load the team's policy and check the alias is allowed.
2. Admission: take tokens from the team's local request bucket and an estimate of input tokens from the input-token bucket (the gateway counts the prompt locally before sending); check the budget projection (spend so far plus this request's estimated cost) against the monthly budget. Reject with 429 and the team's own retry-after if over the limit, 402-style with a budget error if over budget.
3. Route: the router picks the first healthy deployment for the alias whose observed remaining limit can take the request; it orders the messages so the shared prefix is first and marks the cache breakpoint at the end of the prefix, as the provider's prompt caching documentation describes.
4. Proxy with streaming: open the provider stream, forward tokens as they arrive, and record the time to first token. If the provider answers 429 or a server error before any token was sent, back off per its retry-after and fail over to the next deployment (a retry is safe because nothing reached the client). If the stream breaks after tokens were sent, close the client stream with an error marker and do not retry.
5. Settle: read the usage block (input, cached, output tokens), correct the token buckets with the real counts, write the redacted log record, and increment the team's spend counter.
6. Asynchronously, the spend aggregator rolls the log into daily spend per team and deployment; the budget check reads the rolled total plus the live counter, so a team can overshoot by at most the traffic of the aggregation interval (one minute, assumption), which is the stated target.

Scaling and the bottleneck

Fairness under a shared ceiling

The provider's limits are the bottleneck, and they are shared by all teams: the gateway is the place where fairness is enforced so one team's burst does not get every other team a 429 from the provider. Per-team token buckets are the request rate limiter of Stripe's write-up applied to tokens; a concurrency limiter per deployment protects the self-hosted fleet, whose capacity is in-flight sequences rather than requests per minute; and a load shedder drops low-priority traffic (batch jobs, evaluations) first when a deployment's remaining limit runs low. The router tracks each deployment's remaining limit from response headers and 429s, and a breaker per deployment opens on error rate so failover is immediate rather than discovered request by request.

Cost levers and the log

Cost scales with tokens, not requests, so the levers are token levers: the shared prefix kept identical and first so the provider cache hits; an exact-match response cache for deterministic workloads (embeddings of the same text, classification at temperature zero) keyed by a hash of the full request; routing cheap aliases (small models, self-hosted) by policy for workloads that allow it; and output caps per alias. The log is what makes any of this visible: spend per team per deployment per day, cache hit ratio per alias, and the routing decisions a reviewer can question. Logging text is the privacy risk: redaction runs before the write, text is opt-in per team with retention, and the log store is access-controlled separately from metadata.

Failure modes and what degrades

Each row names the failure, what callers see, and the mitigation.

Trade-offs and the alternative not taken

The decisions a reviewer reads first.

What changes at 10x

At 50 million requests per day and every team on it, the gateway is company infrastructure: tenancy becomes explicit (tiers of teams with isolation of counters and queues so a runaway agent cannot starve a customer-facing product), policy and guardrails run inside the gateway (input and output checks, allowed tools, data classification rules per alias), routing decisions are evaluated offline against quality and cost (a routing change is reviewed like a dependency upgrade, with the log as the evidence), cost allocation becomes showback and then chargeback, and capacity planning for the self-hosted fleet is driven by the gateway's observed token demand per alias. The well-architected guidance for generative AI workloads is the checklist for that review: model selection, retrieval, guardrails and cost as explicit design decisions.

Self-check rubric

Read, not scored: find the level your answer reached.

L1, mid-level: applies

• One API in front of several providers with per-team keys, quotas, logging and retries, and the token arithmetic from the assumptions.
• Streaming proxied end to end with the time to first token measured.
• A retry rule that never duplicates output.

L2, senior: chooses with numbers and defends

• Routing and fallback across hosted and self-hosted deployments with per-deployment breakers and observed limit state.
• Token-based rate limits per team summing below the provider ceiling, with the local-first counter design.
• Prompt caching used deliberately (stable prefix first, breakpoint) with the pricing and limit consequences stated.
• Budgets with a bounded overshoot, and personal data kept out of logs by default.

L3, staff: sets the constraint and judges

• A company-wide gateway: tenancy tiers, policy and guardrails in the path, evaluation of routing decisions from the log.
• Cost allocation (showback to chargeback) and capacity planning for self-hosted models from observed demand.
• What the gateway refuses: unbounded output, unredacted text by default, production traffic pinned to one deployment.

The 45-minute interview frame

• 0:00 to 0:05 Requirements: whose limits matter, what a safe retry is, what the cache caches, who chooses the provider, what is logged, hard or soft budgets. Say the numbers aloud: 174 requests per second at peak, 15 billion input tokens and 1.6 billion output tokens per day, 31 million input tokens per minute at peak against per-minute provider limits, about half of input tokens cacheable.
• 0:05 to 0:10 Entities and API: Team with limits and budget, Key, ModelAlias with ordered deployments, DeploymentState, RequestLog with cached tokens, Spend; the completions, embeddings and usage endpoints and the error classes.
• 0:10 to 0:20 High-level design with the diagram: gateway nodes, key and policy store, token counters, router to providers and the self-hosted fleet, response cache, redacted log, spend aggregator. Say which state is local and which is shared.
• 0:20 to 0:40 Two deep dives: limits and fairness under a shared provider ceiling (token buckets per team, concurrency per self-hosted deployment, shedding by priority, breakers and failover), then cost and privacy (prompt caching mechanics and prices, response cache, routing by policy, redaction and retention of logs, the budget overshoot bound).
• 0:40 to 0:45 Failure modes from the table, the 10x platform (tenancy tiers, guardrails, routing evaluation, chargeback, capacity planning), and the L3 sentence: the gateway is a rate limiter, a cost allocator and a privacy boundary that happens to proxy requests.

The design-doc frame

• Context and scope. The applications calling models today, the providers and contracts, the self-hosted fleet's capacity, the spend surprise or the 429 incident that motivates the gateway.
• Goals and non-goals. One API, team limits and budgets, safe failover, streaming, spend visibility, privacy by default; no evaluation product, no prompt tooling, no fine-tuning, no agent orchestration.
• Design. This answer fills it: admission, routing, streaming proxy, settlement, the counter and log design, caching.
• Alternatives considered. An SDK per app, request-based limits, retry-anything, synchronous exact budgets, always-on text logging, with the trade-off table.
• Cross-cutting concerns. Security (key hashing, scopes, revocation propagation, admin audit), privacy (redaction, opt-in text, retention, access control on the log store), observability (first-token latency per deployment, failover ratio, cache hit ratio, spend per team per day, rejected requests by reason), cost (tokens by class per team), rollout by moving one team's alias at a time with the direct provider path as fallback.
• Review questions the doc must answer. What a client receives when a stream breaks after the first token, and by how much a team can exceed its budget before the gate closes.

Depth notes

• L1. One API in front of several providers with keys, per-team quotas, logging and retries.
• L2. Routing and fallback across providers and self-hosted models, streaming, token-based rate limits, prompt caching, budgets, PII handling in logs.
• L3. A company-wide gateway (tenancy, policy and guardrails, evaluation of routing decisions, cost allocation, capacity for self-hosted models).

Sources

• docLiteLLM documentation (docs.litellm.ai)
• docClaude Platform docs: Rate limits (platform.claude.com)
• docClaude Platform docs: Prompt caching (platform.claude.com)
• docvLLM documentation (docs.vllm.ai)
• blogStripe: Scaling your API with rate limiters (stripe.com)
• docAWS Well-Architected: Generative AI Lens (docs.aws.amazon.com)

Related lessons

• API design sd-api-design planned
• Rate limiting, overload and resilience patterns sd-rate-limiting-overload planned
• Timeouts, retries and idempotency sd-idempotency-retries planned
• Caching sd-caching planned
• Observability for designers sd-observability planned
• Cost as a design constraint sd-cost planned
• Multi-tenancy sd-multi-tenancy planned
• Design exercise: API rate limiter