Practice: Scaling and performance-cost tradeoffs for LLM systems

Generated by Claude through GigAI 0.1.11.10 on 2026-10-12.

Main tool: AWS · AWS puts the quotas, the cost attribution services and the queueing and analytics building blocks in one place, so every latency and cost lever in this lesson can be tied to a documented limit and a line on a bill. · Alternatives: Google Cloud and Microsoft Azure offer comparable billing, quota and managed model services with different pricing and tooling; a self-hosted stack of vLLM behind LiteLLM with Redis, Envoy, OpenTelemetry and Prometheus trades managed limits for GPUs you run yourself; calling the Claude API or the OpenAI Batch API directly gives prompt caching and batch discounts without any cloud cost tooling around them.

Every step links into the tool's own documentation; links were checked on 2026-10-12 · version 1

Setup (one time)

1. Sign in to the AWS Management Console with a personal or sandbox account; this needs an AWS account, so stay inside the free tier, and use a local emulator container as the closest substitute if you have none.
Every later item reads quotas or cost data from an account. A sandbox keeps the practice away from customer spend.
Explained in: What is the AWS Management Console? - AWS Management Console (docs.aws.amazon.com), checked 2026-10-12
Done when: The console home page opens and shows your account and a selected region.
2. Create a non-root identity with programmatic credentials for your laptop, following the security credentials overview.
Cost and quota queries from a laptop should never run as the root user. Scoped credentials are also what a customer will expect you to ask for.
Explained in: AWS General Reference - AWS General Reference (docs.aws.amazon.com), checked 2026-10-12
Done when: A command line identity check from your laptop returns the non-root identity you created.
3. Look up the endpoint and quota reference for the model serving service and the region you plan to use.
Capacity for an LLM system is bounded by rate limits, so you need to know where those limits are published before you design anything.
Explained in: AWS General Reference - AWS General Reference (docs.aws.amazon.com), checked 2026-10-12
Done when: You have a note with the regional endpoint and the page that lists its request and token quotas.

Steps

1. Read the introduction of the Cost Optimization Pillar and note its design principles in your own words.
The pillar is the vocabulary a customer's cloud team already uses for cost. It frames the quality, latency and cost choice as something you can defend.
Explained in: Cost Optimization Pillar - AWS Well-Architected Framework - Cost Optimization Pillar (docs.aws.amazon.com), checked 2026-10-12
Done when: You have a short list of the pillar's principles, each mapped to one LLM cost lever such as tokens, output length or capacity.
2. Write a one-page map of the AWS cost management services and what each one answers.
Cost attribution starts with knowing which service reports spend, which forecasts it and which alerts on it.
Explained in: Choosing an AWS cost management strategy - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: Your map names each cost management service with the one question it answers for an LLM workload.
3. Write a cost attribution scheme for one LLM feature: the tags or accounts that separate tenants, and the unit you will report, such as cost per request or per resolved ticket.
Unit economics only work when spend can be divided by a business unit of work. The criteria in the guide show which dimensions are worth separating.
Explained in: Choosing an AWS cost management strategy - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: You have a table of tag keys, the values allowed for each, and the formula for your unit cost.
4. Compare the cost management services against your scheme and choose the ones you will use for reporting, budgeting and anomaly alerts.
Choosing on paper first avoids enabling tools that cannot break spend down by the dimensions you defined.
Explained in: Choosing an AWS cost management strategy - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: Each need in your scheme has one chosen service next to it and a reason of one line.
5. Write a latency budget for one request path, splitting time between network, queueing, time to first token and output generation, using the modern application criteria to place each part.
Latency in an LLM system comes mostly from output length and waiting for capacity. A budget shows which part is worth cutting with streaming, shorter outputs or a smaller model.
Explained in: Choosing a modern application strategy - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: You have a budget whose parts add up to a target response time, with one planned cut marked against the largest part.
6. Compare the generative AI service options and record which support prompt caching and which offer a discounted batch mode.
Caching a repeated prompt prefix and moving non-urgent work to batch are the two provider features with the largest cost effect. Which service you pick decides whether you get them.
Explained in: Choosing a generative AI service - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: You have a comparison table with a caching column, a batch column and the workloads you would send to each.
7. Query the published quotas for your chosen model endpoint and write an overload plan: retry with backoff, then queue, then a smaller model or a cached answer.
Rate limits are the real ceiling on throughput. Graceful degradation has to be decided before the limit is hit, not during an incident.
Explained in: AWS General Reference - AWS General Reference (docs.aws.amazon.com), checked 2026-10-12
Done when: Your plan lists the request and token limits and the behaviour at each stage of overload.
8. Choose an application integration service to buffer batch and overflow requests, and sketch the queue between your caller and the model.
A queue turns a rate limit error into a delay and gives batch work a place to wait. The guide shows which service fits ordered, delayed or fan-out traffic.
Explained in: Choosing an AWS application integration service - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: You have a diagram with the chosen service, its retry and dead letter behaviour, and which requests bypass it.
9. Create the queue and a small worker using the getting started material for your chosen integration service; this needs an AWS account, so stay in the free tier or run a local emulator container instead.
Building the buffer once makes the overload plan repeatable. It can then be checked in rather than explained.
Explained in: Choosing an AWS application integration service - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: A test message sent to the queue is picked up by your worker, and a failed one lands in the dead letter queue.
10. Configure a budget with an alert and a saved cost report grouped by your attribution tags; an AWS account is required, so use the free tier and a very low threshold.
A budget and a saved report make cost attribution automatic. The same view can be reproduced for every customer review.
Explained in: Choosing an AWS cost management strategy - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: The budget shows in the console with its alert, and the saved report opens grouped by your tags.
11. Run a repeatable query that divides spend by request count to produce your unit cost, using the analytics service the guide recommends for cost and usage data.
A scripted query replaces a hand-built spreadsheet. It lets you rerun the unit economics after each change to prompts, caching or model choice.
Explained in: Choosing an AWS analytics service - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: Running the saved query twice returns the same unit cost for the same period.
12. Write a one-page recommendation that picks a point on the quality, latency and cost surface for the customer, citing your budget, quota plan and unit cost.
The skill in this lesson is a choice the customer can defend. Writing it down forces each lever to be backed by a number you measured or a limit you looked up.
Explained in: Choosing a modern application strategy - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: A reviewer can read the page and find the chosen design, the expected unit cost, the latency target and the behaviour under overload.

Back to Scaling and performance-cost tradeoffs for LLM systems