Design exercise: metrics and time series pipeline

Generated by Claude (Fable 5.1) by hand, Scout learning center M4a on 2026-10-12, reviewed by a second reader (Claude Fable 5.1), Scout learning center M4b on 2026-10-12.

Concept: System design and architecture · also covers Observability, Relational and non-relational data stores, Scaling · Part C (Design exercises) of the system design module · design exercise · version 1

A classic design exercise: a metrics system for tens of millions of series with dashboards, alerts and tiered retention. The reference answer sizes the sample stream from stated assumptions and the Gorilla compression figure, decouples ingestion from storage with a samples log, keeps a compressed 26-hour window in memory, compacts into immutable blocks with downsampled tiers, narrows queries through an inverted index, and treats cardinality as the unit of capacity with hard per-tenant limits.

Attempt the design from the prompt, the constraints and the questions before reading the reference answer.

The prompt

Design the metrics system for a company's services: thousands of instances emit counters, gauges and histograms every few seconds, engineers query the last few hours on dashboards and alerts, and a few people need a year of history at coarser resolution. The system must not fall over when one team adds a label that explodes into millions of series.

Constraints and numbers

Functional

Ingest metrics by scrape and by push; store them with labels; answer range and instant queries over labels with aggregation functions; evaluate alerting rules; retain raw data for 15 days, 1-minute data for 90 days and 1-hour data for 15 months (assumption); enforce per-tenant series limits.

Scale (assumption)

20,000 service instances, each exposing 2,000 series, so 40 million active series; a 15 s scrape interval; 50 tenants (teams); 500 dashboard and rule queries per second at peak, most over the last hour. Chosen for the exercise; derive the sample rate, the raw and compressed volumes and the hot-window memory yourself.

Targets (assumption)

A sample is queryable within 30 s of being scraped; a last-hour query answers in under 1 s at p99 and a 30-day query in under 10 s; 99.9 percent availability for ingestion and for alert evaluation; no single tenant can degrade another's ingestion.

Non-goals

Logs and traces (separate systems that share label conventions), anomaly detection, billing, the dashboard product itself.

Clarifying questions worth asking

• What is a series? A metric name plus a unique set of label key-value pairs; every distinct combination is a separate stream of (timestamp, value) samples, which is the Prometheus data model.
• Pull or push? Pull by default (the scraper controls the rate and knows when a target is down); push for batch jobs and for edge locations that cannot be scraped, through a gateway that applies the same limits.
• What is the cardinality rule? A label must never carry an unbounded value (user ids, request ids, full URLs); the Prometheus instrumentation guidance says to keep a metric's label cardinality small, below about 10 as its rule of thumb, because every label set is a separate series, and the ingest layer enforces a per-tenant series limit with rejection and an alert when it is hit.
• How exact must old data be? Downsampled: for data older than 15 days, 1-minute aggregates (min, max, sum, count) are enough; older than 90 days, 1-hour.
• Are histograms first class? Yes, as bucket series; a histogram with 20 buckets is 22 series, and the sizing counts them.
• Can a query span all tenants? No; queries are tenant-scoped, with a platform tenant that may read everything for fleet-level views.

Reference answer

Requirements and estimates

Derived numbers: samples and the hot window

40 million series scraped every 15 s is 2.7 million samples per second. A raw sample is 16 bytes (an 8-byte timestamp and an 8-byte float, the figure the Gorilla paper starts from), so raw ingest is 43 MB per second and 3.7 TB per day. The Gorilla paper reports delta-of-delta timestamp and XOR value compression averaging 1.37 bytes per sample, a 12x reduction; at that rate the hot window costs 3.7 MB per second and 320 GB per day. Gorilla kept 26 hours in memory because most queries were for recent data; with the same window, the hot tier holds about 350 GB of compressed samples plus per-series overhead (a series header of about 100 bytes, assumption, is 4 GB), replicated twice, so about 700 GB of memory across the hot nodes: a few dozen large nodes, or more small ones.

Derived numbers: retention and queries

Long-term storage: raw for 15 days is about 4.8 TB compressed; 1-minute aggregates for 90 days are 4 values every 60 s instead of 1 value every 15 s, so about the same bytes per series per day as raw, 29 TB for 90 days (assumption, no further compression counted); 1-hour aggregates for 15 months are negligible. Object storage holds all of it cheaply. Queries: 500 per second at peak, most touching a few hundred series each over one hour (240 samples per series), so about 100,000 series-hours per second against the hot tier, which is a memory-bound scan, not a disk one.

High-level design

Container view

Scrapers pull targets on a schedule (the Prometheus model) and a push gateway accepts what cannot be scraped; both go through an ingest layer that validates labels and enforces per-tenant series limits before anything is stored. Samples land on a partitioned log keyed by series hash, which decouples ingestion from storage and lets the hot store be rebuilt. The hot store keeps the last 26 hours in memory, compressed, replicated twice; a compactor writes 2-hour blocks to object storage and downsamples into tiers. A query frontend splits a query by time range, caches results, and fans out to hot shards and long-term blocks, which is the shape Monarch describes (regional storage with query pushdown to where the data is) and M3 describes for Prometheus (a coordinator that handles downsampling, retention and rollup rules in front of the storage nodes).

flowchart LR
  T[Service instances: /metrics endpoints] -->|scrape every 15 s| SC[Scrapers, sharded by target]
  PG[Push gateway: batch jobs and edge] --> IN[Ingest and validation: cardinality limits per tenant]
  SC --> IN
  IN --> LOG[[samples log, partitioned by series hash]]
  LOG --> HOT[(Hot store: in-memory, compressed, 26 h window, 2 replicas)]
  HOT -->|every 2 h blocks| CP[Compactor and downsampler]
  CP --> LT[(Long-term blocks in object storage: raw, 1 min, 1 h tiers)]
  IN --> IX[(Series index: labels to series id)]
  Q[Dashboards, alerting rules] --> QF[Query frontend: split, cache, fan-out]
  QF --> HOT
  QF --> LT
  QF --> IX

flowchart LR
  T[Service instances: /metrics endpoints] -->|scrape every 15 s| SC[Scrapers, sharded by target]
  PG[Push gateway: batch jobs and edge] --> IN[Ingest and validation: cardinality limits per tenant]
  SC --> IN
  IN --> LOG[[samples log, partitioned by series hash]]
  LOG --> HOT[(Hot store: in-memory, compressed, 26 h window, 2 replicas)]
  HOT -->|every 2 h blocks| CP[Compactor and downsampler]
  CP --> LT[(Long-term blocks in object storage: raw, 1 min, 1 h tiers)]
  IN --> IX[(Series index: labels to series id)]
  Q[Dashboards, alerting rules] --> QF[Query frontend: split, cache, fan-out]
  QF --> HOT
  QF --> LT
  QF --> IX

Boxes are containers; cylinders are stores; double brackets are logs. Ingestion and storage are decoupled by the samples log.

The data model

Entities

Series {series_id (hash of name and sorted labels), tenant, name, labels{}, first_seen, last_seen}. Sample {series_id, timestamp_ms, value} stored as compressed chunks of about 120 samples (assumption) per series in the hot tier. Block {tenant, from, to, resolution (raw, 1m, 1h), series index, chunk files} immutable in object storage. Index: an inverted index from each label pair to the series ids that carry it, per block, so a query by labels intersects posting lists before touching chunks. Tenant {tenant_id, max_series, max_samples_per_second, retention overrides}. Rule {tenant, expression, interval, for, labels}.

API

GET /api/v1/query?query=&time= (instant), GET /api/v1/query_range?query=&start=&end=&step=, GET /api/v1/series?match[]=, GET /api/v1/labels; the push gateway's POST /write accepts batches of samples with labels; rules are declared in files per tenant. The query language is a label-selector-and-function language in the PromQL shape; the exercise does not design it.

The core flow, step by step

One scrape cycle and one dashboard query.

1. A scraper shard owns a subset of targets (by consistent hashing of the target address) and scrapes each every 15 s with a per-target timeout; a failed scrape produces an up=0 sample so absence is visible.
2. The ingest layer parses the exposition, attaches the tenant label, rejects samples whose label set is malformed or whose series would exceed the tenant's limit (returning a counted error, not silently dropping), and writes the batch to the samples log partition for each series hash.
3. Hot store nodes consume their partitions, look up or create the series, append the sample to the series' current chunk with delta-of-delta and XOR encoding, and acknowledge the log offset; the series index is updated for new series.
4. Every 2 hours the compactor flushes closed chunks into an immutable block with its own inverted index, uploads it to object storage, and writes the 1-minute and 1-hour aggregates for the tiers; the hot store drops data older than 26 hours only after the block is durable.
5. A dashboard query for the last hour arrives at the frontend, is split by time if it spans tiers, is checked against a results cache keyed by (tenant, query, step, time bucket), and is fanned out to the hot shards that hold matching series (the index narrows the shards); each shard evaluates as much of the expression as it can locally and returns partial results that the frontend merges.
6. Alert rules run on the same path on a schedule in a rule evaluator that is replicated and deduplicates notifications, so an evaluator failure does not silence alerts.

Scaling and the bottleneck

Cardinality is the capacity unit

The bottleneck is cardinality, not bytes. One service that adds a user id label turns 2,000 series into millions overnight; memory per series, not per sample, is what fills the hot tier, and a query that touches millions of series is what times out. The ingest limit per tenant is the control: a hard cap on active series with rejection, a soft cap that alerts, and per-metric cardinality reports so the owning team can see which label did it. The Prometheus instrumentation guidance (keep label cardinality small; every label set is another series with its own cost) is the rule the limit enforces: label values must be bounded sets. Series churn (short-lived pods creating new series on every deploy) is the second-order problem; it is bounded by the same limit and by expiring series that have not received a sample for a window.

Query fan-out and downsampling

Reads scale by narrowing before scanning: the inverted index turns a label selector into a small set of series and the shards that hold them, the frontend splits long ranges into cached sub-ranges, and downsampled tiers make a 30-day query read 1-minute data (43,000 points per series) instead of raw (172,800). Pushing aggregation down to the shards keeps the data where it is, which is Monarch's main argument. Fleet-level queries that touch every tenant are run by the platform tenant against pre-aggregated recording rules, never ad hoc at full resolution.

Failure modes and what degrades

Each row names the failure, what engineers see, and the mitigation.

Trade-offs and the alternative not taken

The decisions a reviewer reads first.

What changes at 10x

At 400 million series the system is a multi-tenant platform with zones: regional ingestion and hot tiers with a global query layer that federates across them (Monarch's regional zones), per-tenant isolation of hot-store partitions so a noisy tenant is confined to its shards, quotas and showback per tenant measured in active series and in query cost, and a cost per million series per month that the platform publishes. Recording rules move from a convenience to a necessity for every dashboard that aggregates across the fleet. Exemplars that link a histogram bucket to a trace id, and native histograms that collapse 22 series into one, are the next two design changes a team would plan.

Self-check rubric

Read, not scored: find the level your answer reached.

L1, mid-level: applies

• Scrape and push ingestion, a time series store with labels, a query API and tiered retention.
• The sample rate (2.7 million per second) and raw volume derived from the stated assumptions, with the compressed figure cited.
• A per-tenant series limit enforced at ingest, with the cardinality rule stated.

L2, senior: chooses with numbers and defends

• The in-memory hot window with compression, a samples log between ingestion and storage, and 2-hour immutable blocks with an inverted index.
• Downsampling tiers and query splitting with a results cache; fan-out to the shards narrowed by the index with pushdown of aggregation.
• Replicated rule evaluation and the up=0 convention so absence is visible.
• Series churn and the memory-per-series argument as the capacity model.

L3, staff: sets the constraint and judges

• A multi-tenant metrics platform: regional zones with federation, tenant isolation of partitions, quotas and showback per series and per query.
• Cost per million series per month as the number the platform is held to, and recording rules as a necessity at fleet scale.
• What the system refuses: unbounded labels, cross-tenant ad hoc queries, dropping hot data before blocks are durable.

The 45-minute interview frame

• 0:00 to 0:05 Requirements: what a series is, pull versus push, the cardinality rule, downsampling, histograms, tenant scoping. Say the numbers aloud: 40 million series, 2.7 million samples per second, 43 MB per second raw and about 3.7 MB per second compressed, a 26-hour hot window of about 700 GB with two replicas.
• 0:05 to 0:10 Entities and API: Series, Sample in compressed chunks, Block with an inverted index, Tenant limits, Rule; the query and series endpoints and the push write endpoint.
• 0:10 to 0:20 High-level design with the diagram: scrapers and push gateway, ingest with limits, the samples log, the hot store, compactor and tiers in object storage, the query frontend. Say what the log buys.
• 0:20 to 0:40 Two deep dives: cardinality as the capacity unit (limits, reports, churn, memory per series), then the query path (index narrowing, splitting and caching, downsampled tiers, pushdown, replicated rules).
• 0:40 to 0:45 Failure modes from the table, the 10x platform (zones and federation, isolation, showback), and the L3 sentence: the unit of capacity is a series, not a byte, and the design is about who may create one.

The design-doc frame

• Context and scope. The services and their instrumentation today, the dashboards and alerts that depend on it, the outage or cost that motivates a new system, and the label conventions shared with logs and traces.
• Goals and non-goals. Freshness within 30 s, last-hour queries under 1 s, tiered retention, tenant isolation; logs, traces, anomaly detection and the dashboard product out of scope.
• Design. This answer fills it: ingestion with limits, the samples log, the hot tier, compaction and tiers, the query frontend and rules.
• Alternatives considered. Push everywhere, a disk-based store for everything, direct writes without a log, raw retention forever, best-effort cardinality, with the trade-off table.
• Cross-cutting concerns. Security (tenant scoping on every query, who may change limits), observability of the observability system (ingest lag, rejected samples per tenant, compaction backlog, query p99 by tier, rule evaluation lag), cost per million series per month, rollout by scraping both systems for a tenant and comparing.
• Review questions the doc must answer. What happens on the day a tenant doubles its series in an hour, and what a dashboard shows while object storage is unavailable.

Depth notes

• L1. Scrape or push ingestion, a time series store, a query API and retention.
• L2. Aggregation and downsampling tiers, cardinality limits, hot window in memory with compression, query fan-out across shards.
• L3. A multi-tenant metrics platform (quotas, isolation, cost per series, regional federation).

Sources

• docPrometheus documentation: Overview (prometheus.io)
• paperGorilla: a fast, scalable, in-memory time series database (VLDB 2015) (www.vldb.org)
• paperMonarch: Google's planet-scale in-memory time series database (VLDB 2020) (www.vldb.org)
• blogUber Engineering: M3, a metrics platform for Prometheus (www.uber.com)
• docPrometheus documentation: Instrumentation practices (prometheus.io)
• bookSite Reliability Engineering: Monitoring distributed systems (sre.google)

Related lessons

• Queues, streams and delivery guarantees sd-queues-streams planned
• Storage engines and indexes sd-storage-engines planned
• Data modelling across stores sd-data-modelling planned
• Capacity planning and performance sd-capacity planned
• Observability for designers sd-observability planned