Design exercise: notifications

Generated by Claude (Fable 5.1) by hand, Scout learning center M4a on 2026-10-12, reviewed by a second reader (Claude Fable 5.1), Scout learning center M4b on 2026-10-12.

Concept: System design and architecture · also covers API design, Distributed systems, Reliability · Part C (Design exercises) of the system design module · design exercise · version 2

On this page

The prompt
Constraints and numbers
Clarifying questions worth asking
Reference answer
Requirements and estimates
High-level design
The data model
The core flow, step by step
Scaling and the bottleneck
Failure modes and what degrades
Trade-offs and the alternative not taken
What changes at 10x
Self-check rubric
The 45-minute interview frame
The design-doc frame
Depth notes
Sources
Related lessons

A classic design exercise: a shared notification service for push, email and SMS with preferences, quiet hours, provider failover and no duplicates. The reference answer uses an idempotency key per (event, user, channel), one durable queue per channel and priority class, adapters behind circuit breakers with a secondary provider, collapse identifiers at the provider, and a delivery status store; the design-doc frame follows the outline's worked example.

Attempt the design from the prompt, the constraints and the questions before reading the reference answer.

The prompt

Design a notification service that several product teams use to send push, email and SMS to users. A user must see each event once, on the channels they allow and not during their quiet hours; a provider outage must not stop delivery; the sending team must be able to ask what happened to a notification.

Constraints and numbers

Functional

An API for internal teams to send a notification for an event to a user on one or more channels; per-user preferences and quiet hours; templates per team; delivery status by notification id; provider failover; retries without duplicates.

Scale (assumption)

20 million users; 50 million notifications per day, 70 percent push, 25 percent email, 5 percent SMS; peak 10x the daily average; five sending teams today, more later. Chosen for the exercise; derive the per-second rates per channel and the queue sizes yourself.

Targets (assumption)

Push handed to the provider within 5 s at p95; email within 60 s; SMS within 30 s; provider failover within 60 s of detection; at-least-once delivery to the provider with deduplication so a user sees one message per event; 99.9 percent availability of the send API.

Non-goals

Marketing campaigns and audience selection, an in-app inbox, analytics beyond delivery status, exactly-once delivery across providers (not possible end to end).

Clarifying questions worth asking

Who decides the channel? The sending team proposes channels per event type; the user's preferences and quiet hours filter them; the service never adds a channel the team did not ask for.
What is the duplicate we must prevent? The same event sent twice by a retrying caller, and the same queued notification processed twice by a retrying consumer. The same event sent with two different event ids by the caller is not caught, and the API contract says so.
What happens during quiet hours? Push and SMS are held until the window ends (or dropped if the event expires); email is sent, since it does not wake anyone (assumption; a product decision the doc must state).
Priority classes? Two: transactional (security codes, receipts) which ignores quiet hours and rate limits, and informational which honours both.
How long is delivery status kept? 90 days (assumption), queryable by notification id and by user.
Where does personal data live? Phone numbers and device tokens only inside the adapter boundary and the status store, encrypted at rest; teams send user ids, never addresses.

Reference answer

Requirements and estimates

Derived numbers

50 million per day is about 580 per second average, 5,800 at peak. Per channel at peak: push 4,060 per second, email 1,450, SMS 290. Each notification record is about 1 kB (ids, channel, template id, rendered payload reference, status history), so 50 GB per day and 4.5 TB for 90 days of status, a partitioned table keyed by notification id with an index by user. Queue depth at peak if a provider stalls for 60 s: 4,060 times 60 is about 244,000 push messages, trivial for a durable queue but the number that sets the failover deadline.

The latency budget

Provider latency dominates: a push provider call is a network round trip to an external service of the order of 100 ms (assumption), so 4,060 per second needs about 400 concurrent in-flight requests by Little's law (rate times residence time), spread across the adapter workers. The internal hops (API to queue to adapter) are datacenter round trips of 500 us each, negligible against the 5 s target.

High-level design

Container view

One API in front of a preference lookup, one durable queue per channel and priority class (six queues) with competing consumers, provider adapters behind circuit breakers with a secondary provider each, a delivery status store written from provider callbacks, and a dead-letter queue with an operator view. The channel queues are separate because the channels have different latency targets and different providers; a shared queue would let an SMS backlog delay push. Fan-out of one event to several channels happens at the API, the way a publish-subscribe topic fans one message out to several queues.

flowchart TB
  subgraph Notification system
    API[Notification API: idempotency key, validation] --> P[Preference lookup: channels, quiet hours, per-user limits]
    API --> QP[[push queues: transactional, informational]]
    API --> QE[[email queues: transactional, informational]]
    API --> QS[[sms queues: transactional, informational]]
    QP --> AP[Push adapter + breaker]
    QE --> AE[Email adapter + breaker]
    QS --> AS[SMS adapter + breaker]
    AP --> ST[(delivery status)]
    AE --> ST
    AS --> ST
    AP -.->|retries exhausted on both providers| DL[[dead letter]]
    AE -.-> DL
    AS -.-> DL
  end
  T[Product teams] --> API
  AP --> FCM{{Push providers}}
  AE --> SES{{Email provider}}
  AS --> SMS{{SMS provider}}

flowchart TB
  subgraph Notification system
    API[Notification API: idempotency key, validation] --> P[Preference lookup: channels, quiet hours, per-user limits]
    API --> QP[[push queues: transactional, informational]]
    API --> QE[[email queues: transactional, informational]]
    API --> QS[[sms queues: transactional, informational]]
    QP --> AP[Push adapter + breaker]
    QE --> AE[Email adapter + breaker]
    QS --> AS[SMS adapter + breaker]
    AP --> ST[(delivery status)]
    AE --> ST
    AS --> ST
    AP -.->|retries exhausted on both providers| DL[[dead letter]]
    AE -.-> DL
    AS -.-> DL
  end
  T[Product teams] --> API
  AP --> FCM{{Push providers}}
  AE --> SES{{Email provider}}
  AS --> SMS{{SMS provider}}

Boxes are containers, the outer frame is the system boundary, hexagons are external systems, double brackets are queues; each queue node stands for two queues, one per priority class, and the dotted edges carry messages whose retries are exhausted.

The data model

Entities

Notification {notification_id, idempotency_key = (event_id, user_id, channel), team_id, event_type, priority, template_id, payload, created_at, expires_at}. Preference {user_id, channel, enabled, quiet_hours (start, end, timezone), per_channel_limits}. DeviceToken {user_id, platform, token, updated_at} and Address {user_id, email, phone} inside the adapter boundary only. DeliveryStatus {notification_id, channel, attempts[] of (provider, time, result), state (queued, held, sent, delivered, failed, dead)}. Template {team_id, template_id, channel, body, version}.

API

POST /notifications {event_id, user_id, event_type, channels[], priority, template_id, variables, expires_at} with an Idempotency-Key header (the Stripe write-up and the IETF Idempotency-Key draft describe the semantics; Stripe's API reference keeps a key for at least 24 hours, the lifetime this design adopts) returns {notification_ids[]}; GET /notifications/{id} returns the status history; PUT /users/{id}/preferences; POST /templates. Teams authenticate with per-team credentials and are rate limited per team.

The core flow, step by step

One informational event for a user who allows push and email, sent during the day.

The API validates the request, checks the idempotency key in a 24-hour key store, and returns the stored result if the key is known.
The API reads the user's preferences: push and email enabled, no quiet hours now, under the per-user limit of 10 informational pushes per hour (assumption). SMS was not requested. It renders the templates and writes one Notification per channel in state queued.
The API enqueues one message per channel on that channel's queue and returns the notification ids. Writing the record and enqueueing are made safe by writing the record first and treating a duplicate enqueue as harmless (the consumer deduplicates by notification id).
A push adapter worker takes the message (visibility timeout 30 s), loads the device tokens, and calls the primary push provider with a collapse identifier equal to the event id so that a redelivered or updated notification replaces rather than duplicates on the device (APNs exposes a collapse identifier on its HTTP/2 provider API; the adapter uses each provider's equivalent), and an expiry equal to expires_at.
On success the worker records the attempt and state sent, and deletes the queue message. On a transient failure it retries up to 3 times with exponential backoff and jitter; on a hard failure (invalid token) it marks failed and removes the token.
Provider callbacks (delivered, bounced, unsubscribed) arrive on a webhook, are matched by provider message id, and update the status store; the email adapter maps a bounce to a suppressed address.
A message that exhausts retries on both providers goes to the dead-letter queue with its last error; the operator view lets a team replay or discard it.

Scaling and the bottleneck

Providers, breakers and backlogs

The bottleneck is the provider, not the service: each provider has its own rate limits and its own bad days. The adapter keeps a circuit breaker per provider; when the primary's error rate crosses a threshold the breaker opens and traffic shifts to the secondary within the 60 s failover target, with the breaker probing the primary in a half-open state. Adapter workers scale on queue age (oldest message age), not on queue length, because age is the SLI the targets are written in. The Builders' Library article on queue backlogs is the reference for what follows: a backlog of a slow channel must not starve a fast one (separate queues), one abusive sender must not fill everyone's queue (per-team quotas at the API, and a sidelined queue for a team whose messages keep failing), and an old backlog must drain by age so fresh transactional messages are not stuck behind stale informational ones (the transactional class has its own queue per channel).

Preferences, quiet hours and collapsing

Preferences are read on every send: 5,800 reads per second at peak against a 20-million-row table is a cache job (cache-aside with a short TTL, invalidated on preference change). Quiet hours mean held messages: a held notification is stored with a release time and re-enqueued by a scheduler when the window ends, with the expiry still enforced; the held set per user is capped and the newest wins when the cap is exceeded (collapsing), so a user does not wake up to 40 pings. Per-user rate limits are token buckets keyed by user and channel in the same cache.

Failure modes and what degrades

Each row names the failure, what the user and the team see, and the mitigation.

Failure	Effect	Mitigation
Primary push provider down	push latency rises until failover	breaker opens on error rate; secondary provider takes traffic within 60 s; alert on breaker state
Preference service down	cannot decide channels	serve stale preferences from the cache first; with no cached entry the default is send push, hold SMS and email until preferences return (product decision, stated: an unknown preference may be an unsubscribe, and email's 60 s target tolerates a short hold)
Caller retries with the same event	would duplicate	idempotency key for 24 hours returns the first result
Consumer crashes after the provider call, before the ack	the message is redelivered	collapse identifier at the provider; status store shows the attempt; the second attempt sees state sent and skips
SMS backlog from a provider incident	SMS late	its own queue; push and email unaffected; age-based alert; operator can pause the channel
Template bug renders bad text	wrong message to users	template versions with a canary send to an internal audience; a kill switch per template

Trade-offs and the alternative not taken

The decisions a reviewer reads first, including the library-versus-service question the design-doc frame of the outline raises.

Decision	Options	Chosen	Why
Shape	a library in each team's service, a shared service, a managed notification product	a shared service; the managed product is one provider behind the adapter	shared dedupe and preferences need shared state; provider changes deploy once
Queues	one queue with a channel field versus one queue per channel and priority	one per channel and priority class	channel latency targets differ; a backlog must not cross channels
Duplicates	exactly-once attempts versus at-least-once plus collapse and dedupe	at-least-once with idempotency key, status check and collapse id	exactly-once across a provider boundary is not available; collapse makes a rare duplicate harmless on the device
Preference outage	fail closed (send nothing) versus per-channel default	send push, hold SMS and email	silence is also a failure for transactional events; SMS costs money and wakes people; an unknown email preference may be an unsubscribe, and a held email still meets its 60 s target if the outage is short

What changes at 10x

At 500 million notifications per day and fifty sending teams the service becomes a platform: per-team quotas and budgets (SMS unit cost is the driver), priority classes enforced during incidents by shedding informational traffic first, template governance with review, regional deployments so that a user's data stays in region, and a compliance layer (consent records, unsubscribe handling per channel, retention). The status store becomes partitioned by notification id with time-based tiers. The dead-letter view becomes per team with replay quotas, because at that scale a team's bad deploy can produce millions of failed sends in minutes.

Self-check rubric

Read, not scored: find the level your answer reached.

L1, mid-level: applies

A queue per channel with competing consumers, provider adapters, bounded retries with jitter, and an idempotency key for callers.
A delivery status store written from attempts and provider callbacks, and the per-channel rates derived from the assumptions.
The duplicate story stated honestly: what is caught (caller retries, consumer redelivery) and what is not (two event ids for one event).

L2, senior: chooses with numbers and defends

Preferences and quiet hours with held messages, a release scheduler, collapsing and per-user limits.
Circuit breakers per provider with a secondary provider and a 60 s failover argument, and workers scaled on queue age.
Backlog handling: separate queues per channel and priority, per-team quotas, sidelining of a failing sender, age-based draining.
The preference-service outage default chosen and justified.

L3, staff: sets the constraint and judges

The platform questions: quotas and cost per channel per team, priority classes during incidents, template governance, consent and retention.
Why a library per team or a managed product alone does not solve the problem, with the managed product kept as a provider.
The deprecation plan for the teams' existing direct-to-provider code paths, with shadow mode before cutover.

The 45-minute interview frame

0:00 to 0:05 Requirements: who chooses channels, what duplicate means, quiet hours, priority classes, status retention, where personal data lives. Say the numbers aloud: 5,800 notifications per second at peak (4,060 push), a 244,000-message push backlog after a 60 s provider stall, about 400 concurrent provider calls in flight.
0:05 to 0:10 Entities and API: Notification with its idempotency key of (event, user, channel), Preference, DeviceToken and Address inside the adapter boundary, DeliveryStatus, Template; the send and status endpoints.
0:10 to 0:20 High-level design with the diagram: API, preference lookup, a queue per channel and priority class, adapters with breakers and secondary providers, status store, dead letter. Say why the queues are separate.
0:20 to 0:40 Two deep dives: duplicates and retries (idempotency key, consumer redelivery, collapse identifiers, bounded retries then failover then dead letter), then backlogs and failover (breakers, scaling on age, per-team quotas and sidelining, quiet-hour holds and collapsing).
0:40 to 0:45 Failure modes from the table, the 10x platform (quotas, priority shedding, compliance, regions), and the L3 sentence: the product decision hiding in this design is what to do when the preference service is down, and it should be written down before the review.

The design-doc frame

Context and scope. Five product teams send push, email and SMS from their own code paths with three providers and no shared dedupe; the incidents of the last quarters (duplicate sends after retries, a provider outage with no failover, no quiet hours). Scope: one internal service for all teams; marketing campaigns and an in-app inbox out of scope.
Goals and non-goals. At-least-once delivery with dedupe so the user sees one message per event; failover within 60 s; preferences and quiet hours; status queryable by the sending team. Non-goals: exactly-once across providers, real-time analytics.
Design. This answer fills it: the API with idempotency keys, the preference lookup, one queue per channel and priority, adapters behind breakers, the status store, the dead-letter view.
Alternatives considered. A library per team (no shared dedupe, three deployments of every provider change); one queue for all channels (SMS backlog delays push); a managed product alone (preferences still need our data), kept as a provider.
Cross-cutting concerns. Security: internal API, per-team credentials, personal data only inside the adapter boundary and the status store, encrypted at rest. Observability: queue age per channel, send success by provider, dedupe hit ratio, breaker state; alert on queue age over the channel target. Cost: per-channel unit cost with SMS the driver, budget with the sending team via a quota. Rollout: shadow mode, then one team's push, then all channels; rollback is a config flag per team.
Review questions the doc must answer. What happens to a notification when the preference service is down (stale cache first, then send push, hold SMS and email); how retries are bounded per provider (3 attempts with jitter, then failover, then dead letter); the dedupe key's lifetime (24 h) and the duplicate it does not catch.

Depth notes

L1. A queue per channel (push, email, SMS), provider adapters, retries and dedupe.
L2. User preferences and quiet hours, per-user rate limits, collapsing, delivery tracking, provider failover.
L3. A notification platform for many internal teams (templates, priorities, quotas, cost per channel, compliance).

Sources

docFirebase Cloud Messaging: architectural overview (firebase.google.com)
docApple Developer: Sending notification requests to APNs (developer.apple.com)
docAmazon SNS Developer Guide: What is Amazon SNS? (docs.aws.amazon.com)
docAmazon Builders' Library: Avoiding insurmountable queue backlogs (builder.aws.com)
blogStripe: Designing robust and predictable APIs with idempotency (stripe.com)
docStripe API reference: Idempotent requests (key lifetime) (docs.stripe.com)
specIETF draft: The Idempotency-Key HTTP Header Field (datatracker.ietf.org)
docAzure Architecture Center: Competing Consumers pattern (learn.microsoft.com)