Lesson 14 of 15 Case Studies

Design a Notification System

Notification system design for push, SMS and email: event intake, user preferences, templating, per-channel queues, retries, dedup and rate limits.

40-50 min intermediate 0/15 lessons done
Course contents Lesson 14 of 15 · 0 done
On this page (11 sections)
  1. The problem
  2. 1. Requirements
  3. 2. Architecture
  4. 3. Preferences and templates
  5. 4. Reliability
  6. 5. Not annoying users
  7. 6. Scheduling and bulk
  8. 7. Tracking
  9. Key takeaways
  10. Exercises
  11. Quiz

The problem

Design a service that sends notifications on behalf of other services: “your order shipped” (email + push), OTP codes (SMS), “someone liked your photo” (push), weekly digests (email). Many products, many channels, one platform.

1. Requirements

Functional: send via push (APNs, FCM), SMS and email; respect user preferences and quiet hours; templates with personalisation; scheduled and bulk sends; delivery tracking.

Non-functional: ~10M notifications/day normally, spikes of 10x for marketing campaigns; transactional messages (OTP) delivered within seconds; no duplicates for important messages; never lose a transactional notification; third-party providers will be slow or down sometimes.

2. Architecture

services --> notification API --> validate + dedup --> [ intake queue ]
                                                            |
                                     enrichment workers: preferences, contact info, template render
                                                            |
                     +--------------------+-----------------+------------------+
                     v                    v                 v                  v
               [push queue]         [sms queue]       [email queue]     [scheduled store]
                     |                    |                 |
               push workers          sms workers       email workers
                     |                    |                 |
               APNs / FCM          SMS providers      email provider(s)
                     \___________________ status events ___/--> tracking store / analytics
  • Notification API accepts {event_type, user_id, data, idempotency_key, priority} and returns immediately after enqueuing.
  • Separate queue per channel isolates failures: if the SMS provider is down, push and email keep flowing.
  • Priority: OTP and security messages get their own high-priority queues and workers so a marketing blast never delays them.

3. Preferences and templates

A preferences store holds per-user, per-category, per-channel opt-ins (marketing email off, order updates on) plus quiet hours and locale. Enrichment workers check it before sending; legal unsubscribe requirements make this non-negotiable.

Templates are versioned per event type, channel and language, rendered with the event data ("Hi {name}, order {id} shipped"). Rendering in the platform keeps product services simple and wording consistent.

4. Reliability

  • At-least-once processing through the queues, with idempotency keys checked in a dedup store (key → sent) so retries don’t send twice. A user receiving two OTPs is confusing; two charges notifications is alarming.
  • Retries with exponential backoff for provider errors; after N attempts, a dead-letter queue and an alert (lesson 8).
  • Provider failover: keep two SMS/email providers and switch when error rates rise.
  • Invalid tokens: APNs/FCM report uninstalled apps; remove those device tokens.

5. Not annoying users

  • Rate limiting per user and category (lesson 11): at most N push notifications an hour.
  • Batching/digests: “12 people liked your photo” instead of 12 pushes.
  • Quiet hours in the user’s time zone: hold non-urgent messages in the scheduled store.

6. Scheduling and bulk

Scheduled notifications sit in a store indexed by send time; a scheduler polls due items (partitioned so many schedulers share the work) and enqueues them. Campaigns to millions are expanded in batches and throttled to protect providers and your own downstream systems.

7. Tracking

Workers emit status events (queued, sent, delivered, opened, failed) to a stream, aggregated for dashboards and kept per notification for support (“did the user get the OTP?”).

Key takeaways

  • Accept, enqueue, and return: the API never waits for providers.
  • One queue per channel and separate high-priority lanes isolate failures and protect OTPs.
  • Check preferences and quiet hours before sending; render templates centrally.
  • Idempotency keys, retries with backoff, dead letters and provider failover make delivery reliable; rate limits and digests keep it humane.

Exercises

  1. Design the preferences table and the lookup used for “should we send event X to user Y on channel Z now?”.
  2. Write a Python digest builder that groups like-notifications for the same post within 10 minutes into one message.
  3. The email provider is returning 30% errors. Walk through what your system does, step by step.

Quiz

  1. Why use a separate queue per channel?
  2. How do you stop a retry from sending a duplicate OTP?
  3. Where do quiet hours get enforced?
Show answers
  1. So a slow or failing provider on one channel doesn’t block the others, and each channel can scale independently.
  2. Each request carries an idempotency key; the worker records it on send and skips any repeat with the same key.
  3. In the enrichment step, before routing: non-urgent notifications during quiet hours go to the scheduled store for later delivery.

Finished this lesson?

Mark it complete to track your progress (saved in this browser).