An event backbone that keeps everything

Scenario: producers (apps, partners, devices) send events over HTTPS. Several services react to them: billing, email, a live dashboard, analytics. Nobody wants to run a stream, but everyone wants every message kept, and the occasional "reprocess last Tuesday for analytics" to be possible.

I run this pattern in production. It's the middle ground between a queue and a stream described in Queue, bus or stream?: a log without a stream.

The design

Producers ─▶ API Gateway (REST) ─▶ SNS topic ─┬─▶ SQS: billing ─▶ Lambda, in batches
                                              ├─▶ SQS: email ───▶ Lambda, in batches
                                              ├─▶ Lambda: live dashboard
                                              └─▶ Firehose ─▶ S3: 2026/11/01/14/….gz
                                                                (every message, for good)

Replay:  S3 archive ─▶ Athena or a script (time window, no duplicates, in order) ─▶ one consumer's SQS queue

1. API Gateway takes the HTTPS calls and publishes to SNS with no Lambda in between: a REST API's AWS integration calls Publish directly. (An HTTP API's direct integrations cover SQS, EventBridge and Kinesis, not SNS.) When the producers are your own services on AWS, skip API Gateway and let them publish to SNS with IAM.
2. SNS copies each message to every subscriber. Each consumer that does real work gets its own SQS queue, so it has its own buffer, retries and dead-letter queue, and Lambda reads it in batches. A consumer that must react instantly (the dashboard) can subscribe a Lambda directly.
3. Firehose is one more subscriber. It batches messages into GZIP files under hourly date prefixes in S3, with no code. Every message is kept for good, at S3's price.
4. A new consumer is a new subscription. It sees messages from the day it subscribes, and history through a replay.

What it costs

Per million messages, Sydney, 1 November 2026:

At 100 events a second (about 263 million a month) with three SQS consumers: API Gateway about $920, SNS about $131, the archive about $105, the queues about $378. About $1,535 a month, of which the front door is the biggest share: producers on AWS that publish to SNS directly save the $920. S3 storage adds about $1.60 a month for every month of history kept (about 63 GB, compressed).

Sources, checked on 1 November 2026: API Gateway pricing, SNS pricing (the AWS Price List: $0.225 per million messages delivered to Firehose in Sydney), Firehose pricing, SQS pricing.

Firehose, not a Lambda, for the archive

The obvious archiver is a Lambda subscribed to the topic that writes each message to S3. Mine is one. It works, and the bill is in the wrong place, which is why Firehose is how I'd build it today:

SNS invokes the Lambda once per message, so it can only write one object per message, and S3 bills $5.50 per million PUTs in Sydney. Firehose batches thousands of messages into each file, so the PUTs all but vanish, and it compresses JSON several times over. There's a second win: replaying a few hundred files a day under date prefixes is far simpler than listing millions of tiny objects.

Setting it up takes four pieces, all in Pulumi or Terraform:

1. A Firehose stream (Direct PUT) writing to S3: a buffer of a few minutes or a few MB, GZIP compression, and the default date prefix (YYYY/MM/DD/HH, UTC).
2. An SNS subscription with protocol firehose, and an IAM role SNS can assume to write to the stream (SubscriptionRoleArn).
3. Raw message delivery left off, so each line in the files keeps SNS's envelope: the MessageId and Timestamp are what make a replay free of duplicates and in order. Each message lands on its own line.
4. A dead-letter queue on the subscription (a redrive policy), so a delivery Firehose keeps refusing lands somewhere instead of being lost.

Replaying history to one consumer

1. Pick the window from the prefixes: last Tuesday is 2026/10/27/, all 24 hours of it (UTC).
2. Read it with Athena (a table over the archive, partitioned by those prefixes) or a script that reads the files.
3. Drop duplicates by MessageId: Firehose and SNS deliver at least once.
4. Sort by Timestamp when order matters to that consumer.
5. Send the messages to that consumer's own SQS queue, in batches of 10, never back to the topic: republishing would hand every other subscriber a second copy.
6. Overlap the live traffic a little and rely on the consumer ignoring duplicates, so nothing falls between the end of the replay and the live messages.

It's a script you write once. A stream does steps 1 to 6 with one setting (move the consumer back to a timestamp), which is the honest gap between this design and Kinesis.

Gotchas

• Each message is billed as at least 5 KB by Firehose. With 1 KB messages that's still about $0.18 per million, and messages can't be packed together, because SNS delivers one per record.
• A Firehose stream in Sydney accepts 1,000 requests a second and 1 MiB/s by default. AWS raises a Direct PUT stream's limit automatically when it throttles; above about 1,000 messages a second sustained, ask for more up front, or split across streams.
• The archive isn't live. Files land when the buffer flushes, after a few minutes. Live consumers stay on SNS.
• SNS publishes 1,500 messages a second per account in Sydney by default. It's a soft quota: raise it before launch, not when publishes start failing.

When it stops being enough

Move to a stream (Kinesis, or Kafka on MSK when Kafka itself is required) when one of these becomes true:

• Replay is routine, not exceptional: new consumers often need history, or reprocessing is part of the job.
• Events for one key must be processed one by one, in order, at a volume FIFO queues can't carry.
• Volume and consumers make the bill: from about 100 events a second, the fan-out to three queues costs about 20 times a provisioned Kinesis stream (the chart in Queue, bus or stream?).

Until then, this design is cheaper, keeps more history than a stream, filters per subscriber for free, and has nothing to size.