# An event backbone that keeps everything

> My services publish events through API Gateway and SNS. How do I keep every message, cheaply and for good, and replay history to one consumer when I need to, without running a stream? API Gateway in front, SNS fanning out to SQS queues and Lambdas, and one more subscriber: a Firehose stream that writes every message to S3 as compressed hourly files. The archive costs about $0.40 per million messages, against about $5.80 for a Lambda that writes each message to S3. Replay is a small script: pick a time window, drop duplicates by message ID, sort by timestamp, send to that one consumer's own queue.

- A decision guide from The Stack by Nicolas Dao (Cloudless Labs, Sydney)
- Last reviewed: 1 November 2026, by Nicolas Dao
- Web page: https://cloudlesslabs.com/stack/guides/event-archive/. The whole stack in one file: https://cloudlesslabs.com/stack/brief.md
- Tools: [API Gateway](https://cloudlesslabs.com/stack/aws-api-gateway/index.md) (Use when it fits), [Amazon SNS](https://cloudlesslabs.com/stack/aws-sns/index.md) (Use when it fits), [Amazon Data Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md) (My default), [Amazon S3](https://cloudlesslabs.com/stack/aws-s3/index.md) (My default), [Amazon SQS](https://cloudlesslabs.com/stack/aws-sqs/index.md) (My default), [AWS Lambda](https://cloudlesslabs.com/stack/aws-lambda/index.md) (My default), [S3 + Glue + Athena](https://cloudlesslabs.com/stack/aws-athena-data-lake/index.md) (Use when it fits)

**Scenario:** producers (apps, partners, devices) send events over HTTPS. Several services react to them: billing, email, a live dashboard, analytics. Nobody wants to run a stream, but everyone wants every message kept, and the occasional "reprocess last Tuesday for analytics" to be possible.

I run this pattern in production. It's the middle ground between a queue and a stream described in [Queue, bus or stream?](https://cloudlesslabs.com/stack/guides/queue-or-stream/index.md): a log without a stream.

## The design

```
Producers ─▶ API Gateway (REST) ─▶ SNS topic ─┬─▶ SQS: billing ─▶ Lambda, in batches
                                              ├─▶ SQS: email ───▶ Lambda, in batches
                                              ├─▶ Lambda: live dashboard
                                              └─▶ Firehose ─▶ S3: 2026/11/01/14/….gz
                                                                (every message, for good)

Replay:  S3 archive ─▶ Athena or a script (time window, no duplicates, in order) ─▶ one consumer's SQS queue
```

1. **API Gateway** takes the HTTPS calls and publishes to SNS with no Lambda in between: a REST API's AWS integration calls `Publish` directly. (An HTTP API's direct integrations cover SQS, EventBridge and Kinesis, not SNS.) When the producers are your own services on AWS, skip API Gateway and let them publish to SNS with IAM.
2. **SNS** copies each message to every subscriber. Each consumer that does real work gets its own SQS queue, so it has its own buffer, retries and dead-letter queue, and Lambda reads it in batches. A consumer that must react instantly (the dashboard) can subscribe a Lambda directly.
3. **Firehose** is one more subscriber. It batches messages into GZIP files under hourly date prefixes in S3, with no code. Every message is kept for good, at S3's price.
4. **A new consumer** is a new subscription. It sees messages from the day it subscribes, and history through a replay.

## What it costs

Per million messages, Sydney, 1 November 2026:

| Piece | Per million messages |
|---|---|
| API Gateway, REST API | $3.50 |
| SNS publish (one per request: API Gateway can't batch) | $0.50 |
| The archive: SNS → Firehose ($0.225) + Firehose, each message billed as 5 KB (about $0.18) | about $0.40 |
| Each SQS consumer: the copy into the queue, then receive and delete in batches of 10 | about $0.48 |
| Each Lambda subscribed straight to SNS: one invocation per message (requests only) | $0.20 |

At 100 events a second (about 263 million a month) with three SQS consumers: API Gateway about $920, SNS about $131, the archive about $105, the queues about $378. About $1,535 a month, of which the front door is the biggest share: producers on AWS that publish to SNS directly save the $920. S3 storage adds about $1.60 a month for every month of history kept (about 63 GB, compressed).

Sources, checked on 1 November 2026: [API Gateway pricing](https://aws.amazon.com/api-gateway/pricing/), [SNS pricing](https://aws.amazon.com/sns/pricing/) (the AWS Price List: $0.225 per million messages delivered to Firehose in Sydney), [Firehose pricing](https://aws.amazon.com/firehose/pricing/), [SQS pricing](https://aws.amazon.com/sqs/pricing/).

## Firehose, not a Lambda, for the archive

The obvious archiver is a Lambda subscribed to the topic that writes each message to S3. Mine is one. It works, and the bill is in the wrong place, which is why Firehose is how I'd build it today:

| Archiver | Per million messages | At 100 events a second |
|---|---|---|
| A Lambda writing one S3 object per message (128 MB, about 50 ms) | about $5.80: $5.50 of it is S3 PUTs | about $1,520 a month |
| An SNS subscription to Firehose | about $0.40 | about $105 a month |

SNS invokes the Lambda once per message, so it can only write one object per message, and S3 bills $5.50 per million PUTs in Sydney. Firehose batches thousands of messages into each file, so the PUTs all but vanish, and it compresses JSON several times over. There's a second win: replaying a few hundred files a day under date prefixes is far simpler than listing millions of tiny objects.

Setting it up takes four pieces, all in Pulumi or Terraform:

1. **A Firehose stream** (Direct PUT) writing to S3: a buffer of a few minutes or a few MB, GZIP compression, and the default date prefix (`YYYY/MM/DD/HH`, UTC).
2. **An SNS subscription** with protocol `firehose`, and an IAM role SNS can assume to write to the stream (`SubscriptionRoleArn`).
3. **Raw message delivery left off,** so each line in the files keeps SNS's envelope: the `MessageId` and `Timestamp` are what make a replay free of duplicates and in order. Each message lands on its own line.
4. **A dead-letter queue on the subscription** (a redrive policy), so a delivery Firehose keeps refusing lands somewhere instead of being lost.

## Replaying history to one consumer

1. **Pick the window** from the prefixes: last Tuesday is `2026/10/27/`, all 24 hours of it (UTC).
2. **Read it** with Athena (a table over the archive, partitioned by those prefixes) or a script that reads the files.
3. **Drop duplicates by `MessageId`:** Firehose and SNS deliver at least once.
4. **Sort by `Timestamp`** when order matters to that consumer.
5. **Send the messages to that consumer's own SQS queue**, in batches of 10, never back to the topic: republishing would hand every other subscriber a second copy.
6. **Overlap the live traffic a little and rely on the consumer ignoring duplicates,** so nothing falls between the end of the replay and the live messages.

It's a script you write once. A stream does steps 1 to 6 with one setting (move the consumer back to a timestamp), which is the honest gap between this design and Kinesis.

## Gotchas

- **Each message is billed as at least 5 KB by Firehose.** With 1 KB messages that's still about $0.18 per million, and messages can't be packed together, because SNS delivers one per record.
- **A Firehose stream in Sydney accepts 1,000 requests a second and 1 MiB/s by default.** AWS raises a Direct PUT stream's limit automatically when it throttles; above about 1,000 messages a second sustained, ask for more up front, or split across streams.
- **The archive isn't live.** Files land when the buffer flushes, after a few minutes. Live consumers stay on SNS.
- **SNS publishes 1,500 messages a second per account in Sydney by default.** It's a soft quota: raise it before launch, not when publishes start failing.

## When it stops being enough

Move to a stream ([Kinesis](https://cloudlesslabs.com/stack/aws-kinesis/index.md), or [Kafka on MSK](https://cloudlesslabs.com/stack/aws-msk/index.md) when Kafka itself is required) when one of these becomes true:

- **Replay is routine, not exceptional:** new consumers often need history, or reprocessing is part of the job.
- **Events for one key must be processed one by one, in order,** at a volume FIFO queues can't carry.
- **Volume and consumers make the bill:** from about 100 events a second, the fan-out to three queues costs about 20 times a provisioned Kinesis stream (the chart in [Queue, bus or stream?](https://cloudlesslabs.com/stack/guides/queue-or-stream/index.md)).

Until then, this design is cheaper, keeps more history than a stream, filters per subscriber for free, and has nothing to size.

## Related

- [API Gateway](https://cloudlesslabs.com/stack/aws-api-gateway/index.md): the full entry (verdict, pricing, gotchas)
- [Amazon SNS](https://cloudlesslabs.com/stack/aws-sns/index.md): the full entry (verdict, pricing, gotchas)
- [Amazon Data Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md): the full entry (verdict, pricing, gotchas)
- [Amazon S3](https://cloudlesslabs.com/stack/aws-s3/index.md): the full entry (verdict, pricing, gotchas)
- [Amazon SQS](https://cloudlesslabs.com/stack/aws-sqs/index.md): the full entry (verdict, pricing, gotchas)
- [AWS Lambda](https://cloudlesslabs.com/stack/aws-lambda/index.md): the full entry (verdict, pricing, gotchas)
- [S3 + Glue + Athena](https://cloudlesslabs.com/stack/aws-athena-data-lake/index.md): the full entry (verdict, pricing, gotchas)
- [Messaging or streaming?](https://cloudlesslabs.com/concepts/messaging-or-streaming/index.md): the concept behind it
- [Every decision guide, the choices between tools first](https://cloudlesslabs.com/stack/guides/index.md)
- [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)