# Messaging or streaming?

> Should the pipe remember a message after it has been delivered? Messaging (SQS, SNS, EventBridge) delivers each message and forgets it, so your database is the memory. A stream (Kinesis, Kafka) is a log: it keeps every record in order, and each consumer reads at its own position, replay included. A topic with an S3 archive sits between them: a log without a stream. Start with messaging; move to a stream when ordering, replay and several readers are the job itself.

- A concept by Nicolas Dao (Cloudless Labs, Sydney): the decision before the tool
- Area: [Moving data between services](https://cloudlesslabs.com/concepts/areas/moving-data/index.md)
- Last reviewed: 1 November 2026, by Nicolas Dao
- Web page: https://cloudlesslabs.com/concepts/messaging-or-streaming/

## What you'll be able to decide

Whether your events need a queue, a topic, a stream or a topic with an archive, what each costs a month before it carries anything, and which signal tells you to move from one to the next.

## In short

### The options

- **A. Messaging:** The pipe delivers each message and forgets it: a queue holds it until one consumer takes it, a topic or a bus copies it and keeps nothing. Tools: [Amazon SQS](https://cloudlesslabs.com/stack/aws-sqs/index.md) (My default), [Amazon SNS](https://cloudlesslabs.com/stack/aws-sns/index.md) (Use when it fits), [Amazon EventBridge](https://cloudlesslabs.com/stack/aws-eventbridge/index.md) (My default).
- **B. Fan-out with an archive:** A topic copies every message to one queue per consumer, and one more subscriber writes every message to S3. Replay is possible, but it's a job you write. Tools: [Amazon SNS](https://cloudlesslabs.com/stack/aws-sns/index.md) (Use when it fits), [Amazon Data Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md) (My default), [Amazon S3](https://cloudlesslabs.com/stack/aws-s3/index.md) (My default).
- **C. Streaming:** The pipe is a log: it stores every record in order per key, for a retention you choose, and every consumer keeps its own position. Tools: [Amazon Kinesis](https://cloudlesslabs.com/stack/aws-kinesis/index.md) (Use when it fits), [Amazon MSK (managed Kafka)](https://cloudlesslabs.com/stack/aws-msk/index.md) (Use when it fits).

### The rule of thumb

If you don't need ordering, replay or several independent readers, you don't need Kinesis. From [Amazon Kinesis](https://cloudlesslabs.com/stack/aws-kinesis/index.md).

### How the answer changes with scale

- **Low or spiky volume, any number of consumers:** Messaging: no idle floor, and SQS in batches is nearly free.
- **Someone asks for history, or a consumer needs a re-run after a bug:** Add an archive subscriber to the topic before you add a stream.
- **Sustained volume, order per key, several independent readers:** A stream: on demand while the volume is unknown, provisioned once it's steady.

## The problem, in plain words

Something happens in your system: an order is paid, a sensor reads 21.4 °C, a user signs up. Other code has to act on it. Between the code that knows and the code that acts, you put a pipe.

Every pipe falls into one of two families, and one question separates them: **once a message has been delivered and handled, does the pipe still have it?**

- In **messaging**, no. The pipe's job is delivery. A queue holds a message until a consumer takes it, then deletes it. A topic or a bus copies it to whoever subscribed, and keeps nothing.
- In **streaming**, yes. The pipe is a log: it stores every record in order, for as long as you choose, and each consumer keeps its own position in it.

That one difference decides who can read the past, whether you can replay, whether order is kept, and how the bill grows.

## The picture

The same producer, three pipes. Read each lane left to right: what happens to a message after it's handled is the whole difference.

**Diagram:** Three pipes from the same producer: a queue that deletes each message once it's handled, a topic that copies to a queue per consumer and to an S3 archive, and a log that keeps every record while consumers read at their own positions.

- **Messaging:** Producer → Queue → Consumer → Message deleted. Once handled, the message is gone. The database is the memory.
- **Fan-out with an archive:** Producer → Topic → [Queue → billing / Queue → email / Firehose → S3: every message, for good]. Replay is a script you write: read the files, send them to one consumer's queue.
- **Streaming:** Producer → Log, in order per key → [Consumer A at record 1,204 / Consumer B at record 1,190 / New consumer at record 0]. Records stay for the retention you choose. A new consumer starts from the oldest one.

## The options

**Messaging: the pipe forgets.** [SQS](https://cloudlesslabs.com/stack/aws-sqs/index.md) hands each message to one worker and forgets it once it's done. [SNS](https://cloudlesslabs.com/stack/aws-sns/index.md) broadcasts: one publish, a copy for every subscriber, nothing kept. [EventBridge](https://cloudlesslabs.com/stack/aws-eventbridge/index.md) routes each event by its content, and keeps nothing unless you turn on its archive. The memory of what happened lives in your database, not in the pipe.

**Streaming: the pipe remembers.** [Kinesis](https://cloudlesslabs.com/stack/aws-kinesis/index.md) keeps an ordered log per partition key, for 24 hours up to a year, and every consumer reads it at its own pace: one Lambda writes to Postgres, another feeds a fraud model, a third replays last Tuesday after a bug fix. A consumer added next month can start from the oldest record still kept. Kafka is the same idea in open source; on AWS it's [Amazon MSK](https://cloudlesslabs.com/stack/aws-msk/index.md).

**The middle: fan-out with an archive.** A topic copies every message into one queue per consumer, plus one more subscriber whose only job is to write every message to S3. I used it to move an agtech platform off a single server ([the project](https://cloudlesslabs.com/work/vps-to-aws/index.md)): every raw reading was archived to S3, and at cut-over the readings newer than the last backup were replayed from it. Writes were idempotent, so a reading replayed twice changed nothing. Today the archiver would be a [Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md) subscription, which writes compressed files with no code. It's **a log without a stream**: the history is there, but nobody has a position in it, order isn't kept, and replay is a job you write.

## A concrete scenario

A delivery company, at three stages.

**At launch,** a few thousand orders a day. When an order is paid, the receipt email and the warehouse must both know. SNS copies the event into two SQS queues, each read by its own Lambda in batches, with its own retries and dead-letter queue. The orders live in Postgres, so a pipe that forgets loses nothing. The bill is cents, and there's no idle floor.

**Six months in,** analytics wants every event, and a bug in the warehouse consumer has mishandled a day of orders. Add a Firehose subscription that writes every event to S3. Analytics reads the files, and the warehouse fix is a script that puts Tuesday's files back on its queue. Still no floor.

**In year two,** live tracking arrives. A thousand vans each send a position every second, read by the live map, the ETA model and the lake, in order per van, and the ETA team wants to re-run yesterday after every model change. Ordering, replay and several independent readers are now the job itself. That's a stream: Kinesis, partitioned by van ID, about $78 a month on provisioned shards. Through a topic, three consumers at that rate cost about $3,910 a month, and a thousand publishes a second is two thirds of what a Sydney account can publish by default ([Queue, bus or stream?](https://cloudlesslabs.com/stack/guides/queue-or-stream/index.md)).

The orders stay on the topic and the queues. Two constructs, side by side, each doing the job it fits.

## The trade-offs

| | Messaging (queue, topic, bus) | Topic with an archive | Stream (a log) |
|---|---|---|---|
| Keeps a message after delivery | Queue: until it's consumed. Topic, classic bus: no | In S3, as files, for as long as you keep them | Yes: 24 hours to 365 days on Kinesis |
| A new consumer reads the past | No | From the files, as a batch job | Yes, from any point in the retention |
| Replay | No (EventBridge archives and SNS FIFO topics add a coarse one) | A script you write | Built in |
| Order | FIFO queues and topics only | Not kept | Per partition key |
| A failing message | Retried on its own, then to a dead-letter queue | The same, per queue | Holds up its shard until you set retry and age limits |
| Adding a consumer | Add a queue | Add a queue | Register it; past two, enhanced fan-out |
| The bill grows with | Messages × consumers | Messages × consumers, plus GB archived | Throughput: shards, or GB written and read |
| Floor when idle | None | None | About $13 (one provisioned shard) or $36 (on demand); Kafka on MSK about $558 |

## The rule of thumb

**Let the pipe forget by default: your database is the memory.** Add an archive the day someone asks for history. Move to a stream when ordering, replay and several readers are the job, at a volume that keeps it busy. As the Kinesis entry puts it, most "streaming" workloads are queues in disguise.

## When the answer changes with scale

- **Idle or spiky:** messaging, because it has no floor. 50 million SQS messages cost about $6 batched; an idle on-demand stream still costs about $36 a month.
- **A stream at unknown volume:** start on demand, and move to provisioned shards once the volume is steady.
- **Topics hit a wall first.** In Sydney, SNS publishes 1,500 messages a second per account by default and EventBridge takes 1,200 PutEvents calls a second. Both can be raised, but past about a thousand a second with several consumers, a stream costs a fraction of the fan-out.
- **Streams have walls too.** One partition key tops out at 1 MB/s or 1,000 records a second, and a third standard consumer starts starving the others.

The cost of every design at every volume, as a chart, is in [Queue, bus or stream?](https://cloudlesslabs.com/stack/guides/queue-or-stream/index.md)

## Questions that change the answer

- How many independent readers does each event have? One is a queue; several is a topic with a queue each, or a stream.
- Will anyone need to read last week again? A re-run after a bug is a replay; if it's rare, an archive will do.
- Must events for one device, account or order be handled in order? That's FIFO, or a stream partitioned by that key.
- How many events a second, sustained, at peak? Below a few hundred a second, a stream mostly pays its floor.

Doesn't cover:

- Change data capture out of a database (DMS, Debezium) is its own concept.
- Real-time screens for many viewers (WebSockets, Durable Objects) are a different pipe: see [A live auction on Durable Objects](https://cloudlesslabs.com/stack/guides/live-auction/index.md).

## Check your call

- **A SaaS sends a welcome email, creates a workspace and tells sales whenever a user signs up. About fifty sign-ups a day.** Messaging: a topic with one queue per consumer. The user is in Postgres, nothing needs replaying, and a stream would pay its floor for nothing.
- **Three teams read the same payment events, in order per account, at about 2,000 a second, and must re-run the last 30 days whenever a fraud rule changes.** A stream, partitioned by account ID, with 30 days' retention. Ordering, replay and several independent readers are the job, and 2,000 a second is past what one Sydney account can publish to SNS by default.
- **Analytics wants every order event in the lake. A replay happens maybe twice a year, after a bug.** Keep the topic and its queues, and add an archive subscriber that writes to S3. A rare replay is a script, not a reason to run a stream.

## The technology on this site

- [Queue, bus or stream? SQS, EventBridge, Kinesis or Kafka](https://cloudlesslabs.com/stack/guides/queue-or-stream/index.md): the decision guide
- [SQS, EventBridge or SNS?](https://cloudlesslabs.com/stack/guides/sqs-eventbridge-or-sns/index.md): the decision guide
- [An event backbone that keeps everything](https://cloudlesslabs.com/stack/guides/event-archive/index.md): the decision guide
- [Kinesis or Kafka on AWS?](https://cloudlesslabs.com/stack/guides/kinesis-or-kafka/index.md): the decision guide
- [IoT ingestion into Postgres](https://cloudlesslabs.com/stack/guides/iot-ingestion/index.md): the decision guide
- [Event-driven plumbing](https://cloudlesslabs.com/stack/use-cases/events/index.md): the use case
- [Device and IoT data](https://cloudlesslabs.com/stack/use-cases/iot/index.md): the use case
- [Background jobs and retries](https://cloudlesslabs.com/stack/use-cases/background/index.md): the use case
- [Off a fragile VPS and onto AWS, with almost no downtime](https://cloudlesslabs.com/work/vps-to-aws/index.md): in our client work
- [Real-time ingestion for a large farm-sensor network](https://cloudlesslabs.com/work/realtime-iot/index.md): in our client work

## Related

- [At least once, and idempotency](https://cloudlesslabs.com/concepts/delivery-guarantees/index.md): read next
- [Pay for the work, or pay to be ready?](https://cloudlesslabs.com/concepts/pay-to-be-ready/index.md): read next
- [Moving data between services](https://cloudlesslabs.com/concepts/areas/moving-data/index.md): the area, and the technology for each answer
- [Every concept](https://cloudlesslabs.com/concepts/index.md)
- [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)