What you'll be able to decide
Whether your events need a queue, a topic, a stream or a topic with an archive, what each costs a month before it carries anything, and which signal tells you to move from one to the next.
Messaging (SQS, SNS, EventBridge) delivers each message and forgets it, so your database is the memory. A stream (Kinesis, Kafka) is a log: it keeps every record in order, and each consumer reads at its own position, replay included. A topic with an S3 archive sits between them: a log without a stream. Start with messaging; move to a stream when ordering, replay and several readers are the job itself.
In short
The options
A. Messaging
The pipe delivers each message and forgets it: a queue holds it until one consumer takes it, a topic or a bus copies it and keeps nothing.
B. Fan-out with an archive
A topic copies every message to one queue per consumer, and one more subscriber writes every message to S3. Replay is possible, but it's a job you write.
C. Streaming
The pipe is a log: it stores every record in order per key, for a retention you choose, and every consumer keeps its own position.
The rule of thumb
If you don't need ordering, replay or several independent readers, you don't need Kinesis.
From Amazon Kinesis
How the answer changes with scale
Low or spiky volume, any number of consumers
Messaging: no idle floor, and SQS in batches is nearly free.
Someone asks for history, or a consumer needs a re-run after a bug
Add an archive subscriber to the topic before you add a stream.
Sustained volume, order per key, several independent readers
A stream: on demand while the volume is unknown, provisioned once it's steady.
The problem, in plain words
Something happens in your system: an order is paid, a sensor reads 21.4 °C, a user signs up. Other code has to act on it. Between the code that knows and the code that acts, you put a pipe.
Every pipe falls into one of two families, and one question separates them: once a message has been delivered and handled, does the pipe still have it?
- In messaging, no. The pipe's job is delivery. A queue holds a message until a consumer takes it, then deletes it. A topic or a bus copies it to whoever subscribed, and keeps nothing.
- In streaming, yes. The pipe is a log: it stores every record in order, for as long as you choose, and each consumer keeps its own position in it.
That one difference decides who can read the past, whether you can replay, whether order is kept, and how the bill grows.
The picture
The same producer, three pipes. Read each lane left to right: what happens to a message after it's handled is the whole difference.
Messaging
- Producer
- Queue
- Consumer
- Message deleted
Once handled, the message is gone. The database is the memory.
Fan-out with an archive
- Producer
- Topic
- Queue → billing
- Queue → email
- Firehose → S3: every message, for good
Replay is a script you write: read the files, send them to one consumer's queue.
Streaming
- Producer
- Log, in order per key
- Consumer A at record 1,204
- Consumer B at record 1,190
- New consumer at record 0
Records stay for the retention you choose. A new consumer starts from the oldest one.
The options
Messaging: the pipe forgets. SQS hands each message to one worker and forgets it once it's done. SNS broadcasts: one publish, a copy for every subscriber, nothing kept. EventBridge routes each event by its content, and keeps nothing unless you turn on its archive. The memory of what happened lives in your database, not in the pipe.
Streaming: the pipe remembers. Kinesis keeps an ordered log per partition key, for 24 hours up to a year, and every consumer reads it at its own pace: one Lambda writes to Postgres, another feeds a fraud model, a third replays last Tuesday after a bug fix. A consumer added next month can start from the oldest record still kept. Kafka is the same idea in open source; on AWS it's Amazon MSK.
The middle: fan-out with an archive. A topic copies every message into one queue per consumer, plus one more subscriber whose only job is to write every message to S3. I used it to move an agtech platform off a single server (the project): every raw reading was archived to S3, and at cut-over the readings newer than the last backup were replayed from it. Writes were idempotent, so a reading replayed twice changed nothing. Today the archiver would be a Firehose subscription, which writes compressed files with no code. It's a log without a stream: the history is there, but nobody has a position in it, order isn't kept, and replay is a job you write.
A concrete scenario
A delivery company, at three stages.
At launch, a few thousand orders a day. When an order is paid, the receipt email and the warehouse must both know. SNS copies the event into two SQS queues, each read by its own Lambda in batches, with its own retries and dead-letter queue. The orders live in Postgres, so a pipe that forgets loses nothing. The bill is cents, and there's no idle floor.
Six months in, analytics wants every event, and a bug in the warehouse consumer has mishandled a day of orders. Add a Firehose subscription that writes every event to S3. Analytics reads the files, and the warehouse fix is a script that puts Tuesday's files back on its queue. Still no floor.
In year two, live tracking arrives. A thousand vans each send a position every second, read by the live map, the ETA model and the lake, in order per van, and the ETA team wants to re-run yesterday after every model change. Ordering, replay and several independent readers are now the job itself. That's a stream: Kinesis, partitioned by van ID, about $78 a month on provisioned shards. Through a topic, three consumers at that rate cost about $3,910 a month, and a thousand publishes a second is two thirds of what a Sydney account can publish by default (Queue, bus or stream?).
The orders stay on the topic and the queues. Two constructs, side by side, each doing the job it fits.
The trade-offs
| Messaging (queue, topic, bus) | Topic with an archive | Stream (a log) | |
|---|---|---|---|
| Keeps a message after delivery | Queue: until it's consumed. Topic, classic bus: no | In S3, as files, for as long as you keep them | Yes: 24 hours to 365 days on Kinesis |
| A new consumer reads the past | No | From the files, as a batch job | Yes, from any point in the retention |
| Replay | No (EventBridge archives and SNS FIFO topics add a coarse one) | A script you write | Built in |
| Order | FIFO queues and topics only | Not kept | Per partition key |
| A failing message | Retried on its own, then to a dead-letter queue | The same, per queue | Holds up its shard until you set retry and age limits |
| Adding a consumer | Add a queue | Add a queue | Register it; past two, enhanced fan-out |
| The bill grows with | Messages × consumers | Messages × consumers, plus GB archived | Throughput: shards, or GB written and read |
| Floor when idle | None | None | About $13 (one provisioned shard) or $36 (on demand); Kafka on MSK about $558 |
The rule of thumb
Let the pipe forget by default: your database is the memory. Add an archive the day someone asks for history. Move to a stream when ordering, replay and several readers are the job, at a volume that keeps it busy. As the Kinesis entry puts it, most "streaming" workloads are queues in disguise.
When the answer changes with scale
- Idle or spiky: messaging, because it has no floor. 50 million SQS messages cost about $6 batched; an idle on-demand stream still costs about $36 a month.
- A stream at unknown volume: start on demand, and move to provisioned shards once the volume is steady.
- Topics hit a wall first. In Sydney, SNS publishes 1,500 messages a second per account by default and EventBridge takes 1,200 PutEvents calls a second. Both can be raised, but past about a thousand a second with several consumers, a stream costs a fraction of the fan-out.
- Streams have walls too. One partition key tops out at 1 MB/s or 1,000 records a second, and a third standard consumer starts starving the others.
The cost of every design at every volume, as a chart, is in Queue, bus or stream?
Questions that change the answer
How many independent readers does each event have? One is a queue; several is a topic with a queue each, or a stream.
Will anyone need to read last week again? A re-run after a bug is a replay; if it's rare, an archive will do.
Must events for one device, account or order be handled in order? That's FIFO, or a stream partitioned by that key.
How many events a second, sustained, at peak? Below a few hundred a second, a stream mostly pays its floor.
Doesn't cover
Change data capture out of a database (DMS, Debezium) is its own concept.
Real-time screens for many viewers (WebSockets, Durable Objects) are a different pipe: see A live auction on Durable Objects.
Check your call
Decide before you open each answer.
A SaaS sends a welcome email, creates a workspace and tells sales whenever a user signs up. About fifty sign-ups a day.
Messaging: a topic with one queue per consumer. The user is in Postgres, nothing needs replaying, and a stream would pay its floor for nothing.
Three teams read the same payment events, in order per account, at about 2,000 a second, and must re-run the last 30 days whenever a fraud rule changes.
A stream, partitioned by account ID, with 30 days' retention. Ordering, replay and several independent readers are the job, and 2,000 a second is past what one Sydney account can publish to SNS by default.
Analytics wants every order event in the lake. A replay happens maybe twice a year, after a bug.
Keep the topic and its queues, and add an archive subscriber that writes to S3. A rare replay is a script, not a reason to run a stream.