Moving data between services
Events, messages and records travelling from the code that knows to the code that acts.
Something happens in your system, and other code has to act on it. Between the two you put a pipe, and the pipe decides more than it looks: whether a message can be read twice, whether order is kept, whether a consumer added next month can read the past, and how the bill grows with volume.
Two decisions come first. Should the pipe remember a message after it's delivered? And since every pipe here delivers at least once, how does the consumer stay correct when a message arrives twice?
The decisions, in order
- 01 Messaging or streaming? Should the pipe remember a message after it has been delivered? Messaging (SQS, SNS, EventBridge) delivers each message and forgets it, so your database is the memory. A stream (Kinesis, Kafka) is a log: it keeps every record in order, and each consumer reads at its own position, replay included. A topic with an S3 archive sits between them: a log without a stream. Start with messaging; move to a stream when ordering, replay and several readers are the job itself.
- 02 At least once, and idempotency Every pipe on AWS can deliver the same message twice. How does the consumer stay correct when it does? SQS, SNS, EventBridge, Kinesis and Lambda's retries all deliver at least once: a message can arrive twice. Exactly-once delivery isn't on offer, except as deduplication in an SQS FIFO queue for five minutes, or as transactions inside Kafka. The fix is in the consumer: give every message a key, and make the second write change nothing (a unique constraint, an upsert, a version check).
If you chose this, use this
- The pipe forgets (messaging)
- SQS for work, SNS to copy, EventBridge to route: SQS, EventBridge or SNS?
- Keep everything, replay rarely
- An SNS topic with a Firehose subscription archiving to S3: An event backbone that keeps everything
- The pipe remembers (a stream)
- Kinesis, or Kafka on Amazon MSK when Kafka itself is required: Queue, bus or stream?
- Correct when a message arrives twice
- A unique key in Postgres, partial batch failures on SQS: SQS retries or Durable Functions?
Other areas
- Putting AI into software
Which kind of intelligence each step of a feature needs, and what happens when it's unsure.
- Paying for it
Where the money goes before a single request arrives, and when that's worth it.
- Security, access and residency
Where the data may go, who can reach it, and what runs where it can't do harm.
Want this stack in your AWS account?
Cloudless Labs is a Sydney consultancy. We migrate on-premises systems to AWS, build serverless apps on this stack, and coach teams to ship with Claude Code and Codex.
Talk to us