# Moving data between services

> Events, messages and records travelling from the code that knows to the code that acts.

- Concepts by Nicolas Dao (Cloudless Labs, Sydney). Web page: https://cloudlesslabs.com/concepts/areas/moving-data/

Something happens in your system, and other code has to act on it. Between the two you put a pipe, and the pipe decides more than it looks: whether a message can be read twice, whether order is kept, whether a consumer added next month can read the past, and how the bill grows with volume.

Two decisions come first. Should the pipe remember a message after it's delivered? And since every pipe here delivers at least once, how does the consumer stay correct when a message arrives twice?

## The decisions, in order

1. [Messaging or streaming?](https://cloudlesslabs.com/concepts/messaging-or-streaming/index.md): Should the pipe remember a message after it has been delivered? Messaging (SQS, SNS, EventBridge) delivers each message and forgets it, so your database is the memory. A stream (Kinesis, Kafka) is a log: it keeps every record in order, and each consumer reads at its own position, replay included. A topic with an S3 archive sits between them: a log without a stream. Start with messaging; move to a stream when ordering, replay and several readers are the job itself.
2. [At least once, and idempotency](https://cloudlesslabs.com/concepts/delivery-guarantees/index.md): Every pipe on AWS can deliver the same message twice. How does the consumer stay correct when it does? SQS, SNS, EventBridge, Kinesis and Lambda's retries all deliver at least once: a message can arrive twice. Exactly-once delivery isn't on offer, except as deduplication in an SQS FIFO queue for five minutes, or as transactions inside Kafka. The fix is in the consumer: give every message a key, and make the second write change nothing (a unique constraint, an upsert, a version check).

## If you chose this, use this

- **The pipe forgets (messaging):** [SQS](https://cloudlesslabs.com/stack/aws-sqs/index.md) for work, [SNS](https://cloudlesslabs.com/stack/aws-sns/index.md) to copy, [EventBridge](https://cloudlesslabs.com/stack/aws-eventbridge/index.md) to route: [SQS, EventBridge or SNS?](https://cloudlesslabs.com/stack/guides/sqs-eventbridge-or-sns/index.md)
- **Keep everything, replay rarely:** An SNS topic with a [Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md) subscription archiving to S3: [An event backbone that keeps everything](https://cloudlesslabs.com/stack/guides/event-archive/index.md)
- **The pipe remembers (a stream):** [Kinesis](https://cloudlesslabs.com/stack/aws-kinesis/index.md), or [Kafka on Amazon MSK](https://cloudlesslabs.com/stack/aws-msk/index.md) when Kafka itself is required: [Queue, bus or stream?](https://cloudlesslabs.com/stack/guides/queue-or-stream/index.md)
- **Correct when a message arrives twice:** A unique key in [Postgres](https://cloudlesslabs.com/stack/neon/index.md), partial batch failures on [SQS](https://cloudlesslabs.com/stack/aws-sqs/index.md): [SQS retries or Durable Functions?](https://cloudlesslabs.com/stack/guides/retries/index.md)

## Related

- [Putting AI into software](https://cloudlesslabs.com/concepts/areas/ai/index.md): Which kind of intelligence each step of a feature needs, and what happens when it's unsure.
- [Paying for it](https://cloudlesslabs.com/concepts/areas/cost/index.md): Where the money goes before a single request arrives, and when that's worth it.
- [Security, access and residency](https://cloudlesslabs.com/concepts/areas/security/index.md): Where the data may go, who can reach it, and what runs where it can't do harm.
- [Every concept](https://cloudlesslabs.com/concepts/index.md)
- [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)