# At least once, and idempotency

> Every pipe on AWS can deliver the same message twice. How does the consumer stay correct when it does? SQS, SNS, EventBridge, Kinesis and Lambda's retries all deliver at least once: a message can arrive twice. Exactly-once delivery isn't on offer, except as deduplication in an SQS FIFO queue for five minutes, or as transactions inside Kafka. The fix is in the consumer: give every message a key, and make the second write change nothing (a unique constraint, an upsert, a version check).

- A concept by Nicolas Dao (Cloudless Labs, Sydney): the decision before the tool
- Area: [Moving data between services](https://cloudlesslabs.com/concepts/areas/moving-data/index.md)
- Last reviewed: 1 November 2026, by Nicolas Dao
- Web page: https://cloudlesslabs.com/concepts/delivery-guarantees/

## What you'll be able to decide

Which delivery guarantee each pipe gives, where duplicates come from, and how to make a consumer idempotent with a key and a constraint instead of hoping.

## In short

### The options

- **A. At least once, and an idempotent consumer:** Accept duplicates, and make the second copy a no-op: a unique key and a write that can't apply twice. Tools: [Amazon SQS](https://cloudlesslabs.com/stack/aws-sqs/index.md) (My default), [Amazon SNS](https://cloudlesslabs.com/stack/aws-sns/index.md) (Use when it fits), [Amazon EventBridge](https://cloudlesslabs.com/stack/aws-eventbridge/index.md) (My default), [Amazon Kinesis](https://cloudlesslabs.com/stack/aws-kinesis/index.md) (Use when it fits), [Neon (serverless Postgres)](https://cloudlesslabs.com/stack/neon/index.md) (My default).
- **B. Deduplication in the pipe:** An SQS FIFO queue drops a repeat with the same deduplication ID within five minutes, and keeps order per group, at limited throughput. Tools: [Amazon SQS](https://cloudlesslabs.com/stack/aws-sqs/index.md) (My default).
- **C. Exactly once inside Kafka:** Idempotent producers and transactions across topics: exactly-once processing, as long as the work stays inside Kafka. Tools: [Amazon MSK (managed Kafka)](https://cloudlesslabs.com/stack/aws-msk/index.md) (Use when it fits).

### The rule of thumb

Make every consumer idempotent, with the message's own ID or a business key as a unique key. From [Amazon SNS](https://cloudlesslabs.com/stack/aws-sns/index.md).

### How the answer changes with scale

- **Low volume, one database:** A unique constraint and an insert that does nothing on conflict is the whole solution.
- **Batches at volume:** Report partial batch failures, so one bad message doesn't retry the other nine; the key makes the retried ones harmless.
- **A side effect outside your database (an email, a payment):** Record the key before the call, or pass the provider its idempotency key.

## The problem, in plain words

A consumer takes a message, writes an order, and is about to tell the queue it's done when its Lambda times out. The queue never hears back, so after the visibility timeout it hands the same message to another consumer. The order is written again.

Nothing broke. That's the contract. Every pipe on this site delivers **at least once**: SQS, SNS, EventBridge, Kinesis and Lambda's own retries can all deliver the same message twice, and a retried one can arrive after a newer one. Exactly-once delivery between two independent systems isn't on offer, because the pipe can never know whether a consumer that went silent finished its work.

So the question isn't how to stop duplicates. It's **how to make the second copy change nothing**.

## The picture

The same message, delivered twice: once with a unique key, once without.

**Diagram:** The same message delivered twice: the first copy writes the row, the second hits the unique key and changes nothing; without the key, the customer is charged twice.

- **First delivery:** Queue → Consumer: insert payment pay_1234 → Row written.
- **The same message, again:** Queue → Consumer: insert payment pay_1234 → Unique key: already there, nothing changes. The visibility timeout ran out before the delete: SQS hands the message out a second time.
- **Without the key:** Queue → Consumer: charge the card → Customer charged twice. Checking first, then writing, still races: two copies can both see "not there yet".

## Where duplicates come from

- **A timeout before the acknowledgement.** The work is done, the delete or the acknowledgement isn't: SQS's visibility timeout, a Lambda that ran out of time, an HTTP target that answered too slowly.
- **Retries of a whole batch.** Lambda retries a failed SQS or Kinesis batch, good messages included, unless you report the failures one by one.
- **Producers retrying.** A network error after the write succeeded: the producer sends again. Kinesis's PutRecords returns success even when some records failed, and the retry of those can overlap.
- **Redrives and replays.** A message redriven from a dead-letter queue, or a day replayed from an archive, arrives again by design, and a redriven SQS message even gets a new message ID.

## The options

**At least once, and an idempotent consumer.** The default, and the one I use everywhere. Give each message a key that survives every retry (the order ID, the payment provider's event ID, a device ID and its reading's timestamp) and make the write conditional on it: a unique constraint and an insert that does nothing on conflict, an upsert, or "update only if this version is newer". Checking first and writing second doesn't count: two copies can both see "not there yet". The constraint does the deciding, inside the database, in one step.

**Deduplication in the pipe.** An SQS FIFO queue drops a message that repeats a deduplication ID within five minutes, and keeps order per message group. It helps with a producer that retries, but only inside that window, at FIFO's throughput, and not with the consumer's own retries. Keep the idempotent consumer anyway.

**Exactly once inside Kafka.** An idempotent producer and transactions across topics give exactly-once processing, as long as the input, the state and the output all stay inside Kafka. The moment the consumer writes to Postgres or calls an API, it's at least once again.

## A concrete scenario

An agtech platform's sensor readings came in through API Gateway and an SNS topic, with a queue per subscriber, and one subscriber archived every raw reading to S3 ([the project](https://cloudlesslabs.com/work/vps-to-aws/index.md)). At cut-over, the readings that arrived after the last backup were replayed from that archive. Writes were idempotent, so replaying a reading twice changed nothing, and not one reading was missing. The usual way to get there: a reading's natural key (its device and the time it was taken) under a unique constraint, so the database refuses the copy.

A side effect outside the database is the harder case. A sign-up event that sends a welcome email can't lean on a constraint: the email has gone before the duplicate arrives. Record "welcome sent" for that user in the same transaction that decides to send it, and pass the email provider an idempotency key, so a duplicate event finds the record and stops.

## The trade-offs

| | Idempotent consumer | FIFO deduplication | Kafka transactions |
|---|---|---|---|
| Covers | Every source of duplicates, forever | Producer retries within five minutes | Work that stays inside Kafka |
| Throughput | Unlimited | 300 calls a second per queue (3,000 messages batched), more in high-throughput mode | Kafka's |
| Works across systems | Yes | No | No |
| What you build | A key and a constraint per consumer | A deduplication ID per message | Producers and consumers configured for transactions |

## The rule of thumb

**Assume every message arrives twice, and make the second one boring.** Pick the key from the message's body, not from the pipe, put a unique constraint on it, and let the database refuse the copy. Everything else (FIFO, transactions, careful retries) reduces duplicates; only the consumer makes them harmless.

## When the answer changes with scale

- **Low volume:** a constraint and an insert that ignores conflicts. That's all.
- **Batches at volume:** turn on partial batch failures (ReportBatchItemFailures on SQS, the same on Kinesis), so one bad message doesn't send nine good ones round again.
- **Side effects:** emails, payments and third-party calls get their own key, recorded before the call or passed to the provider, which most payment and email APIs accept.
- **Long workflows:** a step that runs again after a failure must be safe to repeat too; [SQS retries or Durable Functions?](https://cloudlesslabs.com/stack/guides/retries/index.md) covers which steps get a key and which must run at most once.

## Questions that change the answer

- What is each message's natural key: an order ID, a device and timestamp, the provider's event ID? That's the unique constraint.
- Does the consumer call something outside your database (an email, a payment, an API)? Then the key is recorded before the call, or passed to the provider.
- Must events for one key apply in order? Idempotency stops duplicates, not reordering: add a version check, or a keyed stream.

Doesn't cover:

- Ordering: a duplicate and an out-of-order message are different problems. Order is in [Messaging or streaming?](https://cloudlesslabs.com/concepts/messaging-or-streaming/index.md).

## Check your call

- **A payment provider's webhook arrives twice for the same payment.** Store the provider's event ID under a unique constraint, and insert it before acting. The second delivery hits the constraint and returns 200 without doing anything.
- **A Lambda reads SQS in batches of ten, and message seven fails.** Report only message seven as a batch item failure. The other nine are deleted; if one of them comes back anyway, its key makes it a no-op.
- **A failed message is redriven from the dead-letter queue back to its source queue.** It comes back with a new message ID, so a key built from the SQS message ID won't catch it. Take the key from the body: the order ID, the provider's event ID.

## The technology on this site

- [SQS retries or Durable Functions?](https://cloudlesslabs.com/stack/guides/retries/index.md): the decision guide
- [SQS, EventBridge or SNS?](https://cloudlesslabs.com/stack/guides/sqs-eventbridge-or-sns/index.md): the decision guide
- [IoT ingestion into Postgres](https://cloudlesslabs.com/stack/guides/iot-ingestion/index.md): the decision guide
- [Durable functions or Step Functions?](https://cloudlesslabs.com/stack/guides/durable-or-step-functions/index.md): the decision guide
- [Event-driven plumbing](https://cloudlesslabs.com/stack/use-cases/events/index.md): the use case
- [Background jobs and retries](https://cloudlesslabs.com/stack/use-cases/background/index.md): the use case
- [Off a fragile VPS and onto AWS, with almost no downtime](https://cloudlesslabs.com/work/vps-to-aws/index.md): in our client work

## Related

- [Messaging or streaming?](https://cloudlesslabs.com/concepts/messaging-or-streaming/index.md): read next
- [Moving data between services](https://cloudlesslabs.com/concepts/areas/moving-data/index.md): the area, and the technology for each answer
- [Every concept](https://cloudlesslabs.com/concepts/index.md)
- [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)