What you'll be able to decide
Which delivery guarantee each pipe gives, where duplicates come from, and how to make a consumer idempotent with a key and a constraint instead of hoping.
SQS, SNS, EventBridge, Kinesis and Lambda's retries all deliver at least once: a message can arrive twice. Exactly-once delivery isn't on offer, except as deduplication in an SQS FIFO queue for five minutes, or as transactions inside Kafka. The fix is in the consumer: give every message a key, and make the second write change nothing (a unique constraint, an upsert, a version check).
In short
The options
A. At least once, and an idempotent consumer
Accept duplicates, and make the second copy a no-op: a unique key and a write that can't apply twice.
B. Deduplication in the pipe
An SQS FIFO queue drops a repeat with the same deduplication ID within five minutes, and keeps order per group, at limited throughput.
C. Exactly once inside Kafka
Idempotent producers and transactions across topics: exactly-once processing, as long as the work stays inside Kafka.
The rule of thumb
Make every consumer idempotent, with the message's own ID or a business key as a unique key.
From Amazon SNS
How the answer changes with scale
Low volume, one database
A unique constraint and an insert that does nothing on conflict is the whole solution.
Batches at volume
Report partial batch failures, so one bad message doesn't retry the other nine; the key makes the retried ones harmless.
A side effect outside your database (an email, a payment)
Record the key before the call, or pass the provider its idempotency key.
The problem, in plain words
A consumer takes a message, writes an order, and is about to tell the queue it's done when its Lambda times out. The queue never hears back, so after the visibility timeout it hands the same message to another consumer. The order is written again.
Nothing broke. That's the contract. Every pipe on this site delivers at least once: SQS, SNS, EventBridge, Kinesis and Lambda's own retries can all deliver the same message twice, and a retried one can arrive after a newer one. Exactly-once delivery between two independent systems isn't on offer, because the pipe can never know whether a consumer that went silent finished its work.
So the question isn't how to stop duplicates. It's how to make the second copy change nothing.
The picture
The same message, delivered twice: once with a unique key, once without.
First delivery
- Queue
- Consumer: insert payment pay_1234
- Row written
The same message, again
- Queue
- Consumer: insert payment pay_1234
- Unique key: already there, nothing changes
The visibility timeout ran out before the delete: SQS hands the message out a second time.
Without the key
- Queue
- Consumer: charge the card
- Customer charged twice
Checking first, then writing, still races: two copies can both see "not there yet".
Where duplicates come from
- A timeout before the acknowledgement. The work is done, the delete or the acknowledgement isn't: SQS's visibility timeout, a Lambda that ran out of time, an HTTP target that answered too slowly.
- Retries of a whole batch. Lambda retries a failed SQS or Kinesis batch, good messages included, unless you report the failures one by one.
- Producers retrying. A network error after the write succeeded: the producer sends again. Kinesis's PutRecords returns success even when some records failed, and the retry of those can overlap.
- Redrives and replays. A message redriven from a dead-letter queue, or a day replayed from an archive, arrives again by design, and a redriven SQS message even gets a new message ID.
The options
At least once, and an idempotent consumer. The default, and the one I use everywhere. Give each message a key that survives every retry (the order ID, the payment provider's event ID, a device ID and its reading's timestamp) and make the write conditional on it: a unique constraint and an insert that does nothing on conflict, an upsert, or "update only if this version is newer". Checking first and writing second doesn't count: two copies can both see "not there yet". The constraint does the deciding, inside the database, in one step.
Deduplication in the pipe. An SQS FIFO queue drops a message that repeats a deduplication ID within five minutes, and keeps order per message group. It helps with a producer that retries, but only inside that window, at FIFO's throughput, and not with the consumer's own retries. Keep the idempotent consumer anyway.
Exactly once inside Kafka. An idempotent producer and transactions across topics give exactly-once processing, as long as the input, the state and the output all stay inside Kafka. The moment the consumer writes to Postgres or calls an API, it's at least once again.
A concrete scenario
An agtech platform's sensor readings came in through API Gateway and an SNS topic, with a queue per subscriber, and one subscriber archived every raw reading to S3 (the project). At cut-over, the readings that arrived after the last backup were replayed from that archive. Writes were idempotent, so replaying a reading twice changed nothing, and not one reading was missing. The usual way to get there: a reading's natural key (its device and the time it was taken) under a unique constraint, so the database refuses the copy.
A side effect outside the database is the harder case. A sign-up event that sends a welcome email can't lean on a constraint: the email has gone before the duplicate arrives. Record "welcome sent" for that user in the same transaction that decides to send it, and pass the email provider an idempotency key, so a duplicate event finds the record and stops.
The trade-offs
| Idempotent consumer | FIFO deduplication | Kafka transactions | |
|---|---|---|---|
| Covers | Every source of duplicates, forever | Producer retries within five minutes | Work that stays inside Kafka |
| Throughput | Unlimited | 300 calls a second per queue (3,000 messages batched), more in high-throughput mode | Kafka's |
| Works across systems | Yes | No | No |
| What you build | A key and a constraint per consumer | A deduplication ID per message | Producers and consumers configured for transactions |
The rule of thumb
Assume every message arrives twice, and make the second one boring. Pick the key from the message's body, not from the pipe, put a unique constraint on it, and let the database refuse the copy. Everything else (FIFO, transactions, careful retries) reduces duplicates; only the consumer makes them harmless.
When the answer changes with scale
- Low volume: a constraint and an insert that ignores conflicts. That's all.
- Batches at volume: turn on partial batch failures (ReportBatchItemFailures on SQS, the same on Kinesis), so one bad message doesn't send nine good ones round again.
- Side effects: emails, payments and third-party calls get their own key, recorded before the call or passed to the provider, which most payment and email APIs accept.
- Long workflows: a step that runs again after a failure must be safe to repeat too; SQS retries or Durable Functions? covers which steps get a key and which must run at most once.
Questions that change the answer
What is each message's natural key: an order ID, a device and timestamp, the provider's event ID? That's the unique constraint.
Does the consumer call something outside your database (an email, a payment, an API)? Then the key is recorded before the call, or passed to the provider.
Must events for one key apply in order? Idempotency stops duplicates, not reordering: add a version check, or a keyed stream.
Doesn't cover
Ordering: a duplicate and an out-of-order message are different problems. Order is in Messaging or streaming?.
Check your call
Decide before you open each answer.
A payment provider's webhook arrives twice for the same payment.
Store the provider's event ID under a unique constraint, and insert it before acting. The second delivery hits the constraint and returns 200 without doing anything.
A Lambda reads SQS in batches of ten, and message seven fails.
Report only message seven as a batch item failure. The other nine are deleted; if one of them comes back anyway, its key makes it a no-op.
A failed message is redriven from the dead-letter queue back to its source queue.
It comes back with a new message ID, so a key built from the SQS message ID won't catch it. Take the key from the body: the order ID, the provider's event ID.