How to handle duplicate messages from an at-least-once queue
Your queue promises at-least-once delivery, so the same message can arrive twice. The consumer sends an email, and users are getting two.
Deduplicate in the consumer by recording the message id in the same transaction as the effect. On redelivery the insert conflicts and the work is skipped. Checking a separate store before acting is not enough, because two attempts can both pass the check before either writes.
Duplicates are the contract, not a fault
At-least-once means exactly what it says. If a consumer does not acknowledge in time — because it crashed, or was merely slow — the message is redelivered. A consumer that breaks on redelivery is incorrect rather than unlucky, and the fix belongs in the consumer because only it knows what "the same" means.
Why the check must share the transaction
Read-then-write leaves a window. Two deliveries can both read "not processed", both proceed, and both write. Inserting the id with a unique constraint inside the same transaction as the effect closes the window: the second insert fails and the whole attempt rolls back.
Effects you cannot roll back
Sending an email or charging a card cannot participate in a database transaction. Record the intent transactionally first, then perform the external call, then mark it done. A crash between the two leaves a record you can reconcile, which is strictly better than an untracked side effect.
Ordering is a separate guarantee
At-least-once says nothing about order. A redelivered message commonly arrives after messages produced later, so a consumer that assumes sequence will corrupt state even with deduplication in place. If order matters, carry a version and reject anything older than what you already applied.
Deduplication strategies compared
Deduplication strategies compared
| Strategy | Safe under concurrency | Notes |
|---|
| Check a cache, then act | no | Both attempts can pass the check |
| Unique insert in the same transaction | yes | The standard approach |
| Natural idempotence (absolute PUT) | yes | Best when the operation allows it |
| Version check on the target row | yes | Also solves ordering |
| Relying on the broker | no | Deduplication windows are short and best-effort |
Key facts
- At-least-once delivery makes duplicates the expected case, so a consumer that breaks on redelivery is incorrect rather than unlucky.
- Deduplication must record the message id in the same transaction as the effect; a separate check-then-act leaves a window where both attempts pass.
- A redelivered message can arrive after messages produced later, so ordering must be enforced separately from deduplication.
- Effects that cannot join a transaction — email, payment — need the intent recorded first, so a crash leaves something to reconcile.
- Broker-side deduplication windows are short and best-effort, which is why the consumer remains responsible.
Frequently asked questions
Is exactly-once delivery available if I pay for it?
Not end to end. What is sold as exactly-once is at-least-once delivery plus deduplication at a boundary. That is worth having, but it is worth knowing where the deduplication lives, because everything outside it still sees duplicates.
How long should processed message ids be kept?
Longer than the broker maximum redelivery window, plus a margin. Anything shorter and a late redelivery is treated as new, which is the exact failure the table exists to prevent.
Can I use the payload hash instead of a message id?
Only if identical payloads genuinely mean the same event. Two legitimate events with the same content — the same alert firing twice — would be collapsed into one, which is a silent data loss.
Machine-readable copy of this page:
/guide/deduplicate-at-least-once-messages.md