A duplicate is not simply two identical-looking messages. It is two representations of one intended event that must not be acted on twice.
Networks retry. People submit twice. integrations replay old events. scanners read the same item again. distributed systems may deliver the same message more than once because reliability is often easier to achieve by repeating than by guaranteeing a single perfect delivery.
The system therefore needs another mechanism: duplicate suppression.
Duplicate suppression recognises that several arrivals belong to one logical action and prevents repeated arrivals from creating repeated effects. It is the natural companion to How Retry Safety Works.
The First Problem Is Identity
To suppress a duplicate, the receiver must know what counts as “the same event.”
Two payments of the same amount are not necessarily duplicates. A customer may genuinely buy the same item twice. Two identical sensor readings may represent two different moments.
Duplicate detection therefore needs a stable event identity, transaction identity or other evidence strong enough to distinguish repeated delivery from repeated intention.
Content Matching Is Not Enough
It is tempting to say: if two messages have the same fields, treat the second as duplicate.
That can suppress legitimate repeated actions.
A commuter can legitimately make the same journey tomorrow. A retailer can legitimately order 100 units twice. A student can legitimately submit two different attempts with the same answer.
Identity must capture the event, not merely the content.
Seen-Before Memory
Once events have stable identity, the receiver can keep a record of identities already processed.
When event A123 arrives:
- Check whether A123 has been seen.
- If not, process it and record the outcome.
- If yes, do not repeat the effect.
- Return the earlier result or a duplicate acknowledgement.
This simple pattern is powerful because it turns unreliable delivery into stable business meaning.
How Long Must the System Remember?
Duplicate suppression creates a memory problem.
If the receiver remembers every event forever, storage grows without bound. If it forgets too quickly, a very late duplicate may be processed again.
Systems therefore choose a deduplication window based on how late duplicates can realistically arrive and how costly repeated effects would be.
Some domains require long-lived identity because the consequence is durable. Others can safely forget after a bounded operational window.
Exactly Once Is Often an End-to-End Property
A transport layer may deliver a message more than once while the business system still produces one effect.
This distinction matters. “Exactly one message” and “exactly one intended outcome” are different guarantees.
It is often more practical to allow repeated delivery and suppress repeated effect at the layer that understands business identity.
Duplicates Can Arrive Out of Order
Suppose event 2 arrives, then event 1 arrives late. Is event 1 a duplicate, a stale event or a missing predecessor?
Deduplication therefore interacts with sequencing and state machines. The receiver may need both identity and expected transition order.
A unique event is not necessarily a valid event for the current state.
Worked Example: Payment
A payment request is submitted, processed and acknowledged. The acknowledgement is lost. The sender retries with the same transaction identity.
The receiver finds the transaction identity in its processed set and returns the previous result rather than charging again.
The user experienced two transmissions. The financial system experienced one payment intention.
Worked Example: Warehouse Scanning
A carton passes a scanner and the barcode is read twice in rapid succession.
If each scan creates a new inventory movement, stock can appear to move twice. A deduplication rule can combine item identity, checkpoint identity and a short time window to recognise that the two reads likely represent one physical passage.
Notice the domain logic: the same barcode read hours later at another checkpoint may be a legitimate new event.
Worked Example: Public Forms
A citizen submits an online form, sees no confirmation and submits again.
If the application identity is stable, the second submission can reconnect to the first case rather than create two independent cases that later consume staff time to merge.
Good duplicate suppression saves both system capacity and human anxiety.
Worked Example: Learning Evidence
Education has a different but useful analogue. Ten copied examples of the same capability are not ten independent pieces of evidence.
If a learner answers ten near-identical questions, an assessment system should not treat them as ten fully independent demonstrations of transfer. The repeated evidence may share the same cue structure.
The broader lesson is that repeated observations need identity and dependency awareness before they are counted as new information.
False Duplicate vs Missed Duplicate
Deduplication has two error modes.
- False duplicate: suppress a legitimate second action because it resembles the first.
- Missed duplicate: treat the same intended event as new and repeat the effect.
The correct balance depends on consequence. Financial systems may strongly prioritise preventing duplicate charges while still preserving a clear path for genuine repeated purchases.
Replay Is Not Always a Duplicate
Sometimes old events are intentionally replayed to rebuild state, audit behaviour or feed a new consumer.
The system must distinguish replay for reconstruction from a new command to repeat the external effect.
This is another reason event identity, side-effect boundaries and idempotent consumers matter.
A Duplicate Suppression Checklist
- What identifies one logical event?
- Can two legitimate actions have identical content?
- Where should seen-before identity be stored?
- How long must it be remembered?
- What should a duplicate receive as a response?
- Can events arrive out of order?
- How are intentional replays distinguished?
- What happens if the deduplication record itself is lost?
The CivDJ Rotation
- Forward: event arrives → identity checked → first effect or suppressed duplicate → receipt returned.
- Backward: start from “one intended outcome” and reconstruct every delivery attempt that may have led there.
- Rotate: compare sender, receiver, auditor and end-user views of whether two arrivals were truly one event.
Duplicate suppression is memory at the interface: the system remembers that it has already honoured this intention, even if the message comes knocking again.
Continue through How Retry Safety Works, How State Reconciliation Works and the master How X Works hub. The next interface problem is time: how long should one system wait before silence itself becomes a state?