The most dangerous retry is the one made because nobody knows whether the first attempt succeeded.
A payment request times out. A booking screen freezes. A warehouse instruction gets no acknowledgement. A public-service form appears to fail after submission. The user does the natural thing: tries again.
If the first attempt never happened, retry is useful. If the first attempt happened but the acknowledgement was lost, retry can create a second effect.
Retry safety is the set of mechanisms that lets a system try again without turning uncertainty into duplication.
Failure to Hear Is Not Proof of Failure to Act
Consider a payment.
The payer sends a request. The bank executes it. The confirmation message is delayed or lost. From the payer’s perspective, the operation is uncertain.
This distinction is fundamental:
- request failed before effect;
- effect occurred but receipt failed;
- effect is still in progress;
- system genuinely does not know.
A retry policy that treats all four states as identical is unsafe.
Idempotency: Repeating Without Multiplying the Effect
An operation is idempotent when repeating the same request does not keep adding the same effect.
“Set this order status to cancelled” can often be made idempotent. Repeating the same state assignment still leaves one cancelled order.
“Charge this card $100” is not naturally idempotent. Two executions can create two charges.
Retry-safe design therefore either uses naturally idempotent operations or adds identity so the system can recognise repeated attempts as the same intended action.
The Idempotency Key
A common pattern is to assign one unique identifier to the intended operation.
The first request says, in effect: “perform operation 7F42.” If the sender retries, it sends the same identity. The receiver can then say: “I already performed 7F42; here is the previous result,” rather than executing the effect again.
The identifier moves retry safety from guesswork into state recognition.
Retry Identity Must Represent Intention, Not Attempt
If every retry receives a new identity, the receiver cannot distinguish “same intended payment again” from “a second new payment.”
The key should therefore identify the business intention or logical action, not merely the network attempt.
This distinction applies in many domains. One shipment release can have several transmission attempts. One application submission can have several upload attempts. One maintenance job can have several messages without becoming several jobs.
Receipts Close the Uncertainty Gap
Retry becomes easier when the sender can ask for status before repeating the effect.
“Did operation 7F42 complete?” is safer than “please do 7F42 again” when the first outcome is unknown.
This is why receipts, acknowledgement and status lookup are not cosmetic features. They reduce the ambiguity that creates unsafe retry.
The broader handoff mechanism is explained in How Cross-System Handoffs Work.
Retry Has a Time Dimension
Immediate repeated attempts can make an overloaded system worse.
If thousands of clients all retry instantly after a slowdown, the recovery traffic can exceed the original traffic. The system enters a retry storm.
Retry design therefore includes:
- delay before trying again;
- increasing delay after repeated failure;
- randomisation so many clients do not retry together;
- a maximum number of attempts;
- a condition that stops retry when the failure is permanent rather than temporary.
Not Every Error Is Retryable
A malformed request will usually remain malformed on the second attempt. A permission denial does not become valid merely through persistence. An unavailable server may recover later.
Error contracts should therefore distinguish:
- retryable: temporary condition may clear;
- repair-then-retry: input or state must change first;
- do-not-retry: repeated attempts cannot legitimately succeed;
- unknown outcome: query state before deciding.
This is why interface error meaning matters as much as the error signal itself.
Worked Example: Online Payment
A customer presses Pay. The page hangs.
A weak system invites the user to press Pay again with no way to determine whether the first instruction took effect.
A retry-safe system attaches one transaction identity, lets the receiver recognise duplicates and exposes the existing result when the same logical payment is submitted again.
The customer may experience several attempts. The financial system should experience one intended payment.
Worked Example: Warehouse Release
A transport system sends “release trailer 104” to a yard system. The acknowledgement times out.
If the instruction is sent again without identity, the yard may treat it as a second release workflow. If the operation carries a stable release identifier, the receiver can recognise that the action was already completed.
Retry safety preserves one operational intention across unreliable communication.
Worked Example: Education
Human learning offers an analogy with an important limit.
If a student fails one attempt, simply repeating the identical task instantly may reproduce the same failure. The useful retry changes the state first: feedback, representation, prerequisite repair or time.
The lesson is not that cognition is a transaction system. It is that a retry is meaningful only when the conditions that produced failure are understood.
Side Effects Make Retry Hard
A request may create several effects: charge money, send email, reserve inventory and update a ledger.
If the system fails halfway through, retry may need to know which effects already occurred.
This is why complex workflows benefit from explicit state machines and reconciliation rather than pretending every operation is one indivisible action.
See How State Reconciliation Works.
Retry Safety Checklist
- Can the same logical action be identified across attempts?
- Is the operation naturally idempotent?
- If not, can duplicate execution be suppressed?
- Can the sender query the previous result?
- Which errors are retryable?
- How long should the sender wait?
- How are retry storms prevented?
- What side effects may already have occurred?
- When should a human or repair path take over?
The CivDJ Rotation
- Forward: request → uncertain result → retry decision → deduplicated or repeated effect.
- Backward: start from “exactly one intended outcome” and ask what identity and receipts must exist upstream.
- Rotate: compare sender, receiver, customer, operator and auditor views of the retry.
The auditor’s question is especially revealing: can the system explain why several attempts produced one legitimate effect rather than several?
Retry safety is how a civilisation survives unreliable communication without turning uncertainty into duplicate reality.
Continue through How Interface Contracts Work, How State Reconciliation Works and the master How X Works hub. The next article examines the companion mechanism: recognising duplicated events before acting twice.