Distributed algorithms explain how separate computers coordinate work when no participant has an instant, perfect view of the entire system. Messages may be delayed, machines can fail and two honest nodes can temporarily disagree about current events. The gold standard is a precise safety and progress claim under stated timing and fault assumptions—not merely successful communication in a demonstration.
This guide moves from message passing and logical clocks to leader election, global snapshots, consensus, quorum rules, distributed commits and safe recovery. Original examples show why replicated machines can still produce duplicate actions or unavailable services and how to reason about those outcomes.
Choose a practical reading route
Nodes, messages, time and ordering · Chapters 1–8 · Start here
Coordination, agreement and impossibility · Chapters 9–16 · Start here
Replication, consistency and conflicts · Chapters 17–24 · Start here
Partitions, monitoring and safe operation · Chapters 25–30 · Start here
Complete chapter index · Worked failure cases · Additional failure investigations · Connected Gold Standard guides · Gold Standard master
Complete chapter index
Nodes, messages, time and ordering
- 1. Why a network of computers needs a different kind of algorithm
- 2. Distributed algorithms and ordinary algorithm design
- 3. Define the nodes, links and computation model
- 4. Synchronous and asynchronous models imply different knowledge
- 5. Local state and global state are not the same
- 6. Events and the happens-before relation
- 7. Logical clocks track some ordering information
- 8. Broadcast and convergecast move information through networks
Coordination, agreement and impossibility
- 9. Leader election must define what a leader means
- 10. Distributed spanning trees depend on topology knowledge
- 11. Termination detection is more than an empty queue
- 12. Consistent snapshots reveal global properties carefully
- 13. Mutual exclusion coordinates access to a shared resource
- 14. Consensus asks processes to agree under faults
- 15. The FLP result has exact assumptions
- 16. Paxos separates agreement from the application
Replication, consistency and conflicts
- 17. Raft makes replicated logs easier to reason about
- 18. Quorum intersection protects some consistency conditions
- 19. Crash failures and Byzantine behaviour differ
- 20. Two-phase commit solves a different coordination problem
- 21. Idempotency helps when messages are retried
- 22. Linearizability is a strong visibility guarantee
- 23. Eventual consistency requires a convergence contract
- 24. CRDTs support certain coordination-free merges
Partitions, monitoring and safe operation
- 25. Partitions expose operational trade-offs
- 26. Reconfiguration changes the agreement population
- 27. Monitoring and tracing are not proofs of correctness
- 28. Model checking exposes hard-to-test interleavings
- 29. Operational resilience means recovery, not only agreement
- 30. Questions and a practical learning sequence
A claim of reliability always needs a model
| Claim | What to establish | Common missing condition |
|---|---|---|
| One leader | Exclusive authority under membership rules | Old leader still writes after expiry |
| Replicas agree | Consensus safety for defined values | Application data are invalid |
| Requests finish | Liveness under timing and fault model | Coordinator failure or indefinite delay |
| No duplicate effects | Request identity and side-effect handling | Retries across transaction boundaries |
| Recovery succeeds | Consistent data and service restoration | Shared external dependencies |
1. Why a network of computers needs a different kind of algorithm
Distributed algorithms coordinate computations performed by several processors or services without assuming a single perfectly informed controller. Messages take time, one participant may fail while others continue, and each node can see only part of the system. The challenge is not merely dividing work. It is establishing correct shared outcomes despite partial observations and imperfect communication.
Begin by stating a user-visible property: no two clients may receive the same exclusive resource, all functioning participants should eventually receive an answer, or replicas should agree on a committed value. Each property requires defined timing and failure assumptions. A locally correct algorithm does not automatically compose into a globally correct distributed service.
2. Distributed algorithms and ordinary algorithm design
A sequential algorithm can reason about one instruction order and readily accessible state. A distributed algorithm must specify how local actions, remote messages and failures interact to create a global result. It may not be possible for any participant to observe all events as they happen, and information learned remotely may already be stale.
The Algorithm Design root develops correctness and complexity for general procedures. This guide concentrates on message-based computation, agreement, consistency, failure detection and the relationship between local observations and global guarantees. The relevant performance resources include messages, rounds, latency and tolerance of failures, not just instructions executed.
3. Define the nodes, links and computation model
A distributed model names participants and the communication channels connecting them. It must say whether messages are reliable, can be duplicated or reordered, and whether an adversary or failure can interrupt delivery. Node identity, membership changes and permitted local state may influence what is achievable.
A ring of processors with reliable neighbour links differs from a cloud application using a central broker. A graph can describe connectivity, but the algorithm also needs precise operational semantics for sending and receiving. A claim proved for fixed membership and reliable links is not a guarantee for intermittent devices joining and leaving.
4. Synchronous and asynchronous models imply different knowledge
A synchronous model may place known bounds on message delay or computational steps. An asynchronous model provides no fixed upper bound, so a slow participant may be observationally indistinguishable from a failed one for some time. Intermediate partially synchronous assumptions are common in practical designs.
This difference shapes failure detection and consensus. The fact that messages usually arrive quickly in testing does not establish a known upper bound under every allowed execution. State which timing assumptions support each liveness guarantee. Do not silently import clock synchrony from an ordinary desktop programme into an Internet-scale service.
5. Local state and global state are not the same
Each node maintains local variables and observes its own messages. A global state conceptually combines relevant local states and messages in transit. An observer cannot necessarily collect all components simultaneously, and a naive collection may describe a state that never existed as one consistent cut of an execution.
For example, one service records that a transfer was sent while another has not yet recorded receipt. Counting both local records without in-flight messages can appear to lose money. A meaningful global-state method must account for communication and causal ordering rather than merely combine whatever records arrive first.
6. Events and the happens-before relation
Lamport’s happens-before relation captures causal ordering: local events occur in programme order; a send precedes its matching receive; and transitivity carries these constraints. Two events not ordered by happens-before are concurrent in this partial-order sense, even if external wall-clock timestamps look close.
A timestamp can label an ordering convention without proving physical causation. Recording that event A occurred at 10:01 on one clock and B at 10:00 on another does not by itself establish B happened first. Use explicit send-receive relationships and clock assumptions when reconstructing a distributed history.
7. Logical clocks track some ordering information
Lamport clocks assign increasing logical timestamps that respect happens-before: if event a precedes event b causally, the logical timestamp of a is smaller. The converse does not generally hold; a smaller timestamp does not prove causal precedence. A total tie-breaking order can be constructed, but that is not the same as real-world time.
Vector clocks can capture more causal information under fixed-participant assumptions, enabling comparisons that detect certain concurrent events. They have metadata and membership costs. Choose an ordering representation for the actual problem—debugging, conflict detection or causal delivery—rather than claiming one clock method tells every participant what is happening everywhere.
8. Broadcast and convergecast move information through networks
A broadcast disseminates a message from one initiator toward other reachable nodes; convergecast collects results back toward a root along a suitable structure. Trees can reduce duplicate paths in some algorithms, but broken links or dynamic membership affect reachability and completion.
A flooding broadcast may send duplicates through several routes and can create substantial communication overhead. Track message identifiers, termination assumptions and topology changes. The question is not just whether the message arrived once in a demonstration; it is whether the algorithm achieves the specified delivery and resource bounds under relevant network conditions.
9. Leader election must define what a leader means
Leader-election algorithms aim for an agreed or suitably coordinated choice of one participant under specified conditions. In a ring with unique identifiers and reliable links, algorithms can propagate candidate identifiers and select according to a rule. In a changing network, multiple temporary leaders may exist unless the protocol and failure assumptions prevent it.
Do not equate a displayed leader flag with exclusive control over real resources. A formerly elected node may remain active after a partition while another is selected elsewhere. Fencing tokens, quorum rules or other mechanisms may be required to protect downstream operations. Leadership is an algorithmic contract, not a decorative status badge.
10. Distributed spanning trees depend on topology knowledge
A spanning tree connects all reachable vertices of a connected undirected graph without cycles. Distributed algorithms can construct trees through local messages and update rules. Such trees can support broadcast and convergecast, but their correctness assumes enough information and communication to avoid permanent contradictory parent assignments.
If the network changes during construction, the chosen tree may become obsolete. A minimum-weight spanning tree adds a global weight objective and is not equivalent to shortest paths from one root. Read Graph Theory for the mathematical distinction.
11. Termination detection is more than an empty queue
A node with no local tasks may still have a message in transit that will create further work. Distributed termination requires a system-wide condition including appropriate knowledge that no active computation or relevant messages remain. Checking each local queue at separate moments can report a false sense of completion.
In a learning-data pipeline, a worker may finish its current batch before a delayed message arrives from a second worker. A coordinator should not declare all records processed just because the visible inbox is momentarily empty. Define outstanding obligations and acknowledgements so a completion declaration refers to a consistent process state.
12. Consistent snapshots reveal global properties carefully
A distributed snapshot records local states and messages in transit in a causally consistent manner under assumptions appropriate to the algorithm. The Chandy–Lamport approach uses marker messages on suitable channels to capture a consistent cut of an execution.
A snapshot can support monitoring, checkpointing and debugging, but its relation to physical time and business truth still needs interpretation. If one component records a debit and another has not yet seen the corresponding credit, in-flight messages can explain the apparent imbalance. The model’s channel reliability and ordering assumptions must be stated.
13. Mutual exclusion coordinates access to a shared resource
Distributed mutual exclusion prevents incompatible processes from entering a critical section at the same time. A central coordinator, token mechanism or message-based permission method may achieve this under different assumptions. Safety requires exclusivity; liveness and fairness ask whether eligible participants eventually obtain access.
A lock lease is only as reliable as its timing and expiry semantics, especially when workers pause or a partition occurs. Protecting a database write may require transactional constraints or fencing rather than trusting that every worker voluntarily stops the instant a lease expires. Model both the lock and the protected resource.
14. Consensus asks processes to agree under faults
A consensus problem usually includes agreement among deciding correct processes, validity conditions on chosen values and a termination requirement under a stated model. Different definitions vary, so the property set must be explicit. Consensus is not simply a majority survey, and replicas agreeing on an arbitrary value may violate the validity requirement.
A distributed database may use consensus to agree on a log position or leader term, but the surrounding service must still interpret the committed record correctly. Distinguish protocol agreement from data validation, client authorisation and end-to-end transactional consistency.
15. The FLP result has exact assumptions
The Fischer–Lynch–Paterson impossibility result shows that deterministic consensus cannot guarantee termination in a fully asynchronous message-passing system with even the possibility of one crash failure, under the theorem’s model. It does not say practical consensus protocols never make progress or that agreement is impossible in all fault models.
Practical systems often introduce timing assumptions, randomness, failure detectors or conditions that allow progress in ordinary periods. Explain which assumptions change. The theorem’s lesson is that distributed-system guarantees have boundaries; it is not a reason to avoid building reliable services.
16. Paxos separates agreement from the application
Paxos is a family of approaches for achieving consensus under specified network and failure conditions, usually involving proposers, acceptors and quorums. Its safety properties are grounded in how overlapping quorums and proposal ordering constrain conflicting choices.
Teaching simplifications should not be mistaken for production specifications. Different Paxos variants handle leaders, logs, reconfiguration and performance differently. An algorithm may preserve consensus safety during message delays yet stop making progress without a workable leader or timely communication. The operational contract needs the wider state machine.
17. Raft makes replicated logs easier to reason about
Raft structures replicated-state-machine consensus around leader election, log replication and safety constraints under its specified assumptions. A committed log prefix can be applied by replicas to deterministic state-machine commands. Its explanatory appeal is the organisation of the problem into understandable pieces.
A replicated log does not automatically ensure every downstream side effect occurs exactly once. Payments, notifications and external APIs need idempotency and transaction design beyond log agreement. Read the protocol’s terms and membership rules rather than assuming a majority of machines always means universal service availability.
18. Quorum intersection protects some consistency conditions
If each read or write quorum overlaps an appropriate set of other quorums, information can be shared between operations through at least one common member. For fixed N=5 participants, any two sets of three members intersect. But this arithmetic is only one component of a protocol.
The intersecting participant must follow rules that preserve the relevant information and handle versions, stale leaders or failures. Under membership changes or Byzantine behaviour, different quorum conditions may apply. A calculation about set sizes alone does not prove linearizable storage.
19. Crash failures and Byzantine behaviour differ
A crash-fault model allows a participant to stop responding, whereas a Byzantine model allows arbitrary or inconsistent behaviour under the stated adversarial assumptions. Agreement thresholds and protocols can differ substantially because a faulty Byzantine participant may send contradictory messages to different peers.
A service labelled fault tolerant should declare which failures it handles. Malicious compromise, data corruption and operator mistakes are not automatically covered by a design built only for crashes. Safety claims must match the threat model and actual authentication and network protections.
20. Two-phase commit solves a different coordination problem
Two-phase commit coordinates a transaction decision across participants through prepare and commit or abort stages under its assumptions. Traditional two-phase commit can block if a coordinator fails at an inconvenient point; it does not provide the same fault-tolerant agreement guarantee as a properly designed consensus protocol.
A multi-database transaction cannot be judged from the fact that all participants voted yes before a network partition. Recovery logs, coordinator failover and timeouts affect what happens next. The application must specify whether it prefers atomic outcome, bounded availability or compensating actions where global atomicity is impractical.
21. Idempotency helps when messages are retried
A request is idempotent in the relevant sense when applying the same operation multiple times has the intended same effect as one application. In network services, delivery may be repeated after an acknowledgement is lost, so a stable request identity and deduplication rules can prevent duplicated external actions.
An idempotency key must identify the intended logical operation, not accidentally merge distinct legitimate orders. Durable records, expiry and replay rules matter. ‘Exactly once’ at one processing stage does not automatically describe end-to-end external side effects, especially when the latter are not part of the same transaction boundary.
22. Linearizability is a strong visibility guarantee
Linearizability requires each completed concurrent operation to fit a legal sequential behaviour at an instant between invocation and response, while respecting nonoverlapping real-time order. It supports the illusion of atomic access to a shared object under the specified interface.
Other consistency models may allow more stale reads or delayed convergence. Choose based on consequences rather than assuming every system must maximise strictness or availability. A consistent replicated counter is not sufficient if the underlying business records are incorrectly merged.
23. Eventual consistency requires a convergence contract
Eventual consistency broadly describes systems in which replicas may temporarily diverge but converge when updates cease and communication conditions hold, though exact definitions vary. Readers may observe stale states during propagation, which can be acceptable for some activities and dangerous for others.
Specify conflict rules, session guarantees and user-visible delay where important. A public article view count may tolerate temporary inconsistency; an exclusive resource allocation may not. Saying ‘eventually consistent’ without defining what converges and under which conditions is not a sufficient requirement.
24. CRDTs support certain coordination-free merges
Conflict-free replicated data types define update and merge operations that converge under suitable mathematical and delivery assumptions. A grow-only set with union merge is a simple example. More elaborate counters, sets and registers use carefully designed metadata and rules.
Convergence of replica state does not guarantee that the application outcome is fair, secure or legally appropriate. A shopping cart that merges items successfully can still contain a product no longer available. Choose a CRDT only when its semantics match the business operation and deletion or authorisation requirements.
25. Partitions expose operational trade-offs
A network partition prevents some nodes from exchanging messages. A service must decide which operations may continue and what guarantees they retain. CAP-style discussions address specific consistency and availability definitions under partitions, and should not be simplified to an unqualified claim that every system permanently picks two virtues.
Explain what clients observe on each side of a partition. Can they read stale values, write provisionally, or receive a clear refusal? A documented safe rejection may be better than silently accepting conflicting assignments. User-facing behaviour matters as much as a diagram of server groups.
26. Reconfiguration changes the agreement population
Distributed services add and remove participants for growth, maintenance and repair. Changing membership can break fixed-quorum reasoning if old and new configurations do not coordinate correctly. Joint-consensus or other carefully designed transition mechanisms can protect safety through changes.
Do not declare a new set of replicas authoritative before resolving outstanding operations under the old configuration. State which participants vote at each step and how clients discover the new configuration. A protocol that is correct for fixed membership is not automatically correct while operators rotate machines.
27. Monitoring and tracing are not proofs of correctness
Logs, metrics and distributed traces expose observed operations and failures, but clocks, sampling and missing events can make reconstruction incomplete. A service reporting 99.9% successful requests may still experience rare duplicate charges or data loss in unobserved paths.
Choose operational indicators tied to user-visible guarantees. Audit identifiers, message retries, latency percentiles and consistency anomalies. A formal model supplies testable properties, while monitoring checks that the deployed system behaves acceptably within the real environment.
28. Model checking exposes hard-to-test interleavings
A state-machine specification can describe concurrent messages, faults and invariants, and a model checker can systematically explore bounded cases. A short counterexample—one delayed acknowledgement followed by a retry and leader change—may reveal a bug not found during ordinary happy-path tests.
The Formal Methods root develops such specification. A model’s proof or clean bounded run only covers the behaviours included. Validate network assumptions and test actual software as well as the abstraction.
29. Operational resilience means recovery, not only agreement
A distributed algorithm may maintain safety during a disruption but stop serving users until communication returns. Recovery time, backlog processing, data repair and customer communication determine whether the service remains useful.
Define service-level objectives, acceptable stale reads, restoration procedures and safe failure modes. Rehearse realistic failures, including simultaneous dependency outages rather than only one clean node shutdown. Replication is one tool in resilience; shared power, identity services and human operations may still be single points of failure.
30. Questions and a practical learning sequence
Does replication guarantee consistency? No. Does majority agreement prove a payment was correct? No. Does the FLP theorem prohibit every practical consensus service? No. Are network clocks perfectly ordered? Not without assumptions. Can messages be retried safely? Only with the right operation identity and protocol.
Start with three simulated processes passing numbered messages. Model duplicates and delays, compare local records and define when all participants may announce completion. Progress to logical clocks, mutual exclusion and consensus before tackling real storage guarantees. The standard is a precise protocol backed by honest failure assumptions.
Twenty original distributed computation investigations
1. Two nodes both believe they are leader
A network partition separates a former leader from other participants, who elect another leader. Both user interfaces may display Leader. If both can write to the same external resource without a fencing or quorum rule, conflicting actions can occur. A leader flag is not a proof of exclusive authority. Trace the exact authorisation of each write after the split.
2. A message arrives twice after a lost acknowledgement
A worker processes a request and sends an acknowledgement, which is lost. The client retries. Without a stable request identifier, the same charge or registration may be applied twice. Add a defined deduplication record and investigate what happens when its storage fails or expires. Message replay protection is part of end-to-end semantics.
3. A local empty queue is not global termination
Worker A reports no tasks, worker B reports no tasks, and the coordinator announces completion. But a message from B to A is still in transit, and it will create new work. Termination detection needs an account of outstanding messages and a consistent global state, not a snapshot of separate local inboxes.
4. Two Lamport timestamps do not prove causal order
Events X and Y occur independently on two machines. X has Lamport timestamp 4 and Y timestamp 7. The smaller value alone does not prove that X caused or preceded Y in the happens-before relation. Compare communication paths, not only the labels. A total tie-breaking order is a convention beyond the partial causal order.
5. A snapshot counts money that has not arrived
Node A subtracts ten units and sends them to B. The snapshot records A after subtraction and B before receipt. Without recording the in-flight transfer, the apparent total is short by ten. A consistent distributed snapshot includes relevant channel state under its protocol assumptions.
6. Majority sets intersect but stale versions still exist
For five replicas, two sets each containing three necessarily share a participant. However, that overlap alone does not establish that every read receives the most recent authorised value. The overlapping node must participate in appropriate version, term and commit rules. Set arithmetic is necessary in several protocols but not the entire correctness argument.
7. A fixed membership algorithm during a rolling upgrade
A five-node service begins removing two nodes and adding three replacements. A naïve implementation lets old and new majorities commit independently if the configurations do not overlap safely. Model the membership transition, outstanding requests and vote rules. A proof about five fixed members cannot simply be reused for an uncontrolled change.
8. A two-phase commit coordinator disappears
A coordinator tells three services to prepare, and they all record readiness. The coordinator then becomes unavailable before the final decision reaches them. The participants may remain uncertain or blocked under a traditional protocol. A timeout alone does not magically determine the correct atomic decision.
9. An eventual-consistency label hides an unacceptable delay
A learner’s progress record is updated on one server but another server shows an old result for several minutes. For a low-stakes progress display this may be tolerable if explained; for a certification decision it may be unacceptable. State read guarantees and correction pathways rather than advertising generic convergence.
10. A CRDT grows correctly but cannot remove private data
A grow-only set reliably merges updates across replicas, but a user requests deletion. Ordinary union cannot retract an item. A different data design with removal semantics, retention governance and recovery controls is needed. Mathematical convergence is not equivalent to compliance with privacy requirements.
11. An unbounded delay is mistaken for a crash
A node does not reply for ten seconds. In a fully asynchronous model, the observer cannot know whether it crashed or is simply delayed. A practical timeout is an operational suspicion under timing assumptions, not proof of failure. Safety and progress claims should reflect that distinction.
12. An alert network has one shared dependency
Three copies of an application run in different virtual machines, but all depend on one identity provider. When identity is unavailable, none can admit users. Replication did not remove a common operational failure. Map service dependencies as well as replica count.
13. A protocol is correct but the application command is wrong
Three replicas agree to apply an update that accidentally replaces a student’s record with another record using the same display name. Consensus faithfully reproduces the error. Agreement protects shared ordering, not identity verification or the educational validity of application data.
14. The cost of flooding a dense graph
A broadcast floods a message through every link without a visited-message rule. Cycles can create repeated delivery and severe overhead. Add message identifiers and state assumptions, then compare reach and message complexity. Reliable dissemination is not the same as infinite retransmission.
15. A lock expires while work continues
Worker A obtains a short lease and pauses for a long operation. The lease expires and B takes over, but A resumes and writes to the resource. An expiry timestamp in an application is insufficient if the protected resource accepts stale writes. Consider fencing and resource-level checks.
16. A recovery script repeats externally visible actions
A workflow replays a log after a crash. The replicated log state is consistent, but an external notification or payment is sent twice because the replay cannot distinguish already-completed side effects. Design transactional outbox, idempotency or reconciliation mechanisms appropriate to the service. End-to-end guarantees cross boundaries.
17. An impossibility result is misquoted as universal
A report says consensus is mathematically impossible in distributed systems. The classic impossibility theorem concerns deterministic termination in a specific asynchronous crash-fault model. Synchronous or partially synchronous assumptions and suitable protocols permit meaningful progress. State the theorem exactly before making engineering decisions.
18. A protocol is evaluated only on average latency
A benchmark reports a fast median request time but omits the very slow tail during leader changes. Real clients can experience a deadline breach even with excellent typical performance. Measure relevant percentiles, safety anomalies and recovery behaviour, not only averages.
19. A counterexample requires a changed test plan
A model checker finds a trace in which cancellation and delayed confirmation both change an order. Fixing one transition may expose a different liveness problem, such as an order remaining pending forever. Re-evaluate safety and progress properties after the repair, and add an integration test for the discovered sequence.
20. A five-line classroom simulation becomes an operational promise
A classroom exercise simulates message passing among three processes. A production claim about nationwide exam-result delivery would additionally require capacity, legal data handling, monitoring, availability and disaster recovery. Transfer the reasoning about communication and faults, not the toy model’s numerical success rate.
Connected eduKate subjects
Mathematics and protocol reasoning
Real operational dependencies
Research and authoritative teaching references
- MIT OpenCourseWare — Distributed Algorithms Lecture Notes
- MIT OCW — Readings: Consensus and Faults
- MIT Repository — Fischer, Lynch and Paterson Impossibility
- Raft consensus paper and resources
All users, services and numerical cases here are invented teaching examples. A real distributed platform needs security engineering, data governance, operational testing and clearly defined service-level limits in addition to mathematical correctness arguments.
Sixteen further distributed-system design and repair cases
The following fictional investigations distinguish local success from a correct end-to-end result. Each case identifies a missing assumption, a specific possible failure, and the evidence or design change required before promising the user a guarantee.
1. A school result is delivered by three replicas
A school’s result service stores final grades on three replicas. One server receives an update and tells the parent it is final while another still presents the previous value. The service must define which read semantics apply after a confirmed write. If final results require strong read-after-write guarantees, an eventually updated cache may be inadequate. A correct distributed protocol is determined by the contractual observation a legitimate reader should receive, not just how many copies exist.
2. A timeout does not revoke every previous action
Worker A obtains a lease to update an external storage object, but loses contact with the coordinator. A timeout causes B to receive the lease. Worker A can still resume with an old command. Protecting the resource may require rejecting stale fencing tokens, not simply trusting that the old worker has seen its lease expiry. The safety property must be enforced at the side effect that matters, even when participants do not share a timely view of leadership.
3. An event replay is missing its original identity
A queue stores events with human-readable descriptions but no stable unique request identifiers. After a crash, old events replay alongside new ones with identical descriptions. The application cannot reliably distinguish retries from separate legitimate actions. Define an identity and retention policy for idempotent processing. The policy must also handle later requests that happen to have the same user, amount or text but are distinct operations. Data model design is part of distributed correctness.
4. Three clients all think a lock is free
A distributed lock service accepts a read-then-write protocol. Three clients read the state Free before any has written Locked and all begin work. Local correctness of each read does not imply the read-modify-write sequence is atomic. A compare-and-set operation, transaction or other suitable coordination must protect exclusivity. Then examine progress and failure handling, because serialising one action may introduce waiting or deadlock when the client holding the lock disappears.
5. A logical clock does not measure network latency
Two services record events with logical clock labels 14 and 18. A dashboard subtracts them and reports four milliseconds of transit time. Lamport-clock differences count logical steps under a rule, not elapsed physical time. To estimate actual latency, use instrumentation with suitable time sources or trace durations, accounting for clock uncertainty when crossing machines. The event-order tool is valuable, but a numerical difference does not automatically have time units.
6. A quorum read uses an inconsistent value rule
A client reads from three of five replicas and receives values with different versions. The service arbitrarily chooses the most frequent payload rather than applying its defined commit or version-order rules. Although the read contacted a majority, the result may be stale. Quorum intersection helps under carefully designed protocols but does not remove the need for compatible write, version and conflict semantics. Verify the complete read procedure rather than treating a three-of-five count as sufficient.
7. A snapshot records two credits and only one debit
A money-transfer event crosses a boundary while independent services record their portions of an operation. If snapshot markers are handled inconsistently, an observer may include the receiving credit after delivery while also counting the source’s old balance before debit, making money appear from nowhere. Define the cut using causal order and record messages in transit under the snapshot algorithm. This is an accounting lesson about observation, not proof that a real financial system conserves value without audited application rules.
8. A replicated log hides a shared database dependency
A service keeps a consensus log on several machines but stores all large attachments in one storage region. During regional disruption the log can reach agreement about commands while the application cannot retrieve the referenced files. The algorithm’s safety remains valuable, yet service availability is limited by a shared dependency outside the consensus group. Map the whole user transaction including object storage, identity, network routing and recovery when stating reliability guarantees.
9. Network partitions and two incompatible promises
An application promises that a unique booking can always be confirmed within one second from any reachable server, even if those servers cannot communicate during a partition. It also promises that conflicting bookings never occur. Those promises may be incompatible under the specified fault and information model. A responsible product chooses precise behaviour, such as refusing confirmation without adequate coordination, instead of hiding the trade-off behind ‘highly available’ marketing. State which requests succeed and which fail visibly.
10. Byzantine claims without an adversary model
A system uses majority votes among replicas and describes itself as secure against arbitrary malicious nodes. Crash-fault agreement and Byzantine agreement have different requirements, including authentication and thresholds. Majority counts from a crash-only protocol do not by themselves prove resistance to contradictory or forged messages. Write the threat model, which members can be compromised, the permitted network behaviour and the safety objective before selecting or advertising fault tolerance.
11. A rolling upgrade creates a temporary minority
A five-replica service replaces nodes one at a time but accidentally makes an old three-member quorum unavailable before enough new replicas are ready. Safety can be preserved through refusal to commit, yet the service loses availability. A rollout plan should account for eligible voters, transfer state, safe membership change and recovery. An upgrade’s correctness includes its intermediate states, not merely the configuration shown before and after deployment.
12. A duplicate message reaches a third-party provider
An internal message handler is idempotent inside its database, but it sends an email or triggers an external API before persisting the deduplication record. A crash occurs between those actions. On retry, the internal state looks unfinished and the external action repeats. Transactional outbox patterns or suitably idempotent provider interfaces can help, but exact guarantees depend on the external boundary. Trace the side effect end-to-end instead of declaring victory at the queue consumer.
13. A global report cannot infer precise simultaneous truth
A school district aggregates attendance counters from twenty sites. Each site reports at a different local time, and transfers between sites are occurring. The sum may double-count or omit pupils during handoff. Define event time, membership, transfer identity and a consistent aggregation window. The accurate global question is not answered by simply requesting ‘the latest’ figures from each school. Distributed state requires thoughtful reconciliation and metadata about when each component’s observation applies.
14. A leader election ignores unequal identifiers
A ring election algorithm assumes globally unique process identifiers and selects the highest value. A provisioning error duplicates one identifier. The original uniqueness proof no longer applies, and two nodes may each interpret themselves as the winner. Validate identifier allocation or change the algorithm’s assumptions and tie handling. A mathematical guarantee is conditional on its system model, so input-integrity checks are part of deploying that guarantee responsibly.
15. A read replica repairs slowly after an outage
A follower has been offline and returns with old data. The service allows it to serve requests immediately, exposing stale records after the main system has already processed corrections. Define catch-up, verification and eligibility rules for rejoining replicas. An eventual convergence promise without a read gate may be inadequate when users need current records. Observe lag and confirm recovery, not merely that the machine has returned to a healthy network status.
16. A consensus exercise is not a nationwide examination platform
Three students simulate agreeing on a coloured card by passing paper messages. They can learn about message delays, majority overlap and local uncertainty. Deploying an official examination result service requires security, authorisation, accessibility, privacy law, change control, durability and disaster recovery beyond the classroom protocol. The correct transfer is the discipline of specifying agreement and faults while rebuilding the domain-specific service contract, not using a toy simulation as a reliability certificate.
Explore the Gold Standard master, Education and Learning Transfers into Life, and How X Works.