Redundancy becomes useful only when function can actually move.
A spare server, backup power feed, alternate rail route, second supplier or standby team can exist for years and still fail at the moment it is needed. The missing mechanism is often failover: how live function transfers from the normal path to an alternate path without losing state, identity or responsibility.
This article sits beneath How Redundancy Works and beside How Fault Domains Work. Redundancy provides alternatives. Fault domains limit blast radius. Failover performs the transfer.
A Standby Is Not Yet a Working Recovery Path
A standby system must be ready enough to assume the job.
- Does it have current state?
- Can it accept the same identities and permissions?
- Can upstream systems find it?
- Can downstream systems recognise it?
- Does it have enough capacity?
- Has it been tested recently?
If any answer is no, the spare may exist physically while remaining operationally unusable.
Failover Begins With a Trigger
The system needs a rule for when the primary is considered unavailable or unsafe enough to abandon.
Triggers can include missed heartbeats, fault declarations, unacceptable latency, failed health checks, power loss, loss of route authority or operator decision.
The trigger must balance speed and certainty. Fail over too early and transient noise creates unnecessary switching. Fail over too late and the receiver experiences extended outage.
This connects to How Fault Detection Works.
State Is the Hard Part
Stateless services can often fail over more easily because another instance can process the next request with little memory of the previous one.
Stateful systems are harder. The standby must know what transactions committed, what inventory moved, what train authority is current, what session belongs to whom or what work was already completed.
If the primary and standby disagree, failover can preserve availability while corrupting reality.
That is why State Synchronisation and State Reconciliation are part of failover architecture.
Active–Passive and Active–Active
In an active–passive design, the primary handles live work while a standby waits. Failover promotes the standby when the primary is lost.
In an active–active design, several paths handle live work simultaneously. Failure shifts more load onto the survivors.
Active–active can reduce switchover time, but it makes shared-state coordination and split-brain prevention more demanding.
Split Brain: When Both Sides Think They Are Primary
One of failover’s most dangerous states occurs when the primary is not truly dead but becomes unreachable from part of the system. The standby promotes itself, while the old primary continues accepting work elsewhere.
Now two authorities can produce conflicting state.
Robust designs use quorum, fencing, leases, authority tokens or equivalent mechanisms so only one side can own the critical write path at a time.
The principle is broader than software: failover must transfer authority, not merely duplicate capability.
Warm, Hot and Cold Standby
- Cold standby: resources exist but require substantial startup and state restoration.
- Warm standby: the alternate is partly running and reasonably current.
- Hot standby: the alternate is live, closely synchronised and ready for rapid takeover.
Faster recovery generally costs more because the alternate path must remain closer to live readiness.
Failback Is Another Transition
Once the original primary is repaired, should traffic immediately return?
Not necessarily. The repaired primary may have stale state. The alternate may now be the authoritative path. A second transition can create more risk than simply remaining on the standby until a controlled maintenance window.
Failback should therefore be treated as a new handoff with validation, synchronisation and ownership transfer.
Worked Example: Data Service
A primary database becomes unavailable. A replica is current enough to promote.
The failover sequence may include confirming the primary is fenced, promoting the replica, updating routing, verifying writes, monitoring lagging dependencies and recording the authority transition.
The important achievement is not that “another server came up.” It is that the estate now has one trusted writable state again.
Worked Example: Railway
A railway loses one operational path. The alternate may be another track, another power section, another control centre or a degraded operating procedure.
Failover requires more than route availability. Timetable, signalling authority, passenger flow, communications and maintenance access must all agree with the new path.
The canonical railway owner remains How MRT Works | It’s Mathematics.
A Careful Analogy: Logistics
A primary carrier cannot collect freight. An alternate carrier exists, but the booking, labels, customs data, pickup schedule and customer promise all need to move with the shipment.
The failover analogy highlights custody and state: switching provider without transferring the complete operational contract can create stranded freight rather than resilience.
A Failover Checklist
- Define the failure trigger.
- Prove the standby has adequate capacity.
- Synchronise the state required for takeover.
- Prevent two simultaneous primaries.
- Transfer identity, authority and routing.
- Validate end-to-end receiver function.
- Record any state lost during the transition.
- Define failback separately from failover.
- Exercise the whole path regularly.
The CivDJ Rotation
- Forward: primary fault → trigger → authority transfer → standby assumes function → receiver validates continuity.
- Backward: start from a clean failover and ask what state, routing and authority had to be ready before the fault.
- Rotate: compare operator, data owner, safety owner, customer and finance views of standby readiness.
Failover is not the existence of a spare. It is the controlled transfer of live function, state and authority to a path that can carry the receiver safely forward.
Continue through How Redundancy Works, How Fault Domains Work and the master How X Works hub. Next: common-cause failure — why two apparently independent paths can still disappear together.