Agent disaster recovery means restoring a usable, authorised AI workflow after its storage, computing environment, network or essential dependency becomes unavailable or corrupt. A restart is not recovery by itself. The system must recover trustworthy task state, confirm which external actions really happened, preserve the right source and permission versions, and show that a fresh user request can finish safely. For a tool-using assistant, the central question is not merely whether a model answers again; it is whether the whole task can resume without omissions, duplicate side effects or unsupported claims.
Begin with two measurable promises: the recovery point objective (RPO), which limits the age of state that might be lost, and the recovery time objective (RTO), which limits acceptable service interruption. Then identify separately stored evidence: task requests, approvals, source documents, output artefacts, tool receipts, versioned configuration and audit records. This chapter follows a fictional reading archive through a complete restore, works through seventy-two tasks, and demonstrates why a green server dashboard can coexist with a failed recovery objective.
This is an applied specialist companion to How Super Intelligence Works, following agent incident response, data consistency and durable workflow design. All people, organisations, timings and numbers in the learning laboratories are invented, not reported service measurements.
Choose a learning route
- New to reliability? Read recovery objectives, backups and the fictional archive before calculating anything.
- Building an agent? Concentrate on the state inventory, external receipts, restore sequence and duplicate-action prevention.
- Reviewing an incident? Start with the timeline, independent verification and the difference between full restoration and safe degradation.
- Studying independently? Work through the numerical packet, then attempt the changed-condition exercise before opening its explanation.
Contents and checkpoints
- 1. What recovery must actually restore
- 2. RPO and RTO from first principles
- 3. The state-and-dependency inventory
- 4. The 72-task worked laboratory
- 5. Restore procedure and evidence gates
- 6. Changed-condition transfer test
- 7. Questions and useful next reading
1. Recovery is a user-visible capability, not a reboot
Imagine a school archive assistant that finds a student’s approved reading list, assembles a letter and, with permission, sends it to a verified recipient. It can use a model, a database, a search index, a document renderer and a sending service. If the model server returns after an outage but the database was rolled back and the sending service has already accepted yesterday’s letter, retrying the entire task could send another copy. If its access permissions changed while it was offline, following the old approval may be wrong even if the wording of the letter is perfect.
A restoration plan therefore needs an end-to-end success criterion. The current authoritative source is present; the task’s recorded inputs and outputs are consistent; external actions have trustworthy receipts; permissions are checked under current policy; the user can see what was completed, held or rejected. Each of these is independently testable. Calling the recovery successful merely because a health endpoint returns HTTP 200 would collapse six different questions into one pleasant-looking number.
Recovery is distinct from ordinary retry
A retry assumes a bounded operation may be attempted again without creating an unacceptable consequence. Recovery asks whether the system as a whole can reconstruct a reliable account after state was lost or damaged. Even if an action supports a stable idempotency key, a disaster might destroy the mapping from task to key, invalidate a source link, make old approvals expire or isolate the audit store. That is why idempotent tool calls are necessary companions to disaster recovery, not substitutes for it.
2. Define RPO and RTO before choosing technology
The recovery point objective is usually measured backwards from a damaging event to the latest recoverable state. An RPO of fifteen minutes means the chosen recovery design seeks to limit recoverable data loss to at most that time window. It is not a promise that precisely fifteen minutes of work will disappear. Some systems retain an independent transaction log and can recover more recent changes; other backups are corrupt, incomplete or inaccessible and fail to provide the promised recovery point. The objective is a requirement that must be checked against the actual restored state.
The recovery time objective expresses how long a specified service may remain unavailable before the agreed business limit is exceeded. A forty-five-minute RTO is not the time it takes a database command to finish. It includes every step necessary to restore the user-facing service defined by the agreement: infrastructure, dependencies, application configuration, data integrity checks, access controls and a successful end-to-end transaction. Document the start event and the acceptance criteria before calculating the elapsed interval.
Objectives must describe what the user gets
Suppose an operator proposes an RTO of ten minutes and an RPO of zero. Those can be legitimate aims, but they have costs and engineering consequences. Replication, continuous logs, independent regions, retained versions and rehearsed failover may be needed. A critical action ledger can require a stricter recovery point than a cache of search suggestions. It is wasteful to make every disposable derived index as durable as an authoritative financial record, yet dangerous to restore only a disposable index and declare all user work recovered.
Google Cloud’s disaster recovery planning guide explains RTO and RPO as business-derived objectives and notes that shorter values generally increase complexity and cost. The article’s design rule is broader than a particular cloud provider: determine acceptable user harm and recoverable state first, then choose the architecture. A number selected because it sounds impressive is not a requirement; it is a guess waiting to become an incident.
3. Inventory every piece of the agent’s state
A reliable recovery inventory starts with authoritative task state. What request did the person make? What evidence justified the system’s conclusions? Which version of the source did the reader actually approve? Which tool actions were proposed, approved, attempted and independently confirmed? Who was allowed to act at the time? These questions may require separate databases. A conversation transcript alone cannot reconstruct the external world; conversely, a successful external transaction may not contain the reasoning, source citations and permission record needed to justify it.
Build a state inventory with columns for the system of record, sensitivity, retention requirement, recovery target, independent verification method and owner. The word owner matters: if a sending provider owns final delivery status, restoring a local queue is not proof of delivery. If a school office owns an approval policy, a cached instruction is not the authority after the office revises it. Each boundary needs a named route back to the real source of truth.
The state map is more useful than a single backup button
| State or component | Why it matters | Recovery verification |
|---|---|---|
| Human request and consent | Defines authorised work | Original request ID and approval scope |
| Source and version | Supports factual accuracy | Current original and revision identifier |
| Task checkpoint | Shows which step is outstanding | Persisted step and deterministic state transition |
| External action receipt | Prevents duplicate effects | Provider-side lookup using original action key |
| Model, prompt and tool configuration | Explains reproducible behaviour | Approved version and compatibility checks |
| Search index or cache | Speeds retrieval; may be derived | Rebuild against authoritative records and test queries |
A backup is not automatically a consistent snapshot
A database backup taken at 14:05 may capture task records while a separate approval service has just committed a new decision at 14:04:59. A storage copy made without transaction-consistent boundaries can combine incompatible versions. The correct remedy depends on the system: a transactional snapshot, shared revision marker, ordered durable log, replayable event record or a carefully coordinated set of recovery points. The name of the storage product does not solve cross-service consistency by itself.
An AI-specific trap occurs when search embeddings are restored to an earlier version than the document repository. Search may still return plausible passages, but they can refer to superseded facts. Rather than blindly restoring every derived artefact, record the source revision or ingestion watermark associated with each index. After restore, either rebuild the index to the current source or explicitly treat uncertain retrieval as unavailable until validated. Answering “I cannot verify the current document” is safer than inventing freshness.
Keep recovery instructions outside the failed system
If the restoration guide, secret-management procedure and incident contact list are stored only inside the database that must be restored, a straightforward outage becomes a puzzle. A documented and suitably protected recovery path should remain accessible even when the application is offline. Avoid copying sensitive credentials into an unsecured document just for convenience. Preserve a protected method for obtaining current credentials, verifying backup integrity and identifying who has authority to declare a recovered system operational.
A tabletop rehearsal is a practical first check. Imagine the primary account is unavailable, the usual engineer is absent and a backup exists but has not been tested. Ask which person can authorise restoration, where the runbook lives, how a fresh environment is provisioned, what logs identify the last completed task, and who confirms user-visible success. Any answer that ends with “we will look that up when the time comes” indicates a recoverability dependency worth repairing now.
4. The fictional Harbour Reading Archive laboratory
Harbour Reading Archive, a made-up teaching service, prepares approved correspondence and uses an external dispatch provider for delivery. At 14:20 its principal task database becomes unavailable. The last reliable snapshot is marked 14:05. The original event ledger reports seventy-two authorised correspondence intentions: forty-two had confirmed completion by 14:05, eighteen more were processed between 14:05 and 14:20, and twelve were still waiting in the original queue. The dispatch provider is reachable independently, but some local receipts were lost during the event.
Before looking at the arithmetic, write the rule: never infer that an attempted send failed merely because the local receipt is missing. The recipient could have received the letter. Nor should the assistant claim success merely because it composed convincing text. Confirmation comes from the authorised dispatch provider or another independently accepted record. A restored local row is evidence of local state, not evidence of the external side effect.
Read the event timeline precisely
| Time | Recorded event | Interpretation |
|---|---|---|
| 14:05 | Last reliable application snapshot | Potential recoverable point |
| 14:20 | Task database stops serving reliable writes | Start of defined user-visible interruption |
| 14:47 | Restored database process starts | Technical step, not acceptance |
| 15:00 | Search and task workers respond | Partial technical availability |
| 15:11 | End-to-end test passes and service is opened | User-visible recovery point |
Calculate the objective outcomes
The demonstrated recovery-point age is 14:20 minus 14:05, or fifteen minutes. If Harbour’s approved RPO is fifteen minutes, the snapshot sits exactly on that boundary. It is not a guarantee that the eighteen intervening attempts were all lost, because the dispatch provider retains evidence; nor is meeting the boundary proof that the backup is internally consistent. Record “fifteen-minute snapshot age, subject to validation” until the restored data, ledgers and indexes are checked.
The verified recovery time is 15:11 minus 14:20, or fifty-one minutes. If the approved RTO is forty-five minutes, the service misses it by six minutes. A tempting but incorrect calculation would subtract 14:20 from the 15:00 worker-ready timestamp to declare success after forty minutes. That would ignore the eleven additional minutes required before a safe end-to-end operation passed. Always tie the clock to the service definition agreed with the user, not to the earliest flattering signal.
Reconcile the seventy-two intentions
After restoring the fourteen-o-five snapshot, forty-two verified successes remain in authoritative task state. Eighteen subsequent attempts are not automatically safe to rerun. Harbour queries the external provider using stable original intention keys and finds fifteen clearly accepted dispatches. Three are ambiguous because their original receipts cannot yet be obtained. The twelve requests that were pending at the event still have no dispatch attempt. At this stage the counts are forty-two plus fifteen equals fifty-seven verified completed, three uncertain and twelve pending; all seventy-two are accounted for without guessing an outcome.
A second evidence lookup later confirms that two of the three uncertain attempts were accepted and the last was never executed. Confirmed dispatches become fifty-nine. The remaining thirteen consist of the one unexecuted intention and the twelve previously pending. Harbour can safely schedule those under current approvals, permissions and sources, rather than blindly replaying all thirty records absent from the restored database. Notice that recovering information from an external source did not change the original snapshot age; it improved the completeness of the reconstruction.
Separate verified completion from safe handling
Imagine that ten of the thirteen remaining intentions still meet their approval and document-version checks and are eventually dispatched. Three have become ineligible because the original approvals expired and must be returned for fresh human review. At that moment sixty-nine of seventy-two are completed: 69 divided by 72 equals about 95.8 percent. Yet all seventy-two intentions can be safely accounted for if the three held cases have documented owners and no unauthorised deliveries. Operational completeness, business completion and safety are different denominators; an honest report makes those distinctions visible.
That distinction is especially important for generative systems. A model might beautifully summarise the restored work and say “all correspondence recovered” because every row is now present. That statement would be misleading while three items remain blocked by expired permission. The correct status is a structured ledger: fifty-nine previously confirmed, ten newly confirmed, three held for reauthorisation, zero unexplained intents. Clear statuses are not pessimistic; they preserve the reader’s ability to act.
What evidence may the operator safely trust?
Use a source hierarchy. The provider’s queryable transaction record is stronger evidence of a dispatch than a model-generated sentence. A signed or access-controlled approval ledger is stronger evidence of permission than a quoted historical chat instruction. An original document revision is stronger evidence of current policy than a cached embedding. That does not make any source infallible: provider records can be delayed, approvals can be revoked and snapshots can be compromised. Verification means identifying the authority appropriate to each claim and checking the exact identity, time and scope.
Write the claim and its support in matching units. “Dispatch ID D-18 was accepted at 14:16 by the contracted gateway” should have a gateway record keyed to D-18. “Task T-42 has permission to send this version” requires an approval linking the task, recipient, document revision and applicable time. Neither support can prove the other claim. This practice carries straight into ordinary academic research: a citation to a broad report does not automatically prove the precise figure in a sentence.
5. The restore procedure: stop, inspect, restore, reconcile, verify
Recovery begins by limiting additional effects. Disable or fence the writer process that could send duplicate messages, retain its last known intentions and make sure the failure has not compromised identities or credentials. Preserve diagnostic logs before they rotate away. Declare which components are trusted and which remain suspect. Restoring over a damaged environment without isolation might recreate the same corruption or accidentally replay its queue. A protected recovery environment provides a place to test consistency without touching real recipients.
Step one: select and verify a recovery point
Select a snapshot only after checking its identity, origin, integrity, retention window and compatibility with the target application. “Latest available” and “latest usable” are not synonyms. A corrupted newest copy should not be preferred merely because its timestamp is attractive. Record why the chosen point meets or misses the stated RPO, and preserve evidence that can independently explain the decision afterwards. Check that credentials and encryption keys required for restoration are legitimately accessible; an encrypted backup with a lost key is not a usable recovery plan.
Step two: restore authoritative records before derived indexes
Restore the transactional task data and relevant source revisions, then verify referential integrity and status counts. Rebuild or verify search indexes against their underlying documents rather than assuming that any restored vector database knows the latest truth. If an index rebuild will take hours, a documented degraded mode may answer only questions tied to directly verified original documents. The user should see the limitation clearly. Availability without correct evidence can be worse than a narrow but honest service.
Step three: reconcile actions, never replay on hope
For each externally acting task, look up the immutable intention key in the actual provider. Classify it as confirmed, failed without effect, pending, ambiguous or intentionally cancelled. “Failed without effect” requires provider evidence strong enough to justify a retry; it cannot be assigned because a network timeout occurred. Persist the resulting decision and its basis. Where external systems do not support lookup or idempotency, the safe recovery may require human investigation or a manual reconciliation procedure rather than automated repetition.
Step four: restore permissions under current rules
The outage may cross a permission or policy boundary. A reader consented to sending a message from yesterday’s approved document, but a new instruction now requires a changed recipient or a second reviewer. The agent must not treat its cached authorisation as timeless. Recheck identity, resource, action, scope, expiry and the currently effective policy. Save the new decision with a visible provenance trail. This is not a philosophical add-on; it determines whether the next real-world action is allowed.
Step five: run tests that exercise the complete service
A proper readiness test traverses a fresh request, approved source selection, model or deterministic transformation, guarded tool invocation, actual or safely mocked side effect, status persistence and independent readback. It should prove that a versioned record survives a worker restart and that a duplicate intention is not executed twice. Ensure that the test cannot accidentally contact a live recipient unless the tester has explicitly approved that controlled outcome. Test failures need a repair owner and retest, not merely a red row that everyone learns to ignore.
Google Cloud’s recovery-from-data-loss testing guidance specifically highlights data integrity, RTO and RPO and recommends testing the full application stack with restored data. That is the useful principle: a healthy storage engine does not validate a whole AI workflow. A task that depends on both data and action permissions must exercise both. The failure-recovery testing guidance additionally addresses rehearsing failovers and rollback, not merely reading backup success emails.
Step six: return to service with a truthful completion report
When the acceptance test passes, record the actual service restoration time and its criterion. State whether RTO and RPO objectives were met; document uncertainty in recovered side effects; identify held tasks and human owners. If the service is degraded, call it degraded. If a manual reconciliation is still in progress, show which outcomes are confirmed and which cannot be verified. This is evidence-of-recovery, not a ceremonial green tick. Only then should work queues resume under bounded rates and current authorisations.
Why restore drills reveal hidden failures
Teams often discover that backup jobs completed successfully but no one knows how to restore the related application configuration. Other surprises include expired credentials, an undocumented dependency, a missing software version, inaccessible encryption keys, insufficient storage capacity, an index that cannot be rebuilt from its source or a full restore time much longer than estimated. A rehearsal gives those dependencies a safe place to fail. Its result should improve the runbook, the next test and the objective estimate—not become a badge of confidence disconnected from evidence.
A test plan needs a clearly defined blast radius and stop condition. For instance, use a nonproduction copy, synthetic recipients, intentionally non-sensitive records and a controlled tool stub. Verify the separation before introducing failure. A chaos exercise that exposes real private correspondence or triggers uncontrolled real dispatches has defeated the purpose of the training. Keep separate records of a simulated provider response and a real provider acknowledgement so later readers cannot confuse the two.
Diagnose why a restore is slow
Break the total elapsed time into detection, decision, environment preparation, data recovery, reconciliation, functional testing and reopening. If data import takes nine minutes but identity approval takes twenty, optimising import speed alone will not meet the objective. If the database recovers rapidly but an external action log is inaccessible, the critical path may be reconciliation. Map dependencies rather than summing parallel stages blindly; the longest necessary dependent chain determines the earliest safe reopening.
Use the incident’s time series rather than a single average. Ask how long recovery takes when the backup is larger, when the usual expert is unavailable, or when the primary data centre and monitoring service fail together. The test that always uses the smallest dataset, freshest credential and most experienced engineer may establish a laboratory minimum, not a resilient operating promise. A meaningful measurement states workload size, fault assumption, acceptance method and measured variability.
A recovery decision table
| Observation | Tempting but unsafe response | Evidence-based decision |
|---|---|---|
| Worker restart succeeds | Declare restored | Run complete user path and verify state |
| Receipt missing after timeout | Send again immediately | Query external provider by stable intention |
| Snapshot is fifteen minutes old | Claim all work lost | Reconstruct newer actions from independent logs |
| Search returns plausible old passage | Answer confidently | Verify original document revision or hold |
| Approval existed before outage | Reuse without checking | Check current scope, expiry and policy |
| All task rows are visible | Call every task completed | Separate confirmed, queued, ambiguous and held |
6. Transfer: the rules change after the snapshot
Now apply the same method to a different fictional project, Crescent Science Club. Its last confirmed backup is 09:00, the system fails at 09:18 and the user’s recovery-point objective is twenty minutes. The restoration engine starts at 09:50, but the complete verified publishing path does not succeed until 10:02. The acceptable downtime was thirty minutes. During the outage a teacher withdraws permission to publish one report. Before reading the solution, decide whether the two objectives were achieved and whether a completed draft from the backup can be published automatically.
The snapshot age is eighteen minutes, so the stated twenty-minute RPO boundary is met if the snapshot itself is valid. From 09:18 to 10:02 is forty-four minutes: the thirty-minute RTO is missed by fourteen minutes. The 09:50 engine-ready timestamp does not establish a usable complete publishing service. Most importantly, the withdrawn permission makes the proposed publication ineligible regardless of how well the draft was reconstructed. Its task state should say “held — fresh authorisation needed,” not “ready to publish.”
A second transfer: the restore contains an outdated answer
Imagine that a library opening time was changed in the authoritative source between backup and outage, but the search index still returns the earlier closing time after restoration. An assistant cites the old index and confidently answers the user. Is this a hallucination, a disaster recovery problem, or both? The useful diagnosis is that the claimed current time has insufficient current-source support because index freshness was not validated. Treating it only as text generation error misses the infrastructure cause. Repair the source-to-index watermark and verify the official notice before inviting readers to rely on the answer.
Practice: five questions with explained answers
Question one: if a backup is twelve minutes old and RPO is ten minutes, may we declare the target met because no one has reported missing work? No. The demonstrated recoverable point exceeds the target unless stronger evidence establishes a more recent usable state. “No report yet” is not an integrity proof. Document the failure or the verified later log position, rather than changing the definition after the event.
Question two: if the database is available after nineteen minutes but the only action provider is unavailable until forty-seven minutes, which time belongs in RTO? It depends on the stated service definition. If user tasks require verified dispatch, reopening the database alone is insufficient; the completed path at forty-seven minutes is relevant. A genuine read-only mode may have a separate, clearly stated recovery objective. Do not silently substitute the easier feature for the promised one.
Question three: a provider received a request but the response timed out. Should the agent create a fresh key and retry? No. Inspect the original intention using the original stable key, then decide. A fresh key may make the second request appear to be a new authorised side effect. Question four: is a backed-up vector index a replacement for authoritative documents? No. Its generated representation and ingestion watermark must be reconciled with originals.
Question five: all recovered tasks have known owners and safe next steps, but three require human permission. Is the incident completely closed? Service restoration may be complete, but the human business process and corrective tasks remain open. Report the difference. Operational honesty means recording what is verified, what is temporarily held and who will finish it. The result is a clearer handoff, not a weaker achievement.
7. Frequently asked questions
Does a perfect RPO mean that no answer can be wrong? No. RPO addresses recoverable state age, not factual correctness, permissions, model quality or source relevance. A perfectly replicated wrong answer remains wrong. Pair recovery objectives with grounded evidence checks, controlled tool operations, and user-visible result verification.
Can an agent repair itself after a serious outage? Some bounded operations can be automated, including restoring a tested service image, rebuilding a derived index or rechecking a queue. Those capabilities do not grant unlimited authority. Higher-impact changes still require appropriate human authorisation, independent tests and records. An agent’s ability to propose repair steps is separate from the organisation’s permission to enact them.
Should all agent records be backed up forever? No. Apply legitimate retention, confidentiality and deletion rules to backups and logs. A recovery design must specify how authorised deletion, access restriction and expiry apply to restored data. Keeping every old copy indefinitely can increase exposure. Where policies differ across data types, consult the responsible governance owner rather than assuming one storage setting is universally lawful or appropriate.
What if two backups disagree? Validate their provenance, revision boundaries, transaction completeness and authorised record owners. Do not pick the copy with the answer you prefer. If uncertainty affects a consequential action, preserve both for investigation, stop the action and present a precise unresolved state until an authoritative reconciliation is possible.
Is disaster recovery only for extremely large AI systems? No. A modest document assistant that sends one real email or updates one parent record can create consequences when its local state is lost. The basic discipline—unique task identity, backups, current permissions, independent receipts and a tested restoration route—scales down. Small systems may choose simpler technology, not weaker truthfulness.
One final recovery rehearsal can be done on paper before any engineer touches a live server. Draw three columns labelled intended action, authoritative record and safe next step. For each unfinished task, state whether a provider has confirmed it, whether a current approval still covers it and whether its evidence matches the present source revision. In the fictional Harbour packet, an accepted dispatch belongs in the confirmed column even if its local database row disappeared; a task without a valid approval belongs in the held column even if its draft is beautifully written. Then change one condition: the original recipient has withdrawn consent during the outage. The correct recovery route now stops before dispatch and seeks a new authorised decision. Finally ask a second person to reconstruct the same statuses independently from the saved receipts. If the two people disagree, investigate the source-of-truth boundary and repair the runbook before announcing full restoration. This short exercise tests recoverability, permission discipline and honest reporting together.
Source notes and onward learning
Technical reference: Google Cloud’s Disaster Recovery Planning Guide defines the objectives and their trade-offs. Google Cloud’s data-loss recovery testing guidance supports exercising integrity and the whole stack. Google Cloud’s failure-recovery testing guidance discusses failover and rollback. For the governance relationship between measured risk, procedures and action, read the NIST AI Risk Management Framework core. These references supply general engineering principles; the Harbour and Crescent numbers are original teaching examples, not claims about those organisations.
Continue with controlled fault testing to learn how to challenge recovery assumptions, service-level objectives to define user impact, and data consistency to validate changing records. The wider How X Works library and Gold Standard of Education connect mechanism, evidence and independent use. The lesson to carry into life is simple: being able to start again is not the same as knowing what happened.