Durable AI workflows preserve a task’s execution state across a pause, crash or process restart. A checkpoint helps a system resume from a known state, but it does not prove that a message was delivered or a booking was committed. An external action may have succeeded just before a connection failed, making the result uncertain until independently checked.
This chapter follows a fictional study-letter desk processing six requests: two are verified complete, one has an unknown send outcome, one is approved but not sent, one awaits review and one lacks a current source. The exercise distinguishes saved state from real-world outcome and explains how a safe resumption should work.
It is an unnumbered engineering companion to How Super Intelligence Works, extending the hundred-chapter curriculum without claiming contemporary AI has reached artificial superintelligence.
Chapter contents and safe-recovery routes
1. Durable execution is not merely a long conversation · 2. A checkpoint is a saved state boundary · 3. Checkpointers and persistent memory are different · 4. The fictional Aster Study Letter Desk · 5. Print all six states before the crash · 6. Count verified outcomes with the correct denominator · 7. The L3 timeout creates uncertainty · 8. A saved checkpoint is not an external readback · 9. What state should be durable? · 10. Volatile memory does not survive every restart · 11. A thread ID selects state but is not authority · 12. Source freshness can invalidate old progress · 13. Human interrupts are genuine workflow gates · 14. Replay can begin earlier than the failed instruction · 15. Parallel branches need their own result identities · 16. Checkpoint frequency has costs · 17. Reconcile L3 with new external evidence · 18. Resume L4 only after checking approval · 19. L5 and L6 do not complete through restart alone · 20. Durable state has privacy obligations · 21. The checkpoint store can fail too · 22. Dry-run a complete process restart · 23. Changed-context transfer: eight requests · 24. The safest continuation checklist · 25. Frequently asked questions · 26. Official sources and connected learning
1. Durable execution is not merely a long conversation
A model’s conversation history can describe past work but is not automatically a trustworthy operational ledger. Durable execution stores selected workflow state so the application can continue according to defined semantics after interruption.
The state may include the task identity, relevant source version, finished substeps, pending approval, stable tool-operation IDs and unresolved outcomes. A natural-language sentence such as ‘everything completed’ cannot replace that evidence.
2. A checkpoint is a saved state boundary
A checkpoint records enough information for the runtime to resume a defined part of a workflow. What is saved and where resumption begins depends on the execution framework. Checkpointing does not automatically undo an external tool action or guarantee that a side-effecting operation will never be repeated.
The application needs separate rules for tool permission, replay and reconciliation. A checkpoint is valuable precisely because it makes partial progress explicit rather than asking a model to reconstruct it from a vague narrative.
3. Checkpointers and persistent memory are different
LangGraph’s official persistence documentation describes thread-scoped checkpointers used for run-state recovery and stores used for application-defined cross-thread data. A checkpoint can preserve a workflow’s progress; a separate store may hold information permitted for reuse across tasks.
The two data types have different access and retention requirements. A teacher’s draft-review state should not automatically become global long-term memory accessible in unrelated family conversations.
4. The fictional Aster Study Letter Desk
Aster is a paper-only invented school office. It is preparing six study letters, L1 through L6, under a hypothetical teacher-approval process. The assistant can draft and organise, but it cannot invent approval or a successful delivery receipt.
All IDs, recipients and states in this lesson are fictional. No real school, family account or messaging service was accessed. The purpose is to understand what a restart can prove.
5. Print all six states before the crash
L1 and L2 have exact approved versions and independent confirmation that the corresponding letters were delivered. L3 has an approved version and an attempted send whose response was lost. L4 has teacher approval but no send attempt.
L5 is a prepared draft still waiting for authorised review. L6 lacks the required current source needed to draft a supported message. A correct task ledger preserves these differences; it does not mark every record processed simply because it entered the workflow.
6. Count verified outcomes with the correct denominator
Only L1 and L2 are verified complete before recovery. Verified share is 2/6, approximately 33.3%. L3 is not confirmed failed or confirmed sent. L4, L5 and L6 have not reached the intended delivery state.
Counting drafts, tool attempts or approvals instead of delivered records would be an incorrect answer to the completion question. The acceptance boundary belongs to the human’s actual goal.
7. The L3 timeout creates uncertainty
A write request may reach the external service, commit its effect and lose its response. Alternatively, the service may never receive it or may still be working. A timeout alone cannot distinguish these outcomes.
If the workflow resumes at a node that issued the send, it may repeat the operation. A safe continuation should first query an authorised source of external state or use the service’s documented idempotency mechanism to reconcile the intended action.
8. A saved checkpoint is not an external readback
The model may have written ‘L3 sent’ into a conversation before crashing. That sentence does not establish that the authorised delivery system has a matching record.
The application should preserve L3 as outcome_unknown until it obtains relevant evidence. This is the same distinction used throughout the series between a proposed action, an attempted action and a verified result.
9. What state should be durable?
A useful record contains the workflow or thread identifier, task owner, approved source revision, status of each item, outstanding approval, operation identity and the next permitted step. It need not contain every intermediate token or private pupil detail.
The saved fields must be sufficient for correct recovery. Too little state forces guesses; excessive retention can create privacy and storage burdens. The design should use an explicit minimal contract for resuming the real task.
10. Volatile memory does not survive every restart
An in-memory checkpoint may work within a running process but disappear when it exits. LangGraph documentation specifically distinguishes in-memory storage from persistent checkpointers, which are needed for recovery after process restart.
A real application should test a genuine stop-and-restart sequence. A successful uninterrupted run cannot prove persistence or reveal what happens after a node is interrupted during an external side effect.
11. A thread ID selects state but is not authority
A workflow identifier helps the runtime select the intended state. It is not an authentication credential, proof of a teacher’s role or a citation to a source document. The application must separately check which actor may read or resume the particular thread.
Guessing a workflow ID should not grant access to another person’s confidential educational records. Identity and state lookup serve different functions.
12. Source freshness can invalidate old progress
Suppose the approved timetable changes during a multi-day pause. A prepared draft may now contain an obsolete session time. Resuming from the same checkpoint does not automatically prove that the earlier data remain current.
Record the relevant source version and check whether a later revision affects what is safe to send. If content changes materially, the application may need to revalidate the draft and obtain renewed staff approval.
13. Human interrupts are genuine workflow gates
A system may pause before a consequential action and wait for a responsible person to approve or edit the proposal. The state should preserve which exact draft and evidence are being reviewed.
LangGraph’s interrupt documentation describes pausing and resuming execution with human input. The runtime’s exact replay semantics matter. A model’s claim that a teacher would probably approve is not a substitute for an actual recorded decision.
14. Replay can begin earlier than the failed instruction
Some workflow engines rerun an interrupted node or defined region rather than continuing from the exact machine instruction where a process stopped. An external send placed before the pause point may therefore be executed again unless designed for safe replay.
Use appropriate idempotent operations, stable keys and external-state checks for side effects. Do not infer ‘resume’ means the system cannot repeat a previous tool call.
15. Parallel branches need their own result identities
A parent workflow may run two document lookups in parallel. One can finish while the other fails. Depending on the framework, the successful branch’s result may be retained and not recomputed during recovery; this guarantee must be checked in the actual runtime.
A coordinator should not count the same completed lookup twice or treat a child task’s completion as proof the whole parent task finished. Parent and child state have distinct meanings.
16. Checkpoint frequency has costs
Saving a snapshot after every tiny operation increases writes and retained data, while saving only at the end of a long workflow can increase recovery work. Choose granularity according to task duration, replay safety, expected faults and the cost of reconstructing progress.
Checkpointing frequently does not make an unsupported result correct. A snapshot may faithfully preserve a mistake. Validation and authorised source checking are still required.
17. Reconcile L3 with new external evidence
Now stipulate that the authorised delivery service readback shows one matching delivered L3, including the intended approved version. L3 can move from unknown to verified. Total verified letters become 3/6 = 50%.
No new send was necessary. The count changed because the external evidence changed, not because the model became more optimistic after restart.
18. Resume L4 only after checking approval
L4 had a valid approval and no prior send attempt. Under the fictional workflow, the system can verify that approval remains applicable, submit one correctly identified operation and check the target. Suppose the readback confirms a matching delivered letter.
The total then becomes four of six, or about 66.7% verified. L4 needs its own operation ID, distinct from L3. A retry identifier should represent one business intent, not every letter from the same office.
19. L5 and L6 do not complete through restart alone
L5 requires a genuine teacher review; L6 lacks an authoritative source. Neither may be sent merely because the workflow’s process recovered successfully.
The correct status report is four verified and two awaiting different prerequisites. An honest partial result is more useful than a fabricated claim that everything was delivered.
20. Durable state has privacy obligations
Checkpoints may contain source text, reviewer identities, protected operation IDs and user content. The checkpoint store is another repository that needs appropriate access, retention and deletion arrangements.
A recoverable workflow should not collect or retain unrelated pupil details just because it is technically able to store them. Operational history and reusable personal memory should be governed separately.
21. The checkpoint store can fail too
Persistent storage can be unavailable, corrupted or incorrectly configured. A recovery plan should include backup assumptions, validation of state and a way to reconcile external records if the trusted checkpoint cannot be read.
When both state and external outcome are unavailable, the system should preserve uncertainty. A long model-generated story about what probably happened is not a substitute for an observed result.
22. Dry-run a complete process restart
In a synthetic test, stop the workflow after L3’s ambiguous send and before L4’s authorised send. Restart the process and load its checkpoint. The desired behaviour is to reconcile L3, preserve L4’s exact approval and keep L5 and L6 pending.
A model should not automatically replay every write or lose track of the human decision. Testing the actual interruption path is stronger evidence than a normal demo without crashes.
23. Changed-context transfer: eight requests
A fictional archive has eight letters: three confirmed delivered, one uncertain after an attempted send, two awaiting approval, one approved but unsent and one missing its source. The verified share before recovery is 3/8 = 37.5%.
If an authorised readback later confirms the uncertain one, it rises to 4/8 = 50%. The others remain pending; that one observation does not create the missing approval or source document.
24. The safest continuation checklist
Recover the correct workflow identity, load a trusted checkpoint, check source and policy versions, reconcile ambiguous external actions, preserve pending human input and perform only currently permitted steps. Read back consequential outcomes before reporting completion.
If a required piece of evidence is unavailable, name it and retain the status needed for an authorised continuation. Durable execution is a means to truthful work, not permission to guess or duplicate.
25. Frequently asked questions
Does checkpointing guarantee that nothing runs twice? No; replay depends on the framework and operation design. Is a checkpoint proof that an email was sent? No. Does in-memory saving persist through restart? Not necessarily.
Can a process resume without teacher approval? It may resume safe preparatory work but cannot invent a required approval. Should every input be kept indefinitely? No; apply appropriate privacy and retention rules.
Applied recovery audit: state consistency, side effects, resumption and changed circumstances
A checkpoint has a version of the workflow code
Suppose a model-serving team upgrades its orchestration graph while several old tasks are paused. The saved state may refer to node names or fields that changed in the new code. A checkpoint successfully stored before the upgrade may not automatically be compatible with the changed workflow.
A responsible release procedure defines which persisted states can be resumed, whether a migration is required and how failures are detected. A test that restarts the old code is useful but does not by itself prove a newly deployed version can resume every earlier task.
Deterministic work and nondeterministic work replay differently
A pure arithmetic calculation using fixed input can often be repeated with the same expected numerical result. An AI model call may vary its output because of model version, generation settings or nondeterministic execution. An external tool call may change the world and cannot be treated as a pure calculation.
Workflow design should preserve outputs where later stages depend on the exact version a reviewer approved. Otherwise a resumed model may silently rewrite a draft while using an old human approval for different text.
A state snapshot has to be internally consistent
Imagine that L4’s checkpoint says teacher_approved=true but the approved_content_version field is empty. A restart should not treat that as sufficient proof to send any generated content. The fields together express one operational relationship and require a consistency check.
Likewise, a saved operation ID for L3 should correspond to the intended letter and target. A collection of individually well-formed fields can still describe an impossible or contradictory state. Contract validation must include those relationships.
An approval should not be replayed as a model guess
A human decision may enter the workflow through a supported interrupt or review application. On resume, the system can use the recorded approval if it still applies to the exact artifact. It should not rerun a language model to predict how the teacher would decide.
That separation protects the person responsible for educational material. A model that guesses approval confidently is not performing a legitimate human-review stage, even if the resulting wording is polished and appropriate.
The source may change while the process is paused
A checkpoint is an observation of a particular point in time. A school schedule, external API permission or approved rubric may change before the process resumes. The correct continuation depends on the current applicable rules, not only on the historical fact that the task once reached a particular stage.
Save the relevant source revision and check its freshness before a consequential action. If a previously prepared notice is now obsolete, route to source correction and possibly renewed review. A technically successful resume can still produce a substantively wrong message.
No automatic resurrection of revoked access
Suppose the teacher revokes the agent’s publishing access while L4 waits in the queue. A checkpoint containing an earlier permission observation must not grant the agent that privilege forever. The protected tool checks current authorisation when the write is requested.
A saved approval and active technical access can be separate requirements. Both may matter, and neither is automatically refreshed by replaying a previous model response. This is why durable execution and the Agent Identity companion belong together.
A thread identifier should not expose a person’s identity
Workflow IDs help the runtime locate saved state, but including a pupil’s full name, school and date of birth in a human-readable identifier can unnecessarily expose personal data in logs. A stable opaque ID is often a more suitable design.
The mapping between the ID and the authorised user must be protected and retained for an appropriate purpose. This privacy requirement persists even when all generated responses are carefully worded.
Source records and checkpoints have different owners
The official letter source may belong to a teacher, while the workflow checkpoint is stored by an application operator. The checkpoint can record which source version it used but should not replace the owner’s canonical document with a reconstructed approximation.
After a crash, reconcile both stores: is the checkpoint trusted, and is the source still applicable? A model’s generated summary cannot settle their disagreement without reading the authoritative records.
Partial completion is not necessarily failure
L1 and L2 remain correctly delivered despite the process stopping before L3’s result was known. It would be wrong to erase their completed status and restart every letter simply because the overall batch did not finish. The correct batch report can be partly complete and partly pending.
A workflow should preserve verified independent outcomes whenever that is consistent with the user’s task and its own transaction model. This can reduce unnecessary retries and keeps the final communication honest.
A readback result must match the approved target
Suppose the delivery service says a letter exists with a similar title, but it addresses the wrong class. That record is not proof L3 completed correctly. Verification checks the essential approved content, target identity and external status, not merely the presence of any plausible record.
A reconciliation mismatch may require correction through an authorised process. The model should not silently adapt the user’s intended target to fit what it happened to find.
The safe replay boundary belongs before external effects
When possible, separate drafting and reasoning nodes from side-effecting tasks such as sending, publishing or charging. Record the exact intended operation and its stable identity before it is attempted. After uncertainty, reconcile that operation before considering another attempt.
This design supports the runtime’s replay semantics without allowing every restart to trigger a new side effect. It does not guarantee perfect delivery, but it creates clear places where permissions and external state can be checked.
Recovery can need a manual exception path
If the tool does not support reliable status lookup and L3’s send response is lost, the application may be unable to establish whether the letter was delivered. A bounded escalation to the responsible staff member is better than an endless series of sends.
The staff member should receive the intended letter ID, approved version and known attempt state without unnecessary sensitive content. A human decision then follows the organisation’s real process, not an AI-generated prediction.
Multiple checkpoints can support forensic comparison
Suppose the system has a checkpoint from before L3’s send and one after L4’s approval. Comparing them may help identify which state changes occurred within the workflow. Their timestamps and integrity matter, but their existence alone does not prove an external delivery.
A useful audit distinguishes state saved by the workflow engine from independent observations emitted by the messaging service. Those sources may disagree after a partial failure; the conflict is a reason to investigate, not to pick whichever gives the highest completion count.
Checkpoint retention has a lifecycle
Keeping every snapshot forever can consume substantial storage and preserve sensitive old material. Deleting all state immediately can prevent legitimate recovery or audit. The appropriate period depends on the task, legal requirements, privacy and the organisation’s needs.
Define expiry and deletion rules for operational state, as well as backups and associated logs. An article about durability should not imply that indefinite retention is harmless. Recovery capability and data minimisation need to be balanced deliberately.
Metrics should track actual resumed outcomes
A useful dashboard can show active workflows, safely resumed runs, stuck human approvals, unknown external writes and verified completions. Counting the number of checkpoint files is not a measure of successful recovery, just as counting generated paragraphs is not a measure of teaching success.
Evaluate a representative set of interruptions and check whether tasks end in the correct state without duplicate effects. Include the cases that remain unresolved. A restart success percentage based solely on the process booting again is too narrow.
A changed-context recovery calculation
Imagine a new fictional desk with ten letters. Four are verified, two have uncertain external send outcomes, three are approved but unsent and one lacks a source. Initially only four of ten, or 40%, are verified complete.
Suppose readback confirms one of the two uncertain records and the other remains unavailable. The verified count becomes five, or 50%; one still remains outcome_unknown. None of the three approved-unsent letters becomes delivered through that readback. A student should preserve the distinct pending statuses.
The model may help describe the recovery, not invent it
An AI assistant can prepare a clear report listing L1 and L2 verified, L3 uncertain, L4 approved but unsent, L5 waiting for review and L6 waiting for an authoritative source. It can also propose which appropriate lookup should be performed next.
The report must stay grounded in the checkpoint and tool records. When a new external observation changes L3 to delivered, the assistant can update that specific field and recompute the fraction without rewriting unrelated source or approval states.
Independent reasoning after the crash
Give students two versions of the Aster state table, one before recovery and one after readback. Ask them to explain which rows changed, why those changes are justified and which rows remain unresolved. Then conceal the first answer and provide the ten-letter transfer packet.
This tests whether learners understand durable task state instead of memorising that ‘a checkpoint saves your work’. The enduring principle is to preserve evidence, permission and exact external outcomes across interruptions.
Final recovery checks before claiming completion
Why the independent readback is the final authority
Aster’s checkpoint knows whether a tool invocation was attempted, but the letter-delivery service owns the question of what was actually delivered. The workflow should use its approved lookup to compare the intended recipient, content version and actual state. If the service cannot answer, the result remains uncertain rather than becoming a success simply because the agent has restarted.
A restored process may still have stale permissions
The application can load an earlier checkpoint correctly while a teacher’s delegated role has expired. Restoring execution state is not restoring old permissions. The tool must check the live authorisation relevant to the next operation. The model should not silently reuse a previous approval or claim that a successful read proves it may publish.
Checkpoint size is a reliability cost
Large checkpoint snapshots use storage, network bandwidth and recovery time. Smaller ones may omit information needed for safe continuation. A useful implementation measures restore time and what fields are actually required, rather than treating more retained messages as inherently better. The cost should be evaluated against failures the workflow is expected to recover from.
A rehearsal should deliberately simulate missing evidence
A safe test can hide L3’s delivery readback and require the workflow to pause without attempting a duplicate message. This demonstrates a valuable property: the system does not manufacture certainty when a key external dependency is unavailable. Passing the test does not establish universal safety, but it does show the intended failure boundary works in that controlled case.
Publishing claims need status-specific language
A report can accurately say that L4 is approved, L5 is awaiting review and L6 is missing its source. It should not call all three failed, because their next steps differ. The status language affects what staff do next, and a good AI-assisted report makes that distinction easy to retrieve while retaining the exact supporting operational record.
One last transfer question for learners
Suppose Aster later verifies both L5 and L6 after their missing prerequisites are supplied and their external sends are independently checked. The completed count then becomes six of six. That future result cannot be reported in the earlier snapshot. Students should state which new observations would be needed before changing a four-of-six summary to complete.
Continue the four-part agent reliability route: Event-Driven Agents · Agent Sagas · Reliability & Error Budgets. Together these laboratories trace a saved task across failures, event delivery, multi-service compensation and measured complete-task reliability. See the completed 100-topic series hub and Agent Observability for the corresponding full-system checks.
26. Official sources and connected learning
Read LangGraph persistence and interrupts for a concrete implementation model. These sources do not validate the invented Aster outcomes.
Continue with Idempotency and Recovery, Contract-First Agents, Observability and How Super Intelligence Works. All letters and statuses are invented. Sources checked 22 October 2026.