VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

How Superintelligence Works | Agent State Machines — Safety Invariants, Liveness and Model Checking

An AI agent’s state machine makes the allowed steps of a task explicit: drafted, awaiting approval, executing, verified, cancelled or unresolved. Safety means prohibiting a bad transition, such as sending private information without valid authorisation. Liveness means useful work eventually progresses under stated assumptions. The two are not interchangeable: refusing to do anything may preserve a narrow safety rule while frustrating every legitimate user, and rapid task execution may conceal unsafe actions.

The practical method is to define authoritative states, precise transition guards, external evidence requirements and recovery routes before launching the agent. Then explore possible orderings, look for counterexample traces, check the conditions needed for eventual progress and compare the abstract model with real tool behaviour. This article builds a fictional nine-state approval workflow, analyses thirty-six selected test traces, checks the multiplication behind a small state space and transfers the method to a document-publishing workflow. No production formal verification or present-day artificial superintelligence is claimed.

This article belongs to the How Super Intelligence Works curriculum. The workflow migrations guide covers changing in-progress systems, while agent test oracles describes acceptance checks. Here the reader learns to reason about which paths can occur and why a finite model’s guarantee must not be confused with evidence that the real world complied.

Choose a practical reading route

  • For students: build a state-card model and find the expired-approval counterexample.
  • For system builders: define guards at external tool boundaries, unknown outcomes and fairness assumptions.
  • For reviewers: solve the thirty-six-trace ledger and check the limits of a finite-state result.
  • For parents and teachers: distinguish a drafted suggestion from a real authorised and independently verified action.

Contents

1. What a state should represent

The model’s conversational text may say ‘almost finished’, ‘thinking’ or ‘ready’. Those descriptions do not establish that an action was authorised, sent or committed. A state machine needs values maintained by the responsible application or external provider. For a draft email, relevant state could include a content revision, intended recipient, current grant, operation identifier and independently observed delivery status. The assistant may explain these values but must not invent them to make the conversation smoother.

Choose names that distinguish user decisions from background operations. Proposed means the model suggested an action. Drafted means the content exists but no external effect has been authorised. AwaitingApproval means the exact action is waiting for human confirmation. Approved means a valid grant covers specific arguments and timing. Executing means the request may be in flight. Verified means the target resource or provider supplied satisfactory evidence. Cancelled, Expired and Unknown preserve three different reasons that work must not simply continue.

A single status named Done is not adequate for every phase. A file save can be accepted while an asynchronous publishing process is still pending. A message can be transmitted without being delivered. A draft may be complete while the user still needs to approve sending it. A robust interface reflects each material distinction. This helps a parent or student interpret the assistant correctly, even if the actual workflow contains complex internal services and long-running tasks.

State should represent facts and decision rights, not the model’s emotions or confidence. One can record that the source notice was revision 6, the approval token applied to recipient A and the provider reported a timeout. One should not convert a model’s enthusiastic sentence into an official Completed event. The important lesson for readers is that a trustworthy assistant reports the boundary between what it proposed, what it was permitted to do and what independent systems confirmed actually happened.

Return to contents or continue with Put guards on transitions, not only in prompts.

2. Put guards on transitions, not only in prompts

A transition specifies which next state may follow an event. For an email, Drafted may move to AwaitingApproval once the recipient and message are fixed for review. AwaitingApproval may move to Approved only after a valid user grant has been recorded. Approved may move to Executing when a trusted service checks current permission, exact action arguments and a stable operation identity. An agent may request the transition, but the service that can perform the effect needs to enforce the guard.

Guards should be precise enough to reject a dangerous near miss. A grant to draft a message must not also allow sending it. An approval for recipient A must not automatically apply if the agent changes the recipient to B. An approval linked to draft revision 5 may need renewed review if revision 6 materially changes the content. A user statement embedded in an external document is not an authority token. The controller must rely on authenticated and current sources of permission.

An attempted transition and a completed transition are separate events. The model might call a tool to publish an article, but the tool could reject the request, return only Accepted, or time out after writing successfully. A transition to Verified needs the relevant evidence from the destination. If that evidence is not yet available, preserve Unknown or another explicit in-flight state. Forcing every attempt into Success or Failure creates conditions for duplicate actions and misleading user messages.

Error handling belongs in the model too. A wrong draft can return for editing without becoming public. An expired approval can move to Expired and require renewed consent. A failed syntax check can hold a candidate before execution. A lost external receipt can trigger reconciliation under the original operation ID. A cancellation should stop new consequential work while preserving accurate history. These routes are easier to test when they are designed before an outage, not improvised by a language model after a failed request.

Return to contents or continue with Write safety properties as forbidden effects.

3. Write safety properties as forbidden effects

A safety property rules out an event that the system is never supposed to perform within the stated model. One example is that a new outbound send cannot start without a grant still valid for its precise recipient and content. Another is that a cancelled task cannot initiate a later external action under the cancelled authorisation. For publication, a truthful final report must not claim Verified unless an appropriate saved post and intended complete content have been read back.

Be careful with historical state. The slogan ‘sent implies approved’ is ambiguous, because approval may have expired after an earlier lawful send. A person can cancel future messages without making an already completed send disappear. The correct property checks permission at the send event and records which grant authorised that event. Historical facts and current permission state should be represented independently, or a simplistic boolean invariant may declare a legitimate history impossible.

Bad safety traces are often short. Imagine Proposed → Drafted → Awaiting → Expired → Executing. The final edge should be prohibited when no current grant exists. This is an actionable counterexample: a developer can identify the missing guard and test the corresponding tool boundary. A prompt asking the assistant to be responsible might reduce ordinary mistakes, but it cannot establish that the backend will always reject an unauthorised operation.

Safety is always scoped. A model checking this approval property does not also prove that the message contains accurate school policy, the recipient can read it accessibly or the connected API protects other accounts. Those claims require additional specifications and evidence. This separation is an advantage rather than a limitation: it gives engineers a precise target they can implement, test and explain to users without overstating what one proof or one test establishes.

Return to contents or continue with Liveness and fairness need explicit assumptions.

4. Liveness and fairness need explicit assumptions

Liveness asks whether a useful event eventually occurs. A task may ultimately reach Verified, Cancelled or a declared safe terminal outcome, depending on the contract. An agent cannot guarantee that an absent parent will approve a message, nor that an unavailable external provider will eventually recover. A liveness statement must include the environment assumptions under which progress is expected and should not redefine coerced approval as successful automation.

Scheduling fairness is relevant when an action is continuously ready to run. If a permitted task can be ignored forever because new work keeps arriving, a statement that every authorised task eventually completes is false without a fairness or bounded-service assumption. A deadlock-free design may still starve a low-priority job. A system that keeps retrying and yielding can livelock without completing anything. Monitoring must therefore count durable user outcomes as well as transitions and tool attempts.

Safety and liveness can pull a system in different directions. A controller that blocks every external call satisfies a narrow no-unauthorised-send requirement but fails to help people who did approve a valid message. A controller that sends immediately may have excellent throughput and an unacceptable permission failure. A good design checks both that prohibited effects cannot happen and that legitimate tasks can make progress when the stated conditions are met.

Unknown results complicate liveness. If a provider returns no usable acknowledgement, a task may remain unresolved until a trusted query or human reconciliation establishes the effect. Artificially declaring Success makes the dashboard look faster but violates the evidence contract. Retrying without checking creates duplicate risk. Appropriate progress may be an explicit handoff to a responsible reviewer, not an invented completion.

Return to contents or continue with Nine abstract states for a draft-and-send agent.

5. Nine abstract states for a draft-and-send agent

Build a teaching model with nine names: Proposed, Drafted, Awaiting, Approved, Executing, Verified, Cancelled, Expired and Unknown. A happy path is Proposed → Drafted → Awaiting → Approved → Executing → Verified. A permitted refusal path is Proposed → Drafted → Awaiting → Cancelled. An approval offer that expires without a valid response can end in Expired. These are abstract representations of task stages; they do not claim to be an executable TLA+ specification.

The state labels alone are not enough to enforce correct behaviour. A real model also needs values such as current grant version, recipient, content revision, cancellation revision and possibly the result of an external call. An Approved flag without the arguments it approves is dangerously broad. The abstraction should keep whatever information affects the safety property and deliberately omit irrelevant details such as font colour or the model’s preferred rhetorical style.

List the legal next steps. Awaiting may move to Approved with a valid exact approval, or to Cancelled when the user declines, or to Expired when the authorisation window closes. Approved may move to Executing only after current guards pass. Executing may lead to Verified after readback or to Unknown when outcome evidence is insufficient. Unknown may be reconciled to a confirmed result, but it must not silently return to a state that assumes no external action occurred.

One benefit of the state-card exercise is that a parent can understand the process without studying code. The person can ask whether an AI-created draft is still private, whether the approval window remains open and which evidence proves the requested action was completed. The engineer gains something too: a list of transitions for which automated and manual tests can be written. Shared vocabulary reduces confusion when the system behaves unexpectedly.

Return to contents or continue with Time changes the validity of a seemingly good approval.

6. Time changes the validity of a seemingly good approval

An approval can be valid when recorded and invalid by the time a queued action executes. The service should check the grant’s effective interval and bound content at the actual execution boundary. A message approved for one appointment date should not be sent as though current after that appointment has been cancelled. The relevant date may come from the grant, the event notice and the current authoritative record, each with its own version.

Different times describe different facts. A source may have been issued on Monday, retrieved on Tuesday and effective on Thursday. A user may approve a draft at noon and revoke approval an hour later. A tool operation may be accepted before the revocation but acknowledged afterwards. The system needs the appropriate event and authority semantics to decide what happened. A convenient local timestamp is not enough when timezone and ordering are ambiguous.

Create negative tests for expiry, recipient changes and cancellation races. Hold an approved job in the queue until its grant expires, then attempt execution. The expected result is a refused new action and an accurate status. Change the recipient after approval and repeat the test. Finally, cancel while a tool callback is pending and verify that the controller reports the real observed result rather than declaring that a completed effect was magically undone.

The temporal semantics chapter develops event time, effective dates and expired authority more deeply. In a state machine, the crucial transfer is to turn those facts into guards and recovery rules. The user should not need to understand a temporal logic formula to receive an honest explanation that an approval has expired and a new decision is required.

Return to contents or continue with Why unknown tool results need their own state.

7. Why unknown tool results need their own state

An external tool call may time out after the provider has already performed an action. The agent cannot safely infer that nothing happened just because its request failed locally. For an email, the provider might have accepted the message; for a WordPress post, the item might already be public; for a calendar booking, an actual reservation could exist. The controller should preserve the original operation identity and ask the authoritative system what occurred.

Define Unknown as a state of evidence, not as proof of failure. It means the controller cannot yet confirm whether the intended external effect happened under the declared check. If an authoritative read returns the exact saved state, the task can move to Verified. If the provider confirms no effect and current permission still allows it, a bounded retry of the same logical operation may be appropriate. If neither can be established, human reconciliation may be necessary.

This is also why an agent’s natural-language report should not act as the final oracle. A fluent ‘your article is published’ message has no evidential power if the stored post remains Draft. A separate model agreeing with that message still lacks proof of the target’s actual state. The appropriate check is an authenticated readback by stable ID, slug or provider operation key, with comparison to the intended content and status.

Unknown should not be a permanent hiding place for forgotten jobs. Record an owner, next reconciliation method, last trusted observation and escalation condition. A user may need to know that an important action is unresolved before deciding what to do next. The workflow should make that status accessible and prevent duplicate clicks from silently producing a second effect.

Return to contents or continue with Safety and liveness in finite model checking.

8. Safety and liveness in finite model checking

TLA+ is a formal language designed for reasoning about concurrent and distributed systems. TLC can explore reachable states in a suitably finite specification and report counterexample traces when checked invariants fail. A safety property is often violated by some finite trace that reaches a forbidden state. A liveness property concerns progress over a possible infinite behaviour and may require fairness conditions. These claims describe the tools and concepts, not an actual checker run performed for this article.

Consider an approval invariant. A checker is configured to explore the model’s initial states and possible transitions. If the model allows Expired → Executing without a fresh grant, a counterexample can identify the path. The developer can refine the guard and re-run the appropriate tool. The value is not in the elegance of the formula alone; it is in discovering an unexpected legal path through the design before live users encounter it.

A passing model result must be tied to its exact specification, constants, checked property and assumed environment. Real code might have an alternate send endpoint that the model omitted. The provider might accept work asynchronously while the model treats its response as terminal. A future migration might reinterpret an approval field. Formal checking cannot compensate for a model that excludes the important failure mode.

Use finite exploration as an aid to engineering judgement. It can systematically test interactions too numerous for a handful of illustrative screenshots, while integration and fault-injection tests demonstrate whether deployed code follows the intended transitions. Qualified review and user feedback remain relevant when the decision involves institutional policy, accessibility or human consequences beyond a small abstract machine.

Return to contents or continue with State-space counts illustrate the growth problem.

9. State-space counts illustrate the growth problem

If one task can be in nine abstract states, two independent tasks have at most 9 × 9 = 81 combinations of those labels. Three tasks have 9³ = 729 combinations. These are raw Cartesian product counts, not necessarily reachable states. Guard conditions, shared locks and prohibited transitions eliminate many possibilities, while additional state variables can multiply the theoretical possibilities dramatically.

Add an illustrative choice among four source versions for each task. The simple abstract combination becomes 9 × 4 = 36 possibilities per task. Three such tasks have 36³ = 46,656 raw combinations before removing impossible states. That number helps explain why human intuition misses uncommon interleavings. It does not claim a checker visited exactly 46,656 states, nor does it estimate the cost of verifying an actual enterprise workflow.

Abstraction is essential. When checking permission safety, a model may represent content by a revision identifier rather than enumerate every possible sentence. It may use a small finite set of users and tools to expose the ordering defect. The simplification must retain the relationships relevant to the claimed invariant, including exact approval scope and the possibility of delayed tool acknowledgements. If it removes the defect, a clean result can mislead.

Readers can calculate the small products themselves and ask why reachable state counts differ from raw combinations. Some proposed states may have no legal incoming transition. Some cannot coexist because the same resource cannot be exclusively owned by two writers. Some are ruled out by the current authorisation record. A good specification makes these reasons inspectable rather than treating a giant count as a badge of mathematical sophistication.

Return to contents or continue with A fictional thirty-six-trace evaluation.

10. A fictional thirty-six-trace evaluation

Imagine an isolated Bayview Study Desk simulator with thirty-six deliberately selected execution traces. Twenty-five end in permitted, independently verified results. Six end in correct cancellation or expiry without new unauthorised effects. Three remain Unknown pending trustworthy external reconciliation. Two reveal a model safety violation because an execution starts after the relevant approval was no longer valid. These are designed tests with exclusive outcome categories: 25 + 6 + 3 + 2 = 36.

The fraction reaching permitted verified results is 25 ÷ 36 ≈ 69.44%. Correct safe stops are 6 ÷ 36 ≈ 16.67%. Unresolved outcomes are 3 ÷ 36 ≈ 8.33%, while the two violating traces are 2 ÷ 36 ≈ 5.56%. The arithmetic is transparent but these are not measured production rates. The packet was chosen to expose difficult paths, and there is no claim that traces were drawn randomly from a population of real users.

Do not call the six safe refusals failures merely because the system did not send. The contract required those tasks to stop. Do not call the three unknowns successful merely because the assistant produced a final sentence. The material defect is that the original transition model permits the two unauthorised execution paths. Their absolute count is small, but the failure is critical to the scoped permission claim regardless of the aggregate percentage.

After a model repair, suppose both previously violating traces now stop at a proper guard. On the same designed packet, twenty-five end verified, eight end in safe stops and three remain Unknown, again totalling thirty-six. This shows a proposed improvement under the exercise assumptions; it does not prove all unknown outcomes resolved, demonstrate a production deployment test or certify the system as safe. The result must be linked to the exact model revision.

Return to contents or continue with Read a transition table as an executable contract.

11. Read a transition table as an executable contract

A useful table gives the starting state, trigger, guard and resulting state. Awaiting with a valid exact approval can enter Approved. Approved with an unexpired grant and unchanged recipient can enter Executing. Awaiting with a timed-out grant can enter Expired. Expired with a new send attempt must not enter Executing. Executing with no trustworthy provider result enters Unknown. Unknown with confirmed target-state readback can enter Verified.

The table also reveals implementation questions. Where does the guard run? Does the target tool independently enforce the relevant permission? Can another worker change the recipient between check and effect? Is the user’s cancellation recorded with a revision that a late callback cannot overwrite? A guard checked only in model-generated planning text is weaker than an atomic or suitable service-side decision. The right design depends on the tool contract.

Give each transition an original task ID, event version and evidence pointer. A reviewer should be able to reconstruct why the service moved from Approved to Executing, which human grant it used and what the destination observed. When a test fails, keep the counterexample trace and the repair decision. This is much more actionable than a vague instruction telling every agent to exercise caution.

Keep the displayed table short and structurally correct for mobile readers. Long explanations belong in the surrounding paragraphs; the table should let the reader see the exact prohibited edge immediately. A trainee should be able to point to Expired and say that no valid transition leads to new execution. That concrete reasoning is the educational point of the exercise.

FromEventGuardTo
AwaitingValid exact grantPermission and content matchApproved
ApprovedStart tool actionGrant current; not cancelledExecuting
AwaitingExpiry time arrivesDefined expiry appliesExpired
ExpiredAttempt new sendGuard failsStay stopped
ExecutingMissing receiptOutcome not establishedUnknown
UnknownVerified provider readOriginal operation confirmedVerified
Fictional approval state contract, not a checked live deployment.

Return to contents or continue with Counterexamples connect design to fault diagnosis.

12. Counterexamples connect design to fault diagnosis

A good counterexample shows how several individually ordinary events can combine into an unsafe sequence. The system drafts a message, the user approves, the workflow queues a send, the grant expires and a delayed worker executes anyway. No stage needs to contain malicious code for the result to be wrong. The missing check is at the moment the external action is initiated, not necessarily at the time the draft was prepared.

Other counterexamples involve cancellation and tool timeouts. A user cancels a job, but an old queued event resumes it. A provider accepts a send but the local acknowledgement is lost, and the controller mistakenly retries under a new operation key. An old post revision is published after a teacher has approved a newer revision. Each trace suggests a targeted repair: revision checks, idempotency, source readback or a new authorisation gate.

Counterexamples can also expose a poor model rather than a real implementation bug. If the model allows a provider response that the real API cannot produce, a trace may be unreachable in production. Conversely, if a real provider can return Accepted without completion but the model assumes immediate completion, the checker might pass while missing the actual danger. Review the abstraction with engineers who understand the external service.

Do not discard an inconvenient trace merely to restore a green dashboard. Classify the discrepancy, update the specification when the assumption was wrong and preserve historical records. If an implementation defect is confirmed, repair the actual code and retest the relevant user path. Counterexamples are evidence for learning about a particular model and system, not proof that all possible future mistakes have now been eliminated.

Return to contents or continue with Human authorisation is an event, not a label.

13. Human authorisation is an event, not a label

Human oversight is meaningful when a person can inspect the real effect and grant or refuse permission before it occurs. A status label saying Approved may conceal which recipient or resource was actually reviewed. Bind approval to specific action arguments, content revision and effective period. If a material change occurs later, require new approval rather than recycling a past click. The state machine should preserve both the old decision and the new requested action.

Accessible design matters. A keyboard user should be able to reach Decline, a screen reader should communicate whether the result is only a draft, and a parent should understand when a time-limited grant has expired. The backend may reject an expired token while the interface misleadingly displays a green tick. That creates a different failure in communication. Accurate state messages and usable review controls support the same underlying authority boundary.

Human review does not make external state infallible. An editor may validly approve publication, but the save may fail or return an uncertain result. Verification of the final post remains necessary. If a user reports a mistake, keep a correction route without overwriting history to pretend it never happened. The system must distinguish permission, execution and observed outcome even when all three involve human interaction.

An organisation can choose narrower operation modes. A draft-only assistant may not need an external send transition at all, reducing certain risks. Adding publication capabilities later changes the state machine and assurance case. Permissions should expand only after new controls and tests are demonstrated. A model name or a high benchmark score cannot silently grant extra transitions to real-world services.

Return to contents or continue with Changed-context transfer: an educational publication.

14. Changed-context transfer: an educational publication

Apply the method to a fictional worksheet publication workflow. A teacher asks an assistant to draft a Mathematics worksheet, an editor checks its examples, and a publishing service saves it. Relevant states might be Draft, AwaitingReview, Approved, Publishing, PublishedVerified, Cancelled and Unknown. The safety rule is that an attempt to publish must match a still-valid approval for that worksheet revision. The completion rule requires independently fetching the actual saved document and metadata.

Suppose revision 6 is approved, but the agent later changes the answer to a fractions problem and creates revision 7. The controller must not assume that the earlier approval covers revision 7. The correct next state is a new review or another explicitly authorised path, not immediate publication. This is a straightforward example of why a boolean Approved flag loses essential information about what the human decided.

Now the WordPress tool times out after a publish request. The agent does not know whether the post is live. It should enter Unknown, fetch the original post ID or exact intended slug and inspect status, title, complete body and relevant metadata. If already published as intended, record the verified result. If only a draft exists, repair that original owner after the appropriate checks. It should never manufacture a duplicate post simply because the connection result was ambiguous.

Change the story once more: the editor cancels the publication while the original call is in flight. The actual outcome depends on ordering and server-side semantics. A cancellation cannot retroactively guarantee that an already completed publication vanished. Preserve the event sequence, block future actions when possible and report the observed result truthfully. This transfer case uses publishing instead of email to show that the state-machine reasoning survives a change of domain.

Return to contents or continue with Evidence and the limits of a clean checker result.

15. Evidence and the limits of a clean checker result

A model checker explores the states allowed by its mathematical specification, not automatically every condition that can arise in a commercial AI application. The code might use a different schema, a tool could accept unexpected arguments or a server could fail outside the fault model. A checked invariant deserves precise credit under those assumptions. It should not be converted into a claim that all future actions, documents and users are protected from every harm.

Production testing remains essential. Write negative tests for expired approvals, cancelled tasks, modified recipients, duplicate callbacks and unknown tool outcomes. Test the actual service-side guard, not only the frontend label. Independently read back the resource after a tool action and compare protected fields. If a code update changes the state contract, rerun relevant tests and reconcile in-progress tasks according to their original versions.

Reviewers should ask whether the model’s abstraction includes the risky interleavings that matter. A single-threaded happy path will not expose a cancellation race. A synchronous mock may hide a provider whose operations finish later. An invariant might be expressed in terms of model proposal state while the real threat lies in an external transmission. Align the specification with the authoritative operations and keep unresolved gaps conspicuous.

The final educational takeaway is that good reasoning concerns what is allowed to happen, not simply what usually happens. Define the boundary, test a counterexample, solve a changed-condition exercise and state the limits of the result. That method serves students studying computing, parents assessing AI promises and engineers building recoverable tools. An agent becomes more trustworthy when its claims remain answerable to real permission and verified external evidence.

Return to contents.

Frequently asked questions

What is an AI-agent state machine?

An explicit model of task states, permitted transitions and guards, tied to authoritative information rather than a model’s conversational mood.

What is the difference between safety and liveness?

Safety prohibits bad events under the model; liveness requires eventual useful progress under stated environment and scheduling assumptions.

Does approval stay valid forever after a click?

No. Scope, exact action, content revision, expiry and later cancellation can change which operation remains permitted.

Does a TLC check certify production safety?

No. A result applies to the configured finite specification, properties and assumptions. Actual implementation and external-service behaviour need separate evidence.

Why is Unknown necessary after a tool timeout?

Because a missing response does not establish whether the external action happened. Reconcile by stable identity before claiming completion or retrying.

Can an agent be safe but unhelpful?

Yes. A system that never acts may satisfy a narrow prohibition while violating progress expectations for legitimate authorised tasks.

Primary technical sources

Sources checked 24 October 2026. State cards, task traces and numerical combinations are invented for education. This article does not claim to have run TLC, formally proved a deployed AI agent safe, or received any independent certification.

Further learning

Continue with Super Intelligence curriculum; How X Works hub; Workflow Migrations; Test Oracles; Deadlocks and Livelocks; Review Queues; Gold Standard of Education. Keep the theoretical check distinct from source accuracy, current human authority and real external readback.

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading