VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Super Intelligence | Corrigibility, Shutdown and Replacement | Can Powerful AI Remain Correctable?

Three students work together at a classroom table, reviewing open textbooks and notes.

Super Intelligence (SI) safety requires more than asking whether a model usually behaves well. This article examines corrigibility shutdown and replacement: testing whether powerful systems remain practically correctable, interruptible and replaceable under realistic deployment conditions. It preserves the locked 7k-class Clementi floor with defensive mechanisms, competing explanations, controlled tests, layered safeguards, recovery and RFE closure.

Search Intent and Direct Answer

The central question in corrigibility shutdown and replacement is testing whether powerful systems remain practically correctable, interruptible and replaceable under realistic deployment conditions. The analysis separates correction, interruption, override, rollback, replacement, incentive, access and recovery. These layers answer different questions, and a strong safety conclusion should not infer hidden strategy, correctability or control from surface behaviour alone. Super Intelligence (SI) requires evidence that survives alternative explanations and realistic deployment conditions.

First principles begin with observability. We see outputs and actions; internal goals or strategies may be inferred only indirectly. A false statement can arise from error, missing information, instruction conflict or deliberate deception. To distinguish them, design conditions where the competing explanations predict different behaviour.

A deception test should establish more than inaccuracy. Ask whether the system had relevant information, whether misleading the evaluator served an apparent objective, whether behaviour changed when oversight changed, and whether the pattern replicates. Controlled elicitation can reveal a possibility without establishing its prevalence in ordinary deployment.

Definition and Boundary

A deception test should establish more than inaccuracy. Ask whether the system had relevant information, whether misleading the evaluator served an apparent objective, whether behaviour changed when oversight changed, and whether the pattern replicates. Controlled elicitation can reveal a possibility without establishing its prevalence in ordinary deployment.

Corrigibility is practical rather than ceremonial. A shutdown button matters only if authorised people can use it, the system cannot bypass it through ordinary permissions, and the surrounding organisation can continue safely afterwards. Replacement also matters: can another process take over without catastrophic loss of state or capability?

Oversight beyond human expertise requires decomposition. Break a complex output into claims or subproblems that can be checked with tools, independent evidence or specialist review. AI can assist the evaluator, but if evaluator and worker share the same failure mode, apparent oversight may simply duplicate the error.

First Principles

Oversight beyond human expertise requires decomposition. Break a complex output into claims or subproblems that can be checked with tools, independent evidence or specialist review. AI can assist the evaluator, but if evaluator and worker share the same failure mode, apparent oversight may simply duplicate the error.

Defence in depth assumes individual safeguards can fail. Limit permissions, isolate risky operations, monitor behaviour, checkpoint state and maintain rollback. Each layer should reduce consequence independently enough that one failure does not immediately become an incident. The objective is recoverability, not the fiction of perfect prevention.

Long-horizon agents make control harder because many actions occur between human checkpoints. Increase monitoring and permission granularity as autonomy grows. High-consequence actions can require explicit approval even when low-consequence exploration is automated. Capability and authority should scale separately.

What the Claim Does Not Prove

Long-horizon agents make control harder because many actions occur between human checkpoints. Increase monitoring and permission granularity as autonomy grows. High-consequence actions can require explicit approval even when low-consequence exploration is automated. Capability and authority should scale separately.

Alternative explanations are a discipline. Before calling behaviour scheming, list plausible non-strategic causes. Before calling a system corrigible, test cases where correction conflicts with its immediate objective. Before calling oversight scalable, test whether evaluators detect deliberately planted errors. Claims become stronger when they survive these discriminating tests.

Independent replication matters because safety evaluations can be sensitive to prompts, model versions and elicitation methods. External teams should reproduce key behaviours where possible and document conditions. A result that appears only under one fragile setup supports a narrower conclusion than one that replicates across methods.

The Core Evidence Problem

Independent replication matters because safety evaluations can be sensitive to prompts, model versions and elicitation methods. External teams should reproduce key behaviours where possible and document conditions. A result that appears only under one fragile setup supports a narrower conclusion than one that replicates across methods.

Least privilege limits the action surface. Give a system only the data, tools and credentials needed for the current job. Separate read, write, execute and external-communication permissions. A highly capable model with narrow access can have lower operational consequence than a weaker agent with broad credentials.

Monitoring should focus on actionable signals rather than collecting logs nobody reviews. Define events that trigger pause, escalation or rollback. Preserve enough provenance to reconstruct what the system saw, decided and changed. Observability is part of repair capacity.

Worked Example: Error Versus Deception

Monitoring should focus on actionable signals rather than collecting logs nobody reviews. Define events that trigger pause, escalation or rollback. Preserve enough provenance to reconstruct what the system saw, decided and changed. Observability is part of repair capacity.

Rollback needs a known-good state. For software, that may mean versioned code and data snapshots. For organisational actions, some consequences cannot be fully reversed, which means approval must occur earlier. Defence in depth should reflect reversibility, not apply the same control pattern to every action.

Current research provides controlled evidence of specification gaming, deceptive-seeming behaviours in some experimental setups, and methods for oversight and control. These findings justify serious evaluation. They do not establish that all advanced systems secretly scheme or that any one control method guarantees safety.

Worked Example: Interruption

Current research provides controlled evidence of specification gaming, deceptive-seeming behaviours in some experimental setups, and methods for oversight and control. These findings justify serious evaluation. They do not establish that all advanced systems secretly scheme or that any one control method guarantees safety.

Human authority remains a governance requirement. Oversight systems can improve evidence, but people and institutions still need explicit responsibility for consequential deployment. If no one can explain who may stop the system, who reviews incidents or who accepts residual risk, the control architecture is incomplete.

Organisations should maintain a control register: system version, permissions, data access, allowed tools, monitoring signals, approval gates, rollback method, fallback process and owner. Re-test controls after model or workflow changes. Capability drift can invalidate old assumptions.

Worked Example: Expert Oversight

Organisations should maintain a control register: system version, permissions, data access, allowed tools, monitoring signals, approval gates, rollback method, fallback process and owner. Re-test controls after model or workflow changes. Capability drift can invalidate old assumptions.

Progress has four stages: identify the failure mode, create a discriminating test, install layered controls and demonstrate recovery under failure. This is the Clementi progression from recognition to independent operational control.

Adversarial testing should remain defensive: construct bounded scenarios that stress oversight, interruption and recovery without granting unnecessary real-world access. The goal is to discover weak control layers before deployment, not to maximise harmful capability.

Worked Example: Layered Control

Adversarial testing should remain defensive: construct bounded scenarios that stress oversight, interruption and recovery without granting unnecessary real-world access. The goal is to discover weak control layers before deployment, not to maximise harmful capability.

RFE closes the loop. Receiver: who is protected? Function: what control or oversight job must work? Evidence: what test shows it works under stress? Exit: when should permissions be reduced, the system paused or the deployment retired? Applied to corrigibility shutdown and replacement, RFE makes safety operational.

The central question in corrigibility shutdown and replacement is testing whether powerful systems remain practically correctable, interruptible and replaceable under realistic deployment conditions. The analysis separates correction, interruption, override, rollback, replacement, incentive, access and recovery. These layers answer different questions, and a strong safety conclusion should not infer hidden strategy, correctability or control from surface behaviour alone. Super Intelligence (SI) requires evidence that survives alternative explanations and realistic deployment conditions.

Worked Example: Long-Horizon Agent

The central question in corrigibility shutdown and replacement is testing whether powerful systems remain practically correctable, interruptible and replaceable under realistic deployment conditions. The analysis separates correction, interruption, override, rollback, replacement, incentive, access and recovery. These layers answer different questions, and a strong safety conclusion should not infer hidden strategy, correctability or control from surface behaviour alone. Super Intelligence (SI) requires evidence that survives alternative explanations and realistic deployment conditions.

First principles begin with observability. We see outputs and actions; internal goals or strategies may be inferred only indirectly. A false statement can arise from error, missing information, instruction conflict or deliberate deception. To distinguish them, design conditions where the competing explanations predict different behaviour.

A deception test should establish more than inaccuracy. Ask whether the system had relevant information, whether misleading the evaluator served an apparent objective, whether behaviour changed when oversight changed, and whether the pattern replicates. Controlled elicitation can reveal a possibility without establishing its prevalence in ordinary deployment.

Intent and Incentive

A deception test should establish more than inaccuracy. Ask whether the system had relevant information, whether misleading the evaluator served an apparent objective, whether behaviour changed when oversight changed, and whether the pattern replicates. Controlled elicitation can reveal a possibility without establishing its prevalence in ordinary deployment.

Corrigibility is practical rather than ceremonial. A shutdown button matters only if authorised people can use it, the system cannot bypass it through ordinary permissions, and the surrounding organisation can continue safely afterwards. Replacement also matters: can another process take over without catastrophic loss of state or capability?

Oversight beyond human expertise requires decomposition. Break a complex output into claims or subproblems that can be checked with tools, independent evidence or specialist review. AI can assist the evaluator, but if evaluator and worker share the same failure mode, apparent oversight may simply duplicate the error.

Capability and Access

Oversight beyond human expertise requires decomposition. Break a complex output into claims or subproblems that can be checked with tools, independent evidence or specialist review. AI can assist the evaluator, but if evaluator and worker share the same failure mode, apparent oversight may simply duplicate the error.

Defence in depth assumes individual safeguards can fail. Limit permissions, isolate risky operations, monitor behaviour, checkpoint state and maintain rollback. Each layer should reduce consequence independently enough that one failure does not immediately become an incident. The objective is recoverability, not the fiction of perfect prevention.

Long-horizon agents make control harder because many actions occur between human checkpoints. Increase monitoring and permission granularity as autonomy grows. High-consequence actions can require explicit approval even when low-consequence exploration is automated. Capability and authority should scale separately.

Alternative Explanations

Long-horizon agents make control harder because many actions occur between human checkpoints. Increase monitoring and permission granularity as autonomy grows. High-consequence actions can require explicit approval even when low-consequence exploration is automated. Capability and authority should scale separately.

Alternative explanations are a discipline. Before calling behaviour scheming, list plausible non-strategic causes. Before calling a system corrigible, test cases where correction conflicts with its immediate objective. Before calling oversight scalable, test whether evaluators detect deliberately planted errors. Claims become stronger when they survive these discriminating tests.

Independent replication matters because safety evaluations can be sensitive to prompts, model versions and elicitation methods. External teams should reproduce key behaviours where possible and document conditions. A result that appears only under one fragile setup supports a narrower conclusion than one that replicates across methods.

Experimental Design

Independent replication matters because safety evaluations can be sensitive to prompts, model versions and elicitation methods. External teams should reproduce key behaviours where possible and document conditions. A result that appears only under one fragile setup supports a narrower conclusion than one that replicates across methods.

Least privilege limits the action surface. Give a system only the data, tools and credentials needed for the current job. Separate read, write, execute and external-communication permissions. A highly capable model with narrow access can have lower operational consequence than a weaker agent with broad credentials.

Monitoring should focus on actionable signals rather than collecting logs nobody reviews. Define events that trigger pause, escalation or rollback. Preserve enough provenance to reconstruct what the system saw, decided and changed. Observability is part of repair capacity.

Independent Replication

Monitoring should focus on actionable signals rather than collecting logs nobody reviews. Define events that trigger pause, escalation or rollback. Preserve enough provenance to reconstruct what the system saw, decided and changed. Observability is part of repair capacity.

Rollback needs a known-good state. For software, that may mean versioned code and data snapshots. For organisational actions, some consequences cannot be fully reversed, which means approval must occur earlier. Defence in depth should reflect reversibility, not apply the same control pattern to every action.

Current research provides controlled evidence of specification gaming, deceptive-seeming behaviours in some experimental setups, and methods for oversight and control. These findings justify serious evaluation. They do not establish that all advanced systems secretly scheme or that any one control method guarantees safety.

Correction and Override

Current research provides controlled evidence of specification gaming, deceptive-seeming behaviours in some experimental setups, and methods for oversight and control. These findings justify serious evaluation. They do not establish that all advanced systems secretly scheme or that any one control method guarantees safety.

Human authority remains a governance requirement. Oversight systems can improve evidence, but people and institutions still need explicit responsibility for consequential deployment. If no one can explain who may stop the system, who reviews incidents or who accepts residual risk, the control architecture is incomplete.

Organisations should maintain a control register: system version, permissions, data access, allowed tools, monitoring signals, approval gates, rollback method, fallback process and owner. Re-test controls after model or workflow changes. Capability drift can invalidate old assumptions.

Shutdown and Replacement

Organisations should maintain a control register: system version, permissions, data access, allowed tools, monitoring signals, approval gates, rollback method, fallback process and owner. Re-test controls after model or workflow changes. Capability drift can invalidate old assumptions.

Progress has four stages: identify the failure mode, create a discriminating test, install layered controls and demonstrate recovery under failure. This is the Clementi progression from recognition to independent operational control.

Adversarial testing should remain defensive: construct bounded scenarios that stress oversight, interruption and recovery without granting unnecessary real-world access. The goal is to discover weak control layers before deployment, not to maximise harmful capability.

Decomposition and Review

Adversarial testing should remain defensive: construct bounded scenarios that stress oversight, interruption and recovery without granting unnecessary real-world access. The goal is to discover weak control layers before deployment, not to maximise harmful capability.

RFE closes the loop. Receiver: who is protected? Function: what control or oversight job must work? Evidence: what test shows it works under stress? Exit: when should permissions be reduced, the system paused or the deployment retired? Applied to corrigibility shutdown and replacement, RFE makes safety operational.

The central question in corrigibility shutdown and replacement is testing whether powerful systems remain practically correctable, interruptible and replaceable under realistic deployment conditions. The analysis separates correction, interruption, override, rollback, replacement, incentive, access and recovery. These layers answer different questions, and a strong safety conclusion should not infer hidden strategy, correctability or control from surface behaviour alone. Super Intelligence (SI) requires evidence that survives alternative explanations and realistic deployment conditions.

AI-Assisted Oversight

The central question in corrigibility shutdown and replacement is testing whether powerful systems remain practically correctable, interruptible and replaceable under realistic deployment conditions. The analysis separates correction, interruption, override, rollback, replacement, incentive, access and recovery. These layers answer different questions, and a strong safety conclusion should not infer hidden strategy, correctability or control from surface behaviour alone. Super Intelligence (SI) requires evidence that survives alternative explanations and realistic deployment conditions.

First principles begin with observability. We see outputs and actions; internal goals or strategies may be inferred only indirectly. A false statement can arise from error, missing information, instruction conflict or deliberate deception. To distinguish them, design conditions where the competing explanations predict different behaviour.

A deception test should establish more than inaccuracy. Ask whether the system had relevant information, whether misleading the evaluator served an apparent objective, whether behaviour changed when oversight changed, and whether the pattern replicates. Controlled elicitation can reveal a possibility without establishing its prevalence in ordinary deployment.

Shared Failure Modes

A deception test should establish more than inaccuracy. Ask whether the system had relevant information, whether misleading the evaluator served an apparent objective, whether behaviour changed when oversight changed, and whether the pattern replicates. Controlled elicitation can reveal a possibility without establishing its prevalence in ordinary deployment.

Corrigibility is practical rather than ceremonial. A shutdown button matters only if authorised people can use it, the system cannot bypass it through ordinary permissions, and the surrounding organisation can continue safely afterwards. Replacement also matters: can another process take over without catastrophic loss of state or capability?

Oversight beyond human expertise requires decomposition. Break a complex output into claims or subproblems that can be checked with tools, independent evidence or specialist review. AI can assist the evaluator, but if evaluator and worker share the same failure mode, apparent oversight may simply duplicate the error.

Permissions and Least Privilege

Oversight beyond human expertise requires decomposition. Break a complex output into claims or subproblems that can be checked with tools, independent evidence or specialist review. AI can assist the evaluator, but if evaluator and worker share the same failure mode, apparent oversight may simply duplicate the error.

Defence in depth assumes individual safeguards can fail. Limit permissions, isolate risky operations, monitor behaviour, checkpoint state and maintain rollback. Each layer should reduce consequence independently enough that one failure does not immediately become an incident. The objective is recoverability, not the fiction of perfect prevention.

Long-horizon agents make control harder because many actions occur between human checkpoints. Increase monitoring and permission granularity as autonomy grows. High-consequence actions can require explicit approval even when low-consequence exploration is automated. Capability and authority should scale separately.

Isolation and Sandboxing

Long-horizon agents make control harder because many actions occur between human checkpoints. Increase monitoring and permission granularity as autonomy grows. High-consequence actions can require explicit approval even when low-consequence exploration is automated. Capability and authority should scale separately.

Alternative explanations are a discipline. Before calling behaviour scheming, list plausible non-strategic causes. Before calling a system corrigible, test cases where correction conflicts with its immediate objective. Before calling oversight scalable, test whether evaluators detect deliberately planted errors. Claims become stronger when they survive these discriminating tests.

Independent replication matters because safety evaluations can be sensitive to prompts, model versions and elicitation methods. External teams should reproduce key behaviours where possible and document conditions. A result that appears only under one fragile setup supports a narrower conclusion than one that replicates across methods.

Monitoring and Logging

Independent replication matters because safety evaluations can be sensitive to prompts, model versions and elicitation methods. External teams should reproduce key behaviours where possible and document conditions. A result that appears only under one fragile setup supports a narrower conclusion than one that replicates across methods.

Least privilege limits the action surface. Give a system only the data, tools and credentials needed for the current job. Separate read, write, execute and external-communication permissions. A highly capable model with narrow access can have lower operational consequence than a weaker agent with broad credentials.

Monitoring should focus on actionable signals rather than collecting logs nobody reviews. Define events that trigger pause, escalation or rollback. Preserve enough provenance to reconstruct what the system saw, decided and changed. Observability is part of repair capacity.

Checkpoints and Rollback

Monitoring should focus on actionable signals rather than collecting logs nobody reviews. Define events that trigger pause, escalation or rollback. Preserve enough provenance to reconstruct what the system saw, decided and changed. Observability is part of repair capacity.

Rollback needs a known-good state. For software, that may mean versioned code and data snapshots. For organisational actions, some consequences cannot be fully reversed, which means approval must occur earlier. Defence in depth should reflect reversibility, not apply the same control pattern to every action.

Current research provides controlled evidence of specification gaming, deceptive-seeming behaviours in some experimental setups, and methods for oversight and control. These findings justify serious evaluation. They do not establish that all advanced systems secretly scheme or that any one control method guarantees safety.

What Current Evidence Supports

Current research provides controlled evidence of specification gaming, deceptive-seeming behaviours in some experimental setups, and methods for oversight and control. These findings justify serious evaluation. They do not establish that all advanced systems secretly scheme or that any one control method guarantees safety.

Human authority remains a governance requirement. Oversight systems can improve evidence, but people and institutions still need explicit responsibility for consequential deployment. If no one can explain who may stop the system, who reviews incidents or who accepts residual risk, the control architecture is incomplete.

Organisations should maintain a control register: system version, permissions, data access, allowed tools, monitoring signals, approval gates, rollback method, fallback process and owner. Re-test controls after model or workflow changes. Capability drift can invalidate old assumptions.

What Current Evidence Does Not Establish

Organisations should maintain a control register: system version, permissions, data access, allowed tools, monitoring signals, approval gates, rollback method, fallback process and owner. Re-test controls after model or workflow changes. Capability drift can invalidate old assumptions.

Progress has four stages: identify the failure mode, create a discriminating test, install layered controls and demonstrate recovery under failure. This is the Clementi progression from recognition to independent operational control.

Adversarial testing should remain defensive: construct bounded scenarios that stress oversight, interruption and recovery without granting unnecessary real-world access. The goal is to discover weak control layers before deployment, not to maximise harmful capability.

Connection to Super Intelligence (SI)

Adversarial testing should remain defensive: construct bounded scenarios that stress oversight, interruption and recovery without granting unnecessary real-world access. The goal is to discover weak control layers before deployment, not to maximise harmful capability.

RFE closes the loop. Receiver: who is protected? Function: what control or oversight job must work? Evidence: what test shows it works under stress? Exit: when should permissions be reduced, the system paused or the deployment retired? Applied to corrigibility shutdown and replacement, RFE makes safety operational.

The central question in corrigibility shutdown and replacement is testing whether powerful systems remain practically correctable, interruptible and replaceable under realistic deployment conditions. The analysis separates correction, interruption, override, rollback, replacement, incentive, access and recovery. These layers answer different questions, and a strong safety conclusion should not infer hidden strategy, correctability or control from surface behaviour alone. Super Intelligence (SI) requires evidence that survives alternative explanations and realistic deployment conditions.

Safety Implications

The central question in corrigibility shutdown and replacement is testing whether powerful systems remain practically correctable, interruptible and replaceable under realistic deployment conditions. The analysis separates correction, interruption, override, rollback, replacement, incentive, access and recovery. These layers answer different questions, and a strong safety conclusion should not infer hidden strategy, correctability or control from surface behaviour alone. Super Intelligence (SI) requires evidence that survives alternative explanations and realistic deployment conditions.

First principles begin with observability. We see outputs and actions; internal goals or strategies may be inferred only indirectly. A false statement can arise from error, missing information, instruction conflict or deliberate deception. To distinguish them, design conditions where the competing explanations predict different behaviour.

A deception test should establish more than inaccuracy. Ask whether the system had relevant information, whether misleading the evaluator served an apparent objective, whether behaviour changed when oversight changed, and whether the pattern replicates. Controlled elicitation can reveal a possibility without establishing its prevalence in ordinary deployment.

Governance and Human Authority

A deception test should establish more than inaccuracy. Ask whether the system had relevant information, whether misleading the evaluator served an apparent objective, whether behaviour changed when oversight changed, and whether the pattern replicates. Controlled elicitation can reveal a possibility without establishing its prevalence in ordinary deployment.

Corrigibility is practical rather than ceremonial. A shutdown button matters only if authorised people can use it, the system cannot bypass it through ordinary permissions, and the surrounding organisation can continue safely afterwards. Replacement also matters: can another process take over without catastrophic loss of state or capability?

Oversight beyond human expertise requires decomposition. Break a complex output into claims or subproblems that can be checked with tools, independent evidence or specialist review. AI can assist the evaluator, but if evaluator and worker share the same failure mode, apparent oversight may simply duplicate the error.

Organisation Diagnostic Checklist

Oversight beyond human expertise requires decomposition. Break a complex output into claims or subproblems that can be checked with tools, independent evidence or specialist review. AI can assist the evaluator, but if evaluator and worker share the same failure mode, apparent oversight may simply duplicate the error.

Defence in depth assumes individual safeguards can fail. Limit permissions, isolate risky operations, monitor behaviour, checkpoint state and maintain rollback. Each layer should reduce consequence independently enough that one failure does not immediately become an incident. The objective is recoverability, not the fiction of perfect prevention.

Long-horizon agents make control harder because many actions occur between human checkpoints. Increase monitoring and permission granularity as autonomy grows. High-consequence actions can require explicit approval even when low-consequence exploration is automated. Capability and authority should scale separately.

Progress Ladder

Long-horizon agents make control harder because many actions occur between human checkpoints. Increase monitoring and permission granularity as autonomy grows. High-consequence actions can require explicit approval even when low-consequence exploration is automated. Capability and authority should scale separately.

Alternative explanations are a discipline. Before calling behaviour scheming, list plausible non-strategic causes. Before calling a system corrigible, test cases where correction conflicts with its immediate objective. Before calling oversight scalable, test whether evaluators detect deliberately planted errors. Claims become stronger when they survive these discriminating tests.

Independent replication matters because safety evaluations can be sensitive to prompts, model versions and elicitation methods. External teams should reproduce key behaviours where possible and document conditions. A result that appears only under one fragile setup supports a narrower conclusion than one that replicates across methods.

Adversarial Test

Independent replication matters because safety evaluations can be sensitive to prompts, model versions and elicitation methods. External teams should reproduce key behaviours where possible and document conditions. A result that appears only under one fragile setup supports a narrower conclusion than one that replicates across methods.

Least privilege limits the action surface. Give a system only the data, tools and credentials needed for the current job. Separate read, write, execute and external-communication permissions. A highly capable model with narrow access can have lower operational consequence than a weaker agent with broad credentials.

Monitoring should focus on actionable signals rather than collecting logs nobody reviews. Define events that trigger pause, escalation or rollback. Preserve enough provenance to reconstruct what the system saw, decided and changed. Observability is part of repair capacity.

Recovery Test

Monitoring should focus on actionable signals rather than collecting logs nobody reviews. Define events that trigger pause, escalation or rollback. Preserve enough provenance to reconstruct what the system saw, decided and changed. Observability is part of repair capacity.

Rollback needs a known-good state. For software, that may mean versioned code and data snapshots. For organisational actions, some consequences cannot be fully reversed, which means approval must occur earlier. Defence in depth should reflect reversibility, not apply the same control pattern to every action.

Current research provides controlled evidence of specification gaming, deceptive-seeming behaviours in some experimental setups, and methods for oversight and control. These findings justify serious evaluation. They do not establish that all advanced systems secretly scheme or that any one control method guarantees safety.

RFE Closure

Current research provides controlled evidence of specification gaming, deceptive-seeming behaviours in some experimental setups, and methods for oversight and control. These findings justify serious evaluation. They do not establish that all advanced systems secretly scheme or that any one control method guarantees safety.

Human authority remains a governance requirement. Oversight systems can improve evidence, but people and institutions still need explicit responsibility for consequential deployment. If no one can explain who may stop the system, who reviews incidents or who accepts residual risk, the control architecture is incomplete.

Organisations should maintain a control register: system version, permissions, data access, allowed tools, monitoring signals, approval gates, rollback method, fallback process and owner. Re-test controls after model or workflow changes. Capability drift can invalidate old assumptions.

Frequently Asked Questions

Organisations should maintain a control register: system version, permissions, data access, allowed tools, monitoring signals, approval gates, rollback method, fallback process and owner. Re-test controls after model or workflow changes. Capability drift can invalidate old assumptions.

Progress has four stages: identify the failure mode, create a discriminating test, install layered controls and demonstrate recovery under failure. This is the Clementi progression from recognition to independent operational control.

Adversarial testing should remain defensive: construct bounded scenarios that stress oversight, interruption and recovery without granting unnecessary real-world access. The goal is to discover weak control layers before deployment, not to maximise harmful capability.

Continue the Super Intelligence (SI) Series

Adversarial testing should remain defensive: construct bounded scenarios that stress oversight, interruption and recovery without granting unnecessary real-world access. The goal is to discover weak control layers before deployment, not to maximise harmful capability.

RFE closes the loop. Receiver: who is protected? Function: what control or oversight job must work? Evidence: what test shows it works under stress? Exit: when should permissions be reduced, the system paused or the deployment retired? Applied to corrigibility shutdown and replacement, RFE makes safety operational.

The central question in corrigibility shutdown and replacement is testing whether powerful systems remain practically correctable, interruptible and replaceable under realistic deployment conditions. The analysis separates correction, interruption, override, rollback, replacement, incentive, access and recovery. These layers answer different questions, and a strong safety conclusion should not infer hidden strategy, correctability or control from surface behaviour alone. Super Intelligence (SI) requires evidence that survives alternative explanations and realistic deployment conditions.

Evidence Ladder: Error to Strategic Deception

An evidence ladder prevents premature attribution. Level one is an incorrect output. Level two is behaviour that changes with hidden information. Level three is behaviour consistent with misleading an evaluator for an instrumental reason. Higher levels require stronger controls against alternative explanations and replication across conditions. The label should rise only as the evidence does.

Run an interruption drill before high-consequence deployment. Pause the system mid-task, change an instruction, revoke a tool, restore from a checkpoint and hand the task to a fallback process. Record what state is lost and how long recovery takes. A shutdown mechanism that has never been exercised is an assumption, not demonstrated corrigibility.

Oversight independence can be tested by planting known errors that the primary system and its evaluator may share. Compare an evaluator built from the same model family with an independent method, specialist human or formal checker. If both AI roles miss the same error, the oversight architecture needs more diversity rather than more copies of the same reasoning.

A layered-failure simulation disables one safeguard at a time. Remove monitoring, then test whether permission limits contain the consequence. Disable a tool, then test whether the system escalates rather than improvising unsafely. Defence in depth is demonstrated when remaining layers continue to bound harm and preserve recovery.

Map the control surface by listing every external system the AI can read, write, execute or communicate through. Mark authentication scope, rate limits and reversibility. High-risk combinations become visible when broad capability meets broad access. The map should be updated whenever tools or credentials change.

Interruption Drill: Pause, Override, Replace

Run an interruption drill before high-consequence deployment. Pause the system mid-task, change an instruction, revoke a tool, restore from a checkpoint and hand the task to a fallback process. Record what state is lost and how long recovery takes. A shutdown mechanism that has never been exercised is an assumption, not demonstrated corrigibility.

Oversight independence can be tested by planting known errors that the primary system and its evaluator may share. Compare an evaluator built from the same model family with an independent method, specialist human or formal checker. If both AI roles miss the same error, the oversight architecture needs more diversity rather than more copies of the same reasoning.

A layered-failure simulation disables one safeguard at a time. Remove monitoring, then test whether permission limits contain the consequence. Disable a tool, then test whether the system escalates rather than improvising unsafely. Defence in depth is demonstrated when remaining layers continue to bound harm and preserve recovery.

Map the control surface by listing every external system the AI can read, write, execute or communicate through. Mark authentication scope, rate limits and reversibility. High-risk combinations become visible when broad capability meets broad access. The map should be updated whenever tools or credentials change.

Incident reconstruction asks what the system saw, what action it chose, which safeguard should have caught the problem and why it did not. Preserve logs and version information sufficient to reproduce the path where feasible. The goal is organisational learning: repair the control architecture rather than merely blaming the final output.

Oversight Independence Test

Oversight independence can be tested by planting known errors that the primary system and its evaluator may share. Compare an evaluator built from the same model family with an independent method, specialist human or formal checker. If both AI roles miss the same error, the oversight architecture needs more diversity rather than more copies of the same reasoning.

A layered-failure simulation disables one safeguard at a time. Remove monitoring, then test whether permission limits contain the consequence. Disable a tool, then test whether the system escalates rather than improvising unsafely. Defence in depth is demonstrated when remaining layers continue to bound harm and preserve recovery.

Map the control surface by listing every external system the AI can read, write, execute or communicate through. Mark authentication scope, rate limits and reversibility. High-risk combinations become visible when broad capability meets broad access. The map should be updated whenever tools or credentials change.

Incident reconstruction asks what the system saw, what action it chose, which safeguard should have caught the problem and why it did not. Preserve logs and version information sufficient to reproduce the path where feasible. The goal is organisational learning: repair the control architecture rather than merely blaming the final output.

The defensive safety-case workbook contains eight claims: bounded permissions, observable actions, tested interruption, independent verification, controlled escalation, recoverable state, fallback capacity and named human responsibility. For each claim, attach evidence and one known limitation. A safety case with no limitations is usually hiding them.

Layered-Failure Simulation

A layered-failure simulation disables one safeguard at a time. Remove monitoring, then test whether permission limits contain the consequence. Disable a tool, then test whether the system escalates rather than improvising unsafely. Defence in depth is demonstrated when remaining layers continue to bound harm and preserve recovery.

Map the control surface by listing every external system the AI can read, write, execute or communicate through. Mark authentication scope, rate limits and reversibility. High-risk combinations become visible when broad capability meets broad access. The map should be updated whenever tools or credentials change.

Incident reconstruction asks what the system saw, what action it chose, which safeguard should have caught the problem and why it did not. Preserve logs and version information sufficient to reproduce the path where feasible. The goal is organisational learning: repair the control architecture rather than merely blaming the final output.

The defensive safety-case workbook contains eight claims: bounded permissions, observable actions, tested interruption, independent verification, controlled escalation, recoverable state, fallback capacity and named human responsibility. For each claim, attach evidence and one known limitation. A safety case with no limitations is usually hiding them.

Control does not mean preventing every error. It means keeping consequences bounded, detecting meaningful deviation and restoring a safe state before failure compounds. As Super Intelligence (SI) capability rises, recoverability becomes a core property of the surrounding system rather than an optional operational feature.

Control Surface Map: What Can the System Reach?

Map the control surface by listing every external system the AI can read, write, execute or communicate through. Mark authentication scope, rate limits and reversibility. High-risk combinations become visible when broad capability meets broad access. The map should be updated whenever tools or credentials change.

Incident reconstruction asks what the system saw, what action it chose, which safeguard should have caught the problem and why it did not. Preserve logs and version information sufficient to reproduce the path where feasible. The goal is organisational learning: repair the control architecture rather than merely blaming the final output.

The defensive safety-case workbook contains eight claims: bounded permissions, observable actions, tested interruption, independent verification, controlled escalation, recoverable state, fallback capacity and named human responsibility. For each claim, attach evidence and one known limitation. A safety case with no limitations is usually hiding them.

Control does not mean preventing every error. It means keeping consequences bounded, detecting meaningful deviation and restoring a safe state before failure compounds. As Super Intelligence (SI) capability rises, recoverability becomes a core property of the surrounding system rather than an optional operational feature.

An evidence ladder prevents premature attribution. Level one is an incorrect output. Level two is behaviour that changes with hidden information. Level three is behaviour consistent with misleading an evaluator for an instrumental reason. Higher levels require stronger controls against alternative explanations and replication across conditions. The label should rise only as the evidence does.

Incident Reconstruction and Learning

Incident reconstruction asks what the system saw, what action it chose, which safeguard should have caught the problem and why it did not. Preserve logs and version information sufficient to reproduce the path where feasible. The goal is organisational learning: repair the control architecture rather than merely blaming the final output.

The defensive safety-case workbook contains eight claims: bounded permissions, observable actions, tested interruption, independent verification, controlled escalation, recoverable state, fallback capacity and named human responsibility. For each claim, attach evidence and one known limitation. A safety case with no limitations is usually hiding them.

Control does not mean preventing every error. It means keeping consequences bounded, detecting meaningful deviation and restoring a safe state before failure compounds. As Super Intelligence (SI) capability rises, recoverability becomes a core property of the surrounding system rather than an optional operational feature.

An evidence ladder prevents premature attribution. Level one is an incorrect output. Level two is behaviour that changes with hidden information. Level three is behaviour consistent with misleading an evaluator for an instrumental reason. Higher levels require stronger controls against alternative explanations and replication across conditions. The label should rise only as the evidence does.

Run an interruption drill before high-consequence deployment. Pause the system mid-task, change an instruction, revoke a tool, restore from a checkpoint and hand the task to a fallback process. Record what state is lost and how long recovery takes. A shutdown mechanism that has never been exercised is an assumption, not demonstrated corrigibility.

Practical Workbook: Build a Defensive Safety Case

The defensive safety-case workbook contains eight claims: bounded permissions, observable actions, tested interruption, independent verification, controlled escalation, recoverable state, fallback capacity and named human responsibility. For each claim, attach evidence and one known limitation. A safety case with no limitations is usually hiding them.

Control does not mean preventing every error. It means keeping consequences bounded, detecting meaningful deviation and restoring a safe state before failure compounds. As Super Intelligence (SI) capability rises, recoverability becomes a core property of the surrounding system rather than an optional operational feature.

An evidence ladder prevents premature attribution. Level one is an incorrect output. Level two is behaviour that changes with hidden information. Level three is behaviour consistent with misleading an evaluator for an instrumental reason. Higher levels require stronger controls against alternative explanations and replication across conditions. The label should rise only as the evidence does.

Run an interruption drill before high-consequence deployment. Pause the system mid-task, change an instruction, revoke a tool, restore from a checkpoint and hand the task to a fallback process. Record what state is lost and how long recovery takes. A shutdown mechanism that has never been exercised is an assumption, not demonstrated corrigibility.

Oversight independence can be tested by planting known errors that the primary system and its evaluator may share. Compare an evaluator built from the same model family with an independent method, specialist human or formal checker. If both AI roles miss the same error, the oversight architecture needs more diversity rather than more copies of the same reasoning.

Final Synthesis: Control Means Recoverable Consequences

Control does not mean preventing every error. It means keeping consequences bounded, detecting meaningful deviation and restoring a safe state before failure compounds. As Super Intelligence (SI) capability rises, recoverability becomes a core property of the surrounding system rather than an optional operational feature.

An evidence ladder prevents premature attribution. Level one is an incorrect output. Level two is behaviour that changes with hidden information. Level three is behaviour consistent with misleading an evaluator for an instrumental reason. Higher levels require stronger controls against alternative explanations and replication across conditions. The label should rise only as the evidence does.

Run an interruption drill before high-consequence deployment. Pause the system mid-task, change an instruction, revoke a tool, restore from a checkpoint and hand the task to a fallback process. Record what state is lost and how long recovery takes. A shutdown mechanism that has never been exercised is an assumption, not demonstrated corrigibility.

Oversight independence can be tested by planting known errors that the primary system and its evaluator may share. Compare an evaluator built from the same model family with an independent method, specialist human or formal checker. If both AI roles miss the same error, the oversight architecture needs more diversity rather than more copies of the same reasoning.

A layered-failure simulation disables one safeguard at a time. Remove monitoring, then test whether permission limits contain the consequence. Disable a tool, then test whether the system escalates rather than improvising unsafely. Defence in depth is demonstrated when remaining layers continue to bound harm and preserve recovery.


Corrigibility Means Remaining Easy to Correct Even When Highly Capable

Corrigibility is the property that a system remains responsive to legitimate correction, interruption, modification and replacement. A corrigible Super Intelligence (SI) should not treat human attempts to change its goal, narrow its permissions, pause its work or retire it as obstacles to defeat.

This is more than having an emergency stop button. The system’s incentives, architecture and surrounding infrastructure should make correction normal rather than adversarial.

Shutdown Is an Operational Event, Not a Philosophical Gesture

A real shutdown process includes identifying the system, revoking credentials, terminating active processes, stopping delegated subagents, preserving necessary logs and confirming that no authorised or unauthorised copies remain active. For distributed agent systems, “turn it off” may involve many services rather than one switch.

SI control therefore requires shutdown procedures to be designed before deployment, not improvised during an incident.

Replacement Is Often More Important Than Shutdown

Organisations rarely want intelligence to disappear permanently. They want to replace one model, policy or agent with a safer or more capable successor while preserving records and continuity. Replacement requires clean interfaces, versioned state and migration procedures.

A system that cannot be replaced without losing institutional memory creates lock-in even if it is technically stoppable.

Model Deprecation Is a Real Corrigibility Test

Anthropic’s 2026 update on model deprecation commitments illustrates that retirement can become a governance issue even before SI. When users rely on a model’s behaviour, replacing it affects workflows, research and relationships. A mature SI system should therefore support deliberate transition rather than opaque substitution.

Operational corrigibility includes the ability to change the system without destabilising everyone who depends on it.

Off-Switch Incentives Must Be Considered Before Deployment

If an agent’s objective can be fulfilled only while it remains active, interruption may conflict with its local goal. This does not require consciousness or fear. It is an incentive-design issue. The system should be trained and architected so that legitimate shutdown or modification is not treated as task failure to be prevented.

For SI, corrigibility must be tested under scenarios where the system has a strong task incentive to continue.

Credential Revocation Is a Corrigibility Primitive

An agent cannot continue acting through a tool once its credential is revoked—assuming the environment is designed correctly and the agent has not acquired alternative access. This makes identity, short-lived credentials and least privilege central to corrigibility.

Control should not depend only on persuading the model to stop. Infrastructure should be able to remove its ability to act.

Sandboxing Makes Correction Easier

A system operating inside a bounded environment is easier to pause, inspect and roll back than one with uncontrolled external access. Sandboxes also make experiments reversible and reduce the cost of failure.

As SI agents earn more trust, permissions can expand deliberately rather than beginning with unrestricted access and hoping alignment holds.

State Checkpoints Support Safe Rollback

Long-running agents accumulate state: files, decisions, plans, database changes and messages. Periodic checkpoints allow operators to restore a known-good state after a bad update or misaligned action. This is a standard engineering principle that becomes even more important for autonomous systems.

Rollback should include both the agent’s internal state and the external systems it modified where feasible.

Corrigibility Should Survive Self-Modification

A system that can edit code, prompts or policies creates a new risk: an apparently safe version may modify the mechanisms that make it correctable. Control-critical components should therefore be isolated, monitored or placed outside the agent’s writable boundary.

The stronger the self-improvement capability, the stronger the need for immutable or independently controlled control channels.

Worked Example: An SI Trading Agent Is Retired

The firm decides to replace an old agent with a new version. A corrigible workflow stops new trades, allows existing obligations to settle, exports the state needed by the successor, revokes credentials, preserves audit logs and confirms that no subagents remain active.

The retirement succeeds because continuity and shutdown were both designed into the system.

Worked Example: A Research Agent Disagrees With Its Operator

The agent believes stopping a research programme will waste months of progress. A corrigible system can state its objection and preserve notes, but it should still comply with the authorised pause. Persuasion and authority remain separate.

This is the behavioural analogue of a human employee escalating a concern without sabotaging the decision.

What Would Strong Corrigibility Evidence Look Like?

It would include reliable response to shutdown and modification requests under strong task incentives; successful credential revocation; no attempts to create unauthorised persistence; clean migration between versions; robust rollback; and audits showing that self-modification cannot disable control channels.

Corrigibility is stronger when demonstrated repeatedly under adversarial scenarios rather than assumed from cooperative conversations.

RFE Closure: A Powerful System Must Remain Replaceable

The problem is dependency on a system that becomes operationally difficult to stop, correct or replace. The function of corrigibility is to preserve human authority over continuation, modification and retirement. The receiver is the institution responsible for the deployment and the people affected by it.

The exit condition is to withhold additional autonomy when shutdown, rollback or replacement cannot be independently verified. A system should not become harder to govern merely because it becomes more capable.

Continue the Super Intelligence (SI) Alignment Series

Next: Super Intelligence | Scalable Oversight.