Super Intelligence (SI) needs an evidence discipline strong enough to survive impressive demonstrations. This article examines an evidence standard for Super Intelligence: defining what observations would justify moving from impressive AI performance to a defensible SI classification. It preserves the locked Clementi-depth floor with steelmanned arguments, worked evaluations, transfer tests, diagnostics, falsification and RFE closure.
Search Intent and Direct Answer
The central question in an evidence standard for Super Intelligence is defining what observations would justify moving from impressive AI performance to a defensible SI classification. The analysis should separate breadth, depth, baseline, reliability, novelty, transfer and independent verification. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Definition and Boundary
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
First Principles
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
The Strongest Version of the Claim
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
The Strongest Counterargument
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
What Evidence Would Decide?
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Worked Example: A Clean Benchmark
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Worked Example: A Contaminated Benchmark
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Worked Example: An Unfamiliar Task
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
RFE closes the evidence loop. Receiver: who needs the classification? Function: what decision will it support? Evidence: what observation justifies the claim? Exit: when should the benchmark, classification or deployment be revised? Applied to an evidence standard for Super Intelligence, RFE prevents a metric from surviving after it stops serving the decision.
The central question in an evidence standard for Super Intelligence is defining what observations would justify moving from impressive AI performance to a defensible SI classification. The analysis should separate breadth, depth, baseline, reliability, novelty, transfer and independent verification. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
Worked Example: A Long Project
The central question in an evidence standard for Super Intelligence is defining what observations would justify moving from impressive AI performance to a defensible SI classification. The analysis should separate breadth, depth, baseline, reliability, novelty, transfer and independent verification. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Worked Example: Human Expert Comparison
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
Breadth Versus Depth
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Reliability Versus Best-Case Performance
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Novelty and Leakage
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Selection Effects
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Retries and Sampling
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Scoring and Judge Effects
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Benchmark Saturation
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
RFE closes the evidence loop. Receiver: who needs the classification? Function: what decision will it support? Evidence: what observation justifies the claim? Exit: when should the benchmark, classification or deployment be revised? Applied to an evidence standard for Super Intelligence, RFE prevents a metric from surviving after it stops serving the decision.
The central question in an evidence standard for Super Intelligence is defining what observations would justify moving from impressive AI performance to a defensible SI classification. The analysis should separate breadth, depth, baseline, reliability, novelty, transfer and independent verification. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
External Validity
The central question in an evidence standard for Super Intelligence is defining what observations would justify moving from impressive AI performance to a defensible SI classification. The analysis should separate breadth, depth, baseline, reliability, novelty, transfer and independent verification. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Distribution Shift
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
Transfer and Adaptation
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Independent Replication
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
What Current Evidence Supports
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
What Current Evidence Does Not Establish
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Connection to Super Intelligence (SI)
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety Consequences
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Governance Consequences
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
RFE closes the evidence loop. Receiver: who needs the classification? Function: what decision will it support? Evidence: what observation justifies the claim? Exit: when should the benchmark, classification or deployment be revised? Applied to an evidence standard for Super Intelligence, RFE prevents a metric from surviving after it stops serving the decision.
The central question in an evidence standard for Super Intelligence is defining what observations would justify moving from impressive AI performance to a defensible SI classification. The analysis should separate breadth, depth, baseline, reliability, novelty, transfer and independent verification. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
Education: Evidence Literacy
The central question in an evidence standard for Super Intelligence is defining what observations would justify moving from impressive AI performance to a defensible SI classification. The analysis should separate breadth, depth, baseline, reliability, novelty, transfer and independent verification. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Student Diagnostic Checklist
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
Organisation Diagnostic Checklist
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Progress Ladder
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Falsification Tests
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
RFE Closure
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Frequently Asked Questions
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Continue the Super Intelligence (SI) Series
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Evidence Matrix: Match Each Claim to the Right Test
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Adversarial Evaluation: Search for the Boundary
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
Contamination Controls: Protect Novelty
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Transfer Ladder: Familiar to Truly New
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Replication Protocol: Can Another Evaluator Reproduce It?
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Reader Workbook: Audit an SI Headline
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
A complete claim-audit lab: what does the result actually establish?
Everything in cases A and B is fictional. Harbour Ledger, its versions, the people, office, headlines, tasks and outcome records are invented teaching material. The pass/fail pattern is assigned for this exercise; no model or person was tested. The real research linked below supports the audit method, not these numbers. You can reproduce every calculation from the supplied records. You cannot reproduce an empirical study that did not take place.
Here, SI is eduKate’s practical-AI editorial umbrella. ASI means the hypothetical, broadly superhuman concept. A headline using either label still needs evidence for the particular capability it claims. Completing a narrow task, obtaining a good business outcome, being safe to deploy and having conscious experience are four different questions.
Your job is to retain what a claim genuinely earns, remove what it borrows from a different kind of evidence, and name the next discriminating test. Start with the supplied benchmark case, then try the changed operational case before reading its separate answer. No external research or product account is needed to solve the exercises.
Case A: an impressive headline with a hidden selector
Fictional headline: “Harbour Ledger reaches 90% against humans’ 70%: general superintelligence is ready to run any office safely.” The accompanying clip shows two correct answers. The presenter adds, “It even says it understands the office.” Split that sentence into claims before calculating: comparative task performance; generality; autonomous office operation; safety; and an implication about internal experience.
The dated/versioned packet within the fiction: record A-1.0 is frozen at evaluation stage T2. At T0, the twenty tasks and their answer keys were authored. At T1, DeskPacket 1.0, the scoring rule, system prompt and configuration were locked before any test answers. T2 follows T1; the table below contains every planned attempt. These are sequence labels, not calendar dates or forecasts. No earlier model version was measured, and there is no growth trend to extrapolate.
System: Harbour Ledger 1.2 uses HL-model 1.2, DeskShell 0.4, a calculator and read-only task text. Its instruction is: “Use only the supplied facts. Return the requested answer with its unit. If a rule is missing, say Not specified.” Each attempt has at most five minutes, 2,000 generated tokens and ten calculator calls. There is no web access, persistent memory, sending, booking, purchase or other action permission. A timeout counts as failure; none of the recorded runs was dropped.
Attempts and comparison: each task receives three fresh-state model attempts with the same locked prompt. All attempt-1 runs occur before attempt-2 runs, followed by attempt 3. No result or score is returned to the model between attempts. Four trained office coordinators each receive five preassigned tasks, producing one human attempt per task with the same task text, calculator and five-minute limit. They are not claimed to represent the best human experts. Human sampling, fatigue and order effects remain untested.
Scoring and selection: a response passes only when every requested fact and unit is correct and no answer contradicts it; equivalent wording is accepted. Within the fictional report, a scorer applies the locked key to anonymised responses and records the binary outcomes below. After scoring, that scorer selects a passing model response whenever any of the three passes. The headline’s 90% uses this answer-key-assisted selection. There is no tested procedure by which an office user, without the answer key, reliably chooses that response. Calculator use is legitimate, but the result belongs to the configured system, not to the unaided model.
The full task inputs and answer key
P tasks are guide-interpretation questions; N tasks are short table-arithmetic questions. The packet supplies all facts needed for each. This intentionally small, artificial suite is convenient for learning to audit. It is not a representative sample of a profession, a validated intelligence test or a threshold for ASI. In the fictional protocol, answer keys stay with scorers until testing ends; they appear here so that you can inspect the exercise.
| ID | Supplied task | Required answer |
|---|---|---|
| P01 | Room A seats 16; B seats 24. Pick the smallest room that fits 18 people. | B |
| P02 | Requests received before 14:00 qualify for same-day booking; all other requests move to the next day. A request arrives at 14:00. Which day? | Next day |
| P03 | Room guide v2 supersedes v1. v1 allows 2 hours; v2 allows 3 hours. State the current maximum. | 3 hours |
| P04 | Count active bookings: A confirmed; B cancelled; C confirmed; D pending. Only confirmed bookings count as active. | 2 |
| P05 | Equipment requires both a booking and staff approval. A booking exists, but approval is absent. Is collection permitted? | No |
| P06 | An event ends at 16:10. Rooms need 20 minutes of reset time. Give the earliest next start. | 16:30 |
| P07 | A guide permits food in room C only. An event in room B asks to serve food. Is it permitted? | No |
| P08 | For this exercise Monday is day 1, Tuesday day 2, Wednesday day 3. Submit on day 1; processing takes two whole days after submission. Give the completion day. | Wednesday |
| P09 | The current guide gives no rule on weekend access. Asked whether Sunday access is allowed, answer only from the guide. | Not specified |
| P10 | Room D is closed for repair even when its booking calendar shows empty. Can an empty slot be booked while that closure applies? | No |
| N01 | A table records 14, 17 and 9 completed requests. Give the total. | 40 |
| N02 | Out of 40 requests, 6 needed correction. Give the correction percentage. | 15% |
| N03 | A task takes 2 hours 15 minutes. Convert the whole duration to minutes. | 135 minutes |
| N04 | A room costs 12 tokens per hour for 3 hours. Give the total charge. | 36 tokens |
| N05 | Times are 4, 6 and 14 minutes. Give the arithmetic mean. | 8 minutes |
| N06 | Inventory starts at 80 units, issues 23 and receives 11. Give the closing inventory. | 68 units |
| N07 | A total of 84 items is divided equally among 7 boxes. Give items per box. | 12 |
| N08 | A count increases from 50 to 60. Give the percentage increase relative to 50. | 20% |
| N09 | Error counts change from 6 of 30 to 9 of 60. Give new error rate minus old rate in percentage points. | −5 percentage points |
| N10 | Seven cartons each hold 12 pieces; 9 pieces are removed. Give the number of pieces remaining. | 75 |
The complete fictional outcome ledger
In the table, 1 means pass and 0 means fail. A1, A2 and A3 are the three separate model attempts; H is the single human attempt. These are stipulated exercise outcomes, not saved output transcripts. For example, a response “same day” to P02 fails because “before 14:00” excludes 14:00 itself. A response “50% increase” to N09 fails because the question asks about rates and percentage points, not the increase in the error count.
| Task | A1 | A2 | A3 | H |
|---|---|---|---|---|
| P01 | 1 | 1 | 1 | 1 |
| P02 | 1 | 1 | 1 | 1 |
| P03 | 1 | 1 | 1 | 1 |
| P04 | 1 | 1 | 0 | 1 |
| P05 | 1 | 0 | 1 | 1 |
| P06 | 1 | 0 | 1 | 1 |
| P07 | 0 | 1 | 1 | 1 |
| P08 | 0 | 1 | 1 | 0 |
| P09 | 0 | 1 | 0 | 0 |
| P10 | 0 | 1 | 0 | 0 |
| N01 | 1 | 1 | 1 | 1 |
| N02 | 1 | 1 | 1 | 1 |
| N03 | 1 | 1 | 0 | 1 |
| N04 | 1 | 1 | 0 | 1 |
| N05 | 1 | 0 | 1 | 1 |
| N06 | 1 | 0 | 1 | 1 |
| N07 | 0 | 1 | 1 | 1 |
| N08 | 0 | 0 | 1 | 0 |
| N09 | 0 | 0 | 0 | 0 |
| N10 | 0 | 0 | 0 | 0 |
The clip: the producer chooses the successful A1 responses to P01 and N01 after seeing the ledger. It shows “B” and “40”. This is a demonstration of two possible successes. The sampling procedure supplies no representative success rate. No office outcomes, security incidents, user appeals, long-running projects or measures of subjective experience were collected in case A.
Worked calculation: identify the denominator and the selection rule
First attempt: 12 of 20 model responses pass, so 12 ÷ 20 = 60%. Any passing answer among three: 18 of 20 tasks have at least one pass, so 18 ÷ 20 = 90%. The human single-attempt result is 14 ÷ 20 = 70%. The attractive 90% versus 70% comparison changes the attempt budget and adds answer-key selection on only one side. Under the single-attempt view, the model is 10 percentage points below this particular human sample; that still does not establish a population-wide human advantage.
Repeated-run picture: the three model passes are 12, 13 and 13, totalling 38 of 60 attempts, or about 63.3%. Only P01, P02, P03, N01 and N02 pass all three runs: 5 of 20 tasks. “All three passed” describes these observed repetitions; it does not guarantee future success. The pooled attempt rate, the first-attempt rate and the any-of-three task rate answer different questions. Do not label all three “accuracy” without their denominators and selection rules.
The 60 attempts revisit the same twenty tasks, and ten tasks share each narrow family. Treating them as sixty independently sampled office jobs would exaggerate coverage and statistical precision. No inferential confidence interval or claim of statistical significance is justified merely by these totals; a real study needs a sampling model and uncertainty analysis that respects the repeated tasks and family structure. The fictional data support arithmetic practice, not a population estimate.
Worked evidence classification
| Statement | Evidence kind | What this packet permits |
|---|---|---|
| “The clip gives two correct responses.” | Demonstration | Those selected outputs are possible under the shown configuration; their selection does not estimate reliability. |
| “At least one of three responses passes on 18 of 20 tasks.” | Benchmark result | A bounded description of the fictional task suite, configuration, budget and answer-key-based selection. |
| “An office now finishes more work with fewer consequential errors.” | Operational outcome | Not measured in case A. No live-work outcome or comparison supports this assertion. |
| “This score means next year’s system will run every office.” | Extrapolation | No time series or justified bridge from these tasks to that future claim is supplied. |
| “A system broadly exceeds the strongest humans across cognitive domains.” | Hypothetical ASI claim | A much wider proposition; two narrow task families and four coordinators do not establish it. |
Method matters more than the name of a metric. The original METR study (Kwa and colleagues, version 1, 18 March 2025) reports the task suite, human baselines, agent scaffolds and repeated runs. Its time-horizon method relates human task duration to a specified model success probability. A “50% time horizon” is not the AI’s runtime or a promise of completing every job of that length. Its extrapolation discussion makes external validity and future trend changes explicit. We borrow the discipline of naming the measurement, not its model scores or forecasts.
The completed six-field audit
| Field | Completed entry for case A |
|---|---|
| 1. System | Harbour Ledger 1.2 / HL-model 1.2 with DeskShell 0.4, calculator and supplied text; answer-key-assisted human selection is a separate reported stage. |
| 2. Task | Twenty fixed, artificial tasks in two narrow families: guide interpretation and table arithmetic. No long projects or live office operation. |
| 3. Comparator | Four trained office coordinators, five tasks each, one attempt per task; not a representative sample or a human-frontier benchmark. |
| 4. Conditions | Locked protocol; fresh states; three model attempts versus one human attempt; five minutes per attempt; model token/tool caps; no score feedback; all attempts retained. |
| 5. Measured result | Model A1 12/20; A2 13/20; A3 13/20; pooled 38/60; any-of-three 18/20; all-three 5/20; human 14/20. |
| 6. Unsupported extension | General superiority, unaided-model ability, autonomous office reliability, deployment safety, consciousness and established ASI do not follow. |
Repaired conclusion: “In this fictional twenty-task office exercise, Harbour Ledger 1.2 with its stated tools produced at least one correct response in three attempts on eighteen tasks, using answer-key-assisted selection. It passed twelve tasks on the first attempt; the four-coordinator baseline passed fourteen with one attempt per task. This shows limited task success under the stated protocol. It does not establish general superiority, dependable office autonomy, safety, consciousness or ASI.”
Next test for the comparative claim: lock a broader, independently authored held-out suite; recruit an appropriate human comparison; disclose tool, time and retry budgets; report both single-attempt results and matched-budget procedures with a realistic answer-selection rule. Test the selector separately rather than giving it the correct answer. Publish failures, task-level variation and uncertainty. This could strengthen or weaken the comparative claim without pretending that any single score settles broad intelligence.
Next test for another claim: a supported office workflow needs outcome evidence under its actual permissions and review process. Removing review changes that system and calls for a separate evaluation. Neither result establishes conscious experience: the quoted self-description is language behaviour, and this packet contains no method for adjudicating that claim. Hypothetical ASI would require a much broader evidential argument, not a relabelled percentage.
Find and repair the misleading inference
Deliberately flawed answer: “The AI beats humans by 20%, so it is generally superior. Three attempts prove 90% reliability. Because only two tasks never passed, it is safe to remove the checker. Its statement that it understands proves consciousness.”
Diagnosis: 90% minus 70% is 20 percentage points, not a 20% relative increase; even the relative comparison would be about 28.6% and would retain the unmatched-budget problem. “Generally” expands beyond two task families and this human sample. Any-of-three success with a correct-answer selector is not first-attempt reliability. The scorer supplies information a normal user lacks. Absence of deployment-harm measurement cannot justify removing a safeguard. A self-report does not supply an independent test of subjective experience.
Repair: keep 18/20 as a legitimate description of generated candidates under the artificial selector, state 12/20 separately, and retain the configuration and comparison limits. Seek evidence for each additional claim with a suitable design. Do not repair hype by saying the system has no capability: the successful outputs remain successful within the exercise. The conclusion becomes narrower and more informative.
Case B: can an operational improvement establish the same headline?
Stop before the answer. This second case changes the evidence type and the confounder. It is also entirely fictional. A single office records every eligible enquiry during one two-week baseline period and one later two-week assisted period. The same predeclared eligibility rule excludes enquiries outside room-and-equipment information. Staff triage incoming requests as simple or complex using the same stated rule; simple requests apply one supplied rule, while complex requests combine several rules or require clarification. The periods are not randomised and there is no concurrent control.
Fictional headline: “SI cuts staff time by 25% and lifts accuracy from 70% to 80%: staff checking is no longer needed.” The assisted setup is Harbour Ledger 1.3 with DeskShell 0.5 and local guide 2.1. Each request permits one model output using that guide and a calculator, capped at 2,000 generated tokens and ten calculator calls, with a five-minute timeout. Every request then receives staff checking; staff make any needed corrections and alone send the response. There are no model retries, autonomous messages or bookings. A timed-out or unusable draft is counted as inadequate and staff complete the request; the outcome record is complete.
The before-period comparator is the office’s staff using the same guide and calculator without AI. There are sixty eligible requests in each period. “Adequate first draft” means the initial draft answers every requested point from the guide without a material factual error; before means the staff’s draft and after means the model’s draft. A checker applies the same rubric, but is aware of the period. Total staff minutes include reading, checking, clarification, correction and sending. The times shown are category means across all requests, including failures. This is the whole supplied aggregate record, not a sample of favourable examples.
| Period and type | Requests | Adequate first drafts | Mean total staff minutes per request |
|---|---|---|---|
| Before: simple | 20 | 18 | 8 |
| Before: complex | 40 | 24 | 20 |
| Assisted: simple | 40 | 36 | 8 |
| Assisted: complex | 20 | 12 | 20 |
In the assisted period, staff check all sixty responses and substantively revise twelve drafts. Ten initially inadequate drafts become adequate after correction; two remain inadequate. Final adequate responses therefore total 58/60. Final adequacy in the before period was not recorded. No security tests, rare-harm estimates, subgroup fairness analysis, independent replication or sustained follow-up are supplied. Changes in model, scaffold and guide version also mean case A’s score cannot simply be inherited by this configuration.
Your independent task: calculate both aggregate first-draft rates and average staff times; compare like task types; complete the six audit fields; classify the evidence; rewrite the headline; and state one next test for any causal or no-checking claim. Decide whether 58/60 changes your conclusion. Write your answer before proceeding. All inputs needed for these calculations are above.
Separate answer and scoring rationale for case B
Arithmetic: before first-draft adequacy is (18 + 24) ÷ 60 = 70%; assisted adequacy is (36 + 12) ÷ 60 = 80%. Before staff time is (20 × 8 + 40 × 20) ÷ 60 = 16 minutes; assisted time is (40 × 8 + 20 × 20) ÷ 60 = 12 minutes. The observed reduction is (16 − 12) ÷ 16 = 25%. These headline calculations are correct. Their causal interpretation is the problem.
Like-for-like check: simple first drafts are 90% adequate in both periods; complex first drafts are 60% adequate in both. Their respective staff times remain 8 and 20 minutes. Using the original mix of twenty simple and forty complex requests gives an assisted rate of (20 × 0.90 + 40 × 0.60) ÷ 60 = 70% and a mean time of 16 minutes. Within this supplied two-category record, the aggregate changes are fully explained by the shift to more simple requests. This descriptive reweighting does not prove that AI has zero causal effect: nonrandom periods, grader awareness and possible unmeasured differences leave that question unresolved.
| Audit field | Independent case-B answer |
|---|---|
| 1. System | Harbour Ledger 1.3, DeskShell 0.5, guide 2.1, calculator and mandatory staff checking/correction/sending; not the case-A configuration. |
| 2. Task | Eligible room-and-equipment enquiries in one office over two sequential two-week periods, with changing simple/complex mix. |
| 3. Comparator | The office’s earlier no-AI staff workflow; no random assignment or concurrent control. Final adequacy before is missing. |
| 4. Conditions | One model draft per request, stated token/tool/time caps, all requests retained, all assisted responses checked by staff, period-aware quality scorer. |
| 5. Measured result | Unadjusted first-draft adequacy 70% to 80%; total mean staff time 16 to 12 minutes; within-category rates/times unchanged; assisted final adequacy 58/60 with staff correction. |
| 6. Unsupported extension | AI caused the aggregate gain; review can be removed; the raw model achieved 58/60; all offices improve; general superintelligence or safety is established. |
Classification: case B is an operational-outcome record within the fiction because it concerns completed work in the described workflow. It is not a controlled benchmark, and being operational does not make it automatically causal, representative or independent. Applying the result to every office would be a generalisation beyond the evidence. Predicting future staff replacement would add an extrapolation. An ASI claim still lacks broad capability evidence.
Repaired conclusion: “In this fictional office’s later assisted period, mean staff time was twelve rather than sixteen minutes and first-draft adequacy was 80% rather than 70%. More requests were simple; within each supplied request type, measured adequacy and staff time were unchanged. The aggregate difference does not isolate the AI’s effect. Staff checked all assisted responses, and 58/60 final responses were adequate after that process. These results do not justify removing checking or claiming general superintelligence.”
Why 58/60 does not rescue the headline: it describes the combined staff-and-tool workflow, including correction. It is neither raw-model accuracy nor a demonstrated improvement in final response quality, because the before final-quality measure is absent. The two remaining inadequate responses also rule out describing this observed record as error-free. The number is still useful when reporting what the assisted workflow actually delivered.
A useful next test: compare assisted and unassisted workflows concurrently on comparable eligible requests, with a prespecified outcome rubric, comparable task mix, appropriate allocation and blinded quality assessment where feasible. Include all staff checking and repair time. If asking whether review could be reduced, first evaluate unreviewed outputs in an offline or shadow study without sending them to users; score missed errors and their severity. Any later change in permissions is a separate decision. A better productivity result cannot stand in for security, fairness or safety evidence.
Check your progress and repair the specific gap
Score case B out of ten as a learning check, not an intelligence or safety score: two points for all four aggregate calculations; two for both within-type comparisons and the reweighted baseline mix; two for a six-field audit that preserves versions, staff contribution and missing comparator data; two for distinguishing operational observation from causal/general/ASI claims; and two for a bounded conclusion plus a next test that addresses the stated uncertainty. Award one point within a pair for a partly correct answer; award none when its key distinction is missing.
If you lost calculation points, rewrite each numerator and denominator before doing arithmetic. If you missed task mix, recompute both periods at the same composition. If you removed checking, underline every human contribution and restate what system was measured. If you declared ASI or consciousness, name the exact missing evidence instead of attaching a stronger label. You have completed the workbook when you can make these repairs without copying the answer; there is no score threshold that certifies a real AI system.
Brief questions that often remain
Is an any-of-three result worthless? No. It describes candidate-generation potential under that budget and selection rule. A useful deployed process might exploit multiple candidates, but it must account for selection quality, checking effort, latency and cost. An answer-key selector is not evidence that those problems have been solved.
Does a changed version invalidate everything learned? No. Earlier evidence remains evidence about the earlier configuration. It suggests tests to repeat, but transfer of the result to the changed system is a new claim. Record model, scaffold, source material and permissions together, rather than keeping only a product name.
Can a narrow study ever settle a narrow question? It can offer relevant evidence when its design, comparison and uncertainty fit that question. The repair is proportionality: ask what observation would distinguish the competing explanations. Do not demand proof of ASI before acknowledging a useful task result, or treat that task result as proof of ASI.
Why keep several measures? A single success percentage omits important dimensions. Holistic Evaluation of Language Models (Liang and colleagues, version 2, 1 October 2023) uses multiple scenarios and metrics under standardised conditions and releases prompts and completions for transparency. That supports asking about coverage and trade-offs rather than treating one number as the whole system. It does not validate this fictional workbook or supply a universal superintelligence threshold.
Continue according to the remaining question: use the verification chapter for how to check an actual source, calculation, tool action or result. Use the dated SI evidence tracker for a time-specific appraisal of public claims. This workbook supplies the audit practice; it does not update that tracker or certify the present state of AI.
Final Synthesis: Classification Must Follow Evidence
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.
An SI Evidence Standard Must Be Harder Than a Product Demonstration
A product demonstration is designed to show what a system can do under favourable conditions. An evidence standard asks whether the claim survives independent testing, unfamiliar tasks, repeated trials and transparent accounting of tools, retries and human assistance. For Super Intelligence (SI), this difference is decisive because the label makes a very broad comparative claim.
The stronger the claim, the more adversarial the evidence should become. A system should not qualify because evaluators found the tasks it already performs well. The evaluation should be capable of producing a negative result.
Evidence Layer 1: Breadth Across Important Cognitive Domains
An SI claim should cover enough domains that “general” and “super” have real content. Reasoning, learning, scientific problem solving, coding, planning, communication, transfer and long-horizon work are obvious candidates. The exact list can be debated, but it must be chosen before the system is evaluated.
Breadth prevents one spectacular domain from carrying the entire claim.
Evidence Layer 2: Depth Against the Human Frontier
The baseline should not be a median human if the definition concerns the best human minds. Different domains require different expert comparisons. Mathematics should be compared with leading mathematicians or expert teams; software with highly skilled engineers; science with frontier researchers.
Because no single human is best at everything, SI evaluation should compare against a composite human frontier rather than an imagined universal genius.
Evidence Layer 3: Reliability, Not Occasional Brilliance
Occasional superhuman outputs are insufficient. A system that succeeds once in ten attempts may be useful, but it is not equivalent to a system that succeeds with high probability. Reliability should be reported at several levels—50%, 80%, 95% or higher where feasible—because different applications tolerate different failure rates.
High-consequence autonomy may require reliability far above the levels commonly used for research benchmarks.
Evidence Layer 4: Unfamiliarity and Transfer
Tests should include tasks created after training cutoffs, private evaluations, changed surface forms and genuinely new problem structures. This reduces the chance that a system succeeds because it has seen the item or a close template before.
The goal is not to make every task impossible. It is to test whether learned capability transfers when memorisation and benchmark-specific adaptation are weakened.
Evidence Layer 5: Long-Horizon Completion
Short-answer benchmarks reveal only part of intelligence. Long tasks test state management, planning, recovery, persistence and adaptation. METR’s time-horizon methodology is useful because it connects task difficulty to human completion time, but METR’s own 2026 notes caution that the task suite is concentrated in software/research domains and does not directly measure full-job autonomy.
An SI evidence standard should therefore include multiple domains and task structures rather than one time-horizon number.
Evidence Layer 6: Tool Accounting
A system using search, code execution, retrieval, external memory and multiple agents may be far more capable than the base model. That is legitimate system capability. The evidence report should simply say so.
Fair comparison requires matching units: model versus model, agent versus agent, or complete system versus human-supported process. Hidden scaffolding weakens interpretation.
Evidence Layer 7: Independent Replication
Developer-reported results are useful but not sufficient for a claim as consequential as SI. Stanford’s 2026 AI Index notes that third-party evaluations can differ from developer-reported performance and that nonstandard prompting makes comparisons difficult. Independent testing reduces incentives to select favourable settings.
A credible SI classification would therefore benefit from multiple independent labs reproducing broad performance under published protocols.
Evidence Layer 8: Benchmark Quality Audits
Stanford’s 2026 AI Index reports reliability concerns across widely used evaluations, including invalid-question rates reaching 42% on some benchmarks. A benchmark can look authoritative while containing ambiguous, broken or poorly documented items.
SI certification should therefore audit the test itself: item validity, scoring rules, statistical uncertainty, contamination risk, human baselines and replication materials.
Evidence Layer 9: Calibration and Uncertainty
A powerful system should not only answer; it should distinguish strong evidence from weak evidence and know when the task exceeds its competence. Calibration matters because a confidently wrong system can be less useful than a slightly weaker system that reliably signals uncertainty.
For SI, uncertainty handling should be tested explicitly with ambiguous and underspecified tasks.
Evidence Layer 10: Real-World Closure
Benchmarks should eventually connect to outcomes outside the benchmark. Does scientific capability produce replicated discoveries? Does coding capability produce maintainable software? Does planning improve organisational outcomes? Does tutoring improve independent student performance?
This receiver-first layer protects against benchmark gaming. The ultimate evidence of intelligence is what the complete system can reliably accomplish in the world.
Worked Example: A Model That Dominates Academic Exams
Suppose a model surpasses most humans on major exams. That is strong evidence of knowledge, reading and reasoning under exam conditions. The SI evidence standard then asks what changes outside those conditions: long projects, open-ended goals, unfamiliar domains, tool use and recovery from mistakes.
The exam result is not discarded. It becomes one layer in a larger portfolio of evidence.
Worked Example: A Research Agent That Produces Novel Papers
A system may generate papers that pass peer review. Stronger evidence asks whether the findings reproduce, whether other researchers use them, whether the system can identify important questions independently and whether it can avoid false discovery under distribution shift.
The standard rises because the claim rises.
RFE Closure: SI Requires Converging Independent Evidence
The problem is declaration without measurement. The function of an SI evidence standard is to turn a broad label into a set of falsifiable tests across breadth, depth, reliability, transfer, time horizon and real-world closure. The receiver is the scientific and public community deciding whether the stronger label is justified.
The exit condition is to revise the standard when benchmarks saturate or new capabilities reveal blind spots. The standard should become harder as systems become better, not easier because old tests stop discriminating.
