Super Intelligence (SI) needs an evidence discipline strong enough to survive impressive demonstrations. This article examines generalisation and transfer: testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. It preserves the locked Clementi-depth floor with steelmanned arguments, worked evaluations, transfer tests, diagnostics, falsification and RFE closure.
Search Intent and Direct Answer
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Definition and Boundary
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
First Principles
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
The Strongest Version of the Claim
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
The Strongest Counterargument
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
What Evidence Would Decide?
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Worked Example: A Clean Benchmark
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Worked Example: A Contaminated Benchmark
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Worked Example: An Unfamiliar Task
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
RFE closes the evidence loop. Receiver: who needs the classification? Function: what decision will it support? Evidence: what observation justifies the claim? Exit: when should the benchmark, classification or deployment be revised? Applied to generalisation and transfer, RFE prevents a metric from surviving after it stops serving the decision.
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
Worked Example: A Long Project
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Worked Example: Human Expert Comparison
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
Breadth Versus Depth
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Reliability Versus Best-Case Performance
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Novelty and Leakage
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Selection Effects
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Retries and Sampling
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Scoring and Judge Effects
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Benchmark Saturation
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
RFE closes the evidence loop. Receiver: who needs the classification? Function: what decision will it support? Evidence: what observation justifies the claim? Exit: when should the benchmark, classification or deployment be revised? Applied to generalisation and transfer, RFE prevents a metric from surviving after it stops serving the decision.
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
External Validity
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Distribution Shift
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
Transfer and Adaptation
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Independent Replication
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
What Current Evidence Supports
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
What Current Evidence Does Not Establish
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Connection to Super Intelligence (SI)
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety Consequences
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Governance Consequences
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
RFE closes the evidence loop. Receiver: who needs the classification? Function: what decision will it support? Evidence: what observation justifies the claim? Exit: when should the benchmark, classification or deployment be revised? Applied to generalisation and transfer, RFE prevents a metric from surviving after it stops serving the decision.
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
Education: Evidence Literacy
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Student Diagnostic Checklist
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
Organisation Diagnostic Checklist
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Progress Ladder
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Falsification Tests
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
RFE Closure
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Frequently Asked Questions
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Continue the Super Intelligence (SI) Series
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Evidence Matrix: Match Each Claim to the Right Test
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Adversarial Evaluation: Search for the Boundary
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
Contamination Controls: Protect Novelty
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Transfer Ladder: Familiar to Truly New
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Replication Protocol: Can Another Evaluator Reproduce It?
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Reader Workbook: Audit an SI Headline
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
Final Synthesis: Classification Must Follow Evidence
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.