Super Intelligence (SI) needs an evidence discipline strong enough to survive impressive demonstrations. This article examines generalisation and transfer: testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. It preserves the locked Clementi-depth floor with steelmanned arguments, worked evaluations, transfer tests, diagnostics, falsification and RFE closure.
Search Intent and Direct Answer
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Definition and Boundary
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
First Principles
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
The Strongest Version of the Claim
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
The Strongest Counterargument
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
What Evidence Would Decide?
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Worked Example: A Clean Benchmark
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Worked Example: A Contaminated Benchmark
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Worked Example: An Unfamiliar Task
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
RFE closes the evidence loop. Receiver: who needs the classification? Function: what decision will it support? Evidence: what observation justifies the claim? Exit: when should the benchmark, classification or deployment be revised? Applied to generalisation and transfer, RFE prevents a metric from surviving after it stops serving the decision.
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
Worked Example: A Long Project
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Worked Example: Human Expert Comparison
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
Breadth Versus Depth
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Reliability Versus Best-Case Performance
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Novelty and Leakage
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Selection Effects
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Retries and Sampling
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Scoring and Judge Effects
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Benchmark Saturation
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
RFE closes the evidence loop. Receiver: who needs the classification? Function: what decision will it support? Evidence: what observation justifies the claim? Exit: when should the benchmark, classification or deployment be revised? Applied to generalisation and transfer, RFE prevents a metric from surviving after it stops serving the decision.
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
External Validity
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Distribution Shift
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
Transfer and Adaptation
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Independent Replication
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
What Current Evidence Supports
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
What Current Evidence Does Not Establish
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Connection to Super Intelligence (SI)
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety Consequences
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Governance Consequences
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
RFE closes the evidence loop. Receiver: who needs the classification? Function: what decision will it support? Evidence: what observation justifies the claim? Exit: when should the benchmark, classification or deployment be revised? Applied to generalisation and transfer, RFE prevents a metric from surviving after it stops serving the decision.
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
Education: Evidence Literacy
The central question in generalisation and transfer is testing whether capability survives changes in surface form, domain, environment and task structure rather than only familiar conditions. The analysis should separate training distribution, novelty, structural similarity, transfer, robustness, adaptation and recovery. Each dimension can produce a different answer, and broad Super Intelligence (SI) claims require them to line up rather than relying on one favourable measurement.
The strongest argument should be stated before it is tested. Steelmanning matters because weak objections create false confidence. If a sceptical view says data quality, verification or physical resources impose a hard constraint, identify the mechanism and the observation that would show the constraint weakening. If an optimistic view predicts broad transfer, specify where transfer should appear.
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
Student Diagnostic Checklist
Evidence standards work best when chosen before seeing the result. Define the task set, human baseline, permitted tools, number of attempts, scoring rule and success threshold in advance. Pre-specification reduces the temptation to reinterpret an ambiguous outcome after the model performs.
A clean benchmark offers repeatability and a clear score. That is valuable. But cleanliness can remove the ambiguity, changing requirements, missing information and social context of real work. Treat the benchmark as one instrument in a battery rather than as the capability itself.
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
Organisation Diagnostic Checklist
Contamination changes what a score means. If test material or close variants were available during training, apparent generalisation may partly reflect familiarity. Perfect contamination detection is difficult for large training corpora, so evaluation design should include newly created tasks, private test sets or procedurally generated variants where appropriate.
An unfamiliar-task test changes the surface while preserving the underlying skill. Strong transfer means the system identifies the structure and adapts without needing examples that closely match the new case. Failure under small changes suggests the original capability was narrower than the headline implied.
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Progress Ladder
Long projects test a different property from isolated questions. Context must be maintained, plans revised, tools coordinated and errors repaired. METR’s time-horizon work is useful for studying longer tasks, while METR explicitly cautions that its task suites are concentrated in software, machine learning and cybersecurity and are not direct estimates of whole-job automation.
Human comparison requires a named baseline. Average adults, trained professionals, frontier experts and expert teams answer different questions. Give both sides appropriate tools and document time, retries and assistance. A fair comparison is not necessarily an unaided-human contest; it is a transparent comparison aligned with the claim.
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Falsification Tests
Best-case performance can hide reliability. Report the distribution across trials, not only the strongest sample. If a system needs many attempts and a human selector to obtain the showcased answer, that is part of the system configuration. The result can still be useful, but it supports a different claim from first-attempt dependable performance.
Selection effects arise when researchers choose tasks, examples or metrics after seeing where a model performs well. The antidote is representative sampling and held-out evaluation. Broad SI evidence should include domains that are inconvenient for the system, not only those where current architectures shine.
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
RFE Closure
Scoring can introduce its own model. Human graders disagree; automated judges can have biases; exact-match scoring can penalise valid alternatives. A strong evaluation checks whether the scoring method actually tracks the capability of interest and reports uncertainty when judgement is subjective.
Benchmark saturation is a success and a warning. When top systems cluster near the ceiling, the benchmark stops distinguishing frontier capability. Stanford’s 2026 AI Index highlights rapid saturation on several evaluations. The correct response is harder, more representative testing, not declaring that every broader problem has been solved.
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Frequently Asked Questions
External validity asks whether test performance predicts real-world performance. A medical exam is not a clinic; a coding problem is not a maintained production system; a mathematics benchmark is not a research career. Real work contains changing goals, accountability and consequences. Transfer from benchmark to deployment should be measured rather than assumed.
Independent replication matters because developers have more opportunities to tune systems to their own evaluations. External groups, private benchmarks and repeated studies reduce dependence on one measurement pipeline. SI is too broad a classification to rest on a single developer’s test suite.
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Continue the Super Intelligence (SI) Series
Current evidence supports substantial and rapidly improving AI capability across many domains. It also shows unevenness, reliability gaps and changing benchmark ceilings. That combination supports continued investigation of SI without establishing that broad SI already exists.
Safety consequences follow from measurement error. Overestimating capability can lead to brittle automation; underestimating it can leave institutions unprepared for misuse or rapid adoption. Accurate evaluation is therefore not merely academic—it determines which controls, permissions and fallback systems are appropriate.
Education should treat evidence literacy as a core SI skill. Students should be able to distinguish a demonstration from a representative evaluation, correlation from causation, a benchmark score from external validity, and a forecast from an observation. These habits generalise far beyond AI.
Evidence Matrix: Match Each Claim to the Right Test
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Adversarial Evaluation: Search for the Boundary
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
Contamination Controls: Protect Novelty
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Transfer Ladder: Familiar to Truly New
A transfer ladder begins with near transfer: same structure, changed surface. Next comes medium transfer: related domain, different representation. Far transfer asks whether the underlying capability applies in a substantially new context. Adaptation adds another level: can the system learn the new environment efficiently? Report where performance falls rather than compressing every rung into one generalisation score.
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Replication Protocol: Can Another Evaluator Reproduce It?
Replication means another evaluator can reconstruct the conditions and obtain a compatible conclusion. Publish task definitions, scoring rules, tool access, retry policy and human baseline where possible. When private tests are necessary, independent auditors can preserve secrecy while checking the claim. Broad classifications become more credible when they do not depend on one laboratory’s interpretation.
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Reader Workbook: Audit an SI Headline
The reader workbook takes a headline and rewrites it as six fields: system, task, comparator, conditions, measured result and unsupported extension. Circle any words such as “general,” “human-level,” “autonomous” or “superintelligent” that exceed the measured field. Then write the next test needed to justify the stronger word. This is a practical defence against both hype and reflexive dismissal.
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
Final Synthesis: Classification Must Follow Evidence
Classification must follow evidence rather than lead it. The label SI should be the compressed conclusion of a broad evaluation record, not the premise used to interpret every success. As systems improve, the threshold tests should become harder, more novel and more representative. The objective is a classification that remains meaningful even when individual benchmarks are rapidly saturated.
Build an evidence matrix with claims on one axis and tests on the other. A speed claim needs timed comparable work; a breadth claim needs diverse domains; a reliability claim needs repeated trials; a transfer claim needs changed conditions; a long-horizon claim needs extended tasks. Empty cells reveal where the conclusion is outrunning the evidence. The matrix prevents one benchmark from carrying every claim.
Adversarial evaluation deliberately searches for the boundary of competence. Change instructions, introduce irrelevant information, remove a familiar cue, increase task length and test cases where the correct response is uncertainty. The purpose is not to make the system fail for sport. It is to map where independent checking becomes necessary and whether failure is graceful or abrupt.
Novelty needs protection. Use newly authored questions, private datasets, procedural generation, timestamped material or other controls appropriate to the domain. No method eliminates contamination risk completely, but the evaluation should make simple memorisation or benchmark familiarity less plausible. Super Intelligence (SI) classification needs evidence that survives genuinely new material.
Generalisation Is the Bridge From Familiar Success to Real Intelligence
A model generalises when it performs well on relevant cases that are not simple reproductions of its training examples. Transfer is a stronger question: can capability learned in one setting help in another setting with changed surface features, goals, tools or domains? For Super Intelligence (SI), these properties matter more than memorising a large fraction of human knowledge.
A system that has seen nearly every common pattern can appear highly capable until the problem changes. SI requires evidence that competence survives change.
Interpolation and Extrapolation Should Be Distinguished
Interpolation asks the model to operate inside the range of patterns represented in training. Extrapolation asks it to handle cases outside that range. Most machine-learning systems are much more reliable at interpolation. Extrapolation is where hidden assumptions and brittle shortcuts become visible.
A strong transfer test deliberately moves one or more dimensions beyond the familiar training distribution.
Surface Transfer Is Weaker Than Structural Transfer
If a mathematics problem changes names and numbers but preserves the same template, success demonstrates some robustness but limited novelty. Structural transfer changes the relationships that determine the solution. The model must recognise which principles still apply and which no longer do.
For SI, evaluation should progressively move from surface variation to structural novelty.
Domain Transfer Tests Whether Abstractions Are Truly General
A system may learn planning principles in code and then apply them to laboratory scheduling, logistics or project management. Successful cross-domain transfer suggests the model has learned abstractions broader than task-specific patterns.
Failure is also informative. It reveals which capability was tied to the original representation or tool environment.
Distribution Shift Is the Normal Condition of Deployment
Real environments change. Users adopt new vocabulary, regulations change, software versions move, adversaries adapt and rare events occur. A model evaluated once on a static dataset will eventually face inputs from a different distribution.
Robust SI therefore needs monitoring and adaptation after deployment rather than a one-time declaration of capability.
Compositional Generalisation Tests Whether Known Pieces Can Be Recombined
A system may know concept A and concept B separately but fail when the task requires combining them in a new way. Compositional tests create unfamiliar combinations from familiar primitives. This is particularly important for language, mathematics, planning and tool use.
Human intelligence often appears powerful because people can recombine learned concepts flexibly. SI should be expected to do at least as well across important domains.
Tool Transfer Matters for Agentic Systems
An agent trained to use one API may fail when the interface changes even if the underlying goal is identical. Stronger agents infer the role of a tool from documentation, learn the new interface and continue. This is a practical form of transfer that directly affects workplace usefulness.
Evaluation should therefore include unseen tools, changed schemas and partial documentation rather than only familiar environments.
Long-Horizon Transfer Is Harder Than One-Step Transfer
A model may adapt to one unfamiliar instruction yet lose coherence across a long project. Long-horizon transfer requires the system to update plans, preserve state and recognise that earlier assumptions no longer fit the new environment.
METR’s 2026 time-horizon notes show how domain and task structure can change measured capability by orders of magnitude, reinforcing that transfer across task distributions should not be assumed.
Generalisation Can Be Hidden by Benchmark Saturation
When every frontier model scores near the top of a static benchmark, the test no longer reveals which model will handle unfamiliar problems better. Stanford’s 2026 AI Index describes this saturation problem across several modern evaluations.
Fresh benchmarks with private tasks and changed distributions become more important precisely because the models are improving.
Worked Example: A Student Learns Algebra Versus Memorises Templates
A student who memorises ten worked examples may perform well on nearly identical questions. A student who understands the underlying relationship can solve a changed problem with new numbers, wording and constraints. AI generalisation should be tested the same way.
The receiver-focused question is whether the learned capability survives when the examiner removes the familiar cues.
Worked Example: A Coding Agent Moves to a New Repository
An agent performs well on repositories using a familiar framework. It is then given a project in a different language with an unfamiliar build system and sparse documentation. Success requires learning local conventions, identifying dependencies and updating its plan.
This is stronger evidence of software-engineering generalisation than another task inside the original benchmark ecosystem.
Worked Example: Scientific Transfer
An AI develops a causal modelling technique on one biological dataset. Researchers then test whether the method improves inference on a different organism, measurement technology and experimental design. If the gain persists, the evidence shifts from benchmark-specific performance toward a reusable scientific method.
Transfer becomes strongest when the new setting was not chosen because it was known to favour the technique.
Transfer Can Be Negative
Knowledge from one domain can hurt another if the model applies the wrong analogy. This is negative transfer. A pattern learned from ordinary conversation may be inappropriate in legal interpretation; a shortcut learned in one visual dataset may fail in another.
A capable SI system should not only transfer; it should recognise when transfer is unsafe.
Adaptation Speed Is Part of General Intelligence
When a system encounters a new environment, how much new data and feedback does it need before performance recovers? Humans often adapt from a few examples and verbal instructions. Sample-efficient adaptation is therefore a relevant component of generality.
Measure not only final performance but the learning curve after distribution shift.
A Practical Transfer Ladder
Level 1 changes wording. Level 2 changes examples. Level 3 changes task structure. Level 4 changes tools or environment. Level 5 changes domain. Level 6 introduces a genuinely novel problem where no close template is available. Capability that survives higher levels provides progressively stronger evidence of generalisation.
This ladder turns “the model generalises” into a graded, testable claim.
RFE Closure: Generalisation Must Survive the Removal of Familiar Scaffolding
The problem is mistaking large-scale familiarity for general intelligence. The function of transfer evaluation is to test whether learned abstractions remain useful when the environment changes. The receiver is the user who cannot guarantee tomorrow’s problem will resemble yesterday’s training data. Evidence includes performance under distribution shift, adaptation speed, cross-domain transfer and failure recognition.
The exit condition is to narrow the claim whenever performance collapses outside the familiar regime. SI requires capability that travels.
Continue the Super Intelligence (SI) Evidence Series
Next: long-horizon reliability—why completing a project is a much stronger test than answering a question.