How does superintelligence work when an AI seems to score brilliantly on an examination—but has already seen the answers? A learner who memorises the answer key to a practice paper may earn a perfect mark without being ready for new questions. Machine-learning systems can have a related evaluation problem when information from the test set leaks into training, prompts or model-selection decisions.
The mechanism is data leakage: information that should not be available when a prediction is made enters the process used to build, select or evaluate the model. That can produce impressive but misleading accuracy estimates. A trustworthy AI evaluation therefore separates examples used for development from independent examples reserved to test generalisation.
This plain-language guide uses twenty entirely fictional science-answer checks: twelve answers exposed during preparation and eight reserved for an independent check. We calculate the apparent pooled score, the performance on new items and twenty inspectable claims. The figures are illustrative inventions, not measured results from a deployed model. “How Superintelligence Works” is an educational series name, not proof that current AI has achieved artificial superintelligence.
1. What is generalisation?
Generalisation means performing well on relevant new examples rather than only repeating examples the system encountered during development. A spelling model should handle unfamiliar sentences, and a science assistant should answer new questions rather than merely recall a supplied answer sheet.
An evaluation should match the intended future use. Correctly repeating familiar items is useful for checking memory or reproduction but does not by itself establish the ability to solve unseen tasks.
2. What counts as data leakage?
Data leakage occurs when information that would not legitimately be available at prediction time influences model building or evaluation. One obvious form is putting test answers into the material used to prepare the system. More subtle forms include using test statistics during preprocessing or choosing prompts after inspecting the final test answers.
The scikit-learn documentation warns that leakage can give overly optimistic performance estimates. The important question is which information the predictor could actually access when the future decision is made.
3. Leakage need not involve dishonesty
A programmer may accidentally process the whole dataset before dividing it into training and test sets. A school may inadvertently include answer sheets in a prompt pack. A researcher might revise a scoring rule after repeatedly inspecting the supposedly unseen test.
Such mistakes can produce contamination without an intentional plan to cheat. A useful audit focuses on what entered the system, when it entered and how it could influence the evaluated result.
4. Training, validation and test have different roles
Training examples help fit or prepare a model. Validation examples can support development decisions such as selecting settings. A final test set should be reserved for an independent assessment after those decisions are fixed.
The division is not merely a naming convention. If a development team repeatedly tunes itself against the final test, the test no longer serves as a genuinely unseen independent benchmark.
5. The fictional Cedar Science Answer Desk
Imagine a school assembling twenty science questions, labelled Q01 through Q20. A fictional assistant is prepared using the official answers for Q01 through Q12. The answers for Q13 through Q20 are held back until the final scoring stage.
The scenario is invented to make different evaluation populations visible. It does not claim that any actual model was trained or fine-tuned on these questions. Exposure can arise through demonstrations or accessible answer material, not only updates to learned parameters.
6. Define the two partitions
The exposed group contains twelve items, Q01–Q12, whose answers were available during preparation. The holdout group contains eight items, Q13–Q20, whose answers were not used for development decisions in this teaching fixture.
Both groups have known reference answers for the evaluator. Their different exposure statuses are part of the data. We must not claim that all twenty were unseen merely because they appeared in one final score report.
7. The twenty-question reference ledger
- Exposed Q01–Q12: twelve questions, all answered correctly in the invented output record.
- Held-out Q13: correct.
- Held-out Q14: correct.
- Held-out Q15: incorrect.
- Held-out Q16: correct.
- Held-out Q17: incorrect.
- Held-out Q18: correct.
- Held-out Q19: incorrect.
- Held-out Q20: incorrect.
- Exposed correct: 12 of 12.
- Held-out correct: 4 of 8.
- Held-out incorrect: 4 of 8.
- Combined correct: 16 of 20.
- No real model was evaluated or retrained in this illustration.
8. Calculate the exposed-group score
All twelve exposed items were answered correctly, so the exposed-group accuracy is 12/12 = 100%. This is a fact about the fictional answer record, not evidence that the system knows the whole subject.
The score may reflect the system’s access to familiar answers. We cannot separate genuine reasoning from answer recall using this exposed set alone. Its exposure limits how it can be used to assess new-task capability.
9. Calculate the holdout score
Four of the eight reserved items were answered correctly: Q13, Q14, Q16 and Q18. Four were incorrect: Q15, Q17, Q19 and Q20. The holdout score is 4/8 = 50%.
This is the more relevant observed score for truly unseen items within the invented packet. Yet eight questions are too few to establish a precise generalisation rate across an entire subject or an operating AI service.
10. Calculate the pooled score
The combined test contains sixteen correct answers from twenty questions. Therefore the pooled accuracy is 16/20 = 80%. The arithmetic is correct for the mixed collection.
The interpretive problem is that 80% combines twelve exposed items with eight independent items. Calling it an 80% score on unseen questions would misrepresent what was tested.
11. Compare the two reported percentages
The pooled 80% exceeds the held-out 50% by thirty percentage points. The exposed 100% exceeds the held-out 50% by fifty percentage points.
These differences are properties of the invented fixture. They show why exposure status should accompany a score. They do not prove that the amount of leakage causes a particular numerical change in every model.
12. Accuracy is not the whole story
Even the independent holdout score depends on how questions were selected and judged. A set of easy recall items may not test scientific reasoning. A few highly specialised questions may not represent a typical classroom assignment.
A good evaluation reports task coverage, difficulty, source quality and uncertainty in addition to the headline percentage. Removing answer leakage is essential but does not automatically make an evaluation comprehensive.
13. An exact duplicate can leak across the split
Suppose Q02 and Q17 are textually identical, and one is placed in preparation material while the other is labelled held out. The model may have effectively encountered the supposedly unseen question.
A reliable split should check duplicates and closely related variants before training or prompt preparation. A different row ID is not proof of an independent question.
14. Similar templates can also leak
A worksheet may contain twenty questions generated from one fixed template, changing only names or numbers. If half appear in training and half in testing, a model could exploit the shared template instead of demonstrating wider generalisation.
This is not always unacceptable; some applications genuinely require handling the same family of problems. But the evaluation should distinguish performance on familiar templates from performance on new structures.
15. Time leakage means learning from the future
Imagine predicting whether a pupil completes an activity by Friday, while using a database field that is filled only after the result is known. That future-dependent field cannot legitimately be available at prediction time.
A model built using it may look astonishingly accurate during retrospective testing while failing when asked to forecast earlier in the workflow. Time-aware feature checks are needed.
16. Group leakage can break independence
Records from the same school term, document or source may appear in both training and test sets. Shared context can make examples easier than genuinely independent new work.
In tasks involving identifiable people, group-aware splitting can also protect against measuring performance on closely related records that appear in both sets. The appropriate grouping depends on the task and privacy requirements.
17. Preprocessing can leak too
Some statistical models normalise numeric features, select variables or impute missing values using quantities learned from their development data. If the preprocessing method fits to the entire dataset before the split, test information may influence preparation.
The scikit-learn guidance recommends splitting before fitting preprocessing and using pipelines so transformations are fitted only on the appropriate training folds. This is a technical counterpart of not showing the student the answer key.
18. A test becomes stale after repeated tuning
Suppose a team tries twenty prompt variants and chooses the one that scores highest on the final test. Those test outcomes have influenced model selection, so the dataset is no longer independent for evaluating that selected system.
The proper remedy is to maintain a fresh held-out assessment where feasible, document the evaluation history and avoid calling a repeatedly consulted test wholly unseen.
19. Leakage is different from permitted reference retrieval
A tutor may be authorised to consult an official textbook when answering a student’s question. That is legitimate source use for the task. Leakage is an evaluation problem when information that should be unavailable for the intended prediction is included in model preparation or testing.
The distinction depends on the contract. An open-book question can permit retrieval. A closed-book independent test should not secretly include the answer key.
20. The key question is what was available when?
For each data field, example or reference document, ask whether it existed and was available at the time the prediction would be made. If the answer is no, using it may invalidate the intended real-world estimate.
This temporal question often catches issues missed by surface inspection of a spreadsheet. A column may look legitimate while being populated only after the outcome.
21. The checked evaluation scorecard
- Total questions: 20.
- Exposed items: 12.
- Held-out items: 8.
- Exposed correct: 12.
- Exposed incorrect: 0.
- Held-out correct: 4.
- Held-out incorrect: 4.
- Combined correct: 16.
- Exposed-group accuracy: 100%.
- Held-out accuracy: 50%.
- Pooled apparent accuracy: 80%.
- Pooled score minus holdout score: 30 percentage points.
- Exposed score minus holdout score: 50 percentage points.
22. Twenty claims with answer labels
The following statements are classified as SUPPORTED, CONTRADICTED or NOT ESTABLISHED from the frozen invented record. A supported statement follows from the partition and answer ledger. A contradicted statement conflicts with them.
The unestablished claims ask about generalisation or internal training changes for which this small source supplies no reliable evidence.
- 01. Twenty questions were recorded — SUPPORTED.
- 02. Twelve questions were exposed during preparation — SUPPORTED.
- 03. Eight questions were reserved as held out — SUPPORTED.
- 04. All twelve exposed questions were answered correctly — SUPPORTED.
- 05. Four held-out questions were answered correctly — SUPPORTED.
- 06. Sixteen questions were answered correctly in all — SUPPORTED.
- 07. The combined score is 80% — SUPPORTED.
- 08. The held-out score is 50% — SUPPORTED.
- 09. The exposed-group score is 100% — SUPPORTED.
- 10. All twenty questions were genuinely unseen — CONTRADICTED.
- 11. The exposed and holdout groups are equally large — CONTRADICTED.
- 12. The pooled correctness rate is 50% — CONTRADICTED.
- 13. Four held-out questions were answered incorrectly — SUPPORTED.
- 14. Seeing the preparation answers proves strong new-question generalisation — CONTRADICTED as a claim of sufficient evidence.
- 15. The final twenty-item packet is wholly independent of preparation — CONTRADICTED.
- 16. One exposed question was answered incorrectly — CONTRADICTED.
- 17. Q13, Q14, Q16 and Q18 were the held-out correct items — SUPPORTED.
- 18. Exposed and held-out scores differ by fifty percentage points — SUPPORTED.
- 19. The model must achieve 50% on every future science question set — NOT ESTABLISHED.
- 20. The model definitely underwent a weight-update training procedure on the twelve exposed items — NOT ESTABLISHED.
23. Check the label reconciliation
Supported: 01–09, 13, 17 and 18, for twelve. Contradicted: 10, 11, 12, 14, 15 and 16, for six. Not established: 19 and 20, for two. The total is 12 + 6 + 2 = 20.
The distinction between exposure and weight training matters: a model can receive an answer key through a prompt or retrieval tool without being fine-tuned. The record does not establish which technical process prepared it.
24. Why a very high score can trigger investigation
An unexpectedly perfect score on difficult problems is a reason to inspect data partitions, repeated examples and the availability of answer keys. It is not proof of dishonesty or proof of leakage.
Good investigation checks concrete mechanisms. A genuinely strong model could also earn a high score. The evaluation should establish what happened rather than treat statistical surprise as a verdict.
25. What train-test separation aims to achieve
A separate final test estimates performance on cases not used to shape the system. It helps distinguish generalisation from fitting the particular examples that developers have seen.
The test should also represent the intended future task. Even perfectly separated questions can mislead if their difficulty, language or context differs substantially from the application’s real use.
26. Why eight holdout questions are not enough
A score of four correct out of eight has considerable sampling uncertainty. An additional one correct answer would change the observed rate from 50% to 62.5%; one fewer would change it to 37.5%.
The exact eight-case figure is valid for this packet. It cannot establish a tightly measured error rate for future science questions without a larger, appropriately selected evaluation.
27. Leakage can hide in labels
Suppose a dataset contains a column called “teacher correction required”, filled in only after marking is complete. Using it to predict whether an answer is incorrect at the start of marking introduces outcome-related information unavailable at that earlier time.
An assistant should examine field definitions and collection timestamps, not only whether the column looks useful. Predictive features must be available at the moment the real decision is made.
28. Leakage can hide in feature selection
A model developer may test thousands of candidate features against the whole dataset, choose those that best match all outcomes and only then split into train and test. The final test has already influenced feature selection.
The correct workflow fits selection only inside the training and validation process. A genuinely held-out final set remains untouched until model choices are frozen.
29. Cross-validation requires care
Cross-validation divides data into different train and validation folds. It can support model selection and performance assessment, but data transformations and duplicate relationships must be handled separately within each fold.
A pipeline can help ensure that feature scaling or imputation is learned from each fold’s training portion only. The word cross-validation does not automatically guarantee leak-free evaluation.
30. Public benchmarks can become familiar
An open benchmark may be widely discussed, and its example questions could appear in model-development material. A high score on such a benchmark may be less informative about truly novel problems than a carefully held-out evaluation.
Avoid asserting that a particular model memorised a benchmark without evidence. The responsible practice is to investigate contamination, document what is known and supplement public tests with fresh independent tasks.
31. A teacher’s classroom analogy
A pupil who has practised with the exact answer key may earn full marks on that paper. A new paper testing the same concepts provides different evidence about understanding.
Both scores can be educationally useful if labelled honestly. A familiar practice score is progress on learned material; a held-out score tests transfer to new questions.
32. How to repair a contaminated evaluation
Identify which test items, labels or preprocessing statistics entered preparation. Remove or isolate compromised items where feasible, freeze a revised development process and evaluate once against new independently chosen cases.
Document the contamination instead of quietly rewriting history. A new high score is credible only to the degree that the new test and its method actually support it.
33. A practical evaluation checklist
Separate training, validation and final test sets early. Check exact and near duplicates, shared sources, time ordering, source availability and preprocessing. Keep the final test out of model-selection decisions.
Report partition sizes, exposure status, scoring rules and uncertainty. The goal is not to maximise a flattering number but to measure how well the system will handle the task people actually need.
34. The central lesson
When an AI sees an answer before an examination, a strong-looking result may say more about exposure than generalisation. Mixed exposed and unseen scores should not be advertised as though every example were independently held out.
Reliable evaluation protects the test’s independence, records what information was available at prediction time and checks the model on genuinely new tasks. A modest honest result is better evidence than an impressive contaminated score.
Frequently asked questions
- What is data leakage? — Test or future information influences training, model selection or evaluation when it should not.
- Does leakage require cheating? — No. It can occur accidentally in data preparation and preprocessing.
- What is a holdout? — Examples reserved for independent final evaluation.
- Is a perfect training score proof of generalisation? — No. It may reflect exposure to familiar answers.
- Can a publicly available answer key contaminate a test? — Yes, if the evaluation assumes the answer is genuinely unseen.
- Can the same document appear in both sets? — That can compromise independence and should be examined.
- How do I prevent preprocessing leakage? — Split first and fit transformations on training data only.