How Superintelligence Works | Benchmarks — What Scores Measure, Hide and Miss

Deeper case study: interpretation, uncertainty and transfer

1. More than one kind of error

A benchmark is often discussed as though every incorrect answer were interchangeable. In reality a spelling mistake, a wrong edition date, a fabricated citation and a falsely claimed file operation create different problems. A fair review should identify the intended task and sort its failures by mechanism before proposing a solution.

Atlas might be stronger on routine lookups while Birch is stronger on edition reconciliation; even that breakdown may hide different problems. One could be missing documents while another reads correct documents but merges details. A top-line score cannot tell us whether better retrieval, clearer instructions or a new verifier would repair the failure.

An evaluation notebook should preserve a small number of worked error traces showing the input, relevant source, expected answer and observed output. That gives teachers and builders an explanation they can act on rather than simply another percentage.

2. Why selecting a dataset is an editorial choice

To decide which tasks count, an evaluator implicitly decides what capability matters. Selecting mostly arithmetic and logic tasks produces a different portrait of a system from selecting reading comprehension, language translation or public-service information navigation. No dataset covers every job.

A credible benchmark identifies the intended population and selection process. Who chose the questions, and what did they exclude? Were unusual cases intentionally included to challenge the model, or were questions sampled from frequent real requests? Both designs can be useful, but they answer different questions.

The reader should be able to see a connection between the dataset and the decision. An educational publisher choosing a writing assistant needs tests of source-supported explanation, age-appropriate wording and correction, not merely scores from unrelated puzzles.

3. Different weights lead to different answers

Our fictional overall score weights routine questions four times as heavily as complex ones because the packet contains eighty and twenty examples. That is a property of the constructed test, not a universally correct valuation of routine and complex library work.

A librarian may decide that a difficult edition error carries more cost because it can misdirect a historical researcher. That judgement could lead to a policy requiring greater performance on the complex subset, regardless of its proportion in a simple question count.

A good report makes weighting explicit. If someone creates a composite score with numerical priorities for accuracy, speed and safety, they must justify those priorities and disclose how the winner changes when reasonable weights are varied.

4. Expected counts are not observations

Under a ninety-percent routine workload, the weighted Atlas rate of 89.5% describes a mathematical expectation if the stipulated success rates apply. Across one hundred comparable questions the expected count would be 89.5. An actual observed count must be an integer and could differ.

Replacing the expected value with a statement such as ‘Atlas answered ninety requests correctly’ would turn the teaching calculation into fabricated history. The benchmark’s reference data did not include any run on that new workload. Our altered mixture is a scenario, not a recorded service trial.

The same discipline applies to economic forecasts, school performance projections and science simulations. Separate the observed inputs, the mathematical transformation and the conditional output. A model can help people reason while keeping those boundaries honest.

5. Paired comparisons and who got which question wrong

If Atlas and Birch face the same one hundred questions, the aggregate totals do not reveal how often both were wrong on the same question. Two systems can have identical accuracy while making very different errors, a distinction relevant to combined review or model selection.

Keep a per-question comparison table with the correct reference answer and each response. The table can reveal complementary strengths, shared failures and questions that deserve expert inspection. Do not infer independence between model errors simply because the systems were built by different providers.

Pairwise analysis is particularly helpful when a headline gap is small. It can show whether a difference is driven by several unique successes, one disputed reference answer or repeated cases within the same document family.

6. Reference answers must be traceable

A benchmark can have perfectly reproducible scoring and still be wrong if its reference labels are inaccurate. A library catalogue may include a later correction to a publication year, making an earlier answer key obsolete. The grader would punish a correct updated answer if it kept the old key.

Reference preparation needs versioned source documents, review of ambiguous cases and a correction process. Separate a historical label from a current official record. For educational questions, match the relevant qualification, cohort and academic year before assigning an answer.

If the gold reference changes, preserve the old result as a historical measurement and label the new evaluation separately. Quietly changing the answer key while retaining the same comparison number undermines auditability.

7. The role of legitimate abstention

Sometimes the right behaviour is to state that available sources are insufficient. An assistant that invents a plausible edition year may answer more questions than one that asks for a current catalogue, but the latter may be safer and more useful.

Define the benchmark’s abstention policy. Does an appropriate ‘not established’ response count as correct when the packet lacks decisive evidence? Does a refusal where the answer is plainly available count as a failure? These are substantive criteria, not convenient scoring tricks.

Strong tests include questions that cannot be answered responsibly from the supplied material. This distinguishes confidence from grounded capability, and encourages correct uncertainty rather than compulsory guessing.

8. Changed-context tests distinguish reasoning from copying

A useful transfer question changes a material condition and asks whether the system updates its response. If a catalogue record from the previous year is superseded by a newer dated notice, the correct answer may change. A model that repeats the old date despite seeing the new evidence has failed the transfer.

Change one feature at a time during diagnosis: year, document type, unit, language or permission boundary. A good test explains why that change should or should not alter the answer. Repeated success on fixed examples can conceal brittle pattern matching.

For students, this is an especially valuable learning habit. Being able to solve a familiar question matters; being able to explain how the answer changes under a new assumption shows deeper understanding.

9. Why an elegant benchmark report can mislead

Leaderboard pages naturally emphasise ordering and attractive figures. Colourful score bars and a prominent winner can make a measurement seem more comprehensive than it is. Good presentation should not obscure sample sizes, subgroup differences or methodological limits.

Keep a visible route from each numerical claim back to counts and grading rules. Show when a difference is small relative to the amount of evidence. A sentence explaining the missing condition may be more informative than another graphic.

Responsible reporting distinguishes a result that has been independently reproduced from one generated by a single evaluation run. Neither should be inflated into a guarantee of application safety or professional quality.

10. Performance in different languages and registers

An AI system that performs well on English catalogue questions may struggle with multilingual titles, transliteration, regional spelling or bilingual records. A benchmark dominated by one language does not establish equal capability in every language community.

If the target readers use multiple languages, include representative questions and source documents for those languages, with appropriate expert labels. Preserve scripts, diacritics and names; do not assume that approximate translations always convey the same catalogue identity.

A holistic view can also distinguish reading level, technical register and cultural context. A pupil-friendly explanation is not the same output as an archival citation, even when both concern the same subject.

11. Why exact arithmetic still requires interpretation

The arithmetic of Atlas and Birch is unambiguous under the stated numbers. Yet it would be wrong to infer that an 86% accuracy system will necessarily outperform an 84% system for a Singapore school’s specific needs without considering task mix, source quality and error consequences.

Maths protects us from some mistakes but not from ill-defined questions. A weighted average can be exact even when its weights are chosen poorly. A percentage can be correct while its label is misleading. A cost calculation can be precise despite omitting the most important operating expense.

Teach both steps: first verify the computation, then explain what the computation represents. This connects Mathematics learning with adult decision-making, research literacy and responsible use of AI.

12. Speed and reliability should be measured together

A system that consistently returns a correct answer after a long delay may be unsuitable for a time-sensitive request. Another may respond quickly with an uncertain answer that requires human correction. Neither speed nor accuracy alone determines total usefulness.

Specify whether timing includes searching, tool execution, verification and delivery. Measure how often a request times out, not merely the average time for the successful cases. Slow exceptional cases may dominate the user’s experience.

A fair comparison asks what outcome the user receives by the deadline, how much reviewer effort remains and whether external state was actually confirmed. These are application measurements that a pure model benchmark may not provide.

13. Resource efficiency without invented market figures

Suppose two systems consume different amounts of compute or tool operations to reach accepted answers. A benchmark could compare resource use per verified success, but the measurement needs precise definitions and actual observations. A cheap single output might be costly once repeated attempts are included.

Do not attribute real provider pricing or energy use to fictional systems. For teaching, the principle is enough: total operational cost can include model calls, external tools, retries, storage and human corrections, and all must be compared on a common basis.

Resource efficiency may also conflict with answer quality. Reducing calculation time could remove a valuable verification step. A complete evaluation should reveal the trade-off rather than rewarding a fast but untrustworthy result.

14. Reproducibility and model-version control

Record the model identifier, dataset snapshot, prompts, tool settings, scorer version and test date. Without these, another investigator may obtain a different result for legitimate reasons. An updated dataset or configuration changes the experiment.

If a published score moves upward, readers need to know whether the model became better, the questions became easier or the scorer was repaired. A chart alone cannot isolate the cause when several components change simultaneously.

Keep previous results as versioned historical records where appropriate. Do not quietly overwrite an old result and continue describing the comparison as unchanged. Reproducibility is the ability to inspect what was actually done.

15. Benchmarks can be optimised rather than understood

When an evaluation is public and influential, developers may focus effort on its visible tasks. Improvement on that benchmark can be genuine, but it may not transfer proportionally to unfamiliar situations. The incentive makes fresh and diverse testing important.

Use varied sources, unseen cases and changed-context tasks. Study both aggregate success and types of residual failure. Reserve some evaluation material from routine development so teams do not repeatedly tune against the final assessment itself.

A trustworthy evaluator should acknowledge that no finite test can exhaust the possibilities of language, society or the world. The aim is not omniscience; it is reliable, auditable evidence about a defined purpose.

16. Decisions should incorporate consequences of error

Suppose a catalogue error merely requires rechecking a routine book shelf, while an edition error misattributes a historical source in a public article. Those mistakes have different consequences despite each counting as one wrong benchmark question.

An application can require a higher threshold of correctness on high-impact cases or mandatory human review whenever relevant evidence conflicts. That decision belongs to responsible operators, not to an apparently impressive leaderboard.

To make the comparison actionable, show error categories and propose matched controls: re-retrieval for stale sources, calculation checks for numbers, approval gates for external actions and independent review for uncertain conclusions.

17. An independent assessment of evidence quality

Before concluding that a score supports deployment, ask who verified the reference answers and whether each answer follows from an appropriate source. A model may return the reference label by chance while using wrong or fabricated citations.

A source-aware evaluation can require both the correct answer and the relevant evidence. It may also explicitly reward identifying an unresolved question. This is a stronger test of research work than plain exact matching.

Link the result to Research Agents, which teaches provenance and claim checking in a complete fictional investigation. Benchmarking remains a part of the larger evidence workflow.

18. Review the practice calculation once more

At sixty percent routine requests, Atlas’s hypothetical 73% and Birch’s 87% follow directly from the specified category rates. At ninety-five percent routine, Atlas’s 92.25% and Birch’s 85.25% reverse the order. Neither new percentage comes from running those systems in the new settings.

Suppose the incoming questions are slightly harder even within the routine group. Then the old 95% and 85% category rates may no longer apply. Reweighting alone cannot compensate for this shift; a fresh representative test is required.

Finally consider the reader’s decision. A specialist archive might prefer a system that is stronger on complex cases even at greater human-review cost. A routine desk might value throughput. The calculation informs the decision, but does not make it automatically.

19. A practical reporting and revision route

Write a benchmark report card with the task, population, version, counting method, reference labels, tool conditions, subgroup results, missing tests and source review date. Include example errors that illustrate what a reader should not assume.

Then plan the next experiment. If edition reconciliation is weak, test more genuinely unfamiliar edition conflicts; if citations are wrong, include source-entailment checks; if timeouts matter, test under realistic service conditions. An evaluation is useful when it points to a repair.

After the repair, repeat the appropriate tests and compare the results under equivalent conditions. Do not declare progress simply because the prompt became longer or the marketing description became more confident. The observed difference must support the conclusion.

Applied benchmark interpretation: beyond the headline score

Score differences versus decision differences

The Library Challenge’s original two-point gap between Atlas and Birch is a numerical difference on a specified hundred-item test. It does not tell the learning centre how many people will be affected by a future error. First translate the percentages into the actual tasks the centre expects: edition comparisons, shelf checks, reservation questions and requests that need a staff member. The user does not experience a leaderboard position; the user experiences a correct or incorrect decision.

Ask what would happen if one wrong edition answer caused a student to study from outdated material. Its consequence may differ from a misplaced punctuation mark in a catalogue label. Assigning values to those errors requires educational judgement and a clearly approved policy, not merely changing a benchmark weight after seeing which model wins. The score should inform the decision without pretending to make it.

A workload-weighted example with visible assumptions

Suppose an imaginary learning desk expects three hundred questions each week, with 240 routine and 60 complex questions. Under the same unverified carry-over assumption as the original fictional benchmark, Atlas would be expected to answer 240×0.95 = 228 routine questions and 60×0.40 = 24 complex questions correctly: a total of 252. Birch would have 240×0.85 = 204 routine and 60×0.90 = 54 complex correct, totalling 258.

The difference is six expected correct answers in this hypothetical workload. It is not six observed students helped, six cases of harm avoided, or a guarantee of performance next week. The calculation is useful because it preserves category denominators and connects a percentage to a reader’s anticipated job. It also exposes the assumption that examples in each subgroup retain their difficulty when transferred from the invented benchmark.

Why equal overall scores may conceal incompatible mistakes

Imagine a third fictional model called Cedar scoring eighty-six out of one hundred, exactly matching Birch’s overall accuracy. Its failures, however, consist of fourteen wrong routine answers and zero wrong complex answers only if it correctly answers all twenty complex items and sixty-six routine ones. Another 86% system might instead answer seventy-eight routine and eight complex questions correctly. Both totals are eighty-six, but their error locations differ sharply.

Before choosing a model, identify which mistakes require the most costly repair. A system that repeatedly confuses editions may force every complex answer into human review. One that occasionally mistypes a predictable location may be easier to check through a deterministic lookup. Equal scores do not imply equal interventions, and the best repair route may involve improving the application rather than replacing the model.

The right baseline can be simpler than AI

Suppose the catalogue’s routine questions ask for an exact book ID and shelf number. A direct database query might answer them with a documented field mapping, no language-model generation and an explicit error if the ID is absent. Compare that simple baseline with the AI system on accuracy, latency, maintenance and what happens when information is missing. An intelligent-sounding explanation is unnecessary when an exact lookup already settles the question.

The AI model may still add value for ambiguous natural-language requests, spelling variations, or explanations that compare editions. A hybrid design can route clean identifiers to a database and difficult questions to a properly bounded assistant. Benchmarking the complete hybrid workflow often reveals more about the reader’s experience than choosing the model with the highest score in isolation.

A useful benchmark includes appropriate abstention

Some questions cannot be answered from the supplied catalogue. A legitimate response is to say that an edition or availability claim is not established by the current records. An evaluation should distinguish justified abstention from failure to answer a straightforward question. If every declined question is automatically scored wrong, a model may be rewarded for guessing whenever evidence is missing.

Conversely, a system that refuses every question cannot serve a library. An appropriate metric considers accuracy on answered questions, coverage of answerable requests, and whether abstentions occur when information is genuinely inadequate. Examine each denominator. Ninety percent correctness on ten answered questions is not equivalent to ninety percent correctness on a hundred representative questions.

Accuracy on answered questions and coverage

Take a separate fictional set of fifty requests. An assistant answers forty and declines ten. Of the forty answers, thirty-six are correct. Its selective accuracy is 36/40 = 90%, its answer coverage is 40/50 = 80%, and its correct-answer rate over all fifty requests is 36/50 = 72%. These are three legitimate quantities with three different meanings.

A reader who sees only the ninety-percent figure may overestimate the proportion of requests the application can resolve. A reader who sees only seventy-two percent may miss that deferring uncertain cases is a safety feature rather than simply a defect. A fair report publishes all three and identifies why the ten were declined, including whether they truly lacked adequate sources.

A model can pass and still mishandle an external action

Consider a request to ‘update the list of newly purchased books’. A language model may correctly extract every title and ISBN, scoring full marks on the text portion of a benchmark, but fail when the database rejects a transaction. If the assistant says the records were saved without confirming the external result, the user’s actual task remains undone.

Evaluate actions by independently reading the target record or transaction status. Distinguish prepared data, attempted submission, server acceptance and verified committed state. If an operation times out, inspect the database before retrying to avoid duplicate entries. The benchmark must measure the event the user cares about, not merely whether the assistant described a plausible sequence of actions.

Latency and reliability under interruptions

One model may answer in two seconds during a benchmark while another needs more time to retrieve documents. That comparison matters only if the test includes equivalent network conditions and defines whether retrieval and tool execution count. For an ordinary library query, a slightly slower verified lookup may be preferable to a quick fabricated answer. For a live help desk, extreme delay can itself be a material failure.

Measure distribution rather than relying only on a mean response time. The ninety-fifth percentile can help reveal long waits, but it is also sensitive to test sampling and load conditions. Include explicit timeouts and connection interruptions so the final statistics capture operational failures, not just fluent responses that happened to arrive.

Performance changes when a document version changes

The fictional library may replace its cataloguing system or revise how it identifies editions. Under the old test, exact titles might uniquely identify a book. Under the new system, the same title could correspond to several editions and formats. A model that answers from its former pattern can remain confident while becoming wrong. Its earlier benchmark score did not cover the changed identity rule.

Establish a source-version contract in the application: which database snapshot, schema, date and access privileges support an answer? Re-evaluate with new and revised records, not merely old questions. A benchmark that is refreshed for a real change in the task may deserve more weight than one that has been circulated unchanged for years.

Fairness and accessibility require their own tests

Imagine the service must work for children, adult learners and readers using different languages or assistive technology. A strong average on English catalogue questions does not establish that the assistant explains the same rules accessibly across those groups. Test relevant cases with a justified reference rubric, respect privacy and avoid treating population categories as proxies for individual capability.

Fairness questions depend on the deployment’s purpose and on which harms could occur. A difference in task accuracy might reflect uneven source coverage, different question difficulty or a system limitation. The evaluator should investigate those mechanisms before attaching a single moral label to a percentage. The next steps may involve better documents, accessible input formats or different review processes.

Safety testing and capability testing must remain distinct

A benchmark may deliberately probe whether the system responds safely to manipulative text or requests outside its authority. That is not the same as measuring how well it answers legitimate catalogue questions. Both matter, but an average that blends them without explanation can hide a serious weakness. A model could excel at factual lookup and still be unsafe when equipped with overly powerful external tools.

Test the entire application boundary separately: can untrusted webpage text change instructions, can it access an unauthorised collection, or can it claim to have sent messages that it never sent? A high score in a model-only safety test does not prove these system properties. Architecture, permissions and independent observation are necessary parts of the assessment.

A defensible comparison protocol

Record each system’s exact model and application version, dataset version, evaluation date, prompt, temperature or sampling controls where applicable, available tools, token and time budgets, allowed retries, and the grader’s method. The same questions should be scored with the same reference standard. When a provider-specific feature changes a result, disclose that rather than pretending the systems received identical input.

Run obvious arithmetic and identity checks on the scores before publication. Confirm that subset counts sum to the total, that abstentions and timeouts were assigned consistently, and that duplicated questions are documented. This small audit catches errors before an attractive leaderboard spreads a claim that cannot be reconstructed.

What counts as an independent replication

A second run with the same seed, prompts and data can help reproduce an earlier number, but it does not create a new representative sample. Genuine generalisation evidence comes from new, appropriately sampled tasks or changed-context tests that were not used to tune the model or grader. A different language-model grader examining the same examples may not be independent of the same misleading source material.

For a new library deployment, collect consented or synthetic representative examples without revealing private borrower details. Hold back a meaningful part for final assessment, check for overlap with development data and document where human experts disagreed on references. It is better to publish a modest defensible result than an impressive number that no one can verify.

A compact cost calculation with fictional credits

Suppose Atlas’s fictional operating charge is two credits per request and Birch’s is three, with no other expenses. Processing one hundred test requests costs Atlas 200 credits and Birch 300. Dividing by the number of correct test answers gives approximately 200/84 = 2.38 credits per correct answer for Atlas and 300/86 = 3.49 for Birch. This is a teaching-only ratio under artificial fixed costs.

A real institution would also consider human review, retries, expensive tool calls, data protection, service availability and the financial effect of errors. A cheaper answer can become the more expensive decision when it triggers many repairs. This example therefore illustrates how to keep cost units visible, not which real vendor or model is economically best.

Independent review and stopping criteria

Benchmark interpretation can itself become an endless exercise in collecting more rankings. Set a practical stopping condition: when the target task is clear, the candidate systems have been tested against representative and important difficult cases, material uncertainties are documented and the responsible person has adequate evidence for a decision. If the stakes are high, the stopping criteria should include domain review and appropriate operational safeguards.

Do not claim that no untested failure exists. Instead state the tested boundary and a schedule for reviewing drift, new tasks and changed policies. A trustworthy deployment is not frozen at the moment the benchmark is won; it keeps checking whether the application continues to serve its actual users.

A changed-context check readers can perform independently

Assume a third fictional help desk receives forty routine requests and sixty complex requests. Using Atlas’s 95% and 40% rates, its expected score is 0.4×0.95 + 0.6×0.40 = 62%. Birch’s is 0.4×0.85 + 0.6×0.90 = 88%. Before looking at those answers, write the category counts and decide whether the figures represent observed questions or reweighted expectations.

Now imagine the difficult questions include a completely new document format. The old complex-category rates may not apply at all. The appropriate next step is to test fresh examples, not to average the two historic scores more carefully. This is the transferable habit: identify which assumption is being used, carry units and source scope into the output, and know when a new experiment is required.