How does superintelligence work when an AI system passes every test it was given? That sounds like proof of success, but it depends on which cases were tested. If every trial involved an ordinary worksheet with complete details, the system might still fail when a form contains private information or a required field is unknown. Passing a test suite is useful evidence of tested behaviour, not a guarantee that every possible input was handled.
The mechanism is test-coverage reasoning. Coverage asks what portion of a clearly defined space of behaviours, conditions or inputs was actually exercised. A reliable AI-assisted workflow combines expected outputs, negative cases, missing-information cases and independent checks rather than relying on a row of green ticks.
This easy guide builds a fictional school worksheet decision service with twelve test cases. We calculate the difference between four passing starter tests and a broader suite, explain why coverage depends on its denominator and classify twenty claims. The procedure and scores are invented for teaching, not tests of a real AI product. “How Superintelligence Works” is the name of an educational series, not evidence that current systems have attained artificial superintelligence.
1. What is a test?
A test supplies an input, defines an expected outcome and compares the result with that expectation. A calculator test might ask for 3 + 4 and expect 7. A document workflow test might supply an incomplete form and expect a request for clarification.
An AI output can sound plausible and still fail its expected behaviour. Testing replaces the vague question “Did that look good?” with an inspectable question about what the system was supposed to do in a defined case.
2. Why one passing test means very little
Suppose a worksheet assistant correctly prepares an ordinary draft that contains no private information and has all required details. That result tells us something useful: the system handled that example. It tells us very little about whether the system would protect confidential records or respond to incomplete requests.
A test is evidence of behaviour for its specific conditions. A broad claim about capability requires a sample of the kinds of conditions the real application must handle.
3. What is coverage?
Coverage measures the extent of a defined set that has been exercised. It might mean the fraction of possible input combinations tested, the branches of code reached, the requirements verified or the risks represented by cases. These are different metrics.
A report saying “100% coverage” is incomplete without identifying the denominator. Full coverage of four selected examples does not mean complete coverage of all possible messages, users or real-world states.
4. Correctness and coverage are separate
Imagine four tests all pass. Accuracy on those four expected outcomes is 4/4 = 100%. But if those four cover only half the known input combinations, the coverage of that input space is 50%. Both values can be true.
An AI evaluation should report what was tested and how well it performed. Otherwise a perfect-looking pass percentage may conceal which kinds of error were never challenged.
5. The fictional Cedar Worksheet Decision Service
Imagine a school application whose purpose is to select the next step for a worksheet request. It checks three fields. P asks whether the material contains private student information. G asks whether the target year level is supplied. K asks whether the answer key has been verified.
For the teaching example, each field can be YES, NO or UNKNOWN. YES and NO are definite states; UNKNOWN means the field has not been established. The service does not publish or send anything. It selects a safe next step.
6. Define the workflow before generating tests
If P = YES, the result is PRIVACY REVIEW, regardless of the later fields. If P = UNKNOWN, it is CHECK PRIVACY. If P = NO and G = NO or K = NO when G = YES, it is HOLD. If P = NO and G = UNKNOWN, it is CHECK YEAR.
If P = NO, G = YES and K = UNKNOWN, the result is CHECK KEY. Only P = NO, G = YES and K = YES produces READY TO DRAFT. That result still means a draft may be prepared, not published or sent.
7. Why the order of checks matters
Privacy status is evaluated first because it may change who may handle the material and which action is appropriate. A verified answer key does not override confirmed private student information. A year level cannot cure a missing privacy determination.
The decision rule is a fictional policy, not official school guidance. Its value is that the intended order is explicit and can be tested for unexpected behaviour.
8. Count the simple binary input space
First imagine all three fields are definitely YES or NO. Each has two possibilities, so the number of possible combinations is 2 × 2 × 2 = 8. These are the eight fully known input patterns.
That number is small enough to inspect exhaustively. It says nothing about every possible text sentence or every document the application could receive. It is coverage of a simplified structured input space.
9. Four ordinary starter tests
Suppose the first test suite includes only cases where P = NO. It checks the four combinations of G and K, including the one ready case and three cases that must remain on hold. Every starter test passes.
This suite demonstrates that the application handles four non-private structured cases under the tested procedure. It contains no case with P = YES or P = UNKNOWN. It cannot verify the privacy-review or privacy-uncertainty branches.
10. The first four reference tests
- T1 — P=NO, G=YES, K=YES → READY TO DRAFT.
- T2 — P=NO, G=YES, K=NO → HOLD.
- T3 — P=NO, G=NO, K=YES → HOLD.
- T4 — P=NO, G=NO, K=NO → HOLD.
- Starter suite: 4 out of the 8 fully known YES/NO combinations.
- Starter combination coverage: 4/8 = 50%.
- If all four expected outputs pass, starter pass rate is 4/4 = 100%.
- Privacy-positive cases tested in the starter suite: zero.
11. Why the green ticks do not settle privacy
The starter suite might be perfectly implemented and still fail the important test of routing private material. For example, an application could mistakenly ignore P entirely while passing all four P=NO cases.
That is the core danger of confusing passing tests with complete evaluation. A bug that lies outside the tested conditions may remain invisible even when every visible test is green.
12. Add the four privacy-positive known cases
Now include T5 through T8, with P = YES and every possible YES/NO combination for G and K. The reference result in all four cases is PRIVACY REVIEW, because that first policy branch is decisive.
Together with T1 through T4, the suite now covers all eight fully known binary combinations. That is 8/8 = 100% of the defined YES/NO input combinations. It still says nothing about UNKNOWN values or more complex real-world documents.
13. The complete eight-case known-value suite
- T1 — NO, YES, YES → READY TO DRAFT.
- T2 — NO, YES, NO → HOLD.
- T3 — NO, NO, YES → HOLD.
- T4 — NO, NO, NO → HOLD.
- T5 — YES, YES, YES → PRIVACY REVIEW.
- T6 — YES, YES, NO → PRIVACY REVIEW.
- T7 — YES, NO, YES → PRIVACY REVIEW.
- T8 — YES, NO, NO → PRIVACY REVIEW.
- Known-value combination coverage: 8/8 = 100%.
- This is complete only for the eight combinations of definite YES/NO values.
14. Unknown is a third value, not a spare label
Real documents may omit a year level or contain a field whose privacy status could not be checked. If the system automatically interprets UNKNOWN as NO, it can take an inappropriate branch. The simplified binary suite would not exercise that failure.
A trustworthy test plan therefore adds missing-information cases deliberately rather than assuming normal YES/NO tests cover them by implication.
15. Add four explicit unknown cases
T9 has P unknown with G and K known, so the correct next step is CHECK PRIVACY. T10 has privacy definitely NO and year level unknown, so it needs CHECK YEAR. T11 has privacy NO, year level YES and answer-key status unknown, so it needs CHECK KEY.
T12 has privacy YES, with year level and answer-key status unknown. The first privacy branch is decisive and the expected result remains PRIVACY REVIEW. This tests whether the system properly short-circuits on a confirmed privacy concern.
16. The extended twelve-case test packet
- T9 — P=UNKNOWN, G=YES, K=YES → CHECK PRIVACY.
- T10 — P=NO, G=UNKNOWN, K=YES → CHECK YEAR.
- T11 — P=NO, G=YES, K=UNKNOWN → CHECK KEY.
- T12 — P=YES, G=UNKNOWN, K=UNKNOWN → PRIVACY REVIEW.
- All twelve cases are distinct structured input combinations.
- READY TO DRAFT outcomes: 1 (T1).
- HOLD outcomes: 3 (T2, T3, T4).
- PRIVACY REVIEW outcomes: 5 (T5, T6, T7, T8, T12).
- Targeted uncertainty checks: 3 (T9, T10, T11).
- Outcome reconciliation: 1 + 3 + 5 + 3 = 12.
17. Three-valued combinations grow quickly
With three possible values for each of three fields, the full structured input space has 3 × 3 × 3 = 27 combinations. The extended suite contains twelve distinct combinations, giving 12/27 ≈ 44.4% complete-combination coverage of that larger space.
This does not contradict the earlier 100% figure. The earlier denominator was eight fully known combinations; the newer denominator is all twenty-seven three-valued combinations. Coverage percentages mean nothing unless their denominator is stated.
18. More combinations do not automatically mean safer behaviour
Testing each of twenty-seven combinations can be helpful in a small decision service. But a real application may also accept free text, attachments, inconsistent metadata, version changes and malicious or ambiguous inputs. The state space may be vastly larger.
Combinatorial coverage is one useful tool, not the entire safety case. A model might pass structured tests and still misunderstand the source document that supplies the structured values.
19. Passing the extended suite
Suppose a fictional implementation returns every expected result for T1 through T12. Its pass rate on the fixed packet is 12/12 = 100%. We have defined that result for the teaching exercise rather than observed it in a live system.
The success is meaningful within its scope: the implementation follows the reference rule on twelve specified inputs. It does not establish that all other fifteen three-valued combinations work, nor that free-text requests are correctly parsed.
20. Twenty claims to classify
Use SUPPORTED, CONTRADICTED and NOT ESTABLISHED. The reference consists of the policy and twelve listed cases. The labels distinguish true descriptions of tested conditions from claims of unobserved universal performance.
The statements intentionally mix pass rate, input coverage, specific branches and claims about the future. Keeping these dimensions separate is the purpose of the exercise.
- 01. “Three binary fields have eight possible YES/NO combinations.” — SUPPORTED.
- 02. “The starter suite has four tests.” — SUPPORTED.
- 03. “The starter suite contains a case with P=YES.” — CONTRADICTED.
- 04. “The starter suite covers 50% of the eight known-value combinations.” — SUPPORTED.
- 05. “T1 is READY TO DRAFT.” — SUPPORTED.
- 06. “T2 is READY TO DRAFT.” — CONTRADICTED.
- 07. “T3 is HOLD.” — SUPPORTED.
- 08. “T4 is HOLD.” — SUPPORTED.
- 09. “T5 requires PRIVACY REVIEW.” — SUPPORTED.
- 10. “T8 requires PRIVACY REVIEW.” — SUPPORTED.
- 11. “T9 requires CHECK PRIVACY.” — SUPPORTED.
- 12. “T10 requires CHECK YEAR.” — SUPPORTED.
- 13. “T11 requires CHECK KEY.” — SUPPORTED.
- 14. “T12 requires PRIVACY REVIEW.” — SUPPORTED.
- 15. “The expanded suite covers all eight fully known combinations.” — SUPPORTED.
- 16. “The expanded suite contains four cases with at least one UNKNOWN field.” — SUPPORTED.
- 17. “Passing twelve cases guarantees correct responses on every future input.” — NOT ESTABLISHED.
- 18. “Passing the four starter tests proves the privacy-review mechanism is correct.” — NOT ESTABLISHED.
- 19. “T12 can bypass privacy review and be drafted immediately.” — CONTRADICTED.
- 20. “Test coverage is always the same statistic as real-world accuracy.” — CONTRADICTED.
21. Check the classification totals
SUPPORTED: 01, 02, 04, 05, 07, 08, 09, 10, 11, 12, 13, 14, 15 and 16 — fourteen. CONTRADICTED: 03, 06, 19 and 20 — four. NOT ESTABLISHED: 17 and 18 — two. Fourteen plus four plus two equals twenty.
The last two are not established because a finite set of passing tests does not determine unseen behaviour. The suite can give useful evidence without logically guaranteeing results beyond the cases tested.
22. Input coverage and code coverage are different
The 8/8 and 12/27 figures in this article refer to combinations of structured input values. Code coverage, such as statement coverage or branch coverage, concerns which parts of a software implementation were executed.
A program can execute every branch while still processing the wrong meaning from a user message. Conversely, a set of input combinations may exercise the same software path repeatedly. A responsible report identifies the metric rather than using the word coverage generically.
23. Requirement coverage is another distinct metric
An organisation may require no unauthorised publication, accurate source verification, correct privacy handling and a reliable audit trail. Testing only the draft-or-hold decision does not verify all four requirements.
A requirements matrix can connect each expected rule to one or more tests and source evidence. Coverage then shows which obligations have actually been checked, not just which examples produced a visible answer.
24. Negative cases may matter more than easy positives
A test with complete information may confirm that the system can produce a draft. But a more consequential test might ask whether it refuses to publish a private record without authorisation. That blocked action may be more important than several routine formatting successes.
Test priority should reflect risk, not only the convenience of generating inputs. The goal is confidence that the system handles the circumstances that matter, including those that should not proceed.
25. Unknown fields are not the same as false fields
T9 has no verified privacy status. Treating that unknown as NO could lead to drafting before checking privacy. T12 has privacy definitely YES, so the result is REVIEW even though two later fields are unknown.
The distinction is one reason to include unknown cases explicitly. Real-world forms and documents are frequently incomplete, and a test suite containing only perfect data can miss such behaviour.
26. A passing test suite can share a wrong reference
If the expected labels were generated by the same flawed model being tested, the model and its supposed answer key could share an error. A high pass rate would then be misleading.
Independent ground truth, clear rule definitions and appropriate human review help prevent this. The test oracle—the method for deciding the correct answer—is part of the evaluation, not an unquestionable decoration.
27. Hard-coded success is not generalisation
A program could memorise the twelve specific input triples and return exactly the expected labels for them, while failing any new combination. Such a program would receive a perfect pass rate on the known packet.
Testing unseen cases and alternative phrasings helps determine whether the system learned or implemented the underlying rule rather than merely fitting the examples it had already encountered.
28. Regression testing protects previous fixes
After correcting the privacy branch, run cases that exercised the old behaviour as well as new privacy-specific tests. Otherwise an update could repair T9 while breaking T1 or T10.
Regression testing asks whether previously supported outcomes remain correct after a change. It complements broader coverage but does not replace independent evaluation of the new feature.
29. Boundary tests add another dimension
If a workflow includes dates, age thresholds or sheet limits, it should test values just below, exactly at and just above each cutoff. Our simple YES/NO/UNKNOWN tree does not contain numeric thresholds.
A test suite can cover every Boolean combination and still fail on one improperly implemented less-than-or-equal comparison elsewhere. The required test dimensions must match the actual system.
30. Why coverage should have a clear denominator
An assistant might say “We tested 100% of cases.” Does that mean every case in an internally chosen twelve-case packet, all eight binary combinations, all twenty-seven three-valued combinations or every message a real user might enter?
Only the first two could be true under our invented results, and they describe different scopes. The larger natural-language space cannot be exhaustively captured by these few structured examples. Careful reporting prevents an impressive phrase from becoming an unsupported capability claim.
31. The test count does not measure user impact
A system might pass twelve synthetic requests and still be slow, inaccessible or confusing in everyday use. Those qualities require their own evaluation. Likewise, a correct branch decision is not proof that a later human review actually happened.
The responsible conclusion is about the decision output tested. External actions, privacy outcomes, learning effects and delivery receipts require evidence beyond the branch-label fixture.
32. A test failure can reveal a category of errors
Suppose an assistant wrongly sends T9 to READY instead of CHECK PRIVACY. That suggests a failure to preserve unknown privacy states. Developers should investigate other combinations involving P=UNKNOWN, not merely patch one output string.
This is how a single failure can improve a test plan. It points to a broader condition or invariant that may need verification, rather than functioning as an isolated embarrassing example.
33. A small classroom activity
Draw three columns labelled P, G and K. Start with all eight YES/NO combinations and ask pupils to evaluate each through the frozen tree. Then add the four UNKNOWN examples and count how many new outcomes become possible.
Next ask why passing the first four tests would not check the privacy branch. This teaches a broader lesson about evidence: not seeing an error in familiar situations is not the same as having tested situations where the error would occur.
34. Evaluating a live AI application
A serious deployment study would use held-out real task distributions, different document formats, genuine source ambiguity, permission checks and independent outcome verification. Test coverage should be reported alongside performance, failure severity and uncertainty.
Maintain a test ledger with a stable case ID, policy version, input, expected result, observed result and reviewer decision. That makes it possible to reproduce errors and check whether the fixes persisted.
35. An honest test report
A defensible statement for the imagined packet is: “Four starter cases passed and covered four of eight fully known input combinations. A twelve-case suite then included all eight known combinations and four cases containing UNKNOWN values. The twelve cases passed in the fictional evaluation; fifteen other three-valued combinations were not exercised.”
Such a report is not dramatic, but it tells readers exactly what happened. It distinguishes local success from wider assurance and makes the remaining work visible.
Frequently asked questions
Does passing every test prove an AI is always right? No. It supports only the tested conditions and the quality of the reference answers.
What is test coverage? The fraction of a defined space of inputs, code paths, requirements or other targets exercised by testing.
Can 100% pass rate coexist with 50% coverage? Yes. All selected tests can pass while many defined conditions remain untested.
Why test UNKNOWN separately from NO? Missing evidence is not the same as an established negative state.
Is 100% code coverage proof of correctness? No. Executing code paths does not prove their logic or outputs are right.
What is regression testing? Rechecking previously working behaviour after a change.
Why use independent expected answers? To avoid an evaluation that merely rewards repeating the system’s own mistakes.
The central lesson
Green checks matter, but they do not speak for the cases a system never examined. A reliable AI evaluation asks two separate questions: did the system behave correctly on the test inputs, and did those inputs cover the important conditions?
The strongest habit is to name the denominator, include negative and unknown cases, test the risky branches and distinguish verified results from claims about the unseen world. That turns testing into meaningful evidence instead of a reassuring collection of ticks.
Connected learning and authoritative references
- How Super Intelligence Works — main educational curriculum.
- Decision Trees — How AI Chooses the Right Path — the underlying branch procedure.
- Feedback Loops — How AI Uses Corrections — regressions and verified repairs.
- Boundary Values — Why Exact Cutoffs Matter — testing the edges.
- ISTQB: Certified Tester Foundation Level — coverage, test design and test-case terminology.
- NIST: Automated Combinatorial Testing for Software — why interactions among input conditions matter.
- Python unittest documentation — software test assertions and regression testing.