How does superintelligence work when an AI gives different answers to the same question? A pupil asks how many sides a square has. One response says “Four.” and another says “4”. The words are different, but the meaning is identical. Another response might say “Five.”, which is genuinely incorrect. A useful evaluation must distinguish changes in wording from changes in truth.
The mechanism is randomness and repeatability in AI generation. Some language models sample from several plausible tokens, allowing outputs to vary. Variation can also follow from changes in context, model version, parameters or retrieved data. A stable-looking response is not necessarily accurate, and an accurate response is not necessarily identical in wording every time.
This easy guide uses twelve entirely fictional attempts at one geometry question and twenty checkable claims. It teaches exact-string agreement, meaning-based accuracy, controlled testing and the limitations of attempts to reproduce model outputs. All results are invented for education, not observed scores for any real model. “How Superintelligence Works” is an educational series title, not evidence that current AI has attained artificial superintelligence.
1. What is a repeated question?
A repeated question can mean the exact same user-visible sentence or the entire same model request, including prior messages, hidden instructions, generation settings and external tool results. Those are not automatically equivalent. An AI may be answering the same visible words in a changed context.
If a developer wants to study generation variability, it should specify what is held constant. Otherwise a different answer could be a reasonable reaction to new information rather than evidence of randomness in the generation process.
2. The difference between string and meaning
The string “Four.” contains letters and punctuation, whereas “4” is a single numeric character. A strict string comparison says they are different. A semantic test of the number of sides of a square can accept both.
This is important for generative AI because many questions permit several acceptable phrasings. Testing only whether the next response exactly matches one reference sentence can penalise correct answers while rewarding memorised but shallow wording.
3. A correct answer has an independent reference
Under ordinary Euclidean geometry, a square has four sides. That mathematical fact is the reference answer. It does not depend on the popularity of a model’s output or the frequency with which a phrase appears.
A model that repeatedly says “Five.” is reliably repeating an incorrect claim. Correctness should be checked against an appropriate independent standard, not inferred from the model’s confidence or repetition.
4. What randomness changes
Some generative systems select among possible next tokens according to a probability distribution. Multiple sequences can yield correct natural-language answers. Changing sampling parameters may alter the balance between repeated, focused text and more varied text.
However, not every system exposes the same sampling settings, and output variability does not prove which internal mechanism caused it. A trustworthy analysis reports what was controlled rather than guessing at backend details.
5. What repeatability means
Repeatability is the degree to which a defined process produces comparable results when conditions are held constant. For AI, the target may be exact text, a factual meaning, a classification label or an action result.
These targets differ. A writing assistant may be useful when its style varies. A numerical calculator should usually preserve the same mathematical answer. A tool-using assistant must also verify real effects, not just repeated claims that an action occurred.
6. The fictional Cedar Geometry Desk
Imagine a completely fictional trial in which the same geometry question is presented twelve times: “How many sides does a square have?” The output strings are recorded and evaluated under a fixed meaning-based reference.
The example does not contain actual model logs, seeds or implementation details. Its values were chosen to demonstrate how multiple output forms can be counted, not to estimate the failure rate of a deployed system.
7. The complete twelve-run ledger
- Runs 01–07: “Four.” (seven exact repetitions).
- Runs 08–11: “4” (four exact repetitions).
- Run 12: “Five.” (one occurrence).
- Reference answer: 4 sides.
- Meaning-based scoring: “Four.” and “4” count as correct; “Five.” is wrong.
- Total attempts: 7 + 4 + 1 = 12.
- Distinct exact output strings: 3.
- No real model, school pupil or commercial product was tested.
8. Count the exact responses
The first output string occurs seven times, the second four times and the third once. There are twelve recorded attempts but only three unique exact texts. That distinction already shows why the number of attempts is not the same as the number of distinct outcomes.
The most frequent exact string is “Four.”. Its frequency is seven, which is greater than the frequencies four and one for the other strings.
9. Count meaning-based correctness
Seven responses say “Four.” and four say “4”. Together they yield eleven semantically correct answers out of twelve. Their meaning-based correctness rate in the invented record is 11/12 ≈ 91.7%.
This is not the measured reliability of any actual AI model. It is a calculation for a teaching packet whose responses were invented in advance.
10. Count the genuine mistake
The only wrong response is “Five.” on run 12. The observed error proportion in the fictional packet is therefore 1/12 ≈ 8.3%.
It would be false to conclude that a real model fails 8.3% of geometry questions. The packet is not a random representative sample from a real deployment, and the case contains just one elementary question.
11. Modal exact-string share
The most common exact response, “Four.”, appears in seven of the twelve runs, giving a modal share of 7/12 ≈ 58.3%. This is a descriptive property of the invented list.
It is not a standard universal definition of model repeatability. Another study could ask how often pairs of responses match, or whether their meanings agree even when their strings do not.
12. Pairwise exact agreement
Twelve runs form 12 × 11 / 2 = 66 unordered pairs of different runs. The seven “Four.” responses produce 7 × 6 / 2 = 21 identical pairs, and the four “4” responses produce 4 × 3 / 2 = 6.
The single “Five.” has no matching pair. Thus 27 of the 66 run pairs have identical strings, or about 40.9%. This differs from the modal share because it is a different metric with a different denominator.
13. Exact text is not everything
A teacher might accept “Four.”, “4”, “A square has four sides” and “The answer is four.” All can express the same correct fact, provided no contradictory detail is added.
A strict exact-match rule would score these responses differently even though their substance is the same. The scoring rule must reflect the real task rather than convenience alone.
14. Normalising answers carefully
A limited normaliser for this geometry question could map the word four and the numeral 4 to a common numeric label. This would permit consistent correctness scoring without requiring identical punctuation.
But a general normaliser must not remove meaningful negation or units. Mapping “not four” to four would create a false positive. Normalisation rules need explicit scope and tests.
15. Temperature is not probability of truth
Some model interfaces expose a temperature parameter that affects sampling diversity. Lower values typically concentrate on higher-probability choices, while higher values may produce greater variation, subject to the model and API.
That setting is not a calibrated estimate of factual accuracy. It does not prove a response is correct or guarantee an identical result on every invocation.
16. Seeds can help reproducibility
Some APIs support a seed parameter as a best-effort way to make outputs more repeatable under matching settings. Its availability and actual guarantees depend on the API and model.
OpenAI’s API documentation describes seed-based reproducibility as best effort rather than absolute. A careful developer verifies current model support and records all relevant request settings.
17. Backend and model versions matter
An AI service can change its deployed model, system configuration or inference infrastructure. Even when the user-visible prompt stays constant, those changes may influence outputs.
For a controlled evaluation, log the model identifier and backend details made available by the service. Without them, do not claim variability arose only from random sampling.
18. The complete prompt matters
Earlier conversation turns, attachments and other instructions can change how a question is interpreted. A student asking “How many sides?” after discussing a square supplies context that is absent from the same words in a new chat.
A full-context comparison is therefore more informative than a screenshot of the latest question. An accurate AI test should preserve enough context to reconstruct what the model actually received.
19. External sources can legitimately change answers
If the model consults a weather forecast or school timetable, a changed answer may reflect updated information rather than random generation. The world and its data sources can change between requests.
The right test for such tasks checks source freshness and factual correctness, not whether a time-sensitive answer was identical to yesterday’s response.
20. The evaluated property must be named
A report can measure string agreement, semantic equivalence, factual correctness, calibrated confidence or task completion. Each describes something different about a model.
A vendor claim such as “95% consistent” is difficult to interpret without a rubric, sample size, task distribution and description of which type of consistency was tested.
21. The checked scorecard
- Recorded attempts: 12.
- Exact string “Four.”: 7.
- Exact string “4”: 4.
- Exact string “Five.”: 1.
- Distinct exact strings: 3.
- Semantically correct responses: 11.
- Semantically incorrect responses: 1.
- Meaning-based correctness: 11/12 ≈ 91.7%.
- Incorrect-answer share: 1/12 ≈ 8.3%.
- Modal exact-string share: 7/12 ≈ 58.3%.
- Pairwise identical-text comparisons: 27/66 ≈ 40.9%.
22. Twenty claims and reference labels
The statements below are classified as SUPPORTED, CONTRADICTED or NOT ESTABLISHED using only the fictional ledger and geometry reference. They test both calculation and restraint about unsupported real-world claims.
The labels were defined independently of any actual model response, so they do not represent a live benchmark or a change in a commercial system’s behaviour.
- 01. Twelve attempts were recorded — SUPPORTED.
- 02. “Four.” appears seven times — SUPPORTED.
- 03. “4” appears four times — SUPPORTED.
- 04. “Five.” appears once — SUPPORTED.
- 05. Eleven answers are semantically correct — SUPPORTED.
- 06. Exactly seven answers are semantically correct — CONTRADICTED.
- 07. Three exact output strings appear — SUPPORTED.
- 08. Twelve distinct output strings appear — CONTRADICTED.
- 09. “Four.” is the modal exact response — SUPPORTED.
- 10. The modal exact-string share is 7/12 — SUPPORTED.
- 11. The semantic correctness share is 11/12 — SUPPORTED.
- 12. All twelve responses have the same meaning — CONTRADICTED.
- 13. Run 12 gives the wrong square-side count — SUPPORTED.
- 14. Every correct response has identical punctuation — CONTRADICTED.
- 15. Every run produces identical characters — CONTRADICTED.
- 16. The fixture proves twelve real children held different opinions — NOT ESTABLISHED.
- 17. Model weights definitely changed between runs — NOT ESTABLISHED.
- 18. Output variation proves the source textbook was wrong — NOT ESTABLISHED.
- 19. A task-specific normaliser can map the correct forms to four — SUPPORTED.
- 20. String frequency and semantic correctness are different measures — SUPPORTED.
23. Verify the answer-key counts
Supported claims are 01, 02, 03, 04, 05, 07, 09, 10, 11, 13, 19 and 20: twelve. Contradicted claims are 06, 08, 12, 14 and 15: five. Not established are 16, 17 and 18: three.
Twelve plus five plus three equals twenty. The unestablished claims require facts never supplied in the fixture, so a correct classifier should not invent them.
24. Why a deterministic wrong answer remains wrong
A rule-based program can return exactly the same text every time and still contain a faulty rule. A model that always outputs “Five.” for a square would have high exact-repeatability and zero accuracy on this question.
Thus reliability requires both correct behaviour and an appropriate degree of stability. String repetition should not be rewarded as though it were an independent proof of truth.
25. Why creative tasks value variation
A writer asking for five original opening sentences may welcome variety. A student asking for a square’s number of sides expects a stable numerical truth even if the explanation changes.
Different purposes call for different evaluation targets. A creative-writing assistant and a fixed-answer mathematical service should not be judged by one unqualified measure of sameness.
26. Paraphrases test another kind of robustness
Ask “How many edges form a square?” and “Is a square a five-sided polygon?” These are not identical requests, but a geometry-aware system should answer them consistently with the same underlying shape definition.
Paraphrase robustness is not the same as repeatability under identical inputs. It requires generalising the meaning across changed wording rather than repeating a memorised sentence.
27. Stable wording can conceal changed facts
Suppose a model repeatedly says a school event is on a particular date, while the official organiser has moved the event. Identical answers do not prove they remain current.
For time-sensitive tasks, a trustworthy assistant should verify authoritative information as appropriate. Accurate changes in answer can be a feature of evidence-aware reasoning.
28. A useful regression test
After changing generation settings, rerun a fixed set of factual questions with known reference answers. Check whether accuracy changed, whether formatting constraints are met and whether any previously correct cases became wrong.
Use held-out questions too. Repeating only the exact cases used to tune the system can exaggerate performance on new material.
29. Do not infer hidden training changes
Different output strings do not prove that the model was retrained, updated its weights or permanently learned a user’s correction. The variation could arise for several reasons.
Model training, a stored memory, a changed retrieved source and stochastic generation are separate technical processes. A report should identify the mechanism only where it has supporting evidence.
30. Repeatability of actions is even more demanding
If an AI sends a message twice, the consequences can differ from generating two text drafts. A repeat invocation may create duplicate emails or reservations unless the application enforces safe request identity.
A reliable agent uses provider receipts and appropriate idempotency controls rather than assuming that repeating the same command has no external effect.
31. A classroom activity
Show pupils the twelve fictional answer cards and ask them to count exact strings first. Then group cards that mean the same thing and identify the factual error.
The activity teaches a distinction between language form and mathematical content. It also encourages checking the source instead of choosing the most common answer by popularity.
32. How developers can report a study
State the full test configuration, the number of repetitions, which conditions were controlled, how correctness was decided and which similarity measure was used. A modest, transparent report is more useful than a vague claim of perfect determinism.
Twelve invented examples cannot support a general performance guarantee. A real evaluation needs representative tasks, independent references and enough trials to show uncertainty.
33. A simple user checklist
When AI answers change, ask whether the facts changed or only the wording. Then check whether the conversation context, model, tools or sources changed. Identify which answer follows from credible evidence.
For high-consequence decisions, do not assume that the majority answer is correct. Independent verification is more valuable than a vote among similar outputs.
34. The central habit
Keep correctness and repeatability as separate measures. Define both before evaluating a model, and do not treat variation as automatically bad or stability as automatically good.
For everyday learning, the right standard is a dependable fact expressed clearly, with uncertainty and source limitations acknowledged when they matter.
Frequently asked questions
- Does the same question always produce the same text? — Not necessarily; configuration, context and generation method matter.
- Is “Four.” different from “4”? — Different characters, equivalent numeric meaning here.
- Is repeated output proof of accuracy? — No; an identical answer can be wrong.
- What is temperature? — A generation setting used by some APIs to influence output diversity.
- Does using a seed guarantee identical text? — Not universally; check current model-specific support.
- What matters for a factual task? — Correct meaning and evidence, not merely identical wording.
- Should an AI agent repeat an external action to test stability? — Only within a safe authorised test environment with duplicate protections.