Three AI evaluators giving the same answer is not the same as three independent pieces of evidence. They may share training patterns, a misleading retrieved page, an incomplete rubric or the same mistaken assumption. Independent verification means testing a claim through methods whose relevant failure paths are meaningfully different: for example, reading the primary source, checking arithmetic in a deterministic calculation, inspecting the actual saved tool state and obtaining qualified human review where necessary. Agreement can be useful, but it must not be mistaken for truth.
The practical method is to map each evaluator’s information and dependencies, test its judgement on verified counterexamples, inspect how errors overlap, and pair model-based assessment with evidence that does not merely repeat the model’s earlier output. We will solve a fictional hundred-case panel in which all three judges are wrong together fourteen times, distinguish observed consensus from actual correctness, design independent verification gates and transfer the method to a school-event publication. The numbers are invented teaching values, not measured performance of any named product.
This is a specialist continuation of How Super Intelligence Works. The reviewer calibration guide explains agreement between human raters; the test-oracle guide explains what a valid task check is. This article focuses on why apparently separate reviewers can share an error and how to build a more independent evidence path.
Choose a useful route
- Fact-checking a model answer? Begin with shared evidence and identify the real authoritative source.
- Building an evaluation panel? Study correlated errors, reference standards and independent verification gates.
- Learning Mathematics? Work through the 100-case panel and the independence probability calculation.
- Responsible for real actions? Require authenticated target-state readback, permission checks and accountable human oversight.
Contents
- 1. Agreement has a denominator, independence has a mechanism
- 2. A judge is an instrument that must itself be tested
- 3. Recent research shows why consensus can mislead
- 4. Trace dependency through data, prompts and decisions
- 5. Choose a reference standard appropriate to the task
- 6. A fictional three-judge, one-hundred-case laboratory
- 7. Compare observed errors with an independence assumption
- 8. A small table makes false consensus visible
- 9. Validate independence instead of assuming different labels imply it
- 10. Use separate verification gates for different claims
- 11. Prevent shared-source mistakes from propagating
- 12. Model judges are useful as triage, not final authority
- 13. Transfer exercise: a disputed school event notice
- 14. Statistical limits and transparent reporting
- 15. Practical repairs for false consensus
- 16. A useful verification checklist for everyday AI work
- Frequently asked questions
- Research and primary sources
- Related learning routes
1. Agreement has a denominator, independence has a mechanism
A panel can agree on a judgement because the proposition is well supported, because the rubric is easy to apply, or because every member was exposed to the same misleading information. The observed agreement fraction measures how often labels match under the tested cases. It does not reveal by itself which cause is operating. To evaluate reliability, examine how the judges were trained or prompted, which source snippets they received and whether an external ground-truth check exists.
Suppose a teacher asks whether an invented school revision session begins at 09:00. Three language models read the same cached flyer, each says 09:00 and each gives an elaborate justification. If the organiser subsequently corrected the session time to 10:00 and the panel never saw the correction, all three answers are wrong for the current planning decision. The agreement reflects shared evidence, not independent confirmation. Adding a fourth model with the same flyer will not repair the stale source.
Independent evidence is not a personality trait of a model. A second evaluator may use different prose and still depend on the same primary mistake. Independence is relative to the failure mode being studied. Two tools can be independent for arithmetic calculation but share the same incorrectly entered source value. Two human reviewers may independently read an original document yet both apply the wrong intake year. Map the path from claim to source and checking mechanism before using the word independent.
An application’s final assurance should name exactly what was verified. ‘Three models agreed with the draft’ describes consensus. ‘The cited official notice was read directly, its effective date matched the user’s session, and the saved page was independently fetched after publication’ describes stronger, different checks. The latter can still have limitations, but it exposes where evidence came from and how a different reviewer could challenge it.
Return to contents or move to A judge is an instrument that must itself be tested.
2. A judge is an instrument that must itself be tested
A language model can be useful for comparing relevance, spotting omissions and prioritising cases needing human attention. It is not an infallible measurement device. Its output can vary with prompt order, scoring scale, language, verbosity or an adversarial passage embedded in a candidate answer. A trustworthy evaluation programme defines the criterion, tests known positive and negative examples, and records the evaluator’s own failure patterns.
An evaluator’s ability to explain a score does not establish the score’s correctness. A fluent rationale can justify an unsupported decision after the fact. Check whether it cites the correct source span, calculation or tool receipt and whether that evidence really supports the conclusion. If the judge relies on the answer author’s self-assessment, it may become a circular verification loop. The system should not reward a model simply because its explanation sounds like a confident safety report.
Human judgement also needs care. A domain expert might recognise a misleading admissions promise that an automatic rubric misses, while an engineer may detect that a tool result was only queued rather than completed. These are complementary perspectives. A single human vote is not automatically ground truth for every task; the reviewer needs relevant expertise, the actual evidence packet and an escalation route when the question exceeds their authority.
NIST’s September 2026 ARIA Evaluation Planning Manual describes combining model testing, red teaming and user testing in a holistic evaluation approach. Those are distinct vantage points rather than copies of one judge. The guidance does not certify a particular model or dictate one universal panel size. It supports the practical lesson that trustworthy evaluation needs methods designed for the specific system and consequences.
Return to contents or move to Recent research shows why consensus can mislead.
3. Recent research shows why consensus can mislead
Research published by Apple in 2026 under the title ‘Nine Judges, Two Effective Votes’ examined correlations among LLM judging errors on natural-language inference tasks. The reported result was that nine nominal judges gave roughly two independent votes’ worth of information under the study’s analysis. That is not a universal constant for every collection of models. It is evidence that shared error patterns can make the simple number of judges a poor proxy for independent evidence.
Other 2026 research on generator–evaluator agreement, published in Proceedings of Machine Learning Research, examined whether models consistently applied concepts in generation and evaluation. It reported a relationship between some forms of self-consistency and vulnerability to mistakes in the studied settings. The broad lesson is not that agreement is always useless. It is that agreement must be checked against actual correctness and context rather than treated as a safety certificate.
Different models can be useful together when they have complementary strengths or genuinely distinct access to trustworthy information. Even models from separate providers may still share training conventions, benchmark examples or the same retrieval corpus. Diversity should be evaluated empirically. If adding a model only repeats an existing mistake, the relevant independence has not increased. A cheap deterministic check may add more assurance for a numerical claim than an expensive panel of similar text judges.
Research results have scopes and dates. A published finding about a set of inference benchmarks cannot automatically establish how an educational-content workflow behaves, how a medical decision should be made or how every future model family will perform. Link the actual paper, state the setup and use it to motivate a local evaluation design rather than transplanting its headline statistic into a claim of product reliability.
Return to contents or move to Trace dependency through data, prompts and decisions.
4. Trace dependency through data, prompts and decisions
Draw an evidence-dependency graph for one claimed result. Nodes may include the user’s request, source document, extracted citation, retrieved chunks, generating model, first judge, second judge, a calculation tool and the actual external service. An edge shows dependence on information or a previous output. If two judges both consume the same generated rationale and no independent original source, their agreement is downstream of that one rationale. A source error can flow into every branch.
Dependency can arise from apparently harmless design choices. A shared evaluation prompt might tell all models to prioritise the most recent snippet, even when the snippet concerns the wrong audience. A common parser might drop a footnote. A caching layer might serve an obsolete revision. The same category label can be produced repeatedly because the entire panel saw identical incomplete evidence. Fixing prompt variety alone would not address the common parser or stale cache.
Use versioned source identities, not only links. A URL may change. If a factual check depends on the September revision of a policy, record that revision or an appropriate protected evidence snapshot. A new retrieval result should not silently rewrite what the generator saw during the original task. To evaluate the historical output, compare against what was available and applicable then; to give current advice, check the currently effective source. These are distinct questions.
Treat the tool result as a separate branch of evidence. A model might say a file was saved; another model might agree that the message is convincing. Neither knows the actual state unless the application independently fetches the resource. A readback by immutable ID, expected revision and status can break that circular chain. This is where a formal acceptance test often contributes more than another natural-language vote.
Return to contents or move to Choose a reference standard appropriate to the task.
5. Choose a reference standard appropriate to the task
For straightforward arithmetic, an independently written calculation or substituting a solution into the original equation can serve as a reference. For an official timetable, the responsible organiser’s applicable current notice is a stronger reference than a model consensus. For a file update, the stored resource and metadata are primary operational evidence. For a complicated interpretive question, qualified human review may be needed, and genuine uncertainty may remain.
Do not call a convenient answer key infallible. A manually labelled set can contain mistakes, out-of-date policy or ambiguous cases. Record who established the reference, what evidence they used, how disagreements were adjudicated and which versions were in force. A correction to the reference should preserve the historical evaluation and explain which result changed. Otherwise an evaluator may appear to improve simply because its target shifted.
Some tasks lack a unique best answer. A child’s explanation of a Mathematics concept may be valid in several forms. A creative introduction can be clear without matching one expected sentence. In such cases, define essential constraints and a human-usable rubric rather than expecting one exact textual output. Correctness still requires factual and permission boundaries; flexibility in wording should not become a loophole for unsupported claims.
Establish an abstention state when evidence is insufficient. A current official source may be inaccessible, or two equally authoritative notices may conflict. A responsible judge can label the case unresolved and request further verification. Scoring an invented answer as correct merely to complete the dashboard is worse than reporting uncertainty. The user benefits from knowing which fact remains unconfirmed and which next step can resolve it.
Return to contents or move to A fictional three-judge, one-hundred-case laboratory.
6. A fictional three-judge, one-hundred-case laboratory
Consider an invented panel evaluating one hundred short educational AI answers against a carefully checked reference set. Judges J1, J2 and J3 each make exactly twenty errors. Fourteen errors are shared by all three judges on the same fourteen cases because they all rely on one flawed source or assumption. Each judge makes another six errors on a distinct set of six cases, with no overlap among those remaining eighteen mistakes. All other cases are marked correctly by all three. The numbers are chosen for transparent arithmetic, not to describe real judge behaviour.
The hundred cases partition into sixty-eight where all three are correct, fourteen where all three are wrong, and eighteen where exactly one judge is wrong and the other two are correct. Check: 68 + 14 + 18 = 100. Total individual error judgements are 14 × 3 + 18 × 1 = 42 + 18 = 60, or twenty errors per judge across three judges. Each judge is therefore correct on eighty of the hundred cases under the fictional reference.
The panel is unanimous on sixty-eight all-correct and fourteen all-wrong cases, giving 68 + 14 = 82 unanimous cases. But fourteen of the eighty-two unanimous results are wrong. The error fraction among unanimous results is 14 ÷ 82 ≈ 17.07%. A claim that ‘unanimous cases are certain’ would be plainly false in this toy data. Their agreement does not resolve the shared-source error that created the fourteen cases.
With majority voting, the eighteen split cases are decided correctly because two judges are right. The fourteen unanimously wrong cases remain wrong. Majority-correct decisions total 68 + 18 = 86, and majority-wrong decisions total fourteen. Thus the panel’s observed majority accuracy on this invented set is 86%. It improves over the eighty-percent individual judge accuracy, but its remaining failures are concentrated exactly where all judges share the same error.
Return to contents or move to Compare observed errors with an independence assumption.
7. Compare observed errors with an independence assumption
Suppose, counterfactually, that each judge independently made an error on a given case with probability 0.20, and that the simplifying probability applied identically across cases. Then the probability that all three make an error on the same case is 0.20 × 0.20 × 0.20 = 0.008, or 0.8%. That calculation requires genuine independence of the error events. It is not a statistical description of the deliberately correlated fictional panel.
In the toy panel the triple-error fraction is 14 ÷ 100 = 14%, far larger than the independent 0.8% reference. That does not prove a particular deployed ensemble has identical correlation; it shows how dramatically shared sources can violate a naive multiplication rule. The observed concentration points the investigator toward the common evidence, source parser or rubric rather than merely toward changing a model’s temperature.
For three identical independent judges with individual error probability p = 0.20, a majority is wrong if at least two judges err. The probability is 3 × p² × (1 − p) + p³ = 3 × 0.04 × 0.80 + 0.008 = 0.104, or 10.4%. Our correlated fictional majority-wrong fraction is 14%. This comparison illustrates the possible loss of benefit from dependent errors; it is not an estimate of any real service’s future performance.
Independence assumptions also become harder when tasks are clustered. Several questions may come from the same outdated official notice or use the same confusing table. The model’s errors can be correlated across items as well as across judges. A simple binomial interval that treats every rating as an independent coin flip can give misleading confidence. An evaluation plan should preserve case families and design uncertainty calculations around the actual sampling and dependency structure.
Return to contents or move to A small table makes false consensus visible.
8. A small table makes false consensus visible
Show three disposition categories rather than a headline score: unanimous correct, unanimous wrong and split with one wrong. The first two look identical to a system observing only whether judges agree. Only the reference evidence distinguishes them. That is why a dashboard must pair agreement with correctness and the source of the correctness label. Without that separation, more agreement can produce a worse feeling of false certainty.
Add a column for the next action. Unanimous-correct cases can inform ordinary regression testing. Unanimous-wrong cases demand examination of common sources, rubric interpretation and missed facts. Split cases can reveal complementary strengths and may be useful in calibrating arbitration. The correct response to disagreement is investigation, not automatically declaring the majority correct in a high-consequence matter.
Readers can calculate the key measures themselves. Eighty-two of one hundred cases are unanimous, but only sixty-eight of one hundred are both unanimous and correct. The latter fraction is 68%. Fourteen of one hundred are confidently wrong together. A board slide that shows ‘82% judge consensus’ while omitting the fourteen shared failures would omit essential risk evidence. A longer bar is not a substitute for the right denominator.
A table is also easier for students to reason about than claims that consensus is always beneficial or always useless. A panel can recover some isolated individual mistakes while failing on shared error modes. The engineering task is to keep the benefit where it is real and strengthen the common failure path. That often means an independent source or executable tool check, not a fourth near-identical language model.
| Reference result | Cases | Panel pattern |
|---|---|---|
| All judges correct | 68 | Unanimous correct |
| All judges wrong | 14 | Unanimous wrong |
| One judge wrong | 18 | Split; majority correct |
| Total cases | 100 | Eighty-two unanimous |
Return to contents or move to Validate independence instead of assuming different labels imply it.
9. Validate independence instead of assuming different labels imply it
Two judge names or two API credentials do not guarantee independent error processes. If both receive the same flawed extraction, they may fail together. If both use a popular benchmark as prior knowledge, they may reproduce the same outdated answer. If one judge is shown another judge’s rating before deciding, social or algorithmic influence can further reduce independence. Record the evidence and timing each judge actually received.
Try controlled perturbations. Replace an incorrect source with the verified original while keeping the candidate answer constant. Does the panel change its rating appropriately? Remove the generator’s persuasive self-explanation so judges inspect the actual evidence. Change irrelevant order or wording to see whether a judgement depends on presentation rather than correctness. These tests diagnose mechanisms but must preserve the decisive facts of the task.
Independent arithmetic is a useful low-cost comparison. If an AI explanation claims 17 of 20 students is 75%, a calculator or simple division returns 17 ÷ 20 = 85%. Three models praising the writing cannot overrule the number. For an external file action, an authenticated readback after the operation is stronger evidence than two models agreeing that its success message is likely genuine. Test design should match the type of truth being claimed.
Diversifying evaluators can help, but its effect must be measured on relevant reference cases. Different model families may have distinct failure patterns; human reviewers may bring domain expertise; deterministic validators cover exact constraints. Correlation can remain even across different providers, so diversity is not a magic switch. Retain a held-out set of validated cases that detects whether the new panel actually finds errors previously missed.
Return to contents or move to Use separate verification gates for different claims.
10. Use separate verification gates for different claims
An educational article may make factual claims, perform calculations, cite sources and trigger publication. Give each class its proper gate. For factual claims, verify the source’s actual passage, scope and effective date. For calculations, independently recompute with correct units and denominator. For navigation, confirm every anchor points to an existing unique heading. For publication, fetch the post by ID and verify the saved status, slug, body and metadata.
The results should not be averaged into one vague quality score. An excellent prose review cannot compensate for publishing under the wrong account. A correct article cannot be called published while it remains a draft. A tool response that says ‘accepted’ may be only an asynchronous queue receipt. Define critical invariants whose failure blocks the action regardless of how many soft quality dimensions pass.
Some checks should occur before the model runs. Source access permission is best enforced before private documents enter context. A reader’s deadline should be resolved against the appropriate timezone and effective policy. Other checks occur after generation: calculations, attribution, readability and an independently observed external result. Putting each check at the right boundary can remove entire classes of false consensus from the later review panel.
Remember that an independent verifier can still fail. A code test may encode the wrong requirement; a primary official page can be obsolete for the user’s case; a person may misread a figure. Use provenance, versioning, negative tests and a route to correction. Independent verification is an ongoing discipline of making errors detectable, not a metaphysical guarantee of absolute truth.
Return to contents or move to Prevent shared-source mistakes from propagating.
11. Prevent shared-source mistakes from propagating
Start with the authoritative source map. Identify original issuers, revision chains, mirrors, extracted snippets and summaries. If five results are reproductions of one old notice, they are five retrieval hits but only one underlying factual origin. A model panel should not count them as independent corroboration. A source can be useful context without owning the actual decision; distinguish primary authority from explanation.
Date and scope matter. A university may publish last year’s indicative profile and this year’s formal eligibility rules. Both can be genuine documents while addressing different questions. A panel that agrees on the numerical profile may still be wrong to promise current admission. The fix is a claim-specific scope check, not a more elaborate consensus algorithm. The conflicting evidence guide develops that method.
Protect the evidence packet against contamination by the generated conclusion. If a model produces a confident statement, then another model summarises it and a third model verifies the summary, no independent source has entered the chain. The three stages have amplified one initial assertion. Require reviewers to see the original source or actual tool record separately from the proposed answer, with clear roles for each.
Keep a correction route. When a shared source is found wrong, identify which outputs and evaluations relied on it, mark their reference versions and recheck the affected claims. A silent cache refresh can leave old published answers appearing freshly verified if timestamps are reused carelessly. A responsible organisation preserves what was known when the first decision was made and explains how the correction changes the current result.
Return to contents or move to Model judges are useful as triage, not final authority.
12. Model judges are useful as triage, not final authority
A judge model can identify a possibly unsupported claim, highlight contradictory citations or group cases for a human reviewer. That can reduce repetitive work when the task is well bounded and the evaluator’s limits are known. Use it as a diagnostic input with a visible uncertainty state. Do not quietly promote its score into an authorisation to send, delete, publish or make a high-stakes institutional decision.
Evaluate the judge’s own behaviour. Include examples that look fluent but contain wrong units, obsolete dates, unsupported eligibility promises, cancelled permissions and accepted-but-incomplete tool operations. Check not only average agreement with a labelled set but the cost of false acceptance on consequential cases. A judge that reliably praises plausible explanations while missing material conditions needs a targeted repair or narrower role.
Human reviewers need an appropriately designed packet. Show the task, source version, permission and observed effect as separate fields, without immediately revealing other reviewers’ opinions. This reduces anchoring and supports independent first-pass judgement. Record adjudication separately; overwriting original labels to manufacture improved agreement destroys useful diagnostic information.
Use risk-based escalation. Routine stylistic choices can tolerate a flexible rubric and light review. A message to a real parent, an access change or a claim about current university selection may require direct authoritative evidence and a qualified person. A threshold chosen for one task does not become a universal rule for every domain. The decision owner should state the consequence and required proof before looking at the panel’s preferred answer.
Return to contents or move to Transfer exercise: a disputed school event notice.
13. Transfer exercise: a disputed school event notice
Consider an invented school event that has a current official notice stating 10:00. A cached earlier flyer says 09:00. Three model judges all endorse an assistant answer of 09:00 after seeing only the cached flyer. Their agreement is complete and their answer is wrong for the current event. The first task is not to average the times. It is to identify which document is authoritative and applicable, then revise the source context.
After a reviewer fetches the current official notice directly, the correct answer becomes 10:00 with a link to the applicable source and any necessary qualification. The model’s earlier answer should be retained as an error in the evaluation ledger rather than rewritten as though it never occurred. This permits a later test of whether the retrieval pipeline now surfaces the right revision and whether a different judge can detect a stale claim.
Suppose a fictional sixty-case evaluation has forty-three unanimous correct cases, five unanimous wrong cases and twelve split cases. Of the twelve splits, nine are majority correct and three majority wrong against a verified reference. Majority-correct total is 43 + 9 = 52, and majority-wrong total is 5 + 3 = 8. Thus 52 ÷ 60 ≈ 86.67% majority correctness in this invented packet. Unanimous cases total forty-eight, with five wrong, so 5 ÷ 48 ≈ 10.42% of unanimity is erroneous.
Now change the issue from factual text to publication state. The assistant says the school notice has been published, and all three judges praise the response. The actual WordPress record remains a draft. The appropriate oracle is the authenticated saved post ID and status, not the agreement panel. The method transfers because it asks which independent evidence would settle the specific user-facing claim.
Return to contents or move to Statistical limits and transparent reporting.
14. Statistical limits and transparent reporting
Small reference sets can expose failure mechanisms without giving precise population estimates. A hundred deliberately selected cases may overrepresent hard questions or shared-source errors. That makes them useful for stress testing but not necessarily representative of all future traffic. State the selection method and evaluation window. If the dataset is changed after inspecting errors, do not present the resulting score as a preregistered unbiased estimate.
Judges’ errors may correlate across items as well as between judges. Several answers may reuse the same outdated source or same prompt template. A confidence interval that treats every label as independent can be too narrow. Cluster-aware analysis and held-out cases may be appropriate, but they require adequate data and statistical expertise. The article’s toy calculations illustrate mechanisms; they are not a substitute for a proper study design.
Consensus scores can also change with label prevalence. If nearly every example is an easy positive, panels may agree frequently without being effective at detecting the rare dangerous negative case. Report confusion categories and high-severity failures separately from overall agreement. A false approval to transmit private data matters differently from disagreement over a harmless adjective. Design the evaluation for decisions, not just attractive percentages.
Disclose what has not been tested. A review may assess text but not screen-reader usability, source freshness after six months, connected tool permissions or recovery from a network outage. The user should not infer that a model’s language rating covers those absent dimensions. NIST’s evaluation planning guidance encourages context-specific testing approaches; a trustworthy assurance statement names its scope and limitations.
Return to contents or move to Practical repairs for false consensus.
15. Practical repairs for false consensus
Failure one: three judges approve an outdated school policy because all consume the same retrieved summary. Repair the source revision and scope checks, then rerun the factual test with the original responsible notice. Adding identical judges without changing the information path will not address the shared error.
Failure two: the generator writes its own favourable rationale, and judges repeat the rationale rather than checking the actual task. Repair by separating candidate output from independent reference evidence and requiring claim-specific checks. A self-consistent explanation should not be accepted as a verified fact merely because it is detailed.
Failure three: a panel approves a tool operation because the response says ‘done’, while the external record is missing. Repair through stable operation identity and authenticated readback. If the operation’s outcome is uncertain, preserve that status and reconcile before any retry. No majority vote can replace evidence from the destination.
Failure four: a human reviewer accepts the panel verdict before reading the evidence and becomes anchored to its score. Repair through independent first-pass review and delayed disclosure of other ratings where appropriate. Record the human’s reason and relevant source span. A correct-looking majority can still deserve an override when the evidence contradicts it.
Failure five: the application reports a strong aggregate score despite one recurring unauthorised action. Repair the release contract with non-compensatory permission gates, risk-specific reporting and clear stop conditions. Some failures require immediate containment independent of their percentage within a test suite.
Return to contents or move to A useful verification checklist for everyday AI work.
16. A useful verification checklist for everyday AI work
Before relying on a factual answer, identify the actual claim, source, audience, date and relevant exception. Ask whether the cited material supports the conclusion rather than merely containing related vocabulary. For Mathematics or Science calculations, compute independently and check units. For a school decision, verify the responsible institution’s current instructions. A second AI answer can help suggest questions, but should not replace these checks.
Before relying on a completed action, identify the exact target and intended change. Fetch the saved resource or trustworthy provider record after the tool call. Compare title, ID, revision, content, recipient and permission as relevant. A model that says ‘I have completed it’ is reporting its own narrative, which may be mistaken even when sincerely generated. Independent readback turns a claim into inspectable evidence.
For learners, a simple exercise is to give three fictional assistant responses the same wrong starting number and ask why all three calculated the same wrong total. The arithmetic might be perfect while the source number is wrong. Then have the learner inspect the original data and correct the result. This teaches the difference between consistency and truth in a form that transfers to research, media literacy and practical decision-making.
For system builders, create a reference bank of exact failures: stale source, wrong denominator, invalid authorisation, masked private information, ambiguous provider status and successful readback. Test each verification layer against these cases. Keep an account of what the layer cannot catch. A disciplined small set of independent checks can be more valuable than an elaborate panel that shares one hidden assumption.
Return to contents.
Frequently asked questions
If three models agree, is the answer probably correct?
Agreement can be useful evidence only within a validated task context. Shared data, prompts or biases can make all three wrong together; check actual reference evidence.
What is an independent verifier?
A check with a meaningfully different failure path for the claim at hand, such as a source-original reading, deterministic calculation, actual saved-state readback or qualified human assessment.
Can different model providers still share errors?
Yes. Research has documented correlated judging errors even across model families. Measure independence on relevant validated cases rather than assume it from branding.
Does a unanimous wrong result mean model ensembles are useless?
No. They may correct isolated mistakes but leave common-mode failures. Use them where they add value alongside direct reference and action checks.
Why is a saved tool record stronger than the model saying the action completed?
The target system provides observable operation state. A natural-language claim may describe intention or an accepted request rather than the verified final effect.
Is a high agreement percentage an AI safety certification?
No. It measures agreement on a defined sample. Factual validity, authorisation, user impact and untested conditions require other evidence.
Research and primary sources
Sources checked 24 October 2026. The Apple, ICML and NIST publications have their own researched scopes and dates; the hundred-case and sixty-case panels here are entirely invented to teach arithmetic and critical reasoning, not claims about any current commercial model.
- Apple Machine Learning Research — Nine Judges, Two Effective Votes: June 2026 study of correlated judge errors and limited effective independence in its evaluated panels.
- ICML 2026 — The Consistency Dilemma in LLMs: Generator–evaluator self-consistency and its limitations in studied settings.
- NIST ARIA Evaluation Planning Manual (September 2026): Model testing, red teaming and user testing as complementary evaluation perspectives.
- NIST AI RMF Playbook — Measure: Contextual measurement, uncertainty, human assessment and evidence limits.
- W3C PROV-O: Description of source relationships, derivations and accountable evidence chains.
Related learning routes
Read onward through Super Intelligence curriculum; How X Works hub; Reviewer Calibration; Agent Test Oracles; Conflicting Evidence; Evidence Ledgers; Gold Standard of Education. Reuse the habit of separating model agreement from independent source, computation, authority and real-world outcome evidence.