Super Intelligence | Healthcare | Discovery, Clinical Evidence and Access to Care

Super Intelligence master guide · What Is Super Intelligence?

1. Start with the claim, not the excitement

An announcement says that artificial intelligence has made a medical breakthrough. Before deciding whether the news is encouraging, exaggerated or simply incomplete, ask what actually changed. Did a research team identify a candidate worth studying? Did software perform well on stored data? Did a clinical service improve an outcome in a prospective study? Did more people receive appropriate care? These are different achievements. Each can matter, but evidence for one does not automatically establish the others.

This guide is for readers who want to understand those differences without becoming clinicians or machine-learning researchers. Its purpose is evidence literacy: reading studies, evaluating headlines, checking numerical claims and asking better questions about access. It is not a guide to diagnosing symptoms, interpreting an individual’s test result, choosing treatment or running a clinical service. Those decisions belong with appropriately qualified professionals and the relevant care process. The invented exercises below use administrative or abstract research tasks so that the reader can practise evaluation without making a medical decision.

In this eduKate series, practical Super Intelligence, or SI, is an editorial umbrella for contemporary AI-assisted work. Technical artificial superintelligence, or ASI, is a hypothetical capability substantially beyond human performance across a broad range of important domains. An AI system that excels at one medical task has not thereby demonstrated ASI. Nor does a hypothetical future capability establish the safety, effectiveness, regulatory status or availability of a product today. Keeping these meanings separate prevents a powerful name from doing the work that evidence should do.

The most useful first sentence about a system is specific. It identifies the version, the inputs, the people or setting studied, the comparison, the measured result and the remaining uncertainty. For example, an invented software service might help staff find incomplete appointment requests. A report that it identified eighty percent of incomplete requests in one archive says something testable. A statement that it transforms healthcare says very little about what happened to patients, staff or the people who never reached the service.

Think of a healthcare claim as a parcel moving through several checkpoints. A parcel can leave the warehouse without arriving at the intended address. Equally, a promising computational result can leave a research laboratory without becoming an effective service. The relevant question is not whether the first checkpoint was valuable. It is which checkpoints were reached, which were not measured, and whether the final recipient received anything useful. This guide follows that chain from discovery to evidence to access.

You will learn to separate five questions: what the system predicts, whether that prediction is dependable in the intended setting, whether using it changes decisions or work, whether those changes improve meaningful outcomes, and who can actually benefit. A good answer can be optimistic and bounded at the same time. Saying that a result supports a particular next study is a positive conclusion when that is what the research establishes. It does not need to be inflated into a promise of universal clinical benefit.

For an introductory discussion of bounded adoption and review, read Healthcare Promise and the Evidence It Requires. This guide goes further into interpreting discovery claims, comparing study designs, working through denominators and examining the difference between a capable tool and care that reaches people.

Contents · Next chapter

2. Follow the evidence chain without turning it into a rigid ladder

Begin with discovery-stage evidence. A model might identify a relationship in published work, predict a molecular structure or rank hypotheses. The output can make a question more tractable. It is not yet evidence that changing a person’s care will help them. Even an experimentally supported mechanism leaves questions about relevance, alternative explanations and consequences. Discovery is the beginning of a more focused investigation, rather than a certificate that all downstream uncertainty has disappeared.

Technical evaluation asks whether a defined output agrees with an appropriate reference under stated conditions. For a classifier, this may mean comparing its labels with independently established labels. For a summariser, it may mean checking whether source statements were preserved. Technical evaluation can reveal important errors. However, matching a reference in stored data is different from demonstrating benefit when people use the output in a live setting.

External evaluation changes the setting in a meaningful way. It may use a different institution, population, time period, equipment or data process. The word external deserves scrutiny: a dataset from another building may still share the same recruitment, software and documentation habits. External evidence is valuable because it tests whether an earlier result travels. The strength of that test depends on which differences were actually present and how independently the evaluation was conducted.

Prospective evaluation defines a procedure and observes new cases as they occur. This can expose timing, missing-data and integration problems that a cleaned archive conceals. Prospective does not automatically mean randomised or comparative. A prospective observational study can provide useful performance evidence while leaving open whether adopting the system caused better outcomes. Conversely, a carefully designed retrospective study can be very informative about a narrow technical question. The design should fit the claim rather than serve as a prestige label.

An impact study asks what changes when the system is used. The unit under evaluation becomes a configured intervention: software, staff behaviour, available information, training, review and the actions that follow. A randomised comparison can help distinguish the effect of that intervention from differences between the groups receiving it. The measured effect still belongs to the studied package and setting. It does not automatically isolate the model’s contribution from every other component of the service.

Real-world monitoring follows use over time. It can identify changing inputs, new failure modes and consequences that were uncommon or absent in an earlier study. It is not simply the last rung after which evaluation is complete. Changes in software, workflow or population can send questions back to earlier stages. The same system can have strong evidence for one use, weaker evidence for another, and new uncertainty after an update.

Access cuts across the chain. If the people in a study were already able to reach a well-equipped clinic, the study may not answer whether the intervention helps people outside that clinic. If a tool reduces a staff task but requires expensive equipment, its effects may differ between services with different resources. Evidence about access therefore needs its own population and denominators; it cannot be inferred from an accuracy score.

The chain is a map of questions, not a claim that every system must pass identical experiments in an identical sequence. A research tool, an administrative assistant and a regulated clinical device can require different kinds of evaluation. What matters for a general reader is proportionality. If a headline promises improved patient outcomes, look for outcome evidence. If it promises broader access, look for evidence about the people who could and could not receive the service. If it reports better prediction, allow that narrower achievement to stand on its own.

Previous chapter · Contents · Next chapter

3. Discovery can be important before it becomes a treatment

The AlphaFold 3 research paper reports a model for predicting structures of complexes involving proteins, nucleic acids and other molecular components. Its evaluated achievements concern structural prediction. That is substantial scientific work, but the paper’s prediction results are not themselves a clinical trial demonstrating that a particular treatment improves patients’ health. A reader should preserve that distinction even when a headline places drug discovery in the foreground.

An original fictional packet makes the boundary concrete. Imagine a research team called Harbour comparing two ways to rank abstract candidate structures for a laboratory reference task. The team has an untouched evaluation set of one hundred candidates. Under a predeclared matching rule, its earlier method agrees with the reference on forty candidates and the new method agrees on sixty. The packet includes no patient records, treatment assignment, human outcome data, manufacturing analysis or access study. Every number and organisation in this packet is invented for learning.

The new method has twenty additional reference matches per one hundred evaluated candidates. That is a twenty percentage-point increase in the observed match rate, from forty percent to sixty percent. Relative to the earlier forty percent rate, it is a fifty percent increase. Both descriptions are arithmetically correct, but they answer slightly different questions. Neither means that treatments became fifty percent more effective or that half of previously untreatable conditions can now be treated.

Now inspect what the packet leaves unresolved. How were those hundred candidates selected? Were some near-duplicates of training examples? Was the matching rule chosen before results were seen? Were all attempted candidates included? Did the reference capture the scientific property that mattered? Could a change in the evaluation set reverse the ranking? These are questions about the claim actually made, not reasons to dismiss prediction research as unimportant.

Suppose the team initially examined one hundred candidates but later advertised sixty matches among seventy successful runs, excluding thirty runs that returned no usable result. The visible match rate would become about eighty-six percent, but the denominator would have changed. The original result was sixty matches among one hundred attempts. The revised number describes performance conditional on producing an output. An honest report can show both measures, while making the exclusions and failure handling visible.

A second fictional document states that five of the sixty matched candidates were selected for further investigation. That selection does not establish that all sixty are equally promising. It might reflect availability, cost, investigator judgement or other criteria. Nor does a successful follow-up on one selected candidate establish the overall success rate of the entire system. The chain contains selection decisions, and those decisions affect what later evidence can tell us.

The defensible summary is therefore: the new ranking method produced more reference matches on the stated evaluation set and may justify further independent investigation. The unsupported summary is: the system has delivered better treatments to patients. The difference is the missing bridge between a research measurement and a clinical consequence. No rhetorical confidence can replace that bridge.

The broader Scientific Discovery guide explores research loops and experimental infrastructure. Here the key reading habit is simpler: name the object measured. A predicted structure, a candidate, a laboratory result, a trial outcome and an available service are distinct objects. Progress with one can enable the next without guaranteeing it.

Previous chapter · Contents · Next chapter

4. Ask what the technical evaluation actually tested

An evaluation dataset is a sample of a task, not a miniature version of every future situation. Its construction determines which questions can be answered. A collection assembled from unusually clear records may reveal little about ambiguous or incomplete inputs. A dataset with deliberately balanced categories may be useful for development but differ greatly from the frequency of those categories in practice. The reader needs to know why those cases were selected and which cases were excluded.

Training, tuning and testing have different jobs. Training adjusts the model; tuning helps choose settings or versions; testing estimates performance under a defined procedure. If the test set repeatedly influences which system is selected, it has become part of the development process in an important sense. A final score may then be too optimistic about unfamiliar cases. The concern is not merely whether the exact same file appeared twice. Related records or repeated measurements from the same source can also reduce independence.

Time creates another route to misleading evaluation. Imagine a model intended to flag a problem before a deadline, but its input archive includes a field entered after the problem was resolved. That field may strongly predict the label while being unavailable when a real decision must be made. The evaluation can look excellent and still fail to answer the intended question. Asking when each input became available is often more informative than asking how many model parameters were used.

The reference standard also deserves attention. A label might represent a specialist’s assessment, an administrative code, a consensus procedure or a later observation. Those are not interchangeable. If the model is compared with an imperfect label, agreement with that label may not equal agreement with the underlying reality. If ambiguous cases are excluded, the resulting performance may apply mainly to cases on which everyone already agrees. A careful paper explains how disagreement and uncertainty were handled.

The TRIPOD+AI reporting statement addresses transparent reporting of clinical prediction-model studies using regression or machine learning. Reporting guidance helps readers locate essential information about data, methods and evaluation. It is not a stamp that a model is safe or effective, and a completed checklist does not replace critical appraisal. The practical benefit is that a clearly reported study makes its assumptions and limitations inspectable.

For an ordinary reader, the most productive first pass is to write a small evidence card. Record the intended task, data source, recruitment period, inclusion rules, exclusions, reference standard, system version, comparison and output metric. Put unknown beside any field that cannot be found. Unknown does not necessarily mean that the study is bad; it means the report in front of you does not justify filling in that fact from imagination.

Then compare the evidence card with the headline. If the paper tested archived text in one language but the headline promises reliable support for everyone, the expansion is visible. If the study tested one fixed version but the product has changed, ask for the connection between versions. If the reference label was an administrative category, do not quietly convert it into a confirmed diagnosis. This method keeps the reader’s criticism precise and leaves room to recognise genuine technical progress.

Previous chapter · Contents · Next chapter

5. Worked packet: a confusion matrix for a low-risk administrative task

This entire packet is fictional. QueueCheck is an invented assistant that flags incomplete appointment-information requests for staff review. It does not prioritise clinical urgency, diagnose a condition or recommend treatment. The dataset contains two thousand synthetic requests. Independent manual checking finds four hundred incomplete requests and one thousand six hundred complete requests. The system flags three hundred twenty of the incomplete requests and one hundred sixty of the complete requests. These numbers describe an educational exercise, not a deployed product.

Start by reconstructing all four cells rather than trusting the summary score. A true positive is an incomplete request correctly flagged: three hundred twenty. A false negative is an incomplete request not flagged: four hundred minus three hundred twenty equals eighty. A false positive is a complete request unnecessarily flagged: one hundred sixty. A true negative is a complete request correctly left unflagged: one thousand six hundred minus one hundred sixty equals one thousand four hundred forty. The four cells sum to two thousand.

Sensitivity is the fraction of genuinely incomplete requests that were flagged. It is three hundred twenty divided by four hundred, or eighty percent. Specificity is the fraction of complete requests that were left unflagged. It is one thousand four hundred forty divided by one thousand six hundred, or ninety percent. These percentages use different denominators. Neither alone answers how many flagged requests actually need the missing-information review.

Positive predictive value answers that last question. The system creates four hundred eighty flags in total: three hundred twenty useful flags plus one hundred sixty unnecessary flags. Three hundred twenty divided by four hundred eighty is two thirds, approximately 66.7 percent. In this packet, about one in three flags is unnecessary for the stated task. That is not a contradiction of eighty percent sensitivity or ninety percent specificity. It is another view of the same four cells.

Overall accuracy counts correct results among all requests. The correct results are three hundred twenty plus one thousand four hundred forty, giving one thousand seven hundred sixty. Dividing by two thousand gives eighty-eight percent. A simple rule that never flagged anything would already be correct on the one thousand six hundred complete requests, giving eighty percent accuracy. QueueCheck’s higher accuracy is relevant, but it does not by itself tell us whether the remaining eighty missed requests or the one hundred sixty extra reviews are acceptable.

Now change only the illustrative frequency of incomplete requests. Imagine two thousand requests in another synthetic setting, with forty incomplete and one thousand nine hundred sixty complete. Assume, solely for this arithmetic exercise, that sensitivity stays at eighty percent and specificity stays at ninety percent. The resulting cells are thirty-two true positives, eight false negatives, one hundred ninety-six false positives and one thousand seven hundred sixty-four true negatives. The system generates two hundred twenty-eight flags, only thirty-two of which are true positives.

The new positive predictive value is thirty-two divided by two hundred twenty-eight, approximately 14.0 percent. Overall accuracy becomes one thousand seven hundred ninety-six divided by two thousand, or 89.8 percent. Accuracy has increased while the proportion of useful flags has fallen sharply. The result exposes why an attractive single score can conceal the work created by a system. It also shows why the frequency of the target in an evaluation sample matters.

Do not treat unchanged sensitivity and specificity across settings as an empirical fact. We deliberately assumed them to isolate the arithmetic. Real transfer may change those rates as well, because inputs, users and task definitions differ. The lesson is not that one metric is always superior. It is that each metric answers a defined question, and its denominator must be visible before the result can be interpreted.

A complete packet summary would say that QueueCheck identified 320 of 400 incomplete requests, missed 80, generated 160 unnecessary reviews and correctly left 1,440 complete requests unflagged. That sentence preserves the workload and error counts hidden by an eighty-eight percent accuracy headline. It still does not establish that using the assistant improved care. To reach that conclusion, the evaluation would need to follow what happened after staff received its flags.

Previous chapter · Contents · Next chapter

6. Thresholds, calibration, abstention and hidden subgroup differences

Many systems produce a score that must be converted into an action or a flag. A threshold determines where that conversion happens. Changing it can trade one kind of error for another. There is no universal best threshold independent of purpose, consequences and available review capacity. A threshold chosen for a retrospective experiment may also need different scrutiny from one intended for operational use.

Return to the first fictional QueueCheck packet of two thousand requests. Suppose a second configuration flags 360 of the 400 incomplete requests but also flags 320 of the 1,600 complete requests. Its four cells are 360 true positives, 40 false negatives, 320 false positives and 1,280 true negatives. Sensitivity is now ninety percent; specificity is eighty percent. There are 680 flags, of which about 52.9 percent identify an incomplete request. Accuracy is eighty-two percent.

Compared with the first configuration, this version catches forty additional incomplete requests and creates one hundred sixty additional unnecessary reviews. The total number of flags increases by two hundred. Whether that trade-off is worthwhile cannot be settled by declaring that sensitivity improved. The evaluation needs a reason for valuing the extra detections, an account of review effort and a check that increased workload does not undermine handling of the requests already in the queue. This is an administrative illustration, not a recommendation for clinical thresholds.

Calibration asks a different question from ranking. If a system assigns a group of cases a stated probability, how often does the target actually occur in that group? A model can rank cases usefully while overstating or understating their probabilities. A confidence-sounding number on a screen should therefore not be treated as a reliable probability without appropriate evaluation. Likewise, a generated sentence saying that the assistant is certain is not a substitute for an empirical calibration assessment.

Abstention changes the denominator too. Suppose an invented service receives one hundred requests, declines to process twenty and reports ninety percent accuracy on the remaining eighty. It has seventy-two correct processed requests and eight incorrect processed requests. Its processing coverage is eighty percent. Seventy-two percent of all incoming requests were both processed and correct. Reporting only the ninety percent figure conceals the unresolved twenty; reporting only seventy-two percent can conceal that abstention was a deliberate boundary. Both facts are useful when clearly labelled.

Aggregate performance can also conceal uneven results. Split the original two thousand QueueCheck requests into invented format groups A and B. Group A has 1,600 requests, including 320 incomplete ones. It has 288 true positives, 32 false negatives, 128 false positives and 1,152 true negatives. Group B has 400 requests, including 80 incomplete ones. It has 32 true positives, 48 false negatives, 32 false positives and 288 true negatives. Together these reproduce the original four-cell totals exactly.

Sensitivity is ninety percent for format A but forty percent for format B, despite an overall sensitivity of eighty percent. Specificity is ninety percent in both groups. No demographic inference is intended: the groups represent two fictional document formats. The point is that an average may describe neither group’s experience particularly well. A report that only provides the aggregate would leave a reader unable to see where most missed incomplete requests occur.

These calculations do not justify selecting a threshold or deploying QueueCheck. They show what information a responsible reader would request next: uncertainty around the estimates, reasons for subgroup differences, the frequency and handling of abstentions, and the consequences of each type of error. The strongest numerical summary is one that makes disagreement and limitation easier to investigate, rather than one that compresses them out of sight.

Previous chapter · Contents · Next chapter

7. External and prospective evidence answer different questions

Imagine that QueueCheck’s developer tests its system on a second set of records from the same archive. This provides more evidence than showing a handful of successful examples, but the archive may preserve the same conventions and omissions as the first set. Testing records from a different service can reveal additional problems. Testing new requests as they arrive can reveal still others, such as delays, unexpected file formats and inputs that were silently repaired before they reached the archive.

External and prospective are therefore separate descriptions. External concerns the relationship between development and evaluation settings. Prospective concerns how new observations are collected under a defined plan. A study can be external and retrospective, prospective at a development site, or both external and prospective. None of those phrases alone tells us whether using the output caused an improvement. A reader should identify the study’s actual question before ranking its design by a single adjective.

Real research illustrates why this precision matters. A 2021 external validation of the original Epic Sepsis Model studied an adult hospital cohort at Michigan Medicine and reported substantially weaker performance than the developer’s reported performance. Its retrospective design and particular setting are part of the finding. It is an example of why independent evaluation matters, not proof that every future version of that model or every sepsis prediction system must perform identically.

A 2026 multicentre prospective validation of an updated model examined data from four US health systems. It reported improved discrimination while also identifying substantial variation across institutions, low positive predictive values and high alert burden. The two papers concern different versions and evaluations. Reading the newer evidence alongside the older study prevents a reader from freezing a product’s history at either its most favourable or least favourable result.

The general lesson is that model identity includes time and configuration. An evaluation of version one cannot simply be relabelled as evidence for version two. Equally, evidence that a later version improved a measured property should not erase unresolved questions about the intervention’s consequences. The reader needs the chain connecting software changes, evaluated settings, performance measures and actual use.

Prospective studies can also differ in whether people see the output. In a silent evaluation, the system may operate on live incoming information without influencing staff decisions. That can be useful for checking timing and technical behaviour. However, it cannot directly measure how users respond to alerts, whether the interface changes attention, or whether acting on the output improves outcomes. When a report describes prospective validation, ask whether the tool’s output entered the care process and what happened next.

An impact study requires a comparison that fits the question. If staff using AI also receive additional training, more time and better information, the study may estimate the effect of that complete package. That is not necessarily a flaw. It becomes misleading only when the result is described as if the model alone produced the change. Fair interpretation preserves the intervention actually tested.

For independent reading, write two short sentences after the methods section. First: this design can help answer the question of whether a specified thing happened under specified conditions. Second: this design does not by itself establish the broader claim that remains unmeasured. Those sentences are often enough to reveal the difference between a useful study and an overextended headline without pretending to conduct a full specialist review.

Previous chapter · Contents · Next chapter

8. Worked packet: absolute effects, relative effects and missing outcomes

This packet is entirely invented. A community service tests an administrative reminder process for adults who have already requested routine appointment information. Both groups retain the ordinary service and staff support. The new process combines a reviewed AI-assisted message draft with a redesigned reminder schedule. The outcome is whether the request remains administratively unresolved after thirty days. No diagnosis, treatment, clinical prioritisation or health outcome is measured. The exercise is about interpreting study results, not recommending a service design.

Assume a properly randomised comparison assigns one thousand requests to the ordinary process and one thousand to the new process. The packet reports complete thirty-day administrative follow-up for everyone. Forty requests remain unresolved in the ordinary-process group, compared with thirty in the new-process group. The observed unresolved-request rate is four percent versus three percent. The absolute reduction is one percentage point, equivalent to ten fewer unresolved requests per one thousand assigned.

The relative reduction is the one percentage-point difference divided by the original four percent rate. That equals twenty-five percent. Saying that unresolved requests were reduced by twenty-five percent is mathematically compatible with saying that ten fewer of every thousand requests remained unresolved. The second description supplies the baseline scale that the first leaves out. Neither means that twenty-five percent of all people received an additional appointment, or that clinical outcomes improved by twenty-five percent.

The relative risk is three percent divided by four percent, giving 0.75. Its interpretation concerns this defined administrative outcome over this thirty-day window. If the endpoint were successful resolution instead, the rates would be ninety-six and ninety-seven percent. The same data would then show a one percentage-point increase in resolution, with a much smaller relative increase. The choice of framing changes the emotional impression without changing the underlying counts.

The observed difference is an estimate, not a guarantee. The packet does not supply a statistical analysis, confidence interval, prespecified sample-size justification or assessment of all relevant sources of bias. A reader should not declare a conclusive benefit from the point estimates alone. If a professionally conducted analysis supplied an interval, its width and position would help show how precisely the study estimated the effect under its assumptions. An interval would still not repair an unsuitable comparison or an outcome unrelated to the claim.

The study also cannot isolate the contribution of AI from the redesigned reminder schedule. The randomised comparison, as described, evaluates the whole package. A headline saying that the model alone solved the problem would be stronger than the design supports. A useful next research question might compare the redesigned schedule with and without AI-assisted drafting while preserving the other conditions. That is a question for appropriate study design, not a conclusion already contained in these numbers.

Now consider an alternative version of the packet in which follow-up is missing for fifty requests in each group. The report gives the same forty and thirty unresolved requests only among the nine hundred fifty observed requests per group. Those observed rates are approximately 4.21 percent and 3.16 percent. They should not silently be presented as complete outcomes among all one thousand assigned requests. The missing requests may differ from those successfully followed.

Simple extreme bounds make the missingness visible without pretending to solve it. If none of the missing ordinary-process requests were unresolved, that group’s rate among all assigned requests would be four percent; if all fifty were unresolved, it would be nine percent. The corresponding range for the new-process group would be three to eight percent. Depending on the missing outcomes, the apparent advantage could persist, widen or reverse. These are logical bounds for an invented exercise, not a substitute for a justified missing-data analysis.

The balanced conclusion is therefore conditional. Under the complete-follow-up version, the observed administrative unresolved-request rate was lower by one percentage point in the group assigned to the combined process, but uncertainty, costs and other consequences still require evaluation. Under the missing-follow-up version, the incomplete outcome information adds a further limitation. In neither version has a benefit to health been demonstrated. This is how careful wording preserves a promising result without selling an outcome that was never measured.

Previous chapter · Contents · Next chapter

9. A real access study: identify the endpoint that improved

The ACCESS randomised trial examined an AI-enabled diabetic eye-exam pathway in young people with diabetes. Its reported primary result concerned examination completion: the intervention group completed the examination at the point of care, while substantially fewer participants in the referral group completed an examination within the study window. The study also examined follow-up. Its population, specialist oversight and research context matter when interpreting what the result establishes.

This is an instructive example because examination completion is an access-related outcome, rather than simply a classifier score. A service can have good technical performance and still fail to reach people. The study’s question therefore addresses a different part of the chain from a benchmark that compares image labels. At the same time, increased examination completion should not be rewritten as proof of improved long-term vision for every population. An intermediate service outcome and a later health outcome are distinct measurements.

Notice also what the intervention changes. Bringing an examination into a visit that is already happening can alter the effort required to complete it. A model’s contribution exists within that pathway. A reader who attributes the result only to computational intelligence may miss the practical arrangement that made completion easier. Understanding the pathway can make the evidence more useful, because it reveals which parts another service would need to reproduce or evaluate.

The age range and research safeguards should not be ignored when moving from a trial report to a product claim. Research involving a particular population does not, by itself, establish a product’s authorised use for everyone in that population. Nor does an authorisation for one use establish every research application as routine practice. Clinical evidence, regulatory scope and implementation decisions must be read as connected but separate records.

You do not need to memorise the study’s percentages to use its lesson. Ask what event counted as success, who was included in the denominator, what the comparison group received, and which later outcomes were not measured. Those questions prevent both overclaiming and unfair dismissal. A demonstrated improvement in an access-related step can be meaningful without being the final answer about health, cost or long-term equity.

Previous chapter · Contents · Next chapter

10. The human workflow is part of the intervention

A technically correct output can arrive too late, be displayed ambiguously or reach someone who cannot act on it. An incorrect output can be caught by a well-designed review process, but the existence of a reviewer does not guarantee that this will happen. The effects of AI depend on how information moves through people, interfaces, responsibilities and available resources. Evaluating the model in isolation leaves those connections untested.

The CONSORT-AI extension highlights reporting of the AI intervention, its integration, inputs and outputs, human interaction and errors in clinical trials. These details help a reader understand what was tested. They are especially important when two studies use a similar model but give it different responsibilities. A tool that supplies a draft for checking and a tool whose output directly triggers an action are not the same intervention merely because they share a model name.

Consider a fictional record-organising assistant with no authority to change care. It retrieves an old note and writes that an appointment is scheduled, when the source only says it was proposed. A reviewer who checks grammar may miss the difference. A reviewer who compares the output with the dated source has a better opportunity to detect it. The relevant test is not whether the paragraph sounds sensible; it is whether the workflow preserves the source meaning and provides enough time and information for checking.

Automation bias is one reason that review must be evaluated rather than assumed. A fluent answer can anchor attention, making later evidence seem to confirm the initial suggestion. The WHO guidance on large multimodal models in health identifies risks including inaccurate outputs, bias, privacy concerns and automation bias. Its governance recommendations are not evidence that a particular product works; they identify issues that an evaluation should take seriously.

Workload also moves. A service may save time drafting notes but spend more time checking subtle changes, handling exceptions or correcting downstream records. The right comparison follows the completed task, including review and repair. A report of generation time measures only one component. Even genuine time savings should be interpreted in context: do they become more patient-facing time, additional administrative throughput, reduced staff strain or something else? Those outcomes should be measured rather than assumed.

Disagreement is a revealing test of the workflow. What happens when a staff member believes an output is wrong? Can they inspect the relevant evidence, record a correction and continue through an ordinary process? Are they pressured to accept the tool’s result because challenging it is difficult? A formal statement that a human remains responsible is weak if the person lacks authority, time or usable information to exercise that responsibility.

For general readers, a useful report describes the before-and-after pathway. It explains who receives the input, who sees the output, what action is permitted, where uncertainty is recorded and how unresolved cases continue. It also reports failures that were intercepted, not only failures that escaped. Those details show whether claimed assistance is embedded in a functioning service or merely demonstrated in a convenient test environment.

Previous chapter · Contents · Next chapter

11. Bias begins before the model produces an answer

Unequal performance can arise because a model learned an unhelpful association, but it can also arise because some people are missing from the data altogether. An archive describes people whose interactions produced records. It may say much less about people who could not reach the service, did not understand the invitation, could not complete the required form or left before follow-up. If those missing people are absent from evaluation, the resulting average cannot describe their experience.

Selection is therefore an evidence problem before it becomes a modelling problem. Suppose an invented online service reports high completion among registered users. Registration may itself require a device, reliable connection, literacy in a particular language and time to navigate the process. The result can be accurate for registered users while failing to answer the broader access question. The missing denominator is everyone who needed the service, including those who never registered.

Historical labels can also encode previous decisions rather than independent measures of need. A variable recording past service use may reflect availability, affordability or referral habits. Predicting that variable accurately does not automatically mean identifying people who would benefit most from care. The reader should ask what the target label represents and whether that is the same concept used in the headline. This is a conceptual check, not a claim that every proxy is unusable.

Subgroup analysis should be connected to plausible sources of difference. Language, disability access, document format, equipment and local workflow may each matter for different tasks. A long list of demographic percentages is not enough if the relevant access barrier is unmeasured. Conversely, a model with similar technical performance across recorded groups may still be deployed in a way that creates unequal waiting times or follow-up opportunities. Technical fairness and service fairness overlap without being identical.

Small subgroup samples need caution. If an invented evaluation contains only ten examples of a particular format, a result of nine correct outputs does not establish a stable ninety percent performance rate in every future setting. One additional error changes the observed percentage substantially. A transparent report supplies counts and uncertainty, rather than using the apparent precision of a percentage to conceal how little evidence supports it.

Fairness questions can also conflict. A change that catches more target cases may generate more false alarms, and those burdens may fall unevenly. Equalising one numerical metric does not settle every question of acceptable treatment. Decisions about trade-offs require affected people, domain expertise and accountable governance. A general reader’s role is to make the trade-off visible and ask who participated, rather than declaring one mathematical target a universal ethical solution.

The practical repair for a weak claim is specific. Replace everyone with the population actually studied. Replace unbiased with the dimensions and measures actually evaluated. Replace equal access with evidence about invitation, completion, waiting, follow-up and cost. These edits are not merely cautious language. They identify the next missing observations and prevent a narrow result from erasing people whose circumstances were never included.

Previous chapter · Contents · Next chapter

12. Worked packet: follow people through the access pathway

WHO describes universal health coverage in terms of access to needed quality services without financial hardship. This frames access more broadly than possessing an application or receiving a generated answer. The following original fictional packet turns that distinction into arithmetic. It does not estimate any real country’s performance and does not represent a clinical screening programme.

Imagine two invented districts, East and West. Each has one thousand adults eligible for a routine, nonclinical care-navigation information session. The session explains how to contact an existing service; it does not assess symptoms or select treatment. An AI-assisted scheduling tool is one component of the invitation process. Every number in this packet is invented. Eligibility, invitation and attendance are counted separately, with no assumption that attendance itself improves health.

In East, eight hundred eligible people receive an invitation, six hundred book a session, five hundred attend, and four hundred receive the complete information session. In West, four hundred receive an invitation, three hundred book, two hundred fifty attend, and two hundred receive the complete session. The completion rate among attendees is eighty percent in both districts. A dashboard showing only that statistic would suggest equal performance at the last step.

The population-level reach is different. East reaches four hundred of one thousand eligible people with the complete session, or forty percent. West reaches two hundred of one thousand, or twenty percent. East’s invitation coverage is eighty percent; West’s is forty percent. Among invited people, both districts convert half into completed sessions. The observed disparity therefore appears before or at invitation coverage in this simplified packet, even though later conditional percentages match.

Now imagine a software update improves the completion rate among attendees from eighty to ninety percent without changing earlier counts. East would complete four hundred fifty sessions and West two hundred twenty-five. Population-level reach would become forty-five percent and 22.5 percent. Both improve, but the absolute reach gap grows from twenty to 22.5 percentage points. Improving one stage does not necessarily narrow every disparity, particularly when the earlier stages reach different proportions of the eligible population.

This does not prove that the software update caused a harmful inequality. It shows why the chosen comparison matters. Both the gain in completed sessions and the remaining gap deserve attention. A responsible report would ask why invitations failed to reach people in West: incomplete contact information, inconvenient timing, language, trust, device access or something else. The packet does not provide the answer, so a reader should not invent one from the district names.

Suppose a proposal then adds two hundred invitations in West. It would be tempting to predict another hundred completed sessions because half of earlier invitees completed. That prediction assumes the newly reached people face the same conditions as earlier invitees. They may not. The new group could include people who need different contact methods or additional support. The earlier conversion rate is a starting hypothesis, not a guaranteed marginal result.

Cost should be followed through the pathway too. A service may report lower software cost per completed session while leaving users with additional travel, time away from work or communication costs. Another service may spend more per session but reach people previously excluded. Neither conclusion follows from the packet’s attendance counts alone. The relevant question is which costs were measured, whose costs they were, and what outcome those resources purchased.

The completed evidence statement is: both districts had the same completion rate among attendees, but East reached twice the proportion of eligible people because more eligible people entered the invitation pathway. The appropriate next step is to investigate invitation coverage and the experiences of people outside the observed pathway. The unsupported statement is: the AI tool delivered equal access because its final-stage completion percentage was equal.

For a broader discussion of infrastructure, language and institutional capability, see Small States and the Global South. In healthcare evidence literacy, the transferable habit is to retain each denominator. Count eligible people, reached people, participating people and completed pathways separately before deciding who benefited.

Previous chapter · Contents · Next chapter

13. Read regulatory status as a scoped record

Regulatory words are not interchangeable praise. The FDA describes different premarket pathways for medical devices, including 510(k) clearance, De Novo classification and premarket approval. A reader should use the term associated with the actual device record rather than calling every product approved. The agency regulates relevant devices and functions; the presence of AI in a product name is not itself a complete description of its regulatory status.

The FDA’s AI-enabled medical device list links to product-specific marketing-authorisation information and explains that the list is not comprehensive. Its purpose is not to certify that every possible use of a listed product is supported. Follow the linked record and identify the device, intended use and relevant limitations. A general-purpose assistant should not acquire a medical-device claim merely because another product from the same company appears in a regulatory database.

For a general reader, the practical evidence card has four separate entries: the exact product or function, the jurisdiction, the stated use, and the date or version to which the record applies. If one entry is unknown, leave it unknown. A photograph of a logo, a seller’s slogan or a statement that a company works with hospitals cannot fill the gap. Nor does a research article alone settle the product’s legal status.

Jurisdiction matters. A United States record is evidence about a United States regulatory action. It should not be presented as a universal permission covering another country. People evaluating actual procurement or clinical use need the relevant local regulatory and professional advice. This guide explains how to read claims; it does not determine whether a particular product or use is legally permitted.

Also distinguish an agency’s final guidance from draft guidance, a discussion paper, a research project or a press announcement. These documents serve different purposes. The correct question is what status the agency itself assigns to the document and what scope it addresses. A proposed approach should not be reported as an enacted requirement, and a general recommendation should not be rewritten as a product-specific authorisation.

Regulation and evidence remain connected without being identical. A published evaluation can illuminate a limitation worth investigating even when a device has an authorisation. A promising study can also precede a regulatory decision. Readers should resist both shortcuts: treating a regulatory label as the answer to every question about local benefit, or treating an exciting experiment as permission for unrestricted use. The exact record is more useful than the strongest-sounding adjective.

Previous chapter · Contents · Next chapter

14. Monitor the system, the service and the missing information

The FDA’s postmarket-monitoring research programme explains that acquisition systems, protocols and populations can change across sites and over time. Such changes can affect the relationship between development results and clinical use. Its work on monitoring illustrates why evaluation is an ongoing process. It should not be mistaken for a claim that any particular monitoring dashboard is adequate or that every product adapts automatically.

An original fictional incident shows the distinction between observing a signal and explaining it. QueueCheck’s monthly report says that the number of flagged requests has doubled. That observation alone does not establish that the model deteriorated. There may be more incomplete requests, a new input format, a change in the threshold, a routing error or duplicate records. The investigation needs denominators and configuration history before it can identify the mechanism.

Suppose the total number of requests also doubled, while the proportion flagged stayed unchanged. The workload still matters, but the signal no longer implies a changed flag rate. Suppose instead that total requests stayed stable and a new form appeared on the same date. That narrows the investigation without proving causation. A useful monitoring report retains the timeline, the affected inputs, the version and the uncertainty about the cause.

Outcome labels may arrive later than outputs. A system can generate a flag immediately while confirmation or follow-up takes weeks. A dashboard that only tracks recent known outcomes may overrepresent easy-to-resolve cases. Monitoring should therefore make delayed or missing information visible, rather than quietly counting absence of a recorded problem as success. The same concern applies to complaints: no complaints may mean good performance, but it may also mean that people could not find a reporting route.

Privacy belongs in this evidence chain. A learning exercise rarely needs a real medical record. Synthetic or abstract examples can support the same reasoning practice without exposing a person’s health information. In real institutional work, input data, outputs, logs and incident reports all need an appropriate authorised handling process. Removing a name is not a blanket assurance that a combination of details cannot identify someone. Convenience should not substitute for that process.

An evidence-literate reader can ask what happens after a concerning signal. Is there a named responsibility for investigating it? Can use be narrowed or paused through an appropriate process? Are affected records corrected? Is the revised configuration evaluated before a benefit claim is repeated? These questions do not supply a clinical operating protocol. They identify whether monitoring is connected to accountable action or stops at collecting a number.

Finally, look for the cost of preserving reliability. Software maintenance, staff training, review, accessibility support and fallback processes are part of sustained use. A pilot funded with exceptional support may not represent ordinary operation after that support ends. A credible long-term claim explains which conditions must continue, what has changed since the original evaluation and what evidence remains uncertain.

Previous chapter · Contents · Next chapter

15. Build a claim card and rewrite an overextended headline

Here is a complete original fictional capstone. A promotional paragraph reads: Harbour Health AI has achieved ninety percent accuracy, reduced unresolved requests by twenty-five percent and ensured equal access in two districts. Its breakthrough will deliver better care to everyone. The attached packet contains the structure-ranking experiment from chapter three, the QueueCheck matrices, the complete-follow-up reminder comparison and the two-district information-session counts. There is no clinical outcome study and no product-specific regulatory record in the packet.

The first repair is to separate the systems. Harbour’s research-ranking method, QueueCheck’s request flagging and the combined reminder process are different interventions. Combining their most attractive numbers produces a fictional super-product that was never evaluated. A claim card gives each system its own task, version, dataset, comparison and result. If the promotional paragraph does not tell you which system produced which number, that is a reporting problem to resolve before making a general conclusion.

The second repair is to restore exact metrics. QueueCheck’s first configuration has eighty percent sensitivity, ninety percent specificity and eighty-eight percent overall accuracy. Calling it ninety percent accurate appears to have substituted specificity for accuracy. Its second configuration has ninety percent sensitivity but eighty-two percent accuracy. Neither configuration supports an unqualified ninety percent accuracy claim. The correct account can report all four cell counts so that the reader can reproduce each percentage.

The third repair is to restore the baseline and endpoint. The reminder comparison observed forty unresolved requests among one thousand assigned to the ordinary process and thirty among one thousand assigned to the combined process. That is a one percentage-point absolute reduction and a twenty-five percent relative reduction in a thirty-day administrative outcome. It is not a twenty-five percent improvement in health, and the packet does not isolate the AI component from the reminder redesign.

The fourth repair is to restore the access denominator. East and West each completed eighty percent of attended information sessions. However, completed sessions reached forty percent of eligible people in East and twenty percent in West. Equal final-stage completion is not evidence of equal population reach. The card should record both the conditional measure and the eligible-population measure, then identify invitation coverage as an important unresolved question in this simplified example.

A corrected paragraph would read: In separate invented evaluations, a research-ranking method increased reference matches, a request-flagging assistant showed specified error rates, and a combined reminder process had fewer administratively unresolved requests at thirty days. The district exercise showed equal completion among attendees but unequal reach among eligible people. These packets illustrate different evidence questions; they do not establish clinical benefit, universal access, regulatory status or the arrival of artificial superintelligence.

For a real claim, the completed card should end with a next question, not merely a verdict. If the evidence is discovery-stage, ask what independent evaluation follows. If it is a technical study, ask whether the intended setting matches. If it is an impact study, ask whether the endpoint, uncertainty and comparison support the headline. If it concerns access, ask who remains outside the denominator. This gives the reader a constructive route from scepticism to a more informative conversation.

Previous chapter · Contents · Next chapter

16. Practise independently, then check the answers

Exercise one uses a new invented administrative dataset. There are five hundred synthetic information requests, one hundred of which are incomplete. An assistant correctly flags seventy incomplete requests and incorrectly flags forty complete requests. Work out the four-cell matrix, sensitivity, specificity, positive predictive value and overall accuracy. Before calculating, write the denominator for each metric. This prevents the most common mistake: using all five hundred requests for every percentage.

The answer is seventy true positives, thirty false negatives, forty false positives and three hundred sixty true negatives. Sensitivity is seventy divided by one hundred, or seventy percent. Specificity is three hundred sixty divided by four hundred, or ninety percent. Positive predictive value is seventy divided by one hundred ten, approximately 63.6 percent. Accuracy is four hundred thirty divided by five hundred, or eighty-six percent. The cells sum to five hundred, and the positive labels sum to one hundred; these are useful arithmetic checks.

Exercise two changes the frequency of incomplete requests while keeping the same rates solely for an arithmetic thought experiment. There are now twenty incomplete and four hundred eighty complete requests among five hundred. Assume sensitivity remains seventy percent and specificity remains ninety percent. How many useful and unnecessary flags occur? Does ninety percent specificity mean ninety percent of flags are useful? State the assumption that would need empirical testing in a real transfer.

The answer is fourteen true positives, six false negatives, forty-eight false positives and four hundred thirty-two true negatives. There are sixty-two flags, so positive predictive value is fourteen divided by sixty-two, approximately 22.6 percent. Ninety percent specificity describes correct non-flags among complete requests; it does not describe the proportion of useful flags. The untested transfer assumption is that sensitivity and specificity remain unchanged in the new setting. The calculation demonstrates denominator dependence without establishing real-world transportability.

Exercise three concerns an invented reminder comparison with complete follow-up. Sixty of two thousand requests remain unresolved after thirty days under an ordinary process, compared with forty under a combined new process. Calculate the absolute and relative reduction. Then rewrite the sentence, AI improves healthcare by one third, so that it matches the packet. Do not add an interval or a claim of statistical significance that the packet does not provide.

The rates are three percent and two percent. The absolute reduction is one percentage point, or ten fewer unresolved requests per thousand assigned. The relative reduction is one divided by three, approximately 33.3 percent. A suitable rewrite is: the invented comparison observed a lower thirty-day administrative unresolved-request rate under the combined process, falling from three to two percent; uncertainty and the contribution of the AI component remain to be assessed. A clinical benefit has not been measured.

Exercise four asks whether two districts have equal access. Each has two hundred attendees. District A completes one hundred eighty sessions among one thousand eligible people. District B completes one hundred eighty sessions among two thousand eligible people. Calculate completion among attendees and reach among eligible people. Then name one additional observation needed before explaining why their reach differs. All counts describe a fictional nonclinical information service.

Both districts complete ninety percent of attended sessions. District A reaches eighteen percent of eligible people; District B reaches nine percent. The packet does not tell us why. Invitation coverage, booking barriers, transport, timing or other factors could matter, but none should be asserted without evidence. A defensible next observation would be the number of eligible people successfully contacted in each district, using a consistent definition and time period.

Exercise five is a study-design check. A system runs on new incoming records for three months, but staff never see its outputs. The report claims that this proves using the system improves staff decisions. What is missing? The answer is the link between output and human action. The silent prospective evaluation may test timing and technical behaviour, but it does not observe the effect of staff seeing and using the output. A suitable impact comparison would need to examine that intervention and its relevant consequences.

Exercise six tests regulatory reading. A company points to an FDA database entry for one image-analysis function and claims that all of its conversational assistants are authorised for medical use worldwide. What should a reader check? The answer is the exact product and function, intended use, version or record date, jurisdiction and applicable limitations. The existing entry cannot be assumed to cover a different assistant, a broader task or another jurisdiction. The correct response is to request the relevant record, rather than inventing its status.

If you made an error, identify its type before repeating the exercise. A wrong total calls for a matrix check. A wrong percentage calls for a denominator check. A correct number attached to an exaggerated claim calls for an endpoint check. An explanation that goes beyond the packet calls for a missing-information check. Progress means being able to state both the result and its boundary without looking at the worked answer.

Previous chapter · Contents · Next chapter

17. Questions readers often ask

Can one outstanding healthcare result establish artificial superintelligence?

No single narrow task result establishes broadly superhuman capability across important domains. A system can be impressive and useful within a defined medical or scientific task without demonstrating technical ASI. The distinction also works in the other direction: speculation about future ASI cannot stand in for evidence about a contemporary product. For practical evaluation, name the system and the task before using a broad capability label.

Does a randomised trial settle every question?

A well-designed randomised comparison can strengthen causal interpretation for the intervention, population, outcomes and period studied. It does not make every setting equivalent, remove the consequences of missing data or establish outcomes that were not measured. Read recruitment, the comparison process, follow-up, uncertainty and the intervention details. A trial’s strength is the question it can answer well, rather than an unlimited guarantee attached to its design name.

Should readers reject a study if it reports only a technical outcome?

No. A technical evaluation can be exactly what is needed at a particular stage. The problem is not that the study answers a narrow question; the problem is treating the answer as if it established something broader. A reference-matching result, a workflow study and a patient-outcome study can each contribute useful evidence. Give the study credit for what it demonstrates and leave the unmeasured claims unresolved.

Why do the worked examples avoid diagnoses and treatment choices?

The mathematical and evidential lessons do not require readers to make personal clinical decisions. Synthetic administrative packets make denominators, selection and outcome boundaries visible while keeping the exercise separate from care. The same reading habits can help a person discuss a real study with a qualified professional, but the invented thresholds and counts in this guide are not clinical performance targets or instructions.

Can a free or inexpensive AI service solve access by itself?

Price is one part of access. A service may also require connectivity, a suitable device, language support, trust, time and a functioning route to professional care. A low-cost answer is not equivalent to a completed care pathway. An access claim should explain who was eligible, who could enter the service, who completed it, which costs were counted and whether an appropriate outcome followed.

What should I do with a promising but incomplete claim?

Write the strongest statement the supplied evidence supports, then identify the next missing question. That may be independent replication, evaluation in another setting, a comparative impact study, longer follow-up or better information about excluded people. This is more useful than declaring the technology either miraculous or worthless. Evidence literacy preserves the value of genuine progress while keeping its limits legible.

Previous chapter · Contents · Next chapter

18. The final test: did a supported benefit reach the intended people?

Healthcare AI should be judged through a chain of specific claims. Discovery can produce a better question. Technical evaluation can establish a bounded capability. Prospective and external studies can show how that capability behaves in new circumstances. Comparative impact evidence can examine what changes when it is used. Access and monitoring evidence can show who receives the service and whether its performance remains dependable over time.

The reader’s task is to keep those claims connected without collapsing them into one. Preserve the system version, population, comparison, denominator, endpoint and uncertainty. Ask where review and responsibility sit. Notice the people who were excluded, not reached or lost to follow-up. When a result is promising, say why. When a claim exceeds its evidence, repair the claim precisely. The goal is neither automatic enthusiasm nor automatic rejection, but a clearer path from capability to justified benefit.

Return to What Is Super Intelligence? for the wider definitions and future-facing questions. Continue with the linked healthcare overview for bounded adoption, the scientific-discovery guide for research systems, or the Global South guide for infrastructure and access. The essential question remains practical: what supported benefit reached which people, under what conditions, and what still needs to be learned?

Previous chapter · Contents · Sources

Research and official sources

Sources were checked on 1 October 2026. Links support the nearby factual claims. The Harbour, QueueCheck, reminder and district packets, all their numbers, and the independent exercises are original fictional teaching material. They are not clinical performance targets or findings from real services.

AlphaFold 3 research paper

TRIPOD+AI reporting statement

2021 external validation of the original Epic Sepsis Model

2026 multicentre prospective validation of an updated model

ACCESS randomised trial

CONSORT-AI extension

WHO guidance on large multimodal models in health

universal health coverage

premarket pathways for medical devices

AI-enabled medical device list

postmarket-monitoring research programme