eduKateSG · Judgement · The Gold Standard of Education
Compare prediction models by the decisions they improve
Choose the explanation you need, or follow all twelve chapters in order.
Full chapter index · Independent practice · Judgement subject guide
A prediction model can discriminate well and still be unhelpful at the threshold where a real action is taken. Decision curve analysis asks a narrower question: across plausible threshold probabilities, does using this model produce greater net benefit than treating everyone, treating no one, or using another rule?
The method is useful because it connects false positives and true positives to a decision threshold. It is also easy to misuse. Net benefit is not a universal utility score, and a visually higher curve does not rescue poor calibration, biased data or an undefined action. The examples here are fictional and educational, not medical advice.
Continue from The Gold Standard of Judgement within The Gold Standard of Education: learning that transfers into life and the How X Works learning library.
Full chapter index
See the mechanism · Chapters 1–2
Build the worked model · Chapters 3–5
Test what changes · Chapters 6–8
Check evidence and limits · Chapters 9–10
Practise and continue · Chapters 11–12
Define a binary decision such as offer a costly follow-up test versus do not offer it. Define the outcome and the time horizon. The model estimates a probability; the threshold determines when the predicted probability is high enough to act.
A threshold of 0.20 means the decision-maker is indifferent between acting and not acting when the estimated outcome risk is 20 percent, under the simplified tradeoff represented by the method. It does not mean 20 percent is biologically dangerous in every setting.
If several actions, serious side effects or competing outcomes matter, a simple binary decision curve may be incomplete. Write the decision first so that the curve has a real interpretation.
Evidence check: for Start with the action, not the score, preserve the observation before the interpretation. Record population, binary action, outcome and time horizon. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
At threshold p_t, the relative weight assigned to a false positive is p_t/(1-p_t) compared with a true positive. At p_t=0.20, the weight is 0.25. Four false positives have the same penalty as one true positive benefit in the standard net-benefit expression.
This weight arises from the threshold, not from model performance. Stakeholders with different burdens, costs or preferences may have different thresholds. Show a plausible range rather than selecting the point where a favoured model looks best.
Thresholds must be coherent with the action. If nobody would act below 5 percent or above 40 percent because another pathway takes over, curves outside that range are not decision-relevant.
Evidence check: for Threshold probability encodes a tradeoff, preserve the observation before the interpretation. Record whose tradeoff the threshold represents. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
For N people, true positives TP and false positives FP at threshold p_t, net benefit is TP/N − FP/N × p_t/(1-p_t). True negatives do not appear directly because the expression sets a reference and weights avoidable false-positive actions.
The unit can be read as true-positive equivalents per person. Multiply by 100 for equivalents per 100 people. This interpretation depends on the chosen threshold tradeoff and correctly defined outcome.
Net benefit is comparative. Calculate the model, treat-all and treat-none strategies on the same sample and threshold. Treat-none has net benefit zero in the usual formulation. Treat-all changes with outcome prevalence and threshold.
Evidence check: for Net benefit formula, preserve the observation before the interpretation. Record the formula, denominator and threshold weight. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
In a fictional cohort of 1,000, a model at threshold 0.20 identifies 90 true positives and 160 false positives. The false-positive weight is 0.25. Net benefit is 90/1000 − 160/1000×0.25 = 0.050, or five true-positive equivalents per 100.
Suppose treat-all yields 120 true positives and 880 false positives. Its net benefit is 0.120 − 0.880×0.25 = −0.100. Treat-none is zero. At this threshold and in this sample, the model is better than both default strategies.
Change the threshold to 0.10 and recompute the confusion counts; do not reuse the old counts. More people usually cross the lower threshold. The weight also changes to about 0.111. A decision curve repeats this calculation across thresholds.
| Strategy | True positives | False positives | Net benefit |
|---|---|---|---|
| Model | 90 | 160 | 0.050 |
| Treat all | 120 | 880 | -0.100 |
| Treat none | 0 | 0 | 0.000 |
Evidence check: for Worked calculation at one threshold, preserve the observation before the interpretation. Record TP and FP counts recalculated at that threshold. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
CHAPTER 5 OF 12 · Build the worked model
5. Treat-all and treat-none are essential comparators
A model curve above zero is not sufficient if treat-all is higher. The default strategies express what happens without selective prediction. In a high-prevalence setting and at a low threshold, acting for everyone may be difficult to beat.
Where treat-all crosses zero depends on prevalence. This makes prevalence shift important when transporting a curve. A validation population with a different case mix can change both model performance and comparator curves.
Other realistic comparators can be added: an existing clinical score, a resource rule or a simple age threshold. The relevant question is improvement over current practice, not merely improvement over doing nothing.
Evidence check: for Treat-all and treat-none are essential comparators, preserve the observation before the interpretation. Record treat-all, treat-none and current-practice comparators. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
At each threshold, identify the strategy with the highest net benefit, then look for a sustained and meaningful separation. Tiny jagged differences may be sampling noise. Confidence intervals or resampling can show uncertainty.
Do not choose a threshold after inspecting the same curve unless that selection is validated elsewhere. The apparent optimum can overfit. A prespecified decision range and external validation provide a stronger test.
The attached table uses fictional values to show how the preferred strategy can change. A model may add value from 0.10 to 0.30 and lose value beyond 0.40. The conclusion should name the range.
| Threshold | Model A | Model B | Treat all |
|---|---|---|---|
| 0.10 | 0.071 | 0.074 | 0.067 |
| 0.20 | 0.050 | 0.046 | -0.100 |
| 0.30 | 0.031 | 0.025 | -0.257 |
| 0.40 | 0.006 | 0.011 | -0.467 |
Evidence check: for Read a curve across a useful threshold range, preserve the observation before the interpretation. Record the relevant range and uncertainty around differences. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Thresholding assumes predicted probabilities mean what they say. If a model predicts 20 percent risk for groups whose observed risk is 8 percent, the action threshold is misapplied even if rank discrimination is good.
Assess calibration-in-the-large, calibration slope and plots in the validation population. Recalibration may improve decision usefulness, but it must be evaluated without leaking outcome information from the final test sample.
Decision curves do not make calibration checks optional. They can reveal consequences of miscalibration, but a plausible-looking curve from a biased sample is not proof that probabilities transfer.
Evidence check: for Calibration is a decision property, preserve the observation before the interpretation. Record calibration in the intended validation population. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Net benefit depends on outcome prevalence. A case-control dataset sampled to include an artificial fraction of cases cannot be used naively. The analysis must recover or specify the target prevalence with appropriate weighting.
Verification bias, missing outcomes and selective follow-up also distort TP and FP counts. Handle these in the study design and analysis. A curve is only as representative as the data behind it.
State the target population, inclusion criteria, time horizon and prevalence date. Transport to another service or country requires new calibration and decision-curve evaluation.
Evidence check: for Prevalence, sampling and case-control data, preserve the observation before the interpretation. Record target prevalence and sampling weights. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Building and evaluating a model on the same data inflates net benefit. Use bootstrap correction, cross-validation suited to the development process, and preferably external validation. All feature selection and tuning must occur inside resampling folds.
Curves can be unstable at extreme thresholds because few people are classified positive or negative. Show the number acted upon and event counts. A smooth-looking line can hide sparse evidence.
Compare models using paired resampling because their predictions are made on the same people. Report uncertainty and avoid declaring superiority from one crossing point.
Evidence check: for Overfitting and honest validation, preserve the observation before the interpretation. Record resampling boundaries and external validation. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Standard decision curves simplify consequences into a threshold tradeoff and a binary action. They may not capture downstream cascades, capacity constraints, unequal harms, repeated decisions or patient-specific utilities.
Net benefit is not money, quality-adjusted life years or a moral ranking. Converting it to avoided unnecessary interventions can aid interpretation, but only under the formula’s assumptions. Do not compare net-benefit values across unrelated outcomes as if they share a unit.
If resource capacity is fixed, evaluate a policy that selects a defined number of people. If actions have several levels, use a method that represents them. The correct response to a violated assumption is a better decision model, not decorative caveats.
Evidence check: for Limits of the standard framework, preserve the observation before the interpretation. Record capacity, downstream harms and missing utilities. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Practice one: Model A has higher area under the curve, but Model B has higher net benefit from thresholds 0.15 to 0.25. For a service acting only in that range, Model B may be preferable after calibration and validation checks.
Practice two: a curve from a specialist hospital is applied to primary care where prevalence is much lower. Re-estimate calibration, comparator curves and net benefit in primary-care data before transferring the claim.
Practice three: a model beats treat-all but remains below treat-none across the relevant range. It has no demonstrated decision value there, even if its discrimination statistic appears respectable.
Transfer the idea to non-medical screening by defining outcome, action and threshold tradeoff. The mathematics can travel; the ethical and operational meaning must be rebuilt for the new context.
Evidence check: for Independent practice and transfer, preserve the observation before the interpretation. Record which changed context reverses the preferred rule. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Is a higher decision curve always better?
Only within a relevant threshold range, under valid outcome, calibration and sampling assumptions. Consider uncertainty and practical magnitude.
Does net benefit include every cost?
No. It represents a simplified tradeoff encoded by threshold probability. Add explicit analysis for costs, capacity, side effects or unequal consequences that matter.
Can decision curves choose the threshold for me?
They compare strategies across thresholds. The acceptable threshold range comes from the decision context and stakeholder tradeoffs.
What should accompany the plot?
Outcome definition, cohort, prevalence, calibration, confusion counts or intervention rates, validation method, uncertainty and the relevant threshold range.
Authoritative starting points include the original decision curve analysis paper, the open technical note on net benefit calculations and interpretation, and a step-by-step guide. For real clinical use, follow current specialty guidance and perform local validation.
Continue with robust decisions and assumption reversal, value of information and testing delay, The Gold Standard of Inquiry, and The Gold Standard of Evaluation. The next useful move is to write one decision sentence naming the population, action, outcome, time horizon and threshold range before drawing any curve.
Evidence check: for FAQ, sources and the next useful move, preserve the observation before the interpretation. Record the decision sentence that the curve is meant to support. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.