The Gold Standard of Judgement | Value of Information and When Another Test Is Worth the Delay

Full chapter index

Begin with actions, uncertain states and consequences. An action could be launch now, run a pilot, wait, repair, select a course or order a diagnostic investigation. A state is something not fully known that changes consequences: demand is high or low, a component is faulty or sound, an intervention helps or harms.

Assign consequences on a common decision scale where possible. Money is convenient but not universal. Health, safety, learning time, equity and reversibility may need multi-criteria treatment. Value-of-information calculations are only as sound as the consequences and probabilities supplied; they do not convert an omitted concern into zero.

The current best action maximises expected consequence under present information. Uncertainty alone is not a reason to test. If Action A is better in every relevant state, learning the state does not change the choice. Information may still have descriptive or accountability value, but its decision value for this choice is zero.

State the decision deadline. Evidence arriving after an irreversible commitment has little value for that decision. It may still improve future decisions or monitoring. Separating current and future value prevents a long-term research benefit from being used to justify missing an immediate safety window.

Contents · Next chapter

Perfect information reveals the uncertain state before the action is chosen, without cost or delay. It is hypothetical, but it sets an upper bound. Compute the expected consequence when the best action can be selected separately in each state. Subtract the expected consequence of the best action under current information. The difference is the expected value of perfect information, often abbreviated EVPI.

Suppose two actions face two states. If demand is high, A yields 90 points and B yields 65. If demand is low, A yields 20 and B yields 55. With probability 0.4 for high demand, A has expected value 0.4×90 + 0.6×20 = 48. B has 0.4×65 + 0.6×55 = 59, so B is preferred now.

With perfect information, choose A in the high state and B in the low state. Expected value becomes 0.4×90 + 0.6×55 = 69. EVPI is 69 − 59 = 10 points per decision. No information-gathering strategy for this model can be worth more than 10 points before considering wider spillovers.

EVPI is not the value of certainty in an emotional sense. It is the expected improvement from state-contingent action. A large uncertainty can have small EVPI if choices stay the same. A narrow uncertainty near a decision boundary can have high EVPI if it frequently switches the preferred option.

Contents · Previous chapter · Next chapter

Real tests have false positives, false negatives, measurement error and inconclusive results. They may observe a proxy rather than the state that drives consequences. Their expected value is therefore below EVPI when evaluated for the same decision and model.

The ceiling is useful. If a proposed study costs more than EVPI per affected decision, it cannot be justified by improving this decision alone, even if perfect. Stop before building a detailed test model. If study cost is well below EVPI, more analysis may be worthwhile; positive net value is not yet guaranteed.

Population scale changes the ceiling. A ten-point gain for one decision may become large across thousands of repeated decisions, but only if the evidence transfers and can be implemented. Multiply by the effective population and time horizon, not by everyone who could theoretically hear about the study. Discount or otherwise account for when benefits occur.

Perfect information can also reveal model omissions. If the calculation says certainty is nearly worthless but stakeholders still see a high-stakes unknown, check whether the action set, consequence scale or state definition excluded what matters. A low EVPI can be correct, or it can diagnose an impoverished model.

Contents · Previous chapter · Next chapter

Use the high-demand and low-demand example, but add a test that reports Positive or Negative. Sensitivity is 0.80: when demand is high, Positive occurs 80 per cent of the time. Specificity is 0.75: when demand is low, Negative occurs 75 per cent of the time. The prior probability of high demand is 0.40.

The probability of Positive is 0.40×0.80 + 0.60×0.25 = 0.47. Bayes’ rule gives a posterior high-demand probability of 0.32/0.47, about 0.681 after Positive. After Negative, high-demand probability is 0.08/(0.08+0.45), about 0.151.

At a high-demand probability p, A has value 20 + 70p; B has 55 + 10p. A becomes better when 20 + 70p exceeds 55 + 10p, so p exceeds 35/60, about 0.583. Therefore choose A after Positive and B after Negative.

The expected value with sample information is 0.47 times the best posterior value after Positive plus 0.53 times the best posterior value after Negative. Equivalently, sum joint-probability-weighted consequences: for Positive choose A, giving 0.32×90 + 0.15×20 = 31.8; for Negative choose B, giving 0.08×65 + 0.45×55 = 29.95. Total is 61.75. The expected value of sample information is 61.75 − 59 = 2.75 points before test cost and delay.

Contents · Previous chapter · Next chapter

Imagine a weaker test whose Positive result raises the probability of high demand only to 0.52, below the 0.583 switching threshold. Negative lowers it further. B remains preferred after both results. The expected decision value of the test is then zero in this simplified model, even though posterior beliefs change.

This is a powerful screening question: list every possible result and the action you would take. If all arrows point to the same action, either the test is not decision-relevant, the action set is incomplete, or the evidence threshold is too remote. Do not order a test merely because uncertainty feels uncomfortable.

Sometimes a result changes monitoring rather than the initial action. Include that if it has consequences. A positive signal might not reverse a launch decision but could trigger a smaller pilot, extra safeguards or earlier review. Value comes from any adaptive action, not only a binary switch.

Precommitment protects against result theatre. Before seeing the evidence, specify how each result updates belief and which action follows. If decision-makers later ignore an inconvenient result, the realised value of information falls. A test without a credible response plan may have theoretical value but little operational value.

Contents · Previous chapter · Next chapter

Evidence takes time. Waiting can delay benefits, prolong harm, consume a limited window or allow a reversible option to disappear. Add these consequences to the information strategy rather than subtracting an arbitrary “delay penalty” at the end.

Suppose the imperfect test’s expected benefit is 2.75 points, costs 0.8 points and delays action by one month. If expected delay loss is 1.5 points, net value is 0.45. A modest change in delay could reverse the recommendation. If the decision can be staged—begin a reversible part now and wait on the irreversible commitment—the cost of learning may fall.

Delay can also improve the baseline without a formal test because ordinary outcomes arrive. Compare “test and wait” with “wait without testing,” not only with “act now.” If both strategies learn from the same month of operations, the incremental value belongs to what the test adds.

Time affects test accuracy and relevance. An early proxy may be noisy but timely; a later outcome may be accurate but too late. The right question is not which measurement is best in isolation. It is which evidence produces the best feasible decision at the moment action is still available.

Contents · Previous chapter · Next chapter

Financial price is only one cost. A diagnostic procedure may impose discomfort or risk. A school assessment can consume teaching time and change student behaviour. A pilot may divert staff from another service. Data collection can create privacy and governance obligations.

Opportunity cost is especially easy to miss. Analysts assigned to refine a low-value uncertainty cannot investigate another decision with higher EVPI. Laboratory capacity, management attention and participant goodwill are scarce. Compare the proposed evidence plan with the best alternative use of those resources.

Some costs occur only for certain results. A positive screen may trigger an expensive confirmation test. Include the probability-weighted downstream pathway. Likewise, an inconclusive result may require repeat sampling or leave the original decision unchanged after delay.

Distribution matters. A study may have positive average value while burdening one group and benefiting another. Present subgroup consequences and fairness constraints instead of hiding them inside a single mean. Information can worsen inequity if accuracy, access or implementation differs across groups.

Contents · Previous chapter · Next chapter

Irreversible actions increase the value of learning before commitment, but only if useful evidence can arrive in time. Reversible actions create another strategy: act, monitor and adapt. The value of a formal test should be compared with that adaptive path.

A pilot buys information while producing some real-world consequence. Its design should preserve a comparison or clear threshold; otherwise it becomes a small rollout that teaches little. Define what observation would scale, modify or stop the programme and when that review occurs.

Options have value. Reserving capacity, using a short contract or choosing a modular design may cost more initially but preserve the ability to respond to evidence. That flexibility belongs in the consequence table. A seemingly inferior action can be rational because it keeps future choices open.

The existing guide to robust decisions and switching assumptions helps identify which uncertain input can reverse a choice. Value-of-information reasoning is the next step: estimate whether resolving that input is worth the available test, delay and burden.

Contents · Previous chapter · Next chapter

Evidence gathered once can inform many future decisions. Expected value of sample information can therefore be multiplied across an effective population, adjusted for how long the evidence remains relevant, how many decisions occur, and how completely practice changes.

Do not multiply mechanically. A market test in one city may not transfer to another. A clinical estimate may become obsolete when treatment changes. A teaching pilot may depend on staff expertise that cannot be scaled. Use a realistic implementation fraction and explicit time horizon.

Research may also improve the model, not just one parameter. It can reveal a missing state, heterogeneity or a new action. Standard calculations often condition on the current model, so qualitative research and model criticism remain important. Label what the quantitative value includes and excludes.

When several uncertain parameters matter, compute partial value of perfect information for groups of parameters. This shows whether resolving demand, cost, effectiveness or harm could change the decision. Prioritise research where uncertainty is both decision-sensitive and reducible.

Contents · Previous chapter · Next chapter

An expected-value result is not automatically objective. Probabilities may come from small studies, utilities may be contested, and structural assumptions may determine the answer. Show the decision model, sources and sensitivity ranges so that disagreement can be located.

Calibrate test performance for the intended population. Sensitivity and specificity from a selected sample may not transfer. Prevalence or prior probability changes posterior values even when likelihoods stay the same. A positive result can be strongly informative in one setting and weak in another.

Model the inconclusive and missing-result branches. Excluding them makes a test look more decisive and understates delay. Likewise, include implementation error: a correct result has limited value if the organisation cannot change action.

Use scenario analysis around the switching threshold. If the net value is positive only under optimistic accuracy and zero delay, present it as fragile. If the evidence strategy remains worthwhile across plausible assumptions, the case is more robust. Value-of-information analysis guides judgement; it does not remove it.

Contents · Previous chapter · Next chapter

Practice one: a test costs 3 units and EVPI is 2 per decision. For a single decision, stop; no imperfect test can recover more than the perfect-information ceiling. For 1,000 transferable future decisions, do not stop automatically. Recalculate population value, implementation, relevance duration and research cost at the correct scale.

Practice two: a highly accurate test returns after the deadline, while a moderate test returns tomorrow. Accuracy alone does not decide. Model the feasible actions at each arrival time. The moderate test can have higher value because it can still change the commitment.

Practice three: Positive and Negative both lead to Action B, but Positive triggers weekly monitoring and Negative monthly monitoring. The test may have value through the monitoring branch. Include the costs and consequences of those schedules rather than declaring zero value too early.

Transfer the method to everyday choices. Before seeking another review, quote or scan, write what result would change your action. Estimate the size of that improvement and compare it with delay and effort. This small habit turns “more information” into a decision rule and often reveals when a reversible first step is better than waiting.

Contents · Previous chapter · Next chapter

Is value of information the same as test accuracy?

No. Accuracy affects value, but value also depends on prior uncertainty, consequences, switching thresholds, cost, delay and whether action changes.

Can information have value when the main action stays the same?

Yes, if it changes monitoring, safeguards, communication or future decisions. Include those actions explicitly.

Why calculate perfect information if it is impossible?

EVPI is an upper bound. It quickly identifies uncertainties for which no evidence strategy can be worth a high cost under the current decision model.

What if consequences cannot be reduced to money?

Use a transparent multi-criteria or outcome framework and show trade-offs. Do not invent monetary values merely to make the arithmetic look precise.

Authoritative reading includes the NICE methods guidance Incorporating health economics, which explains when value-of-information analysis can indicate whether further research is worthwhile, and the current NICE economic evaluation guidance. These health-technology sources demonstrate the method; decisions in education, engineering or operations need their own outcomes, governance and evidence.

Return to The Gold Standard of Judgement for the subject map. Use The Gold Standard of Inquiry when the evidence question itself needs framing, and The Gold Standard of Evaluation when implementation and outcomes must be assessed after action. The next useful move is simple: draw every result-to-action branch, mark the deadline, and calculate whether the best possible change can repay the test.

Contents · Previous chapter · Return to the Judgement guide