Consensus is useful only when the path to it remains visible
The Delphi method structures several rounds of expert judgement so participants can reconsider views after controlled feedback without collapsing uncertainty into an artificial single answer.
When evidence is incomplete and judgement matters, a group discussion can be dominated by status, speed and social pressure. A Delphi process separates private response from group feedback and repeats the cycle so participants can revise—or deliberately retain—their positions.
The method is not a machine for producing truth. Panel composition, question wording, feedback design, attrition and the chosen definition of consensus all shape the result. A strong report therefore preserves distributions, reasons and dissent alongside any summary conclusion.
Chapter contents
- 1. Define the decision and why Delphi fits
- 2. Pre-specify the protocol and stopping rule
- 3. Construct a panel around relevant expertise and lived knowledge
- 4. Use anonymity to reduce pressure without erasing accountability
- 5. Design questions that elicit judgements rather than desired answers
- 6. Worked example: three rounds on heat preparedness
- 7. Provide controlled feedback that is balanced and reproducible
- 8. Interpret revision without treating convergence as proof
- 9. Define consensus before measuring it
- 10. Preserve dissent, uncertainty and minority reasons
- 11. Manage attrition and changing panels
- 12. Stop, report and use results within their boundary
Choose a reading route
- Route 1: Frame the study — question, decision and protocol
- Route 2: Build the panel — expertise, diversity and independence
- Route 3: Run the rounds — worked iteration and feedback
- Route 4: Interpret agreement — thresholds, dissent and uncertainty
- Route 5: Stop and report — attrition, validity and use
How to use this guide
Start with the chapters that match your immediate decision, then return to the worked example and audit chapters. Each chapter separates a model from the evidence needed to use it responsibly. A claim is not strengthened merely by being technical; it is strengthened when assumptions, observations, alternatives and limits can be inspected.
| Layer | Question | Minimum record |
|---|---|---|
| Concept | What exactly is being represented? | Definitions and boundary |
| Method | How will the reasoning be carried out? | Steps, inputs and assumptions |
| Transfer | Would it work in a changed setting? | A deliberately altered case |
| Audit | What could prove it wrong? | Evidence, limits and revision trigger |
Chapter 1. Define the decision and why Delphi fits
The idea
Delphi is appropriate when informed judgement is needed, evidence is dispersed and interaction benefits from structured reconsideration. It is not automatically preferable to observation, experiment or direct deliberation.
A disciplined method
Write the decision, target population, forecast or priority, time horizon, evidence limits and how results will be used; compare Delphi with simpler alternatives.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is what the panel is being asked to estimate or prioritise. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
A school network asks which infrastructure failures deserve preparedness planning over five years, not which vendor to buy tomorrow.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Change the question to a measurable current failure rate and decide whether direct records should replace expert elicitation.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: A well-run Delphi cannot compensate for a vague construct or unavailable empirical data.
Evidence to retain: protocol, decision owner, alternatives considered, horizon and use statement
Common misconception: Delphi is suitable whenever several experts disagree.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 2. Pre-specify the protocol and stopping rule
The idea
Iteration creates analytic flexibility. A protocol should define rounds, scales, feedback, consensus criteria, missing-data handling and stopping conditions before results are known.
A disciplined method
Publish or register the plan, pilot the instrument, define permitted adaptations and keep a change log with reasons.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is which procedural choices are fixed and which may adapt. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
A three-round study predefines an interquartile-range threshold but allows wording clarification after the pilot, with changes documented.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Change the scale from 1–9 to probabilities and redesign feedback rather than carrying thresholds over mechanically.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Rigid protocols can preserve a flawed question; justified amendments should remain visible.
Evidence to retain: versioned protocol, pilot notes, thresholds, deviations and approvals
Common misconception: More rounds always improve validity.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 3. Construct a panel around relevant expertise and lived knowledge
The idea
Panel quality depends on who can illuminate the question, not prestige alone. Different forms of expertise reveal different assumptions and consequences.
A disciplined method
Create an expertise matrix, recruit across roles and affected perspectives, declare inclusion criteria, manage conflicts and report composition without exposing identities unnecessarily.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is whose knowledge and experience the panel needs. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
A flood-planning panel includes hydrologists, emergency managers, infrastructure operators and community responders rather than one profession only.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Move the problem to a rural context and reassess whether the original expertise matrix still covers local constraints.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Purposive panels do not support simple population inference, and credentials do not eliminate bias.
Evidence to retain: recruitment frame, expertise matrix, conflicts, invitations and participation by round
Common misconception: A larger panel is automatically more representative.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 4. Use anonymity to reduce pressure without erasing accountability
The idea
Delphi usually keeps individual responses anonymous to other panellists, reducing authority and conformity effects. Researchers still need ethical, secure and accountable handling.
A disciplined method
Explain who can see raw responses, separate identifiers from answers, aggregate small cells cautiously and disclose quotations only under consent rules.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is which kind of anonymity is promised to whom. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
Panellists see distributions and coded reasons, while the study team holds a protected recontact key for follow-up.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Use a panel with rare specialties and test whether verbatim comments would reveal identities.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Anonymity can reduce clarification, relationship building and responsibility for careless claims.
Evidence to retain: consent, access controls, coding key, aggregation rules and disclosure review
Common misconception: Anonymous responses are unbiased responses.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 5. Design questions that elicit judgements rather than desired answers
The idea
Questions should define the object, horizon, scale and evidence request. Leading language, double questions and unclear endpoints create consensus by construction.
A disciplined method
Use plain wording, one construct per item, labelled scale anchors and optional rationale; pilot with people similar to the panel.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is what each response value means. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
Experts estimate the probability of a service outage by 2030 and name the evidence that could change their estimate.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Change the horizon to 2040 and check whether participants need scenarios rather than a single forecast.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Numbers can imply precision beyond the evidence, and open responses require careful synthesis.
Evidence to retain: question versions, cognitive interviews, scale definitions and pilot revisions
Common misconception: A nine-point scale makes a vague question precise.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 6. Worked example: three rounds on heat preparedness
The idea
A bounded example shows how private estimates, aggregated feedback and revision interact without forcing unanimity.
A disciplined method
Round one elicits measures and rationales; round two presents median, quartiles and themed reasons; round three asks for final ratings and reasons for remaining divergence.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is whether feedback changes reasoning or merely moves numbers. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
A fictional panel rates cooling centres, building retrofits and outreach. Retrofits retain a wide distribution because cost and tenancy assumptions differ; the report preserves that boundary.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Move the study to a city with reliable home cooling and observe how priorities and rationales change.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: The exercise organises judgement; it does not establish intervention effectiveness.
Evidence to retain: item-level distributions, coded rationales, revisions, assumptions and final dissent
Common misconception: The narrowest distribution identifies the best policy.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
| Round | Input | Output |
|---|---|---|
| 1 | Independent ideas and estimates | Candidate items and reasons |
| 2 | Distribution plus controlled feedback | Revised ratings and comments |
| 3 | Updated feedback | Final distribution and dissent |
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 7. Provide controlled feedback that is balanced and reproducible
The idea
Controlled feedback summarises group responses between rounds. It should show central tendency, spread and important reasoning without becoming a persuasive editorial.
A disciplined method
Use consistent statistical summaries, disclose coding methods, include reasons on both sides and separate external evidence from panellist comments.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is which information participants receive before revising. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
A feedback sheet shows median 7, interquartile range 4–8 and two opposing rationales rather than announcing that the group supports the item.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Replace the median with a histogram for a bimodal item and see whether the conflict becomes clearer.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Researchers influence outcomes through selection, phrasing and ordering of feedback.
Evidence to retain: feedback packets, analysis code, coding frame, omitted comments and review notes
Common misconception: Controlled feedback should steer outliers toward the majority.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 8. Interpret revision without treating convergence as proof
The idea
Participants may revise because of new reasons, social inference, fatigue or misunderstood scales. Stable disagreement can be informative.
A disciplined method
Track individual changes privately, ask for revision rationales where proportionate and distinguish convergence, polarisation and stable spread.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is why ratings moved between rounds. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
Several experts lower risk estimates after learning that a safeguard is already funded; others remain high because implementation is uncertain.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Remove the funding evidence and test whether the direction of revision reverses.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Observed convergence does not reveal causal motivation and can coexist with shared error.
Evidence to retain: paired round responses, change magnitudes, rationales and distribution plots
Common misconception: Convergence demonstrates accuracy.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 9. Define consensus before measuring it
The idea
Consensus has many operational definitions: proportion within a range, interquartile width, rank agreement or stability. Threshold choice must match the scale and decision.
A disciplined method
State the rule, justify it, perform sensitivity checks and report the underlying distribution even when a binary consensus label is used.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is which distributional pattern qualifies as agreement. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
An item meets a predeclared 75-percent-in-7-to-9 rule but fails an interquartile-range rule; both results are reported.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Shift one response at the threshold and show whether the classification is fragile.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: No universal consensus threshold exists, and thresholds can obscure bimodality.
Evidence to retain: raw ratings, scale, rule, sensitivity analysis and label assignment
Common misconception: Consensus is a natural property that requires no definition.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 10. Preserve dissent, uncertainty and minority reasons
The idea
Minority judgements can identify hidden assumptions, rare risks and affected groups. A consensus summary should not erase them.
A disciplined method
Report ranges and clusters, code reasons faithfully, invite final minority statements and flag conditions under which the majority result would fail.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is which disagreement is substantive rather than measurement noise. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
A minority rejects a preparedness priority because the delivery model excludes informal tenants; the condition remains visible beside the ranking.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Change the implementation model and ask whether the minority objection is resolved.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Publishing very small subgroups can threaten confidentiality.
Evidence to retain: distribution, cluster analysis, minority rationale, response and confidentiality review
Common misconception: Outliers should be removed to improve consensus.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 11. Manage attrition and changing panels
The idea
Dropout between rounds can manufacture apparent agreement if dissenters or busy specialists leave. Participation burden and replacement rules must be planned.
A disciplined method
Track invitations and completions by relevant strata, ask non-coercive reasons for withdrawal, compare retained and lost response patterns and avoid silent replacement.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is whether the final panel still represents the intended knowledge mix. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
Round-three agreement rises after frontline responders disproportionately withdraw; the report treats the shift as an attrition warning.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Shorten the survey and test whether retention improves without dropping essential constructs.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: Nonresponse explanations may be unavailable, and small panels limit formal adjustment.
Evidence to retain: flow diagram, completion by round, baseline comparison, withdrawal reasons and adaptations
Common misconception: Only final-round respondents matter.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Chapter 12. Stop, report and use results within their boundary
The idea
A Delphi study should stop according to its protocol or when additional rounds add little justified information. Reporting must connect findings to the selected panel and use case.
A disciplined method
Apply the stopping rule, publish protocol deviations, instrument, panel description, round statistics, dissent, limits and a decision-use statement; schedule review when conditions change.
A useful working record makes the sequence visible: first state the question and unit of analysis; next record the observations and their provenance; then apply the stated rule; finally compare the result with plausible alternatives. For this chapter, the decisive choice is what claim the final results support. Keeping that choice explicit prevents a later reader from mistaking a convenient convention for a fact about the world.
Worked example
The final report says that this purposive panel prioritised three measures under stated assumptions, not that experts have proven them universally best.
Work the example twice. On the first pass, follow the stated assumptions. On the second, reverse one assumption, remove one piece of evidence and ask whether the conclusion, ranking or boundary changes. This small stress test matters because polished outputs can hide sensitivity. The objective is not to force certainty; it is to locate the evidence on which the conclusion actually turns.
Changed-context transfer check
Use the priorities in a pilot, collect outcomes and revise the judgement rather than treating the panel as permanent authority.
A transfer succeeds only when the structure of the problem survives the change. Match roles, dependencies and measurement conditions rather than surface vocabulary. Record what was preserved, what changed, and why the method still applies—or why it does not. If the new context changes the data-generating process or the people bearing the burden, re-specify the model instead of importing the old answer.
Limits, evidence and revision
Limit: External validity, causal effectiveness and implementation feasibility require separate evidence.
Evidence to retain: full report, appendices, data policy, decision record, pilot evidence and review trigger
Common misconception: Delphi consensus is a mandate for action.
A gold-standard conclusion is therefore conditional. It should name the observation period, excluded cases, uncertainty and the event that would trigger review. Negative findings and unresolved contradictions belong in the record. They protect future readers from repeating a hidden assumption and make correction possible without reconstructing the entire analysis from memory.
Practical evidence check
Can another reader recover the input, reproduce the intermediate decision and identify at least one observation that would change the result? If not, the chapter has produced an explanation but not yet an auditable finding.
Field workbook
Draft a two-round Delphi protocol for a non-sensitive planning question. Define the panel matrix, anonymity promise, scales, feedback format, consensus and stopping rules. Create one bimodal mock result and write a report that preserves the disagreement.
- Write the decision question in one sentence and name the decision owner.
- List observations separately from interpretations, with dates and provenance.
- Record at least two plausible alternatives before choosing a preferred account.
- Run the changed-context test and note which assumptions fail.
- State the limit, confidence and next review trigger in plain language.
| Layer | Evidence | Failure signal |
|---|---|---|
| Question | Defined construct and horizon | Leading or double item |
| Panel | Expertise matrix and flow | Prestige-only sample |
| Iteration | Versioned feedback packets | Editorial steering |
| Conclusion | Distribution, dissent and limits | Consensus as truth |
Continue through the verified ecosystem
- Horizon scanning — weak signals and emerging issues
- Scenario analysis — alternative futures and consistency
- Words of estimative probability — calibrated uncertainty language
- Key assumptions check — load-bearing assumptions and reversal
- Assumption-based planning — signposts and contingent action
Frequently asked questions
What makes a study Delphi?
A structured sequence of private responses, controlled group feedback and reconsideration across rounds.
Must a Delphi reach consensus?
No. Stable disagreement can be the most decision-relevant finding.
How many experts are required?
There is no universal number; adequacy depends on the question, knowledge domains and intended claims.
Why use anonymity?
It can reduce status and conformity pressure, though it also limits direct clarification and relationship building.
What is controlled feedback?
A balanced, reproducible summary of distributions and reasons returned between rounds.
Does consensus prove correctness?
No. A panel can share missing information, framing effects or common error.
Authoritative sources and further reading
- RAND: The Delphi Method — foundational method report by Dalkey and Helmer
- The Delphi Method: Techniques and Applications — Linstone and Turoff foundational open book
- The Delphi Technique: Making Sense of Consensus — practical design and consensus discussion
- Conducting and reporting Delphi studies — systematic methodological guidance
- CREDES Delphi reporting guidance — reporting quality and transparency
This guide explains a structured expert-judgement method. It does not turn purposive opinion into representative measurement, causal evidence or universal truth.