Did you know that a team can become better at anticipating events without ever pretending to predict the future with certainty? Collective forecasting works by making a question resolvable, eliciting separate probability estimates, combining judgments with an explicit rule, recording updates as evidence changes, and scoring forecasts against eventual outcomes. The useful intelligence is in the disciplined process, not in the confidence of the loudest participant.
The direct answer is that independent perspectives can contain different information, while structured aggregation can reduce the influence of extreme individual errors. But independence is not automatic, crowds can share the same blind spot, and a numerical consensus can still be wrong. The method works best when the group preserves distinct observations before discussion and evaluates its accuracy over many clearly resolved questions.
Here is the practical reader problem: a school, laboratory, small business or community project must plan for an uncertain delivery date. One person says “almost certain,” another says “unlikely,” and a meeting drifts toward whichever voice sounds authoritative. We will turn that disagreement into a transparent forecasting exercise, check the arithmetic, and show how the group can learn from outcomes without treating a single success as proof of wisdom.
Contents and practical routes
- 1. What exactly is a collective forecast?
- 2. Begin with a question that can actually resolve
- 3. Keep observations, interpretations and decisions separate
- 4. Collect independent first estimates before discussion
- 5. Use reference classes rather than confident narratives
- 6. Make the aggregation rule visible and stable
- 7. Worked example: four estimates and one outcome
- 8. Understand the two common binary Brier conventions
- 9. Why averaging can help without creating new evidence
- 10. Discussion can improve a forecast or contaminate one
- 11. Updating means changing a probability when evidence changes
- 12. Calibration and resolution answer different questions
- 13. Four-event scored practice with checked arithmetic
- 14. Weighting expertise is a governance choice
- 15. Disagreement is information before it is a problem
- 16. Forecasts do not decide how much risk to take
- 17. Use multiple horizons to reward useful warning time
- 18. Do not accidentally reward correlated repetition
- 19. What empirical research actually says
- 20. A practical decision meeting that does not turn into theatre
- 21. Failure diagnosis: six ways group forecasts go wrong
- 22. Changed-context transfer: planning a field experiment
- 23. Ethical and organisational boundaries
- Frequently asked questions
- Research and onward learning
1. What exactly is a collective forecast?
A forecast is a dated statement about a future, observable event or numerical quantity, made with acknowledged uncertainty. Collective forecasting combines two or more people’s judgments under an identifiable rule and may allow discussion, feedback and revision. It is not identical to a vote on what the team wishes to happen. Desires, values and authority determine what action should be taken; forecasts describe what might happen under given conditions.
For a binary event, express the estimate as a probability from zero to one. “There is a 60% chance” means that, across a sufficiently appropriate set of comparable forecasts assigned around 60%, approximately 60% should resolve positively if the forecasts are calibrated. It does not mean that a particular single event will be 60% true. An individual event eventually resolves to yes or no under the defined rules; the probability describes uncertainty before resolution.
Collective intelligence appears when participants contribute genuinely different evidence, when the group preserves enough of that evidence to make meaningful revisions, and when results are checked. Merely averaging four identical copies of one mistaken assumption adds no new information. This guide addresses the distinctive mechanisms of group forecasting, beyond the general ideas of reasoning, evidence and cooperation introduced in the How Intelligence Works hub.
2. Begin with a question that can actually resolve
Forecasting fails before the first number when the event is vague. “Will the project be successful?” cannot be scored unless success, the project, and the date are defined. A better question is: “Will the supplier deliver all twenty tested modules, as accepted under the written checklist, by 17:00 Singapore time on 30 November 2026?” Identify the data source that can establish the result and the handling of missing or disputed evidence.
Freeze the event wording before collecting predictions. State whether partial delivery counts, how late delivery is treated, and what happens if the project is cancelled or the supplier changes the product. In a fictional case, “delivered” might require a receipt signed by a named logistics owner and a quality acceptance record for all twenty units. A truck arriving at the gate with nineteen units does not satisfy that event, although it may still be useful progress.
Good resolution rules make forecasts testable and reduce arguments over who was right. They also separate probability from operational action: a 70% chance of meeting a deadline might justify preparing a contingency plan even if the most likely outcome is delivery. An accurate forecast is a tool for choosing a response, not a substitute for that choice.
3. Keep observations, interpretations and decisions separate
One team member may have spoken to a vendor, another may know a comparable project’s delays, a third may understand testing capacity, and a fourth may notice a public holiday in the schedule. Those are potentially different observations. The probability each member assigns is an interpretation of evidence, and the action selected by management is a third layer. Recording all three prevents the final plan from becoming confused with the forecast used to support it.
Ask each contributor to state what evidence they used, the date of that evidence, the main alternative explanation, and which new observation would change their forecast. Encourage a distinction between direct records and hearsay. A team member can be very knowledgeable about manufacturing yet have old information about the present shipment; the specialist label does not update the record automatically.
Do not require participants to expose confidential personal data or commercially sensitive sources to everyone. A qualified reviewer may attest to a restricted document’s relevance and scope while keeping the document itself protected. The group still needs enough provenance to understand what the claim supports. “I have secret evidence so the answer is 90%” is not a useful substitute for an auditable process.
4. Collect independent first estimates before discussion
If the first person announces a high probability, others may anchor on it, defer to seniority or fear disagreeing. Record individual estimates before group discussion, with a timestamp and a short rationale. Independent first judgments preserve evidence that might otherwise disappear when a meeting converges socially. Independence here means that initial estimates were formed without seeing others’ current numbers, not that people live in separate information universes.
A simple form contains the precise event, relevant base rate, private estimate, confidence in the evidence quality, most important uncertainty, and a possible update trigger. Do not ask for a prose explanation so long that busy staff will invent a rationale after choosing a number. A brief, checkable reason is better than a polished story that hides its origin.
Independence alone is not enough. Four employees may all rely on the same outdated dashboard. Label shared sources so the aggregator does not mistake correlated evidence for four separate confirmations. The article on collective blind spots explores missing evidence and observer coverage; forecasting adds probabilities, event resolution and feedback scores.
5. Use reference classes rather than confident narratives
Before constructing a story about why this particular supplier will succeed, ask what happened in comparable deliveries. Reference classes supply base-rate information: for example, the fraction of genuinely similar jobs completed on time. Define similarity before cherry-picking cases. Twenty routine shipments from the same vendor may be more relevant to next week’s routine shipment than three large, unusual projects, but a pending regulatory change might make the historical comparison obsolete.
Suppose a fictional archive contains 40 comparable projects and 26 met the same acceptance deadline. The observed proportion is 26 / 40 = 65%. That is a descriptive historical rate, not a guaranteed 65% probability for the next project. The new supplier’s backlog, shipping conditions and quality requirements may justify moving up or down from that starting point. Forty observations also leave material sampling uncertainty.
A good rationale might read: “The reference class met deadlines in 26 of 40 cases. This order uses a new component, which raises uncertainty, but the supplier has already passed the first acceptance test. My estimate is 55%, and I would raise it if the remaining testing record arrives.” That is a forecast that can be challenged constructively rather than a claim to exceptional intuition.
6. Make the aggregation rule visible and stable
The most transparent collective rule is an arithmetic mean of individual probabilities. If four forecasters assign 0.20, 0.40, 0.70 and 0.90 to the same binary event, their equal-weight aggregate is (0.20 + 0.40 + 0.70 + 0.90) / 4 = 2.20 / 4 = 0.55. The group forecast is 55%. This calculation says nothing by itself about which participant has the best evidence; it is simply a reproducible transformation of four inputs.
Other aggregation rules are possible, including medians, weighted means, trimmed means and transformations that emphasise consistently informative forecasters. The right choice depends on the outcome history, correlation among participants, incentives, and the decision context. A rule should be fixed or transparently governed before knowing whether the current event occurs. Choosing whichever rule looks best after seeing the result turns evaluation into a retrospective game.
Document which forecasters were eligible, when their estimates were taken, how missing estimates were handled, and whether weights were capped. Four late revisions by one person must not silently count as four independent people. If participants abstain because evidence is insufficient, keep that fact visible rather than imputing a convenient 50% judgment without a declared policy.
7. Worked example: four estimates and one outcome
Continue the fictional supplier event. The four independent probabilities are 20%, 40%, 70% and 90%. We have already calculated the arithmetic-mean forecast as 55%. Now suppose, solely for a solved teaching example, that all twenty tested modules arrive by the defined deadline. The outcome indicator is y = 1. We can measure each forecast’s squared error using (p − y) squared, where p is the forecast probability expressed as a decimal.
The individual errors are (0.20 − 1)² = 0.64, (0.40 − 1)² = 0.36, (0.70 − 1)² = 0.09, and (0.90 − 1)² = 0.01. Their mean is (0.64 + 0.36 + 0.09 + 0.01) / 4 = 1.10 / 4 = 0.275. The group aggregate’s squared error is (0.55 − 1)² = 0.2025. On this one invented event the group mean is better than the mean score of the four individuals, but worse than the best individual’s score of 0.01.
Those numbers illustrate arithmetic, not evidence that this method will outperform strong forecasters in every real situation. Squared error is convex, so the squared error of an arithmetic mean is no worse than the mean squared error of its constituents for a fixed resolved outcome. That mathematical property is not a guarantee that the mean beats the most informed person or a well-chosen external baseline.
8. Understand the two common binary Brier conventions
Forecasting literature does not always use the same scale when reporting a Brier score. For a binary event, one convention scores only the yes probability as (p − y)², yielding values from zero to one. Another represents both yes and no and adds the two squared errors: (p − y)² + ((1 − p) − (1 − y))². The two-category sum is exactly twice the one-category score for a binary outcome.
In our 55% forecast with a positive outcome, the one-category score is 0.2025. The two-category score is 0.2025 + (0.45 − 0)² = 0.405. The 2023 paper Compromising improves forecasting displays the two-outcome summed form when describing its analysis. Other learning materials use the single-probability form. Neither convention is intrinsically dishonest, but comparing raw reported scores without checking the scale is a serious error.
Always report the formula, the number of forecast occasions, the event set, and whether lower is better. Normalise only when the reporting convention warrants it. An apparent score reduction from 0.40 to 0.20 could be nothing more than a switch of convention if the underlying forecasts and outcomes are identical.
9. Why averaging can help without creating new evidence
The mathematical benefit of averaging does not mean four views are independent. Suppose two people rely on the same vendor reassurance and the other two use independent inspection records. The average assigns equal numerical weight to the people, not to distinct sources. The team may be overcounting one claim. Ask whether each estimate adds evidence, a different model of the situation, or merely a repeated conclusion.
Outlier estimates are sometimes noise, but they can also be an early warning. If one forecaster says 10% while the group says 80%, invite the dissenting person to explain which observation could justify such a difference. The group need not automatically accept the dissent; it should preserve and test it. An aggregation algorithm produces a number, not a complete account of why members disagree.
The 2023 Royal Society Open Science study by Ferreiro, Deroy and Bahrami analysed Good Judgment Project forecasting data and examined the benefit of compromise averages over time. That is relevant empirical evidence, not a claim that every workplace meeting improves through the same formula. Its participants, training and question structure were specific. Organisational transfer demands its own evaluation.
10. Discussion can improve a forecast or contaminate one
After independently recording first estimates, structured discussion can reveal missing facts and correct reasoning errors. Ask each person to supply one important observation, identify its source and freshness, and explain how it changes the estimated probability. The chair’s task is not to manufacture unanimous enthusiasm. It is to make evidence inspectable while allowing warranted disagreement to remain.
Separate information-sharing from influence by status. A senior leader can help resolve resource constraints but might also anchor everyone by stating a preferred outcome. Consider discussing the evidence first and updating probabilities privately again afterward. This retains a record of what changed and reduces the pressure to give a publicly approved number. The group can then compare initial, revised and aggregated forecasts.
The 2014 study Psychological Strategies for Winning a Geopolitical Forecasting Tournament reported the value of training, teaming and tracking in a forecasting competition. It is more precise to say that structured collaboration can help under studied conditions than to assert that discussion always improves accuracy. Discussion can also create herding when the same claim circulates without independent checking.
11. Updating means changing a probability when evidence changes
A forecast is a dated state of belief, not a promise to defend an initial guess forever. When a new acceptance report arrives, participants can revise the estimate. Record the previous probability, the new evidence, the new probability, and the time of change. A forecast that moves from 55% to 75% after independently verified test completion can be reasonable; a forecast that jumps because everyone wants a deadline to be met deserves scrutiny.
Do not confuse the timing of an update with quality. Moving to 99% five minutes before a publicly confirmed delivery may score well for that moment but was not equally useful to a decision maker planning a month earlier. Compare forecasts at standard lead times—perhaps thirty, seven and one day before resolution—so a late easy prediction does not crowd out an earlier informative one.
Repeated updates from one person are correlated and should not be treated as independent crowd members. A dataset with one forecaster updating twenty times and another updating once does not contain twenty-one independent people. Scoring systems must define whether they evaluate the latest forecast at fixed horizons, time-weighted trajectories, or each event once.
12. Calibration and resolution answer different questions
Calibration asks whether announced probabilities correspond to observed frequencies across suitable groups of forecasts. If many forecasts near 70% resolve positively about 70% of the time, they are broadly calibrated. Resolution asks whether the forecasts meaningfully discriminate between events likely and unlikely to occur. A person who assigns 50% to every event might be superficially well calibrated in a balanced event set but provide little useful differentiation.
One perfect final score on one event cannot establish calibration. You need a collection of resolved questions, enough examples in probability ranges, and appropriate uncertainty around the estimated frequencies. Small bins can mislead: observing two successes in two 70% forecasts does not prove that 70% was wrong. The relevant event distribution, resolution rules and lead times must be comparable.
Tracking both properties makes improvement concrete. A group might move away from unjustified extremes and become better calibrated while retaining enough variation to inform decisions. Another group may rank events well but systematically call 80% outcomes 95%, producing unwarranted confidence. A useful evaluation should detect the difference rather than rely on persuasive retrospective stories.
13. Four-event scored practice with checked arithmetic
Consider four additional invented binary questions, each resolved under a prespecified rule. The group probabilities are 0.70, 0.30, 0.60 and 0.80. The respective actual outcomes are 1, 0, 0 and 1. Under the one-category squared-error convention the scores are (0.70 − 1)² = 0.09; (0.30 − 0)² = 0.09; (0.60 − 0)² = 0.36; and (0.80 − 1)² = 0.04.
The sum is 0.09 + 0.09 + 0.36 + 0.04 = 0.58, and the mean is 0.58 / 4 = 0.145. A fixed 50% forecast for every event would score 0.25 on each of these four binary outcomes, yielding a mean of 0.25. The group forecast therefore scores 0.105 lower on this fictional four-event set. Lower squared error is better, but four events are far too few to certify a forecasting system.
Notice the single mistake: the group assigned 60% to an event that did not occur, giving the largest error of 0.36. A good review asks whether this arose from bad data, event wording, correlated assumptions, insufficient updates, or ordinary uncertainty. It does not force every previously reasonable 60% judgment to become zero simply because an event failed to occur once.
14. Weighting expertise is a governance choice
A team may consider weighting forecasts by historical accuracy, domain expertise, freshness of evidence or participation quality. That can be defensible, but the rule needs prespecified evidence and safeguards against overfitting. A person excellent at forecasting software delivery may not be especially informed about crop yields or clinic capacity. Weights learned from a tiny or unrepresentative set of questions can reward luck rather than stable skill.
For illustration, use weights 1, 1, 2 and 2 for the four probabilities 0.20, 0.40, 0.70 and 0.90. The weighted mean is (1 × 0.20 + 1 × 0.40 + 2 × 0.70 + 2 × 0.90) / 6 = 3.80 / 6 ≈ 0.6333, or 63.33%. If the fictional event occurs, its one-category squared error is approximately (0.6333 − 1)² ≈ 0.1344. This is better than 0.2025 for that outcome, but the weights must not be chosen afterward simply because they favour a successful participant.
Test weighting schemes on outcomes distinct from those used to design the weights. Consider caps, minimum evidence requirements, transparency and the risk of systematically silencing useful dissent. An elegant weight formula is not a licence to replace contextual judgment with opaque authority.
15. Disagreement is information before it is a problem
A range from 20% to 90% is wide. It might reflect real conflicting evidence, different interpretations of the event, differing reference classes, or simple errors in probability language. Do not average away the disagreement without investigating. Ask each person what observation would move them toward the opposite side, and whether everyone is forecasting precisely the same event.
One member may interpret “delivery” as arrival at the warehouse while another requires acceptance testing. After agreeing on the event wording, both may adjust their probabilities. That is not convergence caused by social pressure; it is a repaired measurement question. Another member may have seen an official maintenance notice absent from the common evidence set. The team can check it before changing any weights.
Record dispersion alongside the aggregate. A mean of 55% formed from four estimates around 55% is not the same information state as a 55% mean formed from 20%, 40%, 70% and 90%. The decision maker may need different contingency arrangements even though both groups report the same central probability.
16. Forecasts do not decide how much risk to take
Suppose a delayed delivery would cost a hypothetical school project S$8,000 in additional venue and staffing expenses, while a contingency booking costs S$1,500 and avoids most of that loss if delay occurs. If the agreed on-time probability is 55%, the implied delay probability is 45%. Under this simplified example, the expected loss without protection is 0.45 × S$8,000 = S$3,600, which exceeds the S$1,500 contingency cost.
That arithmetic suggests that exploring the contingency may be prudent under the stated assumptions, but it is not a complete purchasing instruction. The alternative may not eliminate all loss, the event outcomes may be more than binary, and there can be contractual, equity or operational constraints. A robust decision compares feasible actions using explicit consequences, not just the most likely forecast.
This illustrates the boundary between intelligence and authority. Forecasters help quantify uncertainty; an accountable decision maker approves costs and trade-offs. An agent or manager who says “we are 55% likely to succeed, so do nothing” has skipped the consequence side of the decision.
17. Use multiple horizons to reward useful warning time
A probability announced after a deadline is nearly resolved is less useful for long-range planning than an equally good probability announced while the team can still act. Evaluate fixed horizons such as four weeks, one week and one day before resolution. The precise horizons should reflect the real lead time needed for staffing, procurement, public communication or safety preparation.
Keep the forecast record immutable after submission, except through a separately logged revision. A retrospective edit that makes a prediction look prescient destroys the learning loop. If the external event’s resolution is delayed, preserve the original question and apply the declared delay or voiding rule instead of silently changing the target. If a source later corrects its data, retain both original and corrected records with provenance.
A forecasting dashboard should show historical skill alongside current uncertainty. A group can be good at updating while still poor at long-range accuracy, or vice versa. Stakeholders deserve to see which kind of forecasting capability has actually been measured.
18. Do not accidentally reward correlated repetition
Suppose three committee members independently inspect different supplier records and offer similar estimates. That agreement is more informative than three people forwarding the same vendor email. Source overlap creates correlated errors; extra voices do not erase the correlation. The same principle applies to forecasts produced by multiple AI systems sharing training data, tools or retrieved documents. Diversity of model names is not proof of evidential independence.
Record source families and important dependencies at the level the task permits. A simple note that three participants relied on the same status update may suffice. It can justify seeking one independent inspection rather than adding another round of discussion. If the group repeatedly fails on the same kind of event, map which assumptions and dependencies those events shared.
A surprising minority estimate is worth investigation when it identifies a separate, checkable fact. It is not automatically correct because it is contrarian. The goal is to increase independent evidence and calibrate judgments, not cultivate disagreement for its own sake.
19. What empirical research actually says
The Good Judgment Project and associated research have examined how people form and update geopolitical forecasts. In Mellers and colleagues’ 2014 study, training, teams and tracking were associated with improved forecasting performance in the studied tournament. In their 2015 follow-up, researchers examined more than 150,000 forecasts from 743 participants on 199 events and studied predictors including cognitive flexibility, open-mindedness, training and updating behaviour.
These studies do not show that intelligence is reducible to a score or that anyone can predict every outcome. Participants were selected, questions were structured and outcomes were defined. Applying the procedures in schools, hospitals or companies requires privacy, task relevance, cost awareness and local validation. A public forecasting tournament has different incentives and information constraints from a confidential organisational decision.
The 2023 Ferreiro–Deroy–Bahrami study analysed compromise forecasts in Good Judgment Project data and found benefits to averaging forecasts in the examined setting. The important transfer is methodological: document independent estimates, aggregate explicitly, score results, and study when discussion helps or harms.
20. A practical decision meeting that does not turn into theatre
Start with the resolution question on screen. Give participants a short independent forecasting period. Collect individual probabilities privately, then display the distribution without identifying a rank order of authority. Invite one reason that could increase the forecast and one that could decrease it. Ask who can verify each relevant source and by when. Do not spend twenty minutes polishing an already untestable slogan about confidence.
Next, let participants revise privately. Apply the predetermined aggregation rule and record the final time-stamped group estimate. A separate responsible person decides the contingency policy under the relevant budget and safety constraints. Set the next evidence checkpoint. The team should know what would trigger revision, rather than reconvening only after a surprising event occurs.
After resolution, compute the score under the same formula used for previous questions. Review error mechanisms in a learning-oriented way; do not humiliate an individual for an event that had a legitimate 20% chance and then occurred. If managers punish uncertain but honest probabilities, they will train the group to speak in safe generalities instead of informative numbers.
21. Failure diagnosis: six ways group forecasts go wrong
Failure one: an unresolvable question. Repair with a dated, observable event and a documented source of truth. Failure two: early anchoring by a powerful participant. Repair by collecting first estimates independently. Failure three: false consensus from copied evidence. Repair by marking source overlap and seeking genuinely different observations. Failure four: one charismatic person’s confidence is treated as an empirical weight. Repair by defining and testing weights before outcomes.
Failure five: a group says it is better because it predicted yesterday’s result. Repair by scoring many questions at comparable horizons and using the same convention. Failure six: the estimate is converted into an action with no consideration of consequences. Repair by separating the forecast ledger from a decision record that names feasible alternatives, costs, responsibilities and human authority.
An additional warning concerns event manipulation: a team with control over the event may change behaviour after forecasting. That does not necessarily make the forecast invalid, but the question must say whether it predicts outcomes under a specified intervention or under business as usual. Good forecasting is part of a responsible decision system, not an outside observer pretending interventions never occur.
22. Changed-context transfer: planning a field experiment
Transfer from procurement to a fictional student field experiment. Four researchers must forecast whether all eight data collection stations will provide valid readings by Friday 18:00. The resolution requires a named supervisor to confirm eight valid files using a predefined quality checklist. Historical experiments show 12 of 20 comparable weeks succeeded, a descriptive base rate of 60%. The team records independent estimates of 40%, 50%, 65% and 85%.
Their equal-weight aggregate is (0.40 + 0.50 + 0.65 + 0.85) / 4 = 2.40 / 4 = 0.60, or 60%. Assume a purely fictional outcome in which only seven stations pass the checklist. Under the stated all-eight rule, y = 0. The group squared-error score is (0.60 − 0)² = 0.36 using the one-category convention. Do not relabel the event as successful because most stations worked; that would change the target after the outcome.
The practical response is to determine which station failed and whether redundancy, improved maintenance or a different timetable would reduce future loss. Forecasting accuracy and engineering repair are related but separate. The students can report the failed binary forecast honestly while also recognising that seven successful stations yielded useful measurements.
23. Ethical and organisational boundaries
Probability forecasting should not be used to evade responsibility for decisions affecting people. A school support prediction, for instance, is not a diagnosis or a justification to withdraw access to a service. The group must not collect excessive personal data, pressure participants to reveal protected information, or turn errors into public rankings without appropriate context. Where forecasts concern legal, clinical or safety decisions, qualified human authorities and applicable safeguards remain essential.
Poorly designed incentives can promote extreme or performative predictions. A scoring rule and retrospective review should reward honest estimates across many events, not bravery in predicting improbable disasters or agreement with management. Participants must be able to abstain when evidence is insufficient, even though the organisation may still have to make a decision under uncertainty.
Human intelligence in groups is a process of organised challenge, observation, revision and accountability. It does not require that everyone agrees. It requires that differences can be investigated and that the group remains willing to learn when events do not follow the most convincing narrative.
Frequently asked questions
Is a collective forecast just a majority vote? No. A vote chooses among options or signals preferences. A probabilistic forecast estimates how likely a clearly defined event is, and it can be aggregated and scored.
Can the crowd beat the best expert? Sometimes, but not as a mathematical guarantee. Averaging protects against some individual extremes; a well-informed expert may beat the average on a given question or domain.
Why ask for independent estimates first? It reduces early social anchoring and preserves information that can disappear when people conform to an influential first speaker. Independence is limited if everyone uses the same source.
Can discussion make the result worse? Yes. It can introduce herding, repeated claims and status effects. Structure discussion around evidence and let participants revise without coercion.
Is 70% a promise that the event will happen? No. It is an uncertainty estimate. A single 70% forecast can legitimately resolve negatively; calibration is assessed over many suitable events.
What is the Brier score? A proper scoring method for probabilistic forecasts. Binary reporting may use one-category squared error or a two-category sum; always state which convention and denominator you use.
Should people be rewarded for changing their minds? Evidence-based updates are valuable, but performance must be compared at meaningful horizons. A perfect guess just before the official outcome is not the same as an early useful forecast.
What if the event cannot be resolved? Apply the predetermined voiding or unresolved rule, document the evidence limitation, and avoid retroactively changing the event definition to benefit a forecaster.
Can AI provide one of the estimates? It can supply a clearly labelled forecast when the task and safeguards permit, but sharing the same data and prompts as human contributors can make errors correlated. Never count one model’s multiple paraphrases as independent observers.
Research and onward learning
Primary research includes Mellers et al., Psychological Strategies for Winning a Geopolitical Forecasting Tournament (2014), Mellers et al., The Psychology of Intelligence Analysis (2015), and Ferreiro, Deroy and Bahrami, Compromising Improves Forecasting (2023). These studies investigate forecasting under identifiable conditions; all supplier, school and experiment calculations in this article are invented teaching scenarios.
For the wider account of human and organisational intelligence, visit the How Intelligence Works guide. Continue to Collective Blind Spots for evidence coverage and Institutional Forgetting for keeping practical knowledge available after people move on.
Readers can also explore learning that transfers into life: the forecasting habit is to make a claim testable, respect evidence, update without embarrassment, and separate likely outcomes from responsible action.