Causal inference asks what would change if an intervention, exposure or policy were different. It separates a measured association from an effect supported by a credible comparison and explicit assumptions. The essential question is what would have happened to the same people or system under a meaningful alternative.
This guide connects counterfactuals, causal diagrams and randomised evidence with observational methods such as difference-in-differences, regression discontinuity and target-trial emulation. The original calculations and diagnostic cases show when apparently convincing numerical contrasts can still fail to identify a cause.
Choose a causal question
Counterfactual and causal structure · Chapters 1–9 · Start here →
Identification and fair comparisons · Chapters 10–17 · Start here →
Quasi-experiments and time-dependent evidence · Chapters 18–25 · Start here →
Interpretation, ethics and transfer · Chapters 26–30 · Start here →
Full contents · Worked causal cases · Further causal challenges · Specialist reading · Gold Standard master
Complete chapter index
Counterfactual and causal structure
- 1. Causation asks a different question from description
- 2. The counterfactual and the missing alternative
- 3. Define the estimand before running software
- 4. Scientific method, experimental design and causal inference
- 5. An association does not identify its cause
- 6. Confounding makes comparisons misleading
- 7. Causal diagrams organise assumptions
- 8. Mediators sit on the causal pathway
- 9. Colliders can create an association by selection
Identification and fair comparisons
- 10. Consistency requires a meaningful treatment definition
- 11. Exchangeability: a credible alternative group
- 12. Positivity means both conditions could occur
- 13. Interference occurs when treatment crosses boundaries
- 14. Random assignment and the meaning of intention-to-treat
- 15. Treatment receipt is not the same as assignment
- 16. Regression is a model, not a causal licence
- 17. Propensity scores and weighted comparisons
Quasi-experiments and time-dependent evidence
- 18. Target-trial emulation sets a clear time zero
- 19. Difference-in-differences compares two changes
- 20. Worked difference-in-differences calculation
- 21. Regression discontinuity near a rule-based cutoff
- 22. Instrumental variables are demanding causal designs
- 23. Interrupted time series and competing events
- 24. Time-varying confounding and feedback
- 25. Treatment effects need not be identical for everyone
Interpretation, ethics and transfer
- 26. Measurement and missing data affect identification
- 27. Sensitivity analysis and uncertainty boundaries
- 28. Ethics and the consequences of causal claims
- 29. Transfer the reasoning, not an unrelated percentage
- 30. Frequently asked questions and practical next steps
What the observation establishes—and what it does not
| Observed fact | Supported description | Additional causal question |
|---|---|---|
| Scores rose after tutorials | A before-and-after change | What would they have been without tutorials? |
| Participants outscored others | Difference between selected groups | Why did they participate? |
| Randomly offered groups differed | Difference by assigned offer | How were adherence and outcomes handled? |
| Results jumped near an eligibility threshold | Local discontinuity | Was assignment truly based on the rule? |
1. Causation asks a different question from description
Causal inference investigates what would change if an action, exposure, policy or condition were different. A pupil’s examination scores improve after joining extra tutorials; the change is real, but it does not by itself prove the tutorials caused it. Other teaching, practice, maturation and differences between assessments may also matter.
Begin with a clear intervention, a meaningful alternative and a defined outcome. State who is affected and over which period. The research problem is to estimate what would have happened to comparable people or systems under the alternative condition, not simply to explain why two observed measurements are different.
2. The counterfactual and the missing alternative
Imagine a student receiving a particular learning programme. We observe the result after the programme, not the result the same student would have produced at the same time without it. Potential-outcomes notation writes these alternatives as Y(1) and Y(0). Their difference is a theoretical individual treatment effect, although both cannot ordinarily be observed simultaneously for the same individual.
The challenge is constructing credible information about that missing counterfactual. Researchers use randomised groups, observational comparisons, thresholds or other designs. Every estimate depends on why the compared data can stand in for the unobserved alternative. Without that reasoning, a neat numerical effect is not a causal answer.
3. Define the estimand before running software
An estimand specifies the exact causal quantity sought. The average effect if everyone in a population received a programme differs from the effect among those who chose to participate, or the effect of merely being offered access. These quantities can diverge when effects vary across people and programme uptake is selective.
State the population, treatment versions, outcome, comparison and follow-up horizon in ordinary language. Then choose an estimator suitable for that target. Changing the question after seeing which coefficient looks favourable is a common way to manufacture unwarranted certainty even without making an arithmetic error.
4. Scientific method, experimental design and causal inference
Experimental design plans assignment, controls and measurement before intervention. Causal inference develops the broader reasoning connecting such designs and observational evidence to a causal claim. The scientific method provides the overall discipline of forming and criticising explanations. The fields overlap, but a causal question is more specific than noting a pattern or conducting an experiment.
Use Experimental Design when planning controlled studies, and Scientific Method for wider evidence reasoning. This guide concentrates on counterfactuals, identification assumptions and what different study designs can genuinely establish.
5. An association does not identify its cause
Schools with frequent enrichment activities might have stronger academic results, yet the association could reflect selection, earlier attainment, family support, school resources or the activities themselves. More records can estimate that association very precisely while leaving the causal question unanswered. Temporal ordering helps, but an earlier cause may influence both participation and later outcomes.
Develop competing explanations before selecting variables or statistical methods. Ask what changed, why it changed for one group and not another, and which background factors shape both choices and results. Causal inference requires explaining the data-generating process, not just choosing an attractive regression line.
6. Confounding makes comparisons misleading
A confounder creates or contributes to a noncausal association between exposure and outcome. Previous knowledge can affect whether a pupil seeks tuition and how well that pupil later performs. Comparing tuition attendees with nonattendees may therefore combine the effects of tutoring with differences that existed beforehand.
Measured pre-treatment information can help support adjustment when assumptions are appropriate. However, important differences may remain unmeasured. Adding every available variable is not a guarantee of fairness; variables need to be chosen for their role in the causal structure. A model with dozens of controls can still answer the wrong question.
7. Causal diagrams organise assumptions
A directed acyclic graph represents a proposed causal story with arrows among variables. In a simple educational model, previous attainment points toward both tuition choice and later attainment. A treatment arrow may also point toward the later outcome. The diagram reveals potential paths producing association even when treatment has no effect.
An arrow is not made true by being drawn. The researcher must justify direction and timing using theory, observation and design. Drawing two plausible alternative diagrams can be especially useful: if the required adjustment sets differ, the result may depend on assumptions that should be openly discussed.
8. Mediators sit on the causal pathway
A mediator is a variable through which an intervention may influence the outcome. A new reading curriculum may increase independent reading, which may improve comprehension. If the goal is the total effect of the curriculum, adjusting for new reading habits can remove some of the pathway of interest.
Distinguish baseline habits from habits measured after intervention. The same named variable can play different roles at different times. When direct or mediated effects are the target, additional estimands and assumptions are needed. Do not classify every outcome-related variable as a confounder merely because software offers an adjustment.
9. Colliders can create an association by selection
A collider is influenced by two other variables. Conditioning on it can create an association between its causes. For instance, admission to a selective programme may depend on academic achievement or exceptional artistic accomplishment. Looking only among admitted students can produce an apparent negative relationship between these strengths even when no such population relationship existed.
This selection mechanism matters whenever participation, completion or eligibility is influenced by several causes. A filtered dataset may be more biased than the original data. Before adjusting for a variable or restricting a sample, ask whether that choice opens a noncausal path.
10. Consistency requires a meaningful treatment definition
Consistency links the observed outcome to the potential outcome under the treatment actually received. But labels such as ‘tutoring’ can cover very different interventions: one video, weekly individual teaching or months of specialised support. A causal effect for an undefined mixture is hard to interpret and difficult to transfer.
Specify duration, content, delivery, intensity and timing. If programme versions matter, compare clearly described alternatives or define a justified broader policy. An estimate of an intervention’s effect should not become a generic promise that every activity sharing the same label has the same outcome.
11. Exchangeability: a credible alternative group
Exchangeability concerns whether compared groups can represent one another’s potential outcomes after the relevant assumptions are applied. Random assignment supports this property in expectation for properly designed experiments. Observational studies require stronger claims that sufficient relevant differences have been measured and accounted for.
Similar-looking groups are not automatically exchangeable. Two pupils of the same age can differ substantially in prerequisite knowledge, access to resources and reasons for participating. Present what was adjusted for, what might remain and why the comparison has causal credibility. The assumption cannot be proved from a table of balanced averages alone.
12. Positivity means both conditions could occur
Positivity requires a nonzero opportunity to receive each treatment alternative within the relevant covariate patterns of the population. If every learner with a particular need receives specialist support and none receive the comparison treatment, observational data offer little direct evidence about that missing alternative for those learners.
Investigate overlap before applying weighting or matching. Extreme weights and discarded unmatched cases can reveal weak support. Sometimes the honest estimand concerns only a narrower population where both alternatives occur. A broad population claim cannot be rescued merely by extrapolating a model beyond observed combinations.
13. Interference occurs when treatment crosses boundaries
In a simple individual-effects model, one participant’s assignment is assumed not to affect another’s outcome. But pupils share resources, team members exchange advice and changing a road alters traffic on other roads. Those spillovers make a straightforward individual treatment contrast inadequate for some questions.
Redesign the unit of intervention, define direct and indirect effects where appropriate and consider cluster or network methods. The absence of spillovers is not a default fact about social systems. The policy question may be about the whole affected community rather than only the people directly exposed.
14. Random assignment and the meaning of intention-to-treat
Random assignment helps construct a credible causal comparison by distributing potential confounders through a chance mechanism, but implementation still matters. Allocation concealment, appropriate outcome measurement, follow-up and protocol deviations must be recorded. Chance differences between small randomised groups can also remain.
An intention-to-treat comparison follows original assignment, whether or not participants fully complied. It often addresses the effect of offering or assigning an intervention. The effect among people who actually attend may be different and harder to identify, because participation after assignment can depend on motivation or circumstances.
15. Treatment receipt is not the same as assignment
A project randomly offers optional lessons. Some offered learners do not attend, while some unoffered learners find similar help elsewhere. Comparing scores by original offer preserves a randomised assignment contrast under appropriate conditions. Comparing those who attended with everyone who did not loses that original protection.
Keep assignment, actual receipt and adherence as separate records. If the causal effect of receipt is the question, additional design and modelling assumptions may be needed. Do not replace a small offer effect with a larger attendee association simply because the latter sounds more persuasive.
16. Regression is a model, not a causal licence
A regression can describe conditional associations and, under appropriate causal assumptions, support an estimate of a defined treatment effect. It cannot discover an unobserved cause simply because many variables are supplied. Wrong functional forms, missing interactions and lack of overlap can distort conclusions even with large samples.
Choose controls from the causal design, inspect diagnostics and explain which alternative outcomes are being compared. A regression coefficient is not self-interpreting. Readers should know what was held fixed, which population is represented and what independent evidence supports the causal assumptions.
17. Propensity scores and weighted comparisons
A propensity score represents the probability of receiving treatment conditional on specified measured covariates. Matching and weighting can help compare groups with more similar observed characteristics when positivity and model assumptions are plausible. They do not guarantee that an omitted confounder has ceased to matter.
Evaluate covariate balance rather than only predictive fit. A propensity model that perfectly separates treated and untreated units may reveal almost no overlap, which is a problem for estimation. Report any trimming or exclusions and explain the resulting target population. A matched estimate may not apply to all original applicants.
18. Target-trial emulation sets a clear time zero
A target-trial approach specifies how a hypothetical randomised study would work: eligibility, interventions, time zero, follow-up, outcome, causal contrast and analysis. Observational data are then organised to approximate that protocol as closely as possible. Hernán and Robins’ open text develops this discipline and its assumptions.
Time zero matters. If the treatment group is defined by people who must survive or remain enrolled long enough to receive an intervention, assigning their earlier person-time to treatment can create immortal-time bias. Define comparison strategies at a common starting point, and do not use information from the future to classify past treatment.
19. Difference-in-differences compares two changes
Difference-in-differences compares outcome changes over time between an intervention group and a reference group. It can account for certain fixed group differences, but causal interpretation relies on assumptions about untreated trends. A similar endpoint or visually parallel pair of lines alone does not establish that the assumption holds.
Examine multiple pre-intervention periods when available, changes in composition, anticipation and other shocks. Staggered policy timing needs specialised care, especially when treatment effects differ across groups or time. State the chosen method and how it connects to the estimand rather than treating the two-subtraction formula as a universal causal engine.
20. Worked difference-in-differences calculation
Invented programme averages: the participating group rises from 60 to 70, while a comparable nonparticipating group rises from 65 to 69 over the same interval. The changes are 10 and 4 points respectively, so the difference-in-differences contrast is 10−4=6 points.
The figure is arithmetically correct but not automatically an effect. Different examinations, a concurrent reform or nonparallel underlying trends could explain some of the six points. Identify whether the comparison supports the untreated-trend assumption before turning the number into a claim about the programme.
21. Regression discontinuity near a rule-based cutoff
A regression discontinuity design can compare people or institutions near a threshold that changes intervention eligibility. For example, a known score cutoff may determine access to additional support. When relevant continuity and assignment conditions hold, observations just above and below the threshold can support a local causal estimate.
The result ordinarily concerns the neighbourhood of that cutoff, not everyone. Precise manipulation of scores, other rules changing at the same threshold and inappropriate bandwidths threaten inference. Inspect score distributions, baseline factors and plausible design alternatives rather than trusting a visually sharp plotted jump.
22. Instrumental variables are demanding causal designs
An instrumental-variable strategy uses a variable that affects treatment while meeting additional assumptions about its relation to potential outcomes. Relevance alone is not enough: a proposed instrument can be associated with the outcome through other paths, violating common exclusion conditions.
For instance, distance from a programme site might affect participation but could also affect access to transport or other services. Under certain assumptions an instrumental-variable estimate identifies a local effect for compliers rather than a general population effect. State those assumptions and the resulting estimand before comparing the number with other studies.
23. Interrupted time series and competing events
Interrupted time-series analysis tracks repeated outcomes before and after an intervention, examining changes in level or trend relative to the model. Seasonal patterns, measurement changes and other events at the same time can create a similar appearance. A one-time before-and-after pair supplies much less information about pre-existing trends.
Use sufficient pre-intervention observations, consistent definitions and suitable modelling for autocorrelation when relevant. Add a comparison series if it strengthens the design. Report when data collection methods or participant eligibility changed; otherwise a measurement discontinuity may be falsely credited to the policy.
24. Time-varying confounding and feedback
In a longitudinal study, a factor may influence later treatment while also being influenced by earlier treatment. Standard adjustment for such an intermediate variable can introduce bias or remove part of a causal pathway. Methods such as the g-formula or inverse-probability weighting address particular problems under carefully specified assumptions.
Draw a timeline showing when treatment and confounders were measured. Ensure variables needed for identification were actually observed and measured adequately. Mathematical complexity does not compensate for an unjustified causal model. Be explicit about what assumptions could fail if the situation changes over time.
25. Treatment effects need not be identical for everyone
An average effect may conceal meaningful variation across baseline attainment, age, local circumstances or access conditions. A programme may help some groups and have little effect for others. But subgroup searches conducted only after looking at results can generate apparently dramatic findings by chance.
Prespecify important comparisons when feasible and report uncertainty. Keep differences between groups from becoming ungrounded individual stereotypes. Average effects inform policy but do not guarantee outcomes for every participant. A claim about one subgroup requires data and a design that support that subgroup’s counterfactual.
26. Measurement and missing data affect identification
A treatment may be assigned fairly while its outcomes are measured differently across conditions. A new assessment form, scorer knowledge or instrument calibration can create a misleading difference. Attrition can also select a subgroup of survivors whose outcomes no longer represent everyone originally assigned.
Preserve participant flow, follow-up rates and measured outcome definitions. Distinguish missing data from zero outcomes, and disclose reasons for exclusion. A large sample with incomplete, systematically selected follow-up can produce a misleadingly narrow confidence interval around a biased estimate.
27. Sensitivity analysis and uncertainty boundaries
Confidence intervals describe uncertainty under specified statistical procedures, but do not automatically incorporate uncertainty from unmeasured confounding or an invalid counterfactual. A highly precise association can still be systematically wrong. Sensitivity analysis investigates how conclusions change under plausible deviations from assumptions.
Review alternative model specifications, negative controls, pre-trends or bounding calculations when they fit the design. Do not run checks only until they support a preferred claim. Describe what evidence would weaken or overturn the interpretation, then explain what remains useful for the decision.
28. Ethics and the consequences of causal claims
Some scientifically informative comparisons would be harmful or impermissible. Research involving children, patients, private data or essential services must follow applicable ethical approval, consent, safeguards and professional standards. The value of a clean estimate does not authorise withholding essential support or risking participant safety.
Even a reliable average effect can coexist with unequal benefits or harms across groups. Report decision context, alternative interventions and who bears costs. The goal is evidence for responsible choices, not a race to produce the largest causal coefficient.
29. Transfer the reasoning, not an unrelated percentage
A teaching trial finds an effect in one set of classrooms. A city transport agency considers a new junction design. The numerical educational effect is irrelevant, but the counterfactual reasoning transfers: define options, units, outcome timing, plausible confounders, spillovers and the comparison that could identify change.
Transport can involve rerouting onto neighbouring streets, so interference and network boundaries matter. In education, pupils can share learning strategies, creating a parallel challenge. The generalisable skill is recognising which assumptions must be rebuilt each time the domain or implementation changes.
30. Frequently asked questions and practical next steps
Does correlation prove causation? No, not by itself. Does random assignment guarantee a correct estimate? No: measurement, follow-up and implementation still matter. Can regression turn observations into a randomised trial? No. Does a significant result guarantee a practically important effect? No. Are causal diagrams proof? They make assumptions explicit so those assumptions can be challenged.
Before relying on any effect estimate, ask for the intervention, comparison, eligible population, time zero, outcome, assignment mechanism, identification assumptions and uncertainty. Then write one changed-context question the original study cannot answer without new evidence. That habit turns causal claims into disciplined decisions rather than slogans.
Twelve worked causal reasoning cases
1. A school programme with a changing test
An invented school reports a rise from 65 to 75 after a new revision programme. The second test contains different question types. A defensible claim begins with checking comparability and the absence of relevant concurrent changes. The arithmetic rise is not the same as a programme effect; a stronger evaluation needs a credible alternative outcome.
2. An apparent tutoring benefit with prior differences
Tutoring participants average 80 while nonparticipants average 70. However, participants already averaged 78 before tutoring, and nonparticipants averaged 69. Raw final differences mix selection and possible change. The study needs its estimand, baseline comparability and a defensible design before interpreting the ten-point gap as causal.
3. A mediator hidden by overadjustment
A reading intervention helps some students build a regular reading habit. The habit appears in the data after intervention begins. Adjusting for it while estimating the total intervention effect may block a relevant pathway. Specify whether the goal is a total effect or an effect not mediated through reading habit.
4. A conditional-selection trap
Suppose admission to a selective course occurs through exceptional exam results or exceptional artistic work. Comparing only admitted pupils may induce an artificial association between exam results and artistic achievement. Admission is a common consequence of two pathways, so conditioning on it can distort the original relationship.
5. A matching model with no comparable controls
Highly vulnerable households all receive a support service. The available comparison sample contains no similar untreated households. Matching cannot generate information that was never observed, and weighting may become extreme. Restrict or qualify the estimand rather than claiming a whole-population effect based on unsupported extrapolation.
6. An offer differs from attendance
A programme randomly offers 100 pupils optional extra lessons; only 60 attend. Comparing the original offer groups answers a policy question about offering access. Comparing attendees with non-attendees may be confounded by motivation and availability. Record assignment, uptake and the exact estimand instead of switching comparisons according to which result is larger.
7. A cutoff score that applicants can adjust
Eligibility for a fictional grant begins at score 50. Applicants can influence how their score is reported, creating bunching around the threshold. A simple comparison just above and below 50 may then lack the continuity needed for a local causal estimate. Investigate the score-generation process before interpreting a jump.
8. The two-change calculation
In invented data, intervention mean scores move 60→70 while comparison scores move 65→69. Difference-in-differences equals (70−60)−(69−65)=6 points. Test whether both groups plausibly would have followed comparable untreated trends before calling the figure causal. A different exam scale only in the intervention group could invalidate the claim.
9. A staggered policy with incompatible controls
Three districts adopt a service at different dates. A generic before-after model can confound adoption with broader trends, and simple fixed-effects approaches can struggle when effects differ by district and timing. Choose a design and estimator appropriate to staggered adoption, treatment dynamics and suitable comparison periods.
10. A traffic intervention affects other roads
A junction modification improves the targeted corridor but sends vehicles onto adjacent routes. A study of only the modified road could overstate system-wide benefit. Specify whether the desired effect concerns treated-route speed, all commuters’ delay, safety or a broader network outcome, and measure spillovers accordingly.
11. An unobserved confounder could still matter
After adjusting for initial attainment, programme participants continue to outperform others. Yet family support or other unmeasured factors might influence both enrolment and later progress. Explore plausible sensitivity rather than claiming that a regression coefficient has eliminated all possible confounding.
12. A null result that changes the research question
A trial’s confidence interval includes both moderate benefit and moderate harm. Saying ‘there is no effect’ would overinterpret the data. The study may instead show that the estimate is too uncertain for the decision, inviting clearer outcome measurement, improved precision or a different feasible comparison.
Connected Gold Standard subjects
Statistical and methodological foundations
Research and policy applications
Connected practical investigations
- Economists | Causal Evidence
- Graduate Outcomes and Missing Responses
- Blinded Assessors, Revealing Filenames
Primary methodological references
- Hernán and Robins — Causal Inference: What If (open text)
- World Bank and IDB — Impact Evaluation in Practice, Second Edition
- J-PAL — Introduction to Randomized Evaluations
- J-PAL — Randomized Evaluation Data Analysis
- CDC — Program Evaluation Framework
Every person, programme and numerical comparison used in this guide is an invented teaching example, not an empirical evaluation. Real studies involving people or essential services require appropriate permissions, ethical safeguards and study-specific evidence.
Sixteen further causal identification and transfer challenges
Each case changes a common assumption about selection, timing, intervention or measurement. All figures are hypothetical. Try naming the estimand and the missing comparison before reading the repair: causal reasoning is most useful when it tells us which conclusion we have not yet earned.
1. A target-trial protocol before observing the data
Invented research question: does offering weekly support at the beginning of Secondary 1 improve later independent problem solving? Define eligibility before the term, two offer strategies, time zero, the independent assessment, duration and analysis population. Only then inspect administrative records to determine who could plausibly have been compared. If the treatment group is defined by completing ten sessions, children who dropped out before the tenth session are classified using future information. That can produce immortal-time and selection problems. A clear hypothetical protocol makes the mistake visible before modelling begins.
2. What a well-balanced matching table cannot show
A researcher matches intervention and comparison pupils by baseline marks, school year and classroom. Reported differences in those measured variables become very small. This supports a claim about observed balance, not necessarily exchangeability of potential outcomes. Family study conditions, prior tutoring and motivation might still differ. Explain why the adjustment variables were chosen and what remained unobserved. The honest result is a conditional observational estimate under assumptions, not a reconstructed randomised trial merely because the groups have matching averages.
3. Simpson’s paradox can conceal subgroup patterns
Invented outcomes: among high-risk units, a programme has 30 successes in 100 and the alternative has 2 in 10, rates 30% versus 20%. Among low-risk units, programme success is 9 in 10 and alternative success is 72 in 90, rates 90% versus 80%. Programme outcomes are higher within both groups, yet aggregated rates are 39/110≈35.5% and 74/100=74%. The reversal reflects very different group mixes. Identify the causal question and selection process before averaging rather than interpreting the pooled association as the programme’s effect.
4. An attractive confidence interval under an invalid design
A large sample estimates that programme participation is associated with three extra marks, with a very narrow statistical interval. If programme choice depends on unrecorded previous attainment, the interval may describe sampling uncertainty around a biased association. The calculation can be numerically correct without supporting a causal conclusion. Separate random-error uncertainty from causal assumptions, and report sensitivity to unmeasured confounding. More participants cannot, by themselves, remove systematic selection into intervention.
5. A negative-control outcome as a diagnostic
Suppose an educational intervention begins in September. A researcher checks whether intervention assignment predicts an outcome recorded months earlier that the intervention could not plausibly have changed. An association with this negative-control outcome may warn of confounding, selection or data linkage problems. No negative-control test proves every assumption is true when it returns null. Define why the control should be unaffected and consider whether other pathways could still connect the variables. The test is a diagnostic piece of a wider argument.
6. An interrupted series with changed reporting
A city begins a new transport policy and recorded incidents suddenly decline. On the same date, the incident-reporting app changes its default categories and some cases stop appearing in the published total. An interrupted time-series graph might show a clean downward break even if underlying events did not change. Verify definition, collection process and completeness before estimating the policy effect. If reporting was altered, use compatible measures or explicitly state that causal attribution is not supported by the available series.
7. A causal effect with a different time horizon
An after-school programme increases attendance during its first month, but the question concerns independent academic performance one year later. A short-term attendance effect is not necessarily a long-term learning effect. Define the outcome and horizon for each estimate, and check mechanisms and subsequent support. A researcher should not replace the difficult endpoint with a convenient early proxy without explaining the altered research question. The result may still be valuable for programme implementation while being insufficient for the promised educational outcome.
8. A risk ratio and an absolute change
Invented event rates under two options are 10 in 1,000 and 5 in 1,000. The second is a 50% relative reduction but only a five-per-1,000 absolute reduction, or 0.5 percentage points. Both descriptions are mathematically true. Causal interpretation requires an appropriate comparison; practical importance requires the baseline risk, affected population, uncertainty and costs. A persuasive policy statement should report absolute and relative measures rather than use the more dramatic one alone.
9. One institution per arm cannot isolate its intervention
An education experiment assigns one school to the programme and one school to usual practice. Outcomes for hundreds of children are recorded, but treatment assignment occurs at only two school clusters. Differences in staff, facilities or prior attainment can be indistinguishable from treatment. Treating every pupil as independently assigned creates a false sense of precision. Plan sufficient clusters or a different design, and make the assignment unit explicit. The number of measured people is not the same as the number of independent treatment allocations.
10. An instrument that affects outcomes through a second route
Distance to a training centre might affect the chance of attending, but it can also affect travel cost, fatigue and access to other services. Using distance as an instrumental variable could violate exclusion restrictions. A strong first-stage correlation alone does not repair that pathway. Draw an explicit causal diagram and examine plausible alternative mechanisms. If the assumptions cannot be defended, present the distance association descriptively or seek another credible design instead of a misleading causal estimate.
11. The different meaning of a treatment offer
A policy randomly offers a place in a course. Participants who accept may also receive supplementary materials, while those without an offer can study independently. The intention-to-treat contrast concerns the practical effect of offering the place under those conditions. It does not establish a pure effect of every minute of instruction received. This is a useful policy question, but its meaning should be preserved in the report. A later comparison by attendance may require stronger assumptions because choice of attendance is no longer randomised.
12. Why follow-up can become a collider
An intervention influences whether participants remain enrolled, and previous performance also influences continued enrolment. Analysing only those who remain can condition on a variable affected by both treatment and prior achievement. Even when baseline assignment was randomised, the restricted group can lose comparability. Track all assigned units, investigate attrition and predefine principled missing-outcome methods. Do not assume that analysing only ‘serious participants’ is a harmless way of improving data quality.
13. A neighbourhood programme that crosses geographical borders
A district opens a public library and neighbouring residents begin using it too. A comparison between official district residents and nearby residents may underestimate or distort the effect if the second group is partly exposed. Define exposure through actual access and specify whether interest lies in the direct effect on eligible residents or total community benefit. The comparison strategy must account for spillovers rather than treating administrative boundaries as walls that stop information and services.
14. Subgroups found only after seeing the results
An analysis explores twenty demographic and timing subgroups and highlights the one with the strongest apparent benefit, but this subgroup was not specified beforehand. Sampling noise and multiple comparisons can create seemingly impressive patterns. Label discoveries exploratory, report the scope of the search and verify them with independent evidence or an appropriate confirmatory design. A result can inspire a new hypothesis without being presented as a proven personalised treatment rule.
15. The effect of changing who is eligible
An intervention originally targets pupils with a specified prerequisite gap. A new policy extends it to all pupils, many of whom already master the skill. The average effect in the new population may differ even if the instructional mechanism works as before. Transfer needs consideration of baseline need, treatment fidelity, resources and alternative supports. Report what the original estimate applies to rather than transport its percentage without checking differences. Generalisability is a substantive causal question, not a clerical change in the report title.
16. A decision made responsibly despite residual uncertainty
A carefully designed programme evaluation reports a plausible benefit but retains uncertainty from limited follow-up and possible spillovers. Decision-makers may still choose a monitored pilot if implementation risk is low and reversibility is high. The scientific account should state what is known, what assumptions matter and which observations would trigger correction. Evidence-informed action is not the claim that every uncertainty has disappeared. It is the practice of connecting a transparent causal argument to proportionate, revisable decisions.
Continue through The Gold Standard, Learning Transfers into Life and How X Works.