eduKateSG · How Science Works
Choose your reading route
Calculate an original Wald ratio, identify its complier target, and break exclusion, independence and monotonicity separately.
Build the instrument and its population · Chapters 1–3 · Start this route
Calculate the effect and break its assumptions · Chapters 4–7 · Start this route
Evaluate uncertainty, research and the audit · Chapters 8–10 · Start this route
Full contents · Check your understanding · How Science Works master guide
Someone who chooses to use a tutorial may already differ from someone who ignores it. Comparing their later scores mixes the possible effect of the tutorial with differences in motivation, preparation and opportunity. A third variable can sometimes provide a cleaner comparison, but only if its role is carefully understood.
An instrumental variable supplies variation in the exposure that researchers hope to separate from those other differences. Its usefulness depends on a causal argument, not merely a strong correlation. This lesson builds an original numerical example, then deliberately breaks that argument in several different ways.
For the wider foundations, start with How Causal Inference Works. This article concentrates on the instrument itself: what it changes, what it must not change directly, whose effect it identifies and how uncertainty can undermine an apparently simple ratio.
Contents
Build the instrument and its population · Chapters 1–3
1. Separate encouragement, participation and outcome
Calculate the effect and break its assumptions · Chapters 4–7
4. Calculate the Wald ratio and keep its units
5. Identify the local effect instead of silently changing the question
Evaluate uncertainty, research and the audit · Chapters 8–10
8. A small denominator turns uncertainty into a central issue
CHAPTER 1 OF 10 · Build the instrument and its population
1. Separate encouragement, participation and outcome
Imagine a fictional study involving adult volunteers learning an unfamiliar diagram-reading task. Everyone can access an optional tutorial. A random half receive a neutral reminder that it is available. The other half do not receive that reminder. Later, everyone completes the same quiz under the same conditions.
Call reminder assignment Z, actual tutorial use D, and quiz score Y. Each of Z and D is either zero or one. Y is measured in score points. Randomising Z does not force D: some reminded people ignore the tutorial, while some people find it without a reminder.
Three questions now exist. Does assignment to the reminder change average scores? Does it change tutorial use? What is the causal effect of actually using the tutorial? The first two concern the randomised assignment. The third concerns an action that participants partly choose for themselves.
An instrumental-variable analysis tries to connect those questions. In this example, Z is the proposed instrument for D. Calling it an instrument is initially a proposal, not a certificate that the analysis works. The reminder could affect confidence, effort or access to other material as well as tutorial use.
The example is a paper exercise, not a report of a real study or a recommendation to run one. Its exact averages describe a constructed population with balanced assignment. A real sample would add sampling variation, missing outcomes and practical questions about consent and measurement.
CHAPTER 2 OF 10 · Build the instrument and its population
2. Draw the permitted route and the dangerous shortcuts
Picture a diagram with three main boxes: reminder Z on the left, tutorial use D in the middle and score Y on the right. The intended route is Z → D → Y. Below D and Y, place an unmeasured preparation variable U, with arrows into both. That lower fork explains why simply comparing users with non-users can mislead.
The instrument argument requires that assignment Z be independent of the relevant potential outcomes and participation responses, possibly after justified conditioning in an observational design. Random assignment provides a design-based reason for that independence here. It does not guarantee exact baseline balance in every finite sample.
Relevance means Z actually changes D. A reminder that leaves tutorial use unchanged provides no leverage for this calculation. The first-stage comparison measures that change. It should be reported with uncertainty rather than reduced to a label saying that the instrument passed.
Exclusion means Z affects Y only through D. Add a direct arrow Z → Y to the picture if the reminder itself increases effort during the quiz. That shortcut defeats the simple interpretation, even though assignment is random and tutorial use changes substantially.
The example also assumes a well-defined tutorial and outcome, consistency between assigned conditions and observed responses, and no interference between participants. If reminded people teach their unreminded friends, one person’s assignment can change another person’s outcome. A more suitable design and estimand would then be needed.
CHAPTER 3 OF 10 · Build the instrument and its population
3. Construct the people behind the group averages
Our fictional population contains three response types. Forty percent are compliers: they use the tutorial if reminded and do not use it otherwise. Thirty percent are always-users: they use it with or without the reminder. Thirty percent are never-users: they do not use it in either assignment condition.
These names describe two potential participation decisions under this particular reminder. They are not personality labels. In real data, each person receives only one assignment, so researchers generally cannot read an individual’s type directly from their observed participation.
Give everyone a potential score of 50 without the tutorial. With the tutorial, give compliers a score of 60, always-users 70 and never-users 45. Thus the constructed tutorial effects are respectively +10, +20 and −5 points. These hidden potential outcomes are supplied by the exercise author so we can check the estimator against a known answer.
Assume assignment is independent of these types and scores, and the reminder has no direct effect. In the unreminded group, only the 30 percent always-users participate. Average participation is therefore 0.30. Its average score is 0.30 × 70 + 0.40 × 50 + 0.30 × 50 = 56.
In the reminded group, always-users and compliers participate, giving participation 0.70. Its average score is 0.30 × 70 + 0.40 × 60 + 0.30 × 50 = 60. Notice that the reminder changes the behaviour of only one type. That fact will determine whose tutorial effect the calculation recovers.
CHAPTER 4 OF 10 · Calculate the effect and break its assumptions
4. Calculate the Wald ratio and keep its units
The first stage is the difference in participation between assignment groups: 0.70 − 0.30 = 0.40. This is a 40-percentage-point increase, expressed as a proportion for the calculation. It is not a 40 percent relative increase and should not be entered as the number 40.
The outcome difference, often called the reduced-form effect here, is 60 − 56 = 4 score points. Under random assignment, this is the intention-to-treat effect of receiving the reminder in our constructed population. It is an effect of assignment even though not everybody follows the reminder.
For this binary instrument and binary exposure, the simple Wald ratio divides the outcome difference by the participation difference: 4 / 0.40 = 10 score points per change from non-use to use. Keeping the denominator in proportions is essential. Dividing by 40 would instead give 0.10 points per percentage point, a differently scaled quantity.
Why does division work in this world? The reminder changes tutorial use for 40 percent of people, and each of those people gains 10 points. The population-level score change is consequently 0.40 × 10 = 4. Dividing removes the dilution from people whose participation never changes.
This is a causal explanation of the ratio, not a rule that any pair of group differences can be divided to reveal causation. Without the required assumptions, the arithmetic still produces a number. The number may cease to describe any useful average tutorial effect.
CHAPTER 5 OF 10 · Calculate the effect and break its assumptions
5. Identify the local effect instead of silently changing the question
The recovered 10 points is the effect among compliers for this reminder. It is a local average treatment effect, or LATE, under the stated assumptions. The reminder never changes participation for always-users or never-users, so their tutorial effects are not recovered through its induced comparison.
Because this is an invented population, we can calculate its overall average effect directly: 0.40 × 10 + 0.30 × 20 + 0.30 × (−5) = 8.5 points. That differs from 10. An analyst with ordinary observed study data would not know all three hidden effects merely because the Wald ratio was available.
The naïve comparison of actual users with non-users differs again. With equally sized assignment groups, always-users make up 0.30 of the whole population and participating compliers another 0.20. Half the population therefore uses the tutorial. Their average score is (0.30 × 70 + 0.20 × 60) / 0.50 = 66.
All observed non-users score 50 in this construction, so the as-treated difference is 16 points. We now have three legitimate calculations answering different questions: 4 for assignment, 10 for the identified complier effect, and 16 for the observational user-versus-non-user contrast. Only the author-supplied potential outcomes reveal the overall average effect of 8.5.
A different encouragement could persuade a different subgroup, yielding a different local effect without either study being mathematically wrong. Generalising requires a reason why effects and response types transport to the new population or intervention. The word “causal” does not remove that additional requirement.
CHAPTER 6 OF 10 · Calculate the effect and break its assumptions
6. Break exclusion while keeping randomisation intact
Keep all original participation decisions and tutorial effects. Now suppose receiving the reminder also adds two quiz points through increased attention, regardless of whether the person uses the tutorial. This is a separate hypothetical world, with a direct Z → Y route.
The first stage stays 0.40. The reminded group’s mean rises from 60 to 62, while the unreminded mean stays 56. The observed outcome difference becomes 6, and the Wald ratio becomes 6 / 0.40 = 15. The true complier tutorial effect in the construction remains 10.
Randomisation has not failed. The reminder really causes a six-point average score increase through the combined routes. What fails is attributing the whole increase to tutorial use. Reporting the assignment effect could still answer a useful question about sending reminders; labelling 15 the tutorial effect would answer a different question incorrectly.
A simple sensitivity calculation exposes the issue. If the direct assignment effect were a known δ points, the adjusted ratio in this additive example would be (6 − δ) / 0.40. Values δ = 0, 1 and 2 give 15, 12.5 and 10. These are scenario calculations, not confidence intervals or estimates of δ.
A scientifically justified range for a direct effect can be informative. Choosing that range after seeing which value preserves a preferred conclusion is much less convincing. Negative-control outcomes, design details and external evidence may challenge an exclusion story, but a large first-stage association cannot establish it.
CHAPTER 7 OF 10 · Calculate the effect and break its assumptions
7. Break independence or monotonicity in different ways
Return to the original no-direct-effect world. Suppose assignment is no longer random: people receiving reminders have a three-point baseline advantage unrelated to the tutorial, while the participation difference remains 0.40. The outcome difference becomes 7, and the ratio 17.5. This is an independence failure, separate from the direct-effect example.
Measured baseline balance can reveal some problems, but it cannot prove that every relevant unmeasured difference is absent. Selecting an instrument because it predicts participation is therefore only a first step. Researchers need a convincing account of how the instrument was generated and what else could connect it to outcomes.
A further condition for the simple complier interpretation is monotonicity: no one is made less likely to participate by the encouragement when participation is binary. Our initial population has no defiers, people who would participate without the reminder but refuse with it. Monotonicity concerns paired potential decisions, not an observed group average.
Construct a new population with 50 percent compliers, 10 percent defiers, 20 percent always-users and 20 percent never-users. Give compliers a +10-point tutorial effect and defiers a +30-point effect. Under independence and exclusion, the first stage is 0.50 − 0.10 = 0.40.
The reduced-form difference is now 0.50 × 10 − 0.10 × 30 = 2 points. The ratio is 5, although the complier effect is 10. Opposite participation responses subtract rather than simply averaging together. A positive first stage alone does not rule out defiers, and their presence can radically change the interpretation.
CHAPTER 8 OF 10 · Evaluate uncertainty, research and the audit
8. A small denominator turns uncertainty into a central issue
Real first-stage and outcome differences are estimated rather than handed down as exact population values. Consider a separate scale example with estimated outcome difference 0.04 points and participation difference 0.02. Their ratio is 2 points. Holding the numerator fixed, denominators of 0.01 and 0.03 produce ratios of 4 and approximately 1.33.
Holding the denominator at 0.02 instead, numerators of 0.02 and 0.06 produce ratios of 1 and 3. These perturbations illustrate sensitivity, not a sampling distribution. Both estimates can be uncertain and correlated, so a valid interval cannot generally be made by changing only one number.
The original investigation by Bound, Jaeger and Baker showed why weak instrument relationships can create serious finite-sample and inconsistency problems. Their results also emphasised that slight instrument invalidity can matter greatly when the instrument explains little exposure variation. Weakness and invalidity are different problems that can reinforce each other.
Methods designed for weak instruments can produce wide, unbounded or otherwise nonstandard confidence sets when the data carry little identifying information. Such results may be an honest description of the evidence. They should not be replaced by a neat narrow interval simply because it is easier to display.
First-stage statistics remain useful diagnostics, but no convenient threshold verifies every assumption or guarantees every familiar interval. Weak-instrument-robust inference also does not automatically repair direct effects, confounding or an incorrectly defined outcome. Statistical protection has a scope that must be stated.
CHAPTER 9 OF 10 · Evaluate uncertainty, research and the audit
9. Read the research as an argument with inspectable parts
Angrist, Imbens and Rubin’s 1996 methods paper placed instrumental-variable reasoning within a potential-outcomes framework. It made explicit the assumptions supporting a complier interpretation and examined sensitivity when key assumptions change. The important lesson for this worksheet is that a ratio and a causal estimand are not interchangeable by definition.
A current 2026 article by David S. Lee and Jack Porter examines inference with one endogenous regressor and one instrument. It explains failures of conventional intervals under weak identification and discusses robust alternatives. Its warning against treating a first-stage F above 10 as a universal clearance certificate is especially relevant when reading apparently confident numerical results.
Research on assumption sensitivity also continues. Jung Hyub Lee’s September 2026 preprint proposes an Anderson–Rubin-based procedure allowing prespecified ranges of direct instrument effects under its maintained sampling assumptions. It is a methodological proposal, not evidence that arbitrary direct effects can be ignored. Our elementary δ exercise illustrates the question; it does not implement that paper’s inferential procedure.
For any application, locate the instrument-generation story, the first stage, the assignment–outcome comparison and the claimed target population. Then inspect the exclusion argument and the uncertainty method. A paper that reports only the final ratio hides precisely the information needed to distinguish a credible design from convenient division.
Also inspect outcome availability by assignment group. Differential missingness can spoil a comparison that began with random assignment. Ask whether the reported analysis preserves the original assignment, what it assumes about missing values, and whether the tutorial’s definition remained stable across groups and time.
CHAPTER 10 OF 10 · Evaluate uncertainty, research and the audit
10. A paper-only instrument audit
Copy the original fictional group means: participation 0.30 versus 0.70, score 56 versus 60. Write three labelled answers before interpreting them: first stage 0.40, assignment effect 4 points, Wald ratio 10 points. Then write the population to which the last answer applies and the assumptions that support that claim.
Next change one feature at a time. Add a two-point direct reminder effect and obtain 15. Instead add a three-point baseline assignment imbalance and obtain 17.5. In the separate defier population, obtain 5. Keeping these worlds separate makes clear that the same numerical method can fail for different causal reasons.
Finally, compare the initial ratio with the as-treated contrast of 16 and author-known population effect of 8.5. Explain why disagreement is expected. Do not try to select the largest or most statistically impressive number as the correct effect: identify the question each calculation actually answers.
A useful written conclusion has four parts: what changed the exposure, why that change is plausibly independent, which alternative outcome routes remain possible, and whose effect is identified if the assumptions hold. Add uncertainty appropriate to the design and instrument strength. If any part is unsupported, narrow the conclusion rather than hiding the gap behind technical vocabulary.
Continue through Science Mastery and the How Science Works roadmap. For assignment itself, use How Randomisation Works. An instrument becomes scientifically useful when its permitted causal route survives scrutiny, not merely when its calculation produces an answer.