The Gold Standard of Econometrics

Suppose a town introduces a travel credit for eligible adults. “Did transport improve?” leaves several decisions hidden. A more precise question is: how did offering the credit affect mean monthly bus journeys per eligible adult over the following six months?

Write down the intervention, the eligible population, the outcome, the time window and the comparison. Also distinguish the effect of offering a credit from the effect of actually using one. These are different quantities to estimate, often called estimands.

The practical question may extend beyond journeys. A programme might increase travel while changing crowding, public expenditure or access unevenly. Keep those additional outcomes explicit. A single estimated coefficient cannot settle the entire policy decision.

Learning task: write a one-sentence question that names all five elements. Then identify one decision it could inform and one decision that would require further evidence.

Continue with: The Gold Standard of Economics · The Gold Standard of Economists | Causal Evidence and Clear Policy Trade-offs

Contents · Next chapter · Specialist directory

A cross-section records units at a particular time; a time series follows an aggregate or variable through time; a panel follows the same units across periods. These structures affect the comparisons that can be made. Repeated observations from one person also require attention to dependence when estimating uncertainty.

For the fictional town, decide whether a row represents a person-month, a journey or an area-month. Counting journeys per recorded traveller answers a different question from counting journeys per eligible adult, including people who made no journeys. Denominators belong beside the numbers.

Create a data dictionary: variable name, definition, unit, period, source and missing-value codes. Check whether eligibility definitions or ticket-recording systems changed. A blank value might indicate a failed recording rather than zero travel.

Before choosing how to handle missing observations, investigate why they are missing and whom their absence removes. Preserve the original data and record every cleaning decision. A larger spreadsheet cannot repair an outcome that was measured inconsistently.

Continue with: The Gold Standard of Information · PG1, PG2 and PG3 Graph and Data Literacy | 2027 SEC Mathematics, Science, Geography, Economics and Computing

Contents · Previous chapter · Next chapter · Specialist directory

A regression describes how an outcome varies with explanatory variables in a specified model. Consider the invented teaching equation: predicted monthly journeys = 10 + 2 × credit value measured in tens of dollars. Here, an extra ten dollars corresponds to two additional predicted journeys within that equation. These numbers were chosen for explanation and were never estimated from data.

The intercept is the model’s prediction at a credit value of zero. Its substantive interpretation depends on whether that value and the surrounding range make sense. Extrapolating far beyond the observed values needs separate justification.

With several explanatory variables, a coefficient describes an adjusted relationship conditional on the other included variables and the model’s form. Whether it estimates a causal effect depends on the research design and assumptions. The IMF introduction to regression provides an accessible starting point.

Learning task: explain a coefficient aloud using the outcome’s unit, the explanatory variable’s unit and the relevant population. If any of those are unclear, return to the question and data dictionary.

Continue with: The Gold Standard Of Mathematics · The Gold Standard of Statistics

Contents · Previous chapter · Next chapter · Specialist directory

An estimate should be accompanied by an appropriate account of uncertainty and the assumptions used to calculate it. Reporting a sample size alone leaves important questions unanswered: how were observations selected, how are they related, and how was the standard error calculated?

The American Statistical Association statement explains why a p-value does not measure the probability that a hypothesis is true or the size and practical importance of an effect. A threshold alone provides an inadequate basis for a decision. Report the estimate, its units, uncertainty, analytical choices and context together.

In our transport case, a small increase could still matter to a particular group; a larger increase could be too costly for the stated objective. Assessment needs costs, distribution and alternative uses of resources as well as statistical evidence.

The four means in Chapter 7 supply no sample sizes, variation or dependence information. We therefore calculate the arithmetic contrast without inventing a standard error, confidence interval or significance claim.

Continue with: The Gold Standard of Probability · The Gold Standard of Statistics

Contents · Previous chapter · Next chapter · Specialist directory

People who choose to use a travel credit may differ from people who do not. They may already travel more, live nearer a bus route or have different working hours. Comparing users with non-users could mix those differences with the effect of the credit.

Omitted-variable bias can arise when an excluded factor influences the outcome and is related to an included explanatory variable. Reverse causality creates another difficulty: an explanatory variable may respond to the outcome. MIT’s lecture on omitted-variable bias develops the reasoning behind these problems.

Adding controls needs a causal explanation. A variable changed by the programme may lie on the very pathway being studied; conditioning on it can change the question or introduce bias. A long list of controls does not establish that all relevant differences have been addressed.

Learning task: draw the proposed chain from credit offer to travel. Add route access and prior travel habits, and explain which comparisons would be misleading. Use the diagram to expose assumptions before selecting a regression.

Continue with: The Gold Standard of Epistemology · The Gold Standard of Economists | Causal Evidence and Clear Policy Trade-offs

Contents · Previous chapter · Next chapter · Specialist directory

A well-designed randomized evaluation assigns access to an intervention using a random mechanism. This supports a comparison of groups that are comparable in expectation, although a particular sample can show chance differences. J-PAL’s introduction explains the comparison and its practical implementation.

For a fictional credit lottery, comparing everyone assigned an offer with everyone assigned no offer estimates the effect of assignment, often called the intention-to-treat effect. Comparing only people who redeemed the credit changes the comparison and can restore selection problems.

Random assignment concerns who receives an intervention. Random sampling concerns who enters the study. Keep those operations distinct when discussing whether findings apply beyond the study population. Implementation, missing follow-up outcomes and effects between participants also need examination.

An evaluation report should show how assignment occurred, how many units were assigned, how outcomes were collected and what happened after assignment. A reader needs to understand the intervention as delivered and the comparison that survived implementation.

Continue with: The Gold Standard of a Ministry of Education | Education Research, Evidence Synthesis and Policy Learning · The Gold Standard of a Ministry of Education | Education Reform Pilots, Responsible Scaling and Sunset Rules

Contents · Previous chapter · Next chapter · Specialist directory

Suppose Area A introduces the credit and Area B does not. The table contains wholly fictional mean monthly bus journeys per eligible adult. The two areas begin at different levels.

Area A’s before-and-after change is 28 − 20 = 8. Area B’s change is 21 − 18 = 3. The difference-in-differences contrast is 8 − 3 = 5 additional journeys per eligible adult per month. If A would have shared B’s change without the credit, its counterfactual mean would be 20 + 3 = 23, compared with the observed fictional mean of 28.

The causal argument requires parallel counterfactual trends: without the credit, the groups would have experienced the same mean change. The World Bank researchers’ explanation sets out this two-group, two-period design. Changes in who is observed, anticipation, spillovers or a concurrent change affecting one area can undermine the comparison.

One pre-intervention observation cannot show a pre-intervention trend. Seek earlier outcomes and contextual evidence about how the areas were chosen. Even similar earlier trends would support scrutiny rather than prove the unobserved counterfactual.

Learning task: suppose Area A also introduced a new bus route in the same month. Explain why the subtraction still equals five but attributing all five journeys to the credit becomes difficult. Identify the extra evidence needed before making that attribution.

Continue with: The Gold Standard of Economists | Causal Evidence and Clear Policy Trade-offs

Contents · Previous chapter · Next chapter · Specialist directory

An instrumental variable is a source of variation used to investigate an exposure whose ordinary comparison may be confounded. In a credit-offer lottery, assignment might be considered as an instrument for actual redemption. This requires a separate argument about how assignment changes redemption and outcomes.

The standard binary-treatment framework requires relevance, independence, an exclusion restriction and monotonicity for the usual local average treatment effect interpretation. Exclusion means the instrument affects the outcome only through the treatment as defined. An offer that independently changes travel planning could challenge that argument.

Under the relevant assumptions, the estimate concerns people whose treatment status changes because of the instrument. It does not automatically describe the effect for every eligible adult. MIT’s instrumental-variables notes explain these assumptions and interpretation.

Learning task: name the instrument, treatment and outcome separately. Describe one plausible direct route from the proposed instrument to the outcome. Explain what evidence would make that route less credible before accepting a causal estimate.

Continue with: The Gold Standard of Economists | Causal Evidence and Clear Policy Trade-offs · The Gold Standard of Probability

Contents · Previous chapter · Next chapter · Specialist directory

Forecasting asks what may happen next. A forecast of next month’s journeys can be useful even when the model does not identify the causal effect of the credit. Keep the forecast target and the causal question explicit.

Evaluate forecasts in chronological order. Train on earlier observations, forecast a later period and repeat as appropriate, using the horizon the decision actually needs. Information from the future must stay out of model fitting and preprocessing for each evaluation point. Compare performance with a sensible simple benchmark.

A prediction interval expresses uncertainty about a future outcome under the forecasting framework and its assumptions. Changes in routes, prices or recording systems may make past performance less informative. Forecasting: Principles and Practice explains time-series cross-validation and prediction intervals.

In the fictional case, a forecast of total area journeys would also need an assumption about the future eligible population. A prediction per adult cannot be silently presented as a total. Write down that conversion and test how population scenarios affect it.

Continue with: The Gold Standard of Futures Studies · Ministry of Education V3.0 Reference 30 | Scientific Uncertainty and Forecast Literacy: Decisions Without False Certainty

Contents · Previous chapter · Next chapter · Specialist directory

Residuals record differences between observed and fitted outcomes. Patterns across fitted values, time or groups may reveal aspects of the data the model misses. The NIST discussion of residual analysis introduces diagnostic plots. The checks required depend on the model and inference method.

For the credit example, inspect whether errors are systematically larger during holidays or near particular routes. Repeated errors within an area may matter for uncertainty calculations. A visually tidy residual plot still leaves the causal comparison and its assumptions to be defended.

Sensitivity checks should address plausible alternatives: a different justified time window, a documented measurement change, influential observations or a reasonable comparison group. Explain why each check is informative. Keep an account of the specifications tried and report material changes in the conclusion.

Learning task: write the assumption each proposed check examines. If a check changes the result substantially, explain the source of that sensitivity rather than selecting the most convenient output.

Continue with: The Gold Standard of Statistics · The Gold Standard of Epistemology

Contents · Previous chapter · Next chapter · Specialist directory

Preserve the source version, data dictionary, cleaning code, model specification and instructions needed to recreate each reported result. Label exploratory work and record departures from a prior analysis plan. Where data access must be restricted, explain those limits and provide shareable documentation or appropriately designed teaching data.

Write the conclusion in the same units and population as the original question. Separate what the records show, what the design supports and what remains dependent on assumptions. Describe costs and distributional consequences as additional evidence requirements when they have not been measured.

For our fictional table, a careful statement is: “Area A’s mean increased by five journeys per eligible adult more than Area B’s mean. Interpreting that contrast as a programme effect requires a credible counterfactual comparison and appropriate uncertainty analysis.” The teaching example itself establishes no real policy finding.

A useful handover lets another reader trace a sentence in the conclusion back to a table, calculation, variable definition and source. This is how quantitative learning becomes work that other people can inspect and improve.

Continue with: The Gold Standard of Information · The Gold Standard of a Ministry of Education | Education Research, Evidence Synthesis and Policy Learning

Contents · Previous chapter · Next chapter · Specialist directory

Start with Economics to understand incentives, constraints and the substantive question. Use Mathematics, Probability and Statistics to develop the tools for modelling variation. The teaching and graph-reading guides support the language needed to interpret a table before fitting a model.

Move to the Economists guide for causal evidence and policy trade-offs, and to the education research and reform-pilot articles for evaluation in an institutional setting. Forecast literacy and Futures Studies help distinguish an estimated past effect from a conditional view of the future. Finance and Operations Research connect evidence to particular decisions and constraints.

A transferable learning exercise has four outputs: a precise question, a transparent comparison, an interpreted result and a statement of what could change the conclusion. Ask a learner to explain why the fictional before-and-after change of eight becomes a difference-in-differences contrast of five, then identify the assumption the subtraction cannot verify.

The gold standard is reached when a reader can follow both the calculation and the argument that gives it meaning. Use the directory below to continue the part of that argument that needs more work.

Continue with: The Gold Standard of Teaching Statistics | Questions, Variation and Valid Inference · How to be Good at Statistics · The Gold Standard of Operations Research

Contents · Previous chapter · Specialist directory