Regression works by choosing a mathematical relationship between an outcome and one or more predictors, estimating the unknown parameters of that relationship from observed data, and then using the fitted model to describe conditional associations, predict outcomes, quantify uncertainty or—when the study design and causal assumptions justify it—estimate causal effects. In ordinary linear least squares, the fitted coefficients are chosen to minimise the sum of squared residuals between observed and predicted outcomes.
Regression is one of the great workhorses of modern statistics.
It appears in economics, medicine, education, engineering, psychology, epidemiology, machine learning and policy analysis.
Its familiarity creates a peculiar risk.
A table of coefficients can look like an explanation of reality even when it is only a fitted conditional summary.
The same regression equation can be used to describe association, predict an outcome, adjust for measured covariates or estimate a causal effect.
The equation does not tell us which job it is doing.
The governing question: what quantity does this regression coefficient represent inside this model, over this data range, under these assumptions—and is that the quantity the scientific claim actually needs?
Quick Read
QUESTION → OUTCOME → PREDICTORS → FUNCTIONAL FORM → DESIGN MATRIX → LOSS / LIKELIHOOD → FIT COEFFICIENTS → PREDICTED VALUES → RESIDUALS → STANDARD ERRORS → DIAGNOSTICS → INTERACTIONS / NONLINEARITY / DEPENDENCE CHECK → VALIDATION → INTERPRETATION → CAUSAL BOUNDARY
NIST describes linear least squares regression as the most widely used modelling method and a statistical workhorse. It also emphasises that residual analysis is essential: departures from expected residual behaviour often reveal structure omitted from the model.
1. Regression Gives One Variable a Special Role: the Outcome
Correlation is symmetric.
Regression is not.
We designate Y as the outcome and X as one or more predictors.
Regressing exam score on study time asks a different modelling question from regressing study time on exam score.
This asymmetry is one of the first boundaries between correlation and regression.
2. Simple Linear Regression Models a Conditional Mean
The familiar model is:
E(Y|X) = β₀ + β₁X
The line represents the expected or average Y value at each X under the model.
Individual observations vary around that conditional mean.
Regression is therefore not the claim that every point lies on the line.
3. The Intercept Is the Modelled Outcome When Predictors Equal Zero
β₀ is the intercept.
Its practical meaning depends on whether X = 0 is meaningful and inside the observed range.
If student age ranges from 13 to 16, an intercept at age zero is a mathematical anchor rather than a scientifically useful prediction.
Centering predictors can make the intercept more interpretable.
4. The Slope Is a Conditional Difference per Unit of X
If β₁ = 3.2 in a simple model of score on study hours, the fitted conditional mean increases by 3.2 score units for each additional hour of X.
That is a modelled association.
It becomes a causal effect only under a causal design and identification assumptions.
5. Least Squares Chooses the Line With the Smallest Squared Residual Sum
For each observation, calculate:
residual = observed Y − predicted Y
Square each residual, sum them, and choose coefficients that make this total as small as possible.
NIST describes ordinary linear least squares exactly in this minimisation language.
6. Why Square the Residuals?
Squaring prevents positive and negative errors from cancelling.
It penalises large errors strongly.
It produces convenient mathematics and, under normal-error assumptions, coincides with maximum likelihood estimation.
But squared loss is not the only possible loss function.
7. Least Absolute Deviations Changes the Error Geometry
Instead of minimising squared errors, one can minimise absolute errors.
This reduces the influence of extreme residuals and connects to median regression.
The fitting rule defines which mistakes the model treats as costly.
Regression is partly a choice of geometry.
8. Multiple Regression Adds Predictors to the Conditional Model
With several predictors:
E(Y|X) = β₀ + β₁X₁ + β₂X₂ + …
Each coefficient describes how the fitted conditional mean changes with that predictor while the other modelled predictors are held fixed.
“Holding fixed” is a mathematical conditioning statement, not automatically a real-world intervention.
9. Coefficients Depend on Which Other Variables Are in the Model
A study-time coefficient may be 5 in a simple model.
Add prior attainment and it may become 2.
Add school fixed effects and it may change again.
The coefficient is conditional on the model specification.
There is no context-free regression coefficient floating independently of the other variables included.
10. Covariate Adjustment Can Remove Confounding—or Create Bias
If a variable is a common cause of exposure and outcome, adjustment may be necessary for causal identification.
If it is a mediator, adjustment can block part of the causal effect.
If it is a collider, adjustment can open a biasing path.
Regression cannot decide causal variable roles from correlation alone.
11. The Same Regression Can Be Descriptive, Predictive or Causal
Descriptive regression summarises conditional relationships.
Predictive regression aims to forecast unseen outcomes.
Causal regression estimates intervention effects under identification assumptions.
These jobs can use identical equations and produce identical coefficients.
The scientific meaning comes from the design and target, not the software command.
12. Linear Regression Means Linear in the Parameters
NIST emphasises a frequently misunderstood point.
A linear regression can include X², log(X), sine terms and other transformed predictors while remaining linear in the coefficients.
The curve need not be a straight line in X.
“Linear model” describes parameter structure, not always visual shape.
13. Polynomial Terms Model Curvature
A model with X and X² can bend.
A quadratic model may capture diminishing returns, U-shaped risk or an optimal middle region.
Higher-order polynomials can fit more complex shapes but can behave wildly at boundaries and extrapolate poorly.
Flexibility should follow mechanism and validation, not polynomial enthusiasm.
14. Splines Provide Flexible Local Curvature
Splines join low-degree polynomial pieces smoothly across chosen knots.
Restricted cubic splines are widely used to model nonlinear continuous relationships without forcing one global polynomial.
The resulting model can remain interpretable through plots and contrasts even when no single slope summarises the full relationship.
15. Transforming a Predictor Changes the Meaning of the Coefficient
Regress Y on log(X) and β now represents change in Y per unit change in log X.
For small proportional changes, this can be interpreted in percentage terms.
Transformations can improve fit and scientific representation, but coefficients must be translated back carefully.
16. Transforming the Outcome Changes the Target Scale
A regression of log income on education models the conditional mean of log income, not directly the arithmetic mean of income.
Back-transforming predictions can require bias corrections because E(exp(log Y)) is not generally exp(E(log Y)).
Scale choice changes the estimand.
17. Categorical Predictors Become Indicator Variables
A three-level school type cannot enter ordinary regression as the numbers 1, 2 and 3 unless those numbers represent meaningful spacing.
Instead, the model creates indicator variables relative to a reference category.
Each coefficient describes a conditional contrast against that reference.
18. The Reference Category Changes Coefficients but Not Fitted Values
Choose School A as reference and estimate B−A and C−A.
Choose School C as reference and the coefficient table changes.
The fitted group means do not.
Parameterisation changes the coordinate system, not the underlying fitted model.
19. Interaction Means the Effect of One Predictor Depends on Another
Add X×Z to a model.
Now the slope of X changes with Z.
A tutoring intervention might have a larger effect for lower-baseline students than higher-baseline students.
Interaction turns “the effect” into a conditional surface.
20. Main Effects Change Meaning When an Interaction Is Present
In Y = β₀ + β₁X + β₂Z + β₃XZ, β₁ is the X slope when Z = 0.
If Z = 0 is not meaningful, β₁ can be difficult to interpret.
Centering Z at a meaningful value can make the main effect represent the X slope at that reference condition.
21. “Significant in One Group, Not in Another” Is Not an Interaction Test
If the treatment coefficient is significant in boys and not significant in girls, that does not prove the coefficients differ.
The interaction term directly estimates the between-group difference in slopes or treatment effects.
Differences in p-values are not p-values for differences.
22. Residuals Are What the Model Failed to Explain
NIST defines residuals as observed minus predicted responses and emphasises that examining them is a key part of statistical modelling.
A good residual pattern should look like unstructured variation relative to the job the model claims to do.
Visible structure means information remains in the errors.
23. Residuals Versus Fitted Values Reveal Missed Structure
A U-shaped residual pattern suggests missing nonlinearity.
A funnel shape suggests changing variance.
Distinct bands may reveal omitted groups.
Residual plots are not cosmetic diagnostics after the “real” analysis.
They are evidence about whether the representation is adequate.
24. Linearity Is About the Conditional Mean Structure
Ordinary linear regression assumes the specified linear combination correctly represents the conditional mean over the range of interest.
If the true mean curve bends and the model uses one straight line, coefficients average over that curvature.
Large samples make this wrong average more precise; they do not make the straight-line assumption true.
25. Homoskedasticity Means Constant Conditional Error Variance
Classical ordinary least squares inference often assumes residual variance is constant across predictor levels.
If low-income observations vary modestly and high-income observations vary enormously, the error is heteroskedastic.
OLS coefficients can remain useful under some forms of heteroskedasticity, but conventional standard errors may be wrong.
26. Heteroskedasticity-Robust Standard Errors Repair a Variance Formula, Not the Mean Model
Sandwich or heteroskedasticity-consistent standard errors can provide more reliable uncertainty estimates when error variance changes with predictors.
They do not correct an omitted nonlinear relationship, confounding or measurement bias.
Robustness is always robustness to something specific.
27. Independence Is Often More Important Than Normality
Students within one classroom share a teacher and environment.
Repeated measurements from one person share biology.
Neighbouring observations share geography.
Treating dependent observations as independent can make standard errors dramatically too small.
28. Cluster-Robust Standard Errors Recognise Grouped Dependence
Clustered standard errors allow arbitrary dependence within clusters under asymptotic conditions while treating clusters as independent units.
The number of clusters matters.
Five hundred students across eight schools do not provide the same cluster-level information as five hundred students across one hundred schools.
29. Multilevel Models Represent Cluster Structure Directly
Random intercepts allow groups to have different baselines.
Random slopes allow relationships to vary across groups.
Hierarchical models partially pool information across schools, clinics or individuals.
They model dependence rather than merely adjusting the standard-error calculation.
30. Normality Is Mainly About Conditional Errors for Classical Small-Sample Inference
Linear regression does not require the predictor to be normally distributed.
Nor does it require the outcome itself to be marginally normal.
Classical exact t and F inference uses assumptions about the conditional error distribution.
Large-sample methods can often tolerate non-normal errors when other conditions hold.
31. Q-Q Plots Diagnose Distributional Shape Better Than Ritual Normality Tests Alone
With huge samples, tiny harmless departures can make formal normality tests significant.
With tiny samples, severe non-normality may escape detection.
Graphical diagnostics show where and how the distribution departs.
NIST repeatedly emphasises graphical residual analysis for this reason.
32. Outliers Are Unusual Outcomes Conditional on Predictors
An outlier has a large residual.
It may represent data error, a rare genuine event, an omitted subgroup or model misspecification.
Automatically deleting it because it makes the model inconvenient hides a scientific question.
33. High Leverage Means an Unusual Predictor Configuration
A point far from the centre of predictor space has high leverage.
It has more capacity to pull the fitted regression surface.
High leverage is not necessarily bad.
A well-measured rare case can contain valuable information precisely because it explores a new region.
34. Influence Combines Leverage and Residual Size
Cook’s distance and related measures ask how much the fitted model changes when an observation is removed or down-weighted.
An influential point is not automatically an error.
It is an observation on which the conclusion depends strongly.
That dependence deserves inspection.
35. Multicollinearity Makes Coefficients Hard to Separate
If two predictors move almost together, many coefficient combinations can produce similar predictions.
Individual slopes become unstable and standard errors grow.
The model may predict the outcome well while being unable to assign stable separate effects to the correlated predictors.
36. Variance Inflation Factors Diagnose One Form of Collinearity
A variance inflation factor measures how much coefficient variance is increased because a predictor is linearly predictable from other predictors.
There is no universal magic VIF cutoff that determines scientific validity.
The consequence depends on whether the goal is prediction, individual coefficient interpretation or causal estimation.
37. R² Measures In-Sample Variance Reduction Relative to the Mean Model
R² compares squared residual variation from the fitted model with variation around the outcome mean.
R² = 0.70 means the fitted model accounts for 70% of observed outcome variance in that sample relative to the mean-only baseline.
It does not mean the model is 70% correct.
It does not mean 70% of Y is caused by the predictors.
38. R² Never Decreases When You Add Predictors
Add a useless predictor to ordinary least squares and the model can set its coefficient near zero.
The residual sum of squares cannot become worse in-sample.
R² therefore rewards complexity even when complexity does not generalise.
39. Adjusted R² Applies a Complexity Penalty
Adjusted R² accounts for the number of predictors relative to sample size.
It can fall when a weak predictor is added.
It is still an in-sample model summary, not a substitute for external validation or causal logic.
40. Prediction Error Must Be Tested on Data Not Used to Fit the Model
A flexible regression can fit training data beautifully and predict new data poorly.
Holdout sets and cross-validation estimate out-of-sample prediction performance.
Prediction asks how well the model travels to unseen observations from the deployment distribution.
41. Extrapolation Is Prediction Outside the Support of the Data
A line fitted to ages 13–16 can mathematically predict age 40.
Nothing in the data validates that extrapolation.
NIST lists poor extrapolation behaviour as an important limitation of least-squares models.
Models know their formula, not the boundary of reality unless we enforce it.
42. Confidence Intervals and Prediction Intervals Answer Different Questions
A confidence interval for the mean response at X estimates uncertainty about the conditional mean.
A prediction interval for a new individual adds individual residual variation.
Prediction intervals are therefore wider.
Choose the interval that matches the receiver.
43. Coefficient Standard Errors Quantify Sampling Uncertainty Under the Model
Another sample would produce different fitted coefficients.
The standard error estimates how much the coefficient estimator would vary under repeated samples or assignments according to the inferential framework.
The standard error is conditional on model and dependence assumptions.
44. A Small P-Value Does Not Make a Regression Coefficient Causal
A slope can have p < 10−12 because the association is precise.
Unmeasured confounding can still explain the association.
Statistical evidence about a conditional coefficient is not identification evidence about an intervention.
45. Logistic Regression Models Log-Odds for Binary Outcomes
When Y is binary, ordinary linear regression can predict impossible probabilities below zero or above one.
Logistic regression models the log-odds of the event as a linear predictor.
Exponentiating a coefficient gives an odds ratio for a one-unit predictor change conditional on other variables.
46. Odds Ratios Are Not Risk Ratios
When outcomes are common, an odds ratio can be much farther from 1 than the corresponding risk ratio.
Calling an odds ratio “times more likely” can therefore exaggerate intuitive meaning.
Effect-scale interpretation matters as much as model fitting.
47. Generalised Linear Models Extend Regression Beyond Normal Outcomes
Logistic regression handles binary outcomes.
Poisson and negative-binomial regression handle counts and rates.
Gamma models can handle positive skewed outcomes.
A link function connects the conditional mean to the linear predictor.
48. Poisson Regression Models Event Rates Under Mean-Variance Structure
Counts often depend on exposure time.
An offset can incorporate person-time, distance or population size.
If variation exceeds Poisson assumptions, negative-binomial or robust approaches may be more appropriate.
The outcome distribution is part of the model architecture.
49. Quantile Regression Models Conditional Quantiles, Not Means
Mean regression can hide how predictors affect different parts of the outcome distribution.
Quantile regression can model the median or other conditional quantiles.
A predictor may have little effect near the median but a large effect among the highest outcomes.
“The regression effect” depends on which feature of the distribution is targeted.
50. Survival Regression Models Time-to-Event Structure
Cox proportional-hazards regression relates predictors to event hazards without specifying the baseline hazard parametrically.
The proportional-hazards assumption means hazard ratios remain constant over time.
If effects change with time, one hazard ratio can hide the dynamic relationship.
51. Regularisation Trades Bias for Stability and Prediction
Ridge regression penalises large squared coefficients.
Lasso penalises absolute coefficient size and can set some coefficients to zero.
Elastic net combines both.
Regularisation can improve prediction under many correlated predictors by accepting some bias to reduce variance.
52. Feature Selection Creates Post-Selection Inference Problems
Search hundreds of predictors, select the strongest, then report ordinary p-values as though those predictors were chosen in advance.
The uncertainty from selection is missing.
Model search is itself part of the data-generating procedure for the final coefficient table.
53. Stepwise Regression Is Easy to Run and Easy to Overinterpret
Automated forward, backward or stepwise selection can produce unstable models, biased coefficients and overly optimistic p-values when treated as fixed-model inference.
Prediction-oriented validation or modern penalisation is often more defensible than pretending stepwise selection did not occur.
54. Cross-Validation Selects Models for Prediction, Not Causal Truth
A model can win cross-validation because a proxy predicts well.
Intervening on the proxy may do nothing.
Prediction validation protects forecast performance under similar data distributions.
It does not identify causal pathways.
55. Causal Regression Requires Identification Before Estimation
Causal inference asks what would happen under alternative interventions.
Regression can estimate the identified causal quantity after the design and assumptions specify the right adjustment set, time ordering and estimand.
Running regression first and inventing the causal story afterward reverses the correct order.
56. Conditioning on Post-Treatment Variables Can Change or Bias the Effect
Treatment changes motivation.
Motivation changes scores.
Adjust for motivation and the total effect is partly blocked.
If motivation also shares causes with the outcome, new biases can arise.
Regression adjustment must respect causal timing.
57. Missing Data Can Change Which Cases the Regression Represents
Most software silently drops rows missing any model variable.
If missingness is related to outcomes or predictors, the complete-case regression can represent a selected population.
58. Measurement Error in Predictors Can Bias Slopes
Classical random measurement error in a single continuous predictor often attenuates its slope toward zero.
With multiple variables, differential error or correlated error, bias can move in more complicated directions.
Regression coefficients inherit the quality of variable measurement.
59. Measurement Error in Confounders Leaves Residual Confounding
Adjusting for a noisy measure of socioeconomic status may not fully control the underlying confounding construct.
The regression can appear “adjusted” while important causal imbalance remains.
Variable names do not guarantee variable adequacy.
60. Education Regression Needs Multilevel Structure
Students sit inside classes.
Classes sit inside schools.
Schools sit inside neighbourhoods and systems.
A flat regression can mix within-class and between-school relationships and underestimate uncertainty if clustering is ignored.
The architecture of the data should appear in the architecture of the model.
61. AI Regression Models Can Be Accurate and Unstable Under Distribution Shift
A model trained on one school year may exploit patterns tied to a particular assessment format.
Change curriculum or student composition and the conditional relationship moves.
Regression coefficients and predictions are conditional on the data-generating environment that produced them.
62. The Hostile Test: R² = 0.95 From a Leaking Predictor
A model predicts final grades almost perfectly.
One predictor is a moderation code assigned after the exam.
The model has learned future information unavailable at prediction time.
Excellent fit can conceal invalid task boundaries.
63. The Second Hostile Test: p < 0.001, Wrong Functional Form
The true relationship is sharply U-shaped.
A straight line across an asymmetric sample produces a highly significant positive slope.
The p-value precisely describes the wrong summary.
Model specification comes before threshold interpretation.
64. The Third Hostile Test: Adjust for Everything
An observational study includes forty variables “to be safe”.
Several are post-treatment mediators.
One is a collider affected by both exposure and an unmeasured cause of outcome.
More adjustment has created more bias.
65. The Fourth Hostile Test: High Prediction Accuracy, No Actionable Cause
A regression predicts weak students from frequency of asking questions.
The school tries to reduce questioning.
The predictor was a symptom of difficulty rather than its cause.
Prediction was good.
The intervention logic was wrong.
66. The Fifth Hostile Test: Extrapolation Beyond the Data
A drug dose-response line is fitted between 5 and 20 mg.
The equation predicts benefit at 200 mg.
At that dose the biological mechanism changes and toxicity dominates.
The formula continued; the world did not.
67. Primary School: Regression Begins as “Draw the Trend, Then Check the Misses”
Students plot plant height against days.
They draw a simple trend line.
Then they notice some plants lie above and some below.
The early idea is that a model captures a pattern without replacing individual observations.
A useful line explains some structure. The distance from the line tells you what the line still does not know.
68. Secondary School: Separate Fitted Pattern From Causal Story
A trend line can summarise association.
Students should ask whether a third variable, selection or reverse causation could generate that pattern.
The graph becomes a model to question rather than a story to obey.
69. JC and University: Regression Becomes Conditional Modelling With Explicit Estimands
At higher levels, learners should reconstruct:
- outcome;
- predictors;
- estimand;
- functional form;
- coding;
- interactions;
- loss or likelihood;
- residual structure;
- dependence;
- standard errors;
- diagnostics;
- validation;
- causal identification.
70. Where Regression Fits in the eduKateSG “How Works” Landscape
- How Correlation Works — symmetric pairwise association.
- How Statistical Inference Works — estimands, estimators and uncertainty.
- How Confidence Intervals Work — coefficient precision.
- How P-Values Work — hypothesis testing for model parameters.
- How Causal Inference Works — when regression coefficients can represent intervention effects.
- How Missing Data Works — incomplete-case and imputation assumptions.
Regression owns one precise canonical job: model how an outcome’s conditional distribution or summary changes with predictors, estimate that model from data, diagnose where the representation fails and keep descriptive, predictive and causal meanings separate.
71. What This Article Does Not Claim
- Regression coefficients are not automatically causal effects.
- Linear regression does not require every visual relationship to be a straight line; it is linear in parameters.
- R² is not the percentage of an outcome caused by predictors.
- A high R² does not guarantee out-of-sample prediction.
- Robust standard errors do not repair wrong functional forms or confounding.
- Adding every available covariate can create causal bias.
- A significant coefficient does not prove practical importance.
- Normality of predictors is not a linear-regression requirement.
- Extrapolation beyond observed support is not validated by good in-range fit.
- Model diagnostics are part of inference, not optional decoration.
72. A Compact Regression Audit
- What is the outcome?
- What scientific quantity is the model meant to estimate?
- Which predictors are included and why?
- How are categorical variables coded?
- Are continuous relationships plausibly linear on the chosen scale?
- Are transformations or splines needed?
- Are interactions scientifically expected?
- What loss function or likelihood fits the outcome type?
- Are observations independent?
- Is clustering handled?
- Are residuals patternless enough for the intended model?
- Is variance approximately constant, or are robust methods used?
- Are influential observations driving the fit?
- Is multicollinearity making coefficients unstable?
- What does the intercept mean?
- What does each slope mean conditional on the other variables?
- What confidence intervals surround coefficients?
- Is R² being interpreted correctly?
- Has prediction been validated out of sample?
- Is the model extrapolating?
- How were missing values handled?
- Was feature selection performed?
- Were post-selection uncertainties addressed?
- Is the job description, prediction or causation?
- If causal, what identification assumptions justify the adjustment set?
73. Frequently Asked Questions
What is regression?
Regression is a family of statistical methods that models an outcome as a function of one or more predictors so relationships, predictions or model-based effects can be estimated and uncertainty quantified.
What does least squares do?
Ordinary least squares chooses coefficients that minimise the sum of squared differences between observed and fitted outcome values.
Does regression prove causation?
No. Regression can estimate causal effects only when the study design and causal assumptions identify the target effect and the model estimates that identified quantity appropriately.
What does R² mean?
In ordinary regression with an intercept, R² describes how much observed outcome variance the fitted model explains in-sample relative to predicting the sample mean. It is not a measure of causal contribution.
74. Authoritative Research Corridor
- NIST/SEMATECH — Linear Least Squares Regression
- NIST/SEMATECH — Are the Model Residuals Well-Behaved?
- NIST/SEMATECH — Check of Regression Assumptions
- NIST — Linear Regression Statistical Reference Datasets
- eduKateSG — How Causal Inference Works
Final Thought: A Regression Line Is a Conditional Sentence About the World
Regression is powerful because it can compress enormous datasets into a small number of parameters.
But every coefficient completes an invisible sentence.
For this population.
Across this observed range.
On this measurement scale.
Holding these modelled variables fixed.
Assuming this functional form and dependence structure.
For this descriptive, predictive or causal target.
Read the coefficient without that sentence and the model sounds more certain than it is.
Regression is not reality reduced to a line. It is a carefully conditional representation of how one part of reality changes with another—and its power depends on never forgetting the conditions.