eduKateSG Learning Node Series · 0212
Two learners receive the same entry score. Does that score justify the same forecast of what happens next?
A training programme uses an entry assessment to predict performance at the end of a course. Its overall predictions look respectable. Higher entry scores tend to accompany higher final scores. Yet a closer review finds that predictions are consistently too optimistic in one delivery setting and too pessimistic in another.
The programme now faces several possible explanations. The entry test may function differently. The final assessment may be scored differently. Teaching conditions may differ. Important predictors may be missing. The analysed sample may include only selected completers. Or a common prediction equation may simply be inadequate for the settings in which it is being used.
Differential prediction asks whether the relationship between a predictor and an external criterion differs across groups or contexts. It is a question about a forecast, not an automatic verdict about the predictor’s measurement fairness. A statistical difference deserves investigation, but the difference does not announce its own cause or prescribe a policy response.
The 50-second route
Start with the outcome being predicted, then examine whether a common prediction rule systematically overpredicts or underpredicts that outcome for relevant groups. In a linear model, group differences can appear in intercepts, slopes or both. Check common score support, uncertainty, criterion quality, selection and local transportability. Do not equate a non-significant test with demonstrated equality, or a significant difference with proof of an inherent group characteristic.
Differential item functioning concerns response differences on individual items after matching on the measured construct. Differential test functioning concerns their aggregate score effect. This guide owns a different relationship: how a score predicts an external criterion, such as later course performance. The examples use invented groups and numbers; they are not claims about actual demographic groups, schools or admissions systems.
1. Name the criterion before discussing prediction quality
A predictor is not accurate in the abstract. It predicts something defined at a particular time under particular conditions. An entry score might predict final examination marks, course completion, independent task performance or the need for additional support. Those are different criteria with different measurement properties.
Suppose one campus uses a generously marked final project and another uses a tightly timed written test. Calling both outcomes “course performance” does not make their numerical scores interchangeable. An apparent prediction difference may partly reflect different criteria rather than different relationships between entry capability and later learning.
Define the criterion’s task content, scoring rule, timing, available supports and intended interpretation. Check whether missing outcomes represent noncompletion, absence, transfer or a recording failure. A model can accurately predict an administrative label while poorly representing the capability the programme actually cares about.
The distinction between predictive and measurement questions is central to Meade and Fetzer’s research on test bias and differential prediction. They identify several possible sources of regression-line differences beyond the properties of the predictor itself. Their argument is a reason to investigate the whole prediction system, not a reason to ignore harmful outcomes or dismiss fairness concerns.
2. A common regression line makes a common forecast
Let X be an entry score and Y a later performance score. A simple common model predicts Y using an intercept and a slope: the intercept fixes the expected outcome at a reference entry score, and the slope describes how expected Y changes as X increases.
The prediction is an expected outcome, not a guarantee for an individual. Two learners with identical X can have different Y because of unmeasured preparation, instruction, opportunity, motivation, random variation or measurement error. A residual is the observed outcome minus its prediction. Residual patterns can reveal where the common rule systematically misses.
If one group’s residuals average negative over a relevant score region, the model overpredicts that group’s observed outcomes there. Positive average residuals indicate underprediction. A group mean residual of zero is not enough to show good calibration everywhere: positive errors at one end can cancel negative errors at the other.
Plot predicted and observed relationships over the score ranges actually represented. The graph should show data support and uncertainty, not only two infinitely extended lines. For the broader statistical machinery, see How Regression Works.
3. A worked model with intercept and slope differences
Consider two invented training settings, coded G = 0 and G = 1. Centre the entry score at 60 so the group coefficient refers to a meaningful point in the observed range. Suppose the illustrative expected-outcome model is:
Expected Y = 70 + 0.50(X − 60) − 5G + 0.25G(X − 60) Setting 0: Expected Y = 70 + 0.50(X − 60) Setting 1: Expected Y = 65 + 0.75(X − 60)
At X = 60, the expected outcomes are 70 and 65. The group difference at the reference score is minus five. The slopes also differ: a one-point increase in X corresponds to a 0.50-point expected change in setting 0 and a 0.75-point expected change in setting 1.
| Entry score X | Expected Y, setting 0 | Expected Y, setting 1 | Setting 1 minus setting 0 |
|---|---|---|---|
| 40 | 60 | 50 | −10 |
| 60 | 70 | 65 | −5 |
| 80 | 80 | 80 | 0 |
These are arithmetic implications of a specified toy model, not fitted research findings. They show why “the model is five points wrong for one group” can be incomplete when slopes differ. The gap depends on X. A constant adjustment would not describe this model across the whole range.
The setting indicator is descriptive. It does not establish that changing a learner’s setting would change Y by the displayed amount. That causal claim would require additional assumptions or a suitable assignment design. The group term may be standing in for several differences the model has not measured.
4. Centring changes the question answered by the intercept
In the worked model, the group difference is −5 + 0.25(X − 60). If X were left uncentred, the group coefficient would refer to X = 0. The equivalent expression is −20 + 0.25X. A coefficient of minus twenty would sound dramatic even if nobody in the study had an entry score remotely close to zero.
The two equations are mathematically identical. They describe the same fitted lines. What changed is the reference point. This is why an intercept difference should never be interpreted without asking where on the score scale it applies.
Select reference points tied to real decisions or substantial common data support. Report group differences at more than one relevant score if the lines are not parallel. When uncertainty bands widen sharply near the edges, show that widening rather than extending a confident claim into an unobserved region.
Centring is an interpretive aid, not a remedy for bias or confounding. It can make the coefficient answer a more useful question, but it does not change the information in the data or the quality of the underlying design.
5. Equal correlations do not imply equal forecasts
Imagine two hypothetical groups with identical predictor values and identical residual variation. In one group, every criterion score is five points lower. Subtracting a constant from Y does not change its correlation with X. The two groups can therefore have exactly the same predictor-criterion correlation while their expected outcomes differ at every X.
Correlation describes standardised association. Calibration concerns whether predictions match observed outcomes on the relevant scale. A model can rank learners similarly in both groups while systematically forecasting the wrong level for one group. High correlation is useful evidence about ordering, but it does not settle the calibration question.
Even slope and correlation are not interchangeable. In a simple linear regression with an intercept, the slope equals the correlation multiplied by the ratio of the outcome standard deviation to the predictor standard deviation. Different group variances can produce different slopes with similar correlations.
A responsible report therefore separates association strength, calibration level, slope and prediction error. Combining them into a single claim that the test “works equally well” can conceal the precise difference the audit was meant to investigate.
6. Statistical tests should not replace the size and location of the difference
A conventional linear audit can compare a common-line model, a model allowing a group shift, and a model allowing both a group shift and a group-by-score interaction. A joint test of the group and interaction terms examines whether the chosen common-line restriction is compatible with the data under the modelling assumptions.
Failure to reject that restriction is not proof of identical prediction. A small group, narrow score range or noisy criterion may provide little power to detect a consequential difference. Conversely, a huge sample can detect a difference too small to matter for the intended decision.
Report estimated differences and uncertainty at relevant scores. Where a claim of practically negligible difference is needed, define the tolerance in advance and use an appropriate interval or equivalence framework. A tolerance should be connected to consequences, not chosen after seeing which value makes the model pass.
There is also a multiple-comparison problem when many groups, outcomes, sites and score regions are searched. Predefine primary comparisons, treat exploratory findings as exploratory and seek replication. A dramatic local result discovered among hundreds of unplanned analyses needs a different evidential status from a preregistered, independently confirmed finding.
7. Average evidence and local evidence can tell different stories
Mattern and Patterson’s 2013 study of slope and intercept differences in college admissions analysed a large historical dataset and reported relatively small increments in explained variance for differential-prediction terms under the comparisons studied. It also examined regression plots. Those findings belong to that dataset, predictor combination and analysis; they are not a timeless verdict on every admissions programme.
Aguinis, Culpepper and Pierce’s 2016 study of differential prediction generalisation focused on variation across institutions and samples. Their analysis found remaining heterogeneity after accounting for several statistical and methodological factors. The author-affiliated university record provides the study’s scope and summary.
The useful synthesis is not that one headline defeats the other. A modest average difference can coexist with important local differences. A local difference can also be unstable when based on sparse data. Programmes need to know whether the intended use is supported in their setting, with their criterion and their current learner population.
Pooling can increase precision while hiding variation. Analysing every small site separately can reveal variation while exaggerating noise. Hierarchical models offer one way to share information without forcing every site to have exactly the same relationship, but their assumptions and sensitivity still require review.
8. Common support limits what can honestly be compared
Suppose setting 0 has entry scores mostly between 40 and 65, while setting 1 has scores mostly between 70 and 90. A model can draw two lines across 0 to 100, but there may be little direct evidence comparing settings at the same X. The apparent differential prediction relies heavily on functional-form extrapolation.
Inspect overlapping score ranges, not just total sample sizes. A dataset can be large while having almost no overlap where the policy decision is made. Adding more observations in already-dense regions does not solve missing support at the boundary of interest.
Possible responses include narrowing the comparison to the supported range, collecting additional data, using a more limited claim or examining the result’s sensitivity to alternative functional forms. None requires pretending the unobserved comparison is known.
This is particularly important when a model is used for selection. The most consequential threshold may lie in a region poorly represented in the historical data. A visually smooth fitted line should not lend certainty to a decision that the available observations barely constrain.
9. The outcome may contain measurement bias too
Audits often scrutinise the predictor while treating the criterion as unquestionable truth. Yet final grades, supervisor ratings and course-completion labels are produced by human and institutional systems. Rubrics, task assignments, opportunities to demonstrate performance and rater behaviour can differ.
Imagine equal underlying capability but systematically harsher scoring in one setting. A model may accurately learn lower observed Y there. That can improve prediction of the recorded outcome without making the outcome measure fair or the capability interpretation correct.
Use several relevant outcome measures when feasible and inspect their comparability. A common performance task, moderated scoring or independent evidence may help distinguish predictor issues from criterion issues. Agreement among several measures is still not infallible, especially if they share the same bias or opportunity structure.
The distinction matters ethically as well as statistically. Predicting an unequal institutional outcome is not the same as justifying it. A forecast of what currently happens should not silently become a statement about what a learner is capable of achieving under fairer or more supportive conditions.
10. Measurement error can change regression lines without announcing test bias
Consider a separate theoretical example. Let an observed predictor be X = T + error, where T is a latent true score. Let the outcome depend on T through the same relationship in both groups. Even if the measurement equation and outcome relationship are invariant, conditioning on the same noisy observed X can produce different expected T values when the groups have different latent distributions.
Under a simple Gaussian model with independent measurement error, the conditional expectation can be written as a weighted combination of the observed score and the group’s latent mean. If the weight on X is λ, then:
Expected T given X and group g = μg + λ(X − μg) = λX + (1 − λ)μg
If the outcome relationship is Y = a + bT plus mean-zero error, then its conditional expectation inherits the term b(1 − λ)μg. Different group means can therefore contribute to different observed-score regression intercepts even under this simple invariant measurement setup. This is a mathematical example under explicit assumptions, not a claim about any real population.
The example shows why observed-score prediction and measurement equivalence are different questions. It does not establish that an actual test is unbiased, and it does not authorise group-based score adjustment. Real systems require evidence about their measurement structure, criteria, populations and intended decisions.
11. A 2025 research frontier: distinguish observed-score prediction from true-score relationships
Michael T. Kane and Joanne Kane’s Using Group Differences in True Score Relationships to Evaluate Measurement Bias develops a theoretical model using errors-in-variables regression rather than ordinary least squares for a measurement-bias question. The paper distinguishes measurement bias from predictive bias and examines how regression toward the mean can complicate interpretations. Its reported contribution is theoretical; it does not present a new empirical learner dataset.
The important practical question is the estimand: what relationship are we trying to know? Predicting an observed future outcome from an observed score is not identical to estimating a relationship between latent true scores. Correcting a slope for measurement error may be appropriate for one purpose and inappropriate as a direct prediction rule for another.
Do not treat errors-in-variables modelling as a universal upgrade that automatically makes forecasts fair. It requires information or assumptions about measurement error, and those assumptions can be wrong. The methodological advance is greater clarity about the question, not removal of the need for validation.
This is also why the direction of overprediction or underprediction should not be used as a complete moral verdict on the test. A prediction pattern is one piece of evidence in a larger validity argument. Measurement bias, predictive calibration and the consequences of decisions must be examined separately and then related carefully.
12. Selection changes the data available for validation
Entry assessments are often used to decide who enters a course. Later outcomes are then observed only for those admitted or enrolled. The validation sample is selected by the predictor and possibly by other variables. It may have a restricted score range and a different mix of characteristics from the full applicant population.
A relationship estimated among selected completers cannot automatically be transported to everyone who applied. Selection can affect associations, and dropout can add another layer. Correcting range restriction requires assumptions about the selection process and the distribution of relevant variables; a correction formula is not evidence that those assumptions hold.
Preserve the stages in the data record: applicants, eligible candidates, offers, enrolments, assessment participants and observed outcomes. State exactly which stage the model describes. An analysis of completers should not be advertised as a forecast for all entrants without additional evidence.
The missing counterfactual is especially important for rejected applicants. Their outcome in a course they never attended is not observed. A statistical model may estimate it under assumptions, but the prediction should not be described as a measured fact about their potential.
13. Adding predictors changes the question
Suppose a score-only model shows differential prediction. Adding prior coursework reduces the group difference. That may improve a forecast, but it does not by itself explain why the original difference existed. Coursework may represent preparation, opportunity, selection, or several correlated influences.
For prediction, an additional variable can be useful if it is available at the decision time, measured consistently, permitted for the use and validated out of sample. For causal interpretation, conditioning on the wrong variable can block part of an effect or introduce bias. The correct adjustment set depends on the causal question, not on which variables make a group coefficient disappear.
Also consider whether the variable is a consequence of the institution’s earlier decisions. A model that uses past placement or teacher recommendations may inherit those systems’ limitations. Better numerical prediction can coexist with a more opaque decision process if the added information is poorly understood.
Document the incremental job of every predictor. Ask what it adds, what uncertainty it carries, and how the interpretation changes when it is included. A fairness audit should not become an unrestricted search for controls that produce a preferred conclusion.
14. Prediction uncertainty belongs to the individual forecast
An estimated group mean difference can be relatively precise while individual outcomes remain highly variable. A narrow confidence interval around a regression line is not a narrow prediction interval for one learner. The latter must also account for outcome variability around the conditional mean.
In the toy model, an expected outcome of 65 does not mean the learner will receive 65. Nor does a five-point difference between group means imply that every matched pair will differ by five. Individual variation and measurement uncertainty remain substantial possibilities unless the data establish otherwise.
When predictions trigger support, consider whether uncertainty should lead to additional evidence rather than immediate classification. A short performance task or review of recent work may be more informative than squeezing another decimal place from a broad historical model.
The communication should remain revisable: “This evidence suggests a range of likely outcomes under these conditions” is different from “You are a 65-point student.” Forecasting should support learning decisions, not convert an uncertain conditional expectation into a fixed identity.
15. A common mean model can still produce unequal error patterns
Suppose two groups share the same conditional mean relationship but one has much more outcome variability. The same point forecast can be unbiased on average for both while being much less precise for one. If the programme turns forecasts into pass-risk categories, the resulting error patterns may differ.
Conversely, similar overall accuracy can hide different false-positive and false-negative rates or different performance near a threshold. The appropriate fairness and performance metrics depend on the decision, outcome prevalence and consequences. No single number establishes that every relevant requirement has been satisfied.
Differential prediction of a continuous criterion is therefore one layer of an audit, not the whole audit. Examine distributional assumptions, uncertainty, calibration and the action policy. A model can pass a slope-and-intercept comparison while the eventual decision process still needs substantial improvement.
This does not mean every model must optimise every possible fairness metric simultaneously. Some criteria can be in tension. It means the institution should state which obligations matter for the use, explain trade-offs transparently and avoid presenting success on one statistical check as universal fairness certification.
16. Support can change the outcome the model predicts
A forecast identifies learners at risk of poor performance. The programme gives those learners effective support. Their outcomes improve, and the original forecast now looks too pessimistic. That is not automatically evidence that the model was useless. The prediction helped trigger an action that changed the outcome-generating conditions.
Record interventions alongside forecasts and outcomes. A model trained on unsupported historical cohorts may not remain calibrated after a support policy changes. Conversely, a model trained in a well-supported system may overpredict outcomes when moved to a setting without that support.
Most importantly, risk is not the same as treatment benefit. A learner predicted to do poorly may or may not benefit most from a particular intervention. A learner already predicted to succeed may still benefit substantially. Allocating support requires evidence about the intervention and its effects, not only a ranking of predicted outcomes.
The ethical educational aim is not to make reality conform to the forecast. It is to improve outcomes while learning when the forecast remains useful. A well-designed system expects successful teaching to change what happens next.
17. Auditing groups is not permission to automate group-based treatment
Group information can be important for detecting hidden failures. Removing it from an audit may make unequal calibration invisible. But using it for analysis is a different decision from using it to assign individual scores, admissions outcomes or learning opportunities.
A differential-prediction finding should trigger a structured investigation of the predictor, criterion, context, selection and decision process. It should not automatically trigger separate score conversions or different standards. Operational changes require appropriate institutional, ethical and legal review, plus evidence that the proposed remedy improves the relevant outcomes without creating new harms.
Protect the data used for auditing. Collect only what is justified, restrict access, document permitted purposes and avoid publishing small identifiable cells. Group labels should not be inferred casually from names, appearances or proxies. Uncertainty about group classification can itself affect the analysis.
For individual teaching, return to the learner’s work. A population-level pattern can suggest what the system should investigate, but it should not replace direct evidence about this person’s current knowledge, circumstances and response to support.
18. Cross-domain comparison: a travel-time forecast moved to another city
A travel-time model may predict journeys accurately in one city and overpredict them in another, even for the same distance. Road design, signals, congestion and transport modes differ. The difference does not mean the distance measure is necessarily biased. It may mean that the relationship between distance and travel time is context-dependent.
That is the useful analogy for differential prediction. A score can be a meaningful predictor while its relationship to later performance depends on curriculum, instruction, resources or criterion design. The forecast must be validated where it will be used.
The analogy also has a limit: people can change their behaviour in response to a forecast, and institutions can alter opportunities based on predictions. A model used to route learners is part of the system it predicts. That feedback makes careful monitoring and revisable decisions essential.
19. Cross-domain comparison: forecast accuracy versus intervention benefit
Imagine a maintenance model predicting which machines are most likely to fail. A second question asks which maintenance action would prevent the most failures. These are not identical. Some high-risk failures may be difficult to prevent; some moderate-risk units may respond extremely well to a small repair.
The same logical distinction applies to educational support without reducing learners to machines. A model predicting low final performance is not automatically a model of who will benefit most from a specific lesson, tutor or programme. The support decision needs evidence about the mechanism that can change the outcome.
This comparison provides a useful test for dashboards: does the system merely predict a result, or does it explain why the proposed action is appropriate? A red risk label is not yet a teaching plan. The next step should identify a plausible, addressable barrier and collect evidence that the intervention changes it.
20. A practical review exercise
Equal correlations, different mean residuals: has equal prediction been established? No. Correlation can remain unchanged after a constant outcome shift. Examine calibration levels and slopes on the outcome scale relevant to the decision.
A significant group interaction: does this prove the test measures groups differently? No. It shows a differential relationship with the selected criterion under the fitted model. Investigate measurement, context, selection and specification. DIF and measurement-invariance analyses answer related but distinct questions.
A non-significant group term in a tiny sample: is the common model certified fair? No. The uncertainty may include consequential differences. A claim of negligible difference needs an appropriate tolerance and evidence precise enough to support it.
A model trained on admitted completers: can it predict every applicant’s outcome without qualification? No. The observed sample is selected, and outcomes for nonparticipants are unobserved. Transport requires additional assumptions and evidence.
Predictions become pessimistic after successful support: should the support be removed to restore calibration? No. The objective is improved learning, not preservation of an old forecast. Update the evaluation to account for the changed policy and test the model’s intended use under the new conditions.
21. Build the audit around an actual decision
A useful audit begins with the predictor, criterion, population, time horizon and action. Specify the groups or contexts being examined and why those comparisons are justified. Preserve the analysis sample’s selection history and inspect common score support before fitting elaborate interactions.
Then compare plausible prediction models, display calibration and uncertainty, inspect nonlinear patterns and residual spread, and evaluate the consequences near relevant thresholds. Check criterion comparability and missing outcomes. Use later cohorts or separate sites to examine transportability where the intended claim requires it.
Keep statistical findings separate from explanations and proposed remedies. A finding might be robust while its cause remains unresolved. A plausible cause might suggest a useful experiment while not yet justifying a policy change. An operational remedy should be evaluated for its own effects, not approved merely because it makes a coefficient smaller.
Finally, make the audit revisitable. Record model version, scoring versions, predictor availability, support policies and criterion definitions. Changes in any of these can change prediction. A score-to-outcome relationship is supported in a context; it is not a permanent property engraved inside the test.
The return: a forecast should open a question, not close a future
The two learners with the same entry score do not require a declaration that their futures are identical or fundamentally different. They require an honest account of what the score predicts, how uncertain that prediction is, what conditions it assumes and what additional evidence could improve the next decision.
Differential prediction makes one part of that account visible. It asks whether a common forecast behaves comparably where it is used. The more advanced step is to avoid confusing that answer with a complete theory of fairness, a causal explanation of group differences or a judgement of an individual’s potential.
A responsible educational system does not use statistical precision to make uncertainty harder to challenge. It uses evidence to improve its decisions, creates opportunities for learners to demonstrate change, and remains willing to revise the model when learning makes yesterday’s prediction obsolete.
Research and onward reading
Conceptual distinction: Meade and Fetzer, Test Bias, Differential Prediction, and a Revised Approach for Determining the Suitability of a Predictor in a Selection Context (2009). Historical admissions evidence: Mattern and Patterson, Test of Slope and Intercept Bias in College Admissions (2013). Contextual generalisation: Aguinis, Culpepper and Pierce, Differential Prediction Generalization in College Admissions Testing (2016). Measurement-error frontier: Michael T. Kane and Joanne Kane, Using Group Differences in True Score Relationships to Evaluate Measurement Bias (2025). The equations, tables and institutional scenarios in this guide are original illustrations, not empirical group comparisons.
eduKateSG Learning Node Series · 0212 · Related: Differential Item Functioning, Measurement Invariance and the How X Works Hub.
