eduKateSG Learning Node Series · 0156
Two groups can differ in average scores without any unfair item—and one item can behave unfairly even when the total-score gap looks ordinary.
That is why test fairness cannot be audited by comparing raw group percentages alone.
Suppose two learners have comparable standing on the mathematics construct the test is meant to measure. If one item is systematically easier for learners from one group after that comparison is made, the item deserves investigation.
This is the logic of differential item functioning, usually shortened to DIF.
Differential item functioning occurs when an item has different statistical behaviour for members of different groups who are otherwise matched on the relevant measured proficiency.
The 50-Second Read
- DIF is about conditional item behaviour, not raw group differences.
- The usual question is whether a focal and reference group have different probabilities of success on an item after matching on the construct being measured.
- A DIF flag is not automatic proof of bias. The difference may be construct-relevant, statistically unstable or caused by a poor matching variable.
- Uniform DIF means one group is favoured in a broadly consistent direction across proficiency. Non-uniform DIF means the size or direction of the difference changes with proficiency.
- The Mantel–Haenszel procedure is a classic method for dichotomously scored items and compares group performance within matched score strata.
- Logistic-regression and item-response-theory approaches can model additional forms of DIF, including interactions with ability.
- The matching criterion matters. If contaminated by many DIF items, it can hide or create item flags.
- Criterion refinement or purification can improve detection under some DIF patterns, but research shows it is not uniformly beneficial in every condition.
- Sample size affects power and stability: tiny samples can miss real DIF or produce imprecise estimates.
- The correct workflow is statistical flag → effect size → content review → response-process evidence → retain, revise or remove—not “flag means delete.”
Canonical Owner Boundary
This Learning Node owns group-conditional item behaviour after matching learners on the intended measured construct. How Instructional Sensitivity Works owns whether items respond to relevant teaching. How Assessment Works owns the wider evidence-and-decision system. How Adaptive Testing Works owns next-item selection from accumulating evidence. How Education Works | Assessment Item Banks & Test Form Assembly owns reusable item infrastructure and test-form construction. DIF asks the narrower fairness question: does this item function differently across groups even after we compare learners who are similarly positioned on the construct?
1. Raw Group Differences Are Not DIF
Suppose 80% of Group A answers an item correctly and 60% of Group B does.
That is a group difference.
It is not yet evidence that the item functions differentially.
If Group A also has higher proficiency on the overall construct, the item difference may simply reflect the proficiency difference the test is intended to measure.
DIF asks what remains after relevant matching.
2. The Matching Principle Is the Heart of DIF
Imagine grouping learners by their overall mathematics proficiency.
Within each proficiency band, compare the probability that focal-group and reference-group learners answer Item 12 correctly.
If the probabilities remain systematically different, Item 12 may exhibit DIF.
The phrase “similarly proficient” does enormous work here. If matching is poor, the fairness inference is poor.
3. Reference and Focal Groups Are Analytical Roles
DIF literature commonly calls one group the reference group and another the focal group.
These are analytic labels, not statements that one group is inherently normative or superior.
The practical concern is whether conditional item behaviour differs in a way that threatens the intended interpretation of scores.
4. DIF Is Not the Same as Bias
An item can show statistical DIF for a legitimate construct-relevant reason.
Suppose a language test intentionally measures knowledge of a particular register that one group has encountered more often. The item may show group differences after imperfect matching because the matching score does not fully represent the intended subconstruct.
Conversely, an item can be unfair for a reason a particular DIF analysis fails to detect.
DIF is a statistical signal for fairness investigation. Bias is a substantive judgement about construct-irrelevant advantage or disadvantage.
5. Uniform DIF
Uniform DIF occurs when the item favours one group in a broadly consistent direction across the matched proficiency range.
For example, at low, middle and high proficiency, Group A may remain more likely than comparable Group B learners to answer the item correctly.
The size of the difference can vary, but the direction is comparatively stable.
6. Non-Uniform DIF
Non-uniform DIF occurs when group differences interact with proficiency.
An item might favour the focal group among lower-proficiency learners but favour the reference group at higher proficiency. Or the difference may emerge only at one part of the scale.
This is one reason methods that allow group-by-proficiency interaction can reveal patterns a single common odds ratio may miss.
7. Mantel–Haenszel: The Classic Conditional Comparison
The Mantel–Haenszel procedure became a foundational tool for detecting DIF in dichotomously scored items.
The intuitive procedure is to stratify learners by a matching variable—often test score—and compare focal and reference group success within those strata. The stratum-specific information is combined into an overall conditional association.
Holland and Thayer’s ETS work helped establish the method’s use for item-bias and DIF analysis.
Read: Differential Item Functioning and the Mantel-Haenszel Procedure.
8. Odds Ratios Give the Comparison an Intuitive Shape
At a given proficiency stratum, we can compare the odds of a correct answer for one group with the odds for another.
If these conditional odds differ systematically across strata, the item may be functioning differently.
Operational programmes often transform the raw odds-ratio scale into reporting conventions and flagging categories, but the core idea remains conditional comparison.
9. Statistical Significance Is Not Enough
With very large samples, tiny group differences can become statistically significant.
With small samples, educationally important DIF may fail to reach conventional significance thresholds.
Strong DIF practice therefore combines statistical evidence with effect-size criteria and substantive review.
10. ETS Treats DIF as Part of Fairness and Validity Review
Rebecca Zwick’s 2012 review describes DIF analysis as a key component in evaluating educational-test fairness and validity. The report examined ETS flagging rules, sample-size requirements and criterion refinement rather than treating DIF as one fixed procedure.
This is an important framing: DIF is not a magical single test. It is a family of procedures embedded in a broader quality system.
Read: A Review of ETS Differential Item Functioning Assessment Procedures.
11. The Matching Criterion Can Be Contaminated
Suppose the total test score is used to match proficiency.
But many items in the test themselves contain DIF favouring one group.
Now the matching score is partly built from the same group-dependent behaviour we are trying to detect.
This can distort matching and therefore distort DIF estimates.
12. Criterion Refinement Tries to Clean the Matching Variable
Refinement—or purification—typically identifies suspected DIF items, removes or downweights them from the matching criterion, then repeats the analysis using a cleaner proficiency measure.
The intuition is sensible.
But sensible procedures still need empirical testing.
13. Purification Is Not Universally Better
Zwick, Ye and Isham studied criterion-refinement procedures across simulated conditions. They found that refinement could help substantially when DIF was unbalanced—many items favouring the same group—but could be slightly disadvantageous under balanced patterns where positive and negative DIF offset each other.
That is a useful warning against procedural superstition.
A technique can be generally advisable and still have conditions under which its advantage weakens.
14. Short Tests Make Matching Harder
A total score from five items is a much coarser proficiency measure than a total score from fifty strong items.
ETS simulation work comparing “thin” and “thick” matching found poor behaviour for thin matching on very short tests under studied conditions, including false identification of DIF against the reference group.
The general lesson is not “never test DIF on short tests.” It is that matching precision is part of DIF validity.
15. Logistic Regression Provides Another Lens
A logistic-regression approach can model the probability of a correct response using proficiency, group membership and a proficiency-by-group interaction.
A group term can represent uniform DIF; an interaction can represent non-uniform DIF.
The attraction is interpretability and flexibility. The danger remains the same: a sophisticated regression cannot rescue a bad proficiency match or a badly specified construct.
16. IRT Compares Item Response Functions
Item response theory expresses the relationship between latent proficiency and probability of item success through an item response function.
If the item response function differs across groups after placing learners on a common proficiency scale, the item may exhibit DIF.
IRT methods can distinguish differences in item difficulty, discrimination or other parameters depending on the model.
17. The Definition and the Detection Method Are Not Identical
DIF is a substantive statistical property. Mantel–Haenszel, logistic regression and IRT are methods for investigating that property under particular assumptions.
Zwick’s earlier ETS work examined conditions under which item-response-function and Mantel–Haenszel definitions coincide—and where the equivalence cannot simply be generalised.
Do not confuse the phenomenon with one detection algorithm.
18. Polytomous Items Need Their Own Treatment
Not every item is right or wrong.
Writing rubrics, partial-credit mathematics items, rating scales and performance tasks may use ordered categories.
DIF analysis for polytomously scored items asks whether category probabilities differ across comparable groups, using methods adapted to the richer response structure.
Dorans and Potenza’s ETS work developed an approach to equity assessment for polytomously scored items within the DIF tradition.
19. Sample Size Changes What You Can See
DIF detection is a signal-detection problem.
With too little data, estimates are unstable and true group-conditional differences may be missed. With very large samples, trivial differences can become statistically detectable.
Research on weighted Mantel–Haenszel statistics similarly shows estimation accuracy improving with larger sample sizes under studied simulation conditions.
Sample size should therefore affect both analysis design and confidence in the flag.
20. Multiple Testing Creates a False-Alarm Problem
If a test contains hundreds of items and many groups are compared, a large number of statistical tests can be run.
Even when no item has meaningful DIF, some will look unusual by chance.
Operational programmes therefore need flagging rules, effect sizes, replication across administrations and substantive review rather than a single p-value.
21. Mathematics Example: Context Changes the Item
A ratio item is intended to measure proportional reasoning but is wrapped in a culturally specific sports context.
After matching learners on the mathematics scale, one group still performs unexpectedly poorly.
The DIF flag does not prove the sports context caused the difference. It tells the item-review team where to investigate wording, familiarity, strategy use and construct representation.
22. English Example: Vocabulary Inside a Reading Item
A reading-comprehension question is supposed to measure inference.
But the stem contains an uncommon idiom known much better by one subgroup. Comparable readers differ on this item even though their general comprehension scores are similar.
Now content review asks whether the idiom is part of the intended construct or an irrelevant access barrier.
23. Science Example: Background Knowledge Leakage
A science item is designed to assess interpretation of experimental evidence but uses a scenario that strongly rewards specialised extracurricular knowledge.
If that knowledge is not part of the intended science construct, subgroup differences may indicate construct-irrelevant variance.
DIF becomes a doorway to content investigation.
24. Translation and Adaptation Are High-Risk Zones
An item translated into another language can preserve literal meaning while changing difficulty, ambiguity, syntax or cultural assumptions.
Cross-language and cross-country assessments therefore need both linguistic review and empirical item-functioning evidence.
Equivalent-looking wording is not sufficient evidence of equivalent measurement.
25. Test-Level Fairness Is More Than Item-Level DIF
One item with modest DIF may have little practical impact on a long test.
Many small DIF effects in the same direction can accumulate.
Opposing DIF effects can also cancel in the total score while still indicating item-level problems.
Fairness review therefore considers both individual item flags and aggregate score impact.
26. DIF in Adaptive Testing Has Extra Consequences
In a fixed test, every learner may see the same potentially problematic item.
In computerized adaptive testing, item selection depends on previous responses. A biased or miscalibrated item can alter the ability estimate and therefore change which items are administered next.
Item-pool fairness is therefore part of adaptive-test fairness.
27. DIF in Cognitive Diagnostic Models
Diagnostic classification creates another version of the problem.
If an item mapped to an attribute behaves differently across groups for learners with comparable attribute profiles, the model may misclassify a subgroup as lacking the skill.
This is why DIF methods have also been developed specifically under cognitive diagnostic models.
Read: Assessment of Differential Item Functioning Under Cognitive Diagnosis Models.
28. A Flag Should Trigger Human Item Review
Statistical evidence says the response pattern is unusual.
It does not tell you the cause.
Reviewers inspect wording, examples, visuals, required background knowledge, translation, curriculum exposure, accessibility, response format and possible alternative strategies.
Sometimes the item is revised or removed. Sometimes the DIF is judged construct-relevant and the item is retained with documentation. Sometimes no plausible substantive explanation is found and more data are collected.
29. Cross-Domain Comparison: Sensor Calibration
Imagine two identical sensors measuring the same physical quantity in two environments.
If one sensor systematically reads higher only in one environment, engineers investigate whether temperature, humidity or electromagnetic interference is affecting the instrument.
DIF asks an analogous question about items: does the measuring device behave differently across groups after controlling the target quantity?
30. Cross-Domain Comparison: Medical Assays
A diagnostic assay can perform differently across subpopulations because of biological, procedural or calibration differences.
The correct response is not to declare the assay “biased” from one group mean. Researchers compare performance conditional on relevant states and then investigate mechanisms.
The educational lesson is the same: fairness requires conditional measurement evidence plus domain interpretation.
31. Cross-Domain Comparison: A/B Segmentation
A product metric can look stable overall while one user segment experiences a very different effect.
Analysts condition on relevant characteristics to see whether the system behaves differently for comparable users.
DIF adds stricter psychometric discipline because the matching variable is itself a measurement problem, not a simple observed covariate.
32. A Practical DIF Review Protocol
- Define the construct: know what legitimate group differences the item is allowed to reflect.
- Choose focal and reference comparisons: based on a defensible fairness question, not data fishing.
- Build a credible matching variable: represent the target proficiency as cleanly as possible.
- Choose appropriate DIF methods: consider item type, sample size and possible non-uniformity.
- Inspect effect size as well as significance: separate detectable from consequential differences.
- Refine the criterion when justified: test whether suspected DIF contamination is distorting matching.
- Replicate: look across administrations or samples where possible.
- Review content: inspect wording, context, visuals, translation, accessibility and curriculum opportunity.
- Collect response-process evidence: find whether groups are solving or interpreting the item differently.
- Estimate score impact: ask whether item-level effects accumulate into meaningful test-level consequences.
- Decide: retain, revise, replace or remove with documented reasoning.
- Reanalyse: verify the change rather than assuming revision fixed the problem.
33. Failure Mode: Raw Percentages Are Called DIF
One group gets fewer items correct, so every difficult item is labelled biased.
Repair: compare conditional item performance after matching on the intended construct.
34. Failure Mode: Every Flag Means Delete the Item
The statistical procedure flags an item and it is immediately removed.
Repair: investigate substantive cause, effect size and construct relevance first. DIF detection is triage, not verdict.
35. Failure Mode: The Matching Score Contains the Problem
A test with widespread unidirectional DIF is used uncritically as the proficiency matching variable.
Repair: examine criterion refinement, external anchors or alternative matching strategies and test sensitivity to the choice.
36. Failure Mode: Tiny Subgroups Produce Absolute Conclusions
A small subgroup yields an unstable effect estimate, yet the result is presented as definitive evidence of fairness or unfairness.
Repair: report uncertainty, aggregate evidence across administrations when defensible, and avoid pretending weak power is proof of no DIF.
37. Failure Mode: One Protected Group Is the Entire Fairness Audit
Fairness can vary by language background, disability access, geography, schooling context, age, sex, socioeconomic conditions and intersections among characteristics depending on the assessment and jurisdiction.
Repair: define relevant groups from the validity and fairness argument while protecting privacy and avoiding indiscriminate subgroup fishing.
38. Missing-Node Scan
If one item repeatedly behaves strangely for a subgroup despite similar total proficiency, if a translation looks accurate but item difficulty shifts sharply, if item flags disappear when the matching score is purified, if adaptive-test exposure produces subgroup-specific score effects, if diagnostic classifications differ suspiciously by group, or if reviewers argue about bias using raw pass rates alone, the missing node may be conditional item-functioning analysis.
- Are we comparing comparable learners?
- What construct is the matching variable actually measuring?
- Is the DIF uniform or non-uniform?
- Is the effect educationally meaningful or merely statistically detectable?
- Could the matching criterion itself contain DIF?
- Is sample size sufficient for stable estimation?
- Does the flag replicate?
- What item feature could plausibly create the difference?
- Is that feature construct-relevant or construct-irrelevant?
- What is the aggregate score impact?
- Would revision remove the unwanted difference without weakening construct coverage?
39. Evidence and Limits
DIF methodology is well established, but every procedure rests on assumptions about matching, dimensionality, item form and sampling. Mantel–Haenszel is efficient and interpretable for many dichotomous-item settings, yet IRT and regression methods may reveal forms of non-uniformity or parameter difference it does not express in the same way.
Zwick’s ETS review shows why operational rules require continuous evaluation: flagging thresholds, sample sizes and criterion refinement materially change what the system detects. Simulation work further shows that refinement can help or hurt depending on the pattern of DIF. Large samples can detect small effects, while small samples create unstable estimates.
The strongest fairness argument therefore never ends with “the DIF statistic was nonsignificant.” It asks whether the matching construct is defensible, whether methods had enough power, whether content review found construct-irrelevant features, whether results replicate, and whether the remaining item set supports fair score interpretation.
40. The Return Path
Return to the two learners with similar mathematics proficiency.
They reach the same item.
If group membership should be irrelevant to the construct but systematically changes the probability of success, the item is no longer behaving as a neutral measuring device.
That does not tell us why.
It tells us where to look.
And that is the power of DIF analysis: it converts a vague fairness concern into an item-level investigation that statistics, domain expertise and evidence can examine together.
Differential item functioning works when we stop asking only whether groups score differently and start asking whether the measuring instrument itself behaves differently for learners who should be comparable on what the test claims to measure.
Research and Further Reading
- Holland & Thayer — Differential Item Functioning and the Mantel-Haenszel Procedure
- Zwick — A Review of ETS Differential Item Functioning Assessment Procedures
- Zwick, Ye & Isham — Criterion Refinement in Mantel-Haenszel DIF Analysis
- Zwick, Ye & Isham — Improving Mantel-Haenszel DIF Estimation Through Bayesian Updating
- Lu, Guo & Dorans — Robustness of Weighted Differential Item Functioning Analysis
- Li & Wang — Assessment of Differential Item Functioning Under Cognitive Diagnosis Models
- How Instructional Sensitivity Works
eduKateSG Learning Node Series · 0156 · Previous: 0155 — How Rater Drift Works.