How Differential Test Functioning Works | Find Out Whether Item-Level DIF Changes the Test as a Whole

eduKateSG Learning Node Series · 0180

A test can contain several items with differential item functioning and still show almost no whole-test difference. It can also contain many small item effects that quietly add up.

This is the step after DIF analysis. Differential item functioning asks whether comparable learners from different groups have different probabilities of success on one item. Differential test functioning asks what happens when those item-level differences are aggregated across the test.

The distinction matters because assessment decisions are usually made from total or scale scores, not from one isolated item. Fairness review therefore needs to know not only which items differ, but whether those differences cancel, reinforce or reshape the expected score of the whole instrument.

Differential test functioning works by comparing the expected whole-test scores of matched groups across the latent trait, revealing whether item-level differential functioning accumulates into a practically meaningful score difference.

The 50-Second Read

• DIF is an item-level property; DTF is the aggregate test-level consequence.
• Groups are compared after matching on the relevant latent trait or proficiency.
• DTF can be examined through differences between group-specific expected test scores or test characteristic curves.
• DIF effects in opposite directions can cancel at the total-score level.
• Small DIF effects in the same direction can accumulate into meaningful DTF.
• Signed summaries allow cancellation; unsigned summaries measure total absolute differential functioning.
• A test can show little net DTF while still containing important item-level fairness concerns.
• A test can also have substantial DTF even if no single item appears dramatic.
• Statistical significance is not the same as practical score impact.
• DTF estimates inherit the assumptions of the matching variable and measurement model.
• DTF does not prove why group differences occur or whether an item is biased.
• Fairness review needs statistical evidence, content review, accessibility evidence and consequence analysis together.

Canonical Owner Boundary

This node owns the whole-test aggregate impact of differential item functioning after groups are matched on the measured construct. How Differential Item Functioning Works owns the item-level diagnosis. How Test Characteristic Curves Work owns the general mapping from latent proficiency to expected total score. How Measurement Invariance Works owns broader cross-group and cross-time comparability of latent measurement. This article asks the next fairness question: after all the item differences are combined, do matched groups receive different expected total scores?

1. Start With DIF

Suppose two groups of learners have the same level of the proficiency a test intends to measure. On Item 12, Group A has a higher probability of success. On Item 23, Group B has the advantage. On Item 31, the groups behave nearly identically.

DIF analysis studies those item-specific differences. But score users usually care about the result of all items together. DTF moves the analysis from components to the whole.

2. Compare the Comparable

Raw group score differences are not automatically DIF or DTF. If two groups genuinely differ in the measured proficiency, different average scores can reflect that construct difference.

Differential functioning asks whether group membership predicts different item or test behaviour after matching on the relevant latent trait. ETS DIF methodology has long stressed this principle: compare comparable examinees before interpreting differential performance as an item-functioning issue.

3. Group-Specific Test Characteristic Curves Make DTF Visible

For each group, the calibrated item response functions can be summed into a test characteristic curve. If the curves lie on top of each other, matched learners have the same expected total score across θ under the model.

If the curves separate, the test exhibits differential functioning in that region: group membership changes expected total score even after conditioning on proficiency.

4. One Item Can Show DIF Without Creating Much DTF

Imagine a 60-item test where one item produces a small group difference and the other 59 function similarly. The item deserves review, especially if the difference reflects a fairness concern, but its effect on total expected score may be tiny.

DTF quantifies that aggregate consequence rather than assuming any DIF flag must meaningfully change the total score.

5. Several Small DIF Effects Can Accumulate

Now imagine ten items each produce a small advantage for the same group. No single item looks dramatic, but the expected-score differences all point in one direction.

At the test level, those small effects can accumulate into a score difference that matters for ranking, classification or reporting.

6. Cancellation Is Central

If some items favour Group A and others favour Group B, signed effects can cancel. Net DTF can approach zero even though the test contains several DIF items.

This is why both signed and unsigned perspectives are useful. Signed summaries answer, “What is the net expected-score difference?” Unsigned summaries ask, “How much differential functioning exists in total, regardless of direction?”

7. Zero Net DTF Does Not Mean Zero Fairness Concern

Suppose two biased mechanisms happen to cancel numerically. The total score may show little net difference, but the items can still be unfair, construct-irrelevant or inappropriate.

Cancellation protects the total-score average, not the validity of every item. DTF therefore complements rather than replaces item-level DIF and content review.

8. The Direction of DIF Can Change Across Proficiency

Nonuniform DIF means a group difference changes with θ. An item might favour one group at low proficiency and the other at high proficiency. When such items are aggregated, DTF can also vary across the trait range.

A single average test-level difference can hide this crossing pattern. Plotting group-specific TCCs shows where the score difference actually lives.

9. DTF Can Matter Most Near a Decision Threshold

A small expected-score difference can be operationally important if it occurs near a pass/fail or placement boundary. The same difference in the middle of a broad descriptive scale may have little consequence.

Fairness analysis therefore needs to connect DTF to classification accuracy and decision thresholds rather than treating score impact as context-free.

10. Statistical Significance Is Not Enough

With large samples, very small differential effects can become statistically detectable. A fairness review should therefore quantify magnitude in score units, standardised units, probability of changed classification or another decision-relevant metric.

The question is not merely, “Can we detect DTF?” but, “How much does it change the score interpretation or consequence?”

11. DTF Methods Are Still Developing

Differential test functioning is less familiar to many practitioners than DIF, and methodological work continues. A 2026 article by Sakamoto and Kumagai, A Simple Approach for Differential Test Functioning Based on Sum Scores, proposes a sum-score-based approach and illustrates that DTF remains an active measurement problem rather than a settled single-statistic routine.

Earlier work by Chalmers, Counsell and Flora developed improved DTF statistics that account for sampling variability in item-parameter estimates.

12. Parameter Uncertainty Matters

If group-specific item parameters are estimated with uncertainty, the resulting TCC difference also has uncertainty. Treating item parameters as perfectly known can make a DTF estimate look more stable than it is.

The 2016 paper It Might Not Make a Big DIF addresses this issue by incorporating sampling variability into differential test functioning statistics.

13. The Matching Variable Can Contaminate the Result

DIF and DTF analysis depend on a matching variable representing the target construct. If that matching variable itself contains many biased or differentially functioning items, “comparable” learners may not actually be comparable.

Criterion purification or iterative matching procedures can reduce this contamination, but they add another modelling layer. Fairness statistics inherit the quality of the scale used to match people.

14. A Pooled Calibration Can Hide Group-Specific Curves

If one set of item parameters is estimated across all groups, the pooled model may average over systematic differences. DTF analysis can require group-specific or constrained models that make those differences visible.

This is not an argument for separate scoring by default. It is an argument for testing whether the pooled measurement relationship is actually invariant enough to justify common interpretation.

15. DTF Is Related to Measurement Invariance

Measurement invariance asks whether the measurement model supports comparable meaning across groups. DIF is one manifestation of item-level noninvariance. DTF asks how those item differences affect the aggregate score.

The three levels can therefore be read as a hierarchy: model comparability, item differences, whole-test consequence.

16. DTF Is Not Adverse Impact

Two groups can have different average observed scores because they differ in the measured proficiency distribution. That is impact. DTF refers to differential expected test behaviour among learners matched on the construct.

Confusing group outcome differences with differential measurement functioning can produce both false accusations of bias and false reassurance about genuine measurement problems.

17. DTF Is Not Automatically Test Bias

A statistical difference identifies a measurement pattern. Determining bias requires substantive interpretation: what content creates the difference, whether it is construct-relevant, whether groups have equitable opportunity to demonstrate the intended capability, and what consequences follow.

ETS fairness principles treat DIF as an empirical check that should trigger expert review rather than as automatic proof that an item is unfair. The same caution applies at the test level.

18. One Test Can Show DTF in One Region but Not Another

If group-specific TCCs separate only at high θ, most examinees may experience little differential total-score effect while advanced candidates do. If the curves diverge near the pass standard, the consequences can be concentrated among borderline candidates.

Always inspect DTF conditionally across the trait range instead of relying only on one population average.

19. Subtests Can Behave Differently From the Full Test

A total mathematics test may have little net DTF because algebra favours one group slightly while geometry favours another. At subtest level, however, the differential functioning may be meaningful.

Whether cancellation is acceptable depends on the score interpretation. If subscale scores are reported or used for decisions, subtest DTF deserves its own analysis.

20. Testlets Create Another Aggregation Layer

Several items sharing one passage or stimulus can function differentially as a unit. ETS research by Wainer, Sireci and Thissen defined and studied differential testlet functioning, recognising that the natural fairness unit can sometimes be larger than one item but smaller than the whole test.

This matters because local dependence can make a shared stimulus carry more aggregate group impact than item-by-item analysis suggests.

21. The Composition of the Test Controls Cancellation

Add or remove a few DIF items and net DTF can change. Assemble a new form from the same item bank and the balance of differential effects can shift.

Fairness therefore belongs in test assembly as well as post-hoc analysis. If form construction ignores known DIF directions, one form can accumulate more DTF than another.

22. Adaptive Testing Makes DTF Path-Dependent

In a fixed form, everyone receives the same items. In an adaptive test, different response histories produce different item routes. If DIF items are unevenly available across proficiency regions or exposure controls, matched learners from different groups may encounter different patterns of differential evidence.

Fairness monitoring for adaptive systems therefore needs simulation across realistic routing paths, not only static inspection of the bank.

23. DTF Can Be Small Even When DIF Counts Are Large

A common reporting mistake is to count flagged items and infer test unfairness from the count alone. Twenty small DIF items split evenly in direction can produce less net score difference than three moderate items all pointing the same way.

Counts describe prevalence. DTF describes aggregate consequence.

24. DTF Can Be Important Even When No Item Looks Extreme

The opposite mistake also occurs. Every item effect looks too small to worry about individually, so no one calculates the total. If the effects align, the whole-test difference can still matter.

System effects often live in accumulation.

25. Cross-Domain Comparison: Tiny Biases in a Navigation System

Imagine a navigation system where one sensor is off slightly east, another slightly west and a third correct. Errors can cancel. Replace the west-biased sensor with another east-biased one and the final route drifts.

DIF items behave similarly at the test level. Direction matters as much as count.

26. Cross-Domain Comparison: Portfolio Risk

A financial portfolio cannot be understood by listing the volatility of each asset separately. Correlations and directions determine the aggregate risk. Individual components can offset or reinforce one another.

DTF is the portfolio view of DIF. It asks what the collection does after the item-level effects interact in the total score.

27. Failure Mode: Count DIF Items and Stop

A fairness report says 8% of items show DIF and treats that percentage as the test-level fairness result.

Repair: estimate aggregate expected-score impact and examine whether effects cancel, accumulate or concentrate near consequential score regions.

28. Failure Mode: Celebrate Cancellation

Net DTF is near zero, so all flagged items are declared harmless.

Repair: inspect unsigned differential functioning and the substantive reasons for item differences. Opposing unfairness is not automatically fairness.

29. Failure Mode: Use Overall DTF to Hide a Threshold Problem

The average TCC difference is small across the full proficiency range, but the curves separate sharply near the certification cut.

Repair: inspect conditional DTF where decisions are made and quantify classification impact.

30. Failure Mode: Treat DTF as a Causal Explanation

The curves differ, so the report concludes that a particular cultural feature caused the difference.

Repair: DTF identifies differential measurement behaviour, not its causal mechanism. Substantive review, experimental evidence or targeted follow-up is required to explain why.

31. A Practical DTF Workflow

1. Define the groups and the construct being matched.
2. Establish a defensible measurement model and common metric.
3. Run item-level DIF analyses.
4. Estimate group-specific expected item scores or response functions where required.
5. Aggregate them into group-specific test characteristic curves.
6. Compute signed and, where useful, unsigned whole-test differences.
7. Account for item-parameter uncertainty.
8. Inspect DTF across θ, especially near decision thresholds.
9. Analyse subtests, forms or adaptive routes when those scores are operationally relevant.
10. Review flagged content substantively for fairness and construct relevance.
11. Quantify consequences for reported scores and classifications.
12. Document whether item removal, revision, reweighting or no action is justified.

32. What a Strong Fairness Report Should Separate

A useful report separates group impact, item-level DIF, test-level DTF, model assumptions, statistical uncertainty, substantive fairness review and operational consequences. Collapsing all of these into the word “bias” creates more heat than information.

The strength of measurement is not that it removes judgement; it makes the stages of judgement visible.

33. Classroom Translation

A teacher can use the idea informally when reviewing a class test. Suppose several language-heavy mathematics questions appear to disadvantage learners who understand the mathematics but struggle with the linguistic packaging. One question may barely change the total. Six such questions can shift the whole test.

The classroom lesson is not to eliminate language from mathematics. It is to ask whether the collection of task features changes the score in ways that exceed the capability the test is supposed to measure.

34. Missing-Node Scan

The missing node may be differential test functioning when a programme has a long list of DIF flags but cannot say whether total scores actually differ for matched groups; when opposing DIF items appear to cancel; when many small item effects point in the same direction; when fairness concerns cluster near a pass/fail boundary; when different forms assembled from the same bank show different fairness patterns; when adaptive routing changes who sees DIF items; or when zero net test-level difference is being used to excuse clearly problematic item content.

35. Evidence and Limits

Differential test functioning is an established extension of item-level DIF analysis. Chalmers, Counsell and Flora’s open-access work on improved DTF statistics explains how item-level differential functioning can aggregate at the test level and why sampling variability matters. The 2026 work by Sakamoto and Kumagai shows continuing methodological development using sum-score approaches. ETS fairness guidance situates DIF within a wider validity and fairness review process rather than treating statistical flags as self-interpreting verdicts.

The limits are important. DTF depends on the matching construct, scale, calibration and group model. It can describe expected-score differences without explaining their cause. Cancellation can hide item-level concerns, and aggregate equality does not prove every part of the assessment is fair. DTF is therefore one layer of fairness evidence, not the final judgement.

36. The Return Path

Return to the test with several DIF items.

Counting the items is not enough. Some differences may cancel. Others may align. The effect may disappear across most of the scale but emerge sharply near a consequential threshold. The whole test must be examined as a system.

Differential test functioning is the point where fairness analysis stops staring at individual components and asks what score the assembled machine actually produces for comparable learners.

Differential test functioning matters because fairness lives in both the parts and the whole. Item-level differences become operationally important when their combined effect changes the score or decision the test ultimately delivers.

Research and Further Reading

• Sakamoto & Kumagai — A Simple Approach for Differential Test Functioning Based on Sum Scores
• Chalmers, Counsell & Flora — It Might Not Make a Big DIF: Improved Differential Test Functioning Statistics
• ETS — Review of Differential Item Functioning Assessment Procedures
• ETS International Principles for the Fairness of Assessments
• ETS — Differential Testlet Functioning Definitions and Detection

eduKateSG Learning Node Series · 0180 · Previous: 0179 — How Test Characteristic Curves Work.