How Item Parameter Drift Works | When the Same Test Question Changes Difficulty Over Time

eduKateSG Learning Node Series · 0174

A test question can stay word-for-word identical while the measurement job it performs quietly changes.

A question about a once-unfamiliar technology may become easier as that technology enters everyday life. A curriculum reform can make one topic more heavily taught. Coaching materials can expose a secure item to thousands of candidates. Language usage can shift. A diagram that once demanded interpretation can become a familiar classroom template.

If the probability of a correct response changes across administrations after accounting for the proficiency the item is supposed to measure, an item parameter may have drifted. The item has not necessarily become “bad.” But the assumption that yesterday’s calibration can be carried forward unchanged now needs evidence.

Item parameter drift works when an item’s statistical relationship to the measured trait changes across administrations, making an old calibration an unreliable description of how the item functions now.

The 50-Second Read

Item response models attach parameters such as difficulty and discrimination to items.
Those parameters are useful only if they remain sufficiently invariant for the intended use.
Item parameter drift means an item’s parameter changes across administrations beyond what ordinary sampling variation would reasonably explain.
Difficulty drift is especially common to discuss, but discrimination or category parameters can drift too.
Drift can be caused by curriculum change, exposure, coaching, wording ageing, context change, security compromise or population–instruction interactions.
Item parameter drift is related to differential item functioning but the operational question is different: drift focuses on change over time or administrations.
Drift in anchor items can distort linking, equating and trend interpretation.
Drift in adaptive item banks can misdirect item selection and proficiency estimation.
A significant parameter difference is not automatically an educationally important drift event.
Monitoring needs statistical detection plus substantive item review.
Stable item pools require maintenance, recalibration, replacement and documentation.
The goal is not to freeze items forever; it is to know when the ruler has changed.

Canonical Owner Boundary

This node owns item-level parameter change across administrations and the maintenance problem created when old calibration no longer describes current item functioning. How Differential Item Functioning Works owns group-related item differences among learners matched on the construct. How Measurement Invariance Works owns the broader question of whether a construct is measured comparably across groups or time. Assessment Item Banks & Test Form Assembly owns the infrastructure that stores, selects and assembles reusable items. This article asks: what happens when one of those reusable items no longer behaves like the item we calibrated?

1. Calibration Is a Claim About a Relationship

In item response theory, an item parameter does not merely describe the words on a page. It describes a statistical relationship between responses and a latent trait under a model and population context. Difficulty locates the item on the proficiency scale. Discrimination describes how sharply the response probability changes across proficiency. Polytomous models add threshold or category parameters.

Using those parameters next year assumes that the relationship remains sufficiently stable. Item parameter drift is what we investigate when that assumption becomes doubtful.

2. The Item Can Look the Same While Its Context Changes

Consider a reading item built around the meaning of “streaming.” Twenty years ago, the context might have required inference from surrounding text. Today, widespread familiarity with streaming media could make the same wording easier for reasons unrelated to the intended reading skill.

The physical item did not change. The world around it did. Measurement therefore has to monitor not only edits to items but changes in the relation between items and learners.

3. Difficulty Drift Is the Most Intuitive Case

Suppose an item difficulty parameter was calibrated at 0.5 on a latent scale. Several years later, after placing old and new administrations on a defensible common metric, the same item behaves as though its difficulty were 0.0. Comparable learners are now more likely to answer correctly.

The numerical values here are illustrative. The key signal is not a particular amount but a systematic change that exceeds ordinary estimation noise and matters for the use of the item.

4. Discrimination Can Drift Too

An item can retain roughly the same overall difficulty while becoming less useful for distinguishing nearby proficiency levels. Perhaps a shortcut has become common. Perhaps a clue allows some learners to bypass the intended reasoning. Perhaps instruction has standardised one response route so strongly that the item now separates groups differently.

When discrimination drifts, the item can contribute a different amount of information even if its percentage-correct statistic appears stable.

5. Polytomous Items Add More Ways to Drift

A rubric-scored item may have several score categories. Drift can occur in the thresholds separating those categories, in the relative use of categories, or in how sharply scores relate to proficiency. Rater training, rubric reinterpretation or changing exemplars can all alter category behaviour.

This means drift monitoring is not limited to multiple-choice items.

6. Drift Is Not Ordinary Sampling Fluctuation

Parameter estimates change slightly whenever a new sample is drawn. That alone is not substantive drift. Analysts therefore compare observed changes with the uncertainty expected from estimation and use statistical procedures, effect-size criteria and repeated evidence to decide whether a change is credible and meaningful.

A tiny statistically detectable shift in an enormous sample may have little operational importance. A larger shift in a high-stakes anchor item may matter even when the p-value is not the most interesting part of the story.

7. Drift and DIF Are Close Relatives, Not Identical Jobs

Differential item functioning asks whether an item behaves differently for comparable members of different groups. Item parameter drift usually asks whether the item behaves differently across administrations or occasions.

Mathematically, time can be treated as a grouping variable, so the methods overlap. Operationally, however, the questions differ. DIF often triggers fairness investigation across demographic or linguistic groups. Drift monitoring often triggers item-bank maintenance, recalibration, anchor review or test-security investigation.

8. A Curriculum Can Move an Item

If a national curriculum begins teaching a concept earlier, an item targeting that concept may become easier at the same nominal grade. If instruction de-emphasises a procedure, an old item may become harder. Neither change necessarily means students became globally stronger or weaker.

This is why historical trend interpretation cannot assume that every reusable item is a permanent ruler.

9. Item Exposure Can Move an Item

Secure items can become familiar through repeated use, tutoring materials, memory-based reconstruction, online sharing or informal coaching networks. If familiarity gives future candidates an advantage not captured by the target construct, item difficulty can drift downward.

This connects parameter drift to test security and to the later Learning Node on item-exposure control.

10. Language Can Move an Item

Words change frequency, meaning and cultural familiarity. A phrase that once sounded formal may become ordinary. A reference can become dated. A once-neutral term can acquire a new connotation.

For language-heavy assessments, this can change item functioning even when the intended construct has not changed.

11. Technology Can Move an Item

A data-interpretation question that once required careful reading may become easier when students routinely use similar dashboards. A computing item can become obsolete when interfaces change. A science context can become more familiar after a major public event brings the concept into everyday media.

Items live inside changing technological environments.

12. Security Breach Is a Special Drift Mechanism

If a secure question leaks, future response probability can change suddenly. The pattern may be concentrated in highly exposed regions, coaching groups or administrations. Parameter drift can therefore function as one signal in a wider security investigation.

It is not proof of compromise by itself. Curriculum change, population change and model problems can produce similar statistical symptoms.

13. Anchor Items Make Drift Especially Dangerous

Anchor items are used to connect different forms or administrations onto a common scale. Their job depends on stability. If the anchor itself drifts, the linking transformation can absorb that change and move scores that should have remained comparable.

Feifei Li’s ETS research on drifted polytomous anchor items illustrates the problem: including drifted anchors can bias linking or equating, while identifying and removing problematic anchors can improve results under studied conditions.

14. A Small Drift Can Become a Large Trend Error

Suppose a group of anchor items becomes easier over time because their content is increasingly emphasised in instruction. If analysts treat those items as stable, part of the item change can be misread as a shift in the proficiency scale.

The resulting trend line may attribute movement to learners that partly belongs to the measuring instrument.

15. Adaptive Testing Has a Second Exposure to Drift

Computerised adaptive testing chooses questions using calibrated item parameters. If those parameters are stale, the algorithm may select an item because it expects high information at a proficiency level where the item no longer behaves as expected.

Drift can therefore affect both score estimation and the route through the test.

16. Measurement of Change Is Vulnerable

Longitudinal assessment tries to determine whether a person has changed. If item difficulty also changes across occasions, the observed difference mixes two moving systems: the learner and the ruler.

Research by Cooperman and colleagues on adaptive measurement of change under item parameter drift shows why this matters for individual change classification. When item parameters shift across occasions, apparent change can be distorted unless invariance is examined.

17. The Classic Maintenance Problem

Item parameter drift is not a new concern. Research by Bock and colleagues examined item pool maintenance in the presence of drift using long-run College Board Physics data. Their work showed that item location can change systematically across years and that content changes in schooling can be related to that movement.

The lesson remains current: reusable item banks need maintenance regimes, not just storage.

18. Detection Usually Starts With a Common Metric

You cannot compare item parameters meaningfully if the scales from two administrations are arbitrarily shifted or stretched. Analysts first need a defensible linking or concurrent-calibration framework so that parameter differences are not merely artefacts of scale indeterminacy.

Only then can observed item changes be interpreted as potential drift.

19. Detection Is Not One Universal Test

Methods include direct parameter comparisons, likelihood-based procedures, Wald-type tests, item-characteristic-curve comparisons, sequential monitoring and approaches that evaluate changes in expected scores. Different methods behave differently with sample size, model choice and the pattern of drift.

Guo, Zheng and Chang, for example, proposed a stepwise test characteristic curve method for identifying item parameter drift in repeated testing contexts.

20. Detection Needs a Substantive Review

A flagged item should be read again by content experts. Has the curriculum changed? Is a clue now obvious? Has a term aged? Is the keyed answer still defensible? Has the item appeared in commercial preparation materials? Did administration mode change? Did the surrounding test context change?

The statistics tell us where to look. They do not automatically tell us why the item moved.

21. Cross-Domain Comparison: Sensor Calibration Drift

An industrial temperature sensor can work perfectly when installed and slowly drift as components age. The display still produces precise-looking numbers. The danger is that the mapping between the physical temperature and the reading has changed.

An assessment item is not a metal sensor, but the analogy is useful. Calibration is not a certificate that lasts forever. It is a claim that must remain compatible with current evidence.

22. Cross-Domain Comparison: A Map Whose Roads Move

A navigation map can be internally beautiful and still send drivers into trouble if the road network changes. The map is not wrong because cartography failed; it is wrong because the world moved after the map was made.

Item parameter drift is the measurement version of that problem. The calibrated item is a map of expected responses. When instruction, culture or exposure changes, the map may need updating.

23. Failure Mode: Recalibrate Everything Every Time

A programme becomes so worried about drift that it treats every administration as a completely new scale.

Repair: preserve stable anchors and monitor them. Comparability requires some continuity. The goal is to detect meaningful instability, not abandon invariance as an aspiration.

24. Failure Mode: Freeze Every Parameter Forever

The opposite programme calibrates an item once and treats the parameter as permanent because changing it would complicate reporting.

Repair: schedule drift analyses, especially for long-lived anchors, high-exposure items, changing curricula and high-stakes uses.

25. Failure Mode: Delete Every Flagged Item

A statistical flag triggers immediate retirement.

Repair: investigate magnitude, cause and operational consequence. Some drift can be modelled or recalibrated. Some flags are sampling noise. Some items reveal genuine construct change that should not be hidden by automatic deletion.

26. Failure Mode: Keep a Drifted Anchor Because It Is Convenient

An item is deeply embedded in years of linking designs, so evidence of drift is dismissed to preserve continuity.

Repair: treat anchor status as a responsibility, not immunity. An unstable anchor can contaminate the very comparability it was chosen to protect.

27. A Practical Drift-Monitoring Workflow

Define the intended invariant relationship. State which parameters should remain stable and across which administrations.
Place administrations on a defensible common metric.
Estimate parameter differences with uncertainty.
Use statistical and practical flagging criteria.
Inspect repeated patterns, not isolated noise.
Review item content and administration history.
Check exposure and security evidence.
Evaluate the impact on scores, linking and decisions.
Recalibrate, replace, suspend or retain with documentation.
Re-run trend or equating analyses when drifted anchors materially affected the scale.
Record the reason for every maintenance action.

28. What Good Item-Bank Governance Looks Like

A mature item bank does not store only text, keys and parameters. It keeps calibration dates, sample information, exposure counts, content classifications, administration history, security status, known edits, drift flags and retirement reasons.

That record turns an item from a static file into a maintained measurement asset.

29. Classroom Translation

Teachers can see a simpler version of drift when reusing the same quiz year after year. A question that once separated deep understanding from surface learning may become trivial after it circulates in answer banks. Another may become unusually difficult because the textbook sequence changed.

You do not need an IRT model to notice that yesterday’s diagnostic question may no longer diagnose the same thing. Rotate evidence, inspect response patterns and refresh questions whose instructional environment has changed.

30. Missing-Node Scan

The missing node may be item parameter drift when an old item suddenly becomes much easier without a broad rise in achievement; when trend scores shift after a curriculum revision; when an anchor set produces inconsistent linking results; when adaptive tests repeatedly select items that no longer seem well targeted; when leaked or highly coached items outperform their historical calibration; when longitudinal change appears concentrated in a small subset of items; or when a test programme has a large item bank but no systematic process for deciding whether old calibrations remain current.

31. Evidence and Limits

Item parameter drift has a long measurement literature. Bock and colleagues documented the item-pool maintenance problem in repeated testing. More recent work continues to examine experimental identification, longitudinal change and the consequences of drift for linking and adaptive measurement. Baldwin and colleagues’ experimental study of item parameter drift highlights the continuing importance of distinguishing genuine parameter change from competing explanations.

The limit is that a statistical parameter never drifts in isolation from its model. Apparent drift can come from population differences, scale-linking problems, multidimensionality, mode effects or model misspecification. Detection therefore needs design, statistical evidence and substantive review together.

32. The Return Path

Return to the unchanged question.

Its wording is the same. Its answer key is the same. Its item-bank ID is the same. But the educational environment around it has changed, and comparable learners now respond differently.

The item is not a fossil. It is part of a living measurement system. The responsible response is neither panic nor denial. It is maintenance: detect, investigate, quantify, document and recalibrate when the evidence says the ruler moved.

Item parameter drift matters because a reusable question is only a stable measuring instrument while its relationship to the construct remains stable enough for the decision we ask it to support.

Research and Further Reading

eduKateSG Learning Node Series · 0174 · Previous: 0173 — How Classification Accuracy Works.