How Assessment Mode Effects Work | Check Whether Paper, Computer, Device or Interface Changed the Evidence

eduKateSG Learning Node Series · 0246

The question looks simple: if the Mathematics is the same, should it matter whether the test appears on paper, a laptop, a tablet or a remotely proctored screen?

Sometimes the answer is “very little.” Sometimes the answer is “enough to matter.” And sometimes the most important effect is not a single average score difference but a change concentrated in particular item types, response formats, subgroups, devices or interface actions.

An assessment mode effect is a systematic change in performance or score meaning associated with how an assessment is delivered or responded to rather than with the intended capability itself. Moving from paper to computer can alter navigation, reading, annotation, scrolling, typing, diagram interaction, timing, visibility of previous work and the physical effort required to construct a response. Moving from a test centre to remote proctoring can add another layer: equipment, connectivity, camera requirements, home conditions and different monitoring procedures.

Quick answer

Assessment mode effects work by changing the conditions through which the construct is observed. A change of mode is harmless only when the new conditions preserve the important task demands and do not add construct-irrelevant advantages or barriers large enough to alter interpretation.

The correct question is therefore not, “Are computer tests equivalent to paper tests?” It is: equivalent for which construct, which tasks, which population, which device, which interface, which decision and which period of transition?

The owned reader job

This Learning Node owns the evidence problem created when an assessment changes delivery or response mode. It covers paper versus computer, desktop versus tablet, test-centre versus at-home delivery, typing versus handwriting, scrolling versus page turning, and other interface changes that may alter task behaviour.

It does not replace the broader score comparability owner. It does not replace construct contamination, which owns the general problem of irrelevant barriers entering measurement. Nor does it replace response-process evidence. Mode-effect work uses all three ideas but owns the specific question: did the change in medium or administration alter the evidence enough that scores can no longer be treated as meaningfully interchangeable without adjustment or qualification?

Mode is not one variable

“Computer-based testing” sounds like one condition. It is not. Two computer assessments can differ in screen size, resolution, input device, scrolling behaviour, calculator access, highlighting tools, equation editors, navigation rules, item presentation, timing display, response persistence and accessibility features. A remotely proctored computer test may differ again in room requirements, identity checks, camera behaviour and interruption risk.

That means a mode-effect study should begin by decomposing the mode. What exactly changed?

Presentation: page, screen, scrolling, one item at a time, split panes, zoom, colour, animation.
Response: pencil, keyboard, touchscreen, stylus, drag-and-drop, equation editor, recorded speech.
Navigation: free movement, linear progression, flag-and-return, adaptive routing.
Tools: annotation, calculator, ruler, highlighting, scratch pad, copy-and-paste.
Environment: school, test centre, home, supervised room, remote proctoring.
Technology: device class, screen size, latency, browser, connectivity, peripheral hardware.
Familiarity: experience with typing, device gestures, platform conventions and digital test-taking.

If we collapse all of that into “paper versus computer,” we may detect a difference without knowing what produced it—or miss an important difference because opposite effects cancel in the total score.

Why the same item can become a different task

Consider a reading passage with eight questions. On paper, the passage remains physically visible while the student marks, circles and moves between questions. On a small screen, the same words may require repeated scrolling. If the interface places the question below the passage, the student may have to hold more location information in memory. If a split screen keeps both visible, that burden changes again.

The text has not changed. The task ecology has.

Now consider a writing assessment. Typing can make revision easier for a fluent keyboard user. It can also slow a learner who composes well but types poorly. Spellcheck, autocorrect and editing tools may become part of the response process unless controlled. Handwriting may introduce its own speed, legibility and motor demands. The question is not which mode is universally better. It is whether the mode introduces demands that are relevant or irrelevant to the construct being interpreted.

Three levels of mode evidence

A strong comparability study looks at more than an average score.

1. Test-level differences

Do groups taking different modes show a meaningful difference in total score after accounting for legitimate population differences? A small mean difference can be reassuring, but it is not enough by itself.

2. Item-level differences

Some items may become easier or harder even when the overall mean barely changes. Item-level differential functioning or mode-specific parameter comparisons can reveal whether particular formats are sensitive to the medium.

3. Response-process differences

Even comparable scores can conceal changed processes. Learners may navigate differently, reread less, revise more, use tools differently or experience new interface friction. Process evidence helps distinguish a score that remained stable because the construct was preserved from one that remained stable through compensating changes.

Randomisation is powerful when it is possible

If students choose their own testing mode, the groups can differ before the test begins. Learners choosing at-home remote testing may differ in geography, work schedule, technology access, age or other characteristics from learners choosing a test centre. A raw score comparison would mix mode with selection.

Random assignment is therefore especially valuable. When comparable participants are randomly assigned to paper or computer versions, systematic performance differences can be interpreted more cleanly as mode-related. Large-scale programmes have used randomized transition studies for precisely this reason.

When randomisation is not possible, researchers may use matched samples, propensity-score methods, pseudo-equivalent groups, regression adjustment or other designs. These methods can be useful, but their conclusions depend on the adequacy of measured covariates and assumptions about unmeasured differences. A sophisticated adjustment does not make selection disappear by declaration.

What real studies show: mode effects are empirical, not ideological

Research does not support a single rule that computer testing is always harder, always easier or always equivalent. Results depend on assessment, population and implementation.

An ETS study of an eighth-grade computer Mathematics assessment found the computer version statistically harder than a parallel paper version and found that computer facility predicted online performance after controlling for paper performance. Other ETS comparability studies have found small or nonsystematic overall effects, sometimes alongside item-level differences. A 2024 ETS study of remote-proctored at-home licensure testing, using propensity-score weighting to reduce group differences, reported small and nonsystematic estimated overall mode effects across the examined tests while noting some subgroup variation.

The lesson is not that one study is right and another wrong. The lesson is that mode effects are properties of an assessment system in context. They have to be measured rather than assumed.

Partial invariance: most items can travel while some do not

One of the most useful ideas for large-scale transitions is that comparability does not require every item to behave identically. A programme can test whether item parameters remain sufficiently stable across modes. If most common items behave comparably but a subset shows mode sensitivity, the invariant items may form the bridge used for linking while the affected items are modelled separately, revised or excluded from the anchor.

This is more realistic than two extreme positions: “the modes are identical” or “nothing can be compared.” Educational measurement often lives in the middle. The important task is to identify which evidence travels and which does not.

A worked transition study

Suppose a school system moves a 60-item science test from paper to tablet. The content blueprint and scoring scale are intended to remain stable.

Randomly assign a representative transition sample to paper or tablet where operationally feasible.
Keep content, timing and administration as comparable as possible while documenting unavoidable differences.
Compare total-score distributions and uncertainty, not only means.
Check item-level behaviour for mode-related DIF or parameter shifts.
Separate item families: long reading passages, graph interaction, constructed response, short selected response.
Collect response-process evidence from a smaller subsample: navigation, scrolling, tool use, cognitive interviews and observed friction.
Check important subgroups, including learners with different accessibility needs and digital experience.
Identify a stable common-item set for linking if partial rather than full invariance holds.
Monitor the first operational digital administrations for drift after the novelty of transition changes.

The result might be: total scores are highly comparable, short selected-response items are stable, several long-passage items become harder on smaller screens, and graph-interaction items cannot be treated as direct replicas because the digital version introduces a new interactive affordance. That is a far more useful conclusion than “tablet testing works.”

Device effects can hide inside mode effects

A digital assessment may be designed on a large laptop and administered on a small tablet. If layouts reflow, diagrams shrink, scrolling increases or on-screen keyboards cover response space, the device itself becomes part of the measurement environment.

This matters especially for responsive web testing. “Same URL” does not guarantee “same task.” Programmes should test supported device classes, orientation, zoom behaviour, accessibility technology and interaction patterns rather than treating all digital delivery as one condition.

Remote proctoring adds environment to mode

At-home testing illustrates why mode is a system. The test may be digitally identical, yet the administration can differ through room conditions, internet stability, identity verification, camera requirements, interruptions, equipment or proctor interactions.

A comparability study therefore needs to separate at least three questions: are the populations comparable, is the digital test functioning comparably, and are the administration conditions introducing a relevant difference? A small overall mode effect does not mean every examinee experienced the modes equally. Subgroup and process evidence still matter.

Familiarity is neither automatically bias nor automatically irrelevant

Digital familiarity presents a conceptual challenge. If the target construct is “write an essay using contemporary digital tools,” keyboard competence may be part of the authentic task. If the target is “compose a coherent argument regardless of input device,” severe typing demands may be construct-irrelevant. If the target is digital literacy, interface competence may be central rather than contaminating.

We cannot decide whether familiarity is a problem before defining the construct. Mode-effect research therefore starts with a construct argument, not a preference for paper or screens.

Transition effects can fade—or emerge

The first year of a mode change may not predict the fifth. Students and teachers adapt. Devices become more familiar. Curricula may incorporate digital practices. Testing platforms change. Accessibility tools improve. Conversely, a new feature may introduce a fresh source of noncomparability.

This means comparability is not a one-time certification exercise. Programmes should monitor mode-sensitive indicators after transition, especially when software, devices, task formats or populations change materially.

What not to infer from a mode study

No significant mean difference does not prove every item, subgroup and response process is equivalent.
A significant mean difference does not automatically prove bias; population imbalance or implementation differences may be involved.
High score correlation does not prove identical meaning if a systematic shift remains.
One successful transition does not establish universal computer–paper equivalence.
Digital familiarity predicts performance does not by itself show the test is invalid; the role of familiarity depends on the intended construct.
A new digital feature improves authenticity does not mean the score remains on the old scale without new evidence.

A mode-equivalence decision table

Question	Evidence	If it fails
Were groups comparable?	Randomisation, matching, covariate balance	Do not attribute raw differences to mode
Were total scores comparable?	Means, distributions, effect sizes, conditional comparisons	Investigate source and linking implications
Were items comparable?	DIF, parameter comparisons, item-family analyses	Exclude, model or revise mode-sensitive items
Were response processes comparable?	Cognitive labs, navigation logs, timing, observation	Reconsider whether the same construct is being elicited
Were subgroups affected similarly?	Interaction and subgroup analyses	Investigate access, familiarity and fairness
Can the scale remain linked?	Invariant common items, equating/linking evidence	Qualify trends or establish a new scale bridge

For teachers and schools

Schools do not need a national psychometric laboratory to make better mode decisions. They can run smaller, disciplined checks when changing assessment conditions.

Pilot the new mode before using results for consequential decisions.
Use the same learners or randomized equivalent groups where practical.
Inspect item types separately instead of only comparing overall percentages.
Ask learners where the interface changed what they had to do.
Test the devices that will actually be used, not only the designer’s laptop.
Preserve legitimate accessibility supports.
Do not interpret early transition friction as permanent learner weakness.
If comparability is uncertain, reduce stakes until more evidence accumulates.

For learners and parents

If a student’s score changes sharply after an assessment mode changes, neither “the student got weaker” nor “the computer caused it” should be the automatic conclusion. Compare task types, timing, navigation, input method, accessibility conditions and prior performance. Look for whether the change is concentrated in digital reading, typed writing, graph manipulation or another mode-sensitive demand.

The right response is diagnosis, not mythology about screens or paper.

Failure modes

The platform-equals-mode failure: all digital conditions are treated as identical.
The mean-only failure: an unchanged average hides item or subgroup shifts.
The self-selection failure: voluntary mode groups are compared as if they were randomized.
The novelty failure: first-year unfamiliarity is treated as a permanent property of the mode.
The device-blind failure: screen size and input mechanics are ignored.
The construct-blind failure: digital skills are called bias without asking whether they belong to the target capability.
The scale-presumption failure: a redesigned digital task is assumed to live on the old reporting scale automatically.

Sources and further reading

Return to the core idea: changing the medium can change the measurement because medium is part of the route through which performance becomes visible. Good assessment does not assume that mode matters or that it does not. It makes the change explicit, studies the evidence, preserves what is invariant and refuses to call two scores interchangeable until the bridge between them has earned that claim.