How Missing-Response Cognitive Diagnosis Works | Keep Skipped and Unadministered Items From Becoming Wrong Answers

eduKateSG Learning Node Series · 0220

A blank answer is not a wrong answer until you know why it is blank.

One learner sees Item 18, decides it looks difficult and skips it. Another never reaches Item 18 because time expires. A third is taking an adaptive test in which Item 18 is never administered. A fourth loses the response because the network fails. A fifth produces work that cannot be scored because the uploaded image is corrupted.

All five cells can appear empty in a dataset. They are not the same event. If a cognitive-diagnostic system converts every empty cell into zero, it can manufacture skill deficits. If it simply deletes every empty cell, it can discard information about who skips which items and why.

Missing-response cognitive diagnosis works by separating the response model from the process that makes responses absent, so skipped, not-reached, structurally unadministered and technically missing items are not silently treated as incorrect evidence about mastery.

The 50-second route

A missing response is an observation about data availability, not automatically an incorrect answer.
Skipped items, not-reached items, unadministered adaptive items and technical failures have different mechanisms.
Missing completely at random, missing at random and missing not at random describe different relationships between missingness and observed or unobserved quantities.
In diagnostic assessment, missingness can depend on item difficulty, learner capability, confidence, time or strategy.
When missingness depends on the same latent variables used for diagnosis, ignoring it can bias item parameters and attribute profiles.
A joint model can represent both the probability of a response being missing and the probability of a correct response.
Skipped and not-reached items should not be merged automatically because their missingness processes can differ.
Adaptive-test items that were never selected are structurally absent by design and should not become wrong answers.
Imputation can be useful, but an imputed value is not a recovered fact.
Item position, response time and Q-matrix structure can interact with missingness.
The data system should preserve a reason code for nonresponse whenever possible.
The final educational question is whether handling missingness changes the diagnosis in a way that survives fresh evidence.

Canonical owner boundary

This node owns missing-response mechanisms inside cognitive-diagnostic assessment. How Cognitive Diagnostic Models Work owns the general diagnostic framework. How Response-Time Cognitive Diagnosis Works owns timing as additional diagnostic evidence. How Cognitive-Diagnostic Adaptive Testing Works owns adaptive item selection. This article asks the narrower question: what should a diagnostic model do when the learner’s response is absent, and what can the absence itself legitimately tell us?

1. Start by distinguishing five kinds of empty cell

Skipped: the item was available and the learner moved past it without an answer.

Not reached: a run of items at the end of a timed assessment was never answered because the learner did not get that far.

Not administered: the testing design deliberately did not present the item—for example in adaptive testing or matrix sampling.

Technical missingness: the response was lost, corrupted or never transmitted because of a system problem.

Invalidated response: something was submitted, but the programme has decided the response cannot support scoring under the defined rules.

Collapsing these states into one blank code removes information before modelling even begins.

2. Missing is not wrong

Suppose a learner answers nine of ten diagnostic questions correctly and one item is unadministered because the adaptive algorithm never selected it. Coding that item as incorrect lowers the total and sends negative evidence toward every attribute attached to the item.

The same mistake is more subtle with skips. A learner may skip because they do not know how to solve the item, in which case the missingness is related to mastery. But they may also skip because they intend to return later, because the item display failed or because the instructions encouraged strategic item choice.

The scoring rule should follow the assessment design. A policy may deliberately penalise omitted responses in a particular examination, but that operational score rule is not the same thing as saying an omitted item is direct cognitive evidence of nonmastery.

3. Why deleting missing rows can also be wrong

A complete-case analysis keeps only learners with every required response observed. That can change the population being diagnosed.

If learners with weaker skills are more likely to run out of time, removing incomplete cases preferentially removes lower-performing learners. Item parameters and mastery-profile frequencies can then look more favourable than they really are for the intended population.

Even an available-case approach that uses every observed response can be biased when the probability that a response is missing depends on latent mastery or item properties in ways the model ignores.

4. MCAR, MAR and MNAR describe different missingness stories

Missing completely at random, or MCAR, means the missingness does not depend on observed or unobserved quantities relevant to the analysis. A random server outage affecting arbitrary response cells is an approximate example.

Missing at random, or MAR, means the missingness can depend on observed information once that information is included appropriately in the model.

Missing not at random, or MNAR, means the probability of missingness still depends on unobserved quantities or on the missing value/process itself after conditioning on observed information.

These labels are assumptions about a data-generating process, not properties that can usually be read directly from a dataset. Sensitivity analysis matters because different missingness stories can fit the observed data similarly while implying different diagnoses.

5. Skipping can be diagnostic—and that is exactly why it cannot be ignored

A learner is more likely to skip an item they perceive as impossible. Perceived difficulty can be related to the attributes the item requires. If so, missingness contains information about the same latent knowledge structure the assessment is trying to estimate.

Liang, Lu, Zhang and Cheng’s 2024 study, Modeling Skipped Items in Cognitive Diagnostic Assessments, proposes a joint framework in which a missing-indicator model is combined with a DINA response model. The higher-order structure allows relationships between skipping propensity, higher-order ability and item characteristics.

Their simulations reported improved attribute-profile classification and item-parameter recovery under studied nonignorable missingness conditions. Their empirical illustration used PISA 2018 computer-based data. These findings support modelling skipped responses explicitly; they do not establish that every skip in every classroom means missing knowledge.

6. Not-reached items are a different mechanism

A learner who skips Item 4 and answers Item 5 has shown a different response pattern from a learner whose responses stop after Item 17 and remain empty through Item 30.

The second pattern suggests a dropout or time-limit process. Liang, Lu, Zhang and Shi’s 2022 model for not-reached items in cognitive diagnostic assessments combines a response model with a sequential missing-indicator model and links higher-order ability with dropping-out propensity.

The distinction matters because the remedy differs. A skipped item may call for modelling strategic or difficulty-related omission. Not-reached items may require speededness analysis, timing redesign or a model of dropout under time pressure.

7. A worked example shows how “missing = wrong” can distort a profile

Imagine six items. Items 1–3 require attribute A; Items 4–6 require attribute B. A learner answers Items 1–3 correctly, Items 4–5 correctly, and Item 6 is missing because it was not administered in an adaptive route.

If the missing item is coded as wrong, the observed pattern for B becomes 1, 1, 0. If it remains structurally missing, the observed B evidence is 1, 1 with no response on Item 6.

The first representation adds negative evidence that never occurred. In a diagnostic model with only a few items per attribute, one invented failure can materially change the posterior mastery probability.

The example is intentionally simple. In a real adaptive test, the fact that Item 6 was not administered may itself depend on earlier responses and the adaptive algorithm. The likelihood should reflect the test design rather than treating every missing cell as exchangeable.

8. Structural missingness is often part of good assessment design

Matrix-sampled assessments deliberately give different subsets of items to different learners. Adaptive tests select only items useful for the current estimate. Neither design expects a complete response matrix.

Calling unadministered cells “missing data” can be technically convenient, but operationally they are missing by design. The administration mechanism is known and should enter the analysis appropriately.

A diagnostic system should preserve the distinction between “the learner declined to answer” and “the system never asked.”

9. Missingness can interact with the Q-matrix

Suppose items requiring attribute C are substantially harder and are also more likely to be skipped. If the model ignores missingness, C may be underrepresented in the observed response evidence. If missing responses are coded as wrong, C may be made to look weaker than it is.

The direction of bias depends on the missingness process, item design and model. This is why Q-matrix validation and missing-response modelling are complementary. One determines where item evidence should route; the other determines what an absent response means for that evidence.

10. A joint model gives missingness its own probability

Conceptually, a joint missing-response model contains two related questions for each person–item pair:

1. What is the probability that a response is observed?
2. If it is observed, what is the probability of a correct response?

The first can depend on a missingness propensity and item missingness characteristics. The second depends on the cognitive-diagnostic response model. Higher-order or correlated structures can connect them when the same underlying learner or item characteristics influence both processes.

This is more demanding than simply filling every blank. It makes the missingness mechanism explicit and estimable under assumptions.

11. Correlation does not identify the reason for a skip

If lower estimated proficiency is associated with more skipping, several mechanisms remain possible. Weaker learners may perceive more items as difficult. They may spend longer on earlier items and run out of time. The test may route them differently. Motivation may covary with prior performance.

A joint model can represent association between latent ability and skipping propensity. It does not automatically distinguish every psychological cause of the association.

Use response times, item positions, process data and administration records to investigate competing explanations when the reason matters for instruction or test design.

12. Imputation is a prediction, not recovered history

Another approach fills missing responses with predicted values and then fits a cognitive-diagnostic model. Imputation can preserve sample size and can work well under specified conditions.

But an imputed 1 is not an answer the learner actually gave. An imputed 0 is not an observed failure. Downstream reports should not erase that provenance.

Multiple imputation propagates uncertainty by creating several plausible completed datasets. Single-value methods are simpler but can understate uncertainty when the imputation is treated as known.

13. A 2025 machine-learning imputation method shows the trade-offs clearly

You, Yang and Xu’s 2025 random forest dynamic threshold imputation method was designed for cognitive diagnostic data. Their simulations compared attribute-classification accuracy under several missingness mechanisms and proportions.

The study reported that its RFDTI method outperformed several traditional imputation approaches in its simulation conditions, while performance relative to an earlier random-forest threshold method varied by missingness mechanism. The paper also received a published correction in September 2025, a reminder that current methodological sources should be checked for updates.

Machine-learning imputation does not remove the missing-data assumptions. It changes the prediction machinery. If missingness is strongly nonignorable or the training data do not represent the deployment population, sophisticated imputation can still be confidently wrong.

14. Not reached can be treated as a timing problem as well as a missing-data problem

When a time limit censors the response process, the final unanswered items carry information about both timing and accuracy. Guo, Xu, Ying and Zhang’s response-time censoring approach for not-reached items incorporates the censoring mechanism into the likelihood of item responses and response times.

That work is not specific to cognitive diagnosis, but it supplies an important crosswalk: an unanswered final item may be the result of a time process rather than an ordinary missing response. The appropriate model should reflect the mechanism that generated the blank.

15. Item position can create a false attribute pattern

Suppose all items for attribute C happen to appear near the end. Learners who run out of time now have missing C-items disproportionately. A naive analysis may infer that C is the cohort’s weakest skill when the test has confounded attribute and position.

Balance attribute coverage across positions where feasible. In fixed forms, rotate positions or use multiple forms. In adaptive systems, simulate route distributions to see whether particular profiles encounter systematically different item positions or lengths.

Measurement architecture can create the missing-data pattern that later appears to diagnose the learner.

16. Response time can help distinguish skipped from not-reached mechanisms

A learner who spends unusually long on several preceding items and then leaves the remainder blank presents a different time signature from a learner who quickly skips isolated difficult questions while continuing through the test.

Response-time cognitive diagnosis can contribute to the investigation, but time still does not reveal intention directly. Interface logs and item navigation events may provide stronger evidence about whether an item was opened, revisited or abandoned.

17. Technical missingness should be quarantined from learner inference

If a response disappears because a browser crashes, the missingness is evidence about the delivery system. It should not alter a learner’s cognitive profile as though the learner failed the item.

Store technical event logs separately and create clear invalidation/recovery rules. If the response cannot be reconstructed reliably, preserve the uncertainty and collect new evidence where necessary.

The system that measures the learner must also be able to recognise when it is measuring itself.

18. Missingness can be a consequence of test-taking strategy

In some assessments, strategic skipping is rational. A learner may leave a difficult item and return later. The first blank is therefore a temporary workflow state, not a final omission.

Use final-response status, navigation logs and return behaviour rather than treating the first skip event as permanent missingness. If the examination penalises unanswered items, that consequence belongs to exam strategy; it still does not turn the blank into direct evidence that the learner lacks every required attribute.

19. A classroom diagnostic should often ask again rather than impute

Large-scale assessment may need sophisticated missing-data models because re-testing every missing cell is impossible. A tutor with three students often has another option: ask a fresh, equivalent question.

If a learner leaves a fraction-comparison item blank, the teacher can first ask whether time, wording or uncertainty caused the omission. Then present a new item targeting the same attribute under clear conditions.

Direct new evidence can be more useful than trying to reconstruct a missing response statistically. Model sophistication should not replace an available observation.

20. Cross-domain comparison: an offline sensor is not a zero reading

If a temperature sensor goes offline, the database may show an empty cell. Replacing that cell with 0°C changes “no reading” into a very specific physical claim.

Missing educational responses have the same logical boundary. Absence of an observation is not the observation of failure. A model may infer what probably happened, but provenance should distinguish inference from measurement.

21. Cross-domain comparison: inventory not counted is not zero inventory

A warehouse audit leaves one shelf uncounted. Recording zero stock would create an artificial shortage. Ignoring the shelf entirely may also bias the total if uncounted shelves are systematically the hardest to access.

The correct response is to preserve the missing state, investigate why it is missing, and use an appropriate inference method only with its uncertainty visible.

22. Failure modes

Failure: blank = wrong. Repair: distinguish scoring policy from cognitive evidence and preserve missingness reason.

Failure: delete every incomplete learner. Repair: examine whether missingness is related to skill, item or timing variables.

Failure: combine skipped and not-reached items. Repair: model their different mechanisms or keep them separate.

Failure: impute and forget which values were imputed. Repair: retain provenance and propagate uncertainty.

Failure: interpret adaptive nonadministration as learner omission. Repair: represent the item-selection design explicitly.

Failure: blame a skill for technical missingness. Repair: quarantine delivery-system failures from learner inference.

23. A practical missing-response audit

Create distinct codes for skipped, not reached, not administered, technical missing and invalidated responses.
Preserve item position, timing and navigation information where appropriate.
Check missingness rates by learner, item, attribute, position and administration condition.
Ask whether missingness plausibly depends on latent mastery or item difficulty.
Compare available-case, joint-model and imputation approaches where appropriate.
Run sensitivity analyses across plausible missingness mechanisms.
Inspect how each approach changes item parameters and attribute-profile classifications.
Keep imputed values marked as imputed.
Protect structurally unadministered items from being scored as wrong.
Recollect direct evidence when a consequential diagnosis rests heavily on missing responses.
Revalidate after timing rules, test design or adaptive algorithms change.

24. Rainbolt missing-node scan

The missing node may be missing-response cognitive diagnosis when a diagnostic platform silently codes blanks as zero; when learners who run out of time receive profiles that look like content weaknesses concentrated in late-test attributes; when adaptive tests contain thousands of unadministered cells that downstream analytics treat as failures; when skipped and not-reached items share one code; when imputation changes many mastery classifications but the report does not disclose it; when technical outages produce apparent learning deficits; or when an absent answer is interpreted more confidently than the observed answers surrounding it.

25. Evidence and limits

Current cognitive-diagnostic research increasingly treats nonresponse as a process worth modelling rather than a nuisance to erase. Liang and colleagues’ 2022 not-reached model and 2024 skipped-item model distinguish two important missingness mechanisms. The 2025 RFDTI study develops a machine-learning imputation approach and reports performance differences across missingness conditions.

No method identifies the missingness mechanism from wishful thinking. MNAR assumptions are particularly difficult to verify from observed data alone. The strongest practice combines design knowledge, reason codes, process data, sensitivity analysis and fresh evidence when the consequence of the diagnosis is substantial.

26. The return path

Return to Item 18.

One learner skipped it. One ran out of time. One was never shown it. One lost it to a network error. One submitted an unusable response. The same empty database cell can therefore point to five different systems.

A trustworthy diagnostic model keeps those routes distinct until the evidence justifies combining them.

When an answer is missing, the first diagnostic question is not “Was it wrong?” It is “What process made the evidence disappear?”

Research and onward reading

eduKateSG Learning Node Series · 0220 · Previous: 0219 — How Response-Time Cognitive Diagnosis Works.