How Cognitive-Diagnostic Test Assembly Works | Build a Test That Can Distinguish the Skill Profiles You Intend to Report

eduKateSG Learning Node Series · 0225

A diagnostic test can contain excellent questions, cover every chapter and still be unable to tell two important skill profiles apart.

Imagine a mathematics assessment that promises to tell a teacher whether a learner is weak in equation formation, equation solving, unit conversion or some combination of the three. The item bank contains dozens of respectable questions. Yet nearly every word problem requires formation and solving together. The bank has coverage. It does not have enough diagnostic contrast.

A conventional test blueprint often asks whether content areas, difficulty levels and formats are represented. A cognitive-diagnostic test needs another layer: can the chosen set of items distinguish the latent skill profiles the report intends to name?

Cognitive-diagnostic test assembly works by selecting a feasible set of items whose Q-matrix patterns and response behaviour collectively separate the skill profiles and attributes the assessment intends to report, while still satisfying content, length, time, exposure and operational constraints.

The 50-second read

A diagnostic test is assembled for classification, not only for total-score precision.
The Q-matrix determines which attributes each item is intended to require.
Coverage is necessary but not sufficient: attributes must appear in combinations that let competing profiles be distinguished.
A long test can still be diagnostically weak if it repeats the same attribute bundles.
Automated test assembly can optimise objectives such as average classification error, worst-attribute error or diagnostic-information indices.
Operational constraints still matter: content balance, item format, test length, response time, security and exposure cannot be ignored.
An objective function is not a validity argument. A mathematically optimal form is only as meaningful as the Q-matrix, item model and attribute definitions behind it.
Profile-level accuracy and attribute-level accuracy are different goals.
Parallel diagnostic forms should be compared on diagnostic behaviour, not only on total difficulty.
Field evidence should be used to check whether the assembled test actually separates profiles in real data.

Canonical owner boundary

This node owns the construction of a fixed or module-based test form whose item set is optimised for cognitive-diagnostic classification under practical constraints. How Cognitive Diagnostic Models Work owns the general modelling framework. How Q-Matrix Validation Works owns checking whether item-to-skill mappings are defensible. How Cognitive-Diagnostic Adaptive Testing Works owns item-by-item adaptive selection during administration. Assessment Item Banks & Test Form Assembly owns the broader infrastructure of reusable item banks and form construction. This article asks the narrower question: how should a whole diagnostic form be assembled when the goal is to distinguish skill profiles rather than merely produce one total score?

1. Start with the diagnostic decisions, not the item bank

A useful assembly process begins by naming the distinctions the assessment must support. If the report will distinguish five attributes, what decisions change when one attribute rather than another appears weak? Which combinations matter instructionally? Which profiles are plausible under the curriculum or attribute hierarchy?

This reverses a common workflow. Instead of starting with hundreds of available items and asking how to fill a 40-item test, start with the diagnostic decisions and ask what evidence would make those distinctions observable.

If two reported attributes always appear together in every selected item, the test may be unable to separate them even if both attributes are nominally “covered.” Diagnostic assembly therefore treats the pattern of requirements as part of the measurement design.

2. Coverage is not separability

Suppose a four-attribute assessment contains ten items for each attribute. That sounds balanced. But imagine every A item also requires B, and every B item also requires A. A and B may be mentioned equally often while remaining difficult to distinguish.

Diagnostic assembly asks a stronger question: do the selected q-vectors create enough contrast among the profiles that matter? Items requiring A alone, B alone, A+B, and other informative combinations can supply different kinds of evidence. The exact requirements depend on the diagnostic model and attribute structure.

This is similar to experimental design. Measuring two factors in exactly the same combination on every trial makes their effects hard to separate. More observations of the same confounded design do not manufacture the missing contrast.

3. The objective function decides what “best test” means

Automated test assembly requires an objective. In a conventional IRT setting, an objective might maximise information near a cut score or match a target information function. In cognitive diagnosis, possible objectives include minimising average profile-classification error, protecting the weakest attribute, hitting target attribute accuracies, or maximising a diagnostic-information index.

Finkelman, Kim and Roussos introduced a genetic-algorithm approach that could target average classification error, maximum attribute error or other prescribed diagnostic objectives. A later binary-programming approach showed how attribute-level discrimination indices could be integrated into optimisation more directly. These are algorithmic strategies, not interchangeable definitions of validity.

Finkelman, Kim & Roussos — Automated Test Assembly for Cognitive Diagnosis Models Using a Genetic Algorithm

Finkelman et al. — A Binary Programming Approach to Automated Test Assembly for Cognitive Diagnosis Models

4. Profile accuracy and attribute accuracy are not the same target

Consider four binary attributes. A complete mastery profile contains four bits, such as 1011. A profile is counted wrong if even one attribute is misclassified. Attribute accuracy, by contrast, can be high even when complete profiles are less often exactly correct.

Suppose a system classifies each attribute correctly 90% of the time and, for illustration, the four attribute errors were independent. Exact four-attribute profile accuracy would then be 0.9 to the fourth power, about 0.656. Real attribute errors are not necessarily independent, but the arithmetic shows why profile-level success can be far harder than good marginal accuracy.

The intended report therefore matters. If the teacher will act on individual attribute probabilities, the assembly objective may emphasise attribute-level discrimination. If the system routes learners according to complete profiles, profile-level confusion becomes more consequential.

5. Protect the weakest attribute instead of polishing the average

An average objective can hide a neglected attribute. Suppose four attributes have classification accuracies of 0.95, 0.95, 0.94 and 0.62. The average looks respectable, but the fourth report is too fragile for confident instructional use.

A maximin or worst-attribute objective can force the optimisation to improve the weakest diagnostic dimension. This may require giving up a little efficiency elsewhere. Whether that trade is worth making depends on the consequences attached to each reported attribute.

This resembles reliability engineering: a system with nine excellent components and one critical weak link may fail through the weak link. Diagnostic reporting should not hide one unsupported skill claim behind several strong ones.

6. Attribute frequency needs structure, not quotas alone

Blueprint rules such as “each attribute must appear in at least eight items” are useful capacity controls. They do not guarantee identifiability or useful contrast. Eight items can all repeat the same q-vector.

Assembly should inspect the distribution of q-vectors, not only column totals. Are there items isolating important attributes? Are there enough integrated tasks to test combinations? Are the necessary patterns present across content areas rather than concentrated in one narrow format?

When attribute hierarchies are part of the model, some q-vectors or profiles may be structurally redundant or implausible. Attribute hierarchy should therefore inform assembly rather than being pasted onto the analysis after the form has been built.

7. Item quality and Q-matrix quality interact

A theoretically ideal q-vector does not rescue a poor item. If the question is ambiguous, weakly discriminating, miskeyed or dominated by an unintended reading demand, its nominal attribute pattern can overstate the evidence it supplies.

Conversely, a statistically strong item can be diagnostically unhelpful if it repeats a pattern already oversupplied. Test assembly is therefore a portfolio problem: the value of an item depends partly on what else is already in the form.

Marginal value matters. The twentieth excellent A+B item may add less diagnostic information than a merely good A-only item if the test currently cannot separate A weakness from B weakness.

8. A worked miniature assembly problem

Suppose a bank has twelve candidate items for three attributes A, B and C. The final form can contain only six. A content rule requires at least two algebra-context tasks and two geometry-context tasks. A diagnostic rule requires every attribute to appear in at least three items.

One candidate form contains q-vectors: AB, AB, AB, C, C, ABC. All three attributes meet the frequency rule. Yet A and B are never separated. A second form contains A, B, C, AB, BC, AC. It provides cleaner contrasts among the three attributes.

This does not prove the second form is better in real data. Item parameters, content, difficulty, local dependence and model assumptions matter. It shows why a simple frequency blueprint can certify a form whose diagnostic geometry remains poor.

9. Content constraints are not the enemy of optimisation

A diagnostically informative form that ignores curriculum balance can become educationally invalid. If all high-discrimination items come from one topic, the resulting profile may describe performance on that topic more than the intended domain.

Automated assembly is valuable precisely because it can satisfy several constraints simultaneously: content categories, cognitive processes, format counts, passage limits, item enemies, exposure restrictions and diagnostic targets.

The feasible set is the intersection of those requirements. If no form satisfies all hard constraints, the correct result is “infeasible,” not a quiet violation. An infeasible blueprint is useful information about the item bank.

10. Time is part of the form

A test can be diagnostically excellent on paper and unusable if it systematically exceeds the available testing window. Finkelman, de la Torre and Karp developed a cognitive-diagnostic automated assembly procedure that incorporates response-time constraints so test forms can target diagnostic information while controlling the total-time distribution.

Cognitive diagnosis models and automated test assembly: an approach incorporating response times

Time constraints should not be treated as neutral if speed is not part of the intended construct. A form that achieves its diagnostic precision by adding many slow items can create speededness that changes what responses mean.

11. Parallel forms need diagnostic parallelism

Two forms can have similar total-score means and still differ in their ability to classify individual attributes. If Form A contains clean contrasts for A versus B while Form B bundles them repeatedly, the same learner may receive different diagnostic profiles.

Parallel-form assembly should therefore compare profile confusion, attribute-level precision and Q-matrix structure alongside conventional form statistics. The relevant invariance is not merely equal overall difficulty.

When different forms are used across administrations, the diagnostic-reporting scale itself needs maintenance. A stable total score cannot compensate for unstable skill-level meaning.

12. Multistage forms create a second assembly problem

In multistage testing, the unit of adaptivity is a module rather than one item. Each module must be assembled in advance, yet the complete panel must support multiple routes through the test.

Li and colleagues demonstrated automated test assembly for multistage testing with cognitive diagnosis, using attribute reliability and other statistical and non-statistical constraints. The design problem becomes larger because modules need to function individually and as parts of possible paths.

Automated Test Assembly for Multistage Testing With Cognitive Diagnosis

13. Genetic algorithms and binary programming solve different optimisation landscapes

A genetic algorithm searches by evolving candidate forms through operations inspired by selection and recombination. It can handle flexible objectives but may require substantial computation and does not guarantee that every run finds the global optimum.

Binary integer programming represents each candidate item with a 0/1 selection variable and expresses many constraints mathematically. When the objective and constraints fit that structure, modern solvers can provide strong optimisation guarantees or prove infeasibility.

Heuristics, mixed-integer programming and hybrid approaches are tools. Method choice should follow the actual assembly problem rather than becoming a prestige contest between algorithms.

14. The bank can be the real bottleneck

If no items isolate attribute C, no optimiser can create that contrast. If all C items are slow and difficult, every feasible test may inherit the same weakness. If one important q-vector appears in only two exposed items, security rules can make repeated forms impossible.

Assembly diagnostics should therefore feed back into item development. Instead of reporting merely “solver failed,” identify which constraint or diagnostic target lacks bank capacity. This turns optimisation into a planning tool for future item writing.

The best next item to write is often not another strong item in a rich region. It is an item that fills a missing q-vector, content cell, difficulty band or response-time niche.

15. Robust assembly asks whether the form survives parameter uncertainty

Item parameters and Q-matrix entries are estimated or judged with uncertainty. A form optimised to one exact parameter set can be fragile if small changes produce a different “optimal” selection or sharply worse classification.

Useful sensitivity work perturbs plausible parameter values, alternative Q-matrix specifications or model choices and checks whether important design conclusions persist. A form that remains strong across reasonable alternatives may be preferable to one that is narrowly optimal under one uncertain calibration.

This is the test-construction analogue of engineering tolerance: the goal is not merely to work at the nominal value, but to remain useful when reality deviates slightly from the model.

16. Cross-domain comparison: designing a sensor array

Imagine diagnosing faults in a machine with six sensors. If every sensor reacts to both the pump and the valve in exactly the same way, the system can detect “something is wrong” while failing to isolate the fault. Adding more copies of the same sensor raises confidence in the combined alarm without improving separation.

A cognitive-diagnostic test is similar. Items are imperfect sensors of latent skill states. Useful assembly requires complementary sensor patterns, not merely a large sensor count.

The analogy stops where human learning begins. Learners can change strategy, interpret wording and improve during instruction. The point is structural: diagnostic power depends on the pattern of what each observation can distinguish.

17. Cross-domain comparison: network observability

In control engineering, a system is observable when its internal state can be inferred sufficiently from measured outputs under the model. Placing ten sensors on one redundant location may be less useful than distributing fewer sensors across the variables that distinguish competing internal states.

Cognitive-diagnostic assembly has an analogous concern: does the chosen item set make the intended latent profiles distinguishable? The mathematical details differ, but the design habit transfers—measure where state differences become visible.

18. Failure mode: optimise a wrong Q-matrix perfectly

The solver finds a beautiful form under a Q-matrix that misstates what several items actually require.

Repair: treat Q-matrix validation as an upstream gate. Use content review, response-process evidence and empirical diagnostics. Optimisation magnifies assumptions; it does not validate them.

19. Failure mode: satisfy every quota but separate no important profiles

Every attribute appears eight times and every topic appears in the correct proportion, yet two instructional profiles remain nearly observationally identical.

Repair: add profile-discrimination or attribute-level objectives and inspect the q-vector geometry directly. Frequency is a capacity constraint, not proof of diagnostic usefulness.

20. Failure mode: maximise average quality and abandon one attribute

The global objective improves while one important skill remains poorly classified.

Repair: introduce minimum attribute-level requirements, worst-case objectives or explicit weighting tied to consequences. The optimisation criterion should reflect the reporting obligation.

21. Failure mode: treat solver output as a finished test

A selected form meets all numerical constraints, so it goes operational without content review.

Repair: inspect passage clustering, unintended clues, item enemies, formatting, accessibility, answer-key distribution and substantive coherence. Automated assembly selects from the bank; it does not read the test like a learner.

22. A practical assembly workflow

Define the diagnostic decisions. Name the attributes and profiles whose distinctions change instruction.
Verify the attribute and Q-matrix theory. Do not optimise an unresolved mapping.
Specify hard constraints. Content, format, length, time, exposure, passages, accessibility and administration requirements.
Specify diagnostic objectives. Profile accuracy, attribute accuracy, minimum precision or other explicit targets.
Audit bank capacity. Identify missing q-vectors, weak attributes and thin content cells.
Choose an optimisation method. Binary programming, genetic algorithm, heuristic or another justified approach.
Assemble several candidate forms. Do not rely on one solution when near-optimal alternatives exist.
Review content and operational coherence.
Simulate classification. Examine profile confusion, not only averages.
Stress-test parameter and Q-matrix uncertainty.
Field-test where consequences justify it.
Feed bank gaps back into item development.

23. Classroom translation

A teacher does not need integer programming to use the design principle. Suppose a six-question fractions quiz is intended to separate fraction magnitude, equivalence and common-denominator work. If every question requires all three, the quiz may tell the teacher who succeeds overall while giving weak evidence about which prerequisite is missing.

Add deliberate contrast: one clean magnitude task, one equivalence task, one common-denominator task, then integrated tasks. The aim is not to fragment learning permanently. It is to collect enough discriminating evidence before deciding what to teach next.

24. Rainbolt-style missing-node scan: when assembly is the hidden problem

The missing node may be test assembly when the model looks sophisticated but diagnostic profiles are unstable; when the Q-matrix lists every attribute yet some pairs are never separated; when alternate forms produce noticeably different skill reports; when one attribute has far weaker accuracy than the rest; when the item bank is large but the optimiser repeatedly reports infeasibility; when a timed form becomes speeded; or when adding more items barely improves the distinction the test was designed to make.

25. Evidence and limits

Research on cognitive-diagnostic automated test assembly has developed genetic-algorithm, binary-programming and response-time-constrained approaches, and has been extended to multistage testing. These studies demonstrate that diagnostic objectives can be integrated into form construction under explicit constraints. They do not establish that any one objective function or algorithm is best for every educational use.

The strongest limitation is upstream: a test can only diagnose distinctions represented meaningfully by its attributes, items and response model. Assembly determines which available evidence is collected. It cannot rescue an incoherent skill theory or an item bank that lacks the necessary contrasts.

26. The return path

Return to the mathematics test that covered every chapter yet could not separate equation formation from equation solving.

The problem was not a shortage of questions. It was a shortage of the right contrasts. Once the test is assembled around the distinctions the report intends to make, each selected question earns its place not merely by being good in isolation, but by making a competing learner state more observable.

A diagnostic test is not a pile of diagnostic items. It is a deliberately assembled system of contrasts whose combined evidence must justify the skill distinctions printed on the report.

Research and further reading

eduKateSG Learning Node Series · 0225 · Previous: 0224 — How MAP and EAP Cognitive Diagnosis Work · Explore the How X Works Hub.