EDUKATE SINGAPORE · GOLD STANDARD PRACTICAL CASE STUDY 171
A correct answer must stay correct when the answer choices move.
A fictional school quizzes Secondary 1 pupils on algebra substitution. A correct response worth eighteen appears at D after shuffling, but an old letter-based scorer expects B. This Gold Standard case explains stable choice identity, correct arithmetic and fair automated marking.
All students, providers, results and incidents are invented. Official 1EdTech examples supply technical context, not an allegation against a real school or platform.
Quick answer
- Correct value: 18.
- Original letter: B.
- Shuffled letter: D.
- Failure: the grade checks original position rather than the selected answer identity.
- Immediate response: preserve versions, verify affected outcomes and apply authorised academic review.
Chapter navigation
- 1. The algebra is correct but the quiz says wrong
- 2. What the answer identifier needs to preserve
- 3. The complete worked error in both directions
- 4. 1EdTech QTI has stable identifiers for shuffled choices
- 5. A synthetic twelve-attempt check has a bounded result
- 6. Equivalent mathematical work must receive equivalent scoring
- 7. Correct the decision through an authorised educational route
- 8. An assessment dashboard needs more than completed submissions
- 9. Teach first principles before letter memorisation
- 10. Practice: original B, shuffled D
1. The algebra is correct but the quiz says wrong
A fictional Secondary 1 maths centre lets pupils answer online questions whose answer options are shuffled separately on different screens. The author verifies that 3(x + 4) at x = 2 is 18, and the answer key initially identifies B because 18 is listed there. The delivery engine then rearranges choices without changing the old letter stored by the marking service.
Farah sees A = 24, B = 12, C = 6 and D = 18. She calculates 18 and selects D. The platform marks her wrong. Another pupil selecting displayed B would choose 12 yet could receive the old key’s credit. The numerical answer did not change; the relationship between a screen letter and an underlying choice did.
The Gold Standard requires the centre to verify what a student actually saw and selected before declaring a grade valid. A correct question, a working interface and a successful submitted response do not together establish correct scoring.
2. What the answer identifier needs to preserve
A choice is a mathematical meaning and an internal response identity before it is a displayed position. In the original authoring order A = 12, B = 18, C = 6 and D = 24. In the shuffled order the correct value 18 appears at D. The scorer should compare the stable choice identity against the authorised correct choice, not reuse an earlier visual position.
Useful evidence includes item identifier, item version, original and displayed permutations, submitted choice, scoring rule and actual grade. If one of those links is missing, a future review may not establish what an individual submission meant. A file containing only the letter D without the displayed permutation can be insufficient for reconstructing the student-facing event.
GS-R02 describes the acceptance criteria and GS-R07 addresses the software handoff. The basic test is whether the option representing 18 remains correct regardless of its location on screen.
3. The complete worked error in both directions
The expression is 3(x + 4). Substituting x = 2 gives 3(2 + 4) = 3 × 6 = 18. The options 12, 18, 6 and 24 are possible responses, but only 18 is mathematically correct for the specified expression and value.
The old authoring view associates that result with B; the shuffled view associates it with D. Checking the selected letter against B instead of checking the selected choice’s identity can reject the correct D response and credit the incorrect B response. This is not a matter of exam technique or the pupil failing to remember where an answer used to be.
A good acceptance test selects the correct and incorrect values under several different permutations and checks the recorded score. A single test in the original authoring order would miss the defect while reporting perfect accuracy for that particular view.
4. 1EdTech QTI has stable identifiers for shuffled choices
The 1EdTech QTI 3 Beginner’s Guide describes response declarations and response-processing rules in which correct choices are identified by their stable identifiers. The QTI 3 information model also describes the shuffle property of choice interactions.
An official 1EdTech example sets shuffle to true while retaining stable correct-response identifiers for scoring. The design makes it possible for presentation order to change without changing the identity of the correct answer.
These are authoritative specification examples, not a claim that the fictional platform complies with QTI or that a real assessment vendor failed its standards. The principle is widely useful: the display transformation and response-processing logic must agree about what each selected option means.
5. A synthetic twelve-attempt check has a bounded result
The quality team constructs twelve test attempts with known intended answers and permitted choice permutations. Five receive a result inconsistent with the selected mathematical values. Five out of twelve is approximately 41.7% among those deliberately selected tests.
The percentage does not establish that 41.7% of pupils in the actual programme received wrong marks. The synthetic cases were selected to reveal the suspected defect; they are not a random sample of learner behaviour. Nor does a correct result in seven test attempts establish the rest of the platform is sound.
The accepted conclusion is that the test demonstrated a real scoring problem in specified circumstances. An authorised impact assessment must identify which item versions, students and scoring events were affected before historic results can be revised.
6. Equivalent mathematical work must receive equivalent scoring
Two pupils may calculate 18 independently under the same expression. If one sees B = 18 and another D = 18, equal mathematical understanding should not be graded differently because of an interface permutation. Different orderings exist to change the presentation, not to change the assessment construct.
The maths tutor should also acknowledge that a multiple-choice tick is limited evidence of understanding. A learner could guess 18 correctly or use an unclear method. Where the educational goal concerns algebraic structure, showing working and explaining the balance or substitution steps can provide information beyond a choice.
This case is therefore both software assurance and teaching. The system must not introduce unfairness into the limited evidence it collects, while tutors must not treat a single score as the whole judgement of the learner’s abilities.
7. Correct the decision through an authorised educational route
The centre’s assessment owner preserves the relevant item versions, correct answers, response records, permutations if available and original grades. The software team can fix the mapping, but the authority to adjust academic results belongs to the institution’s actual grading and appeals process.
A quick patch does not reconstruct information that was never recorded. If the student’s displayed choices cannot be recovered reliably, the institution should state that evidentiary limit and apply its authorised procedure rather than guess which response a person selected. Students should not be blamed for software behaviour they did not control.
GS-R03 assigns authority, GS-R09 manages the exception and GS-R10 supports effective correction. A family can receive a truthful update that scoring is under review without being told an invented final mark.
8. An assessment dashboard needs more than completed submissions
The centre reports that forty pupils logged in and all forty submitted an answer. Those figures can be true even while the scoring engine is wrong. Reporting a 100% submission completion rate as evidence of 100% fair marking would silently enlarge the claim.
Separate statuses should cover item delivery, response receipt, stable answer identity, scoring acceptance and authorised results release. A software release that changes question imports, shuffle behaviour or grading rules should prompt proportionate integrated retesting. A passing old test is not a certificate for a newly altered code path.
GS-R08 requires that every statistic retain its proper denominator and meaning. A useful dashboard shows when a technically successful submission still has an unresolved grade.
9. Teach first principles before letter memorisation
One teacher-led demonstration places the same algebra question above four removable cards. Students find the value eighteen first. The teacher rearranges the cards and asks whether eighteen changed. They can see that the numerical result is stable while the letter has moved.
This approach matches the emphasis on structural understanding in eduKate’s Secondary 1 Mathematics Clementi reference. A pupil should know why 3(x + 4) at x = 2 equals eighteen rather than rely on memorising which option label looked right.
An assessment engineer faces the equivalent task: protect the underlying answer identity through changes in representation. Meaning must come before a label, whether the person making the decision is a student or a software service.
10. Practice: original B, shuffled D
Exercise. The original order is A = 12, B = 18, C = 6, D = 24. The shuffled order shown to a student is A = 24, B = 12, C = 6, D = 18. They evaluate 3(x + 4) at x = 2 and select D, but the original letter-B key marks the answer wrong.
The supported finding is that the student selected the correct value eighteen in the displayed version. A letter comparison with the original B is invalid after shuffling; it can also falsely reward a student who selects twelve in the new B position.
The acceptance test must capture the actual displayed order and underlying choice identifiers, submit both correct and incorrect answers under several permutations and verify that legitimate scores do not change with temporary screen letters.
Frequently asked questions
Does shuffled choice order change the correct maths value?
No. It changes where the value is displayed, not what the algebra expression evaluates to.
Can the defect reward an incorrect answer?
Yes. A wrong value appearing at an old correct letter can be credited by the faulty position-based scoring rule.
Can twelve test attempts estimate the error rate for the whole centre?
No. They establish performance in those test conditions, not the prevalence in the untested learner population.
Should every original grade be recalculated automatically?
Only an authorised review can decide how to handle affected records under the actual evidence and assessment rules.
What does QTI contribute?
It provides a model of identifiable choices and response-processing rules so presentation changes need not alter the scoring meaning.
What can a parent ask?
Ask whether the displayed item, student’s actual selection and grading rule have been reviewed together, rather than relying only on a system status that says submitted.
Acceptance checks before a score is trusted
- The mathematical source and expected answer have an accepted version.
- The rendered answer permutations and stable option identities agree.
- The submitted response can be mapped to the actual presented choice.
- Known correct and incorrect values are tested under multiple shuffles.
- Changes to item imports and scoring logic trigger an integrated retest.
- An authorised academic reviewer handles any disputed historical grades.
First principles: a position and a value are different kinds of information
Imagine four labelled cards on a desk. One card says eighteen. Before the cards move, it happens to be at position B. After a student shuffles the cards, it stands at position D. The number eighteen remains eighteen. A system that expects the card to stay at B is confusing a location with the object’s identity. This is the simplest way to see why a shuffled online test needs a stable choice identifier. Students already use this distinction in mathematics: changing how an expression is displayed should not change the value of a correctly evaluated expression.
The teacher can also explain why 3(x + 4) is not necessarily the same as 3x + 4. Distributing three across the bracket gives 3x + 12; at x = 2 this equals 6 + 12 = 18. Alternatively, substituting first gives 3 × 6 = 18. Two valid mathematical routes arrive at the same result. A multiple-choice programme should assess that meaning, not reward the accident of which letter represented eighteen during authoring.
An audit ledger that can reconstruct the learner’s view
The assessment lead can maintain a controlled evidence record with the item identifier and revision, the set of answer choices and their stable identities, the permutation shown to each synthetic learner, the selection captured by the browser and the response processed by the grading service. If the browser transmits only the displayed index, an authorised mapping step has to resolve that index against the exact presented order before the scorer determines whether it matches the correct choice. Every transition needs a consistent interpretation.
A complete record does not require exposing every child’s personal information to software developers. Test accounts and synthetic submissions can establish the defect, while restricted academic records support legitimate real-student review where necessary. The institution should not make a public example of a named pupil to prove its software was faulty. Privacy and grading integrity are complementary responsibilities: keep enough information to decide accurately, and make it available only to the people permitted to use it.
Why both a false negative and a false positive matter
If a pupil who chose eighteen is marked wrong, the record understates demonstrated competence on that question. If a pupil who chose twelve is marked correct, the record overstates that particular response. These two effects can offset in an aggregate class average, making the mean score appear stable even while individual marks are unfair. A manager cannot infer that the bug is harmless because the average class score changed little. Accuracy must be examined at the level of the submitted choice and the assessment decision.
The same concern applies to statistics that describe which distractor is most popular. After a shuffle, a data report that treats the displayed letter B as one fixed answer can mingle choices of twelve and eighteen into one category. Teachers may then draw the wrong conclusion about whether pupils struggle with brackets or substitution. Correct answer mapping is therefore necessary not only for marks but also for interpreting diagnostic evidence and planning the next lesson.
A controlled scoring test should challenge every relevant permutation
A suitable test can generate several allowed shuffles for the fictional item, choose each option once in each view and verify the response’s stable identifier against the expected score. That test demonstrates both that the correct value receives credit and that the other values do not. The verification should include any relevant fixed-position options, partial-credit methods or accessible reading modes supported by the actual item type. The goal is coverage of the deployed semantics, not merely a high count of automated test executions.
One important failure mode is an item-import process that reorders choices but does not update the local cache that associates original letters with values. Another is a scoring service that receives a raw displayed index without the submitted rendering context. These are plausible design risks, not conclusions about the fictional vendor’s precise code. A competent technical review identifies which mechanism caused the observed mismatch before choosing the corrective implementation.
A fair review of older grades has its own limits
If the application stored the actual order presented to every pupil, an academic reviewer may be able to reconstruct what the submitted letter meant under the corresponding item version. If that record was not retained, merely knowing that a learner submitted D may not establish whether D referred to eighteen or another value on the day. Any remedy then requires an authorised assessment decision that accounts for missing evidence. The centre must not invent a reconstructed permutation because a family needs a quick answer.
The repair plan should distinguish future software readiness, correction of already identified cases and uncertainty about other historical records. A passing patch test proves the corrected code works under its tested conditions; it does not retroactively establish every earlier score. The reporting system should preserve an audit trail of the old rule, new rule, review owner and decisions so a parent or educator can understand what has genuinely changed without exposing unrelated learners.
How a parent can check that the learning objective is still protected
A parent should be able to ask whether the original maths item had a verified correct value, whether the actual shuffled choice shown to the learner was preserved, and whether the marking decision was compared with that choice’s identity. The answers do not need to include source code or private data. A useful explanation can say that the expression evaluated to eighteen and that the question’s option position was not correctly mapped under a particular tested version.
The next lesson can still be productive while the grading review proceeds. The tutor can revisit substitution, equivalent expressions and checking solutions by a second method. This reinforces conceptual understanding and avoids letting a software problem become a label attached to the child’s ability. The wider Gold Standard requirement is that teaching decisions rest on genuine evidence of what students understand, not a technology error that turns a correct value into an incorrect letter.
A final publication standard for the quiz provider
A release report should say which versions, item types, permutations and scoring outcomes were tested, which cases remain outside the test scope, and who accepted the result. It should not call the whole assessment bank ‘100% verified’ merely because one demonstration question now scores properly. New content, accessible interfaces and grading configuration changes can introduce new cases requiring proportionate evaluation.
The same standard applies to public claims of fairness. It is legitimate to report that online delivery and submission succeeded for the class while a separate scoring correction remains in progress. That may be less impressive than a uniform green dashboard, but it gives tutors and parents a workable path from the mathematical question to an authorised educational decision. The Gold Standard is a reliable chain from meaning to measurement to action.
What a correct result means at each stage
For the teacher, the correct result means that the value obtained from substituting x = 2 into 3(x + 4) is eighteen. For the quiz renderer, it means that an identifiable choice whose value is eighteen must be displayed at an accepted position. For the browser’s response-capture process, it means that selecting that choice retains its identity even if it occupies a different screen position. For the marking service, it means that the submitted identity is compared with the authorised mathematical answer and the correct score recorded. The word ‘correct’ is therefore tied to a stage and a clear test. A supplier that demonstrates only rendering correctness has not yet demonstrated the scoring stage.
This distinction is especially important when an online assessment has several technology owners. A content author may manage questions, an accessibility team may manage the displayed interface, a vendor may process the quiz, and a teacher may review marks. None should assume the next component preserves meaning without evidence. The accepted interface between systems must identify the option in a way both sides interpret consistently. Otherwise, a harmless-looking reorder of answer cards can have the same practical effect as changing the answer key after pupils have submitted their work.
Accessible presentation must preserve answer identities too
Some students may use a keyboard, screen reader, enlarged text or another legitimate interface to navigate an online quiz. An accessible version might present choices sequentially or announce labels differently from the visual layout. The same stable answer identity must work across those presentations. If a visual interface scores the option correctly but a keyboard or assistive rendering records a different index, the centre has not demonstrated equal assessment meaning across supported routes. Technical accessibility checks and educational scoring checks therefore reinforce one another.
A useful synthetic test has a learner select eighteen using the mouse in one variant, then the equivalent choice using a supported keyboard interaction under the same item. The scorer should identify the same underlying answer. This is not a claim that every real screen reader or quiz format behaves identically, but a reminder that the acceptance conditions should include the interfaces the provider actually promises. A genuine Gold Standard assessment is not correct only when a particular student’s screen happens to match the author’s preview.
Do not let a class average conceal individual marking errors
Imagine five students whose correct eighteen responses are scored as wrong and five others whose incorrect values are credited because a stale letter matches. The total number of marks awarded to the group might be unchanged in that simplified scenario, even though ten individual outcomes are unreliable. A manager looking only at the class average could report that the scoring update made no difference. That would miss the fairness concern: a grade belongs to the student’s actual response, not to the net difference across the class.
For that reason, the reconciliation should compare each affected response against its authorised item version where evidence permits. A summary of averages or total submissions is insufficient for correcting individual records. The school also needs a way to record unresolved cases separately rather than infer that every low score was caused by the defect. The Gold Standard protects both legitimate achievement and legitimate uncertainty; neither can be manufactured from a balanced spreadsheet.
A post-repair assessment should be usable for tutoring decisions
After the software and academic review, tutors can examine which students genuinely understood the substitution step and which still require help with brackets. A pupil who selected twelve may have omitted a multiplication or interpreted the expression differently; that possibility is a teaching hypothesis to explore, not a confirmed diagnosis from one wrong choice. A pupil who selected eighteen can be asked to explain the reasoning, allowing the tutor to distinguish understanding from guessing. Assessment software is most valuable when its corrected data support those sensible next questions.
A short parent update can explain that the question and answer mapping were checked, the published score is now based on the version actually presented, and any remaining uncertainty has been handled under the centre’s procedures. The exact response should not invent a new grade or promise outcomes without evidence. The purpose of the repair is not simply to make a technology dashboard green: it is to make a student’s next lesson and academic record reflect what the student actually did.
Final acceptance question
Before this assessment can be called ready, an authorised reviewer should be able to answer: ‘If I deliberately move the correct option from B to D, can the platform still mark the content rather than the old position, and can I reconstruct that result from its records?’ A confident yes should be demonstrated through a controlled working test, not an assurance that the randomisation feature is popular or that all pages load. That single decision criterion connects technical correctness, educational fairness and the first-principles mathematical understanding illustrated throughout the case.
References and connected Gold Standard cases
- 1EdTech QTI 3 Beginner’s Guide
- 1EdTech QTI Information Model
- 1EdTech Official Shuffled-Choice Example
- Clementi Secondary 1 Mathematics Reference
- Gold Standard Information and Record Integrity
- Gold Standard Core Requirements
- Case 97 · Rubric Version Conflict
- Gold Standard Casebook
1EdTech specifications support assessment interoperability but do not certify the fictional provider. Grades and remedies follow actual school policies. GS-R01–GS-R12 is eduKate’s educational management framework.