Two competent assessors can watch the same medical-device simulation and reach different judgements. The gold standard is not declaring the more senior assessor correct. It is a fair, evidence-based review that preserves safety requirements and identifies what the clinician can actually do independently.
The disagreement that matters
During an approved simulated monitoring scenario, a participant recognises an abnormal alert and eventually escalates it. One assessor marks the performance satisfactory because the final action was appropriate. Another marks it unsatisfactory because a required verification step was omitted. The dispute cannot be resolved by averaging two scores. First, the team must establish which actions the approved competency standard requires, what the participant actually did and whether the omitted step is consequential to safety.
Separate four sources of disagreement
There may be an evidence disagreement about what occurred; a criteria disagreement about what the standard requires; a context disagreement about what the scenario permitted; or an interpretation disagreement about the clinical importance of an observed action. Each has a different remedy. Replaying a video may help the first; consulting approved requirements may resolve the second; scenario validation may be needed for the third.
Protect current clinical responsibilities
A simulation review is not a substitute for professional credentialing or authorisation. Where competence has not been demonstrated for safety-critical use, the relevant institution should follow its established supervision and practice restrictions. The review should not assume that a contested pass is automatic permission to use a device independently, nor should it make an unsupported judgement about a professional’s overall ability.
Preserve observations before debate
Where the training environment allows, record the actions, sequence, instructions and relevant scenario state. Assessors should describe what they observed before arguing about scores. Distinguish an action that was clearly omitted from one that may have occurred outside the visible assessment window. A reliable record protects both patient-safety review and the participant’s right to fair consideration.
Check whether the assessment itself was valid
Did the simulation use the approved device configuration and current procedures? Were instructions clear, equipment functional and the scenario suitable for the role? A confusing simulation can generate assessor disagreement even when the intended competency is well defined. If the exercise was invalid, the appropriate response may be to correct the scenario and reassess, not to award a compromise result.
Use an impartial, authorised resolution route
A practical pathway can include independent review of the evidence, comparison with approved criteria, a moderated discussion and escalation to the responsible clinical education or governance authority if required. A majority vote should not override a mandatory safety check. Record the reasoning behind the decision, not merely which assessor prevailed.
Provide a fair route for the participant
The learner should understand which observable performance is under review and have an appropriate opportunity to clarify or respond under institutional procedure. Avoid public embarrassment and unsupported accusations. Feedback should describe the competency to demonstrate next. An assessment system is more credible when it can correct its own judgement and still maintain clear standards.
Reassessment should answer the disputed question
If disagreement remains because the original scenario did not establish independent competence, a suitable new simulation can test the particular task. The new exercise should be relevant but not depend on memorising exactly what happened the first time. The learner should demonstrate the required decision, verification and escalation independently, under current approved practice.
An applied break-test: the critical check
Suppose one assessor says a missed device identity verification step is a minor omission and another says it is mandatory. The resolution team should locate the current institutional procedure and confirm how it applies to the simulated task. If the step is required, it cannot be waived merely because the participant was otherwise quick. Conversely, the team should not invent requirements absent from the approved competency standard.
Metrics that improve assessment quality
Review the rate and subject of assessor disagreement, repeat disagreements on the same criterion, time to resolution, successful remediation and any changes needed to scenario design. A very low disagreement rate is not automatically evidence of quality; it may reflect weak scrutiny or reluctance to challenge a colleague. The valuable outcome is consistent, defensible assessment aligned with patient safety.
What this teaches a hospital
Assessor disagreements are signals about criteria, training and system design as well as individual competency. When an issue repeats, update examples, clarify descriptors, retrain assessors or repair the scenario. Do not turn every dispute into a personal confrontation. The gold standard makes uncertainty visible, resolves it with evidence and preserves confidence that important clinical tasks are evaluated seriously.
Sources and connected knowledge
Follow applicable hospital governance, device instructions and professional requirements, including relevant information from Singapore HSA Medical Devices. Related eduKate articles cover Assessor Calibration, Assessor Drift Monitoring, Scenario Validation and Drill Evidence Moderation. This article addresses the decision process when a specific disagreement has already occurred.