The Gold Standard of User Experience | Separating Issue Frequency from Severity in Small Usability Studies

An issue needs an observable trigger, task context, evidence and outcome. Avoid labels such as confusing or bad navigation without the event that supports them. Two researchers can only count the same issue consistently when they share an operational definition.

Begin with a measurement ledger. Name the quantity, its unit, the sample or observation that produced it, the time window and the transformation rule. Mark each input as observed, stipulated, estimated or assumed. This prevents a familiar label from importing a new denominator or a stronger claim. Read the ledger as a sentence before calculating; if the sentence is incomplete, the arithmetic is premature.

Define Issue A as: on the account-recovery task, a participant selects the help link, returns to the same screen and cannot identify the next action within the session’s stated observation window. This is countable. The broader phrase participants dislike help is not established by that event.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to change the task while keeping the interface element fixed. Decide whether the original issue definition still applies. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Next chapter

Frequency is a count divided by the opportunities in which the issue could have appeared. The denominator may be participants, task attempts, sessions or eligible encounters. It must match the claim and exclude cases that never reached the trigger only when that exclusion is explicit and justified.

Keep the mechanism beside the number. Ask what physical, biological, statistical or strategic relationship connects the observation to the conclusion, and which conditions could break that link. A correct calculation can answer the wrong question when a changed setting alters the process. Draw one arrow for every inferential step, then challenge each arrow with a counterexample.

If 5 of 12 participants encounter Issue A, participant frequency is 5/12, about 41.7 per cent in this study. If only 8 reached the relevant step, the opportunity-based frequency is 5/8, or 62.5 per cent. Both calculations can be correct and answer different questions.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to add four participants who never reach the relevant step. Recalculate both frequencies and explain which one describes exposure. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

Severity concerns the consequence when an issue occurs: task failure, wrong action, lost work, delay, distress, inaccessible route or recoverable hesitation. Use observable evidence and a declared rubric. A numeral does not make the judgement objective unless the categories and evidence are visible.

Use an independent check that follows a different route. Reconstruct totals from components, compare a bound with a point estimate, inspect a distribution behind an average, or solve a changed case with the original definitions. Agreement does not prove every assumption, but disagreement locates a definition, unit or boundary that needs repair.

A project uses an ordinal four-level rubric: 1 is brief hesitation with independent recovery; 2 needs a detour; 3 prevents the intended route but leaves a workable alternative; 4 blocks an essential outcome or creates serious harm under the study context. This is a project rule, not a universal standard.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to move an issue from a low-stakes practice task to an essential identity-verification task. Explain why the consequence category may change. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

Multiplying an ordinal severity label by a frequency can create a convenient ranking, but the product has no automatic measurement meaning. Preserve the components and the evidence behind them. A decision can then weigh reach, consequence, reversibility and affected groups transparently.

Match the claim to the evidence. A tidy decimal cannot repair selective observation, and one convenient run cannot establish every product, participant, signal or opponent. Report the result, the conditions that make it interpretable, and the next observation needed for the actual decision. Separate the numerical conclusion from the practical conclusion.

Issue A affects 7 of 12 participants and receives severity 1. Issue B affects 2 of 12 and receives severity 4. Products 7 and 8 would rank B slightly higher, but those products depend on arbitrary coding and hide the difference between a blocker and a hesitation.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to renumber the same four severity categories as 1, 2, 4 and 8. Show how the multiplied ranking changes without any new observation. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

Build the evidence table from participant-task records before summarising. Mark whether the issue was possible, observed, recovered from, assisted, or ended the route. Keep missing observation distinct from no issue.

Begin with a measurement ledger. Name the quantity, its unit, the sample or observation that produced it, the time window and the transformation rule. Mark each input as observed, stipulated, estimated or assumed. This prevents a familiar label from importing a new denominator or a stronger claim. Read the ledger as a sentence before calculating; if the sentence is incomplete, the arithmetic is premature.

In a fictional study of 12 participants, A appears for 7 and all recover unaided after a short pause. B appears for 2 and both fail the essential task. C appears for 4 of 8 eligible mobile users; three recover after a detour and one stops. The rows support different frequency and consequence statements.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to reclassify one missing video record as unknown rather than no issue. Recalculate the affected denominator. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

Small usability studies are often designed to reveal mechanisms and barriers, not estimate a population percentage. Report study counts and conditions. Repetition across participants increases confidence that a problem is real in the tested context, but does not automatically supply a sampling margin of error.

Keep the mechanism beside the number. Ask what physical, biological, statistical or strategic relationship connects the observation to the conclusion, and which conditions could break that link. A correct calculation can answer the wrong question when a changed setting alters the process. Draw one arrow for every inferential step, then challenge each arrow with a counterexample.

Seven of twelve in this study is a descriptive study proportion. It is not evidence that 58.3 per cent of all users will encounter the issue. Recruitment, tasks, devices, assistance and prior experience shape what the study exposed.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to replace purposeful recruitment with a documented probability sample while leaving the sample size small. Identify which claims improve and which remain imprecise. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

An issue that affects a smaller access-needs group can be serious even when its overall study frequency is low. Do not dilute a subgroup-specific barrier by dividing it across participants who could never encounter it. Report the subgroup denominator and the whole-study context together.

Use an independent check that follows a different route. Reconstruct totals from components, compare a bound with a point estimate, inspect a distribution behind an average, or solve a changed case with the original definitions. Agreement does not prove every assumption, but disagreement locates a definition, unit or boundary that needs repair.

If 3 of 3 screen-reader users cannot reach a control while 0 of 9 other participants encounter that mechanism, the whole-study proportion is 3/12. The subgroup record is 3/3 among tested screen-reader users. Neither number estimates all users, but the second preserves the relevant opportunity set.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to add nine more participants outside the subgroup. Explain why the overall proportion falls while the barrier evidence does not disappear. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

Assistance changes what an outcome means. Record timing and content. The companion guide on unaided completion and recovery after a facilitator hint develops this chronology.

Match the claim to the evidence. A tidy decimal cannot repair selective observation, and one convenient run cannot establish every product, participant, signal or opponent. Report the result, the conditions that make it interpretable, and the next observation needed for the actual decision. Separate the numerical conclusion from the practical conclusion.

If a participant encounters Issue D, receives a task-revealing hint and completes, the issue was still observed. Completion with assistance does not erase its occurrence or convert the route into unaided success. It may reduce the observed consequence under the assisted study condition.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to replace the hint with planned service support available to every real user. Decide how the product claim changes. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

One participant can encounter several issues and still complete; another can fail after one blocker. Counting issues, people affected and failed tasks are different summaries. Preserve the many-to-many relationship rather than forcing totals to match.

Begin with a measurement ledger. Name the quantity, its unit, the sample or observation that produced it, the time window and the transformation rule. Mark each input as observed, stipulated, estimated or assumed. This prevents a familiar label from importing a new denominator or a stronger claim. Read the ledger as a sentence before calculating; if the sentence is incomplete, the arithmetic is premature.

Across 12 participants, researchers record 18 issue occurrences affecting 9 people, while 3 tasks end without completion. None of these totals is inconsistent. A count of 18 does not mean 18 participants or 18 failed tasks.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to merge repeated occurrences of the same issue within a participant. State which frequency question the new count answers. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

A clear quotation can explain an issue but should not substitute for the complete record. Dramatic moments are memorable and may receive too much weight. Link each illustrative quotation to the participant, task event and observed consequence, with privacy preserved.

Keep the mechanism beside the number. Ask what physical, biological, statistical or strategic relationship connects the observation to the conclusion, and which conditions could break that link. A correct calculation can answer the wrong question when a changed setting alters the process. Draw one arrow for every inferential step, then challenge each arrow with a counterexample.

One participant says this is impossible after a brief hesitation, then completes unaided. Another says nothing and loses entered work. If the report ranks by emotional language alone, it can invert the observable consequences.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to remove every quotation and check whether the evidence table still supports the severity judgement. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

Independent recovery, facilitator assistance, planned support and abandonment are distinct paths. Record them without describing participants as careless or incapable. The interface and service context are part of the mechanism.

Use an independent check that follows a different route. Reconstruct totals from components, compare a bound with a point estimate, inspect a distribution behind an average, or solve a changed case with the original definitions. Agreement does not prove every assumption, but disagreement locates a definition, unit or boundary that needs repair.

Issue E causes four participants to backtrack. Three recover using visible navigation; one uses the browser back button and loses a draft. The shared trigger can be counted together, while the different consequences remain separate.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to add autosave that prevents data loss. Explain why issue frequency may stay constant while severity falls. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

When participants perform similar tasks, later opportunities are not independent replicas of first use. Counterbalancing can distribute order but cannot erase learning. The existing guide on task learning and counterbalancing develops this limit.

Match the claim to the evidence. A tidy decimal cannot repair selective observation, and one convenient run cannot establish every product, participant, signal or opponent. Report the result, the conditions that make it interpretable, and the next observation needed for the actual decision. Separate the numerical conclusion from the practical conclusion.

If Issue F appears in 6 of 12 first attempts but 1 of 12 second attempts, combining them as 7 of 24 hides the order pattern. Report first and later opportunities separately before any pooled summary.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to reverse task order for half the participants and identify which comparison becomes possible. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

A participant who stops before reaching an issue opportunity is not evidence that the issue was absent. The companion article on task times with incomplete attempts shows why outcomes and observation horizons matter.

Begin with a measurement ledger. Name the quantity, its unit, the sample or observation that produced it, the time window and the transformation rule. Mark each input as observed, stipulated, estimated or assumed. This prevents a familiar label from importing a new denominator or a stronger claim. Read the ledger as a sentence before calculating; if the sentence is incomplete, the arithmetic is premature.

Four participants stop on an earlier screen, leaving eight eligible for Issue C. Recording C as 4/12 understates occurrence among observed opportunities; recording 4/8 without mentioning the earlier stops hides a different barrier. Report both layers.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to repair the earlier barrier so all twelve reach C. Explain why a later issue can appear more frequent after an improvement. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

A matrix can organise frequency, consequence, affected groups, reversibility, confidence, implementation dependency and legal or policy duties. It should expose judgement, not automate it. Keep urgent duties and safety-critical barriers from being averaged away.

Keep the mechanism beside the number. Ask what physical, biological, statistical or strategic relationship connects the observation to the conclusion, and which conditions could break that link. A correct calculation can answer the wrong question when a changed setting alters the process. Draw one arrow for every inferential step, then challenge each arrow with a counterexample.

The team marks Issue A high frequency, low immediate consequence; B low frequency, essential-task blocker; and C subgroup-specific with incomplete evidence. The meeting records why B receives immediate action, C receives urgent inclusive investigation and A enters the next iteration.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to change the release deadline without changing evidence. Identify which priority may shift and which duty should not. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

Independent coding followed by evidence-based reconciliation reveals ambiguous definitions. Agreement is useful, but forced consensus can hide uncertainty. Preserve the original observations and note unresolved classification differences.

Use an independent check that follows a different route. Reconstruct totals from components, compare a bound with a point estimate, inspect a distribution behind an average, or solve a changed case with the original definitions. Agreement does not prove every assumption, but disagreement locates a definition, unit or boundary that needs repair.

Two researchers rate Issue G as 2 and 3. They discover that one counted a visible alternate route as recovery while the other required the intended route. Updating the rubric clarifies the disagreement more than averaging the scores to 2.5.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to give both researchers the same rubric but different video segments. Explain why data access, not judgement, is now the first repair. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

A before-and-after comparison needs compatible tasks, opportunity rules and assistance. A lower count can reflect a changed sample or fewer opportunities. Preserve version, device, content and study conditions.

Match the claim to the evidence. A tidy decimal cannot repair selective observation, and one convenient run cannot establish every product, participant, signal or opponent. Report the result, the conditions that make it interpretable, and the next observation needed for the actual decision. Separate the numerical conclusion from the practical conclusion.

Issue H appears in 5 of 10 eligible attempts before a change and 1 of 9 after. The descriptive proportions are 50 per cent and about 11.1 per cent. The small study supports a promising conditional observation, not a guaranteed population effect.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to change the post-test task so the trigger appears only half as often. Explain why the frequency comparison becomes confounded. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

Small counts can change sharply when one record changes. Use counts beside percentages and describe the sample. Avoid significance language unless the design and analysis support it. A range of plausible interpretations is more honest than a decimal-heavy verdict.

Begin with a measurement ledger. Name the quantity, its unit, the sample or observation that produced it, the time window and the transformation rule. Mark each input as observed, stipulated, estimated or assumed. This prevents a familiar label from importing a new denominator or a stronger claim. Read the ledger as a sentence before calculating; if the sentence is incomplete, the arithmetic is premature.

For 2 of 12, the displayed percentage is 16.7. If one ambiguous case is reclassified, it becomes 3 of 12, or 25.0. Showing both counts makes the sensitivity visible; three decimal places would not make the study more stable.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to reclassify one case in each direction and describe the resulting decision sensitivity. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

A useful issue record contains the trigger, observation, affected task, consequence, recovery, opportunity denominator, participants or sessions, platform, version and evidence reference. Assign an owner for investigation without pretending the owner is the cause.

Keep the mechanism beside the number. Ask what physical, biological, statistical or strategic relationship connects the observation to the conclusion, and which conditions could break that link. A correct calculation can answer the wrong question when a changed setting alters the process. Draw one arrow for every inferential step, then challenge each arrow with a counterexample.

The record for Issue B links two session timestamps, the task definition and the lost-work event. It names the product area that can test a fix. It does not publish identifying video or turn a participant’s behaviour into a defect label.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to remove the evidence reference and ask whether another team could verify the finding. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

The NIST Industry Usability Reporting material explains shared reporting structures, while the GOV.UK moderated usability testing guide grounds task-based observation. NHS service guidance on accessible user research helps preserve access needs.

Use an independent check that follows a different route. Reconstruct totals from components, compare a bound with a point estimate, inspect a distribution behind an average, or solve a changed case with the original definitions. Agreement does not prove every assumption, but disagreement locates a definition, unit or boundary that needs repair.

These sources support transparent context, participants, tasks and evidence. They do not prescribe one universal numeric severity formula. This article’s worked rubric is explicitly a teaching device and must not be presented as an ISO scale.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to replace one source with an informal blog and state which claim now needs stronger support. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

Do not say frequent means severe, rare means negligible, a score is objective, completion means no issue, or a small study estimates everyone. Each error collapses a separate evidence dimension. Repair it by restoring the denominator, consequence and study context.

Match the claim to the evidence. A tidy decimal cannot repair selective observation, and one convenient run cannot establish every product, participant, signal or opponent. Report the result, the conditions that make it interpretable, and the next observation needed for the actual decision. Separate the numerical conclusion from the practical conclusion.

When an issue appears once but causes unrecoverable data loss, write once in twelve tested participants, with one essential-task failure under these conditions. That sentence is more useful than rare issue or severity four alone.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to rewrite one misleading dashboard label as a complete evidence sentence. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

Before prioritising, verify issue definition, opportunity set, affected subgroup, occurrences, people affected, task outcomes, assistance, recovery, version, platform, evidence links and coding uncertainty. After prioritising, record the reasoning and the observation that would change it.

Begin with a measurement ledger. Name the quantity, its unit, the sample or observation that produced it, the time window and the transformation rule. Mark each input as observed, stipulated, estimated or assumed. This prevents a familiar label from importing a new denominator or a stronger claim. Read the ledger as a sentence before calculating; if the sentence is incomplete, the arithmetic is premature.

A checklist reveals that Issue C’s denominator mixed mobile and desktop attempts even though only mobile exposed the control. The repair is to reconstruct the eligible set, not merely change the chart colour.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to deliberately use the whole-study denominator, then show the distortion and correct it. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Next chapter

Return to the User Experience root for journey, task and metric foundations. Use the verified branches for assistance chronology, incomplete task times and task-learning limits.

Keep the mechanism beside the number. Ask what physical, biological, statistical or strategic relationship connects the observation to the conclusion, and which conditions could break that link. A correct calculation can answer the wrong question when a changed setting alters the process. Draw one arrow for every inferential step, then challenge each arrow with a counterexample.

The transferable habit is to separate reach from consequence. It works for accessibility barriers, service incidents, classroom misconceptions and operational defects whenever one count cannot carry every decision.

Before moving on, record the observation that would distinguish the two most plausible explanations. If no feasible observation exists, narrow the claim instead of pretending the ambiguity has disappeared.

A changed-context transfer check is to apply the two-column method to a non-digital service and state its opportunity denominator and consequence evidence. State what would change the result and what the example cannot establish. That limit belongs inside the answer.

Contents · Previous chapter · Return to User Experience