The Gold Standard of User Experience | Why Counterbalancing Does Not Erase Task Learning

Imagine two versions of a fictional community-centre booking service, A and B. Both let someone find a class and reserve a place. A research team wants to compare how readily people can complete that journey. Before timing anyone, it needs to say which journey matters: a first encounter, a return visit or a change from a familiar version to a replacement.

Those experiences contain different knowledge. Someone using B after learning A may already know that classes are grouped by venue, that a confirmation page follows payment and that the requested course has a particular name. Their performance on B is an experience of B with that history. Calling it simply “B performance” leaves the history invisible.

Write the intended question in one sentence. For this article’s first model, it is the difference in completion time between A and B without a previous exposure to either version of the particular task. Later we deliberately change that question. A repeated-use question is just as legitimate, but its answer should not quietly replace the first-use answer because the repeated session happened to be easier to arrange.

Contents · Next chapter

A useful session record needs more than design name and seconds. Keep the participant, task variant, presentation period and preceding exposure attached to the observation. In an AB sequence, the A observation comes first and B comes second. In BA, B comes first and A comes second. The second observation has a different history even when the task wording looks identical.

Suppose a participant finds a particular evening course on A, then is asked to find precisely the same course on B. On the second attempt, they may remember the answer before understanding B’s navigation. Their speed is real, but its meaning has changed. It may measure reuse of an answer as well as discovery through an interface.

This is why a blank browser window does not reset the participant. Removing cookies can reset some software state. It cannot remove knowledge from a person’s memory. Keep that distinction in the record before considering any analysis. A carefully preserved sequence lets someone later distinguish a design difficulty from a history-dependent result rather than trying to reconstruct the order from a final average.

Contents · Previous chapter · Next chapter

A period effect is associated with being in the first or second part of a session. Fatigue, settling into the research setting or becoming comfortable with the equipment could affect the second period under either order. A carryover effect depends on what happened in an earlier condition and how it affects the later condition.

For our arithmetic model, use p for a shared change in second-period time. Use cA for the additional effect on B after A, and cB for the additional effect on A after B. Negative values mean faster completion. These quantities are a chosen teaching decomposition, not directly observed labels that a stopwatch can identify independently.

The distinction lets us ask a sharper question. If every second attempt becomes twenty seconds faster regardless of the previous design, equal AB and BA groups can balance that particular effect. If A teaches a shortcut that B does not teach in return, the two sequences carry different benefits. Presenting each design once in each position does not make those benefits equal. We will calculate the remaining difference rather than assume that balanced positions settle it.

Contents · Previous chapter · Next chapter

With two designs, a simple counterbalanced arrangement gives AB to one group and BA to another. Assigning participants randomly to the two sequences can support a fairer comparison, while equal group sizes balance the number of first and second presentations. Neither step establishes that every participant has identical skills or that the two tasks are interchangeable.

I. Scott MacKenzie’s research note on within-subject and between-subject designs discusses counterbalancing and asymmetric skill transfer in human-computer interaction. Its useful starting point here is that ordering is a design decision with assumptions, rather than an automatic cure for all learning.

In the next calculations, the sequence groups have the same stipulated baseline and design effect. That keeps the example small enough to inspect. A real dataset includes participant variation, measurement variation and possibly missing observations. Random assignment does not make those disappear from the realised sample. The paper model demonstrates one reason a balanced mean can remain misleading even before those extra complications are added.

Contents · Previous chapter · Next chapter

Let μ be the stipulated first-use time on A. Let τ be B’s first-use time minus A’s first-use time, so a negative τ favours B for time. Add the shared second-period effect p and the preceding-design effects cA and cB. The four modelled times are A first = μ; B after A = μ + τ + p + cA; B first = μ + τ; A after B = μ + p + cB.

Check what each term means before calculating. cA belongs to the observation of B because it is the effect carried from A. It is easy to attach the subscript to the design currently being measured and reverse the conclusion. Read it aloud as “carryover from A”. The model assumes this additive representation for this exercise only.

There are four observed cell values but five named quantities. Without additional restrictions, the cell values cannot separately identify every component. A neat table therefore does not reveal a hidden pure design effect by itself. The next chapters show exactly what a common balanced comparison contains and which assumption would be needed to interpret it as τ.

Contents · Previous chapter · Next chapter

For the AB sequence, the within-person B-minus-A time difference is τ + p + cA. For BA, maintain the same B-minus-A direction even though B happened first. Its difference is τ − p − cB. Reversing the subtraction just because the order changed would turn a design comparison into a first-versus-second comparison.

Give the two sequences equal weight. Their average contrast is [(τ + p + cA) + (τ − p − cB)]/2 = τ + (cA − cB)/2. The shared period term cancels. The difference between the two carryover terms remains. This is an algebraic conclusion under the declared model, not a general theorem that all order effects are additive.

If cA equals cB, the remaining term is zero. Equality need not mean that nobody learned anything: both directions could carry the same benefit. Conversely, unequal transfer can remain even with exactly equal sequence counts. The mechanism matters. A statement that “order was balanced” describes the allocation; a statement that “the contrast estimates first-use design difference” also requires assumptions about what earlier exposure does.

Contents · Previous chapter · Next chapter

Start with μ = 120 seconds and τ = −10 seconds. B is stipulated to take 110 seconds on first use. Let the shared second-period effect be p = −20 seconds, with cA = cB = 0. The AB times are 120 then 90 seconds; the BA times are 110 then 100 seconds.

In AB, B minus A is −30 seconds. In BA, B minus A is +10 seconds. Equal weighting gives (−30 + 10)/2 = −10 seconds, matching τ. Someone reading only the AB group would exaggerate B’s advantage. Someone reading only the BA group could favour A. The balanced calculation succeeds for the particular shared-effect model we chose.

Notice what has and has not happened. Both second attempts are faster than their corresponding first-use baselines by twenty seconds. That learning-like period effect has not been removed from the participants. It has cancelled from one contrast because its sign reverses under the fixed B-minus-A subtraction. Explaining that cancellation is more informative than saying the study “controlled learning” without specifying which learning model supports the claim.

Contents · Previous chapter · Next chapter

Now keep μ = 120, τ = −10 and p = −20, but set cA = −30 and cB = −5 seconds. Perhaps A makes an otherwise obscure category structure very clear, and that knowledge helps a later attempt on B more than the reverse exposure helps A. This is a fictional mechanism chosen to illustrate the arithmetic.

The AB times become 120 and 60 seconds. The BA times become 110 and 95 seconds. The B-minus-A contrasts are −60 and +15, giving an equal-weight average of −22.5 seconds. The first-use difference remains the stipulated −10 seconds. The extra −12.5 comes from (−30 − (−5))/2.

A polished chart showing B ahead by 22.5 seconds could therefore overstate the intended first-use advantage in this model. It does not mean B’s observed sessions were fabricated or that the stopwatch failed. The chart describes a mixture of design and history. The repair is to preserve the sequence interpretation and investigate the question the design can support, rather than relabel the pooled number as a history-free property of the interface.

Contents · Previous chapter · Next chapter

The second example’s four times are 120, 60, 110 and 95 seconds in their stated cells. Their first-period difference is −10, but the decomposition of the second-period observations is not unique. B after A is fifty seconds faster than its first-use baseline, while A after B is twenty-five seconds faster than its own baseline.

One decomposition is p = −20, cA = −30 and cB = −5. Another is p = −10, cA = −40 and cB = −15. Both produce exactly the same four cells. The separate common-period and carryover values cannot be recovered merely by naming the columns. Their difference cA − cB is the same in these decompositions, but the individual mechanisms remain unmeasured.

This is a useful evidence check for learners. An algebraic symbol is not automatically an identified quantity. Ask what observation or assumption would distinguish two proposed explanations. Additional sessions, task variants or independent observations might inform a carefully designed study, but more records of the same indistinguishable arrangement do not automatically identify every causal component. Keep an explanation’s evidential status attached to it.

Contents · Previous chapter · Next chapter

Equal sequence counts were part of the cancellation. Suppose a fraction w of the complete paired records comes from AB and the rest from BA. The weighted contrast becomes τ + (2w − 1)p + w cA − (1 − w)cB. With w = 1/2, the earlier formula returns. With a different weight, the common period term also remains.

Use the shared-practice example with no extra carryover. If three quarters of the paired records are AB, the weighted result is −10 + (1/2)(−20) = −20 seconds. This differs from −10 even though both sequences were offered. Having some reversed-order sessions is not the same as a balanced analysis of the intended comparison.

Unequal weights can arise through recruitment, incomplete sessions or a deliberate allocation. The appropriate analysis depends on the question and design; automatically reweighting after seeing the outcomes can introduce its own problems. For a first check, show the numbers of participants and complete observations by sequence. Then state the chosen weighting rule. A reported mean needs that information before the reader can assess which cancellation argument, if any, applies.

Contents · Previous chapter · Next chapter

Suppose twelve people begin AB and twelve begin BA, but four BA participants do not complete the second task. The starting allocation is balanced; the complete-pair dataset is not. Dropping the incomplete records may also select participants whose experience differed from the completers’. Equal planned numbers do not establish equal information at analysis.

Record the reason and stage at which a session ended where it is appropriate to do so. Someone may need to stop for reasons unrelated to the design, or the task may be especially difficult. Those situations should not be silently treated as interchangeable. A missing time is not a zero-second completion, and a timed-out attempt is not an ordinary successful completion with a conveniently capped duration.

This article does not specify a missing-data estimator. Its purpose is to keep the design question intact. Report the flow from assignment to observed task outcomes, preserve the success definition and seek analysis appropriate to the actual study. The arithmetic model can reveal an imbalance, but it cannot repair unseen outcomes by inserting the number that would make an attractive chart look balanced again.

Contents · Previous chapter · Next chapter

Imagine the actual service will replace A with B for existing A users. Learning carried from A may now be part of the experience the team wants to understand. The question becomes how people accustomed to A manage the transition to B, rather than which design is faster without prior exposure.

In that setting, an AB observation is not automatically contaminated evidence. It may be relevant evidence for the intended transition, provided the earlier exposure resembles the users’ real familiarity and the study’s other conditions are appropriate. A participant given one short A task is not necessarily representative of someone who has used A every week for two years.

State the population and history explicitly. “B after one practice task on A” is a narrower claim than “B for all existing users”. The same recorded time can answer one question and fail to answer another. Good interpretation begins by matching the observation to the intended experience. It does not treat carryover as always harmful or always helpful; it asks whether that history belongs in the outcome being studied.

Contents · Previous chapter · Next chapter

A tempting repair is to give a different booking task on the second design. That avoids repeating the exact answer, but introduces task differences. Finding a widely advertised sports class may be easier than finding a specialised workshop hidden under an unfamiliar category. Different words alone do not make tasks comparable.

Suppose task X naturally takes twenty seconds less than task Y under either design. If A always receives Y and B always receives X, the task difference can look like a design benefit. Counterbalancing design order alone does not balance the task assignment. The record must preserve both design and task, along with their order and any relationship between them.

For a paper exercise, list the combinations AX, AY, BX and BY and ask which are actually observed. Then ask which comparisons hold the task fixed. A real plan may counterbalance task variants, use a suitable factorial arrangement or choose a different design. The important learner skill is to notice the second factor before averaging. A replacement task is useful only when its role in the comparison has been considered rather than assumed away.

Contents · Previous chapter · Next chapter

Task instructions can teach the route before the participant encounters the design. “Find an evening class” and “Open Activities, choose Evening and select Class 7” ask different things. The latter provides interface knowledge that the former leaves for the participant to discover. The amount of help belongs to the meaning of the result.

The GOV.UK guide to moderated usability testing recommends clear, relevant tasks that avoid supplying the answer. It also emphasises consistent session guidance. Those principles support our narrow concern: a comparison becomes harder to interpret when one condition receives more explanation or a later instruction reveals what an earlier task required the participant to discover.

Prepare the wording before collecting results and retain what was actually said when assistance matters. A moderator can help someone continue for a legitimate research purpose, but that intervention changes how a completion should be interpreted. Separate unaided success from assisted progress under the study’s defined rules. A person’s need for clarification is evidence about the service and task, not a reason to make them feel they have failed an examination.

Contents · Previous chapter · Next chapter

Our additive example uses successful completion times to make the order calculation visible. A real usability comparison may include unsuccessful attempts, errors and assistance. A design that lets more people finish could have a higher mean among successful users because it now includes people who previously could not complete the task.

Do not infer that a slower successful mean alone demonstrates a worse experience. Keep the success criterion, participant denominator and treatment of unfinished attempts visible. The User Experience subject guide develops this broader measurement distinction. The present branch adds the sequence history needed before comparing repeated observations.

For a changed example, let A have four successful users with short times and B have eight successes including several slower completions. The successful-user populations differ. Counterbalancing the order does not make those selected groups identical. Ask which outcome the team intended to compare and which records support it. This is a separate limitation from carryover, so correcting the order calculation cannot be claimed to resolve the whole measurement problem.

Contents · Previous chapter · Next chapter

In the unequal-transfer model, the AB contrast strongly favours B while the BA contrast favours A slightly. That pattern deserves inspection before the results are pooled. It suggests that order may matter, but it does not uniquely identify asymmetric transfer in a real sample. Participant differences, task imbalance or chance variation may also contribute.

Plot or tabulate the observations by sequence with their definitions intact. Look at what people actually did, including whether they remembered an answer, used a learned category or repeated an earlier error. Qualitative observations can help formulate a mechanism without turning a plausible narrative into a proven causal explanation.

A failure to detect an order difference statistically is not proof that carryover is absent. The study may contain little information about it. Conversely, a visible difference does not establish which repair is best without understanding the task. Use the pattern to review the design and its assumptions. An exploratory clue can be valuable when labelled honestly; it becomes misleading when promoted into a certainty merely because the pooled average is inconvenient.

Contents · Previous chapter · Next chapter

Giving each participant only one design avoids exposure to the other design within that study. This can be useful when the question is first use and the task reveals knowledge that cannot meaningfully be forgotten. The tradeoff is that comparisons now involve different participants, whose characteristics and experience need attention.

Random assignment can support a causal design under its conditions, but does not guarantee identical realised groups. Recruitment, task consistency, outcome definitions and analysis still matter. A between-person arrangement also answers a different repeated-use question from a within-person sequence. Choose it because it matches the intended experience and interference concern, rather than because it carries a general label of superiority.

For learners, compare two proposed plans. Plan One times the same person on the same hidden-answer task twice. Plan Two gives one version to each of independently assigned participants. Ask what knowledge can transfer in each plan and which sources of variation remain. A good answer explains both plans’ limits. It does not say that eliminating one source of carryover eliminates every source of uncertainty in the comparison.

Contents · Previous chapter · Next chapter

A pause may reduce fatigue or let a participant rest. It does not necessarily remove a remembered menu location, vocabulary item or answer. Calling the pause a washout period does not establish that the relevant knowledge has disappeared. The property must be justified for the mechanism under study.

The Experimental Design guide discusses persistent learning as a limitation of crossovers. In our booking case, asking someone to wait ten minutes after learning the course name cannot be assumed to restore a first encounter. The time interval is observable; erasure of the useful memory is a separate claim.

A more appropriate question may be how performance changes with repeated use. If so, retain the learning and model or describe it as part of the outcome. Alternatively, change the study design to address first use more directly. The practical lesson is to investigate the mechanism before assigning a reassuring label to a schedule. A well-rested participant can still be a well-informed participant, and those are different properties of the next observation.

Contents · Previous chapter · Next chapter

A fictional library compares catalogue interfaces A and B. Set μ = 100 seconds, τ = +8 seconds and p = −12 seconds. Let cA = −20 and cB = −4. Before continuing, calculate the four times and the equal-weight B-minus-A contrast. Positive τ means B is slower on first use in this stipulated model.

The AB sequence gives 100 then 76 seconds. The BA sequence gives 108 then 84 seconds. Their B-minus-A differences are −24 and +24, whose mean is zero. The model’s first-use difference is nevertheless +8. The remaining carryover contribution is (−20 + 4)/2 = −8, cancelling that difference in the pooled contrast.

Explain the result in words. A zero balanced contrast does not imply equal first-use times under every carryover model. Here it combines a slower first-use B with a stronger benefit flowing from A to B. A learner who can reproduce the earlier calculation but calls this result proof of equality has missed the interpretation. Repair the meaning of the subtraction before trying a more complicated statistical procedure.

Contents · Previous chapter · Next chapter

For a separate summary-only exercise, suppose a reviewer receives the balanced contrast without the four cell values. The reviewer considers an additional assumption: |cA − cB| is no greater than ten seconds. If the balanced contrast is −22.5 seconds, the model implies τ lies between −27.5 and −17.5 seconds. The carryover contribution can move the contrast by at most five seconds.

That interval comes from the stipulated bound, not from the observed times alone. It is not a confidence interval and has no claimed coverage probability. If the ten-second assumption is unsupported, the calculation cannot make it credible. Change the bound to thirty seconds and the compatible interval widens to −37.5 through −7.5.

This is a useful way to expose dependence on an assumption without pretending to estimate it. Ask what evidence could justify the proposed range and whether the conclusion changes under plausible alternatives. Keep sampling uncertainty separate; our exercise has treated the cell values as fixed. A real analysis would need its own account of variation and design, rather than attaching a sensitivity interval to the chart as if it covered every uncertainty at once.

Contents · Previous chapter · Next chapter

Consider the sentence “Twenty-four participants tested both versions in balanced order, and B was faster.” Ask what the report needs before that sentence supports a first-use claim. It should identify the tasks, order allocation, complete observations by sequence, success and assistance rules, timing boundary and the intended population of experience.

Then inspect the sequence-specific results. Does the report show whether the comparison reverses across AB and BA? Does it explain why learning from one version is expected to affect the other similarly, if that assumption is needed? A statement about random order is helpful, but it is not a substitute for this account.

Finally, trace the recommendation back to the question. A recommendation for experienced A users may legitimately rely on different evidence from a recommendation for first-time visitors. The audit should help the team make that distinction explicit. It should not demand a single perfect method for every project. The gold standard is a conclusion whose wording follows the experience actually studied, with the unresolved assumptions visible enough for another person to assess.

Contents · Previous chapter · Next chapter

If the algebra feels difficult, start with four labelled cards: A first, B after A, B first and A after B. Write the times from the shared-practice example on them. Subtract B minus A within each sequence, keeping the direction fixed. Only then average. This repairs the earliest connection between presentation order and comparison direction.

To stabilise understanding, change just p while leaving the other quantities fixed. Verify that the equal-weight contrast stays unchanged in the additive model. Next change only cA. The contrast should move by half that change. These small controlled variations help distinguish a cancellation from a claim that learning did not occur.

To extend, introduce unequal weights or a second task variant and describe which new assumption is required. Do not add every complication simultaneously. A learner shows transfer when they can name what has changed in the question and adjust the comparison accordingly. A page of correctly substituted formulas is less useful than one clear explanation of why a familiar average now includes a different history.

For an assessment analogue, the single-group equating guide examines the same candidates taking two forms. Its score-linking question differs from usability, but its attention to exposure history is useful. Do not import its conclusions as a ready-made analysis of website completion times; carry over the question about what the first attempt changes, then justify the study-specific method.

Contents · Previous chapter · Next chapter

A participant can contribute several task attempts without becoming several independent participants. If one person tests six tasks on each design, twelve timing records share that person’s history, skill and session conditions. Treating all twelve as unrelated observations can overstate how much independent information the study contains. Counterbalancing presentation order does not resolve that dependence.

For a paper audit, give each participant one colour and mark every record from that person. Then ask which comparison is within a person and which comparison is between sequence groups. The appropriate analysis must respect that structure and the study question. Our four-cell calculation is a stipulated illustration of contrasts, not permission to treat every task row as an independent experimental unit.

This also helps explain why a large row count can coexist with a small participant count. Report both with their meanings. A reader can then distinguish more repeated practice by the same people from broader evidence across different people. Keeping those counts separate protects the interpretation before any calculation of uncertainty begins.

Contents · Previous chapter · Next chapter

Does counterbalancing make a usability study pointless? No. It is a useful arrangement for distributing order and supporting comparisons under appropriate assumptions. The lesson is to specify which effects it balances and which histories the result represents. A method becomes more useful when its limits are understood accurately.

Should researchers always discard second attempts? No. Second attempts may be central to learning, repeat-use or transition questions. Discarding them after inspecting the results can also change the planned analysis. Define the intended experience first and use a justified design and interpretation for that experience.

Can a larger sample remove asymmetric transfer? More observations can improve information about a stated model. They do not automatically change the systematic quantity a pooled contrast contains. If the target and contrast differ because of carryover, repeating the same arrangement more often does not make the two quantities identical.

Is a small order difference proof that there is no carryover? No. A small observed difference has to be interpreted with the study’s information, variation and assumptions. The four-cell paper model provides an explanation of one limitation; it is not a significance test, a sample-size rule or a certificate that a real study is unbiased.

Contents · Previous chapter · Next chapter

For measurement and interpretation, return to User Experience for task success, assistance and the wider service journey. Use Experimental Design when the missing connection concerns assignment, repeated measures, carryover or causal claims. These guides ask related questions at different scales.

For the primary reading, MacKenzie’s research note, linked above, supports the distinction between repeated-measures arrangements and asymmetric transfer. The GOV.UK guide supports neutral tasks and consistent moderation. Sources were checked on 24 October 2026. The community-centre and library records, additive model, numerical cases and sensitivity exercise are original teaching constructions; neither source reports those fictional results.

For learning beyond this example, use The Gold Standard of Education: learning transfers into life and the How X Works hub. Carry the central question with you: what did the person already know before the measured attempt? Answering it makes an interface comparison easier to interpret and gives the next research decision a more precise starting point.

Contents · Previous chapter · Return to User Experience