Why English? | Writing an A/B Test Experiment Plan Through Hypothesis, Control, Variation, Metrics, Sample, Duration, Analysis and Decision Rules

ORIENTATION

What this guide covers

Current fact check

Checked on 8 November 2026, GOV.UK A/B-testing guidance said teams should identify a problem, construct a hypothesis, create control and variation, decide sample size and duration, randomly split groups and analyse results. The October 2026 Test, Learn and Grow Playbook said teams should plan learning, success measures, data capture, review and decisions before testing.

What this guide does not cover

This article owns the A/B test experiment plan for one defined comparison. It does not own laboratory experimental design, multivariate testing, clinical trials, usability testing, general analytics, product roadmaps, ethics review or final implementation approval.

Worked example

Every chapter returns to a fictional booking service comparing its current reminder with a clearer reminder that states collection deadline and cancellation route. It is fictional, low-stakes and educational; it does not report a real school, clinical decision or guaranteed outcome.

Warm up with eduKate’s guides to explaining scientific evidence and asking better questions in class.

CHAPTER 1 OF 16

1. Start With the Decision the Test Must Inform

Back to contents

Why this move matters

Name the product or service choice, decision-maker and deadline before defining the experiment.

Keep the causal claim bounded

Name the product or service choice, decision-maker and deadline before defining the experiment. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, The team must decide whether to replace the current reminder for eligible bookings. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

A test result informs rather than automatically makes the decision. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Start With the Decision the Test Must Inform”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 2 OF 16

2. Describe the Problem With Baseline Evidence

Back to contents

Why this move matters

Use dated service data and research to explain what is not working and for whom.

Keep the causal claim bounded

Use dated service data and research to explain what is not working and for whom. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, Some reservations expire uncollected, but records do not yet prove that reminder wording is the cause. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

Correlation in the baseline is not causal proof. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Describe the Problem With Baseline Evidence”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 3 OF 16

3. Write a Testable Hypothesis and Mechanism

Back to contents

Why this move matters

State the change, expected outcome and reason in one bounded causal claim.

Keep the causal claim bounded

State the change, expected outcome and reason in one bounded causal claim. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, A deadline-and-cancellation reminder may increase timely collection because students can act before expiry. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

The mechanism is a theory that the experiment may not directly prove. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Write a Testable Hypothesis and Mechanism”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 4 OF 16

4. Define the Unit of Assignment and Analysis

Back to contents

Why this move matters

Specify whether randomisation and measurement occur by person, booking, class, site or time period.

Keep the causal claim bounded

Specify whether randomisation and measurement occur by person, booking, class, site or time period. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, Each eligible booking is assigned once, with rules preventing the same booking from entering both groups. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

Repeated participation can create dependence that analysis must address. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Define the Unit of Assignment and Analysis”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 5 OF 16

5. Keep the Control Stable and Understandable

Back to contents

Why this move matters

Describe the current experience, version and any simultaneous changes that could affect it.

Keep the causal claim bounded

Describe the current experience, version and any simultaneous changes that could affect it. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, Version A is the existing reminder sent at the current time through the same channel. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

A changing control weakens interpretation. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Keep the Control Stable and Understandable”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 6 OF 16

6. Specify One Coherent Variation

Back to contents

Why this move matters

Write the exact changed elements and keep unrelated design, timing and delivery conditions constant where possible.

Keep the causal claim bounded

Write the exact changed elements and keep unrelated design, timing and delivery conditions constant where possible. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, Version B changes deadline and cancellation wording but not send time or eligibility. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

Bundled changes make it harder to identify what mattered. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Specify One Coherent Variation”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 7 OF 16

7. Define Eligibility, Exclusions and Recruitment

Back to contents

Why this move matters

State who can enter, who must not, consent or notification requirements and how repeat exposure is handled.

Keep the causal claim bounded

State who can enter, who must not, consent or notification requirements and how repeat exposure is handled. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, Only ordinary reservations with approved contact permission are eligible; urgent accessibility arrangements are excluded from randomisation. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

Exclusion can limit transfer and create fairness concerns. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Define Eligibility, Exclusions and Recruitment”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 8 OF 16

8. Plan Random Allocation and Exposure Integrity

Back to contents

Why this move matters

Describe assignment, balance, concealment where relevant and checks that participants receive the intended version.

Keep the causal claim bounded

Describe assignment, balance, concealment where relevant and checks that participants receive the intended version. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, The service assigns equally at random and logs delivery failures separately. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

Randomisation can fail through implementation or attrition. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Plan Random Allocation and Exposure Integrity”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 9 OF 16

9. Choose One Primary Outcome and Time Window

Back to contents

Why this move matters

Name the main outcome, numerator, denominator, measurement source and deadline.

Keep the causal claim bounded

Name the main outcome, numerator, denominator, measurement source and deadline. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, The primary outcome is timely collection among delivered eligible reminders within the stated window. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

Changing the primary outcome after seeing results inflates confidence. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Choose One Primary Outcome and Time Window”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 10 OF 16

10. Add Guardrails and Diagnostic Measures

Back to contents

Why this move matters

Track harms, burden, accessibility, cancellations, support contact and delivery failure without replacing the primary outcome.

Keep the causal claim bounded

Track harms, burden, accessibility, cancellations, support contact and delivery failure without replacing the primary outcome. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, Guardrails include complaints, inaccessible message reports and mistaken cancellations. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

More metrics increase opportunities for accidental stories. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Add Guardrails and Diagnostic Measures”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 11 OF 16

11. Plan Sample Size, Duration and Stopping

Back to contents

Why this move matters

Use expected baseline, effect, traffic, power or precision, seasonality and practical constraints with specialist input.

Keep the causal claim bounded

Use expected baseline, effect, traffic, power or precision, seasonality and practical constraints with specialist input. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, The test runs long enough to cover normal weekly patterns and uses a pre-agreed analysis point. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

Stopping early because a chart looks favourable can mislead. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Plan Sample Size, Duration and Stopping”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 12 OF 16

12. Protect Data Quality, Privacy and Ethics

Back to contents

Why this move matters

Define event logging, identity controls, minimisation, access, retention and escalation for harm or anomaly.

Keep the causal claim bounded

Define event logging, identity controls, minimisation, access, retention and escalation for harm or anomaly. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, The dataset stores the minimum booking and delivery fields needed for analysis. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

A low-risk message test can still affect real people. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Protect Data Quality, Privacy and Ethics”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 13 OF 16

13. Write the Analysis and Decision Rule in Advance

Back to contents

Why this move matters

State comparison, uncertainty, exclusions, missing data treatment and thresholds for adopt, reject, retest or investigate.

Keep the causal claim bounded

State comparison, uncertainty, exclusions, missing data treatment and thresholds for adopt, reject, retest or investigate. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, The team adopts only if the primary outcome improves without breaching guardrails and the uncertainty is acceptable. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

Statistical significance alone does not establish practical value. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Write the Analysis and Decision Rule in Advance”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 14 OF 16

14. Interpret Results With Uncertainty and Context

Back to contents

Why this move matters

Report effect size, uncertainty, actual exposure, anomalies, guardrails and limits of the tested setting.

Keep the causal claim bounded

Report effect size, uncertainty, actual exposure, anomalies, guardrails and limits of the tested setting. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, The variation improves collection modestly, while delivery failures remain unchanged and the confidence interval is wide. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

An inconclusive result is not proof of no effect. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Interpret Results With Uncertainty and Context”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 15 OF 16

15. Combine Quantitative Results With User Research

Back to contents

Why this move matters

Use interviews, support evidence or observation to explain comprehension and unintended effects without rewriting the causal estimate.

Keep the causal claim bounded

Use interviews, support evidence or observation to explain comprehension and unintended effects without rewriting the causal estimate. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, Students say the deadline is clearer, while some still misunderstand the cancellation route. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

Qualitative feedback is not a substitute for randomised outcome analysis. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Combine Quantitative Results With User Research”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

CHAPTER 16 OF 16

16. Record the Decision, Rollout and Next Learning

Back to contents

Why this move matters

State implementation, monitoring, rollback, segments not covered and follow-up questions.

Keep the causal claim bounded

State implementation, monitoring, rollback, segments not covered and follow-up questions. State exactly what changes between control and variation, which outcome may move and which population and time window the test covers.

Run the worked experiment on paper first

For the fictional service, The team rolls out gradually, monitors guardrails and keeps a rollback route for delivery errors. Check eligibility, allocation, exposure, measurement, stopping and analysis before any participant is enrolled.

Protect against convenient interpretation

Write the primary metric, guardrails, exclusions and decision rule in advance. Record inconclusive or harmful results as learning rather than searching for a friendlier metric.

Connect evidence to a proportionate decision

A successful test does not guarantee permanent performance. The experiment can inform a change under its tested conditions; it cannot prove every mechanism, audience or future outcome.

Student, parent and career transfer

Make one evidence card for “Record the Decision, Rollout and Next Learning”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports product analytics, digital services, communication, marketing, policy testing, user experience, research, school computing and evidence-based improvement without promising a grade, admission, funding decision or job.

PRACTICE LAB

Build a A/B test experiment plan decision trail

Student route

1. Choose one original report inside this guide’s scope.
2. Copy its objective and underline actor, action, context, evidence and intended decision.
3. Draw the specific reasoning chain used in this article.
4. Mark each step reported, inferred, contested or unknown.
5. Rewrite the conclusion so it preserves the narrowest important limitation.

Parent or teacher route

Ask the learner to explain turn a promising change into a fair, measurable experiment whose result can guide a real decision without overstating certainty twice: once to a classmate and once to a method specialist. Praise a precise correction and a well-stated unknown; both show disciplined attention to evidence.

Early-career route

Create a one-page A/B test experiment plan audit with source link, date checked, scope, method logic, decisive wording, contrary evidence, limit and next action. A colleague should be able to continue without guessing.

Continue with eduKate’s guides to reading course and career information and giving useful feedback in group projects.

USEFUL NEXT READING

Connect this narrow owner to its neighbour

Read the usability-test-plan guide for a related but different reader job. Return here when the question is specifically how to turn a promising change into a fair, measurable experiment whose result can guide a real decision without overstating certainty. Use each guide for the reader task named in its heading.

For English practice across research, school and career communication, visit the Singapore English Tuition Centre learning site.

FREQUENTLY ASKED QUESTIONS

Questions students and parents often ask

Is A/B testing the same as usability testing?

No. A/B testing compares measured outcomes between versions; usability testing investigates how people understand and use a design.

Must groups be random?

Random allocation is central to a clean A/B comparison; seek specialist advice for designs where it is not possible.

How many users are enough?

It depends on baseline, expected effect, variability, desired certainty and design. Plan with statistical expertise.

Can we stop when one version looks ahead?

Use pre-agreed stopping rules; early fluctuations can mislead.

Does statistical significance mean the change matters?

Not necessarily. Consider effect size, uncertainty, practical value and guardrails.

Can students be experimented on without review?

No. Follow current ethics, consent, privacy and institutional requirements.