Find the question you need
See the treatment gap · Work through the numbers · Choose the effect question · Transfer the method · Sources and connected reading
An experiment can randomise one treatment, deliver another and analyse a third version created by exclusions. This guide keeps the treatment question and the effect being estimated aligned.
Chapter index
Chapters 1–3
1. The intervention on paper is not always the intervention delivered · 2. Define fidelity before measuring it · 3. Work the assignment-based result
Chapters 4–6
4. See why a per-protocol comparison changes the population · 5. Classify deviations by mechanism and timing · 6. Choose the treatment effect before choosing the exclusion rule
Chapters 7–9
7. Do not hide contamination inside one adherence percentage · 8. Fidelity evidence belongs beside outcome evidence · 9. Changed-context transfer: from greenhouse pots to a tutoring programme
Chapters 10–12
10. Common misconceptions and limits · 11. A practical fidelity and estimand audit · 12. Sources, connected reading and the next useful move
CHAPTER 1 OF 12
1. The intervention on paper is not always the intervention delivered
Imagine a fictional greenhouse experiment testing whether Nutrient N increases four-week seedling growth. Forty similar pots are randomly assigned: 20 to Nutrient N and 20 to water-only control.
The protocol says treated pots receive 100 millilitres twice each week. During the study, four treated pots miss one or more doses, and three control pots receive Nutrient N once because of a tray-label error.
Random assignment remains a fact about the design. Treatment received is now more complicated. The experiment must decide which effect question it wants to answer and how the deviations enter that question.
The example is educational, not agricultural advice or a real trial. Its purpose is to distinguish assignment, delivery, receipt and analysis.
Treatment fidelity is the degree to which an intervention is delivered as intended. It can include dose, timing, duration, content, provider behaviour and prohibited co-interventions.
A useful fidelity measure is specified before outcomes are known. “Received enough treatment” should not be invented after seeing which plants grew best.
Record both delivery and receipt. A technician may dispense the correct volume while a blocked dripper prevents the pot from receiving it. Conversely, a pot may receive extra nutrient from spillover even though its assigned procedure was followed.
Fidelity is not one universal percentage. Each critical component should be defined and linked to the causal question.
Suppose final mean growth is 8.4 centimetres among the 20 pots assigned to Nutrient N and 5.2 centimetres among the 20 assigned to control. The assignment-based mean difference is 3.2 centimetres.
This comparison preserves the randomised groups. It estimates the effect of being assigned to the Nutrient N strategy under the delivery conditions that actually occurred in this fictional experiment, including missed doses and accidental exposure.
It does not isolate the biological effect of receiving every planned dose. The label “effect of perfect Nutrient N adherence” would exceed what the comparison directly estimates.
Uncertainty, baseline balance and missing outcomes would also matter in a real analysis. The simple means keep the estimand distinction visible.
| Analysis group | Pots | Mean growth |
|---|---|---|
| Assigned Nutrient N | 20 | 8.4 cm |
| Assigned control | 20 | 5.2 cm |
| Assignment-based difference | 3.2 cm |
Now define protocol adherence as receiving every scheduled assigned procedure with no cross-contamination. Sixteen treated pots and seventeen control pots meet that rule.
Suppose their mean growth is 9.0 and 5.0 centimetres, a difference of 4.0 centimetres. This number is larger than the 3.2-centimetre assignment-based difference.
The difference is not automatically a better estimate. Excluding deviating pots breaks the original randomised comparison if adherence is related to tray position, plant condition or technician behaviour. The retained groups may differ for reasons other than treatment.
A per-protocol analysis can answer a useful question when designed and adjusted appropriately. It should not be presented as though simple exclusion preserves all benefits of randomisation.
A protocol deviation is a departure from the planned design or conduct. Examples include wrong dose, wrong timing, ineligible units, unplanned co-intervention, contamination, missing measurements and analysis changes.
Record when the deviation occurred and whether it could depend on the outcome. A tray-label swap before growth measurement differs from excluding a plant because its result looked unusual.
Distinguish deviations from amendments. A prospective, documented protocol change made without outcome-driven knowledge is different from an undocumented retrospective rule.
Severity should be tied to the question. A five-minute timing shift may be irrelevant for one treatment and critical for another.
The assignment-based question asks about a policy or strategy: what happens when units are assigned to Nutrient N under realistic implementation? A hypothetical question may ask what would have happened if every pot had followed the protocol.
A while-on-treatment question uses outcomes observed before discontinuation, which changes the time window. A composite strategy can build an intercurrent event into the outcome definition. A principal-stratum question concerns a subgroup defined by potential behaviour under assignments and is usually difficult to identify.
These labels come from the ICH E9(R1) estimand framework for clinical trials. The greenhouse analogy borrows the reasoning discipline, not the regulatory application.
The gold standard is to state population, treatment conditions, outcome, handling of intercurrent events and summary measure so the reported effect has one interpretable meaning.
Missed doses and control contamination act differently. Missed treatment may reduce treatment contrast; control exposure can move the comparison group toward the active condition.
Report counts and timing by assigned group. If contamination clusters by tray, technician or date, examine whether the implementation system created dependence.
A single “90% compliant” headline can conceal asymmetric problems. Twelve minor timing deviations are not equivalent to three control units receiving the active treatment.
Sensitivity analyses should vary defensible assumptions, not search for the version that produces the preferred result.
Use checklists, dose logs, timestamps, instrument records or independent observation appropriate to the intervention. Validate that the fidelity measure itself is reliable.
Monitoring can alter behaviour. Staff may follow the protocol more closely when observed. Record the monitoring plan and avoid treating monitored sessions as automatically representative.
Separate process outcomes from final outcomes. A tutor may deliver every planned explanation while students engage differently; a machine may run the programmed cycle while actual temperature drifts.
The purpose of fidelity measurement is to interpret the intervention, not to blame participants or operators for every deviation.
CHAPTER 9 OF 12
9. Changed-context transfer: from greenhouse pots to a tutoring programme
Move the logic to a fictional six-week tutoring programme. Students are assigned to a worked-example sequence, but some sessions are shortened, some students miss lessons and one class receives additional practice.
What stays the same is the distinction between assignment, delivery, receipt and engagement. What changes is the treatment itself: education involves interaction, adaptation and learner agency rather than a fixed liquid dose.
An assignment-based effect can estimate the result of offering the programme under those conditions. A fidelity analysis can show which components were delivered. It cannot reduce teaching quality to attendance alone.
Transfer succeeds when the intervention theory is rewritten for the new context and the effect question remains explicit.
Randomisation does not guarantee perfect delivery. It protects the assignment comparison under its assumptions; it does not erase post-assignment events.
Excluding every deviator is not automatically conservative. It can create selection bias and answer an unclear subgroup question.
High fidelity does not prove effectiveness. A precisely delivered ineffective intervention remains ineffective. Low fidelity can make a null result hard to interpret but does not prove the treatment would have worked.
The estimand framework clarifies the target effect; it does not solve every identification, missing-data or measurement problem.
Write the intervention’s critical components, permitted adaptations and prohibited contamination before outcome analysis.
Create a deviation log with unit, group, timing, type, cause and possible consequence. Keep operational notes separate from outcome-driven judgements.
State the primary effect question in one sentence. Then verify that inclusion, outcome timing and statistical comparison answer that sentence.
Present the assignment-based result and any secondary adherence-focused analysis with distinct labels, assumptions and limits.
| Layer | Record | Question |
|---|---|---|
| Assignment | Randomised group | What strategy was assigned? |
| Delivery | Dose, timing and content | What was actually provided? |
| Receipt | Exposure and engagement | What reached the unit? |
| Deviation | Type, time and cause | Could it change comparison or selection? |
| Estimand | Population, treatment, outcome, summary | What effect does the number mean? |
The ICH E9(R1) Addendum on Estimands and Sensitivity Analysis provides the formal framework for treatment-effect questions and intercurrent events. The ICH E8(R1) General Considerations for Clinical Studies addresses quality by design, while the ICH E6(R3) Good Clinical Practice guideline discusses protocol conduct and justified analysis deviations. These are clinical-research standards; the greenhouse and tutoring examples are explanatory transfers.
Continue through The Gold Standard of Experimental Design, The Gold Standard of Scientific Method, The Gold Standard of Biostatistics and How Research Bias Works.
Return to The Gold Standard of Education: Learning That Transfers into Life, The Gold Standard hub and How X Works.
The next useful move is to place the planned intervention, delivered intervention and received intervention in three adjacent columns. Any mismatch belongs in the effect question before it belongs in an exclusion rule.