Different subgroup results do not automatically mean different treatment effects
A treatment can work differently across baseline groups, but subgroup estimates are noisy and easy to overread. This guide shows how to compare effects directly, control the search and separate plausible heterogeneity from patterns created by chance.
Choose a reading route
- Start from first principles — define the unit and the threat.
- Follow the worked example — make the hidden arithmetic visible.
- Test transfer — change the setting and preserve the rule.
- Audit the evidence — inspect records, assumptions and failure points.
- Use the practical checklist — apply the method to new work.
Chapter contents
Foundations: Chapters 1–3
Method: Chapters 4–6
Transfer and limits: Chapters 7–9
Connections and use: Chapters 10–12
1. Heterogeneity is variation in effects
Treatment-effect heterogeneity means the causal contrast differs across levels of a baseline variable or across people. It is not demonstrated merely because one subgroup has a statistically significant result and another does not.
The relevant question is whether the treatment contrast changes across groups on a stated scale. That is an interaction question.
First principle
Compare effects with each other, not significance labels with each other.
2. Choose the effect scale before looking
A treatment may have a constant risk ratio but different risk differences when baseline risk varies. Absolute and relative heterogeneity therefore answer different questions. Predefine the estimand, outcome time and effect scale.
| Scale | Contrast | Use |
|---|---|---|
| Risk difference | 70% − 50% | Absolute benefit |
| Risk ratio | 70% ÷ 50% | Relative benefit |
| Mean difference | Average score gap | Continuous outcome |
| Hazard ratio | Instantaneous relative rate | Time-to-event model |
Use a scale aligned with the decision and present absolute risks where they help interpretation.
← Previous · Contents · Next →
3. Interaction tests compare contrasts
In a regression model, an interaction term tests whether the treatment effect differs by subgroup. For a binary subgroup, it estimates a difference in treatment effects on the model’s scale.
Subgroup-specific confidence intervals remain useful, but overlapping or non-overlapping intervals are not a substitute for a formal interaction estimate. Small subgroups often produce imprecise interactions even when point estimates look far apart.
- Use baseline variables defined before treatment.
- Include the main treatment and subgroup terms.
- Report the interaction estimate and uncertainty.
- Distinguish exploratory from confirmatory analyses.
← Previous · Contents · Next →
4. Worked example: compare the differences
A fictional trial reports success in 70% versus 50% for treatment and control in Group A, an absolute effect of 20 points. Group B reports 60% versus 50%, an effect of 10 points. The interaction on the risk-difference scale is 20 − 10 = 10 points.
| Group | Treatment | Control | Risk difference |
|---|---|---|---|
| A | 70% | 50% | +20 points |
| B | 60% | 50% | +10 points |
| Interaction | — | — | +10 points |
If each subgroup effect has a standard error near 7 points and estimates are independent, the interaction standard error is about √(7²+7²)=9.9 points. A 10-point interaction is then imprecise, despite the different point estimates.
← Previous · Contents · Next →
5. Multiplicity makes discoveries easier to manufacture
Each additional subgroup, cut point, outcome and model creates another chance for an extreme result. Under ten independent null tests at a 5% threshold, the chance of at least one false positive is 1 − 0.95¹⁰ ≈ 40.1%.
| Independent null tests | Chance of ≥1 false positive |
|---|---|
| 1 | 5.0% |
| 5 | 22.6% |
| 10 | 40.1% |
| 20 | 64.2% |
Prespecification, hierarchical testing, adjusted thresholds, shrinkage and validation reduce overclaiming. None turns a weakly measured subgroup into a reliable one.
← Previous · Contents · Next →
6. Plan, estimate and validate
Limit primary subgroup questions to variables with biological, social or operational rationale. Specify cut points, scale, model, missing-data handling and multiplicity strategy before outcome inspection. Preserve continuous variables when possible.
For exploratory work, use regularisation or partial pooling to stabilise estimates and seek replication in independent data. A subgroup claim should combine statistical evidence, prior plausibility, consistency and practical importance.
Practical check
Create a table of every subgroup analysis attempted, not only the striking ones. Mark prespecified, exploratory and externally validated results.
← Previous · Contents · Next →
7. Transfer test: an education programme
Move the design to a learning programme tested across students with lower and higher baseline attainment. Separate analyses may show significance only in the lower-attainment group because it is larger or more variable, not because the effects differ.
Transfer succeeds when the study tests the treatment-by-baseline interaction on a meaningful score scale, keeps baseline attainment continuous where possible and reports both absolute effects with uncertainty.
If subgroup membership is defined after treatment, it can introduce selection and no longer answers a simple baseline-effect-modification question.
← Previous · Contents · Next →
8. Misconceptions and limits
- “Significant here, not there, means different effects.” It does not test the difference.
- “A forest plot proves heterogeneity.” It visualises estimates; interaction evidence is separate.
- “Prespecified means true.” Planning limits bias but does not remove noise.
- “No interaction means identical effects.” Low power can miss real variation.
- “A machine-learning subgroup is self-validating.” Discovery still needs honest evaluation.
Many trials are powered for an overall effect, not interactions. Treat precision as central, not as a footnote.
← Previous · Contents · Next →
9. Evidence audit
Audit the subgroup rationale, baseline timing, cut points, all tested variables, outcomes, models, missingness, multiplicity plan and validation status. Compare the published report with the protocol and analysis plan.
| Audit question | Evidence | Warning sign |
|---|---|---|
| Was it planned? | Protocol and analysis plan | Outcome-driven subgroup |
| Was difference tested? | Interaction estimate | Separate p-values only |
| How many searches? | Complete analysis inventory | Selective forest plot |
| Is it stable? | Shrinkage or validation | Single small subgroup |
← Previous · Contents · Next →
10. Questions readers often ask
Should every subgroup be tested?
No. Prioritise a small set with clear rationale and decision value.
What if interaction is not significant?
Report the estimate and interval. The data may be compatible with both important heterogeneity and no difference.
Can continuous modifiers be categorised?
They can, but arbitrary cut points discard information and can create unstable patterns. Model continuous relationships where feasible.
← Previous · Contents · Next →
11. Reading routes and ecosystem connections
Begin with the Gold Standard of Education: learning that transfers into life, The Gold Standard and the How X Works hub. The subject foundation is the Gold Standard of Experimental Design.
Continue with guides on treatment fidelity and estimands, attrition and missing outcomes, cluster randomisation, and noncompliance and complier effects. The Gold Standard of Biostatistics and Gold Standard of Causal Inference provide supporting methods.
← Previous · Contents · Next →
12. Sources and practical checklist
Authoritative references include the European Medicines Agency’s Guideline on the Investigation of Subgroups in Confirmatory Clinical Trials, the U.S. Food and Drug Administration’s Multiple Endpoints in Clinical Trials guidance, and the Patient-Centered Outcomes Research Institute chapter Estimation and Reporting of Heterogeneity of Treatment Effects.
- Define the estimand and scale.
- Use baseline modifiers.
- Prespecify priority subgroups and cut points.
- Test interactions directly.
- Count every analysis attempted.
- Address multiplicity and missingness.
- Report uncertainty and seek validation.