The Gold Standard of Experimental Design | Alpha Recycling, Graphical Procedures and Multiple-Endpoint Decisions

1. Define the hypothesis family and claims first

Back to contents

List each null hypothesis, its estimand, endpoint, population, time point and the claim unlocked by rejection. Define the family-wise error rate and whether one or several families are involved. A graph cannot decide which findings deserve confirmatory status; it encodes that scientific and regulatory priority. Exploratory outcomes should not quietly enter the graph after their p-values are known.

2. Allocate the initial alpha budget

Back to contents

Assign each hypothesis a non-negative local significance level whose total does not exceed the one-sided or two-sided family alpha as designed. The initial weights express priority and power trade-offs. Equal allocation is simple but not automatically efficient. A tiny local alpha may make an important endpoint practically unreachable, while giving one node all alpha imposes a fixed-sequence bottleneck unless transfer routes are available.

3. Read nodes and edges as an algorithm

Back to contents

A node holds a hypothesis and its current local alpha. Directed edge weights specify the fraction transferred from a rejected node to remaining nodes. Outgoing weights must satisfy the procedure’s constraints, and the graph updates after each rejection. The same p-values can lead to different decisions under different graphs. A diagram is therefore executable design documentation, not an illustrative figure added after analysis.

4. Understand what alpha recycling does—and does not do

Back to contents

When a hypothesis is rejected, its available local alpha can be redistributed according to prespecified edges. This lets later hypotheses be tested more generously while preserving family-wise error control under the proven procedure. A non-rejected hypothesis does not release alpha merely because its result is disappointing. Recycling reuses error budget made available by a valid rejection; it does not manufacture new significance or rescue arbitrary claims.

5. Connect graphs to hierarchy, fallback and gatekeeping

Back to contents

Fixed-sequence, fallback and many gatekeeping strategies can be expressed graphically. A primary endpoint can pass alpha to a key secondary endpoint, while parallel co-priorities can share initial weight and transfer to each other. The graph should reflect which claims are meaningful after each pattern of success. If a secondary claim is scientifically incoherent without primary success, a permissive edge may be statistically valid yet substantively wrong.

6. Worked example: two doses and two endpoints

Back to contents

An experiment has efficacy hypotheses for high dose H1 and low dose H2, plus function endpoints H3 and H4 linked to those doses. The family alpha is 0.025 one-sided: H1 and H2 each start at 0.0125. If H1 is rejected, its alpha transfers to H3; if H2 is rejected, its alpha transfers to H4. H1 succeeds, H2 does not, so H3 is tested at 0.0125 and H4 receives nothing. H3 succeeds and its prespecified edge sends alpha to H2, which may then be retested at its updated level. The exact update order and weights must follow the validated algorithm, not intuition.

7. Simulate power across plausible effect patterns

Back to contents

A graph has no universally best shape. Simulate global null, all-effective, one-dose-effective, endpoint-specific and correlated-outcome scenarios. Report marginal power, probability of at least one claim, probability of the desired claim set and sensitivity to correlation and missingness. Optimising only for the most optimistic pattern can starve clinically important alternatives. Use simulation to compare scientific decision strategies, not to tune arrows after observing results.

8. Changed-context transfer check: an education programme

Back to contents

A school trial tests mathematics achievement, attendance and teacher implementation across two programme versions. The transferable method defines confirmatory claims, assigns local alpha and prespecifies transfer after rejection. The changed context may make attendance meaningful even without a score effect, so a clinical-style gatekeeping hierarchy should not be copied automatically. Stakeholder priorities and the estimands determine the graph; multiplicity mathematics then protects the error rate.

9. Know the limits and common misconceptions

Back to contents

Alpha is not a probability that a hypothesis is true and does not become a physical substance. A graph does not fix biased endpoints, selective missingness or an incoherent estimand. Retesting and transfer are valid only as specified by the procedure. Correlation may improve power for parametric variants but cannot be assumed without justification. Post-hoc graph changes invalidate confirmatory interpretation unless handled as a new analysis.

10. Audit the graph from protocol to result

Back to contents

Publish the hypothesis list, family definition, sidedness, initial weights, complete transition matrix, update algorithm, adjusted p-values or rejection path and software version. Check that outgoing weights and local levels remain valid after every possible rejection sequence. Reproduce the analysis independently and reconcile the graph with protocol, statistical analysis plan and final report. Show non-rejected nodes and unused paths, not only the successful route.

11. Frequently asked questions

Back to contents

Is recycling the same as alpha spending over time? No; recycling moves multiplicity budget across hypotheses after rejection, while spending handles repeated looks. Can alpha move after a failure? Not under ordinary graphical rejection rules unless the prespecified method says so. Are endpoints tested only once? Some graphical algorithms update and retest remaining hypotheses. Does a graph guarantee useful claims? It guarantees only its statistical property; scientific coherence must be designed. Can weights be changed later? Not for the same confirmatory analysis after results are known.

12. The gold standard turns arrows into accountable decisions

Back to contents

A credible graphical procedure begins with claims, assigns a complete error budget and makes every transfer conditional and reproducible. Its operating characteristics are tested before data are seen, and its final report follows the actual rejection path. The graph is valuable because it exposes priorities and dependencies. When every node, arrow and update has a reason, multiplicity control becomes part of scientific design rather than a last-minute correction.