The Gold Standard of Experimental Design | Selective Reporting Across Learner Libraries, Tuning Grids and Fold Seeds

1. Define the object being compared

Back to contents

Decide whether the claim concerns an algorithm in principle, a fully specified implementation, a tuning procedure or an end-to-end pipeline. Two packages using the same learner name may differ in defaults, stopping rules, regularisation scaling, missing-value handling and numerical precision.

The comparison unit should include preprocessing, search space, budget and selection rule. Otherwise the winning “algorithm” may simply have received a wider grid or a more favourable default.

2. List every adaptive choice before interpreting the score

Back to contents

Create a decision register for learner families, features, transformations, hyperparameter ranges, metrics, resampling scheme, fold count, grouping, seeds, early stopping and exclusion rules. Mark which were fixed in advance and which changed after inspecting results.

Exploration is legitimate. The reporting problem arises when exploratory multiplicity disappears from the final narrative. Preserve unsuccessful and neutral branches sufficiently to show the search that produced the headline.

3. Keep model selection separate from performance assessment

Back to contents

Using the same cross-validation results to choose hyperparameters and report performance creates optimistic bias. Place the entire tuning procedure inside an outer evaluation loop, or use a genuinely untouched test set when the design supports it. The outer unit must respect clusters, people, sites or time.

Nested evaluation estimates the performance of the selection procedure, not the best inner-fold score. Refit choices may use all outer-training data, but the outer test fold must remain unseen until assessment.

4. Make learner libraries comparable

Back to contents

Record package names, versions, wrappers, defaults and computational budgets. Harmonise what can be harmonised—data splits, metrics and eligible information—without erasing meaningful implementation differences. If one library searches fifty configurations and another five hundred, report the budget as part of the intervention.

Defaults are hyperparameters chosen by somebody else, not neutral baselines. Compare both well-documented defaults and an equally resourced tuning protocol when each answers a useful question.

5. Design tuning grids as scientific inputs

Back to contents

Explain parameter ranges, scale and resolution. A linear grid for a parameter spanning orders of magnitude can under-search small values; a hand-refined range after viewing test results is leakage. Conditional parameters and early-stopping budgets also shape which models can win.

Use pilot or prior evidence to set the grid, then lock it for evaluation. If the grid changes, label a new analysis and reassess on untouched data. Report boundary winners because they show that the search space, not only the learner, determined the result.

6. Worked example: the winning classifier moves with the seed

Back to contents

A school-risk dataset is analysed with three libraries, four learners and two tuning grids. One five-fold split makes a boosted tree lead logistic regression by 0.012 AUC. Across twenty prespecified fold seeds, the mean difference is 0.003 and the ranking reverses nine times. The broad grid helps the boosted tree only because it contains a lower learning rate absent from the narrow grid.

The final report presents the distribution of outer-fold differences, the learner–grid interaction and computational cost. It concludes that the data do not establish a stable winner, though several pipelines perform similarly. A single attractive leaderboard would have overstated the evidence.

7. Treat fold assignment as a design variable

Back to contents

Different valid partitions change training composition and which observations test each model. Small, imbalanced or clustered data can make this variation substantial. Generate seeds independently of outcomes, use the same outer splits for paired comparisons and repeat the whole pipeline across a prespecified set.

Do not search seeds and retain the one with the strongest result. Report seed-level estimates or an uncertainty summary that preserves dependence among methods evaluated on common splits.

8. Changed-context transfer check: a time-series forecast

Back to contents

Move from independent pupils to monthly demand forecasting. Random folds leak future information and break temporal dependence. The transferable rule is still to separate tuning from assessment and expose design variability. The changed design uses rolling or blocked origins, realistic forecast horizons and preprocessing fitted only on past data.

Repeat across several origin schedules rather than random seeds alone. A learner that wins only during one unusual economic period should not be described as universally superior.

9. Analyse the multiverse, not just the champion

Back to contents

Summarise performance across libraries, grids, splits and metrics with paired differences, distributions and rank stability. Predefine primary choices while using the wider multiverse to reveal sensitivity. When many comparisons are inferential, address multiplicity or frame them as exploratory.

Do not average away a structural interaction. If one library changes conclusions through a default or preprocessing convention, show that mechanism. Stability means substantive conclusions persist across reasonable choices, not that every score is identical.

10. Audit compute, failures and stopping

Back to contents

Retain configuration files, environments, random-number settings, fold indices, logs, failed runs, timeouts and hardware details that affect results. Excluding failed configurations after seeing poor performance can favour fragile methods. Define how errors and budget exhaustion are counted.

Publish enough code and manifests to reconstruct the selection path. Deterministic replay of one seed is useful, but scientific reproducibility also asks whether the conclusion survives legitimate random variation.

11. Frequently asked questions

Back to contents

Is one fixed seed reproducible? It is replayable, but it does not reveal seed variability. Must every possible learner be tried? No; justify a bounded library. Can nested cross-validation remove all bias? No; it helps separate tuning from evaluation but still inherits data and design limitations. Should the best outer-fold model be refitted? Refit according to the prespecified deployment rule, not by peeking at outer test outcomes.

12. The gold standard reports the procedure that generated the winner

Back to contents

A machine-learning result belongs to a search procedure: eligible pipelines, tuning budget, fold construction, random variation and selection rule. Define that procedure, protect its evaluation data and show the distribution of conclusions across defensible implementations.

The gold standard does not forbid optimisation. It prevents optimisation from masquerading as one fixed confirmatory test. Readers can then judge whether an apparent winner reflects stable evidence, a consequential default or a lucky path through the analytical garden.