The Gold Standard of Experimental Design | Cross-Validated TMLE, Fold Construction and Finite-Sample Coverage

1. Start with the estimand and identification, not the acronym

Back to contents

Define the population, treatment, outcome, follow-up and contrast, then state the assumptions connecting that causal target to observed data. CV-TMLE is an estimator of a specified parameter; it cannot repair unmeasured confounding, interference, informative missingness or a target that changes during analysis. The efficient influence curve and nuisance functions must match the data structure and estimand.

2. Distinguish CV-TMLE from tuning cross-validation

Back to contents

Ordinary cross-validation often selects learners by prediction loss. Cross-validated TMLE uses training and validation folds in the estimation procedure so observations are evaluated with nuisance fits learned elsewhere. An analysis may use internal cross-validation to tune Super Learner inside each training set and outer folds for CV-TMLE. Reporting “five-fold cross-validation” without these roles is ambiguous.

3. Construct folds around independent sampling units

Back to contents

Every observation should enter validation once while its nuisance predictions come from complementary training data. Stratify folds when rare treatment or outcome levels would otherwise disappear, and keep clusters, households or repeated measurements together when dependence is at that level. Fold choices should be set before outcome-driven inspection. Leakage through preprocessing or tuning defeats the separation.

4. Choose pooled or fold-specific targeting deliberately

Back to contents

Implementations can target within each validation fold and average estimates, or stack out-of-fold predictions for a pooled targeting step. These constructions can agree asymptotically under suitable conditions yet differ in finite samples, parameter-bound behaviour and exposure to extreme weights. Name the implementation, software and averaging rule. Do not use the common CV-TMLE label to hide algorithmic differences.

5. Treat finite-sample coverage as an empirical question

Back to contents

Asymptotic normality depends on influence-function behaviour, nuisance convergence, second-order remainder control and overlap. With small samples or near-positivity, influence values can be skewed and fold estimates unstable. Recent simulation evidence supports CV-TMLE in some adaptive-learning settings, but no result guarantees nominal coverage for every learner library and data structure. Use design-specific simulations when stakes justify them.

6. Worked example: five folds and a rare programme group

Back to contents

A study estimates a training programme’s effect on employment in 240 workers, but only eighteen rural workers enrolled. Random folds place almost all rural enrolment in one validation set, producing extreme propensity predictions and a volatile targeting update. Stratified group-aware folds improve balance, yet repeated partitions still yield materially different intervals. The team restricts the supported target population, reports the original analysis as unstable and uses simulations calibrated to the observed overlap. Fold repair helped; changing the causal population addressed the deeper problem.

7. Repeat partitions to diagnose fold dependence

Back to contents

One random split can be unusually favourable or harmful. Repeating a prespecified fold construction with new seeds can reveal partition sensitivity and may be combined by a justified repeated cross-fit procedure. Simply choosing the split with the narrowest interval is selective inference. Report the distribution of point estimates, standard errors, maximum weights and fold-specific failures across repetitions.

8. Changed-context transfer check: cluster-randomised schools

Back to contents

Move from independent workers to schools randomised by cluster. The transferable principle is out-of-fold nuisance learning and targeting. The changed design makes the school, not pupil, the independent fold unit and may require a cluster-level influence curve or variance estimator. Splitting pupils from one school across folds leaks shared practice and can create artificial precision even when treatment itself was randomised.

9. Know the limits and common misconceptions

Back to contents

CV-TMLE does not fix positivity violations, a wrong outcome time, informative loss or a misspecified causal graph. More folds are not always better: training sets grow, but validation folds shrink and computation rises. Cross-fitting can relax particular Donsker requirements without erasing remainder conditions. Good prediction loss is not proof of a well-estimated treatment mechanism or valid interval.

10. Audit folds, learners, targeting and inference

Back to contents

Publish fold assignments or reproducible seeds, grouping and stratification rules, preprocessing boundaries, candidate learners, tuning, out-of-fold predictions, truncation, clever covariates, targeting coefficients, influence values and variance formula. Compare ordinary TMLE, CV-TMLE variants and a cross-fitted one-step estimator where appropriate. Show sensitivity to learner library, folds, overlap handling and repeated partitions.

11. Frequently asked questions

Back to contents

Is CV-TMLE just TMLE with Super Learner? No. Does it avoid every Donsker condition? It can avoid particular empirical-process restrictions under stated theory. How many folds should be used? There is no universal optimum. Should clusters be split? Usually folds should respect the independent sampling unit. Does better coverage in one simulation prove validity here? No; use assumptions and a design-relevant stress test.

12. The gold standard makes sample splitting part of the estimand pipeline

Back to contents

CV-TMLE is strongest when fold construction, nuisance learning, targeting and uncertainty are one transparent procedure. The gold standard separates training from evaluation, keeps dependent units together, challenges overlap and remainder conditions, and treats partition sensitivity as evidence. A credible interval comes from a defensible design and estimator—not from attaching cross-validation to an otherwise hidden workflow.