Choose your reading route
eduKateSG · Why Mathematics? · Double Bootstrap: Nested Resampling and Calibration
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 8–14
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 15–21
Check assumptions
See where design and calibration matter.Route 4 · Chapters 22–28
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 29–34
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important when one bootstrap layer still leaves a confidence level or bias correction imperfectly calibrated? The double bootstrap nests a second resampling experiment inside each first-level resample. It uses that inner evidence to estimate how well the outer procedure is calibrated, then adjusts the final decision.
Did You Know? A full double bootstrap with B outer samples and C inner samples may require about B times C inner statistic evaluations. A modest choice of 1,000 by 200 already implies 200,000 inner fits, so mathematical design matters as much as computing speed.
Full chapter index
Chapters 1–7 · Understand the question
Chapters 8–14 · Reproduce the example
Chapters 15–21 · Check assumptions
Chapters 22–28 · Explore applications
Chapters 29–34 · Study and verify
A single bootstrap estimates the sampling distribution from one empirical distribution. Its interval or test can still miss the advertised level because that estimated distribution is itself uncertain. A second bootstrap layer estimates this remaining calibration error.
Start with the observed data and statistic T. Draw an outer sample, calculate T-star, then treat that outer sample as a temporary empirical population and draw inner samples from it. Repeat this inner experiment for every outer sample.
The R boot package documents nested.corr for the double-bootstrap practical in Davison and Hinkley’s Bootstrap Methods and Their Application. Read the official R documentation.
Outer replicates approximate repeated datasets from the fitted empirical law. Inner replicates estimate how the chosen root behaves inside each outer world. Mixing the two levels destroys the calibration logic.
Let B=400 outer samples and C=100 inner samples per outer sample. The design generates 400 outer statistics and 40,000 inner statistics. Including the original fit gives at least 40,401 statistic evaluations.
| Outer B | Inner C | Inner fits |
|---|---|---|
| 400 | 100 | 40000 |
Suppose 36 of 400 outer samples have an inner tail probability no greater than 0.05. The estimated achieved rejection probability is 36/400=0.09. That differs from the nominal 0.05 and signals a calibration problem.
| Outer cases | At or below 0.05 | Proportion |
|---|---|---|
| 400 | 36 | 0.09 |
The first bootstrap estimates a distributional approximation. The second layer estimates the error of that approximation for the chosen root, allowing an adjusted cutoff or bias correction rather than assuming first-order calibration is exact.
For an interval, inner resamples can estimate an outer-sample-specific quantile or p-value. The outer distribution of those calibrated quantities determines a revised endpoint. The precise algorithm must be named because several double-bootstrap intervals exist.
For a test, each outer statistic is compared with its inner null distribution to form a bootstrap p-value. The collection of outer p-values estimates the critical threshold required to achieve the requested test size.
A second layer can estimate bias remaining after a first bootstrap correction. Repeated correction is not automatically better: unstable statistics and noisy inner estimates can amplify error.
Computational shortcuts use one or a few inner samples or analytic approximations. They can be useful, but they are different algorithms with their own assumptions and should not be described as a full nested run.
At fixed cost, increasing B improves the outer calibration estimate while increasing C improves each inner probability estimate. An efficient allocation depends on the target, tail probability, fit cost and desired Monte Carlo error.
If an outer calibrated proportion is 0.09 from B=400, its simple Monte Carlo standard error is sqrt(0.09×0.91/400)≈0.0143, before accounting for inner simulation noise.
Use independent or deliberately controlled random streams across outer samples. Reusing the same inner draws without a justified common-random-number design can create hidden dependence.
An outer sample may produce a singular fit, and an inner sample may fail even when its outer fit succeeds. Report failures by level and define a principled handling rule before seeing the answer.
A double bootstrap can calibrate a small-sample coefficient interval when residual asymmetry makes a first-level approximation imperfect. The resampling scheme must still respect fixed or random design assumptions.
For a rare failure probability, nested resampling can calibrate an upper limit. Thin tails and repeated zero-failure samples may make the inner layer unstable rather than informative.
Bootstrap-t studentizes each outer statistic using an estimated standard error, often itself calculated analytically or by resampling. A double bootstrap may supply that inner standard error or perform an additional calibration step.
Prepivoting transforms a root by its estimated distribution function. Iterated prepivoting is closely related to nested bootstrap calibration, but the transformation and decision rule should be stated explicitly.
Run a simulation study where truth is known, vary B and C separately, track Monte Carlo error, inspect level or coverage, and compare the nested method with a simpler single bootstrap at equal computing cost.
Double bootstrap cannot repair a wrong sampling unit, dependence ignored by the resampler, severe model misspecification, impossible tail support or an unidentified parameter. More layers do not create information.
Use one row per outer sample and columns for its inner statistics, inner tail probability, fit failures and calibrated indicator. Never flatten the two levels into one undifferentiated list.
Write separate functions for outer generation, inner generation, root calculation and calibration. Unit-test B=400, C=100 and 40,000 inner fits; save seeds and level identifiers.
Nested resampling appears in econometrics, reliability, biomedical statistics, risk and simulation. Work requires probability, software testing, parallel computing and careful reporting.
Simulate skewed samples with known mean, compare single and double-bootstrap interval coverage, and plot accuracy against total fit count rather than against B alone.
A second layer is not a second independent dataset, 95% nominal does not guarantee 95% achieved coverage, and greater computation does not cure invalid resampling units.
CHAPTER 27 OF 34 · Explore applications
27. Connection to recent mathematics articles
The bootstrap-t guide explains studentized roots. The jackknife-after-bootstrap guide explains reuse of stored replicates.
Students should draw the two-level tree and calculate the fit count. Parents can ask what the second level is estimating and whether extra computation changed measured coverage.
- Calculate B×C.
- Recompute 36/400.
- Calculate its Monte Carlo error.
- Distinguish outer and inner uncertainty.
- Compare full and fast variants.
### Does the inner sample come from the original data? It is normally resampled from its outer sample.
### Is every double bootstrap identical? No.
### Does it guarantee coverage? No.
### Why is it expensive? Each outer sample contains another resampling experiment.
Report statistic, root, resampling unit, B, C, seeds, calibration rule, total fits, failures by level, Monte Carlo errors, coverage study and computational shortcut if any.
Total work is approximately B outer fits plus BC inner fits. Parallel execution shortens elapsed time but does not reduce the statistical or energy cost of the evaluations.
Confirm the two levels remain distinct, the calibration target is explicit, inner probabilities are adequately resolved, failures are traceable and improvement is demonstrated against a simpler benchmark.
Continue through the Mathematics Learning Hub. Reproduce one nested, transformed or distributed resampling calculation by hand before trusting a large automated run.
Extended practice laboratory
Practice case 1: Define statistic
Setup: Start from the article’s double-bootstrap mechanism for case 1. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 2: Define root
Setup: Start from the article’s double-bootstrap mechanism for case 2. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 3: Draw outer sample
Setup: Start from the article’s double-bootstrap mechanism for case 3. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 4: Calculate outer root
Setup: Start from the article’s double-bootstrap mechanism for case 4. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 5: Draw inner samples
Setup: Start from the article’s double-bootstrap mechanism for case 5. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 6: Calculate inner roots
Setup: Start from the article’s double-bootstrap mechanism for case 6. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 7: Estimate inner tail
Setup: Start from the article’s double-bootstrap mechanism for case 7. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 8: Repeat outer layer
Setup: Start from the article’s double-bootstrap mechanism for case 8. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 9: Calibrate cutoff
Setup: Start from the article’s double-bootstrap mechanism for case 9. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 10: Count total fits
Setup: Start from the article’s double-bootstrap mechanism for case 10. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 11: Allocate B and C
Setup: Start from the article’s double-bootstrap mechanism for case 11. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 12: Calculate Monte Carlo error
Setup: Start from the article’s double-bootstrap mechanism for case 12. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 13: Change B
Setup: Start from the article’s double-bootstrap mechanism for case 13. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 14: Change C
Setup: Start from the article’s double-bootstrap mechanism for case 14. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 15: Use bootstrap-t
Setup: Start from the article’s double-bootstrap mechanism for case 15. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 16: Use confidence interval
Setup: Start from the article’s double-bootstrap mechanism for case 16. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 17: Use hypothesis test
Setup: Start from the article’s double-bootstrap mechanism for case 17. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 18: Add skewness
Setup: Start from the article’s double-bootstrap mechanism for case 18. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 19: Add failed fit
Setup: Start from the article’s double-bootstrap mechanism for case 19. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 20: Parallelise workers
Setup: Start from the article’s double-bootstrap mechanism for case 20. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 21: Save random streams
Setup: Start from the article’s double-bootstrap mechanism for case 21. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 22: Compare single bootstrap
Setup: Start from the article’s double-bootstrap mechanism for case 22. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 23: Compare fast variant
Setup: Start from the article’s double-bootstrap mechanism for case 23. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 24: Name limits
Setup: Start from the article’s double-bootstrap mechanism for case 24. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 25: Write conclusion
Setup: Start from the article’s double-bootstrap mechanism for case 25. Task: Change the named quantity, recompute the dependent result and compare with the original.
Check: Preserve the resampling unit, verify the probability or weighting identity, and state one diagnostic before interpretation. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.