The bootstrap works by treating the observed sample as an empirical stand-in for the population, drawing many new samples from that empirical distribution—usually by sampling the original observations with replacement—recomputing the statistic of interest on each resample, and using the resulting bootstrap distribution to approximate the statistic’s sampling variability, bias and uncertainty. The method is powerful because it replaces difficult probability algebra with repeated computation, but it is only as trustworthy as the resampling scheme’s ability to mimic how the real data could have been generated.
Imagine having one sample and wanting to know what would have happened if the study had been repeated.
You do not have those repetitions.
You have one dataset.
The bootstrap makes a daring move:
Use the sample you do have to imitate the repeated samples you do not.
That is the entire method in miniature.
Stanford’s statistics teaching notes describe the bootstrap as a resampling technique used to estimate standard errors by repeatedly drawing samples from the empirical distribution and recalculating the statistic. The university’s statistics history notes credit Bradley Efron with developing the modern bootstrap in the late 1970s and early 1980s.
Quick Read
ORIGINAL SAMPLE → EMPIRICAL DISTRIBUTION → RESAMPLE WITH REPLACEMENT → RECOMPUTE STATISTIC → REPEAT MANY TIMES → BOOTSTRAP DISTRIBUTION → STANDARD ERROR / BIAS / INTERVAL → CHECK RESAMPLING UNIT → CHECK DEPENDENCE → INTERPRET
1. The Bootstrap Is About the Sampling Distribution of a Statistic
Suppose you estimate a population mean, median, regression coefficient, correlation, treatment effect or prediction error.
If you repeated the study, the estimate would change.
The distribution of those hypothetical repeated estimates is the sampling distribution.
The bootstrap tries to approximate that distribution from the observed data.
2. The Empirical Distribution Gives Every Observed Unit Equal Mass
If the sample contains n independent observations, the ordinary nonparametric bootstrap treats each observed unit as having probability 1/n.
The empirical distribution is therefore a discrete distribution concentrated exactly on the observed sample values.
Resampling from it means drawing observed units again and again.
3. Sampling With Replacement Is What Creates New Pseudo-Samples
With replacement, one observation can appear several times in a bootstrap sample while another may not appear at all.
That variation is deliberate.
It allows each bootstrap sample to differ from the original while preserving the empirical distribution as the source.
4. A Bootstrap Sample Usually Has the Same Size as the Original Sample
If the original sample has 200 observations, an ordinary bootstrap resample also contains 200 draws with replacement.
The sample size stays fixed while composition changes.
This imitates repeated sampling at the same nominal information scale.
5. About 63.2% of Unique Observations Appear in a Large Bootstrap Sample
Because sampling is with replacement, not every original observation appears in each bootstrap resample.
For large n, the probability that a particular observation is omitted is approximately e−1 ≈ 0.368.
So roughly 63.2% of unique observations appear at least once.
This fact later becomes useful in out-of-bag estimation for bagging and random forests.
6. The Statistic Is Recomputed on Every Resample
Bootstrap uncertainty belongs to a specific estimator.
If your estimator is the median, calculate the median in every bootstrap sample.
If it is a regression slope, refit the regression in every resample.
If it is a complex machine-learning summary, the full relevant estimation step must be repeated.
7. The Bootstrap Standard Error Is the Spread of Bootstrap Estimates
Let θ̂*₁, θ̂*₂, …, θ̂*B be estimates from B bootstrap samples.
The standard deviation of those bootstrap estimates approximates the standard error of the original estimator.
Stanford’s teaching notes present this as the core bootstrap calculation.
8. More Bootstrap Replicates Reduce Monte Carlo Noise
If B is only 50, the estimated standard error can wobble simply because too few resamples were drawn.
Increasing B reduces this simulation noise.
But more resamples do not add new information about the population beyond the original sample.
Ten million bootstrap replicates cannot rescue ten bad observations.
9. Bootstrap Bias Is Estimated by Comparing the Bootstrap Centre With the Original Estimate
If the mean of bootstrap estimates is systematically above the observed estimate, that difference can estimate finite-sample bias under the bootstrap approximation.
A bias-corrected point estimate can sometimes be constructed.
Bias correction is not automatically an improvement; reducing one estimated bias can increase variance or amplify bootstrap approximation error.
10. The Percentile Interval Uses Quantiles of the Bootstrap Distribution Directly
For a nominal 95% percentile interval, take the 2.5th and 97.5th percentiles of the bootstrap estimates.
The method is intuitive because the bootstrap distribution itself provides the endpoints.
Its coverage can be poor when the estimator is biased, skewed or irregular.
11. The Basic Bootstrap Interval Reflects the Bootstrap Errors Around the Observed Estimate
The basic interval uses the bootstrap distribution of θ̂* − θ̂ and reflects its quantiles around the original estimate.
It can be viewed as reversing estimated bootstrap errors.
Like the percentile interval, it relies on the bootstrap distribution approximating the true sampling distribution well.
12. Bootstrap-t Intervals Studentise the Statistic
Instead of bootstrapping the raw estimate, bootstrap-t methods use a statistic such as:
(θ̂* − θ̂) / SE*
Studentisation can improve coverage because it adjusts for changing uncertainty across parameter values.
The price is additional computation because each bootstrap sample may need its own standard-error estimate.
13. BCa Intervals Correct for Bias and Acceleration
The bias-corrected and accelerated interval adjusts percentile cutoffs for median bias and for how estimator variability changes with the parameter.
The acceleration term is often estimated using jackknife influence information.
BCa intervals are among the classic high-quality nonparametric bootstrap intervals for smooth statistics, though no interval is universally best.
14. Bootstrap Confidence Intervals Are Still Frequentist Procedures
A 95% bootstrap confidence interval is not automatically a 95% posterior probability statement about the fixed parameter.
Its interpretation remains about the long-run coverage of the interval construction under repeated sampling and bootstrap approximation.
See How Confidence Intervals Work.
15. The Nonparametric Bootstrap Makes Fewer Distributional Assumptions—but Not No Assumptions
It does not require the analyst to specify a normal, gamma or other parametric population distribution.
But it still assumes the empirical distribution is a useful approximation to the population for the estimator of interest.
It also requires the resampling unit and dependence structure to be correctly represented.
16. The Parametric Bootstrap Resamples From a Fitted Model
Fit a probability model to the observed data.
Generate synthetic datasets from that fitted model.
Refit the model or recompute the statistic in each synthetic dataset.
This can outperform the nonparametric bootstrap when the parametric model is well specified and data are sparse.
It can fail badly when the fitted model is wrong.
17. Pairs Bootstrap Resamples Entire Observation Rows
In regression with independent observational units, one approach resamples complete (X,Y) rows.
This lets both predictor distribution and outcome variation move across bootstrap samples.
It is appropriate for random-design interpretations under suitable independence assumptions.
18. Residual Bootstrap Holds the Predictors Fixed
Fit a regression model.
Resample residuals and add them back to fitted values to create new outcomes at the same predictor values.
This conditions on the design matrix and relies more strongly on the regression error model.
Heteroskedasticity can make naive residual resampling inappropriate.
19. Wild Bootstrap Handles Certain Heteroskedastic Regression Problems
Rather than resampling residuals directly, the wild bootstrap multiplies residuals by random weights with chosen moments.
This can preserve heteroskedastic structure better than ordinary residual resampling.
The method is especially useful for regression inference when conditional variance changes across observations.
20. Cluster Bootstrap Resamples Clusters, Not Individuals
Students within one school share a teacher and environment.
If the independent sampling unit is the school, resampling students individually breaks the dependence structure.
A cluster bootstrap samples schools with replacement and keeps the observations inside each selected school together.
The resampling unit should match the independent information unit.
21. Hierarchical Data Can Require Multi-Stage Bootstrap Designs
Schools may be sampled, then classes within schools, then students within classes.
A multi-stage bootstrap can mimic those sampling stages.
Flattening hierarchical data into independent rows can produce severely underestimated uncertainty.
22. Block Bootstrap Resamples Dependent Time Segments
Time-series observations are not exchangeable one by one because nearby values are autocorrelated.
A block bootstrap resamples contiguous chunks so local dependence is preserved inside each block.
Block length controls a trade-off: blocks must be long enough to preserve dependence and short enough to provide enough resampling units.
23. Moving-Block and Stationary Bootstraps Vary the Block Construction
Moving-block bootstrap samples fixed-length overlapping blocks.
Stationary bootstrap uses random block lengths, often geometrically distributed, to preserve stationarity more naturally.
The correct design depends on the dependence process and inferential target.
24. Spatial Bootstrap Must Preserve Spatial Dependence
Nearby measurements of rainfall, pollution or disease prevalence are often correlated.
Resampling individual locations independently destroys spatial structure.
Spatial blocks, model-based bootstrap or geostatistical simulation may be needed.
25. Matched and Paired Designs Must Resample the Pair
Before–after measurements from one person form a dependent pair.
Matched case–control units also carry a pairing structure.
Resampling individual measurements separately breaks the design.
Resample pairs as units or bootstrap the within-pair differences, depending on the estimand.
26. Stratified Sampling Requires Stratified Resampling
If the original design samples fixed numbers from different strata, the bootstrap should usually preserve that structure.
Resample within strata rather than across the entire pooled sample.
Otherwise the bootstrap imitates a different study design.
27. Survey Bootstraps Can Require Replicate Weights
Complex probability surveys use unequal inclusion probabilities, stratification and clustering.
Specialised replicate-weight methods—such as bootstrap replicate weights, balanced repeated replication or jackknife repeated replication—approximate design-based variance.
Ordinary IID row bootstrap is generally not enough.
28. The Bootstrap Works Best for Smooth, Regular Statistics
Means, regression coefficients and many smooth functionals behave well under ordinary bootstrap conditions.
Statistics with discontinuities, boundaries, extreme-order dependence or model-selection steps can behave poorly.
“Just bootstrap it” is not a universal theorem.
29. Estimating an Extreme Maximum Is Hard for the Nonparametric Bootstrap
The ordinary bootstrap can never generate a value larger than the largest observation already seen.
If the estimand concerns a population maximum or extreme tail beyond the observed range, empirical resampling may fundamentally lack the needed support.
Extreme-value models or parametric tail methods may be more appropriate.
30. Boundary Problems Can Break Standard Bootstrap Approximations
A variance component constrained to be non-negative can have an estimate at zero.
Standard asymptotic normality and naive bootstrap resampling may fail near such boundaries.
Specialised parametric bootstrap or exact theory may be needed.
31. Bootstrap After Model Selection Can Understate Selection Uncertainty
Select a model once using the original data.
Then bootstrap coefficients while keeping that selected model fixed.
This ignores the possibility that another sample would have selected another model.
To study the full procedure, repeat model selection inside every bootstrap sample.
Even then, post-selection distributions can be irregular.
32. Bootstrap Selection Frequencies Reveal Instability
If a predictor is selected in 51% of bootstrap samples, it is hard to defend it as uniquely indispensable.
Selection frequency turns model instability into something visible.
It is not a posterior inclusion probability unless a Bayesian model defines it that way.
33. Bootstrap Can Estimate Optimism in Prediction Models
Fit a model in each bootstrap sample.
Compare performance on the bootstrap sample with performance when that fitted model is evaluated on the original dataset.
The difference estimates overfitting optimism under the bootstrap scheme.
This is common in clinical prediction modelling, though external validation remains stronger evidence.
34. Cross-Validation and Bootstrap Own Different Default Jobs
Cross-validation primarily withholds observations to estimate predictive generalisation or tune a modelling procedure.
The bootstrap primarily resamples observations to approximate estimator sampling behaviour, standard errors, bias or optimism.
Stanford’s teaching notes make this distinction explicitly.
See How Cross-Validation Works.
35. The Bootstrap Is Not a Permutation Test
Bootstrap resampling usually samples with replacement to approximate sampling variability around an estimated data-generating distribution.
Permutation testing rearranges labels or values without replacement under an exchangeability null to construct a reference distribution for a test statistic.
The mechanics look similar because both repeatedly recompute statistics.
The inferential logic is different.
36. Bootstrap Is Not Creating New Independent Evidence
A dataset with 100 observations remains a dataset with 100 original observations no matter how many bootstrap samples are generated.
The bootstrap reuses information to approximate uncertainty.
It does not turn 100 observations into one million independent observations.
37. A Bootstrap Distribution Can Be Smooth Even When the Data Are Poor
Suppose a biased convenience sample is enormous.
The bootstrap distribution of its mean can be beautifully narrow.
The estimate can still be systematically wrong for the target population.
Bootstrap uncertainty reflects the resampling model, not every source of bias.
38. Bootstrap Cannot Repair Measurement Bias
If every thermometer reads 1.5°C too high, resampling those measurements reproduces the calibration error.
The bootstrap can estimate sampling variability around the biased estimate.
It cannot discover the missing calibration truth without external information.
39. Bootstrap Cannot Create Missing Population Regions
If a sample contains no low-income rural participants, nonparametric resampling cannot generate them.
The empirical distribution has zero support where the original sample had zero support.
Representation problems remain representation problems.
40. Small Samples Can Give Fragile Bootstrap Distributions
With six observations, the empirical distribution is a very coarse approximation to the population.
Every bootstrap sample is built from those same six support points.
For smooth statistics this can still be useful, but tail inference, skewed estimators and rare-event structure may be unreliable.
41. Rare Events Can Disappear From Many Bootstrap Samples
If only two events exist in the original sample, some bootstrap resamples contain no events or many duplicates of the same event.
Regression models may fail to converge or statistics may become undefined.
Parametric, exact or specialised rare-event methods may be more stable.
42. The m-out-of-n Bootstrap Can Help in Irregular Problems
Instead of resampling n observations, resample m observations where m grows with n but can be smaller.
This can restore consistency in some nonregular estimation problems where the ordinary n-out-of-n bootstrap fails.
The choice of m introduces another tuning problem.
43. The Bayesian Bootstrap Reweights Observed Values Rather Than Resampling Counts
The Bayesian bootstrap assigns random Dirichlet weights to the observed support points.
It creates a posterior-like distribution over discrete empirical distributions rather than ordinary repeated-sampling bootstrap draws.
Its interpretation is Bayesian and should not be confused with the frequentist nonparametric bootstrap.
44. Bagging Uses Bootstrap Samples to Reduce Prediction Variance
Fit an unstable predictor—such as a deep decision tree—on many bootstrap samples.
Average the predictions.
Bagging reduces variance because the individual bootstrap-fitted models fluctuate differently.
The bootstrap here is not estimating one standard error; it is powering an ensemble learning method.
45. Random Forests Add Feature Randomness to Bootstrap Aggregation
Random forests usually combine bootstrap samples with random subsets of candidate features at each tree split.
The extra feature randomness decorrelates trees so averaging reduces variance more effectively.
Out-of-bag observations can provide an internal performance estimate.
46. Bootstrap Confidence Intervals Need Enough Replicates in the Tails
A 99% interval depends on extreme bootstrap quantiles.
If B is too small, there are too few simulations in those tails to estimate endpoints stably.
The required B therefore depends on the inferential target, not only computational convenience.
47. Reproducibility Requires Recording Random Seeds and the Resampling Scheme
Bootstrap results contain Monte Carlo randomness.
A reproducible analysis should record the random-number generator or seed, number of replicates, resampling unit, interval method and any failed replicate handling.
“We bootstrapped it” is not enough methodological detail.
48. The Hostile Test: Individual Bootstrap for Clustered Schools
Five thousand students come from twenty schools.
The analyst bootstraps individual students as though all 5,000 were independent.
School-level dependence is broken.
The uncertainty can become far too small.
The school—not the student row—may be the appropriate resampling unit.
49. The Second Hostile Test: Ten Million Bootstrap Replicates From Ten Observations
The simulation distribution looks smooth.
The original empirical distribution still contains only ten support points.
Monte Carlo precision has become extraordinary.
Scientific information has not.
50. The Third Hostile Test: Bootstrap After Feature Selection
Two hundred predictors are searched once.
Five are selected.
The five-predictor model is then bootstrapped while selection stays frozen.
The intervals ignore uncertainty about which predictors would have been selected in another sample.
51. The Fourth Hostile Test: Bootstrap a Biased Convenience Sample
One hundred thousand highly engaged app users answer a learning survey.
The bootstrap standard error is microscopic.
Disengaged students are absent from the sampling frame.
The bootstrap quantifies precision around the convenience sample, not representativeness of all students.
52. The Fifth Hostile Test: Time-Series Rows Resampled Independently
Hourly electricity demand has strong autocorrelation.
The analyst resamples individual hours independently.
The synthetic series destroys daily and weekly structure.
The bootstrap distribution answers a different data-generating question.
53. Primary School: Bootstrap Begins as “Make Many Pretend Samples From the Sample We Have”
A class measures ten leaves.
They write each leaf length on a card.
They draw ten cards with replacement, calculate the mean, return the cards and repeat.
The repeated pretend means show how much the answer can wobble.
The bootstrap does not give us more leaves. It helps us understand how uncertain our answer is because we saw only a few.
54. Secondary School: Separate Original Data From Resampling Variability
Students can watch some observations appear twice and others disappear from each resample.
They learn that the bootstrap distribution is generated from the sample rather than newly measured from reality.
The distinction prevents pseudo-replication.
55. JC and University: Bootstrap Becomes a Resampling Approximation to an Estimator’s Sampling Law
At higher levels, learners should reconstruct:
- estimand;
- estimator;
- empirical distribution;
- bootstrap sample;
- bootstrap statistic;
- standard error;
- bias estimate;
- percentile/basic/studentised/BCa intervals;
- dependence structure;
- cluster/block resampling;
- regularity conditions;
- Monte Carlo error.
56. Where Bootstrap Fits in the eduKateSG “How Works” Landscape
- How Statistical Estimation Works — defines the estimator whose uncertainty bootstrap approximates.
- How Confidence Intervals Work — owns coverage interpretation of bootstrap intervals.
- How Cross-Validation Works — distinct resampling method aimed primarily at held-out predictive performance and model tuning.
- How Statistical Model Selection Works — selection steps that may need to be repeated inside bootstrap resamples.
- How Regression Works — pairs, residual and wild bootstrap contexts.
- How Sampling Works — defines the real information-generating unit the bootstrap must imitate.
Bootstrap owns one precise canonical job: approximate the sampling behaviour of an estimator by repeatedly rebuilding pseudo-samples from an empirical or fitted data-generating distribution while preserving the original study’s information structure.
57. What This Article Does Not Claim
- Bootstrap resamples are not new independent observations.
- More bootstrap replicates do not increase the original sample’s scientific information.
- The ordinary IID row bootstrap is inappropriate for many clustered, time-series, spatial and paired designs.
- Bootstrap confidence intervals are not automatically Bayesian credible intervals.
- Nonparametric bootstrap is not assumption-free.
- A narrow bootstrap distribution cannot repair sampling bias or measurement bias.
- The ordinary bootstrap can fail for nonregular estimators, boundaries and extreme-value targets.
- Post-selection bootstrap inference must account for the selection procedure.
- Parametric bootstrap inherits the fitted model’s misspecification risk.
- Cross-validation and permutation testing use resampling for different inferential jobs.
58. A Compact Bootstrap Audit
- What estimand is being estimated?
- What estimator is being bootstrapped?
- What is the independent sampling unit?
- Is the ordinary IID nonparametric bootstrap appropriate?
- Should pairs, clusters, strata or blocks be resampled instead?
- Is the bootstrap parametric or nonparametric?
- If parametric, is the fitted model credible?
- How many bootstrap replicates are used?
- Is Monte Carlo error small enough for the target quantity?
- What standard error is calculated?
- Is bias being estimated or corrected?
- Which interval is used: percentile, basic, bootstrap-t or BCa?
- Is the estimator smooth and regular enough for ordinary bootstrap theory?
- Are there parameter boundaries?
- Are rare events causing degenerate resamples?
- Does time or spatial dependence require block resampling?
- Does complex survey design require replicate weights?
- Was model selection repeated inside each bootstrap sample?
- Could the sample itself be systematically biased?
- Could measurement bias dominate sampling variability?
- Does the bootstrap support the decision actually being made?
59. Frequently Asked Questions
What is the bootstrap?
The bootstrap is a resampling method that repeatedly draws pseudo-samples from the observed empirical distribution or a fitted model and recomputes a statistic to approximate its sampling uncertainty.
Why sample with replacement?
Sampling with replacement makes each pseudo-sample vary while preserving the empirical distribution as the data-generating source. Some observations repeat and some disappear, imitating repeated sampling from that empirical distribution.
How is bootstrap different from cross-validation?
Bootstrap is primarily used to approximate sampling variability, bias and confidence intervals for estimators. Cross-validation primarily withholds data to estimate predictive generalisation or tune model procedures.
Can bootstrap fix a biased sample?
No. It can quantify sampling variability around the observed sample under the resampling model, but it cannot create population groups or calibration information that were missing from the original evidence.
60. Authoritative Research Corridor
- Stanford STATS 202 — Bootstrap
- Stanford Department of Statistics — Bootstrap History and Software Tools
- Bradley Efron — The Bootstrap and Modern Statistics
- DiCiccio and Tibshirani — Bootstrap Confidence Intervals and Bootstrap Approximations
- eduKateSG — How Statistical Estimation Works
- eduKateSG — How Cross-Validation Works
Final Thought: The Bootstrap Is an Honest Use of What We Actually Have
We rarely get to repeat the world on demand.
We get one cohort.
One field season.
One experiment.
One sample.
The bootstrap asks how much that one sample can teach us about the instability of the statistic we computed from it.
Its elegance lies in computational humility.
We do not pretend to possess the missing repeated studies.
We construct an approximation from the evidence actually in hand.
The bootstrap is powerful not because one sample secretly contains infinite data, but because one sample can sometimes contain enough structure to imitate how our estimate would wobble if reality gave us another chance to sample it.