eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How AI Works | Out-of-Bag Estimates — Bootstrap Trees, Unseen Votes and Honest Validation

—

by

eduKate Asia

Did you know? A random forest can estimate how well it predicts observations that individual trees did not see during training, without deliberately holding back a separate validation set for each tree. Every bootstrap tree is trained on a sample drawn with replacement. An observation excluded from one tree’s sample is ‘out of bag’ for that tree, so its prediction can contribute to an out-of-bag, or OOB, evaluation of the forest.

The useful rule is to evaluate each observation only with trees whose bootstrap training sample excluded that observation, count how many eligible predictions it actually receives, and aggregate errors with the correct denominator. OOB evaluation is a convenient internal model check; it is not proof against leakage from preprocessing, repeated identities, time-dependent data or repeated tuning. A final independent test set still matters when making a defensible claim about future performance.

This guide explains the bootstrap mechanism from first principles, computes checked omission probabilities and voting examples, compares OOB evaluation with cross-validation, and offers practical diagnostics and a changed-context exercise. Its numerical datasets are fictional illustrations, not the results of a model run on a real eduKate student or business dataset.

Contents

Why random forests create their own partially unseen observations

A single decision tree can fit training examples very closely, especially when grown deep. That flexibility is useful for discovering patterns, but it can also make the tree sensitive to the particular cases it saw. A random forest trains many trees on varied bootstrap samples and commonly randomises feature choices at splits. Combining their predictions can reduce variance relative to a single unstable tree.

Bootstrap sampling draws observations with replacement from the available training set. If there are 100 training rows, a tree may draw 100 times, but the same row can appear more than once. Other rows will not appear at all in that tree’s sample. The included draws train the tree; the excluded rows are out of bag for that tree.

Different trees receive different bootstrap draws, so each row is excluded from some trees and included in others. The forest can form an out-of-bag prediction for that row by using only the trees that were not trained on it. After this is repeated for all rows, the collection of predictions supplies an internal evaluation signal.

Leo Breiman’s original Random Forests (2001) paper is a primary foundation for the method. The official scikit-learn ensemble guide describes bootstrap forests and out-of-bag generalisation estimates. Their value comes from which observations trained which trees, not from a model mysteriously knowing unseen data.

The phrase ‘internally held out’ should not conceal its limits. Each observation may influence preprocessing performed outside the tree-building process. An OOB prediction can also be statistically dependent on other observations from the same subject or event. Those details matter as much as the exclusion of the exact training row.

Bootstrap sampling with replacement is the core mechanism

Suppose a training dataset contains five observations numbered 1 through 5. A bootstrap sample of length five might be [2,2,5,1,2]. This tree sees observation 2 three times, observation 5 once and observation 1 once. Observations 3 and 4 are absent. A prediction from this tree for observation 3 or 4 is out of bag, while its prediction for observation 2 is not.

Drawing with replacement is important. Without replacement and with a sample size equal to the entire dataset, every observation would appear once and there would be no out-of-bag rows. A method that uses ordinary subsampling without replacement can still leave observations out, but its inclusion probabilities and estimator behaviour differ from the usual bootstrap construction.

The individual bootstrap sample does not need to contain every category or every rare subgroup. That randomness is part of the training procedure. For severely imbalanced data, some trees may see few examples of a rare class. The out-of-bag score can then be dominated by frequent-class performance unless appropriate metrics are reported.

Trees also differ in how features are considered. A forest may choose a random subset of candidate features at each split, making individual trees less correlated. That randomness is separate from the sampling of observations. OOB status depends on whether an observation was included in a tree’s training sample, not which feature was selected at a split.

Keeping the inclusion information is the practical prerequisite for evaluation. Without a record or reproducible method to determine which rows trained each tree, a developer cannot truthfully call the votes out of bag. A script that predicts the entire training set using every tree is merely in-sample ensemble prediction.

Worked probability: why roughly 36.8 percent are left out

Let N be the number of training observations, and let a bootstrap tree make N independent draws with replacement. On any one draw, a particular row has probability 1/N of being chosen and 1−1/N of not being chosen. To be out of bag, that row must be omitted on all N draws.

The exact probability is therefore (1−1/N)^N. For N=100, it is (99/100)^100, approximately 0.3660. The complementary probability of appearing at least once is about 1−0.3660=0.6340. The widely quoted 36.8% is the large-N limit e^(−1)≈0.3679, not an exact value for every finite dataset.

If a forest has 300 independently bootstrapped trees on 100 observations under this simplified design, the expected number of trees leaving one specified row out is 300×(99/100)^100≈109.8. This is an expectation, not a guarantee that every row is excluded from exactly 110 trees. Actual counts vary due to random sampling.

With a dataset of eight rows and eight replacement draws, the exact omission probability is (7/8)^8≈0.3436. The expected number of OOB rows for that one tree is 8×0.3436≈2.749. A particular draw can omit more or fewer than that number. The finite-sample calculation matters especially in very small demonstrations.

These numbers describe the sampling mechanism, not the accuracy of the model. About 36.6% out-of-bag observations for a single tree does not mean the forest has a 36.6% error rate or that 36.6% of predictions are discarded. An OOB estimate aggregates eligible predictions across many trees.

How one observation gets its out-of-bag prediction

Imagine a toy forest with five trees, labelled T1 through T5. The focal observation has true class B. It appears in the training samples of T1 and T3, so their predictions are in-bag and cannot be counted in an honest OOB vote for this observation. T2, T4 and T5 did not see the observation and are eligible to vote.

Suppose T2 predicts A, T4 predicts B and T5 predicts B. The OOB majority class is B, with two of the three eligible votes. The OOB vote fraction for B is 2/3≈0.6667; for A it is 1/3≈0.3333. These fractions are not necessarily calibrated probabilities, especially with only three votes.

If the developer accidentally includes all five trees and T1 and T3 both vote B, the result becomes four of five votes for B. That may look more confident but breaks the OOB definition. The two added votes came from trees trained on the very observation being evaluated. Their inclusion makes the estimate optimistic in an uncontrolled way.

Repeat the eligible-vote procedure separately for every training observation, because each row has a different set of out-of-bag trees. A tree can be eligible for row 17 and ineligible for row 18. An implementation that picks one shared validation forest for every row does not represent the usual OOB construction.

Prediction probabilities may be combined rather than hard votes, depending on the estimator. The invariant remains the same: only predictions from trees whose bootstrap sample omitted that observation can contribute to its OOB prediction. Check the library’s documented aggregation method before interpreting a probability output.

Worked evaluation: count the right rows and errors

Suppose a fictional classifier produces valid OOB predictions for 120 independent records. It predicts the correct class on 94 of them and an incorrect class on 26. The OOB accuracy is 94/120≈0.7833, or 78.33%; the OOB error rate is 26/120≈0.2167, or 21.67%. The two rates sum to 100% up to rounding.

That calculation is straightforward only after defining the population and ensuring every included observation had a valid OOB prediction. A small forest can leave some training rows without any OOB tree. If five rows lack a valid prediction, it is wrong to count 94 correct against all 125 without stating how those five were handled. The denominator must follow the estimator and evaluation protocol.

Even with all 120 observations covered, the accuracy number does not describe subgroup risk. Imagine 100 records from a frequent class and 20 from a rare class. If all 20 rare-class records are misclassified but the model gets 94 of 100 frequent-class records correct, overall accuracy is still 94/120≈78.33%. That is unacceptable for many tasks in which the rare class is important.

Report confusion counts, per-class recall and the actual unit of independence alongside the overall metric. If an observation is one extracted frame from a video and nearby frames are nearly identical, the effective independence may be much lower than the row count. OOB exclusion of one frame does not guarantee the model has not seen near-duplicate content in other frames.

Finally, do not interpret the score as a confidence interval. A point estimate of 78.33% is not a probability that future performance lies above 78%. Quantifying uncertainty requires additional assumptions or resampling and should account for dependence between observations and trees.

Why a small forest can produce missing OOB predictions

For a specified observation, each independent bootstrap tree has a probability q of omitting it. With N=100 and sample size 100, q≈0.366. The chance that none of T independent trees omits the observation is approximately (1−q)^T under the independent bootstrap-draw assumption. That is the probability a particular row receives zero eligible OOB votes.

With T=10, using q≈0.366, the zero-vote probability is about 0.634^10≈0.0105, or roughly one percent for a specific row. With a hundred rows, one might therefore encounter a handful or none, depending on random sampling. This does not imply rows are independent of one another in the forest, so a simple binomial count of missing rows would be only an approximation.

As T grows, zero coverage becomes less likely. But the number of eligible votes can still differ across rows. With a very small number of trees, a row’s OOB probability estimate may be dominated by one or two trees. A forest’s overall OOB score may stabilise as more trees are added, but convergence should be inspected rather than presumed.

The official RandomForestClassifier documentation notes that `oob_decision_function_` can contain NaN if a training data point is never left out when the number of estimators is small. That warning is not a software defect; it follows from the sampling design.

An honest implementation should report the number of rows with usable OOB predictions, check for nonfinite values and avoid silently replacing undefined results with an assumed majority class. A missing OOB vote is an information limitation, not a wrong label to be scored as an ordinary error.

Why each additional tree changes both training and evaluation

An ensemble of more trees generally contains more distinct bootstrap histories. For each observation, the number of eligible OOB voters tends to increase with tree count. At the same time, the forest’s overall prediction averages over more individual tree models. Those two changes can stabilise both the deployed prediction and its internal OOB estimate.

However, adding trees does not necessarily eliminate systematic bias. If all trees learn an unhelpful relation, averaging many of them may create a stable wrong answer. If the training labels are inconsistent or a leakage-prone feature is present, a larger forest may reproduce the error more confidently.

A learning-curve style plot of OOB error versus the number of trees can show whether the estimate remains volatile or appears stable. The official scikit-learn OOB Errors for Random Forests example illustrates this diagnostic. The curve is not a substitute for a final independent test because it can be used repeatedly to choose hyperparameters.

The cost trade-off matters. A forest with 1,000 trees may require more memory and inference time than one with 200, even if OOB error scarcely changes after 150. The model owner must decide whether additional stability justifies operational cost using measured results, not a rule that more trees are inherently better.

Treat OOB error as an aid to model design and validation, not an oracle. An apparently flat curve can reflect the fact that the selected metric ignores a rare but high-consequence error, and a jagged curve can reflect too few trees or an unstable dataset.

OOB evaluation and a genuinely independent test set answer different questions

Out-of-bag evaluation uses data that are part of the overall training dataset, but each individual prediction comes only from trees that omitted that observation during bootstrap fitting. This gives a valuable approximation to performance without fitting a fresh forest for each held-out fold. It can make experimentation more efficient and provide a useful internal cross-check.

Yet a final test set serves a separate governance purpose: it is reserved for estimating how a selected model behaves on data not used to choose its training design and hyperparameters. If a team compares 100 forest configurations by OOB score and selects the best, the chosen score is no longer a completely untouched estimate of generalisation. Selection can exploit favourable random variation.

A common sound workflow sets aside an appropriate final test group, trains and tunes on the remaining training material using OOB results and other validation, then evaluates the selected model once on the untouched group. The exact split design depends on the deployment setting, especially whether future predictions concern new people, new organisations or later time periods.

Cross-validation is another tool. In K-fold validation, each fold is fitted without its held-out rows and predictions are compared across folds. This may be computationally more expensive than OOB estimation, but it can support explicit grouped or chronological splitting strategies. OOB is attractive when its row-wise exclusion matches the independence assumption; it cannot replace a time split merely by being convenient.

Do not compare OOB accuracy and final test accuracy as if they must be identical. Different distributions and sampling variability can produce real gaps. Investigate large discrepancies rather than forcing them to match through more tuning on the test set.

The major trap: a row can be OOB while its information leaked in

Suppose a dataset has 500 observations from 50 individuals, ten observations per person. A random forest uses row-level bootstrap sampling. An OOB prediction for one row may come from trees trained on other rows from the same individual. If those rows share a unique pattern, the prediction is not a clean estimate of performance on entirely new individuals.

Likewise, if feature scaling, selection, encoding or target transformation was fitted using all records before the forest’s bootstrap steps, information from an OOB row may have influenced the model pipeline. The tree exclusion by itself does not undo such preprocessing. It is especially dangerous if a feature encodes a target-related aggregate computed across the full dataset.

The fix is to define the real evaluation unit. If the application predicts outcomes for new patients, group the split by patient. If the application predicts tomorrow’s demand, respect time ordering. If it classifies documents by organisation, avoid near-duplicate documents crossing the boundary. Use a pipeline and validation design that isolate the information the deployed system will genuinely lack.

A deliberate grouped or temporal holdout is often safer than relying on default OOB scoring. A forest can remain a useful estimator even when OOB does not match the task’s validation requirements. The choice is about evidence, not loyalty to one algorithm.

The published data leakage and grouped splits guide covers these information boundaries more broadly. The OOB-specific failure occurs when the same observation is absent from a tree but its correlated or future information is nevertheless present through other records or transformations.

Different tasks require different OOB summaries

Classification often uses an accuracy-like score, but one number is rarely enough. On imbalanced problems examine sensitivity, specificity, per-class recall, precision where relevant and the prevalence of categories. A model can achieve high overall accuracy while failing on the very cases that justify deploying it.

For regression, residual error can be summarised using mean squared error, root mean squared error or absolute error, depending on the loss the application cares about. Keep measurement units explicit. A root mean squared error of 3.2 is meaningless without saying whether the target is days, kilograms or a rating that lacks interval interpretation.

Probability forecasts can be evaluated for calibration and discrimination separately. A random forest’s OOB vote fraction is a model output, not a guarantee of reliable long-run frequency. If a decision depends on probabilities, test them on suitable independent observations and consider separate calibration methods where justified.

When a model serves a particular resource budget, evaluate whether it captures the important cases at that budget. For instance, a reviewer might inspect only the 20 most suspicious records each day. Precision at 20 and the associated consequences can matter more than generic accuracy. Such metrics should be defined before comparing many hyperparameter configurations.

Whatever the metric, preserve the denominator. A failure rate over 100 genuine out-of-bag predictions is not equivalent to a rate over 100 trees or over every prediction produced during training. The numerator and population should be recoverable from the saved evaluation record.

OOB estimates are not the same as feature importance

Random forests often expose feature importance measures. One family uses the total reduction in a splitting impurity criterion across trees. Another can measure how a score deteriorates when values of a feature are permuted. These address how the fitted model uses information, not directly how accurately it predicts unknown cases.

Out-of-bag data can help estimate the effect of feature perturbation without using the same trees that trained on a specific row. But an implementation must ensure that a perturbed row is evaluated only through its eligible OOB trees. Permuting a feature across all rows and then taking a standard training-set score does not automatically become an OOB permutation test.

Correlated predictors complicate interpretation. If two variables carry much the same information, permuting one may cause little loss because the other remains available. Alternatively, some importance methods can give high apparent value to features with many possible splits. Neither pattern proves a causal effect of that feature on the real outcome.

The related permutation feature importance article explains held-out scoring and correlated-feature pitfalls in detail. OOB evaluation is a source of partially unseen predictions; feature importance is an additional analysis with its own assumptions.

When communicating to users, avoid saying ‘the model learned that this factor causes the outcome’ merely because an OOB permutation score changes. Predictive usefulness, correlation and causation are distinct questions.

The hidden selection problem when you keep improving the OOB score

A practitioner may try 50 choices of tree depth, minimum leaf size, maximum features and class weighting. The model with the best OOB score is selected. Even if every individual OOB estimate was calculated correctly, the best observed result can be influenced by random fluctuations across configurations. Optimising the score converts it into part of the model-selection process.

Suppose, in a fictional search, ten models have essentially equal true performance but OOB accuracy fluctuates between 78% and 83% because of sampling. Selecting the model showing 83% does not establish that its future accuracy is 83%. That number benefited from being the maximum among ten observations. A clean final test can expose the resulting optimism.

Keep the search space and selection rule documented. Use robust validation appropriate to the data, such as group-aware folds, nested evaluation or a protected independent test when high-confidence claims are needed. Consider training variability by repeating fits with different random states when feasible.

Stopping a forest search when the OOB score no longer improves is a reasonable engineering heuristic. It is not a proof that the chosen result will reproduce externally. An honest report should say ‘selected using OOB performance’ and provide the independent final evidence separately.

This is another reason to preserve source code, random-state settings, preprocessing and exact dataset version. Without them, the best OOB score becomes an attractive number with no reproducible meaning.

A debugging sequence from suspicious OOB score to trustworthy result

First verify bootstrap training is enabled. A forest trained on the full dataset for every tree has no ordinary bootstrap OOB observations. Read the estimator settings and the official API contract. Some ensemble variants have different defaults, so do not infer behaviour from the model’s name.

Second verify row provenance. Check how inclusion samples were constructed, whether the OOB voter set is nonempty for enough records and whether aggregation follows the estimator’s documented method. Use a tiny hand-crafted dataset to confirm that an in-bag tree is excluded from a given row’s OOB prediction.

Third check preprocessing and leakage. Identify any transformation fitted before bagging and ask whether a withheld row’s label, timestamp or identifier could influence the tree through that transformation or another near-duplicate row. A near-perfect OOB score on a difficult real-world task is a reason for investigation, not an automatic reason for celebration.

Fourth review metric denominators and classes. Confirm that the evaluation does not ignore minority errors, undefined probabilities, invalid labels or missing records. Fifth compare with an appropriate independent split to see whether the OOB estimate transports to the actual deployment population.

Finally document the outcome. Record the number of trees, seed, bootstrap sample size, OOB coverage, score, final test score and acknowledged limitations. State separately whether a causal analysis, calibration study or subgroup assessment was performed. A complete audit trail makes an imperfect model easier to improve than a polished but untraceable accuracy claim.

Changed-context exercise: forecasting requests from new organisations

Imagine a service trying to classify whether support tickets are urgent. Training data contain 2,000 tickets from 40 organisations. Each organisation contributed multiple tickets with repeated language patterns and shared account identifiers. A random forest’s ordinary row-bootstrap OOB score looks promising, but the deployment task is to classify tickets from entirely new organisations.

The important question is whether an OOB tree for one ticket has seen other tickets from the same organisation. Under ordinary row sampling it very likely has. Organisation-specific language can therefore help the tree even though the exact ticket was omitted. The OOB prediction is not a clean test of generalisation to unseen organisations.

A better validation design holds out all tickets from selected organisations together, fitting preprocessing only on the training organisations. It can compare whether the apparent benefit survives that harder setting. The model may still be trained on all authorised historical records for final deployment after selection, but the reported validation must answer the actual future-data question.

Suppose a fictional grouped test includes 200 tickets from never-seen organisations, with 150 correctly classified. Grouped accuracy is 150/200=75%. The earlier ordinary OOB score was 90% on training organisations. The gap is 15 percentage points, not necessarily all caused by leakage; distribution shift and sample variation can contribute. It signals that the internal estimate and deployment context differ.

If the application instead predicts future tickets from organisations whose prior tickets the service will legitimately have seen, the appropriate split may be chronological within organisations. That changed condition requires a different design. The transferable principle is to decide what ‘unseen’ means for the real decision, not to equate the word out-of-bag with universal independence.

Limits of OOB estimates and what they cannot certify

Out-of-bag evaluation is an elegant reuse of bootstrap randomness. It provides an internal estimate without fitting a separate ensemble for every held-out subset. That does not make it a universal unbiased oracle, especially under small samples, heavy grouping, class imbalance, leakage or repeated tuning.

OOB error cannot establish that a dataset is representative of future users. It cannot make mislabeled records correct, remove duplicate identities or detect all forms of drift. It cannot prove a model’s predictions are causal, well calibrated for decisions or fair across every relevant group.

Likewise, a stable OOB curve does not establish that a model is safe in a high-stakes application. External validation, decision thresholds, monitoring and oversight may be necessary. Some tasks require prospective evaluation under a temporal or institutional shift rather than an internal training-set method.

The exact expected share of OOB samples depends on how many observations are drawn for each tree and whether draws use replacement. The familiar approximately 36.8% assumes N draws from N observations for large N. Changing `max_samples` or the ensemble algorithm changes the probability; do not reuse the percentage mechanically.

Every count and model outcome in the worked cases here is explicitly invented for arithmetic and conceptual explanation. The text does not claim that a real random forest was fitted, validated or deployed during publication.

Frequently asked questions about OOB validation

What does out of bag mean? An observation is out of bag for a tree when it was not selected in the bootstrap sample used to train that tree.

How does OOB scoring work? Each observation is predicted using only the trees for which it was absent from the training sample; those eligible predictions are aggregated and compared with observed labels.

Why is approximately one third of a dataset OOB? Under N replacement draws from N rows, the chance a specific row is never drawn is (1−1/N)^N, which approaches about 36.8% as N grows.

Is OOB accuracy the same as test accuracy? No. OOB uses internal bootstrap exclusions; a separate test set can represent different people, times or institutions and remains useful for final performance claims.

Can OOB be undefined for a row? Yes. With too few trees a row might have been included in every bootstrap sample, leaving no eligible OOB voters.

Does every random forest implementation provide OOB scoring? No. Check whether it uses bootstrap sampling and whether the library exposes an OOB score or decision function under the chosen settings.

Will adding trees always improve accuracy? Not necessarily. It can stabilise the ensemble and OOB estimate, but does not automatically repair systematic bias, invalid labels or data leakage.

Can OOB detect target leakage? Sometimes poor generalisation may reveal a problem, but row-level OOB can miss group leakage, preprocessing leakage and near-duplicates. An appropriate independent split is essential.

Is OOB feature importance the same as OOB accuracy? No. Importance uses a perturbation or split-based method to analyse feature reliance; OOB accuracy evaluates predictive correctness of eligible held-out votes.

Verified primary references and learner routes

The original foundation is Leo Breiman, Random Forests (Machine Learning, 2001). For current implementation and definitions, consult the official scikit-learn ensemble guide, RandomForestClassifier API documentation and the out-of-bag error example.

If you are learning, start with five numbered observations and manually create a few bootstrap draws. Mark which observations each tree saw, predict one excluded row and count only eligible votes. Extend to the (1−1/N)^N omission probability and explain how sample size affects OOB coverage.

If you build models, begin with the real deployment unit: row, person, organisation or time period. Decide which forms of leakage matter, then compare internal OOB scores against a protected appropriate final test. Document selection and uncertainty instead of reporting the best tuning score as an external guarantee.

For adjacent challenges, continue with data leakage and grouped splitting and permutation importance. They answer different questions but share the discipline of making the information boundary visible.

The broader How AI Works guide explains how data, models and independent checking fit together. All figures in this chapter are teaching calculations, not unreported execution of a dataset or library.