A trustworthy model score begins with an honest separation of the information a prediction could have used from facts discovered later. Use the contents below to inspect a real evaluation procedure, or work through the fictional cases line by line.
Contents and reading path
What this chapter solves · Choose the reader’s route · A prediction has an information horizon · Three different paths by which information crosses the boundary · The unit of prediction is not always the row · A fully specified fictional dataset · Worked score: a bright number and a darker truth · What if households have different numbers of records? · Future timestamps and labels created after the event · Preprocessing can leak without reading the answer · A pipeline is a boundary, not a decoration · Group-wise cross-validation and its real limits · Separate development choices from the final report · Chronology changes the question · Near-duplicate records hide behind different identifiers · Avoid outcome-dependent filtering · What to save in an evaluation evidence ledger · Diagnose an implausibly good score without guessing the cause · Repair the split without concealing the previous error · Independent practice: reconstruct a contaminated split · Practice solution with the denominators made visible · Changed-context transfer: a product-support example · Changed-context solution and realistic failure modes · What the evidence still cannot establish · Frequently asked questions · Official reading and onward routes
What this chapter solves
Data leakage occurs when a model, its training process or its evaluation uses information that would not legitimately be available for the prediction being tested. The result can look wonderfully accurate during development and disappoint people the moment the system meets genuinely new cases. The remedy is not simply to hide a column named “answer”. Define when a prediction happens, identify what is knowable at that moment, separate related people or records appropriately, and fit every learned transformation using training data alone.
This guide concentrates on a particular reader job: designing an evaluation boundary that survives duplicates, repeated observations, later updates and preprocessing. It complements the wider How AI Works overview, but does not repeat the complete AI pipeline. We will use an explicitly fictional student-support dataset, calculate what its validation percentages mean, construct a grouped split, inspect an innocent-looking future field, and work through what to fix if a promising model fails in use. No figures here are claims about a commercial model or a real school.
Choose the reader’s route
If you are a parent, teacher or student, begin with the examples of repeated pupils, chronological events and a score that seems too good. They show why a model trained on old answers can appear clever without having learned a transferable relationship. If you build models, continue through grouping, transformations, cross-validation and the evidence ledger. If you make decisions about deploying models, concentrate on the acceptance gate, what each score actually measures, and the unresolved risks after release.
A short route through this article is: define the prediction moment; identify related rows; choose the split; fit only on training observations; calculate performance on untouched cases; challenge the design with changed conditions; document a decision. Reading that route backwards also works as an audit: begin with the advertised score and ask which boundary was actually protected at every prior step.
A prediction has an information horizon
Imagine predicting whether a pupil will complete a particular support exercise by Friday. On Monday, you may know the prior week’s attendance, available study time, relevant practice attempts and the exact task assigned. You cannot legitimately know whether the pupil submitted on Thursday or whether a tutor later wrote “completed” in a comment. The prediction moment is Monday. Information created after that moment is outside the admissible horizon, even when it appears beside legitimate fields in a spreadsheet.
That simple time question catches several errors. A model might learn from a course-completion timestamp that was entered after the outcome. A dataset may have been joined to an end-of-term report showing the answer. A “latest status” field may include changes that were not known on Monday. There is no automatic moral judgment in the word leakage: a late field can be excellent for retrospective description. The mistake arises when that field is treated as evidence for a forward-looking prediction.
Before collecting features, write a one-sentence contract: “At time T, using facts observable and authorised by T, predict event Y during interval T to T plus H for unit U.” Specify who requests the prediction and what decision follows. Without that sentence, a technically flawless split may still be testing the wrong problem.
Three different paths by which information crosses the boundary
First, feature leakage puts the answer, or a close consequence of it, into the model’s input. A field such as case_closed_at may practically reveal a case’s completion status. Second, split leakage lets highly related observations appear in both training and testing. Four records from the same pupil may share a distinctive school, timetable, handwriting description or correction pattern that lets the model recognise that individual. Third, procedure leakage arises when the process learns about the test fold through normalisation, feature selection, imputation or model tuning.
These paths require different repairs. Dropping future fields does not fix a child represented on both sides of a split. Separating pupils does not stop a global average calculated from all rows before cross-validation. Wrapping preprocessing in a proper pipeline cannot repair a label that was generated using future information. A team should classify the suspected leak before choosing a control. The first useful question is not “Which algorithm failed?” but “Which information crossed which boundary, by which operation, and when?”
Sometimes there are several boundaries: a repeated customer, a shared clinic, a family, an academic year and a future period. Their importance depends on the intended deployment. A model predicting a new visit by a known person differs from a model predicting for a completely new person. That difference must be named, or apparent agreement between scores is misleading.
The unit of prediction is not always the row
Spreadsheets encourage a row-centred view: split 80% of rows for training and 20% for testing. But rows can be correlated. A pupil might contribute six exercises; a patient may have many appointments; a photograph can be cropped into twenty overlapping images; a customer may open several related requests. If the purpose is to generalise to previously unseen pupils, the pupil rather than the individual record is the independent unit that must stay entirely on one side of the test boundary.
The unit can be even larger. If siblings share a household environment and a model is supposed to generalise to new households, a household identifier may be the grouping unit. If repeated measurements come from one device, a machine ID may be relevant. Conversely, a grouping rule can be unnecessarily strict for a service deliberately predicting new observations from existing known devices. State the reader job before choosing a group key. Calling every row independent by default conceals the question the evaluation is meant to answer.
One practical test is to ask whether a human who had seen the training rows could recognise a held-out row through a stable signature. If so, the held-out row may not represent the novelty promised by the score. Statistical dependence and memorisation are distinct phenomena, but both can weaken the claim that the test is genuinely out of sample.
A fully specified fictional dataset
Consider a teaching demonstration with 60 fictional households, four weekly observations per household: 60 × 4 = 240 rows. Each row records information known on Monday and whether a designated practice task was completed by Friday. There are 120 positive and 120 negative outcomes across the entire teaching dataset. Those totals are stipulated for arithmetic practice, not estimated facts about students. Every row belongs to exactly one household. We want to make a claim about households the model has never seen, not merely about a new week from a familiar household.
A naive row split could allocate 180 observations to training and 60 to testing while placing weeks from the same household on both sides. The count 180 + 60 = 240 tells us nothing about independence. A particular household might have three rows in training and its fourth in testing. If household routines are stable and informative, a flexible model can exploit them. The leakage risk follows from the relationship between records, not from any secret identifier magically being present in the inputs.
Set the proper grouping plan instead: hold out 15 complete households for the final test, exactly four records each, for 15 × 4 = 60 rows. Use the other 45 households, 45 × 4 = 180 rows, to develop the model. The number of rows has not changed, but the meaning of the evaluation has. There are now zero household IDs shared between train and final test. Any cross-validation used for model selection within the 45 development households must also preserve those group boundaries.
Worked score: a bright number and a darker truth
Suppose the naive row-based experiment reports 54 correct classifications among 60 test observations. Accuracy is 54 ÷ 60 = 0.90 = 90%, with six incorrect outcomes. On a separate household-held-out evaluation, suppose the same design family achieves 39 correct classifications among 60 observations. Accuracy is 39 ÷ 60 = 0.65 = 65%, with 21 incorrect. The contrast is 25 percentage points, calculated as 90% − 65%. It is not a 25% relative fall: the relative difference depends on the chosen denominator.
These counts are constructed examples, not performance measurements. The two test populations differ, and changing a split can alter class balance, difficulty and randomness. Therefore the 25-point gap does not isolate the causal contribution of leakage. It is a diagnosis prompt: the evaluation that mixed households may have overstated generalisation to new households. Repeating a properly specified group-held-out evaluation across sensible resamples can estimate variability; a clean, never-used final holdout remains essential for the final claim.
Now inspect denominators. Are 39 correct rows out of 60 independent household-level trials? No: each of 15 households contributes four correlated rows. Row accuracy weights a four-record household four times; because all households here contribute equally, it coincides with an equally weighted average of their individual row accuracies. It does not supply 60 independent households. A confidence interval that assumes 60 independent units would misrepresent uncertainty. Report both record and household counts.
What if households have different numbers of records?
In a realistic dataset, some people appear twice and others twenty times. Row accuracy then answers, “What fraction of the observed interactions did we classify correctly?” A household-level average answers, “How did we perform for a typical household in this evaluation?” Those are different targets. Imagine household A contributes 20 interactions with 18 correct, while B and C contribute five each with two correct per household. Interaction accuracy is (18 + 2 + 2) ÷ (20 + 5 + 5) = 22/30 ≈ 73.3%. Mean household accuracy is (18/20 + 2/5 + 2/5) ÷ 3 ≈ 56.7%.
Neither number is universally right. A help desk paying for each interaction may legitimately care about row-weighted volume. A school making claims about equitable household support may need group-weighted reporting. For a high-stakes choice, present both and inspect the worst-served groups with adequate privacy protection. The split and the metric should be chosen together, because changing the unit of generalisation after seeing the result is a form of post hoc target shifting.
Future timestamps and labels created after the event
Now consider the feature table. A column named `last_contact_time` might seem benign, yet it could record the date of a telephone follow-up after a pupil missed Friday’s task. It was not known on Monday. Likewise, `case_duration_days` may require the case’s closing date, and `next_week_teacher_note` is obviously later information. Even if none is literally the label, each could act as a proxy for the outcome. A model can then receive a clue from the future while the analyst believes it is discovering early predictors.
Define an as-of join: for each prediction at Monday 09:00, include only source events with effective availability by Monday 09:00, under the actual product’s rules. This is stronger than filtering solely by a row’s recorded event time if ingestion happens later. Imagine a practice score taken on Sunday but only uploaded on Tuesday; it may not have been available for Monday’s live decision. Conversely, corrections to historic records need version history so a later revision is not silently treated as knowledge available at the earlier time.
Testing this requires examples that are deliberately awkward: late uploads, time-zone boundaries, retroactive corrections, missing timestamps, and a case reopened after closure. An as-of join is a temporal data contract, not merely a database convenience. The failure mode is logical, so a perfectly calculated accuracy cannot correct it.
Preprocessing can leak without reading the answer
A surprising leak arises even with no target labels in the transformation. Suppose the 180 training rows have a mean value of 10 for a permitted numeric feature and the 60 held-out rows have a mean of 18. The combined mean is (180 × 10 + 60 × 18) ÷ 240 = (1,800 + 1,080) ÷ 240 = 12. A normaliser that computes its centre from all 240 rows has learned the test population’s centre. Using 12 as a training statistic allows the held-out data to influence model development, even though no test labels were read.
The appropriate procedure is to fit the transformation on training observations only. In this example the training mean is 10. Store that fitted value and apply the same transformation to the held-out observations; do not refit it on them. The same logic applies to imputers, learned category vocabularies, feature selectors, principal-component transforms and other operations that estimate parameters from data. Some fixed, a priori transformations are different: converting centimetres to metres using a known factor does not estimate anything from the test fold.
The official scikit-learn guide to common pitfalls explicitly warns against test-data leakage during preprocessing and recommends fitting pipelines inside the training fold. The lesson extends beyond a particular library. Wherever there is a learned statistical state, ask which rows influenced its creation.
A pipeline is a boundary, not a decoration
A well-designed modelling pipeline packages operations so every fold fits a fresh scaler, imputer or selector using its own training portion, then applies the resulting transformation to that fold’s validation portion. This is why pipeline objects are more than syntactic neatness: they make the order and boundary of fitted operations visible. It is still possible to leak by assembling an already-transformed table from all data before the pipeline begins, or by calculating target encodings using the whole dataset. The presence of an object called Pipeline is not itself proof of safety.
Think of each fold as a small laboratory. It starts with unused validation records, fits all learnable preprocessing on the fold’s training records, selects and trains its model under the stated procedure, and evaluates once on its own validation rows. No information returns from the validation laboratory to redesign the same score without appropriate nested selection. A practitioner can check the fitted object provenance or audit which row indices contributed to each fitting step instead of trusting naming conventions.
For worked implementation details, use the official pipeline and composite estimator documentation. Reproduce the principle in any library that provides fit and transform operations. The educational goal is to explain why an apparently harmless mean or feature ranking can make held-out results too optimistic.
Group-wise cross-validation and its real limits
When the deployment requires new households, a group-aware cross-validation method keeps all records from the same household together within a fold. The official GroupKFold documentation describes splitting data into folds with non-overlapping groups. For the 45-household development set, five folds of nine households each are possible in this equal-size illustration. Each validation fold has 9 × 4 = 36 rows, while its companion training portion has 36 × 4 = 144 rows. Across the five validation folds, each of the 45 development households is tested once.
That produces validation diagnostics for model selection, not five independent final external proofs. The folds share much of their training data, and the household population may still be unlike households encountered next year. GroupKFold does not automatically enforce chronology, protect against a dataset-wide transformation performed before calling it, or identify hidden duplicates with missing group IDs. It enforces one specifically declared relationship. Researchers must still decide whether household, class, school, device or project is the correct unit.
For severe class imbalance, a stratified grouped splitter may be considered; however, exact class proportions and strict group separation can conflict. Inspect the actual fold membership, class counts and available positive cases rather than assuming the method name guarantees a perfect balance. For extremely few distinct groups, report the uncertainty instead of pretending a conventional five-fold score is stable.
Separate development choices from the final report
Training data fits the model. Validation data helps choose model variants, transformations, thresholds or tuning settings. A final test is reserved to estimate a previously specified claim after those choices have been fixed. Repeatedly inspecting the final test and revising the model in response turns that test into another development set, whether or not the analyst calls it “validation”. The problem is selection: we gradually choose what happens to look good on a supposedly independent set.
A practical team can write a release card before evaluation: prediction target; feature availability cut-off; group key; anticipated future users; metric; minimum useful performance; disallowed harms; date and method of final split; and the decision maker. Lock the test labels and avoid casual peeking. If fundamental defects are discovered in the test design, repair the procedure transparently and establish a new independent evaluation set where feasible. Do not quietly quote the best result after many unreported attempts.
This does not mean a final test must be sacred forever. After release it may become a historical benchmark, and retraining may include older formerly held-out observations if a fresh forward evaluation is created. The important part is honest naming: the same data cannot simultaneously be a source of tuning decisions and an untouched demonstration of generalisation.
Chronology changes the question
Suppose outcomes depend on school terms, curricula, seasonality or an application redesign. A random household-held-out split can still mix earlier and later periods; it assesses one kind of novelty, not necessarily future performance. To estimate next-term deployment, make a cut between periods and fit only on data that would have existed before the cut. Reserve a later period for prospective-style evaluation. If a household repeats across the time boundary, decide whether the intended user is an existing household’s new week or an unseen household; those may require distinct tests rather than one convenient hybrid score.
Temporal splits demand caution with overlapping windows. A feature defined as “average activity over the last 14 days” and an outcome defined over the next seven days can share information across neighbouring prediction dates. That does not automatically constitute target leakage, but cross-fold overlap and delayed label availability may make the apparent independence misleading. In some designs, a gap or purge interval between development and evaluation periods is necessary. Its length should follow the feature lookback, label horizon and actual availability delays, not a fashionable fixed number.
Inspect the TimeSeriesSplit documentation for the mechanics of time-ordered folds. A time-ordered algorithm does not know the business semantics of your late-arriving labels. You must encode those semantics yourself.
Near-duplicate records hide behind different identifiers
Deduplication is not as simple as dropping identical rows. A support conversation may be exported twice with different internal IDs. An image may be resized, compressed or cropped. A document may have been copied into separate folders or lightly paraphrased. If training contains one version and testing contains another, the model’s apparent performance may reflect recognition of the repeated source rather than generalisation to new material. Hashes catch exact copies; perceptual or semantic similarity reviews may catch near-duplicates but also introduce false matches.
Define an identity hierarchy: exact source ID, original document ID, household or account, shared content source, and close-content family. Choose a conservative grouping rule for the specific claim. A team can audit a sample of the nearest train-test neighbours, ask whether each pair is legitimately separate, and revise deduplication before quoting a final score. The false-match cost also matters: collapsing two genuinely different cases because their wording is similar may hide useful diversity.
Do not claim that every semantic similarity is a leak. Two questions asking “What is 4 plus 5?” may be legitimately common and independent. Leakage becomes important when the evaluation promises novel sources or users but the split supplies recognisable variants of those same sources. Document the claim, then judge proximity against it.
Avoid outcome-dependent filtering
Another subtle failure begins when the dataset is created after the target is known. Suppose a system exports only cases marked “fully resolved”. If the eventual deployment must predict resolution for every incoming case, this export omits failures, unresolved cases and people who never returned. The resulting sample may have a distorted difficulty profile even if there is no shared row between training and test. This is selection bias, distinct from classic column leakage but capable of producing the same false confidence.
Ask who is absent. Were records dropped because a field was missing only for difficult cases? Were outcomes unknown for people who disengaged? Were interventions offered to some users, changing which results became observable? A complete audit should preserve counts before and after filtering with reasons, and report outcome observability as an explicit limitation. Never silently equate “not observed to fail” with “successful”. For some tasks the appropriate response is to abstain until enough evidence arrives.
This concern has a useful boundary with the published agent outcome feedback article. That lesson treats monitoring decisions after agents act; this one treats whether the model development and evaluation datasets had a valid information and sample boundary before deployment.
What to save in an evaluation evidence ledger
A reproducible evaluation needs more than a score and a screenshot. Record the data extraction timestamp, the authorised source versions, the row-level prediction time, unit of generalisation, group key, duplicate policy, development/test split rule, list of features and their earliest availability, preprocessing fit scope, validation design, frozen tuning decisions, target definition, metrics and denominators. Keep a reviewable record of exclusions and any repeated test exposure.
An analyst should be able to select one held-out prediction and trace backwards: which raw records existed at the prediction time; which transformation parameters were used; which training fold estimated them; which group boundaries protected this record; what outcome became known later; and how the metric counted the case. If one step cannot be reconstructed, the deployment claim may be weaker than the attractive number implies. A high-level model card can summarize these facts for a reviewer without exposing sensitive personal data.
The NIST AI RMF Playbook is a voluntary risk-management resource organised around Govern, Map, Measure and Manage; it supports systematic, documented evaluation. It does not certify that a particular model is leak-free. Treat documentation as something that helps other people check the evidence, not as a substitute for the check.
Diagnose an implausibly good score without guessing the cause
A suspiciously high score deserves examination but is not proof of leakage. Some tasks are genuinely easy under the stated conditions. Begin by reconciling the evaluation sample: number of unique people or devices, duplicate clusters, label balance, timestamps, feature origins and missing records. Next inspect which features dominate predictions. A field representing a future disposition, a nearly perfect timestamp proxy or a constant matching an outcome code should prompt source tracing. Then repeat a validation using a stricter group and time boundary.
Compare not only accuracy but failure patterns. Does performance drop on entirely new households? Are scores lower on data recorded after a product change? Do cases with late uploaded information behave differently? Did the team tune a threshold against the final held-out set? Look for the earliest invalid operation, because downstream repairs cannot erase an earlier leak. Do not simply delete the most predictive feature if it is legitimately available and valuable; document why its availability is valid or invalid.
One reasonable audit proceeds in an ordered sequence: validate extraction, recover event chronology, map identities and duplicates, check split membership, replay transformations within training folds, verify tuning history, recalculate metrics and repeat changed-condition testing. If no issue is found, keep the result but narrow its interpretation to the tested population and timeframe.
Repair the split without concealing the previous error
When leakage is found, preserve the original result in an internal correction record and mark why it was invalid for its advertised claim. Identify the specific boundary violation: late feature, shared source, all-data fit, leaked target, repeated test tuning or nonrepresentative sampling. Construct a new clean dataset from source versions and regenerate splits using stable identifiers or timestamps that encode the deployment contract. Every learned transformation must be refit within training data only.
Report the new evaluation separately from the old one. If the new sample is smaller, explain how many independent groups remain. If missing labels create uncertainty, publish that limitation instead of silently imputing a reassuring denominator. Check feature drift, repeated records and population coverage before declaring the repair complete. For a very small sample, qualitative error analysis or a constrained trial may be more appropriate than a precise-looking percentage.
Finally, update process controls: an as-of feature view, grouped split unit tests, pipeline construction checks, a registry of data sources and a ban on using final test information for selection. A reviewer should be able to test the repair on deliberately adversarial example rows. A repaired score is not evidence of broader safety, fairness or educational benefit unless those outcomes were also independently evaluated.
Independent practice: reconstruct a contaminated split
Here is a self-contained exercise. A fictional reading-support model is evaluated on 40 test observations, of which 34 are classified correctly. The model’s team reports 85% accuracy. Ten pupils supplied four observations each. Inspection reveals that eight pupils have earlier observations in the training data; only two pupils are genuinely new. A numeric feature `final_teacher_rating` was recorded at the end of each week, after the Monday prediction moment. The training-data scaler was also fitted to the entire table before splitting.
Identify at least three distinct reasons why 34/40 is insufficient evidence of generalisation to new pupils. Name the valid information horizon and the appropriate independent unit. Suggest a corrected evaluation procedure that can support a modest claim without pretending the same forty observations remain a valid untouched holdout. Then describe which numerical facts you may still report and which you must not reinterpret.
Do the exercise before reading the solution. The goal is not to memorise one pipeline function; it is to learn to recognise when apparently acceptable rows, columns and transformations collectively violate the intended use of a model.
Practice solution with the denominators made visible
The raw arithmetic is correct: 34 ÷ 40 = 0.85 = 85%. The claim is not. Eight of ten test pupils have appeared in training, so at least part of the test evaluates familiar pupils. The end-of-week teacher rating is future information unavailable at Monday’s prediction time. The scaler learns from the complete test table, violating the fit boundary even if no labels are visible during scaling. These are separate failure paths; fixing one leaves the others.
The prediction horizon is Monday to Friday. Retain only attributes available by the actual Monday cut-off, group by pupil so no pupil overlaps between model development and the final test, and fit preprocessing on training folds. Freeze feature rules and tuning before running a genuinely independent new-pupil test. If the original test has informed feature choice, a fresh untouched set is preferable. If a new set is infeasible, the team must explain that evaluation is exploratory rather than an independent final estimate.
The only unconditional numerical fact is that this contaminated setup produced 34 correct labels among 40 observations. We cannot derive the valid new-pupil accuracy, a confidence interval about the relevant population or the improvement from one repair without additional data. Pretending that the two unfamiliar pupils give 8 independent test observations would still confuse row count with pupil count.
Changed-context transfer: a product-support example
Move from pupils to a customer-support classifier that predicts whether a newly opened ticket needs specialist review within 48 hours. Each customer opens several tickets; the company introduced a new triage form in September; some tickets are reopened after an initial resolution. A field named `last_queue` records the final queue after review, and a field named `customer_tier` is available immediately at ticket creation. The business wants to estimate accuracy on new customers next month, using only ticket creation information.
Construct the evaluation: group customers, hold out a later month, exclude the post-decision last_queue field, preserve the creation-time customer_tier, and define a rule for when the 48-hour outcome is considered fully observed. If new-customer data for next month has not yet arrived, the honest answer is that this exact prospective question cannot be finally tested today. Historical backtesting can inform planning but should be labelled accordingly.
The subtlety is that the product change and customer grouping constrain two dimensions at once. A random group holdout across years may not represent the new form. A future-month holdout with known customers may not represent new customers. Plan both analyses if there is enough data and keep their claims separate; do not paste two percentages together as if they proved the same outcome.
Changed-context solution and realistic failure modes
First write the contract: at ticket creation, predict specialist-review need within 48 hours for a customer not previously in the model development sample, under the current triage form. Candidate features are those logged by creation time; `customer_tier` may be legitimate if it truly existed then and its use is authorised. `last_queue` is forbidden because the final queue is determined after triage. Group by stable customer identity, inspect linked accounts and aliases, and hold out only complete future observation windows so a ticket created near the extraction date is not incorrectly marked negative.
A valid estimate for next month requires outcomes from next month. Historical validation might use older unseen customers under the current form, with explicit limitations. For an operational pilot, measure errors and subgroup burdens while preventing the system from making uncontrolled consequential decisions. Diagnose failures by ticket age, customer type, form version and visibility of the eventual outcome. Do not infer that a single statistical split solves distribution shift created by a product redesign.
The transfer lesson is durable: whenever the decision moment, grouping unit or process version changes, the old evaluation contract may no longer apply. Strong machine learning practice updates the contract and reruns evidence rather than reusing an impressive historical score without checking its boundaries.
What the evidence still cannot establish
A leak-free train/test split does not prove that a model is equitable, secure, well-calibrated, robust to future change or beneficial for learning. Nor does it guarantee that feature permissions and privacy requirements are satisfied. A clean split answers a narrower question: did the stated evaluation avoid specific pathways by which development had access to information it should not have? Further investigations may still be needed for harms, subgroup error rates, adversarial inputs, access controls and actual human outcomes.
Likewise, discovering leakage does not mean all earlier work was worthless. It means one inference was unsupported. A team may keep the feature engineering ideas, re-collect evidence properly and report what changed. What matters is honesty about the step that was invalid. A carefully bounded 65% estimate may be more useful for staffing and support decisions than an alluring 90% estimate from familiar records. But even 65% cannot be judged adequate without a benchmark, context and the costs of different errors.
Artificial intelligence systems currently used in products are real, but hypothetical artificial superintelligence is a separate topic. The methods in this chapter are ordinary machine-learning quality controls that apply regardless of promotional claims about model intelligence.
Frequently asked questions
Is data leakage the same as a privacy leak? Not necessarily. In machine learning, this chapter uses leakage to mean forbidden information entering model development or evaluation. A privacy breach is inappropriate disclosure or use of protected information. They can interact, but one term does not prove the other occurred.
Does GroupKFold prevent every form of leakage? No. It protects the supplied group boundary. It does not repair future features, all-data preprocessing, hidden shared identities, invalid targets or a misdescribed deployment population. The group key itself must be correct and stable.
Can I use a random split when records are related? Sometimes, if the intended prediction is another observation from an already-known unit and the validation is designed to match that actual scenario. State the claim. A random row split cannot honestly be described as demonstrating performance on new people when it contains old people.
Why not just use the most recent observations as the test set? Time ordering is often appropriate for future deployment, but repeated entities, late labels and post-outcome fields can still leak. The right split may need both grouping and chronology, with adequate sample size and documented trade-offs.
Does a large dataset make leakage unimportant? No. More rows may replicate the same people, documents or events. An enormous row count can hide a small number of independent sources. Report effective units and the kinds of novelty the evaluation actually tested.
How should a student study this? Draw a timeline with the prediction moment, label observation and feature availability. Then draw lines showing which records belong to the same person. Explain aloud why each line may or may not cross the training/test boundary. That exercise transfers to medical studies, business forecasts and classroom assessments.
Official reading and onward routes
The official scikit-learn Common Pitfalls guide explains leakage from fitted preprocessing and test exposure. The cross-validation user guide explains grouped and time-aware evaluation. The GroupKFold reference states the non-overlapping-group guarantee. The Pipeline API reference documents ordered transformers and estimators. The NIST AI RMF Playbook supplies a voluntary governance context for documenting evaluation claims and risks. These are supporting sources, not assertions that a particular eduKate dataset or tool was tested.
Continue with training label quality when the problem is ambiguous annotations rather than contaminated splits. Read classification thresholds and base rates when a clean score still needs to become a human decision under review capacity. The How Intelligence Works hub connects uncertainty, correction and knowledge transfer beyond machine models; the How X Works hub extends first-principles practice across subjects.
Good learning transfers into the ability to check a claim outside the original example. A secondary-school student can learn to question a denominator; an educator can inspect whether repeated pupils inflated an assessment; an aspiring data scientist can build a reliable validation pipeline; a decision maker can require a precise evaluation contract before trusting an attractive number. Visit The Gold Standard of Education to connect that habit of checking with wider education and work. The purpose of all these routes is not to distrust every model: it is to learn exactly what a result does and does not prove.