VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

How Super Intelligence Works | Training Data — What Models Learn From, What Gets Filtered and What Can Go Wrong

Three students work together at a classroom table, reviewing open textbooks and notes.

Training data is the experience from which a machine-learning model learns statistical structure. It shapes vocabulary, domain coverage, biases, factual patterns, code styles, languages and many of the capabilities a model can later reuse. But “more data” is not a complete strategy. Source quality, mixture, duplication, provenance, rights, privacy and evaluation contamination all matter.

A model cannot learn a pattern it never encounters, and it can learn undesirable patterns that appear repeatedly. Training data therefore belongs inside the core mechanism of Super Intelligence, not as an afterthought outside the model.

This article explains how training data works: collection, cleaning, filtering, deduplication, sampling, token balance, quality, coverage, provenance, contamination, synthetic data, personal information, licensing and data documentation. It also shows why data quantity and model quality do not have a simple one-to-one relationship.

The research proposal Datasheets for Datasets argues that datasets should be documented with their motivation, composition, collection process, recommended uses and other context. That discipline is essential when one dataset can influence a model used by millions of downstream tasks.

Previous: 021 — Pretraining. Here we zoom in on the material being learned from.


The Hidden Transition: Data Is Not Neutral Raw Material

A dataset is created through choices. Someone decides where to collect from, which formats to include, what to remove, how to label language, whether to preserve metadata, how to detect duplicates and how to weight each source during training.

Those choices change the learning environment. A corpus dominated by technical documentation produces different gradients from one dominated by conversational forums. A multilingual model trained mostly on English cannot be assumed to serve every language equally.

Training data is therefore an engineered input to the model’s development, not a passive mirror of the world.

Source Collection

Data can come from licensed corpora, public web pages, books, code repositories, academic material, human-created datasets, synthetic examples, internal records or other sources depending on the model and permissions.

Each source has strengths and failure modes. Documentation can be precise but narrow. Forums can contain diverse everyday language but misinformation. Code can provide executable structure but include insecure patterns. Books can supply long-form coherence but vary in age and domain.

A good mixture uses sources because they serve a learning goal, not merely because they are easy to collect.

Coverage: What Worlds Does the Dataset Represent?

Coverage asks which languages, domains, styles, populations and time periods are present. A model trained on school science textbooks may understand formal explanations but struggle with casual student phrasing. A model trained heavily on US sources may mishandle Singapore-specific educational terms unless those are represented.

Coverage also has depth. Seeing one document about a niche topic is different from seeing thousands of varied examples that teach robust patterns.

Evaluation should therefore be segmented by domain and population. One global score can hide weak coverage.

Frequency Shapes Learning Pressure

Training updates occur when examples are sampled. Common patterns appear more often and generate more gradient signal. Rare languages, rare syntax or niche facts can receive fewer updates.

Developers can alter sampling rates so smaller high-value datasets receive more effective exposure. This means raw corpus size and effective training mixture are not always identical.

Data Quality Is Task-Dependent

“High quality” does not mean one universal writing style. Conversational data may be valuable for dialogue. Messy error logs may be valuable for debugging. Informal explanations may be valuable for tutoring.

Quality means useful for the intended learning objective while controlling noise, harmful content and misleading patterns.

A dataset cleaned so aggressively that it removes informal language can reduce robustness to real users.

Cleaning and Normalisation

Raw data contains broken encoding, duplicated menus, navigation text, corrupted files, malformed code, boilerplate and extraction artefacts. Cleaning removes or repairs material that would waste training signal.

Normalisation can standardise encoding or formats, but excessive normalisation can erase meaningful information such as case, punctuation, code indentation or multilingual distinctions.

The right preprocessing depends on what the model needs to learn.

Deduplication

Large collections contain exact copies, mirrors, syndicated posts and near-duplicates. Deduplication reduces repeated material so a small set of pages does not dominate training.

It can also reduce memorisation risk and benchmark contamination. But deduplication is approximate. Two documents can express the same content with different wording.

A useful pipeline distinguishes exact byte-level duplicates, near-duplicates and semantic repetition rather than pretending one filter solves everything.

A Tiny Deduplication Example

Suppose a corpus contains 100 copies of the same tutorial, 5 paraphrases and 20 genuinely different explanations. Without deduplication, the repeated tutorial can dominate the gradient signal even though it adds little new information.

Removing 99 exact copies creates a more balanced effective dataset. Whether to remove the paraphrases depends on the training goal because linguistic variation can be useful.

Data Mixtures

A foundation model may combine general web text, code, mathematics, scientific papers, books and dialogue. The sampling mixture determines how often each source family appears during training.

A code dataset representing 5% of raw bytes might be sampled at 15% of training tokens if programming capability is a priority. Another model might emphasise multilingual content.

Mixture design is one of the ways developers shape capability before post-training.

Language Balance

A corpus can contain hundreds of languages while still being functionally English-dominated. Token counts, document quality and domain variety matter.

Low-resource languages can suffer from sparse data, poor tokenisation and limited evaluation. Translating English data synthetically can help coverage but may import translation artefacts and cultural assumptions.

Time Coverage and Knowledge Freshness

Training data has a temporal boundary. A model cannot directly learn events that occur after the collected data unless later training includes them.

Older sources can also remain useful for historical knowledge while being dangerous for current policy or prices. The model does not automatically know which statements have expired.

Current applications therefore combine pretrained knowledge with retrieval or databases.

Provenance: Where Did This Example Come From?

Provenance records source identity, collection method, date, licence, transformations and other relevant metadata. It supports audits, removal requests, quality analysis and reproducibility.

Without provenance, a problematic example can be difficult to trace. A model developer may know that an error exists but not which source family introduced it.

Dataset documentation creates organisational memory around the data itself.

Data Rights and Licensing

Technical accessibility does not automatically settle whether material may be used for training. Copyright, contracts, database rights and licences vary by jurisdiction and source.

A robust data pipeline records rights information and applies policy before training rather than assuming public visibility equals unrestricted permission.

This article describes engineering structure, not legal advice. Model developers need qualified legal review for the jurisdictions and data sources they use.

Personal Information

Public and private datasets can contain names, addresses, phone numbers, health information or other sensitive details. Training-data pipelines need rules for identifying and handling personal information.

Removing obvious identifiers can reduce risk, but privacy is broader than a regex that deletes email addresses. Rare contextual combinations can still identify people.

Data minimisation and purpose limitation should happen before training whenever possible.

Toxicity and Harmful Content

Real-world corpora contain harassment, slurs, extremist material, scams and dangerous instructions. Removing all harmful material is neither trivial nor always desirable for models that must recognise and moderate it.

The design question is what exposure is needed for capability and how the model should behave at deployment. Filtering, weighting, post-training and safety evaluation work together.

Bias Through Representation Imbalance

If one profession is repeatedly associated with one gender or one region is described mostly through negative news, the model can absorb those correlations.

Bias is not only offensive text. It can arise from underrepresentation, stereotypes, missing perspectives and historical inequality.

Mitigation starts with measuring representation, not merely counting banned words.

Labels Can Contain Human Error

Supervised datasets depend on annotation. Annotators can disagree, misunderstand instructions or encode subjective judgments as if they were objective truth.

Good datasets define annotation guidelines, measure agreement and preserve ambiguity where appropriate.

A single label can hide uncertainty that the model later learns as certainty.

Self-Supervised Data Still Needs Quality Control

Next-token or masked-token objectives derive targets automatically, but the source text can still be wrong, low-quality or duplicated.

“Self-supervised” means labels come from the data structure. It does not mean the dataset is automatically safe or correct.

Synthetic Data

Synthetic data is generated rather than directly collected from real examples. A model can create practice questions, conversations, code, edge cases or reasoning traces that expand a training set.

Synthetic data can target gaps efficiently, but it can also amplify the generator’s mistakes. Verification and filtering become essential.

Article 029 later in the series will examine synthetic data in depth.

Worked Synthetic Example

Suppose a classifier lacks examples of polite scheduling cancellations. A generator produces 10,000 messages such as “I’m sorry, we need to cancel tomorrow’s lesson.”

If all synthetic messages follow the same phrasing, the dataset looks large but contains little diversity. A better pipeline varies wording, time references, reasons and ambiguity, then samples and validates the results.

Data Augmentation

Augmentation modifies existing examples while preserving the relevant label. Images can be cropped or rotated. Text can be paraphrased carefully. Audio can be perturbed with noise.

The transformation must preserve the task semantics. Changing a number in a maths problem can change the answer; paraphrasing a legal clause can change meaning.

Train–Validation–Test Separation

Training data updates parameters. Validation data guides development. Test data estimates final performance. Leakage between these sets weakens the evidence.

Web-scale pretraining makes leakage difficult because public benchmark questions may already exist online. Contamination analysis and private evaluation sets are therefore important.

Benchmark Contamination

The GPT-3 paper explicitly investigated data contamination because benchmark examples can appear in large web corpora. If the model has seen the exact question and answer, a benchmark may measure recall rather than generalisation.

Contamination can be exact or approximate. Near-duplicate wording can still leak task content.

Evaluation reports should state contamination controls when the result matters.

A Worked Contamination Example

Imagine a public benchmark contains 1,000 questions. Search finds 200 exact copies in the training corpus and 150 paraphrased answer pages.

A high test score can no longer be interpreted purely as unseen problem solving. Remove contaminated items, construct protected variants or report the limitation.

Data Leakage Through Metadata

Sometimes the answer hides in metadata rather than content. A filename like “positive_review_004.txt” can reveal the sentiment label.

A model can exploit these shortcuts and appear accurate while learning the wrong feature. Dataset construction should remove unintended leakage.

Spurious Correlations

A model may learn that snow backgrounds correlate with wolves or that one formatting style correlates with a class. The correlation works on the training set but fails when the environment changes.

Diverse data and targeted stress tests help reveal reliance on spurious features.

Long-Tail Examples

Most datasets have common cases and rare edge cases. Rare failures can matter disproportionately in deployment.

A scheduling model may see thousands of ordinary reschedules but few daylight-saving transitions. A medical system may see common conditions but few rare emergencies.

Evaluation and targeted data collection should include important long-tail cases rather than sampling only by frequency.

Data Documentation

A dataset card or datasheet can record why the dataset exists, who created it, what it contains, collection dates, preprocessing, licences, known limitations and recommended uses.

This turns data from an anonymous file into an accountable component of the model pipeline.

Data Versioning

Datasets change. Bad examples are removed, new sources are added, licences change and labels are corrected. Versioning lets a training run identify the exact data state it used.

Without versioning, two model runs with the same code can differ because the underlying corpus changed silently.

Data Lineage

Lineage tracks transformations from source to training example: original document → extracted text → cleaned document → deduplicated record → tokenized sequence.

When a problem appears, lineage helps identify where meaning was lost or data entered without the expected policy.

Sampling Without Replacement and With Replacement

Training can sample examples in different ways. Reusing examples more often increases their effective weight. Rare datasets can be oversampled; huge sources can be undersampled.

The sampling scheme is part of the learning objective even though it lives outside the model architecture.

Data Quantity Versus Information Diversity

Ten million near-duplicate sentences may contain less useful learning signal than one million diverse, well-constructed examples.

Quantity matters, but information diversity and relevance determine how much new structure the model can learn.

High-Quality Small Data After Broad Pretraining

A large foundation model can be adapted with a smaller curated dataset because broad language and world patterns were learned earlier.

This is one reason instruction tuning and domain fine-tuning can work with datasets far smaller than the pretraining corpus.

Data and Tokenisation Interact

A corpus is consumed through a tokenizer. Rare languages or technical strings can expand into many tokens, changing how much context they occupy and how often subword pieces are updated.

Data coverage alone is not enough. Representation quality matters too.

Data and Architecture Interact

A long-document dataset does not guarantee long-context capability if the architecture or training sequence length is short. An image-text dataset does not help a text-only architecture see images.

Training data must match the model’s representation and objective.

Data and Post-Training Interact

Pretraining may expose the model to unsafe instructions and harmful text so it can recognise them. Post-training can then shape how the assistant responds.

The final behaviour is the product of multiple datasets serving different stages.

Data Deletion and Model Updates

Removing a source from a future training dataset does not automatically erase its influence from an already-trained model. Parameter updates have already occurred.

Model developers may need retraining, targeted unlearning research or other mitigation depending on the requirement. Data governance should therefore act early.

A Training Data Failure Map

Level 1: wrong source rights. Level 2: poor extraction. Level 3: duplicated data. Level 4: weak coverage. Level 5: harmful or biased mixture. Level 6: label error. Level 7: leakage or spurious shortcuts. Level 8: train-test contamination. Level 9: missing lineage. Level 10: downstream use exceeds dataset scope.

This eduKateSG map is designed for diagnosis. “Bad data” is too vague to repair.

Worked Diagnosis: Model Fails Singapore School Terms

Suppose a model understands general education but confuses Full Subject-Based Banding terms used in Singapore.

Inspect whether Singapore-specific educational material was sufficiently represented, whether the tokenizer handles local terms well and whether current policy should be retrieved rather than memorised.

The repair may be targeted data, retrieval or both—not simply a larger model.

Worked Diagnosis: Model Repeats One Website’s Style

If a distinctive phrase appears unusually often in outputs, check whether a source was heavily duplicated in training.

Deduplication and source weighting can reduce overrepresentation.

Worked Diagnosis: Benchmark Is Suspiciously Easy

If evaluation performance is dramatically higher than private paraphrases, investigate contamination or label leakage.

A benchmark score is evidence only under a credible data-separation protocol.

A Practical Dataset Audit

Ask: why does this dataset exist? Which sources are included? What dates are covered? Which languages and domains dominate? How were duplicates removed? What filters were applied? What personal information may remain? What rights attach to the sources?

Then ask: how was the dataset split? What benchmarks may overlap? Which transformations occurred? Is there a version identifier and lineage? Which downstream tasks are outside the dataset’s intended scope?

Independent Exercise 1: More Data or Better Data?

A corpus adds 5 million copies of already common news articles. Does raw size necessarily improve the model?

Answer

No. Repetition may add little new information and increase memorisation. Diversity and effective sampling matter.

Independent Exercise 2: Current Policy

Should a changing private company policy be baked into broad pretraining every week?

Answer

Usually not. Keep current policy in an authoritative retrieval source and use pretraining for general capability.

Independent Exercise 3: Dataset Leakage

A sentiment dataset stores positive examples in a folder named positive and the filename is included in model input. What is wrong?

Answer

The label leaks through metadata. The model can exploit the shortcut instead of learning sentiment from the text.

Independent Exercise 4: Synthetic Data

A model generates 100,000 nearly identical maths questions. Is that automatically a strong training set?

Answer

No. Evaluate diversity, correctness and whether the synthetic examples cover useful variations. Quantity alone is weak evidence.

Independent Exercise 5: Contamination

A benchmark question appears verbatim in pretraining. What should the evaluation report do?

Answer

Flag or remove contaminated items, use protected alternatives where possible and state the limitation so the score is not misread as unseen generalisation.

A Complete Dataset Pipeline From Source to Training Tokens

A dataset is not ready for pretraining merely because a file exists. A robust pipeline has stages. Stage 1 records the source and rights. Stage 2 fetches or receives the material. Stage 3 extracts usable content. Stage 4 normalises encoding and structure. Stage 5 filters unwanted material. Stage 6 deduplicates. Stage 7 assigns metadata and quality signals. Stage 8 splits or protects evaluation material. Stage 9 tokenises. Stage 10 samples examples into the training mixture.

Every stage can change the effective learning environment. Extraction can drop table headers. Filtering can remove minority-language material. Deduplication can accidentally collapse useful variants. Sampling can make a small curated source more influential than a large noisy one.

A strong training-data system therefore treats data transformations as versioned code, not invisible housekeeping.

Source Inventory: Build the Map Before the Mixture

Before training, list every source family. For each one, record owner, date range, language, domain, file format, licence or contract, privacy status, expected quality and intended training role.

This inventory makes gaps visible. If the model is expected to understand Singapore education but the inventory contains almost no Singapore sources, the problem is visible before an expensive training run.

It also makes removals manageable. If one source later becomes unusable, lineage tells the team which processed shards came from it.

Document-Level Versus Passage-Level Quality

A high-quality website can contain low-quality individual pages, and a noisy website can contain valuable passages. Filtering only by domain name is therefore blunt.

Pipelines can score documents or passages using language identification, duplication, formatting, toxicity, classifier scores, metadata and heuristic checks.

Automated quality scoring itself can introduce bias. A classifier trained on formal English may mark dialectal or non-Western writing as low quality. Quality filters need evaluation across the populations they affect.

Boilerplate Removal

Web pages repeat navigation menus, cookie banners, footers and related-links blocks. If those appear on every page, the model may waste training capacity learning site chrome.

Extraction systems try to isolate main content. Errors in that process can remove headings, captions or list structure that carry meaning.

A data audit should compare extracted text with original pages for representative formats rather than assuming the scraper preserved everything important.

Language Identification

Multilingual corpora often run language identification before sampling. Misclassification can remove valuable data or place it in the wrong mixture.

Short texts, code-switching and closely related languages are difficult. A Singapore message mixing English, Malay words and Chinese names may not fit a clean single-language label.

The pipeline can preserve uncertainty or multi-language metadata rather than forcing every document into one simplistic bucket.

Quality Filters Can Create Style Bias

Suppose a filter rewards grammatical formal prose. It may keep textbooks and academic articles while dropping chat messages, learner errors and informal explanations.

That can improve formal writing but weaken robustness to real user language. The right filter depends on the model’s intended interaction environment.

A tutoring model may need both polished explanations and authentic student questions, including misspellings and incomplete sentences.

Code Data Needs Its Own Filters

Code repositories contain generated files, vendored dependencies, duplicated libraries, credentials, insecure patterns and test fixtures. A code training pipeline needs domain-specific filtering beyond ordinary text cleaning.

Licences also matter because software repositories often state explicit reuse conditions. Source-level metadata should survive extraction so governance decisions remain possible.

Mathematics Data Needs Structure Preservation

Equations, proof steps, tables and diagrams can be corrupted by text extraction. Losing a minus sign or superscript changes the mathematical meaning.

A pipeline intended to support strong mathematical capability should inspect rendering and representation quality, not merely count how many pages mention mathematics.

Long Documents Need Segment-Aware Processing

Books, papers and manuals contain chapters, sections, footnotes and cross-references. Splitting them into isolated fragments can remove dependencies.

For pretraining, sequence packing may later mix chunks for efficiency, but document metadata can still help preserve boundaries and analyse source behaviour.

Near-Duplicate Detection

Exact hashing catches identical copies. Near-duplicate detection uses methods such as shingling, MinHash, locality-sensitive hashing or embedding similarity to identify documents that differ only slightly.

The threshold is a policy choice. Too aggressive and the pipeline removes legitimate variants. Too weak and repetition remains.

A good audit samples both removed and retained pairs to estimate false-positive and false-negative behaviour.

Cross-Source Duplication

The same article can appear on an original site, a syndication partner, a scraper and an archive. Removing duplicates only within one source misses this global repetition.

Cross-source deduplication also raises provenance questions: which copy should be retained? Prefer the source with clearer rights, metadata or canonical ownership when possible.

Benchmark Protection Before Web Crawling

If public benchmarks are known, a developer can build fingerprints of their questions and answers before assembling the training corpus. Matches can be excluded or flagged.

This is not perfect because paraphrases can evade exact matching, but proactive protection is stronger than discovering contamination after publication.

Private Evaluation Sets

A protected private set can measure generalisation with much lower contamination risk. Organisations can create realistic internal tasks whose answers have never been posted publicly.

The set should still be versioned and access-controlled. Once repeatedly used for tuning, it gradually becomes a development set rather than a truly independent test.

Data Splits Must Respect Groups

Randomly splitting individual rows can leak near-identical examples across train and test. If multiple samples come from the same student, patient, document or code repository, group-level splitting may be more appropriate.

The split strategy should mirror the kind of generalisation the deployment needs. If the model must handle new users, test on users absent from training.

Temporal Splits

For forecasting or current-information tasks, split by time. Train on earlier data and evaluate on later periods. This better simulates future deployment than random mixing.

Temporal evaluation can reveal drift and whether the model relies on patterns that disappear over time.

Counterfactual Data

A dataset can include examples where irrelevant attributes are changed while the correct label stays the same. This tests whether the model relies on spurious correlations.

Example: keep the job qualifications identical while varying names. If predictions change systematically, the model may be using identity signals it should ignore.

Counterfactual examples are useful for both training and fairness evaluation.

Hard-Negative Mining

A classifier often learns more from confusing near-misses than from obvious negatives. Hard-negative mining finds examples the model currently mistakes and adds them to training.

For retrieval, a hard negative might be an archived policy that is almost identical to the current one. Training the retriever to distinguish them can improve ranking.

Active Learning

When human labelling is expensive, the model can identify uncertain or informative examples for annotation. Humans label those cases, and the new data improves the next model.

This concentrates human effort where the model needs it most. The selection strategy must avoid ignoring rare populations simply because the current model is confidently wrong about them.

Human Annotation Instructions Matter

Two annotators can label the same text differently if the task definition is vague. Annotation guidelines should include definitions, edge cases, examples and escalation rules.

Disagreement can reveal ambiguity in the task itself. Forcing consensus can hide useful uncertainty.

Annotator Population Matters

Cultural, linguistic and professional judgments vary. A dataset for educational feedback should include expertise appropriate to the learners and curriculum being evaluated.

Using one narrow annotator population can encode its preferences as if they were universal.

Inter-Annotator Agreement

Agreement metrics help estimate how consistently a label can be applied. Low agreement may signal unclear guidelines or genuinely subjective cases.

A model cannot be expected to achieve a perfectly objective answer when humans do not share a stable target.

Data Weighting After Annotation

Not every labelled example deserves equal weight. High-confidence expert labels may be weighted differently from uncertain crowd labels, depending on the learning setup.

Weighting choices should be documented because they change the effective objective.

Synthetic Data Verification

Synthetic examples can be checked automatically where rules are exact. Generated arithmetic questions can be recomputed. Generated code can be run against tests. Structured records can be validated against schemas.

For subjective tasks, human review may be needed. The cheaper it becomes to generate examples, the more important it becomes to verify them.

Model Collapse and Recycled Synthetic Data

If future models train heavily on unverified outputs from earlier models, errors and stylistic narrowing can compound. Diverse real-world data and provenance help prevent synthetic material from silently becoming the whole ecosystem.

Synthetic data is valuable when targeted and controlled, not when its origin disappears.

Data Mixtures Are a Resource Allocation Decision

Every training token consumes compute. Allocating 10% of the token budget to code means that budget cannot be used for another source. Mixture design is therefore an optimisation problem over desired capabilities.

Teams should measure marginal value: does another billion tokens from this source improve the target evaluations enough to justify the cost?

Domain Over-Sampling Can Create Trade-offs

Increasing medical text exposure may improve medical terminology but shift behaviour elsewhere if model capacity and total compute are fixed.

A mixture change should be tested across broad regression suites so domain improvements do not hide unexpected losses.

Freshness Sampling

Recent sources can be weighted more heavily when current terminology matters. Historical sources remain useful for history, literature and long-term patterns.

The training objective should not erase time. Metadata or retrieval may still be needed to distinguish historical statements from current ones.

Data Governance Roles

A mature programme assigns owners: source acquisition, legal review, privacy, quality, filtering, versioning, evaluation protection and incident response.

Without ownership, a training corpus becomes an opaque accumulation nobody can confidently update or audit.

Incident Response for Bad Training Data

Suppose a source is later found to contain leaked private records. The team needs to identify affected dataset versions and model runs, stop future use and assess mitigation for already-trained models.

Lineage turns this from a forensic guess into a traceable process.

Data Removal Requests

A removal process should identify the source, processed copies and future dataset versions. The treatment of already-trained models can be technically and legally complex.

This complexity is another reason prevention and provenance matter early in the pipeline.

A Data Quality Scorecard

For each source family, record coverage, duplication rate, extraction error rate, language distribution, rights status, personal-information risk, contamination risk and evaluation value.

Do not collapse everything into one opaque “quality score” unless the dimensions remain inspectable. Different weaknesses need different repairs.

Worked Audit: A Singapore Education Corpus

Goal: improve understanding of Singapore school terminology. Source inventory includes MOE pages, SEAB syllabuses, school websites, tuition explanations and student questions.

Primary-source policy terms should come from official MOE and SEAB documents. Tuition and forum material can provide varied explanatory language but should not override official policy.

The dataset should tag dates because curriculum terminology changes. Old “Express/Normal” references can remain historically useful but should not be presented as current Full SBB terminology.

This example shows why provenance, source role and time matter even when all text is fluent English.

Worked Audit: Programming Corpus

Goal: improve Python debugging. Remove generated build artefacts and vendored dependencies, preserve licences, deduplicate forks and keep tests alongside code when useful.

Evaluate on private repositories or newly written problems to reduce contamination. Include buggy code and error logs, not only perfect finished programs, because debugging requires exposure to failure patterns.

Independent Exercise 6: Quality Filter

A filter deletes all text containing spelling mistakes. What capability might suffer?

Answer

Robustness to real user messages, learner writing, informal text and OCR errors may decline. “Clean” is not always “useful”.

Independent Exercise 7: Group Leakage

Ten essays from one student are randomly split so eight are in training and two in test. What risk appears?

Answer

The model may exploit student-specific style, making the test easier than handling genuinely new students. Group-level splitting may be more appropriate.

Independent Exercise 8: Source Role

A forum post and an official policy disagree. Which belongs as authoritative training evidence for current policy?

Answer

The official policy should control factual current-state labels. The forum can remain useful for language variation or examples of misconceptions if clearly distinguished.

A Final Dataset Readiness Checklist

Before a dataset enters training, verify five categories. First, source legitimacy: where did the data come from, what rights or restrictions apply, and who owns the decision to use it? Second, technical integrity: was extraction correct, is encoding valid, and did tables, equations or code survive preprocessing?

Third, statistical composition: which languages, domains and time periods dominate, what is duplicated, and what is missing? Fourth, privacy and safety: does the data contain personal information, secrets, harmful content or material that requires special handling? Fifth, evaluation protection: are benchmark examples or near-duplicates present?

Then record a version identifier for the processed dataset and the sampling recipe used in the training run. Without those two pieces, “trained on the same data” may hide a completely different effective mixture.

Why Dataset Quality Must Be Measured After Training Too

A dataset audit before training can predict risks, but the model’s behaviour is the final evidence. One source family may be harmless in isolation yet become overrepresented after sampling. A filter may look balanced but reduce performance for a low-resource language.

Post-training evaluation should therefore trace failures back to data hypotheses. If a model confuses two local education terms, search the corpus and mixture before assuming the architecture is at fault. If a stereotype appears, inspect both source representation and post-training behaviour.

Training data is a causal input to the learning process, but model behaviour emerges from data, objective, architecture and optimisation together. The dataset should be documented as a major component, not treated as the sole explanation for every success or failure.

A Worked Data-Mixture Design

Suppose a new educational model has a fixed budget of one billion training tokens for a focused adaptation run. Candidate sources are 500 million tokens of general education prose, 200 million tokens of Singapore curriculum documents, 100 million tokens of worked mathematics, 50 million tokens of code, and 150 million tokens of authentic student questions.

Using the raw proportions would reproduce the source sizes, but the team may want stronger coverage of student language and local curriculum. It could oversample Singapore documents and student questions while down-sampling generic prose. The effective training mixture therefore reflects capability priorities rather than file-system size.

The mixture should then be evaluated by domain: local terminology, maths explanation, learner-error interpretation, general language and code. If local performance improves while general explanation collapses, the adaptation has created a trade-off that needs further tuning.

A Worked Data-Governance Incident

Imagine a source later turns out to contain private records that should never have entered training. A mature data system can identify the source ID, every processed derivative, dataset versions containing it and the model runs that consumed those versions.

The organisation can stop future use immediately, assess whether affected checkpoints require retraining or another mitigation, and document the incident. Without provenance and lineage, the same event becomes a search through anonymous shards with uncertain scope.

This is why dataset engineering is part of model safety. Good lineage does not prevent every mistake, but it makes mistakes bounded, discoverable and repairable.

Training Data Completion Test

A reader has understood training data when they can separate three questions: what material exists, what material actually enters the training mixture, and what behaviour the trained model exhibits. Raw corpus size answers only the first. Sampling and filtering determine the second. Evaluation determines the third.

Keeping those levels separate prevents simplistic claims such as “the model saw this website, therefore it knows every fact on it” or “the dataset contains 100 languages, therefore the model serves all 100 equally well”.

Dataset Repair Pathways

When a model fails because coverage is weak, the repair may be targeted collection. When it fails because one source is duplicated excessively, the repair may be deduplication and resampling. When labels are inconsistent, the repair may be guideline revision and relabelling. When a benchmark is contaminated, the repair belongs in evaluation protection.

These are different operations. “Get better data” is too broad to guide an engineering team.

Dataset Stabilisation Pathway

After obvious defects are repaired, stabilise the corpus with versioning, automated quality tests, representative manual audits and monitoring of language and domain proportions. New data should not silently change the mixture without review.

A stable dataset pipeline makes model comparisons interpretable because a performance change is less likely to come from an unnoticed corpus shift.

Dataset Extension Pathway

Once the baseline is stable, extend deliberately: add a new language, domain, modality or time period and evaluate both the new capability and regressions elsewhere. Treat expansion as an experiment with measurable goals rather than an uncontrolled accumulation of files.

This mirrors the Clementi teaching principle: repair the unstable foundation, stabilise performance, then extend difficulty.

Worked Mixture Check

Suppose two training runs use the same five source families, but Run A samples each source in proportion to raw size while Run B deliberately oversamples local curriculum and authentic learner questions. These are not the same dataset in practice even if the file inventory is identical. The sampling recipe changes how many gradient updates each source generates.

To compare the runs fairly, report both corpus inventory and effective token mixture. Then evaluate the capabilities the mixture was intended to change. If local terminology improves but general writing degrades, the result is a measurable trade-off rather than a vague impression that one dataset was “better”.

Training Data Reality Check

A dataset is not strong because it is large, expensive or diverse on paper. It is strong when its documented composition supports the model’s intended capability and when evaluation shows that the resulting model generalises to representative unseen work.

The final check is therefore behavioural. Trace important model failures back into the data pipeline, test the hypothesis with targeted additions or removals, and measure whether the repair changes the intended outcome without creating unacceptable regressions elsewhere.

That closes the loop from source governance to learning evidence: collect deliberately, transform visibly, train reproducibly and evaluate independently.

Frequently Asked Questions About Training Data

Does an LLM remember every training document?

No. Training updates distributed parameters rather than storing a conventional document database. Some material can be memorised, especially if repeated or distinctive, but the model does not expose a complete indexed copy of its corpus.

Is more training data always better?

No. Quality, diversity, rights, duplication, relevance and sampling matter. More low-value repetition can add cost without useful capability.

What is data contamination?

It is overlap between training data and evaluation material that can make tests easier because the model may have seen the examples before.

What is data provenance?

Information about where data came from, when and how it was collected, what transformations occurred and what rights or restrictions apply.

Why document datasets?

Documentation supports accountability, reproducibility, audits, limitation analysis and appropriate downstream use.

Can synthetic data replace real data?

It can supplement or target gaps, but synthetic data inherits generator limitations and needs verification. It is not automatically equivalent to real-world diversity.

Why does language balance matter?

Underrepresented languages receive less effective training signal and may have weaker tokenisation or evaluation coverage.

Can data filtering remove all bias?

No. Bias can come from representation imbalance, historical patterns and task design. Filtering is one tool among data curation, evaluation and post-training.

Training Data Defines the Learning Environment

Training data is not merely fuel poured into a fixed machine. It defines what examples the model encounters, how often it encounters them and which parts of the world become easy or difficult to represent.

Strong data practice means intentional collection, documented provenance, balanced sampling, careful cleaning, deduplication, privacy protection, rights management and independent evaluation.

Continue through the How Super Intelligence Works hub. Previous: 021 — Pretraining. Next: 023 — Self-Supervised Learning.


How Super Intelligence Works Series Navigation

Previous: 021 — Pretraining · Series Hub · Next: 023 — Self-Supervised Learning