Dataset cartography is a method for learning what a training dataset is doing to a machine-learning model, instead of judging examples only by how they look to a person. During training, it records how much probability the model assigns to each example’s annotated label at successive epochs. An example that remains easy, one on which the model repeatedly changes its mind, and one that stays difficult leave different traces. Mapping those traces can help researchers investigate data quality, choose more informative training examples and test whether the chosen subset improves results beyond the original data distribution.
The method was introduced by Swabha Swayamdipta and colleagues in their EMNLP 2020 paper, “Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics.” Their central insight was that a single training run contains more information than the final score alone. The model’s time series for each training example reveals stability, difficulty and uncertainty-like behaviour that a dataset summary can conceal. The original experiments examined four natural-language-processing datasets and reported distinctive patterns among easy-to-learn, ambiguous and hard-to-learn regions.
This is not geographical mapping, not an automatic label-correction algorithm and not proof that a model knows which examples are objectively true. It is a model-dependent diagnostic built from training behaviour. To use it responsibly, one must define the label probability precisely, keep the epoch and example denominators consistent, separate ambiguous examples from mislabeled ones, test selections on held-out data and preserve the exact training configuration that generated the map.
On this page: The training loop that creates the evidence · Confidence: the time-average of true-label probability · Variability: how much that probability changes · Reading the ambiguous region · A five-epoch worked calculation · An example-selection experiment with honest denominators · A practical workflow for a small team · Changed-condition transfer: six epochs and an unseen domain · Frequently asked questions · Original evidence and the final decision rule
Why a dataset needs a map rather than only a count
Imagine two text-classification datasets with 100,000 labelled sentences each. Both have the same class proportions and similar average sentence length. In the first, most labels are clear but the data contains many near-duplicates. In the second, a substantial minority of examples involve negation, ambiguous references and difficult linguistic distinctions. The equal counts do not imply equal learning value.
A conventional dataset report might include numbers of examples, labels, languages, domains and missing fields. That information is essential but does not reveal how a particular learning system encounters the data. A model can reach high training accuracy by repeatedly seeing simple patterns while failing on important shifts in wording. Another model can have a lower training accuracy because annotations contain genuine disagreements or systematic errors. A global accuracy number cannot distinguish these causes.
Dataset cartography adds a different viewpoint: for every training example, ask how the model behaved across the learning process. Did it assign high probability to the annotated class from early on? Did it alternate between confidence and doubt? Did it persistently assign low probability to the labelled class? The answers form a map of training dynamics, not a replacement for metadata. A complete diagnosis still requires annotation guidelines, manual review, external validation and knowledge of the collection process.
The training loop that creates the evidence
Start with a supervised training set of N examples. Each example i has an input x_i and annotated target y_i. At each selected epoch e, evaluate the current model on the same input and record p_e(y_i | x_i), the probability it assigns to the annotated target. Retain a stable example identifier, the epoch number, the true-label probability and, where needed, the predicted label.
An epoch typically means a pass through the training data, but evaluation schedules differ. If confidence is logged after every parameter update for some examples and once per epoch for others, the traces will not be comparable. The original procedure builds a view from model predictions over epochs. A reproducible implementation should document when probabilities were measured, whether the model was in evaluation mode and whether stochastic augmentation was applied at measurement time.
These traces should be recorded for training examples, not computed by training on an evaluation set whose labels are supposed to remain hidden. Mapping the training set is legitimate diagnostic work; tuning selection thresholds repeatedly against the final test set leaks evaluation information. The most reliable workflow reserves development and test data before making cartographic decisions, then measures how different subset policies perform on genuinely held-out examples.
One training run can produce one useful data map, but the map is not a universal property of each sentence. Change the architecture, random seed, curriculum, class weights or learning rate, and the trace may change. The phrase “training dynamics” highlights this dependency: the observed behaviour belongs to an example and a particular learning process together.
Confidence: the time-average of true-label probability
The first axis is often called confidence, but it has a specific meaning here. For example i across E recorded epochs, average the probability that the model assigns to that example’s annotated class. In words, mean confidence equals the sum of its true-label probabilities across recorded epochs divided by E. It is not the maximum softmax probability of whatever class happens to win, and it is not a human annotation-confidence score.
Suppose a binary classifier assigns one labelled-positive example probabilities 0.90, 0.92, 0.94, 0.95 and 0.96 across five epochs. Their sum is 4.67. Divide by five and the mean confidence is 0.934, or 93.4%. High values indicate that, under this training configuration, the model repeatedly supports the annotated class. They do not establish that the annotation is infallible or that the model is well calibrated on unseen inputs.
Contrast an example with probabilities 0.08, 0.12, 0.10, 0.09 and 0.11. The sum is 0.50 and the average is 0.10, or 10%. The model repeatedly gives little support to the assigned label. That may signal a wrong label, an unusual item, an underrepresented subpopulation, an insufficient representation or simply a training configuration that cannot learn the pattern. Calling it “bad data” without investigation would be unjustified.
When calculating averages across epochs, confirm every example has the same number of valid observations or specify a weighting rule. A missing early epoch cannot be silently counted as zero, and an example observed after a curriculum delay should not be compared blindly with one available throughout training.
Variability: how much that probability changes
A second axis records how widely the true-label probabilities move around their average. A common measure is their standard deviation across epochs: subtract the example’s mean confidence from each observed probability, square the differences, average those squared deviations over the E recorded observations, then take the square root. Consistency of the formula matters more than whether a particular implementation uses E or E minus one for an estimator; report the convention.
Consider probabilities 0.35, 0.75, 0.40, 0.80 and 0.45. They sum to 2.75, so their mean is 0.55. Deviations from the mean are minus 0.20, plus 0.20, minus 0.15, plus 0.25 and minus 0.10. The squared deviations sum to 0.175. Divide by five to obtain 0.035 and take the square root: variability is about 0.1871. The model repeatedly shifts its support toward and away from the annotated class.
By contrast, the high-confidence trace of 0.90, 0.92, 0.94, 0.95 and 0.96 has mean 0.934 and standard deviation about 0.0215 using the same population formula. The low-confidence trace of 0.08, 0.12, 0.10, 0.09 and 0.11 has mean 0.10 and standard deviation about 0.0141. Thus confidence and variability describe different dimensions: an example can be consistently well learned, consistently poorly learned or unstable while training proceeds.
Variability is not a calibrated probability that a label is ambiguous. It depends on the model, the epoch interval and the training path. A model with an unstable learning rate might show high variability for many items for reasons unrelated to inherent linguistic ambiguity.
Correctness and forgetting add useful context
The original method also considers correctness. In a classification task, one can record at each epoch whether the model’s predicted class equals the annotated class, then average those indicators over epochs. With five observations, four correct predictions produce a correctness fraction of 4 ÷ 5 = 0.80. This is not the same number as true-label confidence: a prediction can be correct with a modest softmax probability, especially in a multiclass task.
For the three illustrative binary traces already introduced, the easy example is predicted correctly in all five epochs if the class threshold is 0.5. The unstable example has true-label probabilities exceeding 0.5 in two of five epochs, so its binary correctness fraction is 2 ÷ 5 = 0.40. The consistently hard example never crosses 0.5, so its correctness fraction is zero. In a multiclass model, correctness cannot be inferred from the true-label probability alone; probabilities for competing labels must also be inspected.
Forgetting events offer another perspective. If a model predicts an example correctly at one epoch and incorrectly at the next, that correct-to-incorrect transition counts as a forgetting event under a common definition. A never-learned example is different: it may have zero such transitions because it was never correct in the first place. That is why forgetting counts must not be interpreted without correctness histories.
Logging multiple measures can prevent misclassification. A high-variability item may switch between correct and incorrect, whereas another can vary in probability without ever changing its winning class. Both are dynamic, but they pose different questions for review and sampling.
Reading the easy-to-learn region
On a typical data map, easy-to-learn examples show relatively high mean true-label confidence and low variability. A classifier has repeatedly found features that agree with the annotated class. Such examples help establish the basic training signal and can make optimisation less erratic. Their being easy for the current model does not mean they are redundant for every model or that they should all be removed.
A simple sentiment dataset illustrates the point. “This film was excellent” may be labelled positive and learned quickly when the dataset contains many similar expressions. The same collection may also contain negated or domain-specific uses of familiar adjectives. A model that sees only difficult exceptions might fail to learn stable ordinary patterns, while one trained almost entirely on easy patterns may rely on shallow cues and struggle when conditions change.
The original study reported that easy-to-learn regions were populous and important for optimisation. That is not a universal prescription to include every easy example. It is a warning that selecting only the most ambiguous subset is not automatically efficient or stable. Researchers should compare selection mixes and use held-out learning curves rather than treating a map quadrant as an absolute instruction.
One useful audit asks how many easy items are duplicates or near-duplicates, whether they cover all classes and whether removing them damages calibration or convergence. Repetition may be valuable early in training yet contribute little additional information later. These are empirical questions whose answers depend on the task, architecture and compute budget.
Reading the ambiguous region
Ambiguous examples occupy a region characterised by substantial variability in the model’s support for the annotated class. Across epochs, the model may move back and forth as different features and decision boundaries become salient. This is distinct from simply ranking examples by low final confidence. A difficult example might stay consistently low; an ambiguous one repeatedly changes.
In a natural-language-inference task, a sentence pair containing a pronoun with multiple plausible antecedents may be hard to classify under a single label. Another pair may require world knowledge absent from a shortcut-driven training distribution. A changing true-label probability suggests that the model’s treatment of the example is sensitive to its current parameters. It does not tell us which linguistic ambiguity is present without inspecting the text and annotations.
Swayamdipta and colleagues reported that ambiguous regions were especially useful for out-of-distribution generalisation in the tasks and selection experiments they studied. This is an important observed result, but it has conditions. Some high-variability items may contain noise or inconsistent annotation; some clean examples can vary for training-specific reasons. One must not rewrite the paper’s finding as “always train on the hardest uncertain items.”
For a data steward, the ambiguous region is a place to examine informative boundary cases. It can suggest a richer sample for robustness training or manual analysis, provided the evaluation separates genuinely informative distinctions from accidental instability.
Reading the hard-to-learn region
Hard-to-learn examples have low mean confidence in the annotated class, often with little movement over epochs. A model repeatedly resists the provided label. Such items can include annotation errors, corrupted input, examples beyond the model’s representational reach and valid minority-pattern cases. The original dataset-cartography work found that hard-to-learn regions often exposed label problems, making the region valuable for targeted quality checks.
But “hard” does not mean “wrong.” Suppose a sentiment model trained mostly on contemporary English encounters a grammatical construction from another dialect. It may assign persistently low probability to the correct annotation because its training experience is narrow. Discarding the example could make the model less fair and less robust. A well-designed review distinguishes erroneous labelling from coverage gaps.
An annotation audit should show the item to more than one qualified reviewer when possible, use a documented decision rule and record disagreements. If the gold label changes, maintain version history and understand the consequences for any prior benchmark scores. If the label remains valid, ask whether the model, feature representation or collection strategy is missing an important pattern.
The strongest use of the map is therefore triage. It tells the human investigator where surprising behaviour is concentrated. It does not replace source review, class definitions or domain expertise. Treating it as a truth oracle would reproduce the very data-quality problems it is meant to expose.
Plotting a map without inventing thresholds
A basic visual places mean true-label confidence on one axis and variability on the other. Each point represents one training example, and the density of points conveys the shape of the dataset. Colour can represent correctness or a related independent measure, provided the legend explains the variable. An attractive scatterplot without axis units, definitions or sample counts can be less informative than a modest well-labelled table.
There are no universally correct fixed thresholds such as “0.70 always means easy” or “0.15 variability always means ambiguous.” Different models, epoch counts and training tasks produce different distributions. Researchers often select regional subsets by rankings or quantiles, then inspect how results change with the selection. If thresholds are selected after looking at the test set, the apparent improvement can be contaminated by tuning.
Imagine a dataset of 1,000 items divided, for illustration only, into 600 easy, 250 ambiguous and 150 hard items. The shares are 60%, 25% and 15%, respectively. If a random sample of 100 items reflects the overall proportions, it would contain about 25 ambiguous items in expectation, not necessarily exactly 25 in one draw. A targeted sample of 100 ambiguous items would contain 100 by construction, four times the expected random count in this hypothetical collection. That ratio measures sampling composition, not fourfold generalisation improvement.
Before declaring one selection policy better, report the denominator of the original dataset, the number of selected examples per region, per-class coverage, total training updates and the independent metrics on held-out evaluation data.
A five-epoch worked calculation
The following three traces are invented solely to demonstrate the arithmetic. Example A has annotated-label probabilities 0.90, 0.92, 0.94, 0.95 and 0.96. Its mean is 0.934 and its variability is approximately 0.0215. Example B has values 0.35, 0.75, 0.40, 0.80 and 0.45. Its mean is 0.55 and its variability is approximately 0.1871. Example C has values 0.08, 0.12, 0.10, 0.09 and 0.11. Its mean is 0.10 and its variability is approximately 0.0141.
If these are binary classification probabilities for the annotated class, and the winning label is determined by the 0.5 threshold, example A is correct on five out of five observations, B on two out of five, and C on zero out of five. A is illustrative of the easy region, B of a dynamic ambiguous region and C of a consistently hard region. The categories depend on comparison with the whole dataset; three standalone traces do not establish the proper thresholds for a real map.
Check the sums directly. A totals 4.67, B totals 2.75 and C totals 0.50. Dividing each sum by five yields the reported means. For B, squared deviations from 0.55 total 0.175. The mean squared deviation is 0.035, and its square root is about 0.1871. These calculations illustrate why a high average probability and a high variation are not interchangeable.
A frequent spreadsheet error is to compute variability from the maximum predicted-class probability rather than the annotated-class probability. If the predicted class changes, that substitution can conceal the very instability the map is intended to reveal. Keep the probability column tied to the original annotation throughout the run.
An example-selection experiment with honest denominators
Suppose a fictional text classifier is trained twice under matched conditions. Run A uses 100 examples selected uniformly from a fixed candidate set. Run B uses 100 examples selected for high training-dynamics variability after a separate pilot run. Both runs use the same architecture, update budget and held-out evaluation data. Training set size is 100 examples in both cases; the distinguishing factor is the selection rule.
On a constructed out-of-distribution test set of 200 independent labelled examples, imagine Run A correctly classifies 142, an accuracy of 142 ÷ 200 = 71%. Run B correctly classifies 150, an accuracy of 150 ÷ 200 = 75%. The difference is eight additional correct answers out of 200, or four percentage points. That is a concrete result for this hypothetical sample, not evidence that selecting ambiguous examples always improves performance by four points.
Now inspect an in-distribution test set of 400 items. Suppose Run A gets 360 correct, or 90%, and B gets 352 correct, or 88%. The selection policy that improved the distribution-shift result came with a two-point loss in ordinary evaluation. Whether that trade-off is acceptable depends on the application and the cost of different errors. It cannot be decided by celebrating only the favourable test.
For a real study, repeat the process with multiple seeds and selection samples, report uncertainty intervals and avoid using the final test set to pick a favourable run. The illustrative numbers above are hand-checked examples for reasoning about denominators; they are not observations from the original paper or a claim about current models.
Avoid selection leakage and circular claims
One methodological trap occurs when researchers train a model on the full dataset, use the same model to score all examples, choose a subset and then treat a performance gain as independent evidence that the scoring model objectively discovered good data. The scores are conditional on the model and data used to produce them. That does not invalidate the method, but it requires careful experimental interpretation.
A selection study should specify the pilot model used to calculate training dynamics and the target model used to evaluate the chosen subset. If the target differs, any transfer claim needs its own evidence. If the same model family is used, apparent advantages might depend on shared inductive biases. When measuring an out-of-distribution benefit, the out-of-distribution dataset must not be used for repeated threshold tuning or cherry-picking favourable map regions.
Record random seeds, checkpoint schedules, optimisers, early-stopping criteria, data ordering and duplicate handling. If selection policies use different amounts of computation, state the budget and decide whether the comparison is sample-efficient, compute-efficient or merely accuracy-oriented. A method can require costly pilot training before it allows cheaper subsequent runs.
This discipline prevents a misleading narrative. “We selected difficult examples and obtained a higher score” is weaker than “With a prespecified region rule, fixed budget, separate evaluation domains and repeated seeds, the selected data improved the target metric.” Dataset cartography is a measurement tool; trustworthy conclusions still depend on experimental control.
Why final-epoch loss is not the same diagnostic
A conventional hardness ranking uses the loss at the final checkpoint. For a correctly annotated class, cross-entropy can be written as the negative logarithm of the probability assigned to that class. Low annotated-label probability therefore corresponds to high loss at that checkpoint. But a single checkpoint cannot distinguish consistent difficulty from a model that alternated between confidence and uncertainty earlier in training.
Example B in the worked traces ends at 0.45. A final-checkpoint loss ranking might treat it as difficult. Yet its preceding trace reached 0.75 and 0.80 and fell twice. Example C also ends with low confidence, but it remained near 0.10 throughout. Their training histories are substantially different even though both finish below the binary decision threshold.
Using a training-dynamics map adds information about the path, not merely the destination. This may reveal examples that participate in contested decision boundaries and identify those on which the model never aligned with the provided target. A robust experiment should nonetheless compare the map against simpler baselines such as loss ranking, random selection, confidence-based sampling and class-balanced selection. If a complex procedure provides no benefit over a cheaper baseline under matched conditions, its operational case is weaker.
The decision to use cartography should be tied to a specific need: analysing annotation quality, understanding training instability, or choosing a subset for a measured downstream objective. A colourful plot without an actionable decision does not automatically improve a dataset.
How annotation errors appear in traces
Consider a sentence labelled “positive” that is plainly negative under the published annotation guidelines. A sufficiently capable model trained on many correctly labelled neighbouring examples may persistently predict the negative class, giving low probability to the supplied positive label. On the map, the example may appear hard to learn. Reviewing the raw sentence, label and guideline could reveal an error.
However, model disagreement with the label is not enough to justify changing it. The model may have learned a spurious correlation, may mishandle negation or may be insensitive to a domain-specific meaning. Annotation adjudication should use independent evidence, ideally including more than one reviewer and a recorded rationale. Auto-correcting every low-confidence label risks making the dataset conform to the model’s existing biases.
A useful quality-review queue separates suspected technical corruption from semantic disagreement. Corruption includes truncated fields, reversed labels during import and examples joined to the wrong identifiers. Semantic disagreement includes whether a statement is neutral or negative under a particular policy. The first often has a deterministic fix; the second may require annotation-policy revision or multiple acceptable labels.
After corrections, regenerate relevant diagnostics and preserve a version identifier. A map from the old labels should not be shown as if it describes the corrected dataset. Changes in data definition alter the meaning of every calculated confidence value, so provenance is part of the analysis.
Model dependence and random-seed sensitivity
A data map is not a property engraved into a training example. It is a response of a chosen learning system to a chosen dataset under a chosen training procedure. A larger model may master items that a smaller one finds hard. Different tokenisation can change which words are readily represented. Training with stronger regularisation, augmentation or class weighting can move points even when the text and label are unchanged.
Random seeds matter because parameter initialisation, example order and dropout can change the path. If a decision is costly, run several maps and assess whether a region assignment is stable. One simple diagnostic is to report how frequently each example enters a selected quantile across seeds. That proportion should not be confused with a calibrated probability that the label is wrong. It describes selection stability under the chosen experimental variations.
For transferred applications, a map built by a language model should not automatically govern image or tabular data. The original method’s logic can be adapted, but label probabilities, epochs, model uncertainty and data-specific augmentations behave differently. Even within NLP, an example’s difficulty on a natural-language-inference task may not predict its usefulness for a retrieval or generation task.
The disciplined statement is therefore conditional: “For this model, seed range, dataset version and measurement schedule, these examples showed these dynamics.” Conditional statements are less catchy than “these are the difficult data,” but they support reproducible science.
Calibration and probability interpretation
Softmax probabilities are convenient model outputs, not automatic guarantees of calibrated predictive belief. A classifier may be systematically overconfident on unfamiliar data or underconfident after label smoothing. Mean annotated-label probability across training epochs must therefore be interpreted as a model-internal training measure. It is not itself a calibrated risk estimate for a real-world decision.
The distinction matters especially if the data map is used to flag “uncertain” or “unsafe” examples. High variability can reflect changes in optimisation even when every prediction remains correct. Low variability can reflect a model confidently adhering to a wrong annotation. A calibration analysis on independent evaluation data asks a different question: among cases assigned a certain confidence, how frequently are predictions correct? That analysis requires appropriate binning or scoring rules and enough examples per group.
For instance, if 100 held-out predictions have reported confidence near 0.80 and 60 are correct, the empirical accuracy is 60%, not 80%. The 20-percentage-point gap is a simple illustration of miscalibration. It cannot be inferred from the training-dynamics map without independent outcomes. A method that aims to improve robustness may also need calibration checks because the shift changes which errors are operationally important.
Use cartography and calibration as complementary diagnostics. One tells you how examples behaved while the model learned; the other examines whether confidence supports reliable prediction claims on independent data.
Fairness, minority cases and deletion risk
Low-confidence regions may contain exactly the examples a responsible model should not forget. A speech system might struggle with a minority accent; a language classifier might struggle with code-switching or a regional phrase. An automated rule that discards all persistent low-confidence data could optimise average training performance while making the system worse for underrepresented users.
Before filtering, stratify region membership by relevant, lawfully measured subgroups and data sources where possible. Examine whether minority cases are overrepresented in the discarded region, whether their labels are reliable and whether the target evaluation includes them. This is not an instruction to infer sensitive attributes from data without authority; it is a reminder that quality interventions can change who is represented in the model.
There is a crucial difference between label noise and a difficult valid case. An example may look unusual because a collection process is narrow, not because the example is erroneous. Review processes should give specialists a way to mark “valid but underrepresented” and should preserve that designation across retraining cycles.
A cartographic selection policy can be audited with a table of retained and removed examples by class, source and relevant domain slice. Counts, percentages and performance changes should use explicit denominators. An apparently impressive global improvement is incomplete if the change increases error rates on a mission-critical minority use case.
A practical workflow for a small team
Begin by naming one diagnostic question. Is the aim to find possible mislabels, reduce training expense, understand poor robustness or choose examples for an annotation audit? Choose a dataset version, fix the target labels and reserve held-out evaluation data. Train a baseline model while logging the annotated-label probability for each stable example ID at comparable checkpoints. Validate that the logging itself does not inadvertently apply training-time augmentation to only part of the data.
Next calculate mean confidence, variability and optional correctness for each example. Plot the resulting map and inspect points sampled from all regions. Do not limit human review to the lowest-confidence items; examine the ambiguous and easy regions too so the labels have comparative meaning. Document any proposed region boundaries and check the stability of assignments across seeds where feasible.
Run a bounded selection experiment. Compare a baseline random subset, a class-balanced subset and the cartographic subset at equal example counts and stated compute budgets. Measure in-distribution and out-of-distribution performance, calibration when relevant, subgroup behaviour and variance across runs. Write down costs as well as gains: pilot training time, storage for traces, annotation review and repeated evaluation.
Finally state what decision the evidence supports. It may justify cleaning a set of suspected labels, mixing more boundary cases into training or leaving the original distribution unchanged. A map is successful when it improves a specific decision with measured evidence, not when it makes every plot look neatly divided into three regions.
What to store for reproducibility
A minimal data record contains an immutable example ID, dataset version, target label, epoch or checkpoint identifier, probability assigned to the target and optionally the predicted label. Store the tokenizer version, model checkpoint, training code version, random seed, data ordering and hyperparameters alongside the results. An item that changes ID between epochs can produce a perfectly plotted but meaningless time series.
Data stewardship also matters. Avoid leaking private training text in visual dashboards. A map can use identifiers and aggregate summaries until an authorised reviewer needs to inspect a particular sample. If labelled text comes from users or restricted agreements, logging and review must follow the applicable access policy. The visualisation should not create a new route to disclose the very inputs it seeks to diagnose.
The dataset version should be frozen when comparing selections. Adding or removing examples between epochs changes the universe used for the calculations. If curriculum learning deliberately introduces examples later, either account explicitly for their shorter traces or analyse them separately. The simple mean-across-E formula presumes that E actually describes a meaningful, comparable set of observations.
When publishing research or an internal report, include the uncertainty and the limitations of the mapping model. A saved plotting notebook without the configuration, annotations and splits does not permit independent reproduction of the claimed data-quality conclusions.
Changed-condition transfer: six epochs and an unseen domain
Revisit the worked example with five epochs and high variability. What happens if a sixth checkpoint assigns the annotated class probability 0.95? The new sum is 2.75 + 0.95 = 3.70, and the six-epoch mean is 3.70 ÷ 6, approximately 0.6167. The earlier mean was 0.55. The new observation changes the summary, and a recomputed standard deviation will differ too. Therefore a threshold that identified the example as ambiguous at five epochs might classify it differently after a longer training schedule.
Now suppose the target model is no longer a binary sentiment classifier but a three-class topic classifier. The rule that true-label probability above 0.5 identifies the correct prediction no longer covers every case: an annotated class with probability 0.42 could still be the predicted winner if the two competitors receive 0.35 and 0.23. Correctness requires the full class comparison. The definition of mean annotated-label confidence remains valid, but the interpretation of thresholds changes.
Finally imagine the selected training data improves a new topic-domain test while reducing performance in the original domain. The correct conclusion is a conditional trade-off, not a universal quality improvement. A robust report should name both evaluation populations and each denominator, then describe the application goal. This transfer problem tests the actual principle: preserve definitions while recognising which conditions have changed.
What dataset cartography cannot promise
The map cannot prove that a particular data point is wrong, that the selected subset is optimal, that generalisation will improve for all models, or that high model confidence means verified truth. It does not automatically resolve missing subgroups, hidden annotation-policy disagreements, domain-specific label errors or the ethics of data collection. A noisy training run can create misleading regions, and a stable model can be consistently wrong.
Its original research context matters. The 2020 experiments were performed on natural-language-processing datasets and investigated model-dependent training dynamics. The paper’s findings on ambiguous examples and out-of-distribution performance are valuable empirical results, not invariants of all machine learning. Novel domains require their own tests, especially where distribution shifts and class imbalance differ from those in the original study.
A fair comparison also asks whether simpler procedures can achieve the same improvement. Uniform subsampling, quality checks from annotators and single-checkpoint loss may be cheaper. The complexity of a multi-epoch map is justified when the path-dependent information changes the decision in a valuable way.
The safe governing rule is to treat the map as evidence for a hypothesis about data, then verify that hypothesis through review and independent evaluation. It is a diagnostic instrument, not a substitute for judgement or a licence to delete inconvenient examples.
Frequently asked questions
What is dataset cartography? It is a technique that examines each training example’s confidence trajectory over epochs and maps examples by summary measures such as mean true-label probability and its variability.
Does it work with one training run? The original method can construct a map using a single run. Extra seeds are useful to test whether region assignments and conclusions are robust.
What does “confidence” mean here? It is the probability assigned to the annotated label, averaged across recorded training epochs. It is not necessarily the maximum predicted-class probability or a calibrated probability of truth.
What does high variability mean? The model’s probability for the annotated label changed substantially during training. This can indicate a disputed boundary or unstable learning behaviour, but it does not prove inherent ambiguity.
Are low-confidence examples mislabeled? Some are, but others are valid and difficult for the model. A separate audit must establish the cause before changing or deleting labels.
Should all easy examples be removed? No. They can support optimisation and class coverage. Evaluate subset policies instead of imposing a universal removal rule.
Does selecting ambiguous examples always help out-of-distribution accuracy? No. That was an observed pattern in the original study’s conditions. A new task or model needs its own independent evidence.
What is the denominator in mean confidence? The count of comparable recorded observations for that example, usually epochs in the basic procedure. Missing observations require explicit handling.
Can the same map be used for a new model? Not without qualification. Training dynamics depend on the model and procedure, so cross-model transfer of selection policies must be tested.
What is a good first project? Train a small labelled text classifier, log true-label probabilities at a fixed epoch schedule, compute means and standard deviations per example, inspect a balanced selection from each region and test one prespecified subset rule on untouched evaluation data.
Original evidence and the final decision rule
The canonical source is Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith and Yejin Choi (2020), “Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics,” Proceedings of EMNLP 2020, pages 9275–9293, DOI 10.18653/v1/2020.emnlp-main.746. The ACL Anthology hosts the original paper and its bibliographic metadata. The authors also released a code repository demonstrating data-map computation, selection metrics and plotting.
The paper introduces mean model confidence in the annotated class and variability across epochs as axes of a model-based map. It reports easy-to-learn, ambiguous and hard-to-learn regions across four datasets, with ambiguous-region examples contributing substantially to out-of-distribution generalisation in the tested selection settings and hard-region examples often exposing label issues. Those findings are the empirical anchor for this tutorial. The numeric traces, selection counts and evaluation scores developed above are transparent invented examples, not reproductions of the researchers’ results.
If there is one decision rule to remember, it is this: examine how the model learned each example, decide what the diagnostic suggests, independently check the data and then test the consequence of acting on that suggestion. Do not confuse a point’s position on a map with its truth, value or guaranteed effect on a future model.
Original source links and connected tutorials
Original ACL Anthology paper · Official DOI · Original author-maintained code · How AI Works hub · Deep Sets — Permutation Invariance and Sum Pooling · Neural Additive Models — Interpretable Feature Functions.