A corpus sensitivity analysis is a structured check that repeats an analysis under one or more defensible alternative choices—such as tokenisation, text inclusion, duplicate handling or query representation—to see which conclusions depend on those choices. It is evidence about methodological robustness, not permission to select the most flattering result.
Did you know that a word-frequency report can change even when none of the original sentences change? One analyst counts the exact form run; another counts the verb lemma RUN, including runs and ran. Their totals can differ. Another removes repeated headers or copied passages and changes both numerator and denominator. The important lesson is to discover which claim survives reasonable rules, while recognising when a changed rule actually asks a different question.
This guide explains the difference between reproduction, replication, robustness and generalisation; shows how to change one rule at a time; and offers a full toy worksheet for exact counts, lemma searches, duplicate removal, heading exclusion and domain choice. It applies the approach to responsible vocabulary teaching and corpus evidence without presenting invented statistics as real results.
How the evidence is used: The general method is informed by the linked linguistic research. The classroom datasets, tables, numbers and pupil examples in this guide were constructed for explanation, not drawn from a real education system sample. The examples therefore do not establish actual frequency, performance or learning gains.
EDUKATE · VOCABULARY METHODS · READER ROUTES
Choose what you need first
- Learn the core idea · Chapter 1 onward
- Design a fair test · Chapter 7 onward
- Inspect numbers and examples · Chapter 13 onward
- Apply the method to new texts · Chapter 19 onward
All chapters · Worked table · Decision cases · Practice and solutions · Questions and glossary
Contents in complete reading order
- 1. Begin with a claim that could be wrong
- 2. Understand why changing one rule helps
- 3. Separate robustness from reproducibility
- 4. Fix the estimand before comparing rates
- 5. Make the corpus version part of the question
- 6. Choose alternative rules for defensible reasons
- 7. Inspect tokenisation changes
- 8. Test exact forms against inflectional grouping
- 9. Check case-sensitive and insensitive versions
- 10. Evaluate heading and template inclusion
- 11. Test exact duplicate removal
- 12. Inspect near-duplicates and meaningful revisions
- 13. Calculate the full one-factor worksheet
- 14. Interpret percentage changes cautiously
- 15. Check genre composition separately
- 16. Use stratified checks to expose hidden variation
- 17. Identify interactions among decisions
- 18. Separate exploratory checks from a declared analysis
- 19. Keep tests with constant data distinct from new-source replication
- 20. Teach sensitivity checks with a visible spreadsheet
- 21. Apply the method to word learning
- 22. Respect source rights and learner privacy
- 23. Summarise robustness without a misleading pass mark
- 24. Finish with an eight-question sensitivity audit
CHAPTER 1 OF 24 · LANGUAGE EVIDENCE
1. Begin with a claim that could be wrong
A sensitivity analysis needs a defined starting claim. A vague declaration such as ‘the word is very common’ is difficult to test; a statement naming the exact form, corpus, period and per-token rate can be recalculated under transparent alternatives.
See it in context
Suppose an analyst reports that the exact verb spelling run appears forty times among twenty thousand eligible tokens in a constructed collection. The raw rate is 2,000 per million. The student can now ask whether the conclusion depends on the counting rules.
The mistake to recognise
The mistake treats a precise-looking number as a permanent fact about English rather than a result from one source and method. It can be correct for the original workflow yet fragile when assumptions change.
A safer method
Write the base analysis in one line with source selection, target, numerator, denominator, time and exclusions. Keep the original result available for every later comparison.
Your check: What is a valid starting baseline for the worked exercise?
Explained answer: Forty exact matches among 20,000 eligible tokens, or 2,000 occurrences per million under the declared construction.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 2 OF 24 · LANGUAGE EVIDENCE
2. Understand why changing one rule helps
A controlled variation isolates the effect of a chosen methodological decision. If two or three choices change simultaneously, the resulting difference cannot easily be attributed to any one of them. This resembles a fair comparison in Science, though language data have additional complexities.
See it in context
To test whether duplicated notices inflate a rate, remove only the recognised duplicate content and recompute. Do not simultaneously change tokenisation and add new texts if the goal is to isolate the duplicate policy.
The mistake to recognise
The mistake changes the dataset, search target and frequency formula at once, then assigns the whole observed difference to one preferred explanation.
A safer method
Plan one-factor checks as an accessible starting point. When interactions matter, a fuller design may vary combinations of choices rather than claiming the first single-factor pass is exhaustive.
Your check: What should remain fixed in a duplicate-removal sensitivity check?
Explained answer: The language target and search rule, apart from the explicitly changed data inclusion policy.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 3 OF 24 · LANGUAGE EVIDENCE
3. Separate robustness from reproducibility
Reproducibility usually asks whether the same data and analysis produce the same result. Robustness asks whether a substantive conclusion holds across reasonable alternative analyses of the same data. Replication checks comparable claims on new data, while generalisation asks about broader settings; terminology varies across disciplines, so define it.
See it in context
A class can rerun the exact search and obtain forty matches again: that checks reproduction. It can then exclude repeated templates and obtain another rate: that checks sensitivity to one analysis choice. A new school archive tests a different question about replication or transfer.
The mistake to recognise
The mistaken report says the result is robust simply because a spreadsheet returns the same number twice under identical inputs. Repeatability of arithmetic is not robustness to defensible assumptions.
A safer method
Describe what was held fixed and what was changed. Use ordinary-language definitions of robustness and replication in the report so readers know which kind of confidence has been tested.
Your check: Does rerunning the same calculation twice test robustness to a different tokeniser?
Explained answer: No. It tests repeatability of the original procedure; a tokeniser comparison requires a changed analysis rule.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 4 OF 24 · LANGUAGE EVIDENCE
4. Fix the estimand before comparing rates
An estimand is the specific quantity the analysis intends to describe. An exact-form rate and a lemma rate are not interchangeable; a shift between them may demonstrate operational sensitivity but also a genuine change in the question.
See it in context
The target ‘exact written token run used as a verb’ is narrower than ‘all verb forms belonging to lemma RUN’. If the lemma query finds fifty-eight tokens while exact run finds forty, the two results answer distinct but related questions.
The mistake to recognise
The error calls the lemma result an improved estimate of exactly the same narrow form rate. It is not: more word forms qualified by definition.
A safer method
Label changed estimands explicitly. Use same-target comparisons to assess robustness of the original claim, and use target-expansion comparisons to clarify which vocabulary unit the question actually needs.
Your check: Can 2,900 lemma occurrences per million contradict 2,000 exact-form occurrences per million?
Explained answer: No. The numerators refer to different sets of qualifying tokens.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 5 OF 24 · LANGUAGE EVIDENCE
5. Make the corpus version part of the question
A corpus is not always static. Pages are added, files corrected and metadata revised. To interpret method sensitivity, keep a stable source snapshot unless the planned variation is deliberately about data collection.
See it in context
An archive of family notices may contain twenty files in October and twenty-five in November. Repeating a word search on the expanded archive tests a changed dataset, not merely a changed calculation.
The mistake to recognise
The mistake reports a difference as an effect of tokenisation while the corpus was also updated between runs. Unrecorded updates confound the comparison.
A safer method
Version source IDs, eligible text, token totals and timestamps. Recompute alternative analyses on the same snapshot before comparing with a later archive.
Your check: Why is the corpus version important for an otherwise identical query?
Explained answer: Because adding or removing source material changes the opportunities for observing a word.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 6 OF 24 · LANGUAGE EVIDENCE
6. Choose alternative rules for defensible reasons
A sensitivity check is useful when alternative choices are plausible for the actual task. Testing arbitrary changes merely to create different numbers is not informative. Reasonable alternatives can include case handling, lemma grouping, stopword filtering or duplicate policy.
See it in context
A study of editorial language might reasonably compare a corpus with and without repeated webpage navigation menus. A study of reader exposure might instead keep those menus because people do encounter them.
The mistake to recognise
The misleading practice tries many settings and presents only the variant that produces the most dramatic increase, without explaining why that setting was meaningful.
A safer method
List the alternatives before looking at results where possible, and document the rationale and expected effect for each. Keep exploratory checks separately labelled if added later.
Your check: Should an analysis include every imaginable rule variation?
Explained answer: No. Focus on plausible decisions that could materially affect the substantive conclusion and report the chosen scope.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 7 OF 24 · LANGUAGE EVIDENCE
7. Inspect tokenisation changes
Tokenisers divide text into analysis units using rules for contractions, hyphens, punctuation, numbers and other structures. A changed tokenisation can alter denominators, search results or both. Language variation can make these choices consequential.
See it in context
If one pipeline counts don’t as one word token and another splits it into multiple units, both the total word-token denominator and some pattern-search counts may change. A robust report states its operational unit.
The mistake to recognise
The shortcut divides a match count produced by one tokeniser by a total-token count produced by another. This creates a ratio that may not represent either processed corpus.
A safer method
Keep count and denominator under the same segmentation convention in each variant. Report the original and revised token totals instead of treating the difference as a language change.
Your check: Can a new tokeniser change a frequency rate without adding any sentences?
Explained answer: Yes. It can change the number of eligible units and which strings qualify under a query.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 8 OF 24 · LANGUAGE EVIDENCE
8. Test exact forms against inflectional grouping
For some vocabulary questions, the narrow exact spelling is appropriate. For others, inflected forms should be grouped under a lemma. Comparing both helps show the boundary between form recognition and lexical knowledge.
See it in context
A school writing analysis may want exact run in imperative instructions, whereas a narrative-vocabulary study may group ran, runs and running. The same archive can produce both counts if annotations are reliable.
The mistake to recognise
The error quietly merges past and present forms, then claims to have corrected a mistaken exact-form total. The original narrow result may have been perfectly valid for its target.
A safer method
Define the exact form and lemma+POS queries separately, validate ambiguous uses and avoid describing their difference as pure numerical error.
Your check: Which rate is automatically the correct one: exact run or lemma RUN?
Explained answer: Neither universally. The intended linguistic question determines the correct numerator.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 9 OF 24 · LANGUAGE EVIDENCE
9. Check case-sensitive and insensitive versions
Case handling can change matches. It may help recover sentence-initial forms but can also collapse distinct words, names or acronyms. A sensitivity check should inspect changed hits rather than merely prefer the larger count.
See it in context
The string May can refer to a month; may can be a modal. Case-insensitive counting might combine them in a basic string query, while a grammar-aware study would likely keep distinct lexical functions.
The mistake to recognise
The false inference says additional case-insensitive matches prove higher frequency of one semantic unit, even when some hits refer to a different meaning.
A safer method
Use a spot check of newly included examples. If the goal is the modal, part-of-speech filtering may be more important than raw case folding.
Your check: Does case-insensitive searching always make a rate better?
Explained answer: No. It can improve recall while introducing false matches for a narrow lexical target.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 10 OF 24 · LANGUAGE EVIDENCE
10. Evaluate heading and template inclusion
Many web pages repeat headings, tags, menus and disclaimers. Including them may be appropriate for exposure studies but can distort an attempt to describe authored running prose. The rule should depend on the intended communication unit.
See it in context
A constructed 20,000-token dataset contains repeated section titles carrying twelve of forty target matches. If those titles are excluded from the scope, the numerator and denominator must both be recomputed.
The mistake to recognise
The mistake removes heading hits but keeps heading tokens in the denominator, producing a rate that refers to no coherent analysed text population.
A safer method
Document the before/after source units. If headings contribute 1,000 tokens and twelve hits, the remaining corpus has 19,000 tokens and twenty-eight hits; calculate its rate under the revised prose-only question.
Your check: What is the prose-only rate for twenty-eight hits among 19,000 tokens?
Explained answer: 28/19,000×1,000,000≈1,473.68 per million. It answers a narrower text-scope question.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 11 OF 24 · LANGUAGE EVIDENCE
11. Test exact duplicate removal
Copied notices can make a word look frequent across many separate webpages even when they repeat one authored message. Removing copies according to a declared unique-text policy changes what counts as independent evidence.
See it in context
In the worked workbook, the 20,000-token baseline includes an exact duplicate passage contributing 2,000 tokens and eight hits. Removing it yields 18,000 tokens and thirty-two hits. The new rate is lower.
The mistake to recognise
The faulty shortcut compares raw hit totals but not adjusted token totals, or calls the change proof that real-world usage has fallen. A new processing rule is not a historical trend.
A safer method
Recompute both counts and denominators. Report what qualifies as an exact duplicate and whether the research question wants unique wording or repeated reader exposure.
Your check: How many hits remain after removing eight duplicated hits from forty?
Explained answer: Thirty-two, with 18,000 eligible tokens. The rate is approximately 1,777.78 per million.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 12 OF 24 · LANGUAGE EVIDENCE
12. Inspect near-duplicates and meaningful revisions
Text similarity tools can identify copied or near-copied material, but small wording changes can matter. A version changing ‘must’ to ‘may’ is not necessarily equivalent for language research even if almost every other word matches.
See it in context
A school instruction is revised to change a deadline or obligation. A simple similarity threshold could mark the versions duplicate, while a study of modal verbs needs the changed version retained.
The mistake to recognise
The mistake treats a high similarity percentage as evidence that every difference is unimportant, erasing exactly the linguistic feature under investigation.
A safer method
Flag candidate near-duplicates and review meaningful changes. Keep version relations and describe the policy separately from exact-copy removal.
Your check: Should a near-identical notice with changed negation always be removed?
Explained answer: No. It can contain a substantively different assertion or instruction.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 13 OF 24 · LANGUAGE EVIDENCE
13. Calculate the full one-factor worksheet
The constructed baseline is forty exact target hits in 20,000 tokens, or 2,000 per million. Four alternative analyses illustrate different changes: deduplication, heading exclusion, broader form grouping and a case-insensitive query. Each line should name what was changed and whether the target quantity remains comparable.
See it in context
Deduplication produces 32/18,000≈1,777.78 per million; heading exclusion 28/19,000≈1,473.68; lemma grouping 58/20,000=2,900; case-insensitive exact matching 46/20,000=2,300. The numbers are designed for arithmetic teaching, not collected from English texts.
The mistake to recognise
A misleading table places all four rates under the same heading ‘frequency of run’ without saying that lemma grouping expanded the set of forms or that heading exclusion changed the textual population.
A safer method
Show columns for rule changed, eligible tokens, qualifying hits, rate and interpretation. Compare like with like where appropriate and use the divergent values to discover which assumption drives a claimed conclusion.
Your check: Which variant has the lowest constructed rate?
Explained answer: The heading-exclusion variant, about 1,473.68 per million, within this illustrative table.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 14 OF 24 · LANGUAGE EVIDENCE
14. Interpret percentage changes cautiously
A change in rate can be calculated relative to baseline, but its cause depends on the variant. A relative change may reflect exclusion of repetitive material, a broader target or a source mixture change—not a genuine change in what writers wrote.
See it in context
The duplicate-removal rate is about 11.1 percent below baseline: (1,777.78−2,000)/2,000≈−0.1111. The lemma rate is 45 percent above baseline, but it includes additional inflected forms by definition.
The mistake to recognise
The shortcut treats every relative percentage as a causal effect or a measure of uncertainty. A processing difference is not a teaching intervention or real temporal change.
A safer method
Describe percentage change only alongside the exact rule, raw counts and denominator. Where the target shifts, label the comparison as a difference in operational definition rather than a direct robustness estimate.
Your check: Does a 45 percent larger lemma rate mean learners improved 45 percent?
Explained answer: No. It is a different counted target with more qualifying forms, not a measured learning outcome.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 15 OF 24 · LANGUAGE EVIDENCE
15. Check genre composition separately
A corpus may mix laboratory instructions, news, personal stories and notices. Removing or reweighting one genre can alter word rates because the genres serve different purposes. Such a change tests domain composition as much as an analysis parameter.
See it in context
Suppose run is common in sports reporting but uncommon in administrative letters. A sports-heavy corpus and a letter-heavy corpus will yield different rates even if each is analysed consistently.
The mistake to recognise
The mistake interprets a changed genre mixture as an improvement or decline in universal English vocabulary, rather than a different population of language events.
A safer method
Break down rates by genre and inspect their relevance to the target domain. If a mixture estimate is needed, state credible target shares; if unknown, avoid inventing them.
Your check: Can corpus-wide rates change while each genre’s own rate remains constant?
Explained answer: Yes. Changing the proportions of genres with different rates changes their pooled average.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 16 OF 24 · LANGUAGE EVIDENCE
16. Use stratified checks to expose hidden variation
A pooled result may appear stable even though particular subgroups behave differently, or unstable because one group is small and distinctive. Checking relevant strata can reveal what drives the overall pattern.
See it in context
A phrase appears in almost every scientific report but in few stories. If the sample adds more scientific reports, its overall frequency rises. Within-genre rates may tell a more useful educational story than the new pooled rank.
The mistake to recognise
The error treats one averaged statistic as the whole pattern. The aggregate can hide differences in source purpose and distribution across texts.
A safer method
Report per-stratum counts and source volumes. If one stratum dominates the result, say so and limit the conclusion accordingly.
Your check: Why is a per-genre table often more informative than one overall frequency?
Explained answer: It reveals where the word occurs and whether the pooled number is driven by sample composition.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 17 OF 24 · LANGUAGE EVIDENCE
17. Identify interactions among decisions
Changing one setting at a time is useful for diagnosis, but analytic choices may interact. The effect of lemmatisation could differ after duplicate removal because copied passages contain certain forms disproportionately.
See it in context
If all duplicated instances of run are past tense ran, a lemma query and duplicate policy interact. The difference produced by one variant depends on whether the other has already been applied.
The mistake to recognise
The shortcut concludes one-factor checks exhaust every possible result. They reveal selected sensitivities but can miss combined effects.
A safer method
When a conclusion is important, analyse a small planned grid of plausible combinations. Report which combinations were tested and why; do not quietly select the highest or lowest figure.
Your check: Can two individually small changes combine into a larger effect?
Explained answer: Yes. Rules can interact when they affect overlapping sets of tokens or source material.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 18 OF 24 · LANGUAGE EVIDENCE
18. Separate exploratory checks from a declared analysis
Researchers may discover new concerns after inspecting initial results. Exploring them is legitimate, but the final report should distinguish the primary analysis from later checks so readers understand how the conclusion was developed.
See it in context
A student runs the planned exact-form query, notices unusual repeated headlines, then adds a duplicate-removal check. That follow-up can improve the project while remaining labelled as a new diagnostic step.
The mistake to recognise
The mistake rewrites the history of the project and claims every helpful variant was planned in advance. This obscures the selection process and makes results harder to evaluate.
A safer method
Keep a simple chronological log: primary rule, findings, reason for each added variation, method and revised interpretation. Report unhelpful as well as helpful checks when they bear on the claim.
Your check: Is a post-hoc sensitivity check automatically invalid?
Explained answer: No. It can be informative when transparently identified as exploratory and interpreted cautiously.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 19 OF 24 · LANGUAGE EVIDENCE
19. Keep tests with constant data distinct from new-source replication
Changing analytic rules on the same source collection addresses robustness. Repeating the same rules on another independently sampled collection addresses replication. Extending to new populations or genres addresses broader generalisation.
See it in context
A class analyses two fictional school archives with identical queries. It finds similar rates in both; that is a different kind of support from analysing one archive twice under two tokenisation policies.
The mistake to recognise
The error merges all checks into one ‘verified’ status and conceals whether evidence came from independent new texts or only a recalculation of old ones.
A safer method
Record which dimension changed—analysis, data or target population. Different kinds of checks answer different credibility questions.
Your check: What does another independent corpus test that a tokeniser comparison cannot?
Explained answer: Whether a similar pattern appears in new source material under a comparable analysis.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 20 OF 24 · LANGUAGE EVIDENCE
20. Teach sensitivity checks with a visible spreadsheet
A useful classroom worksheet shows one row per version: baseline, duplicate removal, heading exclusion, lemma grouping and case handling. Students calculate each rate, then write one line explaining why the number changed.
See it in context
The table below uses simple invented counts so every rate can be checked without research software. It makes the numerator and denominator visible rather than letting a chart announce a corrected value without explanation.
The mistake to recognise
The weak activity asks pupils to use the largest rate as the ‘best answer’. That rewards choosing favourable settings instead of explaining the source and meaning of each estimate.
A safer method
Have learners identify which variants hold the target constant, which change the counted vocabulary item, and which change eligible text. Ask them to describe each difference in complete sentences.
Your check: Which skills are needed beyond division and multiplication?
Explained answer: Understanding source eligibility, the lexical target, what was changed and the scope of inference.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 21 OF 24 · LANGUAGE EVIDENCE
21. Apply the method to word learning
The practical educational goal is not a sensitivity statistic itself, but reliable decisions about which words to teach and how. A common word list may be unstable across genres; one spelling may be absent while related forms are widespread.
See it in context
A teacher considering run for a lesson may need its irregular past ran and varied senses. A strict exact-form list could underestimate encounters with the verb lemma. That does not make the original list mathematically wrong; it tells the teacher to choose a target aligned with the lesson.
The mistake to recognise
The mistake builds a universal teaching plan from one frequency ranking and assumes every corpus-processing decision is irrelevant.
A safer method
Combine frequency evidence with real student tasks, word meaning, register, morphological relations and retrieval practice. Let different measures inform rather than dictate the teaching sequence.
Your check: How does a sensitivity check help a teacher choose vocabulary?
Explained answer: It reveals whether a teaching recommendation depends heavily on a narrow query or source mixture and prompts more appropriate examples.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 22 OF 24 · LANGUAGE EVIDENCE
22. Respect source rights and learner privacy
Some corpora contain pupil essays, private messages or copyrighted articles. Reproducibility does not require publicly exposing every raw document when lawful access is restricted. The method can still be described and safe derived records can be shared where permitted.
See it in context
A school may publish an anonymised frequency table and detailed query rules without distributing identifiable original essays. Recreating the exact analysis may then require authorised access under the institution’s policies.
The mistake to recognise
The shortcut demands publication of sensitive material to prove methodological transparency or treats a public filename as blanket permission to reuse its contents.
A safer method
Share the maximum lawful, safe detail: inclusion rules, transformations, derived counts, code or query strings and a privacy note. Do not invent public accessibility that the material does not have.
Your check: Can a study explain its sensitivity checks without publishing private student prose?
Explained answer: Often yes, through permitted aggregated data, reproducible procedures and controlled access when necessary.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 23 OF 24 · LANGUAGE EVIDENCE
23. Summarise robustness without a misleading pass mark
There is no universal percentage-change threshold that certifies a corpus claim as robust for every research question. A small change may matter for an exam-word teaching decision; a larger rate change may be irrelevant if the conclusion stays the same.
See it in context
One analysis could have rates of 2,000 and 1,778 per million with identical ranking of teaching priorities, while another could change from a clear presence to a zero hit. The substantive conclusion matters as much as relative magnitude.
The mistake to recognise
The mistake applies an arbitrary ‘less than ten percent means robust’ rule without considering sampling, task and the choice that produced the change.
A safer method
State which part of the claim held, which changed and why. Use a consistent reporting table and a limitation statement rather than a decorative success badge.
Your check: What is a better final sentence than ‘the result passed all tests’?
Explained answer: ‘The observed pattern persisted under these specified checks, but depended on the target’s exact-form definition and the included text regions.’
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 24 OF 24 · LANGUAGE EVIDENCE
24. Finish with an eight-question sensitivity audit
Before publishing an adjusted vocabulary claim, identify the baseline, target, corpus snapshot, rationale for alternatives, exact change per run, updated numerator and denominator, magnitude of change and substantive conclusion. A strong report lets another reader reproduce the decisions.
See it in context
A cautious conclusion from the constructed workbook reads: ‘The exact-form baseline is 2,000 per million; duplicate and heading policies lower that observed rate, while lemma grouping counts a broader target at 2,900 per million.’ These are different operational calculations on an illustrative dataset.
The mistake to recognise
The overclaim says all alternative rates measure one universal quantity or prove students use the word more or less often. The numbers are invented and the variants alter source or target definitions.
A safer method
Keep the audit next to the final figure. If the claim only survives one arbitrary setting, narrow it and seek better evidence rather than hiding the sensitivity.
Your check: What is the most useful product of a sensitivity analysis?
Explained answer: A transparent statement of which conclusions survive justified choices and which depend on decisions the reader should know.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
Complete worked example with visible assumptions
The following table is an original constructed counting exercise, not a real corpus finding. Baseline target: exact written form run used as a verb; baseline eligible corpus: 20,000 word tokens; observed baseline hits: 40. Each subsequent row changes exactly one stated assumption relative to that baseline, without implying the altered rows are identical estimands. Every rate is hits ÷ eligible tokens × 1,000,000.
| Analysis version | What changes from baseline? | Eligible tokens | Qualifying hits | Occurrences per million | Interpretation |
|---|---|---|---|---|---|
| Baseline | Exact verb form run; full text | 20,000 | 40 | 2,000 | Narrow original target |
| D1 | Exclude one exact duplicate passage | 18,000 | 32 | 1,777.78 | Same target, new source inclusion |
| H1 | Exclude repeated headings | 19,000 | 28 | 1,473.68 | Same target, prose-only scope |
| L1 | Group inflected verb forms under lemma RUN | 20,000 | 58 | 2,900 | Different, broader lexical target |
| C1 | Use case-insensitive exact-form matching | 20,000 | 46 | 2,300 | Broader matching rule; inspect new hits |
Baseline: 40/20,000 × 1,000,000 = 2,000 occurrences per million. The denominator is the declared eligible token total, not the number of documents.
D1 excludes a duplicate passage contributing 2,000 tokens and eight hits, leaving 18,000 tokens and 32 hits. D1 rate = 32/18,000 × 1,000,000 ≈ 1,777.78. It is around 11.1 percent lower than the baseline for the resulting unique-text scope.
H1 excludes repeated headings contributing 1,000 tokens and twelve hits, leaving 19,000 tokens and 28 hits. H1 rate ≈ 1,473.68. This row removes heading material from both numerator and denominator and should be labelled a prose-only analysis.
L1 includes additional verb inflections, bringing qualifying hits to 58 while leaving the source total at 20,000. Its rate is 2,900. This is not the same exact-form estimand as baseline: the lexical target is broader by design.
C1 finds six extra case-variant matches for a total of 46 and a rate of 2,300. A real analysis must inspect those additional examples to ensure proper names, sentence starts and grammatical use did not introduce false positives.
All variant rows are independently compared with the same baseline rather than chained sequentially. Combining D1 and H1 would require checking whether their excluded material overlaps. Without that information, subtracting both exclusions and asserting a combined rate could be wrong.
The table supports methodological reasoning, not claims about actual prevalence of run in English or learning changes. A published report would also document source IDs, processing code or query settings, version timestamps and within-document variation.
Twelve situations that test transfer
Each scenario describes a decision a reader or researcher could face. The explanations are not empirical findings about the people named in the examples; they show how to keep the observation and its inference in proportion.
1. Unannounced duplicate policy
Situation. An updated frequency list removes copied messages.
What could be misleading. The rate changes because source rules changed.
Better reasoning. Report old and new counts and the duplicate criterion.
Ask next. Does the target lexical unit remain the same?
2. Different tokeniser
Situation. A software update splits contractions differently.
What could be misleading. Denominator or match conditions change.
Better reasoning. Recompute consistently and version processing.
Ask next. Were all texts identical?
3. Broader lemma search
Situation. A search adds ran and running to run.
What could be misleading. The estimand changes from form to lemma.
Better reasoning. Name the new target and compare as related but distinct analyses.
Ask next. Which language question needs the lemma?
4. Headline exclusion
Situation. A website repeats the target in every page title.
What could be misleading. Boilerplate may inflate a running-prose rate.
Better reasoning. Compare with a prose-only scope and document exclusions.
Ask next. Did the denominator also change?
5. Genre imbalance
Situation. A science corpus gains a large sports section.
What could be misleading. Pooled rate shifts with text mixture.
Better reasoning. Report genre composition and within-genre results.
Ask next. What population does the final rate represent?
6. Case-folding
Situation. Uppercase names accidentally enter a lowercase word search.
What could be misleading. More matches need not be more correct.
Better reasoning. Inspect newly included concordance examples.
Ask next. Do all extra hits share the intended sense?
7. Unseen data
Situation. The same method runs on a second independent archive.
What could be misleading. This is a different kind of evidence from analytic sensitivity.
Better reasoning. Label replication, not one-factor robustness.
Ask next. Are the source domains comparable?
8. Combined exclusions
Situation. A duplicate passage includes a repeated heading.
What could be misleading. Subtracting exclusion totals independently can double-remove overlap.
Better reasoning. Compute unique excluded tokens once before combining rules.
Ask next. Which source IDs overlap?
9. Extreme score change
Situation. A rare collocation’s MI changes with one event.
What could be misleading. Small samples can have fragile rankings.
Better reasoning. Display raw counts, source spread and score definition.
Ask next. Does the teaching recommendation survive?
10. Privacy constraint
Situation. Raw pupil essays cannot be shared publicly.
What could be misleading. Transparency must respect data protections.
Better reasoning. Provide permitted derived counts and method details.
Ask next. Can authorised reviewers reproduce the analysis?
11. Cherry-picked variant
Situation. Many settings are tested and only the highest rate is published.
What could be misleading. Selected results conceal instability.
Better reasoning. Keep a log and report material alternatives.
Ask next. Was the reported setting planned?
12. Word-list teaching
Situation. A frequency list changes after grouping irregular verbs.
What could be misleading. The list’s lexical unit changed.
Better reasoning. Link the definition to the student’s actual task.
Ask next. Is form recall or lemma understanding being taught?
Eight practice tasks and explained answers
Work each problem before reading the answer. Show the original observations, the selected rule and the interpretation. A useful response can be cautious without being vague: it should say exactly what the supplied evidence establishes.
Practice 1: Baseline arithmetic
Question. Forty hits in 20,000 tokens: calculate per million.
Worked answer. 40/20,000 × 1,000,000 = 2,000 per million.
Why a shortcut fails. Displaying 40 as a per-million rate forgets the denominator.
Practice 2: Duplicate removal
Question. Remove 2,000 tokens and eight hits. What remains and what is the rate?
Worked answer. 18,000 tokens, 32 hits, 32/18,000×1,000,000≈1,777.78.
Why a shortcut fails. Subtracting only hits but not tokens distorts the ratio.
Practice 3: Heading exclusion
Question. Remove 1,000 heading tokens and twelve hits from the baseline. Calculate the rate.
Worked answer. 19,000 tokens, 28 hits, 28/19,000×1,000,000≈1,473.68.
Why a shortcut fails. Reusing the old 20,000 denominator measures neither clear scope.
Practice 4: Lemma comparison
Question. The form query has forty hits; the verb lemma query has fifty-eight. Does the difference prove an error?
Worked answer. No. The lemma query includes additional inflected forms by definition and answers a broader question.
Why a shortcut fails. They are not directly identical estimands.
Practice 5: Case sensitivity
Question. Six new hits appear after case folding. What is the new toy rate?
Worked answer. 46/20,000×1,000,000=2,300; the six additional contexts must be checked for relevance.
Why a shortcut fails. A higher count does not certify correct sense or grammatical role.
Practice 6: One-change rule
Question. A researcher changes tokenisation and genre inclusion in one step. What is difficult?
Worked answer. Attributing the result to either factor independently, because both could contribute.
Why a shortcut fails. Without controlled variations, the cause of change is confounded.
Practice 7: Reproduction versus robustness
Question. A script is rerun on identical data with identical rules and returns the same count. What has been tested?
Worked answer. Reproducibility or repeatability of the original calculation, not robustness to alternative rules.
Why a shortcut fails. Repetition under unchanged assumptions does not challenge those assumptions.
Practice 8: Write a qualified finding
Question. Summarise the toy sensitivity exercise in two sentences.
Worked answer. ‘The constructed exact-form baseline was 2,000 per million; deduplication and heading exclusion reduced it under revised text scopes. A lemma query returned 2,900 for a different, broader target.’
Why a shortcut fails. A claim that language changed in reality is unsupported by synthetic recalculations.
Frequently asked questions: a corpus sensitivity analysis
What is corpus sensitivity analysis?
A documented test of how conclusions change under reasonable alternative data-processing or analytic choices.
Why is it useful?
It reveals whether a finding is driven by one narrow choice such as tokenisation, duplicates or source selection.
Is it the same as repeating an analysis?
No. Repeating identical operations checks reproducibility; changing defensible analytic settings checks robustness.
What is replication?
Checking a comparable claim using different suitable data under a stated approach.
Should one rule change at a time?
It is a useful diagnostic starting point; important interacting choices may also need joint checks.
Is the largest rate automatically best?
No. The target, source domain and method determine what is appropriate.
Can tokenisation affect both counts and denominator?
Yes. It changes how the text is segmented and sometimes which matches qualify.
Is a lemma search a sensitivity check of exact-form frequency?
It can expose dependence on lexical definition, but it estimates a broader target and must be labelled as a different question.
Why report raw counts?
They show the actual evidence and permit verification of every scaled rate.
Does removing duplicates always lower a rate?
No. The direction depends on how many target hits and eligible tokens are removed.
Can a large corpus still be biased?
Yes. Size does not automatically repair missing genres or selection bias.
What is a baseline?
The initial, fully documented analysis against which alternatives are compared.
What is robust if some numbers change?
A specified substantive conclusion may remain stable even when details vary under reasonable choices.
Do sensitivity checks prove causal learning gains?
No. They test analytic assumptions and do not replace learner outcome evaluation.
How should a student report the result?
Name the baseline, variant, changed rule, updated counts, interpretation and scope limit.
Words and concepts in this lesson
- Sensitivity analysis: A structured variation of plausible choices to examine dependence of results.
- Baseline: The original specified analysis.
- Robustness: Consistency of a conclusion under reasonable alternative analyses.
- Reproducibility: Ability to obtain the same result with the same data and procedure.
- Replication: Testing a comparable finding with additional data.
- Estimand: The precisely defined quantity a method intends to estimate.
- Tokenisation: Division of text into counted units.
- Lemma: A lexical base that groups relevant inflectional forms.
- Deduplication: Handling repeated text according to declared identity rules.
- Denominator: The eligible total used to calculate a rate.
- Genre: A form of communication whose purposes can affect language choices.
- Exploratory analysis: A later inquiry added to examine patterns or diagnose unexpected results.
Teach the method through four practical moves
Begin with a situation that requires a decision: choosing a phrase for a school essay, reading a scientific account or checking whether a word comparison is fair. State the decision before introducing its technical label. This puts the world and the communication problem before vocabulary terminology.
Next, show a compact table or a few complete examples. Have learners write down what is observed and which assumptions matter. Ask them to predict one outcome before revealing the result, then reconstruct the calculation, search or classification rather than copying it. Reproduction tests whether the procedure is understood.
Third, introduce one deliberately difficult case: an unseen word, a changed corpus filter, a rare co-occurrence or a contradictory example. Ask a learner to correct the conclusion, not simply apologise for the previous answer. The correction should identify the exact assumption that changed.
Finally, revisit the idea after a delay and transfer it to another subject. For example, a source-selection question in English can become a fairness question about Science experiments or a provenance question in History. A method becomes genuine vocabulary knowledge when the learner can explain and use it outside the original sentence.
Published research and method resources
- Flanagan (2025), reproducibility, replicability, robustness and generalizability in corpus linguistics
- Meylan and colleagues (2021), sensitivity and multiverse analyses for large language datasets
- Wegmann, Nguyen and Jurgens (2025), tokenisation and language variation
- Egbert, Biber and Gray (2022), effects of sampling and coverage on corpus representativeness
- LADAL, Reproducible Research: Principles and Practice
External publications support the conceptual guidance. Numerical examples above were independently constructed and do not reproduce their research datasets.
Related vocabulary methods and teaching hubs
- Corpus sampling frames — know what could enter the study
- Collocation association scores — understand what the statistic measures
- Collocation windows — see how direction and distance affect a count
- Subcorpora — compare source and genre groups explicitly
- Corpus deduplication — inspect repeated passages
- Vocabulary Learning Hub
- Vocabulary Article Library
Continue with the complete vocabulary evidence batch
The four guides address distinct questions: how a label is assigned, what a missing search result means, how word-partner rankings behave across samples, and whether a vocabulary finding survives defensible alternative rules.
- Annotation Guidelines — Define labels, boundary cases, review rules and unresolved decisions
- Zero-Hit Corpus Results — Check search failures, genre coverage and rare-word absence
- Collocation Ranking Stability — Compare top-k word-partner lists without assuming ranks are universal
- Vocabulary Learning Hub — return to reading, meaning, retention and vocabulary OS
- Vocabulary Article Library — follow related vocabulary concepts