Vocabulary | What Is Document-Level Bootstrapping? How Resampling Texts Reveals Word-Frequency Uncertainty

CHAPTER 1 OF 21

1. Start with the quantity the corpus reports

A typical corpus report combines occurrences of a target word with the eligible number of tokens. The resulting rate describes the observed collection. A question about how the rate might vary across other comparable source documents requires information about document-to-document variation, not only one summed numerator.

Look at a concrete example

Our constructed four-document example contains fourteen matches of the target word among eight hundred eligible tokens. The pooled rate is 14/800 = 0.0175, or 17.5 per thousand. It does not say how a new set of four documents would behave.

The mistaken shortcut

An analysis reports 17.500000 per thousand as though extra decimal places capture uncertainty. A mathematically exact observed ratio can be a fragile summary of a small, unbalanced collection.

What to do instead

State the target expression, corpus domain, source documents, counting rule and pooled rate. Then ask how much each document contributes before making claims about a broader population.

Check: What is the baseline rate for fourteen hits in eight hundred tokens?

Explained response: 14 divided by 800 multiplied by 1,000 equals 17.5 occurrences per thousand tokens, restricted to the toy corpus.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Next

CHAPTER 2 OF 21

2. Notice when source texts differ

Words cluster by topic, author, genre and communicative purpose. A set of four documents may contain very different numbers of target occurrences, so treating every word token as an independent draw can understate variation.

Look at a concrete example

A fictional report about filing requests may contain many instances of submit; a narrative description may contain none. The words within each text were not composed as unrelated coin flips.

The mistaken shortcut

A tempting shortcut pretends that 800 individual tokens are interchangeable independent observations even though hundreds come from the same writer or template.

What to do instead

Inspect per-document counts and lengths. Keep document IDs intact and choose a resampling unit that reflects the source design and plausible dependence.

Check: Why can words within one report be dependent?

Explained response: They share topic, writer, genre and often repeated phrasing; the occurrence probability is not identical across all document contexts.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 3 OF 21

3. Understand the bootstrap idea

A nonparametric bootstrap constructs many alternative samples from the observed units, with replacement, and recalculates the statistic in each sample. The resulting spread approximates sampling variability under assumptions about what units are exchangeable.

Look at a concrete example

With four source documents, one bootstrap draw samples four document IDs and can select the same ID more than once. The resampled set may include D01 twice and omit D02 entirely. This is intentional.

The mistaken shortcut

The mistake calls a repeated ID a data-entry error and deletes it from the bootstrap sample. Repetition is exactly what sampling with replacement permits, even though the original corpus contains one physical copy.

What to do instead

Define a draw as selecting the same number of source units with replacement. Keep the selected IDs visible and recompute total hits and total tokens for that draw.

Check: Can D01 appear twice in one bootstrap replicate?

Explained response: Yes. Sampling with replacement permits repeated selection of the same source record.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 4 OF 21

4. Choose documents rather than arbitrary words

A document-based approach preserves the internal word distribution of each selected text. Token-by-token resampling breaks the grouping and can create implausible pseudo-documents whose words have no actual shared context.

Look at a concrete example

When D02 is selected in the toy data, its eight hits and two hundred tokens enter together. The method does not randomly scatter those eight hits across the other three source documents.

The mistaken shortcut

The shortcut samples individual word tokens from the pooled list and assumes the result captures variation across writers or document topics. It does not reproduce document-level clustering.

What to do instead

When the intended sampling units are documents, resample complete documents; if the design samples speakers or classes, consider whether an even higher-level cluster should be the unit.

Check: Is a whole document always the best unit?

Explained response: No. The correct unit follows the sampling design and dependence structure; speakers, writers, schools or time clusters may be more appropriate.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 5 OF 21

5. Preserve different document lengths

Documents contribute differing token totals. A bootstrap replicate that repeats a long document can produce a larger denominator than the original sample, because it is a new pseudo-sample rather than a rearrangement of the old tokens.

Look at a concrete example

D04 is a four-hundred-token source. If selected twice instead of once, it adds eight hundred eligible tokens and eight target occurrences to a replicate’s total.

The mistaken shortcut

The mistaken calculation takes the resampled target-hit total and divides by the original fixed eight hundred tokens, even though the chosen source documents now contribute a different amount of text.

What to do instead

For a pooled token-rate estimand, sum the chosen documents’ hits and tokens within every replicate, then calculate their ratio using that replicate’s denominator.

Check: Why can a bootstrap replicate contain 1,200 tokens when the original four-document corpus has 800?

Explained response: It can select longer source documents more than once under sampling with replacement.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 6 OF 21

6. Recognise what a replicate is

One bootstrap replicate is one resampled collection and the statistic calculated from it. Several replicates reveal different possible totals. Thousands of replicates are commonly used for interval estimation, but their number alone does not guarantee a valid inference.

Look at a concrete example

A first illustrative draw may select D01, D01, D03 and D04. Another may select D02, D02, D04 and D04. Their rates differ because the source mix differs.

The mistaken shortcut

A misleading statement calls each replicate an independent new set of real documents. It is a constructed sample from the same observed sources, not additional field evidence.

What to do instead

Label replicated IDs and simulated totals explicitly. Treat the bootstrap distribution as a model-based view of variation, conditional on the observed corpus and resampling assumptions.

Check: Did each replicate collect new documents in the real world?

Explained response: No. It reuses the observed source records to approximate sampling variability.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 7 OF 21

7. Learn why the number of source units matters

A corpus can have millions of tokens but only a handful of genuinely distinct source documents. The number and diversity of independent primary units limit what resampling can reveal about populations beyond them.

Look at a concrete example

Imagine one large textbook and three short handouts. A bootstrap that samples four documents cannot invent new authors, topics or genres absent from the original collection. Its many replicates still depend on four source units.

The mistaken shortcut

The shortcut equates ten thousand bootstrap draws with ten thousand independent observed documents. Computation does not multiply the source information.

What to do instead

Report both tokens and distinct source units. Be cautious when only a few independent documents exist; seek additional relevant sources before making strong population claims.

Check: Are 10,000 bootstrap replicates equivalent to 10,000 new observations?

Explained response: No. They are repetitions of a calculation using the existing observations.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 8 OF 21

8. Keep the sampling frame explicit

Bootstrap inference presumes that the observed units can stand in for the population the study targets, subject to the chosen resampling model. If only examination notices were sampled, bootstrapping them cannot describe informal conversation.

Look at a concrete example

A set of school science worksheets may give a stable bootstrap distribution for that type of material while revealing nothing direct about students’ spoken vocabulary.

The mistaken shortcut

The mistake treats a narrow convenience collection as representative because the software produced a confidence interval. The interval may be precisely calculated for an inappropriate domain.

What to do instead

Describe source eligibility, selection mechanism, genre, time and known omissions. Narrow the inference when the source frame cannot support the broader target.

Check: Does a well-computed interval repair a corpus lacking the relevant genre?

Explained response: No. The source domain remains a limitation of what any statistic can mean.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 9 OF 21

9. Understand the replacement rule

Each draw samples from the observed source IDs, permitting repeats and omitting some records. Without replacement from the same four documents, drawing four items would simply reproduce the original complete set and show no variability in its pooled rate.

Look at a concrete example

A no-replacement draw of all four documents contains D01, D02, D03 and D04 in some order every time. Its pooled hits and tokens remain fourteen and eight hundred.

The mistaken shortcut

The error calls shuffling four file IDs a bootstrap. A permutation of a complete fixed set does not alter which documents are included.

What to do instead

Sample with replacement for the classical nonparametric bootstrap and record each draw. Distinguish the task from permutation tests and subsampling without replacement.

Check: Why is drawing all four documents without replacement insufficient here?

Explained response: It changes only order, not the included source units, so the pooled rate stays constant.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 10 OF 21

10. Compute the first replicate correctly

The first teaching draw is D01, D01, D03 and D04. D01 contributes two hits and one hundred tokens each time; D03 contributes zero and one hundred; D04 contributes four and four hundred. The replicate’s total is eight hits in seven hundred tokens.

Look at a concrete example

The resampled rate is 8/700×1,000≈11.43 per thousand. That differs from the observed original 17.5 per thousand because D02, which contained eight hits, was absent.

The mistaken shortcut

The shortcut divides eight by four because there are four documents, yielding a number that is not the requested per-token rate.

What to do instead

Sum document-level hits and denominators before taking the ratio. Use per-document averaging only if the estimand is explicitly the mean document rate.

Check: What are the first replicate’s total hits and tokens?

Explained response: Eight matches and seven hundred eligible tokens, giving roughly 11.43 per thousand.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 11 OF 21

11. Compute a replicate dominated by longer texts

The second draw selects D02 twice and D04 twice. Their hit counts are eight, eight, four and four, and their token counts are two hundred, two hundred, four hundred and four hundred.

Look at a concrete example

Total hits are twenty-four among 1,200 tokens. The result is 20 per thousand, higher than the original rate. D02 and D04 dominate the pseudo-sample, but they do not imply a real-world increase.

The mistaken shortcut

The mistake uses the original total-token denominator for this draw, converting the twenty-four hits into an inflated 30 per thousand rate.

What to do instead

Use the actual selected-document token totals. This makes each bootstrap replicate an internally coherent pooled-rate calculation.

Check: What rate follows from twenty-four hits in 1,200 tokens?

Explained response: 20 per thousand, or 20,000 per million.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 12 OF 21

12. Do not mistake four draws for a confidence interval

A few hand-calculated replicates are excellent for understanding the mechanism but cannot support a stable bootstrap percentile interval or a strong population inference. Actual interval work normally requires a much larger collection of replicates and attention to sample design.

Look at a concrete example

The table includes four illustrative draws with rates around 11.43, 20, 30 and 5.71 per thousand. This is a teaching selection, not a random ensemble adequate for estimating percentiles.

The mistaken shortcut

The misleading step takes the minimum and maximum of these four handpicked draws and calls them a 95 percent confidence interval. They have no such coverage justification.

What to do instead

Explain that the four rows demonstrate the arithmetic only. For a real study, produce a suitable number of programmed draws, document randomness and use an interval method appropriate to the design.

Check: Can the smallest and largest values of four demonstration replicates be labelled a 95 percent interval?

Explained response: No. They are illustrative extremes of four chosen examples, not a defensible interval estimate.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 13 OF 21

13. Interpret the bootstrap distribution

A full bootstrap run produces a distribution of a chosen statistic over many resampled datasets. Its centre and spread help describe sensitivity to resampling; formal interval interpretation requires suitable assumptions and methodology.

Look at a concrete example

If many draws that favour D02 return high rates while those omitting D02 return lower rates, the resulting distribution illustrates the role of source heterogeneity.

The mistaken shortcut

The mistake assumes the bootstrap distribution is the probability that a word itself has a certain fixed frequency in every setting. The distribution is conditional on the observed source design and chosen estimator.

What to do instead

Report the statistic, resampling unit, number of draws, interval construction and a statement about what population, if any, is being inferred.

Check: Does a narrow bootstrap distribution prove an unbiased sample?

Explained response: No. It can reflect repeated similar source documents from an unrepresentative domain.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 14 OF 21

14. Contrast token-level and file-level resampling

Gries (2022) argues that corpus statistics should account for uncertainty and demonstrates how word-based resampling can be inappropriate compared with text- or file-based approaches. The general lesson is that the chosen unit matters in language data.

Look at a concrete example

Words within a report often share the same topic, so randomly treating them as isolated interchangeable observations discards structure that a file-based resample preserves.

The mistaken shortcut

The shortcut declares that every corpus statistic can use an ordinary independent-token confidence interval because the dataset contains many word tokens.

What to do instead

Match the inference model to how language was sampled and clustered. Document when the source hierarchy calls for text-, author- or higher-level resampling.

Check: Why might a token bootstrap be overconfident for source-dependent language?

Explained response: It may ignore variation between documents and overstate how many independent observations the corpus provides.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 15 OF 21

15. Know when stratification should be preserved

A corpus often deliberately samples genres or periods in certain proportions. Naively resampling from the pooled source list can change those proportions, producing a distribution that mixes sampling uncertainty with a change in the intended target mixture.

Look at a concrete example

Suppose the target is half science reports and half family notices. A pooled bootstrap could select all science documents in some draws; a stratified bootstrap within each genre retains their intended allocation.

The mistaken shortcut

The error assumes every corpus should be resampled by blindly drawing all documents from one undifferentiated set. The original design may have controlled strata.

What to do instead

If the estimand relies on specified strata, consider sampling within each stratum and applying correct weights. Explain whether the purpose is to hold mixture fixed or examine sensitivity to it.

Check: Should every replicate change the genre mixture?

Explained response: Not necessarily. A stratified design may need its allocation maintained.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 16 OF 21

16. Think about author and source dependence

Documents by the same author or produced from a shared template may not represent independent draws. Resampling each document separately can still understate uncertainty if the intended independent unit is actually the writer or institution.

Look at a concrete example

Ten documents written by one person share style and vocabulary; documents from twenty writers may represent a broader sample of linguistic choices. The correct independence assumption depends on the question.

The mistaken shortcut

The misleading shortcut calls every page an independent speaker or writer. Duplicate templates can compound the error.

What to do instead

Keep author or source clusters when available and permitted. Where dependence occurs at the author or organisation level, consider a cluster bootstrap or a model appropriate to the sample.

Check: Is one hundred files necessarily one hundred independent authors?

Explained response: No. Files can share authorship, source templates or production context.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 17 OF 21

17. Interpret uncertainty alongside dispersion

A pooled word rate says how often an expression appears per eligible token; dispersion shows how widely it appears across documents. Two word pairs can have equal pooled rates but very different source concentration and uncertainty.

Look at a concrete example

One candidate may appear fourteen times spread across four reports; another may appear fourteen times in one report. The second is more concentrated and may behave differently under document resampling.

The mistaken shortcut

The mistake reports identical normalised rates as identical evidence of general usefulness across writers or genres.

What to do instead

Show the source spread as well as frequency. When relevant, report intervals for more than one property, rather than reducing a multidimensional language pattern to one point.

Check: Do equal token rates guarantee equal document coverage?

Explained response: No. A rate can be driven by a small number of texts while another is widespread.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 18 OF 21

18. Use a small study only for the conclusions it supports

Classroom demonstrations help learners grasp probability, counting and corpus design. They should never be presented as measured evidence that particular vocabulary occurs at a given rate in real English.

Look at a concrete example

The four-document exercise can teach why D02 matters, why denominators vary and how repeated IDs are intentional. It cannot establish the uncertainty of word use in schools.

The mistaken shortcut

The erroneous headline upgrades fabricated training numbers to national frequency statistics or treats them as empirical evidence of teaching success.

What to do instead

Label constructed data clearly and use authentic, permitted corpus evidence for substantive prevalence claims. Read original research separately from the illustrative arithmetic.

Check: Can the toy data support a national English frequency claim?

Explained response: No. They are made-up observations designed only to illustrate the calculation.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 19 OF 21

19. Relate the method to vocabulary teaching

Teachers can use source-level variation as a reminder that an apparent target word’s frequency may reflect the chosen worksheets rather than its usefulness for all writing tasks. More pertinent examples matter more than a numerically impressive pooled total.

Look at a concrete example

The verb justify might recur in exam instructions but not in storybooks. A teacher should consider exam function, semantics and learner readiness alongside any corpus rate.

The mistaken shortcut

The shortcut selects words from the top of a single frequency list and assumes that every corpus variation is irrelevant to student tasks.

What to do instead

Use frequency as one signal. Offer varied example sentences, retrieval practice and checks that learners can apply the word in the relevant discipline.

Check: What should a teacher do when a word’s rank varies sharply by genre?

Explained response: Inspect the genres and choose examples aligned with the student’s actual reading or writing task.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 20 OF 21

20. Audit the assumptions before computing many draws

A reliable resampling report names the target word or construction, the unit of source selection, corpus frame, document lengths, number of source units, bootstrap sampling method, number of draws, statistic and interval approach.

Look at a concrete example

A reproducible toy report would show D01–D04 with raw counts, then specify four draws with replacement and the pooled-rate formula. A real study would additionally specify the actual random procedure and report a validated interval.

The mistaken shortcut

The mistake publishes only one confidence interval without telling readers whether the method resampled documents, sentences or individual tokens.

What to do instead

Provide a concise method box that makes it possible to repeat the calculation and judge whether the assumptions suit the corpus.

Check: What is the most important question before using a file bootstrap?

Explained response: Whether files are defensible resampling units for the target population and the dependence among source texts.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Next

CHAPTER 21 OF 21

21. Finish with the inference boundary

Document-level bootstrapping is a useful way to model the influence of source selection. It is not a substitute for relevant coverage, independent sampling, appropriate clustering and contextual reading. The goal is calibrated confidence, not certainty.

Look at a concrete example

A careful conclusion might say that the observed 14/800 rate varied substantially across four demonstration resamples and that this illustrates document dependence. It would not claim a real population interval from four synthetic files.

The mistaken shortcut

The misleading conclusion treats many resamples as independent newly observed data or assumes precise intervals correct an unrepresentative collection.

What to do instead

Report exactly what was resampled, what changed and what remains unknown. If source diversity is poor, improve the underlying data rather than polishing the statistic.

Check: What does a bootstrap add to an observed word count?

Explained response: A structured view of how a statistic varies when plausible source units are resampled, conditional on the method and sample.

To apply this idea outside the demonstration, first name what was counted or classified, then show which sources and rules made that result possible. Ask what would change under a reasonable alternative and avoid turning one local finding into a claim about every learner, document or language variety. A precise, narrower statement is more useful than an impressive statistic without a defensible scope.

Contents · Previous · Worked evidence

Worked evidence: inspect every row

Here are four entirely fictional documents. Their target counts and token totals are invented to make manual calculations possible. Every bootstrap draw selects four whole document IDs with replacement, recomputes summed hits and token totals, and reports occurrences per thousand. Only four illustrative draws are shown; these do not form a formal confidence interval.

Document or draw	Selected source IDs	Target hits	Eligible tokens	Rate per 1,000
D01	Original document	2	100	20
D02	Original document	8	200	40
D03	Original document	0	100	0
D04	Original document	4	400	10
Baseline	D01 D02 D03 D04	14	800	17.50
Illustration B1	D01 D01 D03 D04	8	700	11.43
Illustration B2	D02 D02 D04 D04	24	1,200	20.00
Illustration B3	D01 D02 D02 D03	18	600	30.00
Illustration B4	D03 D03 D03 D04	4	700	5.71

The original collection contains 2+8+0+4 = 14 hits and 100+200+100+400 = 800 eligible tokens. Its pooled rate is 14/800×1,000 = 17.5 occurrences per thousand. The observed document-specific rates range from zero to forty per thousand.

Bootstrap B1 repeats D01 and omits D02. Its hits are 2+2+0+4 = 8, its token total is 100+100+100+400 = 700, and its rate is approximately 11.43 per thousand.

Bootstrap B2 includes two copies of each D02 and D04 in the constructed pseudo-sample. It has 8+8+4+4 = 24 hits in 200+200+400+400 = 1,200 tokens, so its rate is exactly 20 per thousand.

Bootstrap B3 has 2+8+8+0 = 18 hits in 100+200+200+100 = 600 tokens, for 30 per thousand. Bootstrap B4 has four hits among 700 tokens, about 5.71 per thousand. Source mix changes the observed pooled rate.

These four examples deliberately explore noticeably different source combinations. They are neither a random sample from thousands of computed replicates nor an empirically calibrated interval. A proper interval analysis requires an appropriate resampling scheme, many draws and consideration of design and source independence.

Source-level resampling preserves each document’s count and length together. If the project sampled authors or schools rather than documents independently, that higher-level sampling structure may require a different bootstrap.

Twelve applied decisions: what the method catches

These fictional cases connect the vocabulary concept with school reading, writing, language research and everyday communication. Each is designed to expose one boundary or incorrect inference, not to claim measured outcomes.

1. Copied worksheets

Situation. Fifty documents all reproduce one template.

Why it is difficult. The apparent sample size exaggerates independent variation.

Sound response. Deduplicate or account for common source lineage before resampling.

Transfer question. What is the independent sampling unit?

2. One long handbook

Situation. One file contributes most eligible tokens.

Why it is difficult. One source can dominate the pooled rate.

Sound response. Report document lengths and examine its influence.

Transfer question. Would an equal-document average answer a different question?

3. Genre quotas

Situation. The study deliberately uses equal science and narrative samples.

Why it is difficult. Pooled draws can alter the intended mixture.

Sound response. Use a design-conscious stratified bootstrap.

Transfer question. Which target shares must remain fixed?

4. Author clusters

Situation. Ten essays come from one writer.

Why it is difficult. Treating them as independent authors is misleading.

Sound response. Resample writers or use an appropriate clustered design.

Transfer question. Which independence assumption matters?

5. Short texts

Situation. One document holds ten tokens.

Why it is difficult. Ratios can swing dramatically with its inclusion.

Sound response. Show raw denominators and avoid overclaiming.

Transfer question. Would more relevant source data help?

6. Missing genre

Situation. The corpus has no spoken dialogue.

Why it is difficult. Resampling does not create unseen genres.

Sound response. Narrow claims or collect ethically authorised speech.

Transfer question. Which population is actually covered?

7. Four hand draws

Situation. A classroom uses four demonstration replicates.

Why it is difficult. The endpoints are not formal 95% quantiles.

Sound response. Label them teaching examples and avoid interval claims.

Transfer question. What full procedure would be needed?

8. Token bootstrap

Situation. A method resamples each word independently.

Why it is difficult. Source clustering is destroyed.

Sound response. Use whole-text units where justified.

Transfer question. How do repeated phrases affect dependence?

9. Changed query

Situation. One run counts run; another RUN lemma.

Why it is difficult. Different targets produce different numerators.

Sound response. Hold the target fixed for a resampling variability study.

Transfer question. Is this a bootstrap or an estimand change?

10. Rare phrase

Situation. A collocation appears in one text only.

Why it is difficult. Bootstrap replicates often omit that source.

Sound response. Report concentration and source coverage.

Transfer question. Does a high score prove general popularity?

11. Reader exposure

Situation. Repeated newsletters are actual reader encounters.

Why it is difficult. Unique-text deduplication could erase exposures.

Sound response. Choose unit to match the exposure question.

Transfer question. Are we counting independent authors or encountered pages?

12. Reporting intervals

Situation. A published chart shows an interval with no method note.

Why it is difficult. Readers cannot judge what was resampled.

Sound response. Document unit, draws, statistic, interval method and limits.

Transfer question. Could an auditor reconstruct one replicate?

Eight questions with model explanations

Answer from the stated rules before viewing the solution. For quantitative work show the units and denominator. For language labels cite the exact sentence evidence and preserve ambiguity when the supplied context is inadequate.

Practice 1: Baseline rate

Question. Calculate a pooled per-thousand rate for 14 hits in 800 tokens.

Worked answer. 14÷800×1,000=17.5 per thousand.

Why the easy shortcut fails. Dividing by the four documents gives a different, not per-token, quantity.

Practice 2: Replacement meaning

Question. May D01 appear twice in a bootstrap sample of four document IDs?

Worked answer. Yes. Resampling with replacement permits repeated source IDs and omission of others.

Why the easy shortcut fails. Shuffling documents without replacement would leave pooled counts unchanged.

Practice 3: Replicate B1

Question. Find total matches, tokens and rate for D01 D01 D03 D04.

Worked answer. 8 matches, 700 eligible tokens; 8/700×1,000≈11.43.

Why the easy shortcut fails. Keeping the baseline denominator 800 would miscalculate the replicate.

Practice 4: Replicate B2

Question. Find the rate when D02 and D04 are each drawn twice.

Worked answer. 24 matches in 1,200 tokens = 20 per thousand.

Why the easy shortcut fails. The pseudo-sample can exceed the original word count because long files are repeated.

Practice 5: Replicate B3

Question. D01 D02 D02 D03: calculate the per-thousand rate.

Worked answer. 18 hits in 600 tokens = 30 per thousand.

Why the easy shortcut fails. A mean of document rates answers a different estimand.

Practice 6: Four-draw interval

Question. Can 5.71 and 30 be called a 95% confidence interval?

Worked answer. No. They are illustrative minimum and maximum of four constructed examples; proper interval construction needs a suitable bootstrap distribution.

Why the easy shortcut fails. Handpicked extremes offer no coverage guarantee.

Practice 7: Independent source

Question. If ten files are all from the same author, what uncertainty remains?

Worked answer. Author-specific dependence and limited writer coverage; consider higher-level clusters or new independent authors.

Why the easy shortcut fails. Counting ten files as ten independent writers would be misleading.

Practice 8: Interpret source coverage

Question. Can bootstrapping scientific reports estimate slang in casual peer talk?

Worked answer. Not without relevant source coverage; resampling only uses the observed scientific reports.

Why the easy shortcut fails. Repeated calculations cannot create language data from a missing genre.

Frequently asked questions: document-level bootstrapping

What is document-level bootstrapping?

Resampling complete documents, usually with replacement, to investigate how a corpus statistic varies under a specified source-sampling model.

Does the bootstrap create new real documents?

No. It creates constructed pseudo-samples from the observed source collection.

Why sample documents rather than individual words?

Words within a document share writer, topic and style, so token-level independence may be inappropriate.

Does every corpus require a document bootstrap?

No. The correct method depends on target, sampling design and dependence; files may not be independent units.

What is a bootstrap replicate?

One resampled set and the statistic calculated from it.

Can the same document appear more than once in a draw?

Yes. That is permitted under resampling with replacement.

Why can a replicate have more tokens than the original?

It may select long original documents multiple times.

Do four illustration draws form a confidence interval?

No. They explain the mechanism only.

What affects a trustworthy interval?

Source sampling assumptions, unit choice, sample size, number of draws and interval method.

Can bootstrapping fix an unrepresentative corpus?

No. It cannot generate sources outside the collected domain.

What is a stratified bootstrap?

A design that resamples within defined groups, often preserving intended allocation or weighting.

When should authors or schools be sampled instead?

When dependence and the target population are organised primarily at those levels.

Should token rates be recalculated each draw?

Yes. Sum hits and tokens from the selected source units in that replicate.

What should accompany the final statistic?

Corpus scope, observed counts, independent source units, resampling method and honest limits.

Glossary for careful readers

• Bootstrap: Resampling observed units with replacement to examine variation in a statistic.
• Replicate: One constructed resampled dataset and its calculated result.
• Source unit: The document, author or other unit that the sampling scheme treats as one observation.
• Cluster: A group of related observations that may not be statistically independent.
• Token rate: Occurrence count divided by eligible word-token count.
• Stratum: A predeclared source group such as genre or year.
• Dispersion: How widely occurrences spread across sources or segments.
• Confidence interval: An inferential interval whose interpretation depends on an appropriate procedure and assumptions.
• Source heterogeneity: Variation among documents or writers in topic, style and rates.
• Sampling frame: The eligible domain from which source units may be selected.
• Exchangeability: A modelling assumption allowing selected units to stand in for one another under resampling.
• Robustness: Whether a substantive conclusion survives reasonable changes in analysis choices.

A small teaching sequence

Begin with the real question before defining the technical label. Present one short example that a learner can understand, then ask the learner to make a prediction. Use the prediction to reveal which observation, rule or source detail determines the correct result. This approach develops reasoning instead of asking students to collect terminology without purpose.

Next, let students reconstruct one complete worked example. They should name the source unit, write the rule, find the relevant evidence and report a conclusion with a stated limit. A peer can inspect each step. If the conclusion cannot be reconstructed from the displayed observations, revise the explanation rather than adding more confident language.

Then deliberately vary one condition: a different document, an ambiguous token, a missing source or a changed counting rule. Ask the students which parts of the previous conclusion remain defensible. Where the question itself changes, require them to say so instead of treating every numerical difference as an error.

Finally, revisit the idea after a short delay and apply it to unfamiliar reading material. Check recognition, explanation and independent transfer separately. The learning outcome is the ability to use a concept to make a better language decision, not merely to recall its definition.

Research and methodological sources

• Gries (2022), Toward more careful corpus statistics: uncertainty estimates for frequencies, dispersions and association measures
• Cambridge University Press, Designing and Evaluating Language Corpora — domain considerations
• Vocabulary | What Is a Corpus Sensitivity Analysis?

The external resources support the methodological distinctions. Numerical examples and classroom scenarios here remain original teaching illustrations.

Continue through related vocabulary guides

• Vocabulary | What Is a Corpus Sensitivity Analysis?
• Vocabulary | What Is an Annotation Guideline?
• Vocabulary | What Is Inter-Annotator Agreement?
• Vocabulary | What Is a Gold-Standard Corpus?
• Vocabulary | What Is a Corpus Sampling Frame?
• Vocabulary | What Is Normalised Word Frequency?
• Vocabulary Learning Hub
• Vocabulary Articles Index

Return to contents · Vocabulary Learning Hub