Vocabulary | What Is a Corpus Sampling Weight? How Uneven Text Genres Change Frequency Estimates

CHAPTER 1 OF 23

1. Begin with the target domain rather than the spreadsheet

An estimate is meaningful only relative to the population or language domain it aims to describe. A corpus might describe school announcements, public articles, classroom explanations or a specified mixture. The target, not the neatness of the sample table, determines whether weighting is sensible.

See the method in context

A project with equal numbers of tokens from notices and project reports may be excellent for comparing those genres. It does not automatically reproduce the actual composition of an archive in which notices dominate.

The mistake hiding inside a plausible answer

The mistake labels an intentionally equal sample ‘representative of all school messages’ because each genre received the same amount of attention. Equal allocation and population proportionality are different goals.

Make a better decision

Write the target before choosing weights. Specify the source population, unit, period and intended inference. If the target distribution is unknown, do not invent a convenient benchmark to create a corrected-looking number.

Try this: Does 50/50 sampling guarantee that the population itself is 50/50?

Explanation: No. It describes the chosen sample allocation, which may deliberately oversample a rare but important category.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Next

CHAPTER 2 OF 23

2. Distinguish balance from representativeness

Balance is used in different ways across corpus linguistics. It can mean equal coverage of categories, a chosen proportional mixture or broader variation across genres. Representativeness must be judged relative to the intended domain and the particular language features being estimated.

See the method in context

A corpus with equal portions of fiction, science and notices can be balanced for comparing these three genres, yet unrepresentative of a domain in which one genre accounts for most eligible language tokens.

The mistake hiding inside a plausible answer

The shortcut treats balanced as an automatic quality certificate. It overlooks which genres were included, how text lengths differ and whether the target population is even known.

Make a better decision

Explain what was balanced—document count, token count, writers, genres or periods—and why that choice serves the analysis. Do not use an undefined balance label as a substitute for a sampling description.

Try this: Can a corpus be balanced for one research purpose and not another?

Explanation: Yes. Equal genres may support comparisons while not supplying the proportions needed to estimate total domain-wide token exposure.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 3 OF 23

3. Identify the real unit of the weight

A weight applies to some observational unit. In survey statistics it often represents persons or households. In a corpus it may be associated with documents, source texts, speakers, writers or token groups. The correct formula depends on the unit used to define both the sample and target.

See the method in context

If a domain contains eighty percent of its eligible tokens in notices, that is a token share. If eighty percent of its documents are notices, that is a document share. Those facts can differ greatly when notices are shorter than reports.

The mistake hiding inside a plausible answer

A misleading calculation multiplies token-frequency rates by the percentage of documents without showing that this is the appropriate estimand. The resulting number may be neither a valid document average nor a valid token average.

Make a better decision

Declare whether the target is average word rate per token, per document, per writer or per communication event. Align the benchmark and weight formula with that target.

Try this: Are eighty percent of documents and eighty percent of words interchangeable?

Explanation: No. Text lengths vary, so document proportions and token proportions need not match.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 4 OF 23

4. Understand equal allocation across strata

Stratification divides the sampling domain into meaningful categories before selection. Equal allocation assigns similar amounts to each selected category, often improving the ability to examine a rare group. It does not necessarily reflect the domain’s natural composition.

See the method in context

A school corpus designer may collect 10,000 notice tokens and 10,000 report tokens so both genres support detailed classroom comparison. The collected token shares are fifty percent each, regardless of the archive’s actual proportions.

The mistake hiding inside a plausible answer

The false interpretation says a word rate calculated from that pooled sample must describe the entire school’s written communication. It actually describes the deliberately selected 50/50 mixture.

Make a better decision

Use equal allocation for its stated purpose, including comparison or instruction. For a population-wide token estimate, obtain justified target token shares and consider an appropriate mixture-weighted estimate.

Try this: When is a 50/50 text collection useful even if the domain is 80/20?

Explanation: It can provide sufficient examples from the smaller genre for meaningful comparison; its pooled rate must be interpreted as a 50/50 sample unless adjusted appropriately.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 5 OF 23

5. Learn proportional allocation

Proportional allocation attempts to match known strata sizes when drawing the sample. If the target mixture is 80% notice tokens and 20% report tokens, an idealised 20,000-token proportional collection would contain 16,000 notice tokens and 4,000 report tokens.

See the method in context

Such a design can estimate the target token mix more directly if selection within each stratum is appropriate and coverage is sound. It does not remove errors caused by missing sources, wrong genre labels or unusual document lengths.

The mistake hiding inside a plausible answer

The shortcut assumes a proportional allocation guarantees representativeness. A poorly selected set of 16,000 notice tokens could still come from one repetitive document, failing to capture variation across writers and purposes.

Make a better decision

Describe both proportional targets and within-stratum selection. Where possible, sample source units using a defensible method and then inspect their token counts; achieving exact token quotas can itself require care.

Try this: Why is matching 80/20 token shares not a complete scientific method?

Explanation: It says little about whether chosen texts within each stratum represent the domain’s language variation or are duplicated.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 6 OF 23

6. Understand why a weighted average differs from a simple average

A token-domain rate across disjoint genres can be expressed as the sum of each genre’s within-group rate times its target token share, provided the shares and rates match the intended domain. The weights describe how much each category should contribute to the final rate.

See the method in context

In the fictional workbook, notices have ten target matches per thousand tokens and reports have two. The 50/50 sample rate is six per thousand. With target shares of 0.8 and 0.2, the mixture rate is 0.8×10 + 0.2×2 = 8.4 per thousand.

The mistake hiding inside a plausible answer

The error uses the unweighted six per thousand as though it describes an 80/20 domain. The arithmetic is correct for its sample but corresponds to a different mixture.

Make a better decision

Show both figures with labels: observed 50/50 sample rate and hypothetical target-mixture-weighted estimate. State that the target share must be known or credibly estimated before interpreting the weighted result as a real domain estimate.

Try this: How can a sample have rate six while an 80/20 target mixture has rate 8.4?

Explanation: Notices have the higher within-genre rate; increasing their target contribution from 50% to 80% raises the combined estimate.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 7 OF 23

7. Calculate a simple target-share adjustment

For an illustrative stratified token mixture, a convenient adjustment factor for each genre is its target token proportion divided by its observed sample token proportion. This form is valid for re-expressing a mixture of observed stratum rates; it is not automatically the design weight for every corpus-sampling scheme.

See the method in context

If notices occupy 0.8 of target tokens but 0.5 of sample tokens, their mixture factor is 0.8/0.5=1.6. Reports occupy 0.2 of the target and 0.5 of the sample, so their factor is 0.2/0.5=0.4.

The mistake hiding inside a plausible answer

The mistake treats 1.6 as a claim that each notice word occurred 1.6 times. A weight is part of an estimating formula, not a new observation. The underlying sample still has its original raw counts.

Make a better decision

Label the weights and benchmark, keep observed hits and tokens untouched, and compute the weighted numerator and denominator separately if using a weighted-count demonstration.

Try this: Why does a genre with target share 20% but sample share 50% receive factor 0.4?

Explanation: It was oversampled relative to the target mixture and is downweighted in that particular descriptive estimate.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 8 OF 23

8. Keep raw and weighted counts distinct

Weighting can generate pseudo counts for an estimate. Those weighted values should not be described as observed occurrences, documents or participants. Readers need both actual evidence and transformed results to assess how much information the estimate rests upon.

See the method in context

The toy sample records 100 matches in 10,000 notice tokens and 20 matches in 10,000 report tokens: 120 observed hits. Multiplying by 1.6 and 0.4 yields a weighted numerator of 168, but nobody observed 168 actual target tokens.

The mistake hiding inside a plausible answer

The headline ‘We found 168 uses of submit’ would be false for the sample. The observed count is 120; 168 is a scaled quantity in a particular mixture-adjusted calculation.

Make a better decision

Display ‘observed count’ and ‘weighted estimate component’ in separate columns, retain units and explain the benchmark. Never replace the source register with a rounded weighted table.

Try this: How many target tokens were actually observed in the constructed sample?

Explanation: One hundred twenty. The weighted sum of 168 is a calculation, not a count of extra observed word occurrences.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 9 OF 23

9. Link weights with the correct denominator

A weighted frequency must use a numerator and denominator transformed under consistent weights. Weighting the hits without accounting for the corresponding token totals yields a rate for no coherent target mixture.

See the method in context

Our notice portion has 10,000 sampled tokens and 100 hits, both multiplied by 1.6. Its weighted components become 16,000 tokens and 160 hits. The report portion becomes 4,000 weighted tokens and eight weighted hits.

The mistake hiding inside a plausible answer

The error reports weighted hits 168 divided by the original denominator 20,000 without confirming how the weighting scheme defines its denominator. This toy scheme happens to preserve that denominator, but that is a consequence of its factor normalisation—not a general rule.

Make a better decision

Write the weighted ratio explicitly as sum of weighted hits divided by sum of weighted eligible tokens. In the toy example both sums are visible, and the rate is 168/20,000=8.4 per thousand.

Try this: Why do both the weighted numerator and weighted denominator matter?

Explanation: Because a rate is a ratio describing the same weighted target distribution, not a collection of unrelated adjusted counts.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 10 OF 23

10. Recognise unequal inclusion probabilities

Probability sampling can give some eligible source units a higher chance of selection than others. Basic design weights often use the inverse of an inclusion probability so selected units represent different amounts of the frame.

See the method in context

If one eligible type of source is sampled with probability one half and another with probability one quarter, their basic inverse-probability weights are two and four respectively. The latter selected unit represents more eligible units under that simplified design.

The mistake hiding inside a plausible answer

The mistake assumes all corpus sampling weights are merely percentages of genres. Some are design-based weights tied to how texts were chosen; these require a documented probability sampling process.

Make a better decision

Keep selection probabilities and source-unit IDs. If selection is purposive or convenience-based, do not fabricate inverse probabilities. The right estimator depends on the target unit and design.

Try this: What is a basic inverse-probability weight for inclusion probability 0.25?

Explanation: 1/0.25=4, under the simple design-weight definition. Whether this applies to a particular word-rate calculation requires matching the estimand and sampling units.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 11 OF 23

11. Do not convert document weights into token weights casually

A short notice and a long report may have different token totals. Applying source-unit weights to estimate a word-token rate requires accounting for each source’s eligible tokens as well as its target hits. The weighted ratio need not equal a simple average of document rates.

See the method in context

Imagine one notice contains 100 words with five hits and one report contains 10,000 words with twenty hits. An equal average of per-document percentages gives a different result from the fraction of all observed tokens that match.

The mistake hiding inside a plausible answer

The misleading step says ‘each document counts equally’ without specifying whether the estimate concerns the typical document or the typical word token. Neither unit is inherently superior; the question chooses.

Make a better decision

Record each document’s token denominator and target count. For a token-rate estimand with source-level design weights, sum weighted hits and weighted tokens; for a document-level estimand, define and report a document-average measure.

Try this: When could equal document weighting be legitimate?

Explanation: When the declared target is the average document’s rate rather than the share of target words among all eligible tokens.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 12 OF 23

12. Understand calibration and benchmark sources

Post-stratification or calibration uses known or credible population totals to adjust representation. A benchmark must match the target domain, time, categories and measurement unit. An outdated or irrelevant reference can make an estimate worse.

See the method in context

A school may have a reliable archive listing the total eligible token volume of notices and reports for a defined year. That could support a token-share benchmark for a corresponding sample. A different school’s archive is not automatically a valid substitute.

The mistake hiding inside a plausible answer

The error downloads an arbitrary 80/20 mixture from a convenient infographic and treats it as a factual description of the target population. Weighting is no better than its input assumptions.

Make a better decision

Document who produced the benchmark, when, from which eligible texts and under which tokenisation rule. If the distribution is a hypothetical teaching assumption, label it hypothetical rather than pretending it is measured.

Try this: Can a benchmark from another institution establish this school’s text composition?

Explanation: Not without evidence that the populations align. A convenient external percentage is not automatically applicable.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 13 OF 23

13. Work through the complete two-genre example

The constructed dataset provides two equal token samples: 10,000 tokens from notices containing 100 matches of submit, and 10,000 report tokens containing twenty. The target-domain token shares are stipulated as 80% notices and 20% reports solely to demonstrate the arithmetic.

See the method in context

Within notices, the rate is ten per thousand tokens. Within reports, it is two per thousand. Because the sample has equal token shares, its unweighted pooled rate is 120/20,000×1,000=6 per thousand.

The mistake hiding inside a plausible answer

A false conclusion calls 8.4 ‘the true frequency in the school’s writing’. The model has no real school data, and the 80/20 shares are assumed rather than observed.

Make a better decision

Compute target weights of 1.6 and 0.4; form weighted components, and verify the equivalent mixture average 0.8×10+0.2×2=8.4. Preserve all original counts for inspection.

Try this: What rate corresponds to the declared hypothetical 80/20 domain mixture?

Explanation: 8.4 occurrences per thousand eligible tokens, equivalently 8,400 per million. It is an illustrative mixture estimate, not a measured institutional fact.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 14 OF 23

14. Explore what happens when target shares change

A weighted estimate depends on the target composition. This is a feature, not a flaw: different populations can genuinely have different rates when their component genres use words differently. But estimates should not be tuned to make an outcome impressive.

See the method in context

If the target were 20% notices and 80% reports, the same within-genre rates would imply 0.2×10 + 0.8×2=3.6 per thousand, less than half the 80/20 estimate. The sample itself would be unchanged.

The mistake hiding inside a plausible answer

The mistaken story says one of these targets must prove that a word changed in real usage. The change here occurs entirely in the assumed population mixture, not in observed within-genre language.

Make a better decision

Report the target proportions with the final rate. If the benchmark is uncertain, consider a transparent range of plausible mixtures and identify which conclusion remains stable.

Try this: What changes when notice share moves from 80% to 20% in this toy model?

Explanation: The estimated overall mixture rate changes from 8.4 to 3.6 per thousand because the high-rate genre receives less weight.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 15 OF 23

15. Confront missing strata that weights cannot create

If no texts from a relevant genre were collected, multiplying the observed genres cannot reveal the missing genre’s true vocabulary rate. Weights redistribute contributions among observed groups; they do not create unseen language evidence.

See the method in context

Suppose the target domain contains informal student messages, but the corpus consists only of official notices and reports. Even perfect knowledge of the missing messages’ target share cannot reveal their word rate from the other two sources.

The mistake hiding inside a plausible answer

The shortcut assigns a large weight to notices and declares that informal language is now represented. It is not; the missing type is still missing from the actual evidence.

Make a better decision

Acquire permitted texts from the omitted genre or narrow the target domain. When collection is impossible, report that weighted estimates apply only to the observed, covered domain or depend on explicit unverifiable assumptions.

Try this: Can a weight of ten substitute for an absent genre?

Explanation: No. Multiplying existing observations cannot supply language patterns that were never observed.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 16 OF 23

16. Check whether strata are internally representative

A genre label alone does not ensure that the selected texts represent that genre. If a notices subset comes mostly from one repetitive template, a genre weight may faithfully amplify the wrong pattern.

See the method in context

A sample of official notices might contain only examination reminders, while the target notice population includes transport, events, safety and routine administration. A correct notice share cannot repair the missing topic variation.

The mistake hiding inside a plausible answer

The misleading conclusion says the rate is representative because the 80/20 proportions match, ignoring selection bias within the notice stratum.

Make a better decision

Inspect within-stratum diversity of source, date, author, topic and document length. Use better selection and coverage, not only adjusted proportions. Weighting addresses composition, not all forms of bias.

Try this: Is a matched genre share proof that the texts within each genre were well sampled?

Explanation: No. The distribution of words within each stratum also depends on which specific texts were selected.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 17 OF 23

17. Understand variance and extreme weights

Weights can help align an estimate with a target, but very unequal weights may make the result sensitive to a small number of high-weight observations. Uncertainty calculations must respect the sampling design and clustering of words within documents.

See the method in context

If one rare genre contains two short texts and receives a large target weight, an unusual phrase in one of them can dominate the overall estimate. Treating every word token as an independent random draw would understate document-level dependence.

The mistake hiding inside a plausible answer

A polished report reports a highly precise weighted rate without revealing the number of source documents or influence of extreme weights. Precision can be illusory when effective evidence is sparse.

Make a better decision

Show weight ranges, document counts and sensitivity checks. Use design-appropriate uncertainty methods when making statistical claims; otherwise frame the result as a descriptive adjusted estimate with limits.

Try this: Can a huge weight compensate for only one observed text while preserving confidence?

Explanation: It can change a point estimate, but it cannot create independent evidence. It may increase sensitivity and uncertainty.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 18 OF 23

18. Separate descriptive weighting from causal claims

Genre weighting can produce a descriptive mixture estimate. It does not establish that genre caused differences in word choice or that a teaching intervention changed language. Causal conclusions require a different design.

See the method in context

The word submit may be frequent in notices because notices often request action. The weighted rate reflects that mixture. It cannot show that exposure to notices made students better writers or caused them to prefer the word.

The mistake hiding inside a plausible answer

The shortcut uses an adjusted chart to claim a programme improved vocabulary. Weighting corrects a specified composition imbalance; it does not identify a causal effect of instruction.

Make a better decision

Describe the estimated language pattern and avoid turning it into a causal statement unless additional evidence supports such an inference.

Try this: Can a weighted frequency prove that a vocabulary lesson was effective?

Explanation: No. A learning-effect evaluation needs learner outcomes and an appropriate comparison or design, not just text-corpus mixture adjustment.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 19 OF 23

19. Choose whether weighting is needed at all

Weights should address a concrete mismatch between sample and the target estimand. If the purpose is to compare genre-specific word use, the two separate rates may be the best result. Forcing them into one weighted average can hide the very distinction the class hopes to learn.

See the method in context

A teacher asking whether submit appears more often in official notices than in student project reports can compare ten and two per thousand directly. A single target-mixture rate of 8.4 tells a different story about aggregated token exposure.

The mistake hiding inside a plausible answer

The mistake applies weights because doing so sounds more advanced, regardless of whether the research question calls for any pooled population estimate.

Make a better decision

Match the output to the task: per-genre comparisons, document averages, an overall token estimate or exploratory context analysis. Use weights only when they make the target clearer and their assumptions can be defended.

Try this: What result is most useful for comparing notices and reports as genres?

Explanation: The two separately defined, comparable within-genre rates and example contexts, not necessarily one pooled weighted number.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 20 OF 23

20. Teach weights with a physical mixture analogy

A simple teaching approach uses two coloured containers to represent genres: eighty blue counters and twenty orange counters for a hypothetical target. The selected sample instead has fifty of each. Give each colour an illustrative word rate and ask how the overall mix changes.

See the method in context

If blue notices use the target at ten per thousand and orange reports at two per thousand, the half-and-half mixture rate is six. An eighty-to-twenty mixture rate is 8.4. Students can verify the weighted average without needing survey software.

The mistake hiding inside a plausible answer

A confusing lesson presents inverse-probability notation before learners know which population and unit are being estimated. That can produce mechanical formula use without interpretative understanding.

Make a better decision

Begin with shares and per-genre rates; then introduce the adjustment factor. Finish by changing the target shares and asking what does not change—the observed source records and within-genre toy rates.

Try this: What does the mixture analogy reveal?

Explanation: That an overall rate depends on component rates and their relative contributions; changing the composition can change the average without changing the component texts.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 21 OF 23

21. Audit a spreadsheet carrying sample weights

A good worksheet separates raw records, source token counts, genre membership, target proportions, sample proportions, adjustment factors and final weighted components. Each formula should be reproducible and checked on one row.

See the method in context

For the workbook, the notice weight is 1.6 and the report weight 0.4. Multiply notice hits and tokens by 1.6 and reports by 0.4. Then divide weighted hits by weighted tokens and multiply by the reporting base.

The mistake hiding inside a plausible answer

A spreadsheet error may hard-code the 80/20 target in one chart and the 50/50 sample in another, or copy a denominator from the wrong subset. The final percentage can still look plausible.

Make a better decision

Add a validation column checking that target shares sum to one, that no required stratum is absent, and that the equivalent weighted-average formula matches the weighted-count ratio for this constructed example.

Try this: What simple cross-check should the two-genre workbook pass?

Explanation: The weighted component ratio and 0.8×10 + 0.2×2 should both produce 8.4 per thousand.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 22 OF 23

22. Read the assumptions before trusting the point estimate

No amount of arithmetic establishes that an 80/20 benchmark was measured correctly, that documents within each genre were selected fairly, or that the target word has comparable senses across contexts. Those are substantive evidence questions.

See the method in context

A source can have an exact word count and still mix a noun meaning in one genre with a verb meaning in another. Weighting the form total does not solve sense ambiguity. Metadata, concordance lines and careful questions remain necessary.

The mistake hiding inside a plausible answer

The shortcut announces a highly precise corrected ‘national vocabulary rate’ after applying a model from a school-only convenience sample. The headline outruns both coverage and selection.

Make a better decision

State the assumptions: target domain, benchmark, unit, source coverage, query, within-stratum comparability, observed raw data and uncertainty. If one assumption is weak, narrow the claim or run a transparent sensitivity analysis.

Try this: Can accurate multiplication by 0.8 and 0.2 make an unsupported national claim valid?

Explanation: No. Arithmetic fidelity does not repair a mismatched source domain or uncertain benchmark.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Next

CHAPTER 23 OF 23

23. Finish with a seven-part weighting audit

Before publishing an adjusted vocabulary rate, confirm the target population, estimand and unit, observed strata, source selection, benchmark shares, weighted calculation and uncertainty. Then ask whether a simpler unweighted comparison is more useful for the reader.

See the method in context

A safe conclusion reads: ‘In this constructed two-genre exercise, the observed 50/50 sample gives six target matches per thousand tokens; a hypothetical 80/20 token-share mixture gives 8.4. The example does not measure real school language.’

The mistake hiding inside a plausible answer

The overclaim says ‘most Singapore school writing contains submit at 8.4 per thousand’. The dataset is fictional; no authentic population proportions or corpus frequencies were collected. Such language cannot be justified by the calculation.

Make a better decision

Publish the source table and method in a way another reader can reconstruct. Prefer a narrow reproducible statement over an inflated claim, and use weighting only when the intended population mixture can be explained.

Try this: What makes a weighted corpus estimate credible?

Explanation: An appropriate target, correctly matched units, defensible benchmarks and selection, transparent arithmetic and honest limits.

A reader can usually strengthen the result by asking three questions: Which source material was eligible? Which item was actually counted or classified? What is the smallest conclusion justified by that evidence? Write those answers explicitly before claiming that a pattern applies to every learner, writer, genre or period. When the evidence is too narrow, a more careful explanation is a better result than a dramatic statistic.

Contents · Previous · Worked table

Worked data: all decisions visible

In this deliberately fictional two-genre corpus, the sample has 10,000 eligible word tokens from official notices and 10,000 from project reports. The sample is therefore 50% notice tokens and 50% report tokens. Its target form is submit. For teaching arithmetic, we stipulate a hypothetical domain that has 80% notice tokens and 20% report tokens. This target distribution is assumed, not measured.

Genre	Observed tokens	Observed hits	Rate per 1,000	Sample share	Hypothetical target share	Factor
Official notices	10,000	100	10	50%	80%	1.6
Project reports	10,000	20	2	50%	20%	0.4
Observed pooled sample	20,000	120	6	100%	Not the target mix	Unweighted
Weighted pseudo-total	20,000	168	8.4	Adjusted	80% / 20%	Not observed hits

Within-group rates are 100/10,000×1,000=10 target hits per thousand notice tokens and 20/10,000×1,000=2 per thousand report tokens. The sample pooled rate is 120/20,000×1,000=6 per thousand. This is what the unweighted 50/50 sample actually describes.

The target mixture is assumed to be 80% notice tokens and 20% report tokens. Thus the target-rate expression is 0.8×10+0.2×2=8.4 per thousand. The expression is a descriptive post-stratified mixture of within-genre rates under the assumed token shares.

To check with adjustment factors, notices receive 0.8/0.5=1.6 and reports 0.2/0.5=0.4. Weighted notice component: 100×1.6=160 pseudo hits and 10,000×1.6=16,000 pseudo tokens. Weighted report component: 20×0.4=8 pseudo hits and 10,000×0.4=4,000 pseudo tokens.

The weighted sum is 168 pseudo hits over 20,000 pseudo tokens, giving 8.4 per thousand, the same as the mixture average. The originally observed corpus still has 120 actual hits; the weighted 168 is an estimating device, not a field count.

If instead the hypothetical target were 20% notices and 80% reports, the mixture rate would be 0.2×10+0.8×2=3.6 per thousand. This changed estimate would not indicate a real-world language change; it would come solely from choosing a different target composition.

Document proportions, writer proportions and token proportions are not interchangeable. Applying 80/20 weights to tokens is appropriate in this teaching demonstration only because the 80/20 benchmark was explicitly defined as target eligible-token shares.

Twelve transfer cases: the limits of an attractive number

Each scenario is an original classroom-style illustration. Read the case, identify what the analyst can actually observe, and compare the suggested repair with the evidence needed to carry it out.

1. Equal quotas

Situation: A student collects the same number of tokens from two genres.

Hidden danger: The target domain might contain very unequal genre shares.

Reasoned response: Present genre comparisons or justify mixture weights; do not call 50/50 representative by default.

Next question: What are the target token shares?

2. Short versus long documents

Situation: Eighty percent of archived files are notices, but most words are in reports.

Hidden danger: Document share is mistaken for token share.

Reasoned response: Define whether the estimand concerns average document or word-token exposure.

Next question: Which denominator is appropriate?

3. Missing student messages

Situation: A corpus contains no informal conversations.

Hidden danger: Weights redistribute observed genres but cannot create missing speech.

Reasoned response: Collect permitted sources or narrow the target.

Next question: What source type is unobserved?

4. Oversampled rare category

Situation: A researcher deliberately includes extra rare notices.

Hidden danger: The pooled rate describes the selected mixture, not necessarily the target.

Reasoned response: Document the allocation and use appropriate weights for the relevant target.

Next question: Did the sampling probability vary by stratum?

5. Unreliable benchmark

Situation: The 80/20 target proportion came from an unrelated school.

Hidden danger: A plausible external percentage may not fit this domain.

Reasoned response: Verify provenance or report sensitivity scenarios.

Next question: Can the target distribution be measured?

6. One repeated source

Situation: A notice genre is dominated by one copied template.

Hidden danger: A genre weight could amplify duplication bias.

Reasoned response: Deduplicate according to purpose and improve within-stratum coverage.

Next question: How many independent source texts remain?

7. Two senses

Situation: Submit appears in different constructions across genres.

Hidden danger: Form-frequency weighting does not reveal meaning.

Reasoned response: Inspect concordance lines and specify the lexical target.

Next question: Which senses are relevant to teaching?

8. Extreme weight

Situation: One small genre receives a very large factor.

Hidden danger: A few examples can dominate the estimate.

Reasoned response: Report counts, sensitivity and design-aware uncertainty.

Next question: Would one omitted text change the result?

9. Changed population

Situation: An archive’s genre mixture changes next year.

Hidden danger: Last year’s weights may no longer match.

Reasoned response: Update benchmarks and label time periods.

Next question: Which event caused the composition change?

10. Within-genre sampling bias

Situation: Only examination notices were collected.

Hidden danger: A correct target share cannot fix a distorted notice sample.

Reasoned response: Broaden coverage and examine subgenre variation.

Next question: Are the selected notices typical of the target notices?

11. Teaching comparison

Situation: A teacher simply wants to compare report and notice vocabulary.

Hidden danger: One weighted pooled rate hides the useful contrast.

Reasoned response: Use separately normalised within-genre rates.

Next question: What practical task will students perform?

12. Causal leap

Situation: A weighted word rate increases after a course.

Hidden danger: Composition change is mistaken for teaching improvement.

Reasoned response: Evaluate independent learner outcomes with an appropriate study.

Next question: What observation would show learning?

Eight practice questions and explained solutions

Answer before reading the solution. Show raw evidence and units for a calculation; for a qualitative decision, state the scope and the reason. The purpose is transferable judgement rather than memorising a list of answers.

Exercise 1: Unweighted pooled rate

Question. Notices have 100 hits in 10,000 tokens and reports 20 hits in 10,000. Find the sample rate per thousand.

Explained answer. 120/20,000×1,000=6 per thousand eligible tokens.

Why the shortcut fails. A 10-per-thousand notice rate is not the entire sample’s rate.

Exercise 2: Target mixture

Question. Using 80% notice tokens and 20% report tokens, calculate the weighted rate.

Explained answer. 0.8×10+0.2×2=8.4 per thousand.

Why the shortcut fails. Simple 50/50 averaging answers the sample mixture, not the target mix.

Exercise 3: Weight factors

Question. Sample is half notice and half report, target is 80/20. Find genre factors.

Explained answer. Notices 0.8/0.5=1.6; reports 0.2/0.5=0.4.

Why the shortcut fails. The factors are not newly observed text occurrences.

Exercise 4: Weighted ratio

Question. Use factors 1.6 and 0.4 on hits and token totals. Compute the result.

Explained answer. Weighted hits 160+8=168, weighted token totals 16,000+4,000=20,000; rate 8.4 per thousand.

Why the shortcut fails. Weights must affect numerator and denominator consistently.

Exercise 5: Change the target

Question. If the target is 20% notices and 80% reports, compute its rate.

Explained answer. 0.2×10+0.8×2=3.6 per thousand.

Why the shortcut fails. The difference does not prove changed word choice: the assumed target mixture changed.

Exercise 6: Choose the unit

Question. A school states 80% of files are notices. Can we use 0.8 as notice token share?

Explained answer. Not without data showing the token proportions or a justified estimator for the intended unit.

Why the shortcut fails. File counts and word counts are not interchangeable.

Exercise 7: Missing genre

Question. The target includes student chats, but none were sampled. Can upweighting reports replace their rate?

Explained answer. No. The missing stratum lacks its own observed language evidence. Collect authorised examples or narrow scope.

Why the shortcut fails. A large weight cannot manufacture the unseen lexical distribution.

Exercise 8: Inverse probability

Question. An eligible document has known selection probability 0.25. What is a basic design weight?

Explained answer. 1/0.25=4 under the simple inverse-probability rule; integrating it into a word-rate estimate requires the correct document and token units.

Why the shortcut fails. The weight is not automatically a multiplier for every displayed per-million score.

Frequently asked questions: Corpus sampling weight

What is a corpus sampling weight?

An adjustment assigning contributions to sampled units or strata for a specified target estimate, under an explicit sampling or benchmark model.

Why would a language corpus need weights?

Its selected text composition may not match the target domain’s composition, or source units may have unequal selection probabilities.

Is a balanced corpus always representative?

No. Balance is purpose-specific and may mean equal categories rather than target-population proportions.

Can weights correct an equal-quota design?

They can help re-express a mixture when appropriate population shares and within-stratum assumptions are justified.

What is a stratum?

A predeclared category in the target domain, such as genre, medium, period or source type.

What is the difference between sample share and target share?

Sample share describes collected material; target share describes the distribution whose rate is being estimated.

Are weighted hits actually observed?

No. The source count stays unchanged; weighted totals are transformed components of an estimator.

Do I weight numerator and denominator?

For a weighted word-token rate, yes: compute a consistent weighted ratio or equivalent justified mixture average.

Can I use document shares to weight token rates?

Not automatically. Clarify the estimand and account for document lengths and sampling units.

What happens when a genre is missing?

Weights cannot recover its unobserved word distribution. The target or data collection must change.

Do weights remove selection bias?

No. Bias within strata, coverage gaps, duplicates and wrong labels may remain.

How do extreme weights affect conclusions?

They can make estimates sensitive to a small amount of observed evidence and increase uncertainty.

What if population genre shares are unknown?

Do not invent weights; report stratum-specific results or transparent scenarios until a defensible benchmark exists.

Can weighting prove learning improvement?

No. It is an estimation adjustment, not a causal study of learner outcomes.

When is an unweighted comparison better?

When the goal is to understand how genres differ rather than to estimate a pooled target-domain rate.

Compact vocabulary glossary

• Stratum: A defined category in a target domain used for sampling and analysis.
• Sample share: The observed proportion of an eligible unit type in collected material.
• Target share: The benchmark proportion relevant to the intended population or estimand.
• Weight: A numerical contribution factor assigned under a specified estimation design.
• Post-stratification: Adjustment of estimates using a known target distribution across categories.
• Inclusion probability: A unit’s probability of selection under a probability-sampling scheme.
• Base weight: Often the reciprocal of a sampling inclusion probability.
• Token rate: Word occurrences divided by eligible token counts.
• Pseudo count: A weighted component used for calculation, not a literally observed number of tokens.
• Representativeness: The extent to which selected evidence supports the intended domain inference.
• Coverage bias: A distortion caused by missing or misrepresented sections of the target domain.
• Sensitivity analysis: Inspection of how conclusions change when assumptions or parameters vary.

A classroom routine for learning the method

Begin with one surprising example and have students make a prediction before revealing its count or classification. Ask them to state the rule that would settle the question. A learner who can make and defend a prediction is doing more than remembering a definition: the learner is reasoning from language evidence.

Next, introduce a counterexample. Change a short phrase’s word order, add a negation, move one noun beyond the selected position, or alter the sampling mixture. Hold everything else constant. Let students identify which conclusion becomes invalid, and why. This deliberate small change makes the scope of a rule visible.

Then ask students to recreate a compact evidence table without copying the supplied answers. They should name each column, check one calculation or label independently, and write a caption saying precisely what the table represents. Invite a peer to test whether the reported number follows from the chosen method.

Finally, apply the idea to a new reading or writing task. A good transfer exercise ends with two sentences: one specific observation supported by the materials, and one limitation that would matter when applying the result elsewhere. Revisit the exercise after a delay to test retrieval, and correct a mistake without hiding the original reasoning.

Methodological reading and verification

• Cambridge: domain considerations, stratification and proportionality
• Cambridge: corpus representativeness, selection and coverage bias
• US Census Bureau: SIPP survey weighting and post-stratification
• US Census Bureau: CPS weighting and inverse inclusion probabilities
• Pew Research Center: how different weighting methods work
• Lancaster Corpus Linguistics: balanced corpora and sampling frames

The sources support the methods described. Illustrative counts and example messages are original to this article and should not be cited as empirical study results.

Continue through connected vocabulary guides

• Vocabulary | What Is a Collocation Association Score?
• Vocabulary | What Is a Word Sketch?
• Vocabulary | What Is a Concordance Line?
• Vocabulary | What Is a Subcorpus?
• Vocabulary | What Is a Corpus Sampling Frame?
• Vocabulary | What Is Normalised Word Frequency?
• Vocabulary Learning Hub
• Vocabulary articles index

Return to contents · Vocabulary Learning Hub