A text can use many different words yet rely mostly on common vocabulary. Another can repeat a smaller set that includes specialised or low-frequency items. Lexical diversity and lexical sophistication therefore answer different questions.
The central invented comparison gives Text A 100 tokens, 50 types and a 10% outside-band share. Text B has 200 tokens, 80 types and a 6% outside-band share. More types in B do not produce a higher simple TTR or sophistication proportion.
These examples teach measurement choices. They do not diagnose a learner, grade real work, define good writing or replace qualified corpus and assessment analysis.
This specialist guide belongs to the Vocabulary root. It also connects to The Gold Standard of Education | Learning That Transfers into Life and the How X Works hub. The worked cases are original teaching examples; their numbers are stipulated so that the reasoning can be checked.
eduKateSG · Vocabulary
Find your next learning step
Keep internal variety, external word properties, text length and reference corpus in separate columns.
ROUTE 1 · CHAPTERS 1–4
Define the constructs
Name the quantities and the reference conditions before calculating.
ROUTE 2 · CHAPTERS 5–8
Work the text examples
Carry the central worked case through each transformation.
ROUTE 3 · CHAPTERS 9–12
Audit the pipeline
Inspect the evidence, assumptions and competing mechanisms.
ROUTE 5 · CHAPTERS 17–20
Report and continue
Repair shortcuts, report limits and continue through verified routes.
Chapter contents
Define the constructs · Chapters 1–4
1. Name diversity and sophistication
2. Choose the lexical unit first
Work the text examples · Chapters 5–8
5. Work the central two-text comparison
6. Calculate a frequency-band profile
Audit the pipeline · Chapters 9–12
9. Audit tokenisation and lemmatisation
10. Audit corpus domain and date
Transfer the comparison · Chapters 13–16
13. Transfer to learner writing
14. Transfer to speaking transcripts
Report and continue · Chapters 17–20
17. Repair “more types means more sophisticated”
18. Repair “rare always means better”
Lexical diversity asks how varied the word choices are within a sample. Lexical sophistication asks about properties of those choices relative to external evidence, often frequency, range, association strength or psycholinguistic information. A text can rise on one and fall on the other.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 1: A 100-token text repeats fifty everyday types; a second uses forty types including several low-frequency technical items. The first may be more diverse while the second is more sophisticated under the declared corpus.
Changed-context check: Hold the types fixed and change their reference frequencies; only sophistication should move.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Limit: No one metric captures vocabulary knowledge, writing quality or learning.
Tokens, written types, lemmas and word families create different inventories. Sophistication scores must declare which unit is looked up in the reference resource, while diversity must declare which unit is counted inside the sample.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 2: Count “analyse, analysed, analysing” as three written types but one lemma under a stated lemmatiser; show how both diversity and lookup results change.
Changed-context check: Hold word choices fixed and switch from forms to lemmas; document both score changes.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
A reproducible analysis begins with two inventories rather than one score. The sample inventory lists the eligible tokens and the unit used for within-text counting. The reference inventory supplies frequency, range or another external property for those same units. A diversity calculation can be completed from the first inventory alone; a sophistication calculation cannot. When a form has no reference match, keep it in an explicit unmatched category until spelling, names and segmentation have been checked. This two-inventory design prevents a missing corpus entry from being celebrated as rare vocabulary and lets another analyst trace every word from the learner text to the reported summary.
Limit: Lemmas and families depend on language and analysis rules.
A word is not intrinsically “low frequency” without a reference corpus, counting unit and frequency scale. A school corpus, spoken corpus and news corpus can assign different ranks to the same item.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 3: Look up “cell” in a general corpus and a biology corpus. Preserve both values instead of calling one the true frequency.
Changed-context check: Switch corpora and predict which domain terms may change rank.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Limit: Corpus counts are samples, not universal facts about a word.
Diversity indices and sophistication summaries can both react to sample length. Keep token count, truncation rule and any moving window visible rather than treating a score as length-free.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 4: Truncate two responses to the first 100 eligible tokens, then compare the result with the full-text scores and report the sensitivity.
Changed-context check: Double a text by repeating it; simple TTR changes while token-proportion sophistication can remain.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Limit: Length controls reduce particular biases but do not remove task differences.
Text A has 100 tokens, 50 types and ten tokens outside the chosen high-frequency bands. Text B has 200 tokens, 80 types and twelve such tokens. A has TTR 0.50 and a 10% outside-band share; B has TTR 0.40 and a 6% share.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 5: For A, 50/100=0.50 and 10/100=10%. For B, 80/200=0.40 and 12/200=6%. More types in B do not produce a higher TTR or outside-band proportion.
Changed-context check: Swap the twelve outside-band tokens for high-frequency synonyms; B’s diversity can stay while sophistication changes.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
The central comparison becomes clearer in a small matrix. Text A has fewer total types but a higher simple TTR and a higher outside-band proportion. Text B has more total types yet repeats them across twice as many tokens and places a smaller share outside the chosen bands. None of those statements says which text is better. They describe different sample properties under one tokenisation and corpus. Add confidence by repeating the calculation with matched one-hundred-token windows. If Text B’s result changes sharply across windows, report that instability. If the direction persists, say it is robust to that particular length check rather than universally length-free.
Limit: The teaching numbers are invented and do not score a real learner.
A lexical frequency profile groups eligible tokens into declared bands. The denominator must state whether proper nouns, numbers, function words and unlisted items are included, excluded or reported separately.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 6: If 150 of 200 eligible tokens are band1, 30 band2, 12 outside and 8 unlisted, report 75%,15%,6%,4% rather than hiding unlisted items.
Changed-context check: Move unlisted tokens into a separate denominator and show the reported percentages change.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Frequency bands should be treated as an ordered reporting system rather than a line dividing ordinary and impressive words. Preserve counts in each band, the unmatched category and the denominator. Then inspect the actual sentences containing items that drive the result. A technical noun may be unavoidable for the prompt, while a rare adjective may be unnecessary or inaccurate. If the analysis is repeated with a second reasonable band list, report which conclusions remain and which depend on the boundary. This sensitivity check turns a list choice into visible evidence instead of allowing it to determine the learner judgement silently.
Limit: Band boundaries and lists are analytical choices.
CHAPTER 7 OF 20 · Work the text examples
7. Compare mean frequency with proportions
Average log frequency compresses a distribution into one number; a band profile preserves proportions in several regions. Two texts can share a mean while one mixes very frequent and very rare words.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 7: Frequencies 1000 and 10 average505, as do 600 and410; the same mean masks very different distributions.
Changed-context check: Use log frequencies and compare the ordering with raw means.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Limit: Means can conceal mixture and skew.
Frequency counts total occurrences; range or dispersion records how broadly an item appears across documents or contexts. A word can be frequent inside one narrow domain yet have restricted range.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 8: A technical term occurs 500 times in ten documents; a common verb occurs 500 times across 400 documents. Equal totals do not mean equal range.
Changed-context check: Hold total frequency fixed and redistribute occurrences across documents; range changes.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
A mean frequency can hide the very distribution a teacher needs to see. Suppose one response uses almost entirely mid-frequency vocabulary, while another mixes very common function words with a few extremely rare technical terms. Their average frequency may be similar, but the learning needs differ. Report at least a compact profile or quantiles beside the mean, and keep content and function words separate when the question requires it. Log frequency can reduce domination by a few enormous counts, but it does not choose the reference corpus or judge appropriateness. The transformation changes the summary scale; it does not convert corpus frequency into a direct measure of writing quality.
Limit: Range needs a well-designed document structure.
Tokenisation changes what reaches both metrics. Hyphens, apostrophes, multiword units, spelling variants and punctuation need stable rules; lemmatisation can merge forms before lookup.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 9: Treat “decision-making” first as one token and then as two. The changed denominator and lookup path must be reported.
Changed-context check: Apply one tokenizer to both texts and verify identical strings receive identical treatment.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Limit: Automatic preprocessing needs manual checks for consequential cases.
Reference corpora represent particular periods, genres, populations and sampling designs. A word labelled sophisticated against old news may be ordinary in current classroom science.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 10: A digital term absent from a 1990s corpus may be common in 2026. Update the corpus comparison before inferring learner sophistication.
Changed-context check: Use a contemporary corpus and record which classifications are robust.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Limit: Older and newer corpora answer different historical questions.
Independent writing, source-based writing and a prompted explanation invite different vocabulary. Sophistication can reflect topic borrowing or necessary terminology rather than general lexical control.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 11: A source-based response repeats specialised source words. Mark source overlap before attributing every item to independent productive choice.
Changed-context check: Change from independent to source-based writing; expect source overlap to matter.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Preprocessing uncertainty should be measured on the items that can change the conclusion. Draw a small audit sample containing hyphenations, apostrophes, compounds, proper names, spelling errors and ambiguous lemmas. Have a second reader apply the written rules, then compare outcomes. If disagreements cluster around a few high-impact tokens, show the score with and without them. This is more informative than quoting many decimal places from an automatic pipeline. The objective is not perfect linguistic annotation for every purpose. It is to know whether the diversity or sophistication claim depends on fragile decisions that were hidden before the calculation began.
Limit: Topic effects and source use can be legitimate, not contamination.
A misspelling can appear absent from the frequency list and be misclassified as rare. Names, codes and copied strings create the same problem unless they are audited before interpretation.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 12: The misspelling “enviroment” receives no corpus match. Classify it as an error or unmatched form, not a rare sophisticated word.
Changed-context check: Correct spelling and rerun lookup; unmatched counts should fall.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Limit: Error handling must not reward misspellings.
For learner writing, keep diversity, sophistication, accuracy and appropriateness in separate columns. A less frequent word used inaccurately is not automatic evidence of stronger writing.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 13: Compare “use” with accurately used “employ” in context; the corpus rank is only one part of the writing judgement.
Changed-context check: Judge appropriateness separately and see whether the corpus result alone predicts it.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Limit: Appropriateness requires contextual human judgement.
Speaking transcripts add filled pauses, repairs, contractions and transcription choices. A sophistication pipeline designed for polished writing cannot be transferred silently.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 14: Remove filled pauses under a declared rule, then show that the denominator and type inventory changed.
Changed-context check: Change transcription conventions and report score sensitivity.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
For speaking or source-based work, the task design becomes part of the vocabulary evidence. A speaker may repeat words while planning, repair a phrase or use discourse markers that are appropriate in speech. A writer answering from a source may reuse necessary technical terms. Removing these items silently can inflate or depress both metrics. Keep a declared transcript or source-overlap layer and publish sensitivity results when the choice matters. Then interpret the remaining score at the sample level. The same learner can produce different profiles across an interview, a narrative and a science explanation because the communicative jobs and available language differ.
Limit: Speech and writing should not share defaults without validation.
Academic-word analysis depends on the selected list or corpus and on the discipline. Academic status is not identical to low general frequency, and neither is identical to diversity.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 15: A word can sit on an academic list and still be frequent. Report list membership and frequency as separate fields.
Changed-context check: Change the academic list while retaining raw words; list coverage changes.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Limit: Academic vocabulary varies across disciplines.
CHAPTER 16 OF 20 · Transfer the comparison
16. Transfer across unequal text lengths
Unequal samples need declared control: common truncation, moving windows, matched tasks or a model that includes length. No single repair makes every index perfectly comparable.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 16: Calculate metrics on matched 100-token windows and full samples; keep both results to show robustness.
Changed-context check: Use matched windows of two sizes and report whether the conclusion holds.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Unequal texts can also be compared through repeated windows rather than one truncation point. Draw several fixed-length windows from each text, calculate the selected metrics for every window and show the distribution. This preserves more of the longer response and reveals internal variation that one first-hundred-token sample could miss. The windows are not independent evidence about the learner, so avoid exaggerated significance claims. Their value is diagnostic: they show whether the direction is stable across parts of the texts, whether one unusual passage controls the result and whether a full-text difference is mainly a consequence of length.
Limit: Very short texts may remain unstable after adjustment.
CHAPTER 17 OF 20 · Report and continue
17. Repair “more types means more sophisticated”
Adding ten new high-frequency types can raise diversity while leaving sophistication unchanged or lower. Count variety and external word properties separately.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 17: Add ten new function-word types to an artificial text. Diversity rises, but a content-word sophistication index may not.
Changed-context check: Add new common types and confirm that variety can rise without rarity.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Teacher feedback improves when metric results are translated into observable choices. Instead of telling a learner to “use more sophisticated words”, identify a repeated general verb, a missing precise term or an inaccurate low-frequency substitution. Offer alternatives that fit the sentence, then check whether the revision improves meaning before looking at the corpus score. This sequence keeps communication first. A learner should not replace a clear common word merely to cross a frequency threshold. The metric can reveal a pattern and suggest places to inspect; the sentence, audience and task decide whether a change is useful.
Limit: Diversity can be valuable even when words are common.
An obscure word is not automatically precise, appropriate or learned. Sophistication metrics describe selected corpus properties, not a moral ranking of vocabulary.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 18: Replace a precise common word with an obscure but wrong word; the frequency score can improve while communication worsens.
Changed-context check: Replace rarity with contextual fit as the decision question; conclusions may reverse.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Limit: Rare-word pursuit can reduce clarity.
A defensible learner claim states the task, sample length, tokenisation, unit, corpus, frequency measure, exclusions, score and uncertainty. It avoids turning one text into a diagnosis of a person.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 19: Write a result that begins “In this 200-token response under corpus X…” rather than “This learner has sophisticated vocabulary”.
Changed-context check: Ask a second reader to reproduce the score from the receipt.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
Limit: A sample-level claim is not a person-level diagnosis.
CHAPTER 20 OF 20 · Report and continue
20. Continue through verified vocabulary routes
The reader can continue to type-token ratio, word-form versus lemma counting and document dispersion. Together these routes keep internal variety, lexical unit and external distribution distinct.
Keep the mechanism visible. For Vocabulary, write the measured or computed quantity, its reference condition and its time, spatial or corpus basis before comparing it with another number. A label without those fields invites a correct calculation to answer the wrong question.
Worked example 20: Use the type-token article for internal variety and the dispersion article for external distribution before combining interpretations.
Changed-context check: Follow each sibling link and identify which denominator it owns.
Build an evidence receipt before interpreting the result: record the source, sampling or rendering rule, units, denominator, transformation, intermediate values and changed-context prediction. A second reader should be able to reproduce the distinction without guessing which convention was used.
A final reporting sheet can carry four columns: internal variety, external lexical properties, accuracy and contextual fit. Under each, state the measure, evidence and uncertainty. The conclusion should begin with the text and task, such as “In this two-hundred-token explanation under corpus X…”. It should not jump to a permanent trait of the learner. Then add one changed-context result: another window size, corpus, token unit or task. If the claim survives, its scope becomes clearer. If it changes, the analysis has still succeeded because it has found the condition on which the earlier interpretation depended.
Limit: Connected articles support interpretation but do not create a single universal score.
Continue through the verified reading route
Type-token ratio and lexical diversity
Type-token ratio and lexical diversity. Use this for length-sensitive internal variety and moving-window checks.
Word forms, lemmas and families
Word forms, lemmas and families. Use this before choosing the lexical unit.
Word dispersion across unequal texts
Word dispersion across unequal texts. Use this when external frequency and document range must remain separate.
Frequently asked questions
Is lexical sophistication the same as lexical diversity?
No. Diversity describes variety inside a sample; sophistication uses external properties such as corpus frequency, range or association.
Are rare words always better?
No. Accuracy, appropriateness and clarity remain separate judgements.
Can unequal texts be compared?
They can be studied with declared controls and sensitivity checks, but no adjustment removes every task and length effect.
Which corpus should I use?
Use one that fits the language, genre, period and reader question, then report it explicitly.
Authoritative sources and further reading
Laufer and Nation: Vocabulary Size and Use. The primary study introduces a lexical frequency profile for analysing vocabulary use in learner writing.
Kyle and Crossley: Automatically Assessing Lexical Sophistication. The primary study treats sophistication as a family of indices involving frequency, range, n-grams and other properties.
Kim, Crossley and Kyle: Lexical Sophistication as a Multidimensional Phenomenon. The primary study tests multiple dimensions rather than reducing sophistication to one rare-word count.
Return to the Vocabulary subject guide, The Gold Standard of Education, or the How X Works hub.