VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

The Gold Standard of Vocabulary | Separating Type-Token Ratio from Lexical Diversity Across Unequal Text Lengths

Three learners review open books together at a classroom table, with stacks of textbooks, stationery and a whiteboard in the bright room.

Simple type-token ratio divides the number of distinct types by the number of running tokens. It is easy to calculate and easy to misread. Longer texts create more opportunities for repetition, so a lower TTR can arise even when the vocabulary repertoire has not narrowed.

The central invented Text A contains 100 tokens and 50 types, TTR 0.50. Text B repeats the same 50-type cycle across 200 tokens, so TTR falls to 0.25. With a 100-token moving window, each periodic window still contains 50 types and MATTR remains 0.50. The changed result comes from the metric and scale, not a halving of known vocabulary.

These examples teach measurement. They do not score a real learner, validate a high-stakes assessment, diagnose language ability or replace qualified corpus and assessment analysis.

This specialist guide belongs to the Vocabulary root. It also connects to The Gold Standard of Education | Learning That Transfers into Life and the How X Works hub. The worked cases are original teaching examples; their numbers are stipulated so that the reasoning can be checked.

eduKateSG · Vocabulary

Find your next learning step

Keep tokens, types, counting unit, text length and index settings visible before comparing lexical diversity.

ROUTE 1 · CHAPTERS 1–5

Define the quantities

Name the object, reference condition and decision before calculating.

ROUTE 2 · CHAPTERS 6–10

Work the central example

Carry a complete example through its arithmetic and interpretation.

ROUTE 3 · CHAPTERS 11–15

Audit the evidence

Inspect measurement choices, competing mechanisms and uncertainty.

ROUTE 4 · CHAPTERS 16–20

Transfer the reasoning

Change context while keeping the distinction intact.

ROUTE 5 · CHAPTERS 21–24

Report and continue

Repair shortcuts, state limits and follow verified routes.

Full chapter contents · Vocabulary subject guide

Chapter contents

Define the quantities · Chapters 1–5

1. Define tokens, types and lexical diversity

2. Show why text length changes TTR

3. Separate richness from fluency and length

4. Declare the unit counted

5. Use the short-answer gateway

Work the central example · Chapters 6–10

6. Work a fixed-length comparison

7. Compute moving-average TTR

8. Explain the window decision

9. Introduce MTLD carefully

10. Complete a three-text ledger

Audit the evidence · Chapters 11–15

11. Audit tokenization

12. Audit case and spelling normalization

13. Audit lemmatization and morphology

14. Audit prompt and genre

15. Audit very short texts

Transfer the reasoning · Chapters 16–20

16. Transfer to learner progress

17. Transfer to reading materials

18. Transfer to spoken language

19. Transfer across languages

20. Build a reproducible diversity receipt

Report and continue · Chapters 21–24

21. Repair “higher TTR means stronger vocabulary”

22. Repair “length-aware means length-proof”

23. State the supported decision

24. Continue through verified vocabulary routes

CHAPTER 1 OF 24 · Define the quantities

1. Define tokens, types and lexical diversity

Back to contents

Begin with the reader’s actual decision. Tokens are running word occurrences. Types are distinct units under a declared counting rule. Lexical diversity is a broader construct that a chosen index attempts to represent.

Work the teaching case in a visible order. The teaching text has 100 tokens and 50 types, so simple TTR=50/100=0.50.

Make the reasoning independently checkable. State capitalization, punctuation, contractions and lemmatization rules.

Test transfer by changing one condition while holding the others still. Count word forms first, then lemmas, and compare.

Keep the claim boundary visible. A TTR value is index output, not a direct count of a learner’s vocabulary knowledge.

Before accepting the result, predict what should move and what should stay fixed. Apply this check: State capitalization, punctuation, contractions and lemmatization rules. Then run the changed case: Count word forms first, then lemmas, and compare.

The evidence receipt for “Define tokens, types and lexical diversity” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Next chapter

CHAPTER 2 OF 24 · Define the quantities

2. Show why text length changes TTR

Back to contents

Keep the observation, transformation and conclusion on separate lines. As a text grows, repeated words accumulate and the chance that every new token is a new type usually falls.

Work the teaching case in a visible order. Extend the 100-token text to 200 tokens by repeating the same 50-type cycle. The type count remains 50 and TTR falls to 0.25.

Make the reasoning independently checkable. Keep the vocabulary set and token order rule visible.

Test transfer by changing one condition while holding the others still. Add 50 genuinely new types in the second half.

Keep the claim boundary visible. The constructed repetition is a mechanism example, not a model of natural prose.

Build a compact evidence receipt with input, unit, method, calculation, observation and limit. Use this check: Keep the vocabulary set and token order rule visible. For transfer, Add 50 genuinely new types in the second half.

The evidence receipt for “Show why text length changes TTR” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 3 OF 24 · Define the quantities

3. Separate richness from fluency and length

Back to contents

Use the short answer only after the reference condition is fixed. A longer text can contain more unique words yet have a lower TTR because its denominator grew faster.

Work the teaching case in a visible order. Text A has 50 types in 100 tokens; Text B has 70 types in 200 tokens. B uses more types but has TTR 0.35 versus 0.50.

Make the reasoning independently checkable. Report tokens and types beside every ratio.

Test transfer by changing one condition while holding the others still. Compare equal 100-token samples from both texts.

Keep the claim boundary visible. Neither raw type count nor simple TTR alone establishes overall writing quality.

Ask a second reader to reproduce the decision from the stated evidence. They should be able to apply: Report tokens and types beside every ratio. Next, Compare equal 100-token samples from both texts.

The evidence receipt for “Separate richness from fluency and length” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 4 OF 24 · Define the quantities

4. Declare the unit counted

Back to contents

Trace the mechanism in an order another reader can reproduce. Word forms, lemmas and word families produce different type counts. The unit must remain stable across texts.

Work the teaching case in a visible order. Counts of walk, walks, walked and walking yield four forms but may collapse under a chosen lemma rule.

Make the reasoning independently checkable. Repeat the ratio using both units and label them.

Test transfer by changing one condition while holding the others still. Add a homograph with two meanings.

Keep the claim boundary visible. Lemmatization does not automatically separate senses or register.

Turn the claim into a test that could fail. First, Repeat the ratio using both units and label them. Then change one consequential condition: Add a homograph with two meanings.

The evidence receipt for “Declare the unit counted” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 5 OF 24 · Define the quantities

5. Use the short-answer gateway

Back to contents

Treat a plausible number as a prompt for a check rather than as proof. For unequal text lengths, do not compare simple TTRs as though length were irrelevant. Align samples or use a length-aware measure whose assumptions fit the question.

Work the teaching case in a visible order. The central 100- and 200-token texts need token/type totals plus a fixed-window comparison.

Make the reasoning independently checkable. Write the reader job before choosing an index.

Test transfer by changing one condition while holding the others still. Change the task from comparing texts to tracking one learner over time.

Keep the claim boundary visible. No index removes the need to inspect genre, prompt and language sample.

Before accepting the result, predict what should move and what should stay fixed. Apply this check: Write the reader job before choosing an index. Then run the changed case: Change the task from comparing texts to tracking one learner over time.

The evidence receipt for “Use the short-answer gateway” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 6 OF 24 · Work the central example

6. Work a fixed-length comparison

Back to contents

Begin with the reader’s actual decision. Equal-length excerpts remove one major denominator difference while introducing sampling variability.

Work the teaching case in a visible order. Take the first 100 tokens of both teaching texts and compute their type counts under the same rule.

Make the reasoning independently checkable. Repeat with several windows rather than one convenient excerpt.

Test transfer by changing one condition while holding the others still. Shift the window by 25 tokens.

Keep the claim boundary visible. A fixed excerpt may miss vocabulary appearing elsewhere.

Build a compact evidence receipt with input, unit, method, calculation, observation and limit. Use this check: Repeat with several windows rather than one convenient excerpt. For transfer, Shift the window by 25 tokens.

The evidence receipt for “Work a fixed-length comparison” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 7 OF 24 · Work the central example

7. Compute moving-average TTR

Back to contents

Keep the observation, transformation and conclusion on separate lines. MATTR calculates TTR in successive windows of fixed length and averages them, reducing direct dependence on total text length while retaining a chosen window scale.

Work the teaching case in a visible order. In the periodic 50-type teaching sequence, every 100-token window contains 50 types, so each window TTR and MATTR are 0.50.

Make the reasoning independently checkable. Show the first, middle and last windows before averaging.

Test transfer by changing one condition while holding the others still. Change the window from 100 to 50 tokens.

Keep the claim boundary visible. MATTR values from different window sizes are not automatically comparable.

Ask a second reader to reproduce the decision from the stated evidence. They should be able to apply: Show the first, middle and last windows before averaging. Next, Change the window from 100 to 50 tokens.

The evidence receipt for “Compute moving-average TTR” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 8 OF 24 · Work the central example

8. Explain the window decision

Back to contents

Use the short answer only after the reference condition is fixed. A short window emphasizes local variety; a long window reflects broader repetition and requires sufficiently long texts.

Work the teaching case in a visible order. The same sequence can give MATTR 1.00 with a 50-token window and 0.50 with a 100-token window.

Make the reasoning independently checkable. Report the window size beside the result.

Test transfer by changing one condition while holding the others still. Use a text shorter than the proposed window.

Keep the claim boundary visible. Choosing a window after seeing the desired result is a form of analytic flexibility.

Turn the claim into a test that could fail. First, Report the window size beside the result. Then change one consequential condition: Use a text shorter than the proposed window.

The evidence receipt for “Explain the window decision” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 9 OF 24 · Work the central example

9. Introduce MTLD carefully

Back to contents

Trace the mechanism in an order another reader can reproduce. MTLD estimates the mean length of word strings that maintain TTR above a criterion, commonly using a specified threshold. It answers a different operational question from simple TTR.

Work the teaching case in a visible order. Walk through a short fictional sequence until its running TTR crosses the declared threshold, then start the next factor.

Make the reasoning independently checkable. Preserve threshold, direction, partial-factor rule and software version.

Test transfer by changing one condition while holding the others still. Reverse token order and compare.

Keep the claim boundary visible. MTLD is less length-sensitive in many studies but is not length-free in every sample.

Before accepting the result, predict what should move and what should stay fixed. Apply this check: Preserve threshold, direction, partial-factor rule and software version. Then run the changed case: Reverse token order and compare.

The evidence receipt for “Introduce MTLD carefully” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 10 OF 24 · Work the central example

10. Complete a three-text ledger

Back to contents

Treat a plausible number as a prompt for a check rather than as proof. A useful comparison table includes tokens, types, simple TTR, MATTR window, MATTR result and MTLD settings.

Work the teaching case in a visible order. Record A: 100 tokens/50 types/TTR .50. Record B: 200/50/.25. Record C: 200/100/.50.

Make the reasoning independently checkable. Check that every ratio uses the declared unit and denominator.

Test transfer by changing one condition while holding the others still. Add a fourth text from another genre.

Keep the claim boundary visible. The table compares indices, not the writers’ complete lexical competence.

Build a compact evidence receipt with input, unit, method, calculation, observation and limit. Use this check: Check that every ratio uses the declared unit and denominator. For transfer, Add a fourth text from another genre.

The evidence receipt for “Complete a three-text ledger” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 11 OF 24 · Audit the evidence

11. Audit tokenization

Back to contents

Begin with the reader’s actual decision. Hyphens, apostrophes, numbers, URLs and punctuation can change token and type counts before any diversity formula is applied.

Work the teaching case in a visible order. Count “state-of-the-art” as one token and then under a split rule; record both outcomes.

Make the reasoning independently checkable. Save the tokenizer rules and exceptions.

Test transfer by changing one condition while holding the others still. Add emoji and code fragments.

Keep the claim boundary visible. Tool defaults may differ across languages and software versions.

Ask a second reader to reproduce the decision from the stated evidence. They should be able to apply: Save the tokenizer rules and exceptions. Next, Add emoji and code fragments.

The evidence receipt for “Audit tokenization” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 12 OF 24 · Audit the evidence

12. Audit case and spelling normalization

Back to contents

Keep the observation, transformation and conclusion on separate lines. Case-folding and spelling normalization can merge forms that were visibly different in the text.

Work the teaching case in a visible order. Compare Singapore and singapore under case-sensitive and insensitive counts.

Make the reasoning independently checkable. Keep raw text and normalized tokens.

Test transfer by changing one condition while holding the others still. Normalize British and American spelling variants.

Keep the claim boundary visible. Normalization supports a defined question; it is not inherently more truthful.

Turn the claim into a test that could fail. First, Keep raw text and normalized tokens. Then change one consequential condition: Normalize British and American spelling variants.

The evidence receipt for “Audit case and spelling normalization” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 13 OF 24 · Audit the evidence

13. Audit lemmatization and morphology

Back to contents

Use the short answer only after the reference condition is fixed. Automated lemmatizers can make errors, especially with ambiguous forms, names and learner language.

Work the teaching case in a visible order. Inspect a sample of mapped lemmas before trusting the total.

Make the reasoning independently checkable. Report manual correction or confidence procedure.

Test transfer by changing one condition while holding the others still. Use an inflected-language sample.

Keep the claim boundary visible. An English-tuned pipeline should not be transferred silently to another language.

Before accepting the result, predict what should move and what should stay fixed. Apply this check: Report manual correction or confidence procedure. Then run the changed case: Use an inflected-language sample.

The evidence receipt for “Audit lemmatization and morphology” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 14 OF 24 · Audit the evidence

14. Audit prompt and genre

Back to contents

Trace the mechanism in an order another reader can reproduce. A constrained prompt can require repeated terms, while narrative, explanation and dialogue invite different vocabulary patterns.

Work the teaching case in a visible order. Compare two equal-length texts written to different prompts.

Make the reasoning independently checkable. Keep prompt, time limit, reference materials and audience.

Test transfer by changing one condition while holding the others still. Ask the same writer to produce a lab report and story.

Keep the claim boundary visible. Index differences can reflect task design rather than vocabulary growth.

Build a compact evidence receipt with input, unit, method, calculation, observation and limit. Use this check: Keep prompt, time limit, reference materials and audience. For transfer, Ask the same writer to produce a lab report and story.

The evidence receipt for “Audit prompt and genre” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 15 OF 24 · Audit the evidence

15. Audit very short texts

Back to contents

Treat a plausible number as a prompt for a check rather than as proof. Short samples provide few windows and unstable type counts. A length-aware index still needs enough observations for its algorithm.

Work the teaching case in a visible order. A 40-token response cannot support a 100-token MATTR window without an explicit fallback.

Make the reasoning independently checkable. State minimum sample rules before scoring.

Test transfer by changing one condition while holding the others still. Combine several comparable responses from the same learner.

Keep the claim boundary visible. Pooling changes the object from one response to a collection.

Ask a second reader to reproduce the decision from the stated evidence. They should be able to apply: State minimum sample rules before scoring. Next, Combine several comparable responses from the same learner.

The evidence receipt for “Audit very short texts” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 16 OF 24 · Transfer the reasoning

16. Transfer to learner progress

Back to contents

Begin with the reader’s actual decision. Longitudinal comparison needs aligned tasks, counting rules and enough text, plus direct inspection of new vocabulary use.

Work the teaching case in a visible order. Compare two 200-token responses to parallel prompts with the same tokenizer and MATTR window.

Make the reasoning independently checkable. Retain raw texts and annotate genuinely new, appropriate uses.

Test transfer by changing one condition while holding the others still. Increase the later writing time.

Keep the claim boundary visible. A higher diversity score does not prove durable receptive or productive knowledge.

Turn the claim into a test that could fail. First, Retain raw texts and annotate genuinely new, appropriate uses. Then change one consequential condition: Increase the later writing time.

The evidence receipt for “Transfer to learner progress” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 17 OF 24 · Transfer the reasoning

17. Transfer to reading materials

Back to contents

Keep the observation, transformation and conclusion on separate lines. Lexical diversity of a passage differs from vocabulary difficulty, coverage and dispersion.

Work the teaching case in a visible order. A passage can use many types yet still contain mostly high-frequency words.

Make the reasoning independently checkable. Pair diversity with frequency-profile and reader-knowledge evidence.

Test transfer by changing one condition while holding the others still. Concentrate unfamiliar words in one paragraph.

Keep the claim boundary visible. Diversity does not measure where unfamiliar items occur.

Before accepting the result, predict what should move and what should stay fixed. Apply this check: Pair diversity with frequency-profile and reader-knowledge evidence. Then run the changed case: Concentrate unfamiliar words in one paragraph.

The evidence receipt for “Transfer to reading materials” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 18 OF 24 · Transfer the reasoning

18. Transfer to spoken language

Back to contents

Use the short answer only after the reference condition is fixed. Transcripts include repetitions, fillers, repairs and transcription decisions that affect counts.

Work the teaching case in a visible order. Compute with fillers included and excluded under declared rules.

Make the reasoning independently checkable. Preserve audio, transcript convention and segmentation.

Test transfer by changing one condition while holding the others still. Compare a monologue with an interview.

Keep the claim boundary visible. Speech and writing scores should not be equated without validation.

Build a compact evidence receipt with input, unit, method, calculation, observation and limit. Use this check: Preserve audio, transcript convention and segmentation. For transfer, Compare a monologue with an interview.

The evidence receipt for “Transfer to spoken language” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 19 OF 24 · Transfer the reasoning

19. Transfer across languages

Back to contents

Trace the mechanism in an order another reader can reproduce. Morphology, segmentation and writing systems change what counts as a token or type.

Work the teaching case in a visible order. A word-form TTR in English and a segmented-character measure in another language do not share a natural unit.

Make the reasoning independently checkable. Use language-appropriate tools and validation.

Test transfer by changing one condition while holding the others still. Switch from word forms to morphemes.

Keep the claim boundary visible. A shared formula does not guarantee construct equivalence.

Ask a second reader to reproduce the decision from the stated evidence. They should be able to apply: Use language-appropriate tools and validation. Next, Switch from word forms to morphemes.

The evidence receipt for “Transfer across languages” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 20 OF 24 · Transfer the reasoning

20. Build a reproducible diversity receipt

Back to contents

Treat a plausible number as a prompt for a check rather than as proof. Store raw text, task, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and every intermediate count.

Work the teaching case in a visible order. Another reader recomputes .50, .25 and the fixed-window result from the teaching texts.

Make the reasoning independently checkable. Remove the counting unit and identify the ambiguity.

Test transfer by changing one condition while holding the others still. Version the receipt after changing tokenizer rules.

Keep the claim boundary visible. Reproducibility confirms the calculation, not the validity of a high-stakes interpretation.

Turn the claim into a test that could fail. First, Remove the counting unit and identify the ambiguity. Then change one consequential condition: Version the receipt after changing tokenizer rules.

The evidence receipt for “Build a reproducible diversity receipt” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 21 OF 24 · Report and continue

21. Repair “higher TTR means stronger vocabulary”

Back to contents

Begin with the reader’s actual decision. A shorter text can have a higher TTR simply because fewer opportunities for repetition have occurred.

Work the teaching case in a visible order. Text A’s .50 exceeds Text B’s .25 even though both use the same 50 types in the constructed example.

Make the reasoning independently checkable. Compare aligned lengths or a prespecified length-aware index.

Test transfer by changing one condition while holding the others still. Double Text A by repetition.

Keep the claim boundary visible. Vocabulary quality, precision, appropriateness and knowledge depth remain separate.

Before accepting the result, predict what should move and what should stay fixed. Apply this check: Compare aligned lengths or a prespecified length-aware index. Then run the changed case: Double Text A by repetition.

The evidence receipt for “Repair “higher TTR means stronger vocabulary”” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 22 OF 24 · Report and continue

22. Repair “length-aware means length-proof”

Back to contents

Keep the observation, transformation and conclusion on separate lines. MATTR, MTLD and related indices reduce particular length effects under studied conditions; they still depend on settings, sample length and language structure.

Work the teaching case in a visible order. Change MATTR window size and watch the central score change.

Make the reasoning independently checkable. Report sensitivity across defensible settings.

Test transfer by changing one condition while holding the others still. Apply the method to a 50-token sample.

Keep the claim boundary visible. Research findings about stability have stated corpora and ranges.

Build a compact evidence receipt with input, unit, method, calculation, observation and limit. Use this check: Report sensitivity across defensible settings. For transfer, Apply the method to a 50-token sample.

The evidence receipt for “Repair “length-aware means length-proof”” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 23 OF 24 · Report and continue

23. State the supported decision

Back to contents

Use the short answer only after the reference condition is fixed. The method supports explaining why simple TTRs from unequal texts are not directly comparable and building a transparent alternative comparison.

Work the teaching case in a visible order. It can reject the claim that the repeated 200-token text shows half the vocabulary merely because TTR halved.

Make the reasoning independently checkable. Keep raw type and token counts beside the chosen index.

Test transfer by changing one condition while holding the others still. Move from fictional sequences to learner writing.

Keep the claim boundary visible. High-stakes assessment needs validated instruments and qualified interpretation.

Ask a second reader to reproduce the decision from the stated evidence. They should be able to apply: Keep raw type and token counts beside the chosen index. Next, Move from fictional sequences to learner writing.

The evidence receipt for “State the supported decision” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Next chapter

CHAPTER 24 OF 24 · Report and continue

24. Continue through verified vocabulary routes

Back to contents

Trace the mechanism in an order another reader can reproduce. This branch joins counting units, dispersion, cue-dependent test scores and practical vocabulary learning without merging their constructs.

Work the teaching case in a visible order. Choose the sibling whose mechanism owns the next question.

Make the reasoning independently checkable. Keep lexical diversity, frequency, dispersion, coverage and learner knowledge on separate evidence lines.

Test transfer by changing one condition while holding the others still. Move to word-form versus lemma counting before recomputing the ratio.

Keep the claim boundary visible. This article owns simple TTR versus length-aware lexical-diversity reasoning across unequal texts.

Turn the claim into a test that could fail. First, Keep lexical diversity, frequency, dispersion, coverage and learner knowledge on separate evidence lines. Then change one consequential condition: Move to word-form versus lemma counting before recomputing the ratio.

The evidence receipt for “Continue through verified vocabulary routes” records source, input, transformation, changed-context prediction and limit separately. Keep raw text, prompt, tokenization, normalization, type unit, sample length, window, threshold, software version, exclusions and intermediate counts together.

Contents · Previous chapter · Return to Vocabulary

Word dispersion across unequal text lengths

Word dispersion across unequal text lengths. Use this when the question concerns where a target word occurs across corpus parts.

Word forms, lemmas and families

Word forms, lemmas and families. Use this to choose the unit counted as a distinct type.

Test scores when cues change

Test scores when cues change. Use this when vocabulary scores differ because prompts or retrieval cues changed.

Frequently asked questions

Why can TTR fall when a text gets longer?

Repeated tokens accumulate as the denominator grows, so length affects the ratio.

Does MATTR remove every length effect?

No. It reduces one direct dependency but still depends on window size, sample length, tokenization and language.

Does a higher diversity score prove better writing?

No. Accuracy, appropriateness, precision, cohesion, task fit and vocabulary knowledge require other evidence.

Authoritative sources and further reading

Covington and McFall: Moving-Average Type–Token Ratio. The primary paper explains TTR’s text-length problem and introduces MATTR with a fixed moving window.

McCarthy and Jarvis: MTLD, vocd-D and HD-D. The primary validation study evaluates MTLD and other lexical-diversity approaches.

Koizumi and In’nami: effects of text length on lexical-diversity measures. The empirical study tests several indices across 50–200-token learner texts and reports the conditions under which length still matters.

Return to the Vocabulary subject guide, The Gold Standard of Education, or the How X Works hub.