Document frequency is the number of documents in a defined collection that contain a word or phrase at least once under a stated search rule. It answers ‘How many different documents contain this expression?’ rather than ‘How many times does the expression appear?’ One text that repeats a word twenty times still contributes only one to its basic document frequency.
Why is this a useful vocabulary technology? Because repeated language can mislead readers. A Science textbook might repeat a technical word in one chapter while an ordinary connector appears across many kinds of documents. Document frequency helps us distinguish concentration from breadth. But the number does not show whether the word is understood, whether the text is accurate or whether several sources independently support a claim.
This guide explains what document frequency is, why term frequency and collection frequency are different, how document counts connect to inverse document frequency (IDF) and TF–IDF, and when tokenisation, duplicates, genre or meaning can distort conclusions. It includes original classroom calculations and real-life applications across Primary and Secondary English, Mathematics, Science, History, GP and translation.
Editorial note: All worked numerical data below are fictional teaching examples. They are not observations from an actual British National Corpus, student cohort or search index.
Contents: when does a word appear widely, not just often?
Open 44 chapters and classroom investigations
- 1. Document frequency counts documents, not repeated mentions
- 2. The difference between term frequency and document frequency
- 3. Why the distinction matters for vocabulary
- 4. A four-document fictional collection
- 5. Presence is binary within a document
- 6. N is the number of documents examined
- 7. Document-frequency proportion
- 8. Collection frequency can hide concentration
- 9. Document frequency is not the same as corpus dispersion
- 10. The choice of document boundary changes the answer
- 11. A class corpus may be too small to generalise
- 12. Document duplicates distort perceived reach
- 13. A document may include quotation rather than endorsement
- 14. Exact forms and lemmas produce different df values
- 15. Case and named entities need deliberate handling
- 16. Multiword phrases count differently
- 17. Document frequency and inverse document frequency
- 18. What happens when df is zero
- 19. IDF is not a difficulty score
- 20. TF–IDF combines different signals
- 21. Document frequency and stopword detection
- 22. Minimum document-frequency thresholds
- 23. A threshold and a statistical test are different
- 24. Document size changes the opportunity for presence
- 25. Document frequency does not measure exact location
- 26. Document frequency does not tell us the right sense
- 27. A document-frequency table is an audit tool
- 28. A simple binary matrix
- 29. Document frequency in a school library
- 30. Primary English: a four-card exercise
- 31. Secondary English: topic words across genres
- 32. Mathematics: percentages depend on the denominator
- 33. Science: a specialist word can be vital despite low df
- 34. History: avoid treating mention as support
- 35. General Paper: evidence can be repeated without being independent
- 36. Translation: words are not equivalent counting units
- 37. Search rankings may use more than df
- 38. Beware of a corpus built from templates
- 39. Time periods change which words are widespread
- 40. Zero frequency is not evidence of impossibility
- 41. How to choose which measure belongs in an article
- 42. A privacy-safe classroom dataset
- 43. A repeatable ten-minute practice routine
- 44. The final lesson: spread and meaning are different
Original binary document–term example
The following four small records are invented classroom texts. For simplicity, matching is case-insensitive, the target words are counted as whole words, and documents are the four named cards.
| Fictional document | evidence (present?) | claim (present?) | Invented sentence |
|---|---|---|---|
| A | 1 | 0 | Evidence improves a careful explanation. |
| B | 1 | 1 | A claim needs evidence. |
| C | 0 | 1 | A claim was challenged. |
| D | 0 | 1 | Another claim was questioned. |
Document frequency of evidence is 2 because two cards contain it. Document frequency of claim is 3. No assertion about real English frequency follows from this invented dataset. The denominators are the four cards, not all words in English.
1. Document frequency counts documents, not repeated mentions
Document frequency, often abbreviated df, is the number of documents in a defined collection that contain a specified term under a documented matching rule. If a word occurs thirty times in one essay and nowhere else, its document frequency is one, not thirty.
This is an important change of perspective. Ordinary word frequency asks how often something appears; document frequency asks how widely it is present across separate texts. Neither number alone tells us whether the word was used correctly, what it means in context or how important the subject is to a learner.
2. The difference between term frequency and document frequency
Term frequency counts occurrences of a term within a particular document, often represented as tf. Document frequency asks in how many documents the term appears at least once. Collection frequency totals a term’s occurrences across the collection.
The same term can therefore have high collection frequency but low document frequency when one source repeats it many times. A classroom report should not substitute one number for the other. Choose the unit that answers the research question.
3. Why the distinction matters for vocabulary
A child might encounter the word evaporation repeatedly in one Science chapter yet rarely in storybooks. A raw count from the chapter makes it look common in that text, while document frequency across several genres could show it is concentrated in specialist material.
This can help a tutor plan exposure. If the aim is to understand a Science topic, concentrated repetitions may be valuable. If the aim is broad reading readiness, words encountered across multiple kinds of text may deserve a different teaching priority. Both decisions need context beyond one statistic.
4. A four-document fictional collection
Imagine four original text cards. Card A says ‘Evidence supports evidence-based explanations.’ Card B says ‘The evidence was checked.’ Card C says ‘A claim requires reasons.’ Card D says ‘A claim can be challenged.’ We will define the term evidence as an exact lowercase word and treat each card as one document.
Within this invented collection, evidence occurs in A and B, so its document frequency is two. The precise collection frequency depends on the matching rule and whether hyphenated forms are split. This is why the unit, tokenisation and casing must be stated before counting.
5. Presence is binary within a document
For the basic document-frequency calculation, a document contributes one if the term appears and zero if it does not. Its contribution is not increased when the term appears a second or tenth time in that same document.
This binary rule prevents a single very repetitive document from automatically counting as many different sources. It does not establish independence of sources: two separate documents may still be copies of one another. Source provenance is a further question.
6. N is the number of documents examined
Researchers often denote the total number of documents in a collection by N. Document frequency is evaluated relative to N: a term found in 20 of 25 documents has a different spread from a term found in 20 of 20,000 documents.
When reporting a percentage, write both numerator and denominator. ’20 documents contain this word’ is incomplete if the goal is to compare coverage across collections. A credible statement names the corpus and selection method.
7. Document-frequency proportion
A simple document-frequency proportion is df divided by N. If a fictional term appears in 3 of 5 documents, the proportion is 0.6, or 60%. This is a descriptive measure of presence in that imaginary collection.
It is not the probability that a random person knows the word. It does not automatically mean that 60% of English writing uses the term. Such conclusions would require a relevant sampling design and clearly defined population.
8. Collection frequency can hide concentration
Suppose an invented six-document collection contains twelve occurrences of process entirely within one document. A different term might occur once in each of all six documents. The first has more total mentions; the second has a larger document frequency.
That contrast reveals why document frequency is useful alongside raw counts. A keyword heavily repeated in a specialist paper may be less broadly distributed than a quiet linking term that appears in almost every document. A good analyst inspects both.
9. Document frequency is not the same as corpus dispersion
Document frequency says how many files include a term. Dispersion describes its distribution across documents or segments and sometimes its internal spread. Two terms with identical document frequency can have very different patterns of repetition or concentration.
Use document frequency for the simple question ‘How many documents contain it?’ and a dispersion plot or other analysis when positions and clustering matter. Giving each measure a clearly defined job makes vocabulary research more precise.
10. The choice of document boundary changes the answer
A ‘document’ may mean a complete book, an article, a student composition, a web page or a paragraph, depending on the research design. Splitting one long book into chapters increases the number of documents and can change document frequency.
The boundaries must therefore be fixed before comparing counts. Do not compare df across datasets when one uses paragraphs and another entire books without explaining the difference. The method is only as comparable as the units.
11. A class corpus may be too small to generalise
Five student paragraphs can demonstrate document frequency, but they cannot represent all Primary English writing. The class may share a lesson topic, writing prompt and vocabulary taught the previous week.
A tutor can use the local collection to decide what to review in that class. The finding should be labelled as class-specific. A teacher’s strong lesson comes from accurate interpretation, not an exaggerated claim that the data reveal the habits of every learner in Singapore.
12. Document duplicates distort perceived reach
Two webpages can be copies of the same press release. Counting them as separate documents may increase document frequency without showing wider independent use. A school dataset can similarly contain repeated worksheets or model answers.
Check duplication and note whether the goal is to count files as stored or independent sources. Neither interpretation is automatically wrong, but they answer different questions. When measuring breadth of language use, duplicated material deserves special scrutiny.
13. A document may include quotation rather than endorsement
A term appears in a document even when it is quoted only to be rejected. Document frequency records presence, not agreement. A news report that quotes a false claim still contains the words of that claim.
Do not infer a speaker’s belief from a term’s document frequency. For History or media literacy, inspect who used the expression, why it appears and how the writer evaluates it. The count is a map to sources, not a replacement for source analysis.
14. Exact forms and lemmas produce different df values
Counting write exactly is different from counting occurrences of the lemma write, which may include wrote and written. A document containing only wrote contributes to the lemma’s df but not necessarily to the exact-form df.
The reported quantity needs a matching rule. If the task is about the word family, lemmatisation may be appropriate; if the task is about a spelling or tense, exact matching may serve better. Automatic lemma tags should still be spot-checked.
15. Case and named entities need deliberate handling
A case-insensitive search can combine May as a month with may as a modal. A document-frequency calculation then reports the spread of a spelling pattern, not a single grammatical meaning.
For a study of possibility or permission, manually check or grammatically filter the relevant uses. Proper names and sentence-initial capitals can create false matches. A high df can be real for the surface string while misleading about the intended concept.
16. Multiword phrases count differently
A phrase such as as a result can be counted by the number of documents containing the whole sequence under a chosen query. That is not equivalent to the number of documents containing result on its own.
Students should keep phrase, word-form and lemma queries distinct. If punctuation, inserted words or alternative expressions are allowed, explain the rules. A well-described phrase frequency answers a specific question about expression, not every possible way to communicate the concept.
17. Document frequency and inverse document frequency
Inverse document frequency, or IDF, is a weighting idea used in information retrieval. In an elementary unsmoothed formulation for a term present in at least one document, IDF is the logarithm of N divided by the term’s document frequency.
The intuition is that a term appearing in nearly every document distinguishes those documents less well than a term appearing in relatively few. The exact formula, smoothing and implementation vary. IDF measures a form of retrieval distinctiveness, not the moral or educational importance of a word.
18. What happens when df is zero
A word with document frequency zero was not matched in any included document under the stated query. In an unsmoothed logarithmic IDF formula, dividing by zero is undefined, so implementations use particular conventions or smoothing strategies.
This is why the formula should not be copied without reading the tool’s documentation. Zero observed matches may also result from spelling, corpus scope or search settings. It does not prove that the word is nonexistent in the language.
19. IDF is not a difficulty score
A rare term in one corpus could be elementary in another domain or language variety. A word such as a local place name may have low document frequency in a general corpus but be well known to the intended readers.
Do not translate retrieval rarity into a direct measure of student vocabulary knowledge. Difficulty also depends on learner experience, sense, morphological transparency, conceptual familiarity and task. IDF solves a different problem.
20. TF–IDF combines different signals
Term frequency–inverse document frequency combines within-document occurrence information with a document-level distinctiveness factor. Variants of the scheme exist, with different normalisation or smoothing choices.
For an educational example, the idea is enough: a word repeated in a particular text and relatively uncommon across the collection can help distinguish that text from others. The score does not establish whether a claim is true or which word a student should use in an essay.
21. Document frequency and stopword detection
Some text-vectorisation tools can filter terms with document frequency above a chosen threshold. The idea is that a word found in nearly all documents might offer little value for distinguishing documents in that collection.
The threshold is task-dependent. It would be dangerous to assume that an extremely common word such as not is harmless to remove from every analysis. Negation, time, permission and logical operators can be essential even when they appear across many documents.
22. Minimum document-frequency thresholds
A minimum document-frequency setting can exclude terms appearing in too few documents when constructing a feature vocabulary. This may reduce noise or sparse features in some machine-learning workflows.
It can also erase rare but important technical terms, minority expressions or signals about unusual events. A school project should assess the intended task and check what disappears. Convenience in modelling is not a reason to suppress meaningful language.
23. A threshold and a statistical test are different
Selecting a minimum df does not establish that a pattern is statistically significant. Nor does a high df prove the word is typical of a wider population. Thresholds are rules for retaining or excluding features in a specified dataset.
If the project needs claims about populations, it requires appropriate sampling, model assumptions and analysis. The classroom calculation can remain descriptive and honest. The absence of a significance test is not a problem if the report does not pretend to have performed one.
24. Document size changes the opportunity for presence
A very long document gives a term more opportunities to occur than a short note. Document frequency treats each as one possible document containing the term, even though their lengths are different.
For a careful study, report length distribution and consider whether segmentation or a different measure would answer the question better. The correct method depends on whether the object is editorial breadth, independent sources or probability of encounter in running text.
25. Document frequency does not measure exact location
The word might appear on the first line, in a heading or only in the final appendix. Basic df gives no position information. A dispersion plot or source view can help locate matches.
This is especially useful when a tutor asks whether a key term is introduced, explained and revisited in an essay. Presence alone is not enough to assess how well the concept is developed.
26. Document frequency does not tell us the right sense
The term cell can mean a biological unit, a room or an item in a spreadsheet. A surface-form df across mixed documents merges these meanings.
For subject-specific vocabulary teaching, classify senses before claiming that a concept is widely used. A high number of matched spellings may conceal a smaller number of examples with the desired meaning. Context and subject matter remain essential.
27. A document-frequency table is an audit tool
A good worksheet lists each document, whether the term is present, and optionally its occurrence count. The student can inspect how the aggregate df was obtained, making errors easier to find.
For a small class activity, a tick or zero is sufficient. More advanced students may use a binary document–term matrix. The important feature is traceability: every summary count can be connected to the texts it represents.
28. A simple binary matrix
Imagine three fictional documents and two terms. Write a row for each document and a column for each term, placing 1 if the term occurs and 0 if it does not. The sum down a column is the document frequency of that term.
This exercise connects vocabulary analysis to Mathematics, spreadsheets and information retrieval. It also helps learners see why repeated occurrences do not change a binary presence value. A separate count table would answer the occurrence question.
29. Document frequency in a school library
Suppose a teacher reviews a small set of licensed reading passages to see which specialist terms are encountered across topics. A term appearing in several independent passages may have broader distribution than a term confined to one scientific chapter.
The result can inform exposure planning but cannot replace assessment of which meanings students understand. Students may recognise a common term incorrectly or understand a rare technical term through a clear explanation. Frequency supports teaching judgement; it does not replace it.
30. Primary English: a four-card exercise
Give pupils four original short passages. Ask them to circle the word journey each time it appears, then tick the cards where it appears at least once. Count all occurrences and count the ticked cards separately.
The teacher can explain the difference in everyday language: ‘How many times did we see the word?’ versus ‘How many stories contained the word?’ The labels term frequency and document frequency can be introduced after the idea is secure.
31. Secondary English: topic words across genres
A Secondary learner can compare original Science notes, an editorial paragraph, a narrative and a school notice. The learner chooses one target expression, tracks presence and reads the surrounding sentences.
This shows that one spelling can appear across genres with different meanings or functions. A strong explanation notes that difference instead of merely ranking the documents. Vocabulary learning improves when frequency questions lead back to meaning and audience.
32. Mathematics: percentages depend on the denominator
If a target appears in 6 of 10 documents, the observed proportion is 60%. If another corpus has 6 of 100, it is 6%. The raw document count is the same but the proportions differ.
Students must label numerator and denominator and ensure the document unit is comparable. This is a practical lesson in proportional reasoning and in avoiding misleading statistics. A percentage alone is still not evidence that the sampled corpus represents the entire language.
33. Science: a specialist word can be vital despite low df
A rare technical term may occur in only one investigation but name exactly the mechanism under study. Its low document frequency reflects how narrowly the concept appears in the collection, not its lack of importance.
When revising for an examination, choose vocabulary based on the syllabus, underlying concepts and likely tasks. Corpus statistics can show distribution but must not override the definition required by Science.
34. History: avoid treating mention as support
A term such as reform may occur in several sources because authors support, criticise or quote the idea. Document frequency tells us that the word is present; it cannot tell us whether those writers agree.
Source evaluation requires author, date, audience, purpose and corroborating evidence. A vocabulary analysis should distinguish recurring labels from proven historical interpretation. The reader must not turn a count of mentions into a count of endorsements.
35. General Paper: evidence can be repeated without being independent
Five articles can mention one claim because they copy the same press release. A raw document-frequency count of five does not establish five independent confirmations. This matters in GP essays and media literacy.
Check the primary source and the relationship among articles. A responsible student cites independent evidence where possible and distinguishes reporting from verification. Document frequency is a useful diagnostic but not a substitute for provenance.
36. Translation: words are not equivalent counting units
One language may express a concept through one word and another through a multiword construction. Comparing document frequency of identical strings across languages may therefore be meaningless.
Instead, define a concept or communicative function and identify appropriate target-language expressions. Record which forms were searched and verify context. Cross-language semantic equivalence requires more than shared spelling or raw counts.
37. Search rankings may use more than df
Modern search engines combine many signals, including document statistics, contextual matching and relevance assessments. Basic TF–IDF provides a useful foundational idea but does not describe every current ranking system.
Students should avoid claiming they know why a live search result ranked first based solely on the term’s document frequency. Use official documentation and actual evidence for system-specific claims. The general concept remains valuable even as implementations vary.
38. Beware of a corpus built from templates
If hundreds of school notices share the same footer and headings, certain words may appear in nearly every document for formatting reasons. The document-frequency figure may reflect the template rather than the main content.
A researcher could remove or separately analyse boilerplate using documented rules. Keep the original documents accessible where permitted and state the preprocessing choice. Search counts should not silently inherit artefacts from website design.
39. Time periods change which words are widespread
Document frequency in a historical dataset may differ from a more recent dataset because new technologies, events or writing conventions appear. The comparison can be interesting, but collections must be comparable in topic, source and size.
A change in document frequency is not automatically a change in people’s knowledge. Publication patterns, digitisation and genre selection also affect what survives in the corpus. Make the scope explicit before drawing social conclusions.
40. Zero frequency is not evidence of impossibility
A term absent from a small set of writing may simply have had no reason to appear. It may also be present under a different spelling or inflection. A zero df therefore describes the search result and dataset, not the existence of the word.
For vocabulary instruction, consult a credible dictionary and task-relevant examples rather than telling a student that a word is invalid because it was not found in a local corpus.
41. How to choose which measure belongs in an article
Use term frequency when the question asks about repetition inside a text. Use document frequency when it asks how many independent files contain a term. Use collection frequency when it asks how many total occurrences exist across the whole set.
For position and clustering, consider dispersion; for retrieval distinctiveness, consider IDF or related weighting. Each tool has a narrow job. Better analysis comes from choosing the right measurement before seeing the result.
42. A privacy-safe classroom dataset
Teachers may want to analyse student compositions, but real writing can contain names, identifying details or personal events. A document-frequency table does not automatically anonymise those materials.
Use fictional practice texts for general demonstrations. For legitimate institutional studies, follow permissions, data-protection policy and access limits. No analytical convenience justifies exposing student material outside approved systems.
43. A repeatable ten-minute practice routine
Select five invented or openly licensed short passages, define one term and matching rule, mark its presence in each document, calculate df and total occurrence count, then compare the two measures.
Finally, read two sentences containing the term and identify its meaning. This last step prevents the lesson from turning entirely into arithmetic. The result should be a student who understands both what was counted and what the count cannot tell them.
44. The final lesson: spread and meaning are different
A word can be repeated intensely in one document or appear quietly across many. Document frequency helps distinguish those patterns of presence at the document level, but it does not reveal sense, truth, difficulty or value without context.
Used properly, the measure is another instrument in a vocabulary toolkit. It helps students locate evidence, compare sources and avoid mistaking repetition for broad usage. The most important outcome is better judgement in reading and writing.
A six-step calculation and interpretation exercise
- Choose a fixed set of identifiable documents and define each document boundary.
- Select a target word, lemma or phrase and write down the exact matching convention.
- Mark every document with 1 if the target is present, 0 otherwise.
- Sum the binary values to obtain document frequency; count occurrences separately if needed.
- Calculate df divided by N only if a proportion is useful, and label the denominator.
- Read a sample of the matching sentences before claiming anything about meaning, style or source agreement.
Frequently asked questions
1. Can repeated mentions raise document frequency?
Not within the same document. Basic df counts whether each document includes the term at least once.
2. Is document frequency the same as word frequency?
No. Word or term frequency counts occurrences; document frequency counts documents containing the term.
3. What does N mean?
The total number of documents in the specified collection.
4. Does df measure how many people use the word?
No. Documents may share authors, templates or quoted material.
5. Can a word have a low df and be essential?
Yes. A specialist term can be crucial to one document even if absent from most others.
6. What is IDF?
Inverse document frequency, a retrieval-weighting idea that generally gives lower discriminative weight to terms found across many documents.
7. Is TF–IDF a truth score?
No. It concerns term weights for information retrieval, not the truth of claims.
8. What if the word never appears?
df is zero for that dataset and query; it does not mean the word does not exist.
9. Can document frequency help teaching?
Yes, as one signal of distribution alongside meaning, curriculum relevance and learner needs.
10. What makes the measure trustworthy?
A defined collection, document unit, matching rule, inspection of duplicates and appropriate limits on conclusions.
Primary references and continued vocabulary learning
- Introduction to Information Retrieval — Stanford/Cambridge University Press — the source text for term frequency, document frequency and inverse document frequency.
- Scikit-learn: CountVectorizer — document-frequency thresholds min_df and max_df in a text vectoriser.
- Scikit-learn: Feature Extraction — term representations, stopwords and weighting.
- NLTK Book: Language Processing and Counting — token, type and frequency basics.
- Vocabulary | What Is a Dispersion Plot? — where occurrences lie inside texts.
- Vocabulary | What Is a Corpus Comparison? — compare appropriate text collections.
- Vocabulary | What Is Normalised Word Frequency? — choose denominators carefully.
- Vocabulary | What Is a Stopword List? — understand the consequences of excluding common words.
- Vocabulary Learning Hub — continue the school-to-life vocabulary pathway.
Document frequency is one clear tool among many. It answers a narrow question about presence across a collection. Strong vocabulary learning starts with that precision, then returns to full sentences, source credibility and the decisions people need to make.