In a library about cars, the word car tells you almost nothing.
In that same library, the term hydropneumatic may narrow the collection immediately.
Keyword discrimination is the ability of a term or phrase to distinguish a smaller, more relevant portion of a corpus from the rest of that corpus.
This is the third pillar beneath How Keywords Work. The master owns lexical cues broadly. This article owns corpus rarity: document frequency, inverse document frequency, common terms, rare terms, stop words, field context and why discriminating power is useful without becoming a synonym for importance or truth.
Quick Read
A term that appears in almost every document is usually weak at distinguishing which document is relevant. Information retrieval therefore uses document frequency—the number of documents containing a term—and often inverse document frequency, which increases as document frequency falls. The classic Stanford information-retrieval model defines IDF from the ratio between corpus size and document frequency. Rare terms can therefore contribute more discriminating evidence than common terms. But rarity is corpus-relative and not equivalent to semantic importance: a typo can be rare, a critical common word can matter greatly, and exact identifiers may deserve priority for reasons other than IDF. Good search combines discrimination with field, phrase, intent, authority and semantic context.
term → count documents containing it → compare with corpus size → estimate discriminating power → combine with within-document evidence → apply field/context/phrase rules → retrieve/rank → verify meaning
Document Frequency Counts Documents, Not Occurrences
Suppose insurance appears 100 times in one document.
Its document frequency is still one if no other document contains it.
Document frequency asks:
How many documents in this collection contain this term at least once?
Collection Frequency Is a Different Statistic
Collection frequency counts total occurrences of the term across the corpus.
The Stanford Introduction to Information Retrieval gives a classic reason to prefer document frequency when measuring discrimination: two terms can have similar total occurrence counts yet be distributed across documents very differently.
For retrieval, the number of documents containing a term often better captures how much that term separates one document from another.
IDF Makes the Intuition Explicit
In the classic formulation:
idf(t) = log(N / df(t))
where:
- N = number of documents in the corpus;
- df(t) = number of documents containing term t.
As document frequency falls, IDF rises.
The exact formula varies in practical ranking systems, but the central intuition survives:
a term seen everywhere distinguishes weakly; a term seen rarely can distinguish strongly.
Rarity Is Relative to the Corpus
Photosynthesis may be relatively rare on the general web.
Inside a biology textbook corpus, it may be common.
Mitochondria may narrow a general school corpus strongly but weakly inside a cell-biology journal collection.
Discrimination must therefore be computed against the collection actually being searched.
Corpus Changes Can Change IDF Without Any Document Changing
Add one million chemistry documents.
Terms such as molecule and reaction become much more common in the collection.
The same document text now lives in a different statistical environment.
Ranking evidence is partly a property of corpus state, not just document content.
Within-Document Frequency Solves Another Part of the Problem
A rare term appears once.
Another relevant term appears repeatedly throughout the page.
Term frequency asks how strongly the document itself uses the term.
IDF asks how strongly that term distinguishes documents in the corpus.
TF-IDF-style thinking combines the two.
BM25 Refines the Same Basic Evidence
Modern lexical systems commonly use BM25-family scoring rather than raw TF-IDF.
BM25 includes:
- term-frequency saturation;
- document-length normalisation;
- inverse-document-frequency-style weighting.
The exact implementation can vary, but uncommon query terms generally carry more discriminating weight than ubiquitous ones.
Rare Does Not Mean Important
A random typo can be unique.
A meaningless serial fragment can be unique.
A rare surname can be extremely important for one lookup.
Rarity measures discrimination in the corpus.
Importance depends on the user’s task.
rare ≠ valuable; common ≠ irrelevant.
Common Terms Can Be Essential to Meaning
to be or not to be.
The Who.
Legal clauses where not reverses meaning.
Stop-word removal based only on global commonness can destroy phrase or negation semantics.
Stop Words Are an Engineering Choice
Historically, search systems often omitted extremely common function words to save storage and computation.
Modern systems can afford to retain more of them, especially when phrase search, natural-language queries and exact quotations matter.
A “stop word” is not linguistically meaningless.
It is a term an indexing/search configuration chooses to treat differently.
Field Context Can Override Global Commonness
Singapore may be common across a Singapore website.
In a structured country field, it can still be an exact eligibility constraint.
Discrimination in full text and exact field semantics solve different jobs.
Title Presence Can Matter Even When a Term Is Common
A term appears in every footer across the site.
It appears in the title of only twelve pages.
Field-aware indexing lets the ranking system treat title occurrence as different evidence from generic body/footer occurrence.
Phrase Rarity Can Exceed Individual-Term Rarity
learning is common.
machine is common.
machine learning is more specific.
Longer combinations can become strongly discriminating even when each component term is ordinary.
The first pillar, Keyword Specificity, owns how phrase structure and extra constraints narrow retrieval.
Rare Technical Terms Are Powerful Bridges
A learner knows one rare term from a worksheet.
Searching it can jump directly into the right topic cluster.
Technical vocabulary often has high discrimination precisely because it appears in fewer unrelated documents.
But Technical Jargon Can Hurt Novice Recall
The corpus uses myocardial infarction.
The user searches heart attack.
The rare technical term is highly discriminating once known.
It is useless if the user never supplies or reaches it.
Controlled synonyms and semantic search bridge this lexical gap.
Controlled Labels Can Preserve a Rare Alias Without Making It Preferred
A rare historical label may be extremely discriminating for old documents.
It need not remain the public preferred label.
The second pillar, Controlled Labels, owns how that alias can remain searchable under a stable concept identity.
Vocabulary Drift Changes Discrimination Over Time
A term is dominant in 2005.
A new term replaces it by 2026.
The old term becomes rarer in recent documents and may appear highly discriminating—but now toward historical material.
The fourth pillar, Vocabulary Drift, owns this time dimension.
Temporal Corpora Produce Different Term Statistics
Search only documents from 2026.
Search the full 1990–2026 archive.
The same term can have different document frequencies in the two search spaces.
Time filtering changes not only candidate eligibility but lexical statistics.
Tenant and Domain Boundaries Change Discrimination Too
Term student is common in an education tenant.
It may be rare in a manufacturing corpus.
Shared global term statistics can therefore behave differently from per-domain statistics.
Permission-aware systems should also ask whether global statistics themselves reveal sensitive corpus composition.
A Unique Identifier Is Maximally Discriminating for Lookup
Invoice 8472.
ISBN.
Accession number.
If the field is trustworthy and the identity is exact, rarity is not merely statistical—it expresses a lookup key.
Do not let fuzzy ranking dilute exact key semantics.
Discrimination Does Not Prove Relevance
A document contains a rare query term in a bibliography.
Another document discusses the concept deeply using a synonym.
The rare lexical hit is discriminating but may be less useful.
Ranking needs field, frequency, proximity, semantic and task evidence too.
Discrimination Does Not Prove Authority
A fringe page invents a rare phrase.
The phrase has enormous IDF because almost nobody uses it.
That does not make the page authoritative.
Authority and evidence need independent signals.
Discrimination Does Not Prove Truth
A rare false claim can be highly distinctive.
Search relevance scores help find candidate material.
How Evidence Works owns whether the claim should be believed.
Search Optimisation Should Not Manufacture Artificial Discrimination
A publisher invents awkward rare phrases only because a keyword tool shows low competition.
The page becomes lexically distinctive but reader-hostile.
Google’s current spam guidance rejects keyword stuffing and search-engine-first manipulation.
Useful specificity should arise from real subject distinctions and reader jobs, not synthetic rarity.
Discrimination Can Guide Internal Linking
Anchor text should describe the destination accurately.
Highly generic anchors such as read more carry little lexical context.
More descriptive anchors expose the relationship between pages without requiring unnatural repetition.
Corpus Diagnostics Can Reveal Over-Broad Pages
A page contains only very common terms for its domain.
It may still be excellent prose.
But if it owns a specialised job, the absence of domain-specific vocabulary can indicate that the page has not developed enough conceptual precision.
This is an editorial diagnostic, not a density recipe.
Keyword Research Can Reveal Language, Not Automatically Value
A keyword tool reports search volume and perhaps competition.
That tells us something about user language and ecosystem usage.
It does not tell us:
- whether a separate page should exist;
- whether the term is accurate;
- whether the page is authoritative;
- whether the user will be helped.
Discrimination is one lexical property inside a much larger publishing and retrieval system.
A Better Keyword-Discrimination Model
query term → corpus boundary → document frequency → inverse rarity signal → within-document frequency/field/position → phrase and synonym context → candidate score → semantic/authority checks → user task
A 30-Lens Keyword Discrimination Audit
- Term: what lexical cue is being measured?
- Corpus: what collection defines commonness?
- N: how many documents are in that corpus?
- Document frequency: how many documents contain the term?
- Collection frequency: how many total occurrences?
- IDF: how much rarity weight follows?
- Term frequency: how often inside this document?
- Saturation: are repetitions diminishing?
- Document length: is frequency normalised?
- Field: title, body, identifier, subject?
- Position: does the term occur in a phrase?
- Exact key: is this actually an identifier?
- Phrase rarity: is a multiword combination more discriminating?
- Synonym: could lexical rarity hide vocabulary mismatch?
- Alias: is a rare alternate label useful?
- Stop word: is commonness being mistaken for meaninglessness?
- Negation: does a common term reverse meaning?
- Domain: is rarity different inside a specialist corpus?
- Tenant: should statistics be scoped for privacy?
- Time: does historical/current corpus change frequency?
- Drift: is the rare term simply old terminology?
- Typo: is rarity caused by error?
- Novel term: is rarity caused by emerging language?
- Relevance: does the hit actually answer the query?
- Authority: who owns the underlying claim?
- Truth: is rare wording being confused with evidence?
- SEO: is artificial rarity being manufactured?
- Anchor: does descriptive linking add useful context?
- Editorial: does specialised content contain necessary domain vocabulary?
- World return: did the discriminating term lead to the right real resource/concept rather than merely a rare string?
Laboratory 1: Compute IDF Intuitively
Create a ten-document corpus. Let one term appear in ten documents, another in five and another in one. Compare their document frequencies and IDF intuition.
Laboratory 2: Change the Corpus
Take the same query term and compute its document frequency in a general school corpus and a specialised chemistry corpus. Explain why the term’s discriminating power changes.
Laboratory 3: Rare but Wrong
Create one relevant page using a common synonym and one irrelevant page containing a rare exact query term. Show why IDF alone cannot determine final relevance.
For Primary Readers
If every child in class is wearing shoes, “shoes” does not help you find one child. If only one child has a purple telescope badge, that clue narrows the class immediately.
For Secondary Readers
Define document frequency and explain why inverse document frequency rewards terms that occur in fewer documents while still requiring other relevance evidence.
For Advanced Readers
Model lexical discrimination as corpus-relative information contributed by a term’s document distribution. IDF is a monotonic transform of inverse document frequency, useful for ranking but semantically agnostic; relevance requires interaction with within-document evidence, structured fields, intent and other retrieval signals.
Common Misconceptions
- “Rare keywords are more important.” They can be more discriminating, not inherently more valuable.
- “Common words are useless.” They can be essential in phrases, negation and structured fields.
- “IDF is fixed for a word.” It changes with corpus boundary and document population.
- “A rare exact match should always outrank a common synonym.” Relevance depends on context and task, not rarity alone.
- “Low-competition rare keywords justify new pages.” Publishing still requires an independent reader job and substantive content.
Research Corridor
- Stanford — Introduction to Information Retrieval — document frequency, inverse document frequency, TF-IDF and retrieval scoring.
- Apache Lucene — current lexical search implementation family including BM25-style scoring.
- Google Search Central — Spam Policies — current keyword-stuffing guidance.
- eduKateSG — Inverted Index & Postings.
Frequently Asked Questions
What is keyword discrimination?
It is how strongly a term or phrase distinguishes a subset of documents within a particular corpus, often related to how many documents contain that term.
What is inverse document frequency?
IDF is a term-weighting idea that increases as the proportion of documents containing the term decreases, giving rarer terms more discriminating weight in lexical retrieval.
Does a high-IDF term mean the document is authoritative?
No. IDF concerns corpus rarity, not authority, factual accuracy or evidence quality.
Final Thought: A Useful Keyword Is Often the Clue the Corpus Cannot Say Everywhere
Common words describe the room.
Discriminating words help us find the chair.
Keyword discrimination works when rarity helps narrow the search without being promoted into a false theory that unusual words are automatically more meaningful, more authoritative or more true.
KEYWORDS · FOUR PILLAR LEGS
Return to How Keywords Work, or continue through Keyword Specificity, Controlled Labels and Vocabulary Drift. Return to the Information & Representation Hub.