Collocation ranking stability describes how consistently the same candidate word partnerships appear at similar positions in a ranking when the underlying sample, corpus partition, search settings or association method changes. It is a property to investigate, not a promise that every word has one universal top-ten list.
Did you know that the number-one word partner in one corpus sample can fall to second place in another, while both results are correctly calculated? Imagine a class searching for nouns near strong. In one constructed sample, strong evidence is the highest-count adjacent phrase; in a second, strong coffee leads. The difference need not indicate that English itself changed. It may reflect which texts were sampled and which question the ranking asks.
This guide distinguishes frequency rankings from association-strength rankings, explains top-k overlap and the Jaccard calculation, and shows how to check sample composition, rare pairs, source independence, query parameters and lexical meaning. It uses independently constructed arithmetic and draws methodological context from corpus linguistic studies of collocation sampling and measure choice.
How the evidence is used: The general method is informed by the linked linguistic research. The classroom datasets, tables, numbers and pupil examples in this guide were constructed for explanation, not drawn from a real education system sample. The examples therefore do not establish actual frequency, performance or learning gains.
EDUKATE · VOCABULARY METHODS · READER ROUTES
Choose what you need first
- Learn the core idea · Chapter 1 onward
- Design a fair test · Chapter 7 onward
- Inspect numbers and examples · Chapter 13 onward
- Apply the method to new texts · Chapter 19 onward
All chapters · Worked table · Decision cases · Practice and solutions · Questions and glossary
Contents in complete reading order
- 1. Begin with the word partnership and its purpose
- 2. Recognise that a ranking is an output of choices
- 3. Separate raw pair count from association strength
- 4. Define the units of the sample
- 5. Understand downsampling
- 6. Keep sample size and source diversity distinct
- 7. Control for genre composition
- 8. Distinguish a stable list from a stable order
- 9. Use top-k overlap as a simple first check
- 10. Calculate Jaccard similarity correctly
- 11. Recognise that rank position needs its own measure
- 12. Understand ties and close counts
- 13. Follow a complete miniature ranking example
- 14. Inspect every candidate before teaching it
- 15. Separate ranking stability from linguistic correctness
- 16. Check dependence across documents
- 17. Compare a full corpus and a downsample without assuming equivalence
- 18. Check alternative association measures separately
- 19. Use a stability check before making a claim
- 20. Turn rank movement into a teaching opportunity
- 21. Version the corpus and query to make comparisons possible
- 22. Interpret ranks without invented certainty
- 23. Finish with a nine-part stability audit
CHAPTER 1 OF 23 · LANGUAGE EVIDENCE
1. Begin with the word partnership and its purpose
A collocation list is useful only if its target and purpose are known. A student choosing natural wording for an essay may need common, clear examples; a linguist exploring rare fixed expressions may need a different ranking. Stability asks whether a chosen analysis preserves the partners and ordering that matter for this task.
See it in context
An instructor might investigate strong before nouns in school reports. The selected corpus, adjacency rule and focus on noun partners should be stated before the ranking. Without those, the top item cannot be interpreted.
The mistake to recognise
The shortcut treats any list sorted from one interface as a dictionary of universally preferred combinations. It may instead reflect a narrow genre or a statistical measure emphasising unusual rather than common pairs.
A safer method
Define the node, candidate grammatical class, context, data source and ranking metric. Only then can a change in ranking be meaningfully evaluated.
Your check: Does a high-ranked pair automatically belong in every writing lesson?
Explained answer: No. The purpose, meaning, register and enough authentic examples must be considered.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 2 OF 23 · LANGUAGE EVIDENCE
2. Recognise that a ranking is an output of choices
A rank is not stored inside a word. It is computed from observed data by a procedure. Change the corpus size, tokenisation, part-of-speech filter or measure and the rank may change without any error in the arithmetic.
See it in context
Two sources can both contain strong evidence and strong coffee. A scientific report collection may favour evidence; a café review collection may favour coffee. The difference reflects communicative domains, not a contradiction.
The mistake to recognise
A weak claim says ‘the computer has discovered English’s favourite phrase’ without naming the sampled genre or scoring procedure. It confuses a contextual measurement with a universal lexical fact.
A safer method
Write the result as ‘highest-ranked by [metric] in [corpus] under [query]’ rather than a bare absolute superlative. Retain the original counts.
Your check: Why can a rank change even when both tools are functioning correctly?
Explained answer: They may use different eligible texts or ranking definitions, and those choices change the data or statistic.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 3 OF 23 · LANGUAGE EVIDENCE
3. Separate raw pair count from association strength
Raw co-occurrence frequency counts eligible pair events. Association statistics consider other quantities, such as how often each word occurs separately. A common pair can be frequent but not unusually selective; a rare pair can receive a high association score.
See it in context
A constructed list may contain twelve strong–coffee events and five strong–signal events. An MI-based measure might prefer the rarer signal pair when its expected count is especially small, while a frequency ranking prefers coffee.
The mistake to recognise
The false shortcut calls an MI score ‘the number of times the phrase appeared’. A change from raw count ranking to MI ranking compares different properties, not necessarily a disagreement about the observed examples.
A safer method
Label the metric in every table. Analyse ranking stability within a fixed measure first; compare different measures deliberately as a separate question.
Your check: If one pair has more observed hits but another has higher MI, must one result be wrong?
Explained answer: No. They rank different quantitative properties, and their interpretation should keep that difference visible.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 4 OF 23 · LANGUAGE EVIDENCE
4. Define the units of the sample
Sampling complete documents, paragraphs, sentences or isolated concordance lines gives different opportunities for collocation. A small fragment may omit a word’s usual partners or change the marginal counts needed by an association formula.
See it in context
An archive of school reports can be split by complete report, by page or by sentence. A word whose partners appear in introductions may be overrepresented when sampled by the first page of each report.
The mistake to recognise
The error treats a thousand independently sampled lines as equivalent to a thousand independent writers. Many lines may come from one source and therefore share language habits.
A safer method
Record document IDs, sampling unit and sample size. For some analyses it is preferable to sample complete documents and inspect how token totals vary.
Your check: What is lost when source IDs are discarded?
Explained answer: The ability to detect clustering, duplicated passages or domination by a few documents.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 5 OF 23 · LANGUAGE EVIDENCE
5. Understand downsampling
Downsampling selects a smaller portion of a larger corpus for efficiency or qualitative inspection. It can be useful, but it changes the data available for ranking. Evidence suggests downsampled collocation lists can resemble full-corpus results in many cases while diverging substantially for some target words.
See it in context
A researcher extracts 10 percent of a large archive to inspect examples. If the extract happens to contain fewer scientific texts, strong evidence may move down relative to everyday descriptions.
The mistake to recognise
The mistake assumes that because a downsample is convenient and often broadly similar, its top ten collocates will always reproduce the full corpus ranking. That requires empirical checking, not faith.
A safer method
Compare rankings from the full corpus and appropriately sampled subsets when feasible. Report target-specific variation rather than relying on one average across all words.
Your check: Does a 10 percent sample necessarily preserve the same top three?
Explained answer: No. Ranking overlap depends on the target word, source distribution, sample design and metric.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 6 OF 23 · LANGUAGE EVIDENCE
6. Keep sample size and source diversity distinct
A larger token count can improve the chance of observing rare collocations, yet many tokens from one repetitive source are not equivalent to a diverse sample of text types. The intended population determines what suitable coverage means.
See it in context
A hundred thousand tokens of café menus may provide many examples of strong coffee but little evidence about academic writing. A smaller, well-targeted archive of essays may better answer a school composition question.
The mistake to recognise
The wrong repair for unstable rankings is simply ‘add more text’ without examining whether the new text resembles the domain the claim is supposed to represent.
A safer method
Track source count, text length, genre and duplication. Add relevant material, not just any convenient words, and compare whether the important partners persist.
Your check: Can adding unrelated material make a ranking less useful for teaching?
Explained answer: Yes. It can shift the mixture away from the communication task that originally motivated the analysis.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 7 OF 23 · LANGUAGE EVIDENCE
7. Control for genre composition
Partners of the same adjective can vary across domains. Scientific prose, narrative description, news and conversations use different subjects and meanings. A ranking drawn from a mixed corpus often reflects its blend.
See it in context
A corpus with 70 percent scientific explanation may rank strong evidence above strong coffee; an evenly mixed general-life archive may show a different ordering. Those results need not conflict about grammaticality.
The mistake to recognise
The mistake labels a genre difference a sudden vocabulary change while ignoring that one sample contains different source proportions.
A safer method
Compare within a common genre or report genre-specific rankings before constructing an overall list. If an overall mix is needed, define the target shares.
Your check: What should be checked before comparing rankings from two schools?
Explained answer: Whether the sampled text types, writers, dates, task prompts and processing are comparable.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 8 OF 23 · LANGUAGE EVIDENCE
8. Distinguish a stable list from a stable order
The same five candidate partners may appear in both samples but occupy different positions. List-membership stability and rank-order stability are therefore different properties. A teaching decision may only need the examples to remain useful, not every exact rank.
See it in context
Imagine evidence, coffee and support remain among the top three, but evidence moves from first to third. The membership is unchanged although the order shifts. A measure of set overlap would report full overlap while an order-sensitive measure would not.
The mistake to recognise
A reader reports ‘100 percent stable rankings’ merely because the same candidate names appear. That ignores changed ordering and potentially large changes in frequency.
A safer method
State whether the comparison concerns top-k membership, order, scores or practical usefulness. Choose a measure that corresponds to the actual decision.
Your check: Do identical top-three members imply identical ranking?
Explained answer: No. Set membership can remain the same while the order changes.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 9 OF 23 · LANGUAGE EVIDENCE
9. Use top-k overlap as a simple first check
Top-k overlap counts the candidates shared by two ranked top-k lists. For two top-three lists, there are at most three shared items. This makes a transparent classroom introduction before more elaborate rank-correlation measures.
See it in context
Suppose Sample A has evidence, coffee and support; Sample B has coffee, evidence and argument. Two members—evidence and coffee—are shared. Top-three overlap is 2/3, approximately 66.7 percent when expressed relative to either list’s size.
The mistake to recognise
The mistake claims the overlap percentage is a statistical confidence level or a probability that the top item is true. It is simply a description of the two displayed sets.
A safer method
Show both lists, the shared items and k. Keep the sample definition alongside the overlap value so readers know what was compared.
Your check: What is top-three overlap when two of three candidates are shared?
Explained answer: Two shared items out of three positions, or about 66.7 percent by the stated definition.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 10 OF 23 · LANGUAGE EVIDENCE
10. Calculate Jaccard similarity correctly
Jaccard similarity compares sets by dividing the size of their intersection by the size of their union. It ignores rank order. For two top-three sets sharing two items, their union has four distinct members, so the Jaccard value is two divided by four.
See it in context
In the constructed example, A={evidence,coffee,support} and B={coffee,evidence,argument}. The intersection is {evidence,coffee}, size two. The union adds support and argument, size four. Jaccard=0.5.
The mistake to recognise
A common error divides two shared items by six combined list positions, incorrectly failing to remove duplicates from the union. Another calls 0.5 a measure of matching rank order, which it is not.
A safer method
Write the two sets explicitly and count unique union members. Use a rank-aware measure when the order itself is central to the research question.
Your check: What is the Jaccard value for the displayed top-three sets?
Explained answer: 2/4=0.5. It describes membership overlap, not agreement on first, second and third place.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 11 OF 23 · LANGUAGE EVIDENCE
11. Recognise that rank position needs its own measure
Rank-order correlation and rank displacement can capture aspects of ordering that set overlap misses. Kendall’s tau, Spearman correlation and related measures have assumptions about common candidates and ties. The correct choice depends on the comparison.
See it in context
If strong coffee moves from second to first while strong evidence moves from first to second, the top-two set is unchanged. A simple overlap score is one, even though the first-place recommendation changed.
The mistake to recognise
The error treats a set metric as a universal quality measure. An interface that presents exactly the same three candidates in reverse order may pass a membership test while altering the suggested priority.
A safer method
For a simple class activity, list each shared candidate’s rank before and after. More formal work can use an appropriate rank-correlation method with documented treatment of ties and missing items.
Your check: Would Jaccard notice two top-three lists that contain the same candidates in reverse order?
Explained answer: No. The sets are identical, so Jaccard is one even though the rankings differ.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 12 OF 23 · LANGUAGE EVIDENCE
12. Understand ties and close counts
Two partners can have nearly equal raw counts or association scores. Small changes can swap their order even if the practical difference is negligible. A ranked list hides the distance between candidates unless the underlying figures are reported.
See it in context
If evidence has fourteen matches and coffee eleven in one equal-sized sample, then evidence leads by three. Another sample could have coffee sixteen and evidence fifteen. That changes first place despite both remaining common in those samples.
The mistake to recognise
The wrong response announces a dramatic preference reversal without observing that the counts are close and uncertain. A one-place movement may be less informative than the size of the difference.
A safer method
Show raw pair counts, the score definition and, when appropriate, uncertainty or replication. Consider whether the teaching decision materially changes with one place of rank.
Your check: Does moving from first to second always mean a big linguistic change?
Explained answer: No. Rankings can swap around small score differences without indicating a large or stable population difference.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 13 OF 23 · LANGUAGE EVIDENCE
13. Follow a complete miniature ranking example
Our teaching table uses two fictional, nonoverlapping 20,000-token samples from the same intended domain and identical immediate adjective–noun query settings. We count six constructed strong+noun pairs in each. The samples are invented to demonstrate arithmetic and ranking differences.
See it in context
In Sample A the top counts are evidence fourteen, coffee eleven and support ten. In Sample B the top are coffee sixteen, evidence fifteen and argument twelve. The top-three lists share evidence and coffee but differ at the third position.
The mistake to recognise
The misleading headline says real English shifted from evidence to coffee based on these imaginary numbers. Even with real corpora, sample differences could arise from source mix or chance rather than historical change.
A safer method
Report both tables, identical source rules and the observed top-three sets; calculate overlap 2/3 and Jaccard 0.5. Interpret the result as a property of these two samples only.
Your check: Which item leaves the top three between the two invented samples?
Explained answer: Support leaves and argument enters. Evidence and coffee remain but swap their first/second order.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 14 OF 23 · LANGUAGE EVIDENCE
14. Inspect every candidate before teaching it
A ranking can contain typing errors, proper names, quotations, unusual metaphors and words that merely occur nearby. A high position is a prompt for concordance inspection, not a complete proof of natural phrase usage.
See it in context
A class sees strong signal near the top of a technical corpus list. In some contexts this is a natural expression, but its meaning differs from strong evidence and strong coffee. Students need full sentences rather than isolated pair counts.
The mistake to recognise
The error teaches every ranked two-word item as a ready-made phrase without checking its grammatical relation, meaning or register.
A safer method
Open concordance examples for the candidates, identify the relationship and select expressions relevant to the learner’s task. Keep score and context distinct.
Your check: What extra evidence is needed after a pair ranks highly?
Explained answer: Authentic contexts, grammatical interpretation and relevance to the intended writing or reading domain.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 15 OF 23 · LANGUAGE EVIDENCE
15. Separate ranking stability from linguistic correctness
A collocation may appear consistently in many samples and still be inappropriate in a particular sentence. Stable usage in one genre does not establish universal grammatical suitability; conversely, a rare correct expression can rank unstably.
See it in context
A learner choosing heavy rain rather than powerful rain needs evidence about conventional adjective–noun use in that context. A technical corpus ranking of a different construction is not enough to settle the choice.
The mistake to recognise
The mistake equates ‘stable high rank’ with ‘always correct’. Statistical repeatability and semantic appropriateness are separate questions.
A safer method
Inspect target meaning, audience, genre and contextual fit. Use dictionaries and attested sentences alongside numerical corpus evidence.
Your check: Can a rare but legitimate expression be unstable in small samples?
Explained answer: Yes. Its few occurrences make it sensitive to which texts were selected.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 16 OF 23 · LANGUAGE EVIDENCE
16. Check dependence across documents
Collocation events within one document may not be independent. A repeated rubric or copied speech can contribute many hits to one pair, creating an apparently stable ranking across overlapping extracts that reuse the same source.
See it in context
If strong evidence appears in a standard assessment rubric printed on every worksheet, the phrase may dominate both Sample A and Sample B although those samples share the same template origin.
The mistake to recognise
The false interpretation calls shared copied material independent confirmation. Overlap in original sources can conceal rather than reduce uncertainty.
A safer method
Keep source IDs, check deduplication and compare results after withholding dominant documents. Define what counts as an independent source under the question.
Your check: Do repeated worksheet templates guarantee independent popularity?
Explained answer: No. They can be one linguistic decision multiplied across documents.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 17 OF 23 · LANGUAGE EVIDENCE
17. Compare a full corpus and a downsample without assuming equivalence
A smaller concordance sample may be convenient for qualitative reading yet insufficient to reproduce the full-corpus rank order. Research on downsampled collocation corpora reports meaningful word-specific variation even where the overall agreement is often moderate to high.
See it in context
A teacher downloads fifty examples from a larger archive and ranks candidate partners there. That selection can be useful for learning, but its top three should not automatically be described as the full archive’s top three.
The mistake to recognise
The mistake uses a convenience subset as though every count and marginal frequency still represented the full corpus. Association statistics may change when the relevant denominator and candidate frequencies change.
A safer method
Where possible, compute full-corpus statistics separately from sample-based qualitative examples. State what was sampled and compare rankings before treating the smaller set as an adequate proxy.
Your check: Is a concordance sample always a valid stand-in for full-corpus collocation statistics?
Explained answer: No. It should be tested for the particular target and metric, especially when rare or domain-specific pairs matter.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 18 OF 23 · LANGUAGE EVIDENCE
18. Check alternative association measures separately
Pointwise mutual information, logDice, t-score and other measures weight pair frequency and expectation differently. If one method changes the top ten, that is not necessarily sampling instability; it is a change in the property being ranked.
See it in context
A rare pair may lead an MI list while a frequent pair leads a t-score list. This can occur on exactly the same corpus data. Comparing the two rankings can still be informative about different notions of association.
The mistake to recognise
The error calls one measure unreliable solely because it disagrees with another without stating which aspect of collocation the task wants to prioritise.
A safer method
Treat change of measure as a deliberate secondary analysis and consult each measure’s definition. For ranking reproducibility, first hold the metric constant while varying the sample.
Your check: When is disagreement due to metric choice rather than sample?
Explained answer: When the corpus and query stay fixed but the scoring formula changes, producing a different ranking.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 19 OF 23 · LANGUAGE EVIDENCE
19. Use a stability check before making a claim
A simple robustness experiment draws several defensible subsets from the same target domain, applies identical queries and ranks, then compares top-k membership, rank movement and qualitative context. It does not prove absolute stability, but it shows sensitivity to sampling decisions.
See it in context
For a classroom exercise, two nonoverlapping groups of short constructed texts can produce rankings under the same rule. Students record which pairs remain common and which shift. A third group can test a prediction.
The mistake to recognise
The temptation is to repeat a search until the favourite phrase wins and then publish that run. That is selection of a flattering result, not stability testing.
A safer method
Define the sampling procedure and comparison criterion in advance where possible. Keep all attempted runs and report variation, not just the neatest outcome.
Your check: What is the smallest honest product of a stability check?
Explained answer: A comparison of rankings under clearly documented, defensible samples plus a conclusion limited to that comparison.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 20 OF 23 · LANGUAGE EVIDENCE
20. Turn rank movement into a teaching opportunity
If students see partners change between genres, they can learn that word choice is contextual rather than arbitrary. This supports better academic and everyday writing because learners see why some phrases fit one communicative purpose better than another.
See it in context
Strong evidence suits an argumentative paragraph; strong coffee suits a description of taste; strong signal suits many technical explanations. A teacher can use rankings as prompts to explore sense and setting.
The mistake to recognise
An unproductive lesson makes learners memorise rank positions as though first place is a permanent vocabulary fact. The educational value lies in appropriate selection, not the number itself.
A safer method
Ask learners to write one sentence for each partner, then explain why transferring the noun to a different context may change the adjective’s interpretation.
Your check: Which skill matters more than recalling the top-ranked partner?
Explained answer: Choosing and explaining a natural word partnership in the learner’s intended sentence.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 21 OF 23 · LANGUAGE EVIDENCE
21. Version the corpus and query to make comparisons possible
Two rankings can change because the source archive was updated, duplicates were removed or the tool changed its tokeniser. A reproducible result needs a snapshot of the data and operational settings.
See it in context
Suppose a school archive gains twenty new scientific reports. A new ranking can shift without any individual author changing their vocabulary choice. The corpus mixture has changed, and the report should say so.
The mistake to recognise
The mistake labels an analysis made in November as a direct replication of an October result while its underlying text collection and matching algorithm differ.
A safer method
Store corpus version, source selection, date, tokenisation, node and candidate definitions, window, statistic and frequency cutoffs. If these change, explain the difference explicitly.
Your check: Can two rankings be directly compared when one has a different dataset and undocumented preprocessing?
Explained answer: Only with major caveats. The sources and procedures must be aligned or the differences transparently accounted for.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 22 OF 23 · LANGUAGE EVIDENCE
22. Interpret ranks without invented certainty
A stable ranking may support a practical decision in a defined task, but no small sample or single statistic proves a universal law of language. Uncertainty can arise from sampling, variation across texts and the meaning of the expressions.
See it in context
A report might say ‘strong evidence appeared among the three highest-count adjacent strong+noun pairs in both constructed samples’. That sentence is accurate for the table without implying the phrase is universally top-ranked.
The mistake to recognise
The exaggerated alternative says the expression is the single most natural partner in English because one list ranked it first. It erases source scope, alternative uses and sampling uncertainty.
A safer method
State the observed property, the test conducted and what it cannot establish. Keep external references and sample numbers accessible so the reader can evaluate the strength of evidence.
Your check: What is an appropriately cautious summary of the worked example?
Explained answer: Two constructed equal-sized samples shared two of their top three pairs, with different top positions; no population prevalence claim follows.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
CHAPTER 23 OF 23 · LANGUAGE EVIDENCE
23. Finish with a nine-part stability audit
Before relying on a word-partner list, check the target, corpus domain, unit, sample design, text overlap, search relation, measure, observed counts and comparison criterion. A good audit can fit in a caption if the reasoning is already clear.
See it in context
The final teaching report can show the two six-row count columns and highlight the two shared top-three members. It can note that the calculation is original illustrative material, not a measurement of real English use.
The mistake to recognise
The false conclusion uses the visually striking swap from evidence to coffee to claim students should avoid one of those perfectly legitimate phrases.
A safer method
Choose word partners based on context and evidence; treat ranking stability as one diagnostic of whether an ordering withstands reasonable data changes.
Your check: What makes an unstable ranking still useful?
Explained answer: It can identify the need for more varied evidence and draw attention to genre-sensitive word choices.
When transferring this method, identify the exact target, the set of eligible texts and the operation that produced the result. A reader should be able to reconstruct the choice instead of trusting a headline. If a different but reasonable setting changes the finding, report the change and explain what it teaches; if the setting changes the underlying question, do not pretend the two numbers are direct estimates of the same thing.
Complete worked example with visible assumptions
Here are two entirely fictional, nonoverlapping 20,000-token samples from the same hypothetical target domain under the same exact adjective–noun adjacency counting rule. Each row counts an observed strong+noun pair in the sample. The six candidates are not a full inventory; they serve to show how top-three sets can move.
| Candidate strong + … | Sample A pair count | Sample B pair count | Rank A | Rank B | Interpretive note |
|---|---|---|---|---|---|
| evidence | 14 | 15 | 1 | 2 | Present in both top-three lists |
| coffee | 11 | 16 | 2 | 1 | Present in both, reverses order |
| support | 10 | 7 | 3 | 5 | Leaves top three |
| argument | 8 | 12 | 4 | 3 | Enters top three |
| signal | 6 | 4 | 5 | 6 | Lower observed pair count |
| link | 3 | 5 | 6 | 4 | Lower-ranked in A, higher in B |
Sample A top three are evidence (14), coffee (11) and support (10). Sample B top three are coffee (16), evidence (15) and argument (12). The shared members are evidence and coffee; support leaves and argument enters.
Top-three membership overlap relative to list size equals two shared members divided by three, or 2/3 ≈ 66.7 percent. This number does not describe agreement on order: coffee rises to first and evidence falls to second.
The Jaccard similarity for membership sets A={evidence, coffee, support} and B={coffee, evidence, argument} is size of intersection over size of union = 2/4 = 0.5. The union has four distinct candidates, not six positions.
Both sample corpora were assumed equal in size and used the same pair-count rule, so comparing their raw constructed counts does not create an unequal-token problem in this teaching table. In real analysis, display eligible token totals and document composition; a scored association ranking would require consistent marginal frequencies and a declared statistic.
This example supports a conclusion about the two artificial rankings only. It does not establish that these six expressions have the displayed prevalence in any language community, genre or school.
Twelve situations that test transfer
Each scenario describes a decision a reader or researcher could face. The explanations are not empirical findings about the people named in the examples; they show how to keep the observation and its inference in proportion.
1. Rank swap
Situation. Evidence and coffee exchange first place in two samples.
What could be misleading. A rank change is mistaken for changing English norms.
Better reasoning. Inspect counts, source mix and context first.
Ask next. Are the pair counts close?
2. Identical membership
Situation. Two top-five lists contain the same candidates in different order.
What could be misleading. A set-overlap measure masks rank differences.
Better reasoning. Report membership and order separately.
Ask next. Which rank-aware measure fits the task?
3. Rare one-hit phrase
Situation. A phrase gets high MI from a very low expected count.
What could be misleading. A high score may be unstable across samples.
Better reasoning. Show raw count and validate contexts.
Ask next. Would it survive a larger relevant corpus?
4. Different genres
Situation. Science reports are compared with coffee reviews.
What could be misleading. Genre mixture changes which nouns occur.
Better reasoning. Use task-specific or comparable sources.
Ask next. What is the intended target domain?
5. One dominant author
Situation. Most phrase tokens come from one textbook.
What could be misleading. A ranking may represent one writer’s style.
Better reasoning. Report document spread and check with that source held out.
Ask next. How many independent writers contribute?
6. Duplicated template
Situation. A phrase repeats across one hundred copied forms.
What could be misleading. Copies mimic independent recurrence.
Better reasoning. Audit provenance and deduplicate according to purpose.
Ask next. How many unique authored texts remain?
7. Changed metric
Situation. One ranking uses MI and another raw count.
What could be misleading. Different properties are ranked.
Better reasoning. Keep the same metric for a sampling-stability test.
Ask next. Why might an unusual pair rise under MI?
8. Short concordance download
Situation. Only fifty lines are used from a huge corpus.
What could be misleading. Candidate marginal counts may shift.
Better reasoning. Compute full-corpus statistics when making full-corpus claims.
Ask next. Was the sample intended only for reading examples?
9. Top-k choice
Situation. A report compares top two one day and top ten another.
What could be misleading. Different k values change overlap interpretation.
Better reasoning. Fix the comparison criterion.
Ask next. What is the decision that top-k represents?
10. Grammar mismatch
Situation. One query uses adjacent words; another uses syntactic dependencies.
What could be misleading. Event definitions differ even if the data do not.
Better reasoning. Label relation and candidate restrictions.
Ask next. Are these two counts measuring the same pattern?
11. Regional spelling
Situation. One sample contains more colour, another color.
What could be misleading. Form definition affects candidate identification.
Better reasoning. Apply comparable variant rules or report orthography separately.
Ask next. Is the target a lemma or exact form?
12. Teaching example
Situation. A pupil memorises the top pair but writes it in the wrong sentence.
What could be misleading. Rank is not contextual competence.
Better reasoning. Teach sentence meaning, register and transfer.
Ask next. Can the student explain the partnership?
Eight practice tasks and explained answers
Work each problem before reading the answer. Show the original observations, the selected rule and the interpretation. A useful response can be cautious without being vague: it should say exactly what the supplied evidence establishes.
Practice 1: List the top three
Question. Which pairs lead Sample A and B in the constructed table?
Worked answer. A: evidence, coffee, support. B: coffee, evidence, argument.
Why a shortcut fails. Ranking only by which item appears first omits the membership change.
Practice 2: Count shared candidates
Question. How many top-three pairs appear in both samples?
Worked answer. Two: evidence and coffee.
Why a shortcut fails. Support and argument are different, despite both being legitimate noun candidates.
Practice 3: Calculate top-three overlap
Question. Report shared fraction for the two lists.
Worked answer. 2/3 ≈ 66.7 percent, relative to the three-item top lists.
Why a shortcut fails. The number is a set-overlap description, not statistical confidence.
Practice 4: Calculate Jaccard
Question. Compute the Jaccard similarity for the top-three sets.
Worked answer. Intersection has 2 candidates; union has 4; Jaccard=2/4=0.5.
Why a shortcut fails. Dividing by six fails to remove repeated candidates from the union.
Practice 5: Explain order change
Question. If top-three membership is identical but rank order reverses, what does Jaccard say?
Worked answer. Jaccard=1 because both sets contain identical members. A rank-aware measure is needed to show order reversal.
Why a shortcut fails. Set overlap does not measure rank order.
Practice 6: Interpret a rare MI top rank
Question. A one-hit pair ranks first under MI. What should be checked?
Worked answer. Observed count, expected count, source context, errors and stability across relevant samples.
Why a shortcut fails. MI rank alone cannot establish that the pair is common or suitable for instruction.
Practice 7: Control corpus changes
Question. One sample removes duplicates, another keeps them. Can they be called the same analysis?
Worked answer. Not without explaining changed source preprocessing; compare under compatible deduplication or report the difference as a separate sensitivity test.
Why a shortcut fails. Hidden changes to source independence alter the evidence.
Practice 8: Write a suitable headline
Question. Describe the fictional result without claiming a universal English preference.
Worked answer. ‘Two equal-sized constructed samples shared two of their top three strong+noun pairs, while their first-place ordering changed.’
Why a shortcut fails. ‘English now prefers strong coffee’ is unsupported by the synthetic sample.
Frequently asked questions: collocation ranking stability
What is collocation ranking stability?
The consistency of word-partner ranking membership or order under defined changes to samples or methods.
Does a high rank prove a phrase is the most natural English?
No. Source domain, meaning and measure determine the list.
Can full and downsampled corpora give different partners?
Yes. Research documents variation, even when many rankings resemble the full-corpus results.
What is top-k overlap?
The number of shared candidates in two top-k lists, often expressed as a fraction of k.
What is Jaccard similarity?
Set intersection size divided by union size; it ignores rank order.
Why might rare collocations rank unstably?
Few observed occurrences can be sensitive to which texts are selected and to the association measure.
Is stable membership the same as stable order?
No. The same candidates can appear in different positions.
Can different measures produce different rankings?
Yes. MI, t-score and other statistics emphasise different aspects of co-occurrence.
Does a bigger corpus guarantee stability?
No. It can help represent rare patterns, but source coverage and genre composition still matter.
Why inspect concordance lines?
They reveal grammar, sense, copied material and other context hidden by the score.
How can teachers use stability results?
Choose several well-attested relevant expressions and teach how their meaning changes with context.
Should ranking methods be reported?
Yes. Include target, corpus, sampling unit, window, statistic, counts and comparison criterion.
Can an unstable list be informative?
Yes. It may signal topic variation, insufficient evidence or methodological sensitivity.
Do these constructed counts describe actual English?
No. They are an arithmetic exercise, not an observed linguistic dataset.
Words and concepts in this lesson
- Collocate: A candidate partner found near or grammatically linked to a target.
- Node: The target word for collocation extraction.
- Ranking: An ordering computed from counts or scores.
- Stability: Consistency of results under stated data or method changes.
- Downsampling: Selecting a smaller portion of a larger collection.
- Top-k: The first k ranked items.
- Top-k overlap: Shared membership of two lists relative to k.
- Jaccard similarity: Intersection over union of two sets.
- Rank correlation: A method comparing how similarly candidates are ordered.
- PMI/MI: A logarithmic association measure based on observed relative to expected co-occurrence.
- T-score: A different association measure influenced by observed pair frequency.
- Genre: A type of communicative task that influences words and patterns.
Teach the method through four practical moves
Begin with a situation that requires a decision: choosing a phrase for a school essay, reading a scientific account or checking whether a word comparison is fair. State the decision before introducing its technical label. This puts the world and the communication problem before vocabulary terminology.
Next, show a compact table or a few complete examples. Have learners write down what is observed and which assumptions matter. Ask them to predict one outcome before revealing the result, then reconstruct the calculation, search or classification rather than copying it. Reproduction tests whether the procedure is understood.
Third, introduce one deliberately difficult case: an unseen word, a changed corpus filter, a rare co-occurrence or a contradictory example. Ask a learner to correct the conclusion, not simply apologise for the previous answer. The correction should identify the exact assumption that changed.
Finally, revisit the idea after a delay and transfer it to another subject. For example, a source-selection question in English can become a fairness question about Science experiments or a provenance question in History. A method becomes genuine vocabulary knowledge when the learner can explain and use it outside the original sentence.
Published research and method resources
- Kanner (2025), Collocations in downsampled corpora
- Evert and colleagues (2017), large-scale evaluation of association measures and context sizes
- Su, Gu and Liu (2024), association measures for collocation extraction
- Gries (2022), which properties are measured by association statistics?
- Corpus Frequencies and Collocations — interactive educational lab
External publications support the conceptual guidance. Numerical examples above were independently constructed and do not reproduce their research datasets.
Related vocabulary methods and teaching hubs
- Corpus sampling frames — know what could enter the study
- Collocation association scores — understand what the statistic measures
- Collocation windows — see how direction and distance affect a count
- Subcorpora — compare source and genre groups explicitly
- Corpus deduplication — inspect repeated passages
- Vocabulary Learning Hub
- Vocabulary Article Library
Continue with the complete vocabulary evidence batch
The four guides address distinct questions: how a label is assigned, what a missing search result means, how word-partner rankings behave across samples, and whether a vocabulary finding survives defensible alternative rules.
- Annotation Guidelines — Define labels, boundary cases, review rules and unresolved decisions
- Zero-Hit Corpus Results — Check search failures, genre coverage and rare-word absence
- Corpus Sensitivity Analysis — Test how a defensible changed rule affects vocabulary claims
- Vocabulary Learning Hub — return to reading, meaning, retention and vocabulary OS
- Vocabulary Article Library — follow related vocabulary concepts