Vocabulary | What Is Corpus Deduplication? Why Repeated Text Can Distort Word Frequency

CHAPTER 01 OF 22

1. Understand what a duplicate actually is

In a language corpus, two records can hold the same text, substantially overlapping text or different versions of the same underlying communication. A duplicate is not defined only by matching filenames or page URLs. The relevant notion of sameness depends on the research question.

A concrete example

A message titled ‘Sports Day Reminder’ appears as a website page and an attached PDF. Their titles and file formats differ, but the text may be identical. Two students independently composing ‘Good morning’ are a different kind of repetition.

Where an attractive answer goes wrong

The easy mistake calls every repeated phrase a data error. Some repetitions are genuine linguistic events, while another is merely an accidental copy of the same document. Deduplication must preserve that distinction.

A better decision

Start with the unit and inferential target: unique source texts, versions, independent authors or encounters by readers. Then classify repeated material rather than deleting it automatically.

Check your understanding: Are two files with different names automatically independent evidence?

Explained answer: No. They can contain the same source text. Compare provenance and content before counting them as independent units.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Next chapter

CHAPTER 02 OF 22

2. Decide whether the purpose is wording or exposure

A unique-text corpus aims to avoid counting the same document repeatedly merely because it exists in several storage locations. An exposure corpus may intentionally count every separate delivery to readers. A record of repetition must be interpreted relative to that purpose.

A concrete example

An announcement sent to parents three times is one underlying authored text but three communication events. A researcher of parent exposure may count the three send events; a researcher of lexical variation across authored documents may collapse them.

Where an attractive answer goes wrong

The false assumption says deduplication always improves data quality. Removing legitimate repeated encounters can damage a project that explicitly studies how often readers saw a phrase.

A better decision

Write the purpose above the processing log. For each suspected repeat, ask whether it is a new authored language event, a new exposure event, an archive duplicate or an informative revision.

Check your understanding: Should an email reminder sent again a week later always be discarded?

Explained answer: Not always. For unique wording, it may be a repeated text; for communication exposure, the second send can be a real observation.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 03 OF 22

3. Separate identical files from identical text

Byte-identical files are the easiest duplicates to recognise. But identical text can exist inside HTML, PDF and email bodies with different code and formatting. Conversely, two files with similar names can contain substantially changed content.

A concrete example

A school PDF and webpage show the same five paragraphs; one contains a logo and pagination while the other has navigation menus. The files differ at the byte level, but the core authored passage is the same.

Where an attractive answer goes wrong

The shortcut uses only file hashes and concludes that no duplicates exist. That misses text copied across formats or with changed line breaks, punctuation and boilerplate.

A better decision

Preserve original files, extract comparable text carefully, and decide which formatting changes should be ignored for the question. Record the extraction procedure to make the comparison reproducible.

Check your understanding: Can a byte hash alone find every copied document?

Explained answer: No. It detects byte-identical material, not necessarily identical text rendered in different formats.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 04 OF 22

4. Identify exact text duplicates after extraction

An exact text duplicate matches under a declared text-normalisation procedure, which may remove irrelevant whitespace or normalise encoding. Even this relatively simple classification needs a policy because removing punctuation or case can merge meaningful differences.

A concrete example

Two pages containing the same notice differ only because one uses Windows line endings and the other uses Unix line endings. A normalised text comparison can detect that they communicate the same message.

Where an attractive answer goes wrong

A dangerous method strips every mark and lowercases everything before comparing, accidentally merging a correction that changes ‘can’ to ‘cannot’ or different proper names that matter to the claim.

A better decision

Use conservative normalisation and preserve the raw version. Test a few apparently matching pairs manually, especially where a one-word change could reverse the meaning.

Check your understanding: Should ‘Students may enter’ and ‘Students may not enter’ be considered identical?

Explained answer: No. The added negation changes the meaning and must not disappear under normalisation.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 05 OF 22

5. Understand near-duplicates

Near-duplicates are similar but not completely identical passages. They can arise from updated articles, copied templates, translated abstracts, repeated introductions or edits correcting an error. Whether they should be joined is not decided by a single universal similarity percentage.

A concrete example

Version 1 of a field-trip notice says ‘return the form by Monday’; version 2 says ‘return the form by Wednesday’. Nearly all tokens match, but the revised deadline changes a real instruction.

Where an attractive answer goes wrong

An aggressive similarity filter might remove the corrected version and leave readers with obsolete information. A relaxed filter might count dozens of copied versions as independent authors.

A better decision

Use similarity measures to flag candidate pairs, then review meaningful differences. Keep a version relationship and explain whether the analytic task treats revisions as separate events.

Check your understanding: Does 95% text similarity prove the remaining 5% is unimportant?

Explained answer: No. A small change may alter a date, quantity, negation or legal condition. Similarity is a detection cue, not a final editorial judgement.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 06 OF 22

6. Recognise syndication and copied reporting

A news article or announcement may be republished through several outlets with little wording change. This can inflate the apparent prevalence of distinctive expressions even if those outlets did not independently compose them.

A concrete example

A press release is quoted across five sites. A corpus of webpages may then contain the phrase ‘substantial improvement’ five times. For a unique-authoring study, those instances likely share a common original text.

Where an attractive answer goes wrong

The mistake interprets five web locations as five independent witnesses to a popular phrase without checking the common source. Web distribution is not equivalent to lexical invention.

A better decision

Track original source and republication links when possible. For audience exposure, count delivery sites separately and label the result accordingly. For independent usage, flag syndication clusters.

Check your understanding: Would five copies of one agency report demonstrate five unrelated writers favour a phrase?

Explained answer: No. They may all reproduce a single underlying authored choice.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 07 OF 22

7. Treat boilerplate as a separate mechanism

Boilerplate is recurring text such as navigation menus, footers, consent notices, institutional disclaimers and repeated article templates. It can appear across hundreds of documents without being the central language of those documents.

A concrete example

A set of school pages ends with ‘Contact the school office for further assistance’. If that sentence appears on every page, it can dominate a word list even when the main page texts are diverse.

Where an attractive answer goes wrong

The mistaken approach removes all repeated language indiscriminately, including real slogans, ritual formulas or text whose recurrence is the actual subject of research.

A better decision

Specify which boilerplate regions are excluded from the running-text analysis. Retain raw copies and, if relevant, analyse boilerplate as its own subcorpus rather than pretending it never existed.

Check your understanding: Is a frequently repeated disclaimer always an error?

Explained answer: No. It may be genuine language exposure or an institutional convention. The analytical purpose decides whether and how to include it.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 08 OF 22

8. See why frequent n-grams reveal copying

An n-gram is a sequence of adjacent units. A long identical string occurring in many documents can be a clue that text was copied or reproduced. It can also be a legitimate quotation or standard formula, so interpretation still requires review.

A concrete example

If an eleven-word sentence appears verbatim in dozens of unrelated-looking pages, the pattern is worth investigating. In some cases it may belong to a common footer rather than the pages’ substantive content.

Where an attractive answer goes wrong

A naive algorithm calls every common phrase a duplicated article. Short n-grams such as ‘as a result of’ recur naturally and cannot identify document duplication on their own.

A better decision

Use appropriately long repeated spans as candidate indicators, then check surrounding text, source and document-level similarity. Document the threshold, because short and long spans have different false-positive risks.

Check your understanding: Can a shared four-word phrase prove two texts are duplicates?

Explained answer: No. Many legitimate expressions recur. Longer repeated passages and provenance offer stronger evidence.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 09 OF 22

9. Know what a similarity score does and does not do

A document-similarity measure can rank candidates according to shared shingles, tokens or other features. The score is calculated under a model. It does not reveal why texts match or whether a revision is substantively significant.

A concrete example

Two school notices may share 200 template words and have different one-line instructions. A similarity algorithm could rate them highly alike even though the central content differs. That may still be useful for candidate detection.

Where an attractive answer goes wrong

The error treats 0.95 similarity as a categorical truth that the documents are duplicates for every purpose. It is a numerical approximation based on a selected representation.

A better decision

Write down tokenisation, shingle size, similarity metric and candidate threshold. Review edge cases and error rates in a sample. Maintain human-verifiable source relationships.

Check your understanding: Does an algorithm’s similarity score determine what counts as a new communication event?

Explained answer: No. It indicates a text-pattern relation. Communication provenance and research purpose determine analytic identity.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 10 OF 22

10. Avoid false-positive deletions

A false positive occurs when a deduplication rule flags two meaningfully distinct items as the same. Such a mistake can erase rare variants, corrections, alternative viewpoints and genuinely different writers.

A concrete example

Two emergency messages share the same template but one says ‘do not use the stairs’ while another says ‘use the stairs’. Removing one after a coarse similarity test would erase a crucial distinction.

Where an attractive answer goes wrong

A wrong shortcut equates near-identical formatting with equivalent communicative meaning. The result can be a cleaner-looking but less faithful corpus.

A better decision

Audit pairs containing changed dates, numbers, negation, locations or named entities. Keep a review queue for cases where a small difference could change meaning or study conclusions.

Check your understanding: Why can one changed word matter more than hundreds of shared words?

Explained answer: Because meaning and truth conditions are not proportional to the number of unchanged tokens.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 11 OF 22

11. Avoid false-negative duplicates

A false negative occurs when repeated material is not flagged. Copies may evade detection through formatting conversion, rearranged paragraphs, quotations embedded in longer pages or minor punctuation changes.

A concrete example

A report copied into a newsletter may have a new introduction and heading but preserve eight pages of the original content. A full-document hash will not detect that overlap.

Where an attractive answer goes wrong

A lax filter treats every independently hosted page as a fresh text and may count large reused passages repeatedly, inflating the distribution of words concentrated in that passage.

A better decision

Use multiple cues: exact hashes, long repeated spans, near-duplicate similarity and source metadata. Review a sample from the unflagged collection where duplication risk is high.

Check your understanding: Could a text be mostly copied even if its document hash is unique?

Explained answer: Yes. A new introduction or formatting layer can change the file hash while most prose remains shared.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 12 OF 22

12. Count documents, passages and tokens separately

Deduplication can operate at the level of whole documents, paragraphs or individual repeated spans. A corpus may keep both documents but remove shared boilerplate from token counts, or collapse entire copies into one source record. These strategies answer different questions.

A concrete example

Two reports each contain a common legal disclaimer and distinct substantive sections. Removing the whole second report would lose unique language; excluding the repeated disclaimer for a topic-word analysis may be more appropriate.

Where an attractive answer goes wrong

The error applies one all-or-nothing duplicate label to documents that contain both shared and original sections. This treats text reuse as a binary property even when overlap is partial.

A better decision

Decide whether the analytic unit is document, authored passage or token exposure. Preserve segment-level provenance where needed so readers can see what was shared and what remained unique.

Check your understanding: Should two partially overlapping documents always be merged into one?

Explained answer: No. Segment-level handling can retain genuinely different content while controlling repeated material.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 13 OF 22

13. Build a transparent duplicate register

A review table should record record ID, original source, suspected relationship, exact or near match, action and reason. Such a register helps another person reproduce why some copies contribute to counts and others do not.

A concrete example

Our fictional register includes an exact duplicate of record A and a revised version of record B. Removing the exact copy changes the total count; retaining the revised version recognises a changed communication event under the example’s chosen rule.

Where an attractive answer goes wrong

A weak report announces that the corpus has been ‘cleaned’ without recording what was removed. Later researchers may be unable to test whether the cleaning altered a vocabulary finding.

A better decision

Keep separate fields for observed text length, target hits and the decision to include, exclude or retain as a version. Publish non-sensitive aggregate details and preserve private records securely.

Check your understanding: What information makes a duplicate decision auditable?

Explained answer: Stable IDs, source relationships, similarity evidence, inclusion rule, reviewer decision and any resulting change in counts.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 14 OF 22

14. Calculate the raw before-and-after difference

The worked register contains five records: A, an exact copy of A, B, C and a revised B. Before deduplication, the illustrative set contains 1,820 eligible tokens and sixteen target matches. Removing only the exact copy yields 1,420 tokens and twelve matches.

A concrete example

The before rate is 16/1,820×1,000,000≈8,791.21 per million. The after rate is 12/1,420×1,000,000≈8,450.70 per million. Both numerator and denominator changed, so the rate moves less dramatically than the raw total.

Where an attractive answer goes wrong

The common mistake subtracts duplicate matches from the numerator but leaves duplicate tokens in the denominator. That creates a rate that refers to no consistent processed corpus.

A better decision

Recompute denominator and numerator together. Record the unique-text policy that justifies removing the copy, and do not treat the constructed numbers as real frequencies for English.

Check your understanding: Does removing four duplicate hits necessarily divide the normalised rate by 16/12?

Explained answer: No. The denominator also shrinks, so the new rate is calculated from both updated counts.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 15 OF 22

15. Know why deduplication may increase some rates

Removing duplicated text does not always reduce a word’s normalised frequency. If the copied material contains relatively few matches of a chosen term, removing it can raise the share of that term in the remaining corpus.

A concrete example

Suppose an archive has two hundred repeated footer tokens and no occurrences of the target word. Deleting those tokens leaves the numerator unchanged and reduces the denominator. The per-token rate therefore increases.

Where an attractive answer goes wrong

A superficial story claims deduplication always makes every frequency lower. It assumes the deleted material has the same rate as the rest of the collection, which is not generally true.

A better decision

Use the equation count divided by eligible tokens to check direction. Report both counts and token totals rather than predicting a universal effect.

Check your understanding: What happens if duplicate removal leaves matches unchanged but reduces eligible tokens?

Explained answer: The normalised rate increases, because the same numerator is divided by a smaller denominator.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 16 OF 22

16. Distinguish source independence from repeated exposure

A replicated notice may be encountered by many readers, so it can be important for comprehension or communication studies. But repeated copies do not supply independent evidence about how many distinct authors chose the phrase.

A concrete example

A public-health slogan printed on many posters may be widespread as an exposure event despite being written once. An exposure study might count posters; a study of original lexical choices might count one slogan and its authorship.

Where an attractive answer goes wrong

The error assumes that because a phrase appears on ten signs, ten different people spontaneously selected the wording. Publication channels and authorship remain distinct variables.

A better decision

Record dissemination events and unique authored texts as separate fields if both matter. This gives researchers and educators richer evidence without forcing incompatible goals into one count.

Check your understanding: Can a phrase be common in public exposure but rare in independent authoring?

Explained answer: Yes. A widely repeated slogan is a clear example.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 17 OF 22

17. Keep revised versions when the question requires them

Updates may encode meaningful changes in circumstances, wording, instructions or policies. A corpus tracking how language changes across versions should retain each published version as a distinct event while linking the common lineage.

A concrete example

An event notice changes from ‘Friday’ to ‘Thursday’, or from ‘recommended’ to ‘required’. The documents are near-identical, but the revision matters for anyone studying modals or instructions.

Where an attractive answer goes wrong

A careless deduplication step discards all but the earliest or longest version, potentially preserving a superseded or false statement and erasing a meaningful language shift.

A better decision

Create version IDs and timestamps; mark unchanged passages where useful. State whether the analysis follows original authorship, final approved text or all public revisions.

Check your understanding: Is the latest version always the correct one to use for a historical language study?

Explained answer: Not necessarily. Earlier versions may be central evidence of the change being studied.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 18 OF 22

18. Prevent train-test contamination

In computational evaluation, duplicated or near-duplicated texts can occur in both model training and test sets. This may inflate apparent generalisation because the system is evaluated on language closely resembling material it has already seen.

A concrete example

A classroom analogy is to teach students the exact eight example sentences and then use those same sentences to claim performance on previously unseen vocabulary. The test no longer measures transfer to new items.

Where an attractive answer goes wrong

A simplistic fix checks only exact duplicate documents and overlooks near-identical examples or shared answer templates. The amount of overlap relevant to a model depends on the task.

A better decision

For evaluations, separate data appropriately, check overlap at relevant granularities and report the splitting and duplicate-handling policy. Do not claim an independent test if its examples were used in development.

Check your understanding: Why is a correct score on repeated training examples different from a score on unseen examples?

Explained answer: The first can reflect memory of observed items; the second better tests generalisation under an appropriate independent evaluation design.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 19 OF 22

19. Respect copyright, permissions and privacy

A corpus can include publicly readable text without unrestricted redistribution rights. Deduplication does not change ownership or confidentiality. Some copied passages carry private names or other details that should not be exposed in a public sample.

A concrete example

School records may contain a repeated standard notice with student names inserted in different places. A similarity system can flag the shared template, but publishing raw examples would still disclose identifiable information.

Where an attractive answer goes wrong

The misconception says removing duplicates makes a collection automatically safe and lawful to share. Technical cleaning does not determine consent, copyright permission or protection of minors.

A better decision

Minimise protected information, honour licence terms, restrict sensitive source access and publish synthetic teaching cases or lawful aggregates. Maintain enough internal provenance to permit authorised checking without unnecessary disclosure.

Check your understanding: Does deleting duplicated text remove confidentiality obligations?

Explained answer: No. The remaining records can still contain personal or protected material.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 20 OF 22

20. Apply deduplication to vocabulary teaching

Teachers sometimes reuse example sentences deliberately to strengthen retrieval and fluency. That is not a flaw in pedagogy. The role of deduplication is to clarify what a count represents, not to forbid repetition wherever it appears.

A concrete example

If the same example of ‘justify’ is printed on five worksheets, a learner may benefit from seeing it repeatedly. A vocabulary researcher must nevertheless distinguish five encounters from five independent examples of natural language use.

Where an attractive answer goes wrong

The mistaken lesson tells students that repetition is always bad because a corpus researcher removes duplicates. Repeated practice and statistical independence address different questions.

A better decision

Explain the purpose before the rule. For teaching, preserve purposeful repetition and gradually introduce varied contexts; for counting independent language examples, identify repeated source material explicitly.

Check your understanding: Can the same sentence be valuable in teaching and problematic in a corpus estimate?

Explained answer: Yes. Repetition may support learning while distorting a claim about independent lexical choices.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 21 OF 22

21. Compare deduplication versions openly

Cleaning decisions change both the dataset and the conclusion. A responsible project versions the raw archive, extracted text, duplicate clusters and final analytic set. It also saves the exact query used to calculate results.

A concrete example

After one round of cleaning, a teacher notices that a file was incorrectly treated as a duplicate because of a changed negative instruction. The next version should restore it, recompute rates and document the correction.

Where an attractive answer goes wrong

The common failure silently overwrites the corpus and leaves old charts online without version labels. Later readers cannot explain differences between numbers presented as results of the same collection.

A better decision

Maintain a change log showing source IDs added, removed or restored, reason, affected token totals and revised word counts. A correction is evidence of responsible review, not necessarily project failure.

Check your understanding: Why preserve an earlier rate after a corrected deduplication policy?

Explained answer: So readers can understand the change, reproduce the correction and avoid mistaking data-cleaning effects for a language trend.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Next chapter

CHAPTER 22 OF 22

22. Audit the final claim, not just the file count

The final question is not ‘did the algorithm remove enough files?’ but ‘does the remaining evidence answer the declared question?’ An excellent deduplication process can still yield a poorly sampled corpus, and an exposure study may rightly retain repetitions.

A concrete example

A defensible sentence could say: ‘After excluding one exact archive copy, this constructed five-record example contains twelve target matches among 1,420 eligible tokens; the revised notice is retained as a separate version.’ Every component can be checked.

Where an attractive answer goes wrong

The overclaim says ‘this proves the phrase is less common in schools after deduplication’. There are no real school measurements here, and a change in processing policy is not evidence of a real-world decline.

A better decision

Check unit, purpose, duplicates, versions, exclusions, counts, denominators, independent sources and uncertainty. Publish both what was observed and the limit on generalisation.

Check your understanding: What is the clearest sign that deduplication supported a trustworthy vocabulary conclusion?

Explained answer: The report makes the retained units, reasons, before-and-after counts and resulting scope explicit.

This distinction is most useful when it changes what the reader can safely conclude. Write down the source or counting rule you would need to inspect, say what remains unknown, and resist upgrading a local observation into a claim about all writers, students or texts. Where a method offers several reasonable choices, keep those choices visible rather than presenting one decision as a universal rule.

Contents · Previous chapter · Worked dataset

Worked dataset: inspect every assumption

This five-record register is invented to illustrate a unique-text analysis that retains substantive revisions as separate communication versions. Record A-copy duplicates A exactly; B-revised changes a substantive instruction and is retained under this declared version-aware policy. The target word count is fictional and used only for arithmetic.

Record	Eligible tokens	Target hits	Relationship	Action
A	400	4	Original unique notice	Retain
A-copy	400	4	Exact text copy of A	Exclude from unique-text count
B	200	2	Original short message	Retain
C	600	3	Independent report	Retain
B-revised	220	3	Edited version of B, meaningful update	Retain as separate version
Before removal	1,820	16	Five supplied records	Uncleaned total
After exact copy removal	1,420	12	Four retained version-aware records	Cleaned total

Before removal, total eligible tokens are 400+400+200+600+220=1,820 and target hits are 4+4+2+3+3=16. The raw rate is 16/1,820×1,000,000≈8,791.21 per million tokens. The pair count and denominator are internally consistent, though the example intentionally contains a duplicate.

A-copy contributes 400 tokens and four hits but is identical to A. Under the declared unique-text policy, removing this record yields 1,420 tokens and twelve hits; the rate is 12/1,420×1,000,000≈8,450.70 per million. The computation uses the revised denominator rather than recycling 1,820.

B-revised remains because the project counts published substantive versions separately. Another project investigating only unique initial authoring may instead link B and B-revised differently. Neither choice is a mathematical fact independent of research purpose; the objective is to make the chosen policy explicit.

Removing B-revised as well would yield 1,200 tokens and nine hits, or 7,500 per million. This alternative demonstrates that a plausible change in deduplication policy can alter the result. It does not tell us which policy is correct without knowing the question.

In a public report, the register should also name the source lineage, date, author or publisher where lawful, and the checks used to classify exact and near-duplicates. The arithmetic is only one stage of a trustworthy explanation.

Twelve transfer cases: from tidy counts to defensible conclusions

These cases are purposefully varied across school communication, research, workplace writing and general reading. They ask for a decision, not simply a recited definition; each answer is bounded by its supplied evidence.

1. PDF and webpage

Situation. A school notice exists as identical HTML and PDF files.

Hidden issue. Different containers conceal the same authored text.

Reasoned response. Use a unique-text rule and preserve both source locations as provenance.

Further question. Would an exposure study count two delivery channels?

2. One changed negation

Situation. Two pages differ by the word not.

Hidden issue. High similarity hides a crucial semantic difference.

Reasoned response. Retain the changed version or explain its relationship and impact.

Further question. Could a similarity threshold safely delete it without review?

3. Repeating footer

Situation. A standard footer appears in eighty downloaded pages.

Hidden issue. Navigation boilerplate may dominate a running-text count.

Reasoned response. Exclude or analyse separately, consistently with the purpose.

Further question. Are readers’ actual exposures under study?

4. Copied quotation

Situation. Several essays quote the same published sentence.

Hidden issue. The repetition may be appropriate citation, not independent phrasing.

Reasoned response. Separate quoted material from author-generated prose where the task requires.

Further question. Which field identifies the quoted source?

5. Syndicated article

Situation. One news item is reprinted on several sites.

Hidden issue. Distribution appears as independent wording.

Reasoned response. Link common origin and choose a unique-text or exposure policy.

Further question. How many independent writers actually chose the phrase?

6. Minor formatting change

Situation. One message replaces smart quotation marks with plain ones.

Hidden issue. Byte comparison misses identical communicative wording.

Reasoned response. Apply conservative text normalisation and preserve originals.

Further question. Can normalisation accidentally erase meaning?

7. New date

Situation. A reminder is reissued with a new event date.

Hidden issue. Near-duplication and genuine update coexist.

Reasoned response. Record version lineage and decide whether the new event counts.

Further question. Which study question requires both?

8. Two independent authors

Situation. Two students independently write the same sentence.

Hidden issue. Identical string does not mean copied document.

Reasoned response. Check provenance before deduplicating independently authored examples.

Further question. What kind of independence matters to the project?

9. Train-test overlap

Situation. An evaluation file shares examples with training data.

Hidden issue. Performance may reflect remembered instances.

Reasoned response. Check relevant duplicate spans and use an independent test set.

Further question. Would an exact-document check catch paraphrases?

10. Repeated classroom practice

Situation. A sentence is deliberately revisited across lessons.

Hidden issue. The repetition is pedagogically intentional.

Reasoned response. Retain exposure counts for learning analysis; mark repeated sources for diversity claims.

Further question. Is the outcome learning or independent lexical variety?

11. False positive cluster

Situation. Two distinct instructions share a school template.

Hidden issue. An automated similarity score treats them as identical.

Reasoned response. Review changed fields and retain distinct message content.

Further question. Would a human reviewer notice the changed instruction?

12. After-cleaning chart

Situation. A report changes counts after duplicate removal but not the stated method.

Hidden issue. The reader cannot reproduce the new numbers.

Reasoned response. Version the dataset, show excluded IDs and recompute numerator plus denominator.

Further question. What would a transparent caption contain?

Eight practice questions with explanations

Answer each question before opening the explanation. For a numerical question, show the denominator and unit; for an eligibility or annotation question, state the exact boundary of the decision. A correct number with a false generalisation is not a complete answer.

Exercise 1: Name the duplicate

Question. A webpage and an attached PDF contain identical notice wording. For unique authored texts, how many units?

Explained solution. One underlying text, with two representations or locations. Keep both source records linked to one analytic text.

Common wrong turn. Different file formats do not guarantee independent wording.

Exercise 2: Recompute totals

Question. Five records have 1,820 tokens and sixteen hits. Exclude an exact 400-token, four-hit copy. Give new totals.

Explained solution. 1,420 eligible tokens and twelve hits. These counts refer to the retained four records under the declared policy.

Common wrong turn. Changing only the numerator produces an inconsistent processed corpus.

Exercise 3: Calculate before and after

Question. For the totals in the worked dataset, compare normalised rates.

Explained solution. Before: 16/1,820×1,000,000≈8,791.21 pmw. After: 12/1,420×1,000,000≈8,450.70 pmw.

Common wrong turn. The change in rate is not equal to the proportional change in hits because corpus size also changed.

Exercise 4: Recognise a meaningful revision

Question. A notice changes ‘must’ to ‘may’. Should 99% similarity settle the deduplication choice?

Explained solution. No. The modality changes obligation. Review and record versions according to the task.

Common wrong turn. A numerical similarity value cannot determine semantic equivalence.

Exercise 5: Exposure or independence

Question. The same poster is displayed at ten locations. What should an exposure study count?

Explained solution. It may count ten distinct display opportunities, while a unique-authored-wording study counts one original text. State the analytic unit.

Common wrong turn. One universal deduplication rule cannot serve both aims.

Exercise 6: Boilerplate choice

Question. Every page repeats an irrelevant 50-word menu. What should happen in a study of article-body vocabulary?

Explained solution. Exclude menus from both match search and eligible token denominator, documenting the rule; preserve raw pages.

Common wrong turn. Removing menu hits but not menu tokens distorts the rate.

Exercise 7: Test overlap

Question. A model’s test examples repeat training examples. What kind of evidence is compromised?

Explained solution. Evidence of generalisation to unseen inputs; evaluate on appropriately separated, audited material.

Common wrong turn. A high repeated-example score is not independent transfer performance.

Exercise 8: Audit a tool’s result

Question. A detector flags a 95%-similar notice pair with changed dates and names. What is the next action?

Explained solution. Review the changed fields, identify the shared source and decide under the declared purpose. Document the human-verifiable decision.

Common wrong turn. Automatic detection is not a safe substitute for meaningful interpretation.

Frequently asked questions: Corpus deduplication

What is corpus deduplication?

Identifying and handling repeated or near-repeated text according to explicit analytic rules so that counts describe the intended evidence.

Are exact duplicates and near-duplicates the same?

No. Exact matches meet a declared text-equivalence rule; near-duplicates share substantial text but may contain meaningful differences.

Can file hashes find every copied article?

No. Different formats, layout changes and inserted text can hide identical underlying prose.

Should every repeated sentence be deleted?

No. Repetition can be genuine, instructive, quoted or an actual exposure event.

How does boilerplate affect word lists?

Repeated menus, footers and templates can create many tokens that do not reflect the main authored content.

Can a high similarity score prove two notices are equivalent?

No. A short difference may alter negation, instructions, dates or obligations.

Why does deduplication change normalised frequency?

Both target matches and eligible token totals may change; their ratio determines the rate.

Can a frequency increase after deduplication?

Yes. Removing repeated low-target material can reduce the denominator more than the numerator.

What is a duplicate cluster?

A recorded group of related copies or versions, ideally with source provenance and the decision governing their use.

What is source independence?

The extent to which observations reflect distinct underlying authorship or events rather than repeated instances of the same text.

Why does deduplication matter in machine learning evaluation?

Overlapping training and test examples can make performance look better than generalisation to unseen examples.

Is deduplication always desirable?

No. Exposure, language repetition, revised publication and teaching-practice studies may legitimately retain repeated events.

What metadata should be kept?

Source ID, date, provenance, duplicate relationship, similarity evidence, action, reason and affected counts.

Does deduplication solve sampling bias?

No. It addresses one source of distortion; a cleaned corpus may still omit important writers or genres.

Can schools share a cleaned corpus freely?

Not necessarily. Copyright, permissions and privacy still apply to the retained material.

Short glossary for working readers

• Exact duplicate: A repeated item matching under a specified equality rule.
• Near-duplicate: Highly similar but not completely identical content.
• Boilerplate: Recurring template or site text, such as menus and footers.
• Syndication: Republication or distribution of a common source text.
• Provenance: The source and lineage of a text or passage.
• Version: A distinct state of a document at a given stage or time.
• N-gram: A sequence of n adjacent linguistic units.
• Shingle: A text fragment used to compare local overlap.
• False positive: A distinct item incorrectly flagged as duplicate.
• False negative: A repeated item that the detector fails to flag.
• Normalised frequency: Occurrences divided by eligible corpus size and scaled to a common unit.
• Train-test leakage: Overlap that compromises the independence of an evaluation set.

A four-stage teaching and verification plan

First, begin with the specific claim that a learner wants to make. Reframe it as a question about observable language, not as a contest to find the largest number. The learner writes the target, units and intended audience on one line. That starting line is a useful quality control because it can be compared directly with every later result.

Second, work with a miniature source register and deliberate edge cases. Ask learners to label the obvious examples and then spend more time on duplicated files, ambiguous sentences, unbalanced genres or missing metadata. Differences of judgement are productive when each person can explain the rule and what evidence would settle the case.

Third, reconstruct the worked example without the answers. Keep the same declared rules but change one detail, such as a document’s genre, an inclusion condition, a window length or whether a repeated notice is counted twice. Record what changes in the numerical result. This makes the new vocabulary a working method rather than a definition to memorise.

Fourth, practise transfer. Have the learner write one cautious conclusion for a reading lesson, one for a research report and one for a public-facing explanation. Each should identify what the observed evidence supports and one limitation. If a conclusion overreaches, revise its scope before polishing its style. Clear reasoning is a more durable writing skill than a dramatic claim.

Methodological sources

• Cambridge University Press: collecting corpus data and checking duplicated text
• ACL Anthology: Deduplicating Training Data Makes Language Models Better (Lee et al., 2022)
• International Journal of Corpus Linguistics: Deduplicating corpora (Bednarek and Lee, 2026)
• BYU Law and Corpus Linguistics: Principled Text De-duping

Sources provide conceptual grounding; the fictional worked datasets in this guide are not findings reported by the linked authors.

Related vocabulary evidence guides

• What Is a Corpus Sampling Frame? — Define your source population
• What Is Normalised Word Frequency? — Compare different-sized corpora
• What Is Corpus Metadata? — Source details that make word evidence interpretable
• What Is a Corpus Query? — Know what the search actually finds
• What Is a Gold-Standard Corpus? — Keep checked references distinct from raw examples
• Vocabulary Learning Hub
• Vocabulary Article Library

Return to chapter contents · Vocabulary Learning Hub