How does superintelligence work when one event appears twice in a report? An AI assistant might read two identical library-loan entries and conclude that two books were borrowed. But one event can be recorded twice because a file was imported again, a message was retried or two databases copied the same record. Counting rows is not always the same as counting what happened.
The practical mechanism is duplicate detection, identity and reconciliation. A trustworthy AI system distinguishes separate objects, repeated records of one object, different events involving the same object and uncertain matches that need human review. This helps with study records, school libraries, invoices, registrations, research and automated agents.
This accessible guide follows an invented lending ledger with fourteen rows and ten distinct events. We will count duplicates, preserve returns as separate events, inspect the risks of merging by name alone and show an auditable method for reconciling records. Every person and entry is fictional, with no real pupil information or measured model result. “How Superintelligence Works” is an educational series title, not a declaration that present-day AI possesses artificial superintelligence.
1. One thing can appear in many places
A school could have a file in an email attachment, a cloud folder and a backup. Three copies of the file do not necessarily describe three independent pieces of work. Likewise, one lesson may appear in a timetable, an attendance sheet and a parent’s calendar.
A model that counts every appearance as a new event can inflate a report. The first question is what unit the user wants to count: rows, documents, physical objects, people, bookings, loans or changes in state.
2. Duplicate rows are not always duplicate events
Two rows with identical text may be exact duplicates of one event. But two events can also legitimately look similar. A pupil could borrow the same book on two different days. A tutor might deliver the same lesson topic to two different groups.
The same title, name or amount is therefore not sufficient evidence of a duplicate. The system needs the right identifiers and event context. Removing records because they resemble each other can destroy valid history.
3. What is a stable identifier?
A stable identifier names one entity or event within a defined system. In our fictional ledger, every lending event has an event ID such as EVT-01. Books have asset IDs such as BL-07. Borrowers have local identifiers such as M1.
Those identifiers represent different kinds of identity. An asset ID tracks a particular book copy; an event ID tracks an issue or return. A borrower ID tracks a person within the authorised register. Confusing these identifiers can lead to false merges.
4. Why a title is not an identifier
Two copies of the same textbook can share the title, author and edition while being different physical assets. If both are borrowed, merging them because their titles match would erase a real loan.
A unique asset identifier prevents that mistake. Similarly, two people can have the same first and last names. Matching a person by name alone is unreliable, especially when the result affects records or contact information.
5. Distinguish physical identity from event identity
A book can be issued, returned and issued again. Those are separate events involving the same object. They should not be merged into one record merely because the book’s asset ID repeats.
The event sequence matters. A return changes the outstanding-loan state. If an AI deletes the return as a duplicate of the earlier issue, it may incorrectly report the book as overdue or still out.
6. The fictional Cedar Lending Ledger
Imagine an invented school library importing fourteen rows from two local exports. Each row carries an event ID, event date, book asset, action and borrower ID. The system’s explicit rule is that an identical repeated event ID with identical fields represents a duplicate copy of the same confirmed event.
We assume the data were already checked against the source for this exercise. In real systems, event IDs and record authority may differ between sources, and matching needs stronger validation. Our simple example makes the duplicate arithmetic visible.
7. Fourteen imported rows
- A — EVT-01, 1 Oct 2026, BL-07, ISSUE, M1.
- B — EVT-02, 2 Oct 2026, BL-08, ISSUE, M2.
- C — EVT-01, 1 Oct 2026, BL-07, ISSUE, M1: duplicate of A.
- D — EVT-03, 2 Oct 2026, BL-07, RETURN, M1.
- E — EVT-04, 3 Oct 2026, BL-09, ISSUE, M3.
- F — EVT-02, 2 Oct 2026, BL-08, ISSUE, M2: duplicate of B.
- G — EVT-05, 4 Oct 2026, BL-08, RETURN, M2.
- H — EVT-06, 5 Oct 2026, BL-10, ISSUE, M4.
- I — EVT-07, 5 Oct 2026, BL-11, ISSUE, M5.
- J — EVT-04, 3 Oct 2026, BL-09, ISSUE, M3: duplicate of E.
- K — EVT-08, 6 Oct 2026, BL-09, RETURN, M3.
- L — EVT-09, 6 Oct 2026, BL-12, ISSUE, M6.
- M — EVT-10, 7 Oct 2026, BL-13, ISSUE, M7.
- N — EVT-10, 7 Oct 2026, BL-13, ISSUE, M7: duplicate of M.
8. The first accounting check
The raw import contains fourteen rows, A through N. But four rows repeat an earlier event ID with matching data: C repeats A, F repeats B, J repeats E and N repeats M.
Removing only those four redundant copies leaves ten distinct events, EVT-01 through EVT-10. Fourteen minus four equals ten. We have removed extra representations of events, not removed any real event that the fictional ledger intended to record.
9. Count issues and returns after reconciliation
Among the ten unique events, issues are EVT-01, EVT-02, EVT-04, EVT-06, EVT-07, EVT-09 and EVT-10: seven. Returns are EVT-03, EVT-05 and EVT-08: three. Seven plus three equals ten.
Counting raw rows instead would produce eleven ISSUE rows and three RETURN rows. That inflated issue total would be a reporting error caused by duplicated imports. Reconciliation restores the intended unit of counting.
10. Identify the outstanding loans
Three issued books have matching returns: BL-07, BL-08 and BL-09. Four issued books have no return event in this complete fictional event sequence: BL-10, BL-11, BL-12 and BL-13.
Under the exercise’s complete-ledger assumption, there are four open loans. This is a statement about the records as of the last event. It does not prove the books are physically missing or that a particular borrower is at fault. A real system may have pending returns that were not yet synchronised.
11. Reconciled reference totals
- Raw imported rows: 14.
- Repeated copies identified: 4 (C, F, J and N).
- Unique event IDs after reconciliation: 10.
- Unique ISSUE events: 7.
- Unique RETURN events: 3.
- Unique physical asset IDs in the listed events: 7 (BL-07 through BL-13).
- Assets with recorded returns: 3 (BL-07, BL-08, BL-09).
- Assets with no return event in the toy ledger: 4 (BL-10 through BL-13).
- Excess-row share: 4/14 ≈ 28.6%, a property of this invented import.
- Final reconciliation: 7 issues − 3 returns = 4 open loans, given one issue per listed asset.
12. The difference between a repeated event and a related event
Rows A and D both concern BL-07 and M1. But A records an ISSUE and D records a RETURN, with different event IDs and dates. They are related events, not duplicates.
A model that merges by asset ID alone would lose the return. A model that treats every shared borrower ID as duplicate might collapse all a borrower’s activity into one invented event. Correct identity requires both the right key and the kind of event being recorded.
13. Why “same person” is a harder question
A name may be spelled differently across files. Two records can have different transliterations or an added middle name and still concern one person. But two people can also share the same name and birthday. Name similarity alone is not an adequate basis for consequential identity decisions.
Entity resolution systems compare authorised identifiers and other evidence while controlling the costs of false merges and missed matches. For our toy ledger, the borrower IDs are already known. We do not pretend to establish real-world identities from names.
14. False merges and missed duplicates
A false merge happens when the system combines two genuinely distinct records. That can erase a real transaction or link information from different people. A missed duplicate leaves repeated copies that may inflate counts or trigger repeated actions.
These are distinct error types. The right balance depends on the application. A search index may tolerate showing a duplicate title, while financial, educational and identity records require much more careful handling.
15. A matching event ID with conflicting fields is not an easy duplicate
Suppose another export shows EVT-04 with a different book asset or date. The ID matches, but the details conflict. It may be a correction, corruption, misuse of an identifier or an attempted update. The model should not silently discard one value or choose whichever arrived last.
The correct response depends on the system’s versioning and authority rules. Preserve both observations for review, identify the source timestamps and determine whether a valid amendment exists. A simple deduplication rule is not a substitute for conflict resolution.
16. Exact duplicates versus near duplicates
An exact duplicate can have matching keys and fields. A near duplicate may differ in formatting, punctuation or harmless whitespace while describing the same event. For example, “BL-07” and “BL 07” might represent one asset under a known normalisation rule.
But automatic normalisation can also collapse genuinely different identifiers. A trustworthy system uses rules appropriate to the source, records what was normalised and flags uncertain cases rather than assuming every small text difference is harmless.
17. Version history is not duplication
A worksheet saved as Version 1 and Version 2 may share most paragraphs but include a corrected answer. If a system deletes Version 2 as a duplicate of Version 1, it loses the correction. The desired task may be to identify the current approved version rather than remove all similar files.
Documents should carry version identifiers, effective dates and provenance where relevant. Similar content is evidence worth inspecting, not proof of duplicate identity.
18. A record’s source matters
Two databases may copy an event from a common upstream system. Treating them as independent confirmations of the same event would inflate the weight of evidence. One source repeated twice is not automatically two independent witnesses.
An AI research assistant should preserve provenance: which system originally recorded the event, which export copied it and whether a later record is independent or derived. This is especially important in multi-source reports.
19. Why a duplicate can create a duplicate action
Suppose an automation processes each imported row as a request to send a reminder. If the same reminder request appears twice and the system lacks safeguards, a user might receive two messages. Removing duplicated reporting rows is one defence, but a safe action system also needs a method to recognise an already completed request.
A stable request ID and an appropriate idempotency design can help ensure that retrying the same authorised action does not create an unintended second effect. The language model should not be the only component responsible for duplicate protection.
20. Sixteen claims to classify
Use three labels: SUPPORTED, CONTRADICTED and NOT ESTABLISHED. The fictional ledger defines matching repeated event IDs as copies when their event fields agree, and it is complete for the listed issue and return activity. Claims about legal ownership, physical whereabouts or unrecorded actions remain outside the evidence.
The goal is to test the interpretation of event IDs, assets and duplicate count, not to infer the personal circumstances of imaginary borrowers.
- 01. “There are fourteen imported rows.” — SUPPORTED.
- 02. “There are fourteen distinct lending events.” — CONTRADICTED.
- 03. “C duplicates event A.” — SUPPORTED.
- 04. “F duplicates event B.” — SUPPORTED.
- 05. “J duplicates event E.” — SUPPORTED.
- 06. “N duplicates event M.” — SUPPORTED.
- 07. “D duplicates A because it mentions BL-07.” — CONTRADICTED.
- 08. “Seven distinct ISSUE events remain.” — SUPPORTED.
- 09. “Three distinct RETURN events remain.” — SUPPORTED.
- 10. “The ledger records a return of BL-09.” — SUPPORTED.
- 11. “The ledger records a return of BL-13.” — CONTRADICTED under the complete stated toy event list.
- 12. “There are four open loans at the end of the recorded sequence.” — SUPPORTED.
- 13. “Every repeated row refers to a different physical book.” — CONTRADICTED.
- 14. “The borrowers legally own their issued books.” — NOT ESTABLISHED.
- 15. “All open-loan books are certainly physically lost.” — NOT ESTABLISHED.
- 16. “EVT-03 is the return event for BL-07.” — SUPPORTED.
21. Verify the classification totals
Supported statements are 01, 03, 04, 05, 06, 08, 09, 10, 12 and 16: ten. Contradicted statements are 02, 07, 11 and 13: four. Not-established statements are 14 and 15: two. Ten plus four plus two equals sixteen.
An external system might have a more recent return, but the invented fixture is expressly a complete source for its defined period. A real-world assistant would instead say “no return is recorded in the source checked” unless it had an authoritative complete transaction history.
22. A simple manual deduplication method
First define the thing being counted. Next choose a stable event key. Compare records with the same key and inspect whether their event fields match. Mark exact extra copies while retaining an audit trail. Finally recompute the totals and check that the underlying event sequence remains meaningful.
This is easier to verify than asking an AI to “remove anything that looks duplicated”. The vague instruction has no clear unit of identity and can mistakenly remove legitimately repeated actions.
23. What a safe deduplication tool should preserve
A reconciled record should retain the canonical event, links to the original source rows, import timestamps where available, the rule used to identify duplicates and any unresolved conflicts. This allows someone to trace why fourteen rows became ten events.
Deleting the evidence of earlier imports may make the summary look tidy while making corrections difficult. The purpose is trustworthy counting and state reconstruction, not merely a shorter spreadsheet.
24. How a human reviewer can challenge the result
Ask the AI to identify every row it proposes to merge and which stable key supports the merge. Inspect at least the cases that could change obligations or people. Check that issue and return events are kept distinct. Then compare before-and-after counts.
If the assistant reports that four duplicates were removed but the row count falls by six, the reconciliation is inconsistent. Numerical invariants give reviewers a practical way to catch mistakes.
25. The danger of fuzzy matching without a threshold
A system might calculate similarity between titles, dates or names and decide that records above a chosen score are duplicates. Such scores require validation for the target data. A high similarity score is not a guarantee of same identity.
A safe workflow can send borderline cases to review, using a second source or authorised identifier when available. The cost of a false merge can be larger than the cost of keeping a possible duplicate temporarily.
26. Different fields need different identity rules
A document might be the same exact file across two folders but have different permissions; a pupil might have two account IDs after a migration; a class session might be rescheduled while keeping its event identity. One rule cannot correctly reconcile all such cases.
Specify whether the key represents a person, an object, an event, a version or a relationship. The right answer depends on that choice, not simply on how similar two lines appear.
27. Borrowing history is not ownership
The imaginary record shows that books were issued to borrowers. It does not establish that borrowers bought the books, own the intellectual property or have authority to dispose of them. Those are different relationships.
This links duplicate reasoning with part–whole and ownership reasoning. A system should not let the nearest named person become the presumed owner of every item in the record.
28. A transfer exercise with classroom forms
A school receives two uploads with the same verified submission ID and identical contents. Under a policy that defines submission IDs as unique events, these may be two deliveries of one submission. If the uploads contain different answers despite the same ID, the system must determine whether a valid revision occurred.
Treating every second upload as an unauthorised duplicate could discard a legitimate corrected submission. Treating every second upload as a new submission could inflate the number of participating pupils. The source’s version rule decides.
29. A transfer exercise with an email reminder
A staff member drafts one approved reminder, but a network retry causes the same send request to appear twice. The application should consult the sending service’s receipt and safe retry mechanism before issuing another message.
An AI that says “the first attempt timed out, therefore it definitely failed” may cause duplicates. A timeout can mean the result is unknown. The correct next step is status reconciliation, not blind repetition.
30. How to teach the idea with cards
Give pupils fourteen paper cards representing the ledger rows, including the four duplicated event IDs. Ask them to count cards, then distinct events, then assets and finally open loans. The four questions have different correct answers: fourteen, ten, seven and four.
This is a powerful demonstration that counting is never only arithmetic. You must first define the object being counted. An AI may perform perfect addition while answering the wrong counting question.
31. How to evaluate a real record-matching model
Prepare examples with genuine duplicate events, repeated actions involving the same object, name collisions, formatting differences, corrected versions and conflicting sources. Independently label which pairs refer to the same identity or event. Measure false merges separately from missed duplicates.
Report how the system handles uncertain pairs and what happens after a correction. A model that performs well on identical copies may fail when formats differ or when two distinct people share similar names.
32. What responsible reporting sounds like
A good summary of this toy ledger says: “Fourteen imported rows contain ten unique lending events, including seven issues and three returns. Four rows repeat earlier confirmed events. Four loans remain open in the recorded sequence.”
It does not say: “Four borrowers lost their books,” “Fourteen books were borrowed,” or “The library made four errors.” Those stronger claims do not follow from the defined records.
Frequently asked questions
Are two identical rows always duplicates? Only if they represent one event or entity under the data’s identity rules.
Can a book appear in two events? Yes. Issue and return are different events involving the same asset.
Does the same name prove the same person? No. Names can collide or vary; consequential identity matching needs stronger evidence.
What is a false merge? Combining records that actually represent different objects, people or events.
Is an unknown return the same as a confirmed missing book? No. It is a record state that may require further checking.
How can duplicates affect an AI agent? They can inflate counts or trigger repeated actions if request identity is not enforced.
Should original duplicate rows be deleted permanently? Not automatically. Preserve provenance and an audit trail according to the applicable record-retention rules.
The central lesson
An AI system cannot know what “twice” means until it knows the thing being counted. Fourteen rows, ten events, seven books and four open loans are all correct descriptions of different aspects of the same fictional ledger.
Reliable deduplication means identifying stable objects and events, preserving genuine state changes, recording how duplicate copies were recognised and refusing to invent identity or ownership from resemblance. The goal is a traceable record, not merely fewer rows.
Connected learning and reliable sources
- How Super Intelligence Works — main curriculum.
- Parts and Wholes — distinguishing objects, members and ownership.
- Unknown, None and Zero — distinguishing absent records from recorded absence.
- False Positives and False Negatives — measuring false merges and missed problems.
- W3C PROV-O — representing entities, activity and provenance.
- PostgreSQL: INSERT and ON CONFLICT — database approaches to unique constraints and repeated writes.
- NIST AI Risk Management Framework — verification and controlled use of AI results.