AI agents should treat Unicode as a boundary between how text looks and which exact record or command it identifies. Canonically equivalent character sequences can differ in raw code points, while visually similar letters from different scripts can represent distinct identifiers. A safe agent preserves original text, uses field-specific normalisation, flags suspicious confusables and authorises actions only against verified source-controlled IDs—not a label guessed from its appearance.
This guide explains the actual engineering mechanisms for modern AI-assisted applications: text encoding, NFC and NFKC choices, lookalike detection, invisible characters, multilingual fairness, versioned comparison policies and destination-side identity checks. Fictional worked examples show precisely how a normalisation mistake can become a false merge, and how an explicit review path prevents it. Present-day systems benefit from these controls; no existing artificial superintelligence is assumed.
Choose a reading route
New readers can begin with code points, graphemes and the Café example. Builders should focus on field types, comparison pipelines and versioned IDs. Reviewers can read the mixed-script case and the 80-comparison ledger. Educators can jump to the multilingual study-table transfer task.
Contents
- 1. A string that looks right may refer to the wrong thing
- 2. Start with bytes, code points and rendered graphemes
- 3. Use canonical normalisation for its actual purpose
- 4. Confusable characters are not canonically equivalent
- 5. Treat different text fields as different security objects
- 6. Construct a simple, inspectable comparison pipeline
- 7. A byte-level exercise with two spellings of Café
- 8. A confusable identifier exercise with an explicit block
- 9. Worked laboratory: eighty identifier comparisons
- 10. Invisible and directional characters change inspection
- 11. Confusability is not a reason to flatten multilingual names
- 12. Avoid confusing search convenience with account authority
- 13. Identifier uniqueness needs namespaces and versions
- 14. Test the pipeline at every representation boundary
- 15. Diagnose failures without making a model the final judge
- 16. Independent practice: classify four identifier observations
- 17. A changed-context transfer: preserving a multilingual study table
- 18. Limits, governance and a practical learning route
1. A string that looks right may refer to the wrong thing
An agent can read a visually convincing name, type it into a database filter and still select the wrong record. Unicode makes it possible for one displayed word to have different underlying code points, for canonically equivalent character sequences to use different bytes, and for different scripts to contain characters with remarkably similar shapes. These properties enable global writing systems; they are not defects. The defect is treating a rendered appearance as a trusted identity key.
Unicode handling becomes a security and correctness boundary when a model copies user names, file paths, account aliases or tool parameters into software that compares exact values. A natural-language explanation can mention a plausible match; the external system must perform a defined comparison. The safe pattern preserves original text, records encoding and origin, applies only documented field-specific normalisation, checks suspicious identifiers and authorises an action against a stable source-controlled ID. A visually matched label is not permission to operate on an account.
The objective is not to ban multilingual text. It is to accept legitimate names and scripts without silently conflating different identifiers. Good internationalisation and good security often support each other: retaining source forms and showing uncertainty respects people’s names while preventing agents from “repairing” unknown characters into invented identities.
2. Start with bytes, code points and rendered graphemes
A computer stores encoded bytes; Unicode defines abstract code points; text shaping renders sequences into visual glyphs. The distinctions matter because two strings that appear alike may contain different code points, while one visible symbol may use several. In UTF-8, many common Latin letters occupy one byte, but that is not a general rule for all characters. In JavaScript, string length traditionally counts UTF-16 code units rather than user-perceived characters. A user interface that truncates by the wrong measure can cut through a sequence and change what appears.
Consider the letter é. It can be represented as U+00E9 LATIN SMALL LETTER E WITH ACUTE, or as U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT. Both normally display as é, yet their raw code-point sequences differ. If a service hashes raw input strings, looks for exact bytes or uses a strict binary database comparison, it may treat the two spellings as different until an agreed canonical normalisation step is applied.
A grapheme cluster may include a base character and combining marks or multiple code points joined for a single visible emoji sequence. Neither code-point count nor byte count is a reliable proxy for visual width or reading difficulty. An agent producing UI text should not assume that a 40-character limit enforced on bytes is also a 40-visible-character limit. Preserve clear field constraints and test with realistic multilingual examples.
3. Use canonical normalisation for its actual purpose
The Unicode Consortium defines NFC, NFD, NFKC and NFKD as normalisation forms. NFC and NFD manage canonical equivalence through composition and decomposition; compatibility forms additionally apply broader compatibility mappings. NFC can make the two canonical representations of é comparable in a well-defined way. The purpose is to create consistent representations of equivalent text, not to conclude that similar-looking human records are the same. The operation should be specified by Unicode version, field and comparison context.
For ordinary display labels, retaining the original spelling may be important even when a canonical comparison key is also stored. A safe record may hold original_text, nfc_comparison_text, source_encoding, normalisation_version and source_record_id as separate fields. An agent should never present the comparison key as if it were the user’s exact submitted name. If a later complaint alleges altered accents, the originals allow investigation rather than relying on the transformed value.
NFKC can be useful for certain technical comparison tasks but may change distinctions a particular domain wants to preserve. Fullwidth and compatibility characters can become other forms; mathematical or stylistic symbols may lose a distinction meaningful to users. Do not apply NFKC indiscriminately to passwords, legal names, document evidence or opaque identifiers. For a security-critical identifier, the issuer’s published specification should decide permitted characters and canonicalisation.
4. Confusable characters are not canonically equivalent
Unicode security guidance distinguishes strings that render similarly from strings that are equivalent under Unicode normalisation. The Latin lowercase a is U+0061; the Cyrillic lowercase а is U+0430. In some fonts they are difficult to distinguish, yet they are different letters. For a fictional account called case-123, replacing Latin a with Cyrillic а creates a different underlying identifier despite the near-identical on-screen appearance. NFC does not make two separate scripts the same.
Unicode Technical Standard #39 provides mechanisms for detecting confusable text, mixed scripts and identifier restrictions. Its “skeleton” mapping supports detection of potential confusability; it is not intended as a normalised account identifier to display or store as the canonical identity. An application that converts every visually similar identifier into the same value would create the very collisions it intended to prevent. A confusable warning is a reason to verify the correct source and restrict action, not to auto-merge.
The affected user should see a calm, accurate message: “The account label contains characters from more than one writing system. Confirm the account by its verified ID.” This avoids character-based prejudice and does not assume malicious intent. Legitimate multilingual names can mix scripts, transliterations or symbols. Risk policy should be sensitive to whether the field is a public display name, a free-form quotation, a filename, an authentication identifier or an instruction to transfer money.
5. Treat different text fields as different security objects
A person’s name, a file path, a case-sensitive API key, an email address, an internationalised domain name and a natural-language quotation do not share one safe comparison rule. A surname may permit accented letters and culturally meaningful spacing; an API token should normally be compared exactly according to its issuer specification. A document title may be searched approximately but must not be used alone to select a sensitive record. A web domain has its own registration and IDNA rules that cannot be replaced by a general-purpose Unicode normaliser.
Define field classes in a data dictionary: free text for display; human-readable labels used only to help search; source-controlled opaque IDs; filesystem paths; executable commands; and destination addresses. Attach character, length, encoding and comparison rules to each. An AI agent may help explain an exception, but should not override these rules because a name “looks close enough”. Validating after unsafe transformations can create accidental equivalence and security holes, so the order of operations matters.
A structured input parser must identify where text originated. Quoted file contents, retrieved websites and OCR text are not executable instructions simply because they contain words that look like commands. Unicode deception adds another reason to separate data from instruction: invisible control characters or visually ambiguous identifiers can make a human reviewer misunderstand what the model is requesting. Record tool arguments in an inspectable escaped form as well as their exact stored form.
6. Construct a simple, inspectable comparison pipeline
Begin by decoding bytes with the documented encoding, usually UTF-8 when the protocol specifies it. Reject malformed input or handle it under an explicitly defined replacement policy instead of silently inventing a clean value. Preserve the original bytes or exact decoded string where lawful and necessary for provenance. Classify the field by type before normalising. For a selected searchable label, create an NFC comparison value and, if authorised, a separate case-insensitive search index. For an issuer-defined opaque identifier, preserve exact semantics.
Next examine suspicious conditions that matter to that field: leading or trailing invisible characters, unexpected bidirectional controls, unexpected scripts in an identifier, mixed-script confusables, and discrepancies between the displayed label and its source-controlled ID. The action should depend on the finding. A harmless combining accent in a human name is normal; a confusable payment-destination identifier warrants a stronger identity check. Do not turn all international input into errors.
Finally resolve the record using a stable ID or a verified source selection and enforce permission at the destination. If the agent’s language-model answer refers to a display string that conflicts with the target ID, block the operation until resolved. This pipeline separates text hygiene from identity proof and authorisation. Each stage should have a visible reason code so an analyst can tell what changed and what was merely flagged.
7. A byte-level exercise with two spellings of Café
Take the string “Café” with a precomposed é and another string displayed identically but using e plus the combining acute accent. The first may have five Unicode code points including C, a, f and precomposed é? Count carefully: C, a, f, é makes four code points. The decomposed form C, a, f, e, combining acute makes five code points. Both normally display as four user-perceived letters. This example is intentionally stated by code-point names so the visible formatting is not asked to prove the sequence.
Under NFC, each of those two sequences becomes the same canonical representation, C, a, f, U+00E9. Under NFD, each becomes C, a, f, U+0065, U+0301. That is a reason to use documented canonical normalisation when comparing a field whose intended identity is ordinary canonically equivalent text. It is not a reason to remove the accent altogether or match every cafe-related business. A separate source ID must still distinguish Café A from Café B.
If an agent merges two customer accounts merely because their displayed name normalises to “Café”, the error is not a Unicode library bug. It is an identity-model failure: the application promoted a nonunique display label into a unique identifier. Test both layers. First confirm canonical string comparison; then confirm the correct distinct account record is selected through issuer-scoped identity evidence.
8. A confusable identifier exercise with an explicit block
Suppose an agent receives an instruction to update an authorised internal project “case-123” using a trusted project ID P-77. A retrieved file contains a similar-looking label in which its a is a Cyrillic code point. The project registry has no verified relationship between that alternate string and P-77. The model may describe the two labels as visually equivalent, but the correct action is to continue using the trusted P-77 record only if the user’s requested scope and the retrieved evidence both support it. Otherwise the agent should pause and surface the discrepancy.
An attacker-controlled document could exploit the same confusion by presenting an apparent staff name with deceptive letters. But not every script mixture is an attack; false alarms are possible. The tool should report objective properties such as U+0430 in a field expected to contain an ASCII identifier, rather than claiming that a person is malicious. Human reviewers need source location, code-point names, the exact target ID, and the proposed action before deciding.
A confusable skeleton can help produce a review alert, but it should not overwrite the original string or assign a record. If the two strings are genuinely intended as aliases of the same project, add an explicit verified alias edge under the project owner’s policy. That turns a guess into a reviewable data-management decision. If they are separate, the near visual resemblance remains important evidence of collision risk, not evidence of shared identity.
9. Worked laboratory: eighty identifier comparisons
Consider a fictional batch of 80 comparisons against an authorised identifier directory. Sixty cases are exact, issuer-scoped ASCII matches. Eight are labels that differ only by independently verified canonical Unicode representation and normalise to the expected value under the directory’s agreed NFC rule. Seven contain visually confusable characters that correspond to different, verified source IDs and therefore must not be auto-merged. Five contain unsupported directional or control-character patterns whose meaning remains unresolved. The groups are exhaustive: 60 + 8 + 7 + 5 = 80.
A cautious policy completes the 60 exact and eight canonical-equivalent cases when they also meet the source-ID and permission rule. It sends the other twelve to a review path rather than treating their display forms as proof. Automatic coverage is (60 + 8) / 80 = 68 / 80 = 85%. The held proportion is (7 + 5) / 80 = 12 / 80 = 15%. If a naive visual comparison had wrongly merged all seven verified-distinct confusable cases, its false-merge count would be seven in this teaching set: 7 / 80 = 8.75% of all candidate comparisons, not a measured real-world attack rate.
Do not infer that the cautious policy’s 85% coverage proves an 85% correctness rate. Coverage counts decisions completed automatically. Verification quality requires independent ground truth for accepted and held cases and an examination of the cases never retrieved. This fictional grouping is designed to teach why the action criterion must combine text comparison with issuer-scoped identity, not to benchmark a Unicode security product.
10. Invisible and directional characters change inspection
Some characters have little or no obvious glyph but still affect storage, cursor movement or text layout. Examples include zero-width formatting characters and directional controls. In right-to-left and mixed-direction writing, the visual ordering of numbers and punctuation may differ from the logical sequence stored. This is legitimate language functionality, yet an agent copying identifiers from a visual screenshot can misread what belongs to a command, path or field. A trustworthy inspection view should reveal control-character presence without mangling the person’s actual text.
A careful system offers two views: a faithful display for people using the language, and an escaped diagnostic representation with code points for review of identifiers or commands. A reviewer should not have to infer a hidden mark from a blinking cursor. Avoid sanitising a complete document into plain ASCII merely to make logs easier; that would destroy meaning and still not solve identity. Instead define where controls are permitted and where an exact, machine-readable identifier is required.
Right-to-left source content deserves the same evidence fidelity as left-to-right content. If the task involves international names, translate or transliterate only as a separate annotation and keep the original. If the task involves an internal API key that forbids bidi controls, reject the invalid value with a clear reason. The field’s semantics, not the user’s language, determine which rule is justified.
11. Confusability is not a reason to flatten multilingual names
A multilingual education service may receive Arabic, Chinese, Hindi, Russian and English names. Simplifying them all to one Latin-only string can produce false matches and exclude legitimate distinctions. Names can also have multiple common romanisations. The right model recognises aliases and transliteration links as claims backed by sources, not irreversible transformations of identity. The original script, alternate representation and provenance should travel together.
For example, a parent might write a student’s name in one script while the official roster uses another. A model’s transliteration can be a candidate-search aid, but the identity decision should use the school-controlled record ID or a documented verification process. A small spelling discrepancy may be normal. The system should neither refuse reasonable service because characters are unfamiliar nor disclose another student’s record because two transliterations look alike.
Data-quality reviewers can test outcomes across scripts without using one script as the only gold standard. Keep evaluation sets representing the actual population, measure false merges separately from missed matches, and review held cases with appropriate linguistic expertise. A global English-language similarity threshold is not a substitute for culturally and technically informed evaluation.
12. Avoid confusing search convenience with account authority
Search often tolerates approximation: it can return nearby spellings, normalised aliases and a shortlist of relevant documents. Account changes, payment destination selection and data disclosure cannot use the same permissive semantics. A search result should carry the stable ID and its source information; before taking action, revalidate that ID under current permissions. The agent must not execute a tool call solely because a displayed string looks like what the user typed.
This split also prevents a common confused-deputy problem. An untrusted page might contain a visually deceptive account label that resembles an authorised destination. If the model copies the label into a higher-privilege tool call, the agent can be steered toward the wrong target. Keeping retrieved text as evidence rather than command and requiring source-controlled IDs interrupts that path. Unicode hygiene is one layer of defence, not a replacement for tool permission checks.
Use a confirmation interface that shows the actual record’s trustworthy human-readable name alongside a stable key, source, scope and proposed action. For high-consequence actions, a reviewer can be asked to verify an independent record or receipt. Showing only a candidate’s visually similar display label is not enough to establish what will change.
13. Identifier uniqueness needs namespaces and versions
Even exact Unicode strings can collide if their namespace is missing. “item-042” in an inventory database may not be “item-042” in a research catalogue. If an agent aggregates records across platforms, always attach the issuer and object type to the identifier. Then enforce uniqueness on the complete scoped key and preserve the original source label as a separate display field. Unicode normalisation cannot replace namespace design.
Schemas can evolve. A system may move from ASCII-only names to international identifiers, change its normalisation policy or upgrade Unicode tables. Record which rules apply to stored identifiers and design a migration before reindexing. A seemingly harmless new comparison rule might make two previously distinct stored values collide. Reindexing must surface those conflicts for deliberate review, rather than silently selecting the older or newer record.
A unique canonical string is useful only within the scope where its uniqueness is guaranteed. If the system does not control the issuer, a matching value alone may not be authoritative. The safest long-term design is to use generated opaque IDs for internal operations and allow richly international display names around them without forcing each human name to be unique.
14. Test the pipeline at every representation boundary
Begin with paired test inputs: NFC versus NFD, visually confusable Latin/Cyrillic letters, legitimate mixed-script names, leading and trailing nonbreaking spaces, combining marks, directional controls and emoji grapheme clusters. For each, specify the permitted source field, expected comparison key, alert status and authorised downstream operation. Test the input parser, database collation, serializer, log viewer and final UI. A safe transformation at one layer is not sufficient if another layer treats bytes or graphemes differently.
Test a migration from an older canonicalisation version to the current policy using a copy of real schemas with synthetic sensitive fields. Count newly colliding keys, missing rows and changed decisions. Ensure each conflict produces an actionable review record. A “successful index rebuild” says nothing about whether two customers silently acquired the same searchable identifier; test the identity invariant directly. Avoid production experiments that could expose real private records.
When a service handles a document with several scripts, ask a competent reader to verify text direction and appearance under common fonts and assistive technology. A raw code-point dump can diagnose machine comparison, but it does not prove that a human presentation is clear or culturally correct. Accessibility and Unicode safety belong together in end-to-end tests, even though their failure modes differ.
15. Diagnose failures without making a model the final judge
Failure one: an agent stores two canonically equivalent labels as different search keys. Repair with the documented canonical normalisation suited to that field while preserving originals. Failure two: an apparent duplicate account uses a confusable character from a different script. Repair with source-ID validation, an alert and an abstain path rather than blindly mapping skeletons. Failure three: a system strips every accent and merges two legitimate names. Repair the data model so display labels are not unique account keys.
Failure four: a model reads an identifier from a screenshot and inserts an invisible character into a tool parameter. Repair through a typed destination ID picker or strict identifier validation, never by guessing at intent. Failure five: a log viewer hides directional controls and makes a malicious or malformed value appear benign. Repair with escaped diagnostic views and provenance. Failure six: a Unicode library upgrade creates a new collision. Repair with versioned migration tests and a stop condition before destructive reindexing.
The diagnostic question is always which boundary first lost the distinction. Was it decoding, normalisation, candidate search, source identity, UI rendering or final authorisation? Repair at that boundary and add a regression test. A generic language-model instruction to “be careful with Unicode” is not a substitute for enforcing the correct representation and permission rules.
16. Independent practice: classify four identifier observations
Imagine a fictional project directory in which the authoritative ID “project-52” is stored under a controlled ASCII-only scheme. Four observations arrive: the exact string; the same string with an unexpected invisible character; a display alias with a non-Latin lookalike letter; and a natural-language project title in Arabic accompanied by the exact authoritative ID. Which should be used to change the project record? The key question is not which visible text feels most familiar but whether the exact source-controlled ID and access scope are verified.
The exact validated ASCII identifier can select the project within its issuer’s namespace when the user also has permission. The string containing an invisible disallowed character should fail strict ID validation and enter diagnostics. The visually deceptive alias is not an authoritative ID; it may be recorded as an unresolved candidate after review. The Arabic title is legitimate display data and should be preserved without transliteration loss, while the accompanying verified ID is used to select the record. None of those observations alone authorises an action.
Now change the requirement: the task is only to search for documents about the project, not to change a record. The alias and multilingual title may become valid search aids, provided results remain scoped and suggestions carry provenance. The acceptable comparison becomes more permissive precisely because a search suggestion is not a sensitive write. This transfer demonstrates why field purpose and action severity should determine text-handling policy.
17. A changed-context transfer: preserving a multilingual study table
A tutoring assistant receives a vocabulary table containing English, Mandarin, Hindi and Russian words. It must identify duplicated rows but must not erase distinctions between languages or visually similar characters. The previous identifier rule of strict ASCII acceptance would be inappropriate for the words themselves. Define row identity instead as the worksheet ID plus language column plus an instructor-assigned vocabulary entry ID. Normalise text only to support search or checking within the correct language and keep the original spelling.
Suppose a 40-row worksheet contains eight rows with legitimate accents or combining marks, six with script-specific characters that resemble Latin ones, and two genuine duplicated entry IDs accidentally pasted twice. A validator should not mark the fourteen unusual-looking rows as duplicates; it should detect the two repeated IDs in their exact issuer-scoped column and ask for a correction. If the two duplicated entries each appear twice, the set of unique IDs has 38 distinct IDs across 40 rows; there are two surplus rows. That count is about ID uniqueness, not whether any vocabulary translation is correct.
The explained lesson is transferable: a safe equality relation depends on what kind of object the application is comparing. User-facing multilingual content deserves faithful preservation. Security-critical identifiers deserve an issuer-defined exactness rule. The agent can assist with both only if it keeps their comparison contracts separate.
18. Limits, governance and a practical learning route
Unicode is a living standard with versioned specifications; as of October 2026, the Unicode Consortium’s current published normalisation and security reports should be checked directly rather than paraphrased from an old library blog post. Confusable analysis is not exact visual science because glyph shapes vary across fonts and contexts. A detected confusable is a warning, not proof of impersonation, and absence of a warning does not prove a string is safe. Apply layered controls and test the environment actually used.
For learners, the first useful exercise is to inspect code points in two visually identical strings and learn why text representation differs from identity. For builders, the next step is designing typed identifiers, canonical comparison values, source provenance and safe UI displays. For security reviewers, the focus is whether an untrusted value can select an unauthorised destination. For multilingual educators, preserving original scripts while documenting meaningful comparisons becomes a form of respect as well as accuracy.
The enduring principle is simple: representation is not authority. A model may interpret a string, but the application must identify the exact object, validate the field’s rules, enforce permission and provide enough evidence to repair any wrong choice. That discipline is useful in AI-assisted tools today without relying on claims about artificial superintelligence.
Frequently asked questions
Should every user name be normalised to ASCII?
No. That destroys legitimate script and accent distinctions and can create collisions. Preserve original names and use typed stable IDs for consequential operations.
Does NFC remove lookalike attacks?
No. NFC handles canonical equivalence, not arbitrary cross-script visual resemblance. Use field-specific checks and trusted identity evidence.
Is a confusable skeleton a safe identifier?
No. Unicode describes it as an internal comparison aid for confusability detection, not a canonical user-facing or stored account identifier.
Can a name with mixed scripts be legitimate?
Yes. Names and documents may naturally mix languages or scripts. Evaluate the field and action context instead of making a blanket accusation.
How can a reviewer inspect invisible characters?
Provide an escaped diagnostic view with code-point labels and original source provenance, while preserving a faithful human-readable view for the user’s language.
What happens when a Unicode policy changes?
Version the rule, test reindexing against actual keys and hold newly colliding identifiers for review; never silently overwrite an earlier record.
Official technical sources
The Unicode Consortium’s UAX #15 Normalization Forms defines canonical and compatibility normalisation. Its UTS #39 Unicode Security Mechanisms covers confusable detection, mixed-script analysis and identifier policy; the document explicitly limits what skeletons can prove. UAX #29 Text Segmentation explains grapheme clusters, and UAX #9 Bidirectional Algorithm covers text layout in mixed writing directions. The fictional project and worksheet figures above are original classroom exercises, not reported case studies.
Follow the wider learning route
Continue through the How Super Intelligence Works hub and How X Works master hub. Read agent entity resolution for the separate question of whether two records refer to the same entity, and multi-tenant isolation for enforcing account boundaries. The Gold Standard of Education connects precise technical understanding with everyday responsible decisions.