Two names look identical on the screen.
The database says they are different.
Neither display is necessarily wrong.
The strings may simply use different Unicode sequences that are defined to be equivalent.
Unicode normalization converts equivalent text representations into defined standard forms so systems can compare and process them consistently.
This is the second pillar beneath How Encoding Works. The master owns encoding contracts generally. This article owns canonical and compatibility equivalence: why text can look the same while its underlying code-point sequence differs, and when that difference should or should not be erased.
Quick Read
Unicode defines four normalization forms. NFD performs canonical decomposition. NFC performs canonical decomposition followed by canonical composition. NFKD and NFKC use compatibility decomposition, with NFKC recomposing afterward. NFC/NFD preserve canonical distinctions while giving canonically equivalent strings consistent forms. NFKC/NFKD intentionally erase additional compatibility distinctions and therefore can change semantics or formatting in contexts where those distinctions matter. Normalization improves comparison, search and interoperability, but it is not a substitute for collation, case folding, grapheme segmentation or identifier-security policy.
Unicode text → choose equivalence policy → decompose → canonical reorder → optionally recompose → normalized form → comparison / storage / search
The Same Visible Accent Can Have Two Encoded Forms
Some accented characters can be represented as:
- one precomposed code point; or
- a base character followed by a combining mark.
To the reader, the result can look the same.
To a byte-by-byte comparison, the sequences differ.
Unicode calls such representations canonically equivalent when they are intended to represent the same abstract text.
Normalization Gives Equivalent Strings a Unique Form
The current Unicode 17 version of Unicode Standard Annex #15 states a central design goal: canonically equivalent strings should normalize to precisely the same form under NFC or NFD.
That makes ordinary binary comparison meaningful after both strings have been transformed according to the same normalization policy.
NFD: Canonical Decomposition
NFD expands canonically decomposable characters into their canonical components and orders combining marks canonically.
Conceptually:
precomposed form → base + combining components
This is useful for some internal processing and makes canonical structure explicit.
NFC: Canonical Decomposition Then Composition
NFC begins with canonical decomposition and then recomposes sequences where Unicode’s composition rules permit it.
The Unicode Normalization FAQ describes NFC as the best general form for much ordinary text because it is compatible with a large body of existing encoded text and gives canonically equivalent strings stable comparison behaviour.
Canonical Ordering Matters Even When Composition Does Not
A base can carry several combining marks.
Different code-point orders may render similarly.
Normalization defines canonical ordering so equivalent sequences do not remain arbitrarily permuted internally.
NFKD and NFKC Go Further
The K stands for compatibility.
Compatibility normalization can collapse distinctions that Unicode preserves under canonical equivalence.
Examples include certain presentation or compatibility characters whose distinction is useful for round-trip or formatting purposes but may be undesirable in restricted identifiers or loose matching.
NFKC Is Not “Better NFC”
Compatibility normalization is more aggressive.
UAX #15 explicitly warns against blindly applying NFKC/NFKD to arbitrary text because compatibility distinctions can matter semantically or for round-trip conversion.
Normalization policy must therefore match the job.
Canonical Equivalence and Visual Similarity Are Different
Two strings can look alike without being canonically equivalent.
Latin “a” and a similar-looking character from another script are not made identical merely because the glyphs resemble each other.
Normalization is not a general “make visually similar text equal” function.
Normalization Is Not Case Folding
“A” and “a” are not canonically equivalent just because a case-insensitive search may want them to match.
Case folding is a separate comparison transformation with its own language and Unicode rules.
Do not mix transformations conceptually simply because they are often used in the same search pipeline.
Normalization Is Not Collation
Normalization can make equivalent representations consistent.
Collation decides ordering and comparison behaviour under linguistic conventions.
Two normalized strings may still compare differently depending on locale-sensitive collation rules.
Normalization Is Not Grapheme Segmentation
NFC does not guarantee that one visible character becomes one code point.
Emoji sequences and many script structures remain multi-code-point user-perceived characters.
The first pillar, Grapheme Clusters, owns user-perceived text segmentation.
Search Benefits From Canonical Equivalence
A user types a composed accented form.
The indexed document contains the canonically equivalent decomposed sequence.
Without normalization-aware processing, a naive exact match can miss it.
Normalization can therefore prevent representation accidents from becoming search failures.
But Search Often Needs More Than Normalization
Case.
Diacritics.
Word boundaries.
Language-specific collation.
Tokenisation.
Stemming or morphology.
Those are additional search-policy layers, not consequences of NFC alone.
Identifiers Need a Stronger Policy Than Free Text
Usernames, domain labels, programming identifiers and security principals cannot rely on display appearance alone.
A robust identifier policy can specify:
- allowed scripts;
- normalization form;
- case handling;
- confusable checks;
- length units;
- prohibited control/format characters.
The Unicode FAQ notes that NFKC can be appropriate in restricted identifier domains, but security-sensitive identifier design should follow the relevant Unicode security and identifier specifications rather than inventing ad hoc transformations.
Normalization Before Hashing Changes Identity Semantics
Hash raw bytes and canonically equivalent sequences produce different hashes.
Normalize first and then hash, and canonically equivalent text can produce one normalized byte representation.
Which behaviour is correct depends on what the hash is identifying:
- exact original bytes;
- canonical Unicode text identity;
- a higher-level application object.
Canonicalisation should never be inserted into identity pipelines without defining the owner of truth.
Digital Signatures Make This Boundary Critical
A signature normally authenticates exact bytes.
Normalize the text after signing and the byte sequence may change even though a reader sees equivalent text.
If canonical text signing is required, the canonicalization rules must be part of the signed protocol before hashing/signing—not an afterthought at verification.
Normalization on Ingest or on Compare?
There are several architectures.
- normalize on ingest: store a chosen normalized representation;
- preserve original + normalized key: retain source bytes/text while indexing a normalized form;
- normalize on compare: transform both sides at comparison time.
Each has trade-offs for provenance, storage, performance and round-trip fidelity.
Preserving Original Input Can Be Valuable
Even when an application uses NFC internally, retaining the original representation can matter for:
- forensics;
- digital preservation;
- exact round trips;
- auditing user input;
- reproducing a source document.
Normalized identity and source provenance are different jobs.
Concatenation Can Break Normalization
Two strings are individually normalized.
Concatenate them.
The result is not guaranteed to remain normalized because a combining sequence can interact across the join boundary.
The Unicode Normalization FAQ explicitly calls this out and describes efficient normalized-concatenation strategies.
Streaming Normalization Needs Boundary State
If text arrives in chunks, normalization cannot always process every chunk independently and assume concatenating the outputs is equivalent to normalizing the complete stream.
A correct streaming implementation must preserve enough boundary context to handle combining sequences safely.
Normalization Is Designed to Be Idempotent
Normalize an NFC string to NFC again.
The result should remain unchanged.
UAX #15 lists idempotence as a core design property of the normalization forms.
This makes normalized representations operationally stable once the chosen form is reached.
Normalization Stability Matters Across Unicode Versions
Unicode treats normalization stability as a major compatibility guarantee. Systems can store normalized text without expecting ordinary future Unicode releases to arbitrarily rewrite established normalization results for already-assigned characters.
Version discipline still matters for newly assigned code points and broader identifier/security rules.
Do Not Normalize Corrupted Bytes
Normalization operates on decoded Unicode text.
If a UTF-8 byte sequence was decoded with the wrong charset and became mojibake, normalization normally does not magically recover the intended original bytes.
The third pillar, Mojibake, owns the wrong-decoder problem.
Malformed UTF-8 Is an Earlier Error Boundary
Before normalization, the byte stream must decode into valid Unicode scalar values according to the required decoder policy.
Invalid byte sequences are not an NFC/NFD problem.
The fourth pillar, Malformed Input, owns decoder error handling.
Normalization Can Remove Distinctions on Purpose
NFKC/NFKD compatibility decomposition can make some distinct encoded forms compare the same.
This is useful in domains where the difference is irrelevant or dangerous.
It is lossy if the application later wishes it had preserved the distinction.
Compatibility normalization therefore belongs conceptually beside the site’s Information & Representation work: usefulness comes from discarding a distinction under an explicit job.
Normalization Should Have a Declared Scope
Free text display.
Search key.
Identifier.
Cryptographic canonical form.
Legacy round-trip.
The same transformation policy should not be assumed safe in every one of these contexts.
A Better Normalization Model
decoded Unicode text → define comparison/identity job → choose canonical or compatibility equivalence → normalize → preserve source if needed → compare/index/store under explicit policy → verify receiver expectations
A 30-Lens Unicode Normalization Audit
- Input: are we operating on valid decoded Unicode text?
- Job: display, search, identifier, hash or storage?
- Equivalence: canonical or compatibility?
- NFD: would decomposition help?
- NFC: is composed canonical text preferred?
- NFKD: can compatibility distinctions be discarded?
- NFKC: is compatibility folding justified?
- Canonical ordering: are combining marks normalised?
- Binary comparison: are both sides in the same form?
- Case: is separate case folding required?
- Collation: is locale-sensitive comparison required?
- Grapheme: is user-perceived segmentation a separate need?
- Search: which matching transformations follow?
- Identifier: what additional security profile applies?
- Confusables: can lookalike scripts still deceive?
- Hash: are exact bytes or canonical text being identified?
- Signature: was canonicalization defined before signing?
- Original: should source representation be retained?
- Ingest: normalize at write time?
- Compare: normalize at comparison time?
- Concatenation: can joins disturb the normalized form?
- Streaming: is boundary state preserved?
- Idempotence: does re-normalization remain unchanged?
- Version: which Unicode normalization data apply?
- Legacy: is round-trip compatibility required?
- Security: are transformations being used as a false anti-spoofing solution?
- Mojibake: was decoding correct before normalization?
- Malformed input: were invalid byte sequences handled upstream?
- Loss: what distinction does compatibility normalization erase?
- Receiver: does the resulting equality match the application’s intended notion of sameness?
Laboratory 1: Same Appearance, Different Sequence
Create canonically equivalent composed and decomposed strings. Compare them before and after NFC and NFD. Observe how code-point sequence changes while user-perceived text remains equivalent.
Laboratory 2: Compatibility Loss
Choose a set of compatibility characters and compare NFC with NFKC. List which distinctions NFKC intentionally removes and whether that loss is acceptable for display text versus a restricted identifier.
Laboratory 3: Normalize Then Concatenate
Use a base character at the end of one normalized string and a combining mark at the start of another. Concatenate them and test whether the combined result remains normalized.
For Primary Readers
Write the same accented letter using one printed tile and then using a letter tile plus an accent tile. The page can look the same even though the pieces underneath differ.
For Secondary Readers
Explain canonical equivalence and why normalizing both strings before comparison can prevent false mismatches.
For Advanced Readers
Model Unicode normalization as a stable canonicalization transform over defined canonical or compatibility equivalence relations. Correct use depends on whether application identity is exact-byte, canonical-text or restricted-identifier identity.
Common Misconceptions
- “Normalization makes visually similar text identical.” It operates on defined Unicode equivalence, not arbitrary visual resemblance.
- “NFKC is simply a stronger NFC and therefore better.” It erases compatibility distinctions and can be inappropriate for ordinary text.
- “Normalization handles case-insensitive search.” Case folding is a separate operation.
- “NFC makes every character one code point.” Multi-code-point grapheme clusters remain common.
- “Normalization can repair mojibake.” Wrong decoding must be diagnosed at the byte/charset boundary first.
Research Corridor
- Unicode Standard Annex #15 — Unicode Normalization Forms.
- Unicode Normalization FAQ — practical guidance on NFC, NFD, NFKC and NFKD.
- Unicode 17 Core Specification — Normalization Definitions.
- eduKateSG — How Encoding Works.
Frequently Asked Questions
What is Unicode normalization?
It is a family of standard transformations that converts Unicode text into defined canonical or compatibility forms so equivalent sequences can be processed consistently.
What is the difference between NFC and NFD?
NFD uses canonical decomposition. NFC canonically decomposes and then recomposes sequences where permitted, producing a more composed canonical form.
When should NFKC be used?
Only when compatibility distinctions can deliberately be ignored under the application’s defined policy, often in restricted identifier or matching contexts—not blindly across arbitrary source text.
Final Thought: Equality Is a Policy Before It Is an Operator
The screen says the strings are the same.
The bytes say they are different.
Normalization works when the system first decides which differences are representational accidents—and which differences must remain part of identity.
ENCODING · FOUR PILLAR LEGS
Return to How Encoding Works, or continue through Grapheme Clusters, Mojibake and Malformed Input. Return to the Information & Representation Hub.