Two tokens look almost the same.
Instead of carrying both forward, a model combines them.
Two become one.
Repeat that operation across a sequence and the model becomes cheaper to run.
That is token merging.
Quick Read
Token merging reduces Transformer sequence length by combining token representations judged sufficiently similar or redundant. Unlike token pruning, which deletes selected tokens, merging attempts to preserve their combined contribution in a new representative token.
One-sentence answer: token merging is lossy because several distinguishable intermediate representations become one, preserving shared structure while collapsing differences among the merged tokens.
Merging Is More Conservative Than Deletion
Pruning says, “this token can disappear.”
Merging says, “these tokens are similar enough that one combined representative can stand in for them.”
The second operation preserves more aggregate information.
It also creates a different failure mode: the merged token may no longer preserve the distinctions that made the originals separately useful.
Similarity Is the Gate
Tokens can be compared by cosine similarity, feature distance, attention-derived relationships or learned criteria.
If two representations are close enough, they become candidates for merger.
The metric therefore decides which differences the model is allowed to forget.
A Representative Token Must Carry Mass
If ten image patches merge into one representative, the new token should not behave as though it came from one patch only.
Weighted averaging, token-size tracking or related mechanisms can preserve how much source mass the merged token represents.
The model forgets individuality but tries to remember multiplicity.
Vision Transformers Made Token Merging Intuitive
Large smooth regions of an image can produce many similar patch tokens.
Sky next to sky next to sky may not deserve full independent processing through every layer.
Merge those tokens and broad scene structure can remain while computation falls.
But a tiny object hidden inside a visually similar region can be swallowed by the merger.
Local Detail Is the Price
Once two positions share one token, later layers cannot treat the original positions completely independently unless extra positional or routing information survives.
That can blur fine spatial boundaries, small text, thin structures and tiny objects.
The representation becomes more global and less local.
Merging Can Be Progressive
Early layers may keep many tokens because local detail is still being interpreted.
Later layers may merge more aggressively once broader semantics emerge.
This mirrors a common compression principle: preserve fine detail until the system has enough context to know what can safely collapse.
Token Merging and Clustering Are Related
Both operations group similar representations.
Clustering usually produces a grouping as the output of analysis.
Token merging uses similarity operationally inside the model so future computation happens on fewer representatives.
One describes structure. The other rewrites the computational state.
Merging and Aggregation Are Not Identical Either
Aggregation may reduce many values to a statistic such as a mean or total.
Token merging typically operates on learned high-dimensional representations and may preserve token mass, position or other metadata.
The merged token is intended to continue participating in the model, not merely report a summary outward.
The Merge Schedule Matters
Merge too slowly and efficiency gains vanish.
Merge too quickly and distinct features collapse before the model understands them.
A fixed merge rate may work on ordinary inputs and fail on dense scenes or long, information-rich sequences.
Merged Tokens Can Become Semantically Strange
A representative token may stand for several positions or semantic fragments simultaneously.
Later attention layers then interact with a compressed object that never existed in the original input.
The merged token is a model-created construct.
It is useful precisely because it is not a literal source token anymore.
Language Makes Merging Harder
Two words can have similar embeddings and very different grammatical roles.
Two repeated names can refer to different people.
Two clauses can look semantically similar but differ in negation, time or modality.
Merging language tokens therefore requires much more than surface similarity.
Token Merging and KV-Cache Merging Meet Later
Token merging can reduce active sequence representations during forward computation.
KV-cache merging can compress stored historical attention state during autoregressive inference.
Both turn many states into fewer representatives, but at different points in the runtime.
Education: Good Notes Merge Repetition
A textbook may explain the same idea across several examples.
A good student notices the invariant and compresses the repeated examples into one principle plus a few exceptions.
This is an analogy, not Transformer mechanics. The danger is identical in structure: merge too aggressively and the exception disappears inside the rule.
A CivDJ View: Ask What the Representative Cannot Separate Anymore
Forward: does the merged model preserve ordinary accuracy?
Backward: which original tokens contributed to each representative?
Rotate: do small objects, rare words, boundaries or minority features disappear more often?
A merged token should be judged not only by what it represents well, but by which distinctions it can no longer express.
When Token Merging Is Exactly the Right Loss
Use it when token redundancy is high, similar tokens genuinely carry overlapping information, later layers benefit from shorter sequences, and merge schedules are validated against fine-detail and minority cases.
When Token Merging Becomes Dangerous
- Similarity metrics merge semantically distinct tokens.
- Small objects disappear inside large regions.
- Spatial boundaries blur.
- Rare linguistic distinctions collapse.
- Merge rates are fixed despite varying input complexity.
- Representatives are treated as if they preserve every original token independently.
A Practical Token-Merging Audit
- Metric: how is similarity measured?
- Schedule: when and how fast do tokens merge?
- Mass: does the representative retain token weight or size?
- Locality: can distant or unrelated tokens merge?
- Detail: which small structures degrade?
- Groups: do rare features merge disproportionately?
- Baseline: is the full-token model retained for comparison?
Research Anchors
Recent dynamic-token surveys continue to distinguish token merging from pruning because merging compresses multiple token states into representatives rather than deleting them outright, creating a different efficiency–fidelity trade-off.
Continue Through eduKateSG
Continue with How Lossy Works | Token Pruning, How Lossy Works | Clustering, and the companion article How Lossy Works | Sparse Attention.
Final Thought: Merging Saves Work by Turning Several Voices Into One
The question is whether the voices were truly saying the same thing.