A mixture-of-experts model contains specialists.
One handles one region of the input space.
Another handles another.
Then an efficiency engineer asks:
Do we really need all of them separately?
That question leads to expert merging.
Quick Read
Expert merging reduces the number of distinct expert modules in a mixture-of-experts model by combining experts judged redundant or sufficiently similar. Contemporary MoE optimisation surveys distinguish expert merging from pruning: pruning removes experts, while merging attempts to preserve several experts’ shared function inside fewer modules.
One-sentence answer: expert merging is lossy because several independently trained specialist functions are collapsed into fewer parameter sets, preserving common behaviour while sacrificing some specialised freedom.
Why Experts Become Expensive
Mixture-of-experts models can hold enormous parameter capacity while routing each token through only a small number of experts.
This reduces per-token arithmetic relative to activating every parameter.
But all experts still need to be stored, loaded, distributed or made available somewhere in the serving system.
Memory and communication become major deployment costs.
Redundant Experts Are the Opportunity
Training does not guarantee perfectly differentiated experts.
Several experts may learn similar transformations or serve overlapping token populations.
If two experts are functionally close, storing both may buy little additional capability.
But Similar Parameters Do Not Guarantee Similar Function
Two neural networks can implement similar behaviour with different internal parameter arrangements.
Conversely, two experts with superficially similar weight statistics can diverge strongly on particular inputs.
Expert similarity should therefore be tested functionally, not only geometrically.
Parameter Averaging Is the Simplest Merge—and Often Too Simple
Average two sets of weights and the result sits between them numerically.
It does not follow that the resulting expert sits between them behaviourally in a useful way.
Nonlinear networks can be sensitive to parameter alignment, permutation and activation geometry.
A successful merge may require matching, alignment or recovery fine-tuning.
Routing History Reveals Functional Overlap
If two experts receive similar tokens, produce similar outputs and contribute similarly to downstream loss, they are stronger merge candidates.
Routing data therefore becomes evidence about specialisation.
The system asks not merely “are these experts close?” but “do they solve the same reader job inside the model?”
The Router Must Change Too
Merge experts and the routing space changes.
Tokens previously assigned to two separate specialists may now share one destination.
Load balancing, expert capacity and communication patterns can all shift.
Expert merging is therefore an architectural intervention, not merely a file-size operation.
Specialisation Collapse Is the Central Risk
One expert may look redundant on average while preserving a rare linguistic, domain or reasoning capability.
Merge it into a dominant expert and that niche can disappear.
The merged model becomes smoother and cheaper.
It may also become less plural.
Expert Diversity Is Not Waste by Definition
Diversity can look inefficient because several experts solve overlapping cases.
But overlap can provide robustness, load balancing and alternate routes under distribution shift.
Removing diversity can make the model more brittle even when average benchmark scores remain stable.
Expert Merging and Pruning Are Different
Expert pruning says one specialist can disappear.
Expert merging says several specialists should become one combined specialist.
Pruning discards.
Merging attempts to preserve common function while sacrificing independence.
Expert Merging and Parameter Sharing Also Differ
Parameter sharing constrains components to use the same weights by design or training architecture.
Expert merging usually begins after specialists have already differentiated and then collapses them.
One prevents independence from emerging.
The other removes independence after learning.
Recovery Fine-Tuning Can Hide the Merge Scar
After merging, the model can be fine-tuned so the combined expert adapts to its wider workload.
Headline performance may recover.
Yet rare behaviours can still vanish.
Recovery must therefore be measured by capability slices, not only global loss.
Distributed Serving Makes Expert Count a Systems Problem
Experts may be distributed across devices.
Every routing decision can imply communication.
Fewer experts can simplify placement, reduce memory footprint and lower all-to-all communication pressure.
This is why contemporary 2026 MoE surveys study expert merging alongside routing, scheduling and hardware-aware optimisation.
Education: Combining Two Specialists Can Produce a Generalist
A mathematics specialist and a literature specialist can both teach reasoning.
Merge their common principles into one general reasoning framework and efficiency improves.
But if the merger discards notation from mathematics or close reading from literature, the generalist has lost the very expertise that made the specialists valuable.
This is an analogy, but the structural lesson holds: commonality and specialisation must be separated before compression.
A CivDJ View: Rotate the Specialist Before You Merge It
Forward: do the merged experts preserve ordinary performance?
Backward: what unique inputs did each original expert handle?
Rotate: do domain, language, rare-class and distribution-shift behaviours survive?
The merger is safe only when overlap remains overlap under more than one viewing angle.
When Expert Merging Is Exactly the Right Loss
Use it when experts demonstrate functional redundancy, deployment is limited by expert memory or communication, routing can be recalibrated, and unique specialist capabilities are explicitly tested before and after the merge.
When Expert Merging Becomes Dangerous
- Similarity is measured only from weights rather than behaviour.
- Rare specialist capabilities disappear.
- Router load becomes concentrated after merging.
- Recovery fine-tuning hides subgroup regressions.
- Expert diversity needed for distribution shift is removed.
- The original expert configuration is discarded before comparison is complete.
A Practical Expert-Merging Audit
- Similarity: parameters, outputs, routing overlap or all three?
- Specialisation: what unique traffic does each expert own?
- Merge rule: averaging, alignment, learned fusion or another method?
- Router: how is routing recalibrated?
- Load: does traffic become imbalanced?
- Edges: which rare domains regress?
- Master: is the pre-merge model retained?
Research Anchors
A 2026 ACM Computing Surveys review of mixture-of-experts inference explicitly includes expert merging among model-level optimisation strategies and warns that compression must preserve expert diversity rather than collapse specialisation indiscriminately.
Continue Through eduKateSG
Continue with How Lossy Works | Parameter Sharing, How Lossy Works | Activation Sparsity, and the companion article How Lossy Works | Sparse Attention.
Final Thought: Specialists Look Redundant Until the Rare Case Arrives
Expert merging is powerful when overlap is real.
It becomes dangerous when common behaviour is mistaken for complete equivalence.