Mixture-of-experts models route tokens to selected learned subnetworks, but token capacity, expert overload and load balancing can determine whether a theoretically sparse architecture really works. This guide explains routing and checks with original numerical cases, drawing on primary Switch Transformer and expert-choice research.
Contents
- What a mixture-of-experts layer actually does
- Dense versus sparse computation: total parameters are not active parameters
- How a token-choice router selects an expert
- Expert capacity: what does a token slot mean?
- Worked example: 120 tokens and a surprising overflow
- What does it mean when an expert overflows?
- Balancing losses help but do not impose hard guarantees
- Expert-choice routing reverses the selection direction
- Latency and the hidden cost of moving activations
- Does each expert become a human-readable specialist?
- Training stability, precision and representation drift
- Who loses when one expert is congested?
- Nine engineering errors and how to repair them
- Changed-context exercise: an eight-expert batch with top-2 routing
- Limits of the mechanism and research evidence
- Frequently asked questions
- Reader pathways and original publications
What a mixture-of-experts layer actually does
A mixture-of-experts neural network keeps several candidate subnetworks, often called experts, and a router decides which experts process a particular token or example. In a sparse MoE layer, only a subset of experts is activated for each token. This can increase the model’s total parameter capacity without evaluating every expert for every token. The word expert does not mean each subnetwork is a verified human specialist; it describes learned components within a statistical model.
The useful engineering answer is to inspect routing, per-expert capacity and overflow together. A router may send many tokens to the same popular expert. If that expert has a finite token budget per batch, some assignments cannot be processed under the intended path. Training must therefore balance useful specialisation against congestion, communication cost and reliability. Model quality cannot be judged from total parameter count or router confidence alone.
A Switch Transformer-style token-choice router can select the highest-scoring expert for each token, whereas other MoE systems use top-k selections or expert-choice assignment. These are different computational contracts. Expert-choice routing allows each expert to choose tokens up to a fixed capacity and a token may be selected by a variable number of experts. Its load behaviour should not be mechanically confused with top-1 token choice.
The primary sources are Fedus, Zoph and Shazeer, Switch Transformers (JMLR 2022) and Zhou and colleagues, Mixture-of-Experts with Expert Choice Routing (NeurIPS 2022). This article explains their architectural distinctions but does not assert that one method is universally optimal or that any named production model uses a particular unpublished configuration.
Dense versus sparse computation: total parameters are not active parameters
A dense feed-forward layer uses the same learned weight matrices on every token entering that layer. A model with several expert feed-forward networks can store many more parameters, but a router may activate only one or two of them for a given token. The distinction between total stored parameters and active per-token parameters is central to understanding MoE scaling.
Suppose four expert subnetworks each contain two million parameters and the router activates exactly one expert per token. Across experts, the system stores eight million expert parameters, ignoring shared layers and router weights. For a token processed by just one expert, only about two million expert parameters participate in that expert computation. The exact arithmetic cost also depends on matrix dimensions, activation functions, kernels and hardware, so do not turn a simple parameter count into a precise floating-point operation estimate.
Training still incurs storage, optimisation and communication costs. Expert weights may be sharded across devices, so tokens or activations must move to the chosen expert and results must return. A bottleneck at one expert can delay an entire synchronised step. Sparse activation does not magically eliminate network latency, memory overhead or the need to maintain many parameters.
The JMLR Switch Transformers work specifically addresses practical challenges such as complexity, communication costs and training instability. Its reported speedups come from its studied models, baselines and hardware resources. They should not be repeated as a universal guarantee that an MoE conversion will accelerate every local workload.
Treat sparsity as a design option with trade-offs. Dense models avoid routing-related overflow, though they face their own compute limits. Sparse models can offer attractive capacity efficiency, but the router becomes part of the quality and reliability system rather than a simple dispatch utility.
How a token-choice router selects an expert
A routing network takes a token representation and produces a score for each available expert. One common implementation converts scores to softmax probabilities. For a token with four routing logits, a top-1 policy selects the highest-scoring expert and sends that token’s representation there, subject to capacity or dispatch constraints.
The routing probabilities need not be calibrated statements of semantic truth. A 0.7 score for expert A does not mean a human can say with 70% confidence that A is the correct domain specialist. The values arise from an optimised neural routing objective and may be affected by shared representations, auxiliary balancing terms and training instability.
Top-2 or top-k routing may select multiple experts per token and combine their outputs with learned weights. That increases the available computation and may improve coverage for some inputs, but it consumes more expert slots and communication. The choice of top-k is an architectural hyperparameter, not a free quality improvement.
Routers can be deterministic conditional on fixed logits and tie-breaking rules, or use noise or other stochasticity in training. This affects how assignments spread across experts and whether a rare expert receives enough examples. Reproducibility requires recording the routing algorithm, capacity policy, random seeds where relevant and expert topology.
An engineering review should compare the actual dispatch map with the model’s intended mathematical description. If the implementation accidentally routes every token to expert zero, a large sparse model might behave like a much smaller and unbalanced one while still producing syntactically valid output.
Expert capacity: what does a token slot mean?
An MoE implementation usually cannot route an unbounded number of tokens to one expert in a single batch while promising fixed memory and compute. It therefore budgets a certain number of token assignments per expert. The exact capacity formula depends on the implementation, token count, number of experts, capacity factor, dispatch strategy and any rounding or grouping rules.
For a simple illustrative top-1 system with T tokens and E equally provisioned experts, the average token load is T/E. A capacity factor c might be translated into a per-expert budget near ceil(c×T/E). This formula is a pedagogical example, not a universal implementation contract. Real systems can have expert parallel groups, padding constraints, dedicated shared experts or distinct capacity handling during training and inference.
Suppose T=120, E=4 and c=1.2. Average load is 120/4=30 tokens per expert, and illustrative capacity is ceil(1.2×30)=36 token slots per expert. Total nominal slot capacity is 4×36=144, exceeding the 120 assignments by 24 slots. Yet a router that sends 60 tokens to one expert can still overflow that expert’s 36 slots despite ample unused capacity elsewhere.
That distinction is the heart of routing congestion. Global spare capacity does not guarantee local acceptance. The distribution of assignments matters, as do overflow handling and router training. If no mechanism redistributes the extra tokens, the model may bypass their expert computation or apply some defined fallback.
Some models pad each expert’s batch to capacity to simplify hardware execution. Others use variable shapes or alternative dispatch kernels. Padding wastes work when experts are unevenly loaded; irregular communication creates another cost. The capacity factor trades memory and throughput against overflow risk, not just a mathematical target.
Worked example: 120 tokens and a surprising overflow
Consider a fictional top-1 router with four experts, numbered A through D. There are 120 tokens in one routing group and each is initially assigned to exactly one expert. The counts are A=60, B=30, C=20 and D=10. They sum to 60+30+20+10=120, so every token is accounted for. Under the illustrative per-expert capacity of 36 slots, A exceeds its budget by 24 tokens while the other experts do not.
If the implementation simply accepts the first 36 assigned tokens at each expert and bypasses expert computation for the 24 overflow assignments, exactly 24/120=20% of token assignments overflow at this layer. The other 96/120=80% have an accepted route. There are 144 available slots globally but only 96 are occupied because unfilled capacity in B, C and D cannot automatically absorb A’s excess under this policy.
Expert utilisation relative to the configured 36 slots is A=36/36=100%, B=30/36≈83.33%, C=20/36≈55.56% and D=10/36≈27.78%. The sum of accepted assignments is 36+30+20+10=96. Total slot utilisation is 96/144=66.67%. These denominators matter: a 20% overflow rate refers to tokens, while 66.67% utilisation refers to all provisioned expert slots.
Now suppose a balancing intervention changes assignment counts to A=36, B=32, C=28, D=24 while keeping 120 assignments. Every expert remains within capacity and no assignment overflows in this invented example. But that does not prove the balanced router has better prediction quality. Some tokens may have been sent away from their preferred learned expert, changing the network’s output.
The lesson is that capacity, overflow and task loss must be measured together. A routing distribution can be operationally tidy yet semantically poor, or accurately specialised yet costly and congested. An evaluation that reports only aggregate route entropy or parameter count misses the engineering trade-off.
What does it mean when an expert overflows?
The word dropped can mislead readers into thinking an entire input token disappears from the neural network. In some MoE designs, an over-capacity token skips the expert sublayer, perhaps retaining the residual pathway. In others it may be redirected, processed by a shared expert, delayed, or handled by a different inference schedule. The exact behaviour belongs to the implemented layer.
That difference affects correctness. A residual path may preserve numerical stability while providing less conditional expert processing for affected tokens. Rerouting to a less preferred expert may preserve compute but alter learned specialisation. Waiting for spare capacity may increase latency. There is no universally correct overflow response across tasks and deployment budgets.
Overflow can also vary by token position and input type. If the first tokens in a group have priority, later tokens may suffer systematically when an expert is popular. If token order correlates with language, document segment or user cohort, this can introduce uneven errors. A fair audit looks at overflow by meaningful subgroup, not merely total average.
The Expert Choice Routing research notes that poor balance and finite expert capacity can leave tokens unprocessed by the intended MoE layer. It proposes a distinct assignment method to address these limitations in studied models. This is evidence for a concrete architectural problem, not a guarantee that all implementations of expert choice eliminate every task-quality trade-off.
To diagnose overflow, log aggregated expert loads and accepted counts without leaking private token content. Align measurements to the same routing group and capacity policy. If groups vary dramatically in size, comparing absolute overflows without denominators can hide the actual rate.
Balancing losses help but do not impose hard guarantees
Training may add an auxiliary load-balancing term to the main predictive loss. Such a term encourages the router to spread assignments or routing probability mass across experts rather than repeatedly overusing a few. Its exact formula differs across architectures. A small coefficient may fail to affect a strongly peaked router; an excessively large coefficient can overwhelm the primary task and force unhelpful allocations.
Balancing signals can be defined from actual top-choice counts, mean router probabilities or other differentiable proxies. Those quantities are related but not identical. A router can have fairly even average probabilities while its argmax choices pile onto one expert. Conversely, an apparently balanced top-choice count can conceal very unequal confidence and training gradients.
An auxiliary loss is a soft preference during optimisation, not a hard capacity check at execution time. The paper on Expert Choice Routing explicitly observes that auxiliary balancing does not guarantee equal loads under token-choice routing. A model still needs a defined overflow or capacity management policy. Removing that policy because a training curve looks balanced is unsafe.
Distinguish training and inference. Some implementations use capacity limits primarily during training, or alter routing groups and batching at inference. Others dynamically schedule token assignments. A model trained with one overflow regime can behave differently under another. Compare the intended deployment pathway with the training and evaluation setup.
Watch for collapse during the early phase of training, when few experts receive most tokens. An expert deprived of inputs may not learn features that would later make it attractive, reinforcing the imbalance. Balancing loss, initialisation and routing noise can affect that dynamic. Treat it as an empirical optimisation problem requiring instrumentation.
Expert-choice routing reverses the selection direction
In conventional token-choice routing, each token chooses a fixed number of experts, such as one or two. A crowded expert may then receive more requests than its slot budget allows. Expert-choice routing changes the direction: each expert selects a fixed number of high-affinity tokens. This makes the expert’s assignment count controllable, but individual tokens may be selected by varying numbers of experts.
Consider an illustrative group of 12 tokens and three experts with a budget of four token slots each. Under expert choice, A selects its four favourite tokens, B selects four, and C selects four. Every expert has exactly four assignments, so all 12 slots are filled. However, if A and B both select token 3, that token receives two expert assignments. Another token may receive zero. The 12 assignments do not necessarily cover 12 distinct tokens.
That is an important difference from saying ‘every token gets one expert’. The assignment count and unique-token coverage are different denominators. Expert choice can balance expert loads by construction under its selection rule, while requiring the model to handle variable experts per token and potentially unselected tokens.
Zhou and colleagues describe this heterogeneous token-to-expert mapping in their NeurIPS 2022 paper. They report stronger efficiency and task results than selected studied baselines under matched conditions. Those experimental outcomes should be attributed to that comparison, not promoted into a universal preference for all workloads.
Choosing between token choice and expert choice depends on the model’s training objective, batch structure, hardware communication, inference latency and requirements for token coverage. A simple implementation is not automatically worse than a mathematically elaborate assignment, especially under tight serving constraints.
Latency and the hidden cost of moving activations
Routers often send activations to experts distributed across accelerators. This may require an all-to-all communication pattern, local expert computation and return movement. If expert A takes much longer than B, a synchronised operation may wait for A. A model can have the same number of active expert parameters per token but very different wall-clock latency depending on balance and hardware layout.
Suppose four devices host one expert each. Under one hypothetical group, expert compute times are 4, 4, 4 and 4 milliseconds, plus a separate 2 milliseconds for communication. A rough synchronised step might need about 4+2=6 milliseconds under this simplified model. Under an imbalanced group with expert times 10, 2, 2 and 2 milliseconds, the same simplistic bound becomes 10+2=12 milliseconds. This is illustrative timing arithmetic, not a real benchmark.
Total batch throughput may not track single-request latency. Large batches make it easier to amortise dispatch overhead yet can create uneven group loads. Small batches can underutilise experts and suffer disproportionate communication overhead. The right reporting unit depends on whether the service prioritises tokens per second, request latency, energy or another constraint.
Padding can hide inefficiency. If each expert processes a padded tensor of 36 slots but only one has 36 real tokens, some computation or memory may be spent on empty positions. Alternative kernels reduce some waste, yet their complexity and memory behaviour also deserve measurement.
An honest comparison reports hardware, batch size, sequence lengths, expert parallelism, capacity policy and achieved quality. Parameter sparsity without these details is not a trustworthy cost claim.
Does each expert become a human-readable specialist?
A trained MoE may develop different distributions of activity across input features, languages, task types or token patterns. But expert identity is learned from optimisation, not necessarily an explicit taxonomy created by teachers or engineers. Calling one expert ‘maths’ because it activates on a few equations can be an overinterpretation of anecdotal examples.
Inspect route distributions on representative held-out inputs to discover patterns, while recognising that correlation is not a causal description of an expert’s function. An expert might respond to syntax, frequency, token position or mixed features that are difficult to name. Apparent specialisation may also change during fine-tuning or after a routing update.
Probe with interventions cautiously. If disabling expert B changes performance on a subgroup, that suggests B contributes under those conditions, but other experts and residual pathways may compensate. A small set of prompt examples cannot establish a unique role. Carefully designed ablations and error analyses are needed.
Highly specialised routing is not always desired. A task may need information that spans several learned patterns. Top-k routing, shared experts or alternative architecture choices can trade specialisation for coverage. The optimisation target should be task usefulness under real constraints rather than an appealing story about invisible experts.
For educational explanations, the analogy to a team of specialists can help intuition only if its limitations are explicit. Neural experts do not possess human responsibility, independent goals or verified occupational credentials. They are computational modules whose outputs require statistical and operational evaluation.
Training stability, precision and representation drift
MoE models introduce additional optimisation concerns. Router scores and gradients interact with expert load. Some experts may receive few tokens early, leading to poor learning and persistent disadvantage. Router outputs can become extremely peaked, causing imbalance or unstable gradients. Mixed-precision execution and communication may introduce further numerical considerations.
The Switch Transformer paper describes methods intended to make sparse models easier to train and transfer at scale, including attention to lower-precision formats and simplified top-1 routing. ST-MoE work continues the design discussion for stability and transfer. These original studies justify investigating stability mechanisms, not claiming all MoE configurations are inherently stable.
Record router statistics over time: selection counts, dropped or bypassed assignment rates, auxiliary loss, main task loss, gradient health and per-expert update frequency. Sudden changes can precede task-quality failures. Validation should include both model outputs and these operational signals.
Checkpoint resumption deserves care. A saved model must include the router, expert weights, optimiser state where needed and configuration defining expert identities and device mapping. Reloading experts in a different order without consistent mapping can silently change outputs even if each weight tensor retains plausible dimensions.
Fine-tuning may update only part of a sparse model. That can be useful economically but can also distort routing to fixed experts. The evaluation must match the actual trainable parameters and downstream tasks. A successful training script exit is not proof that the deployed router dispatches tokens as intended.
Who loses when one expert is congested?
Aggregate accuracy can hide asymmetric failures. Suppose one language’s tokens regularly prefer expert A while other languages spread across B, C and D. If A is often over capacity, the first language may experience more bypassed expert computation, even when overall overflow is modest. This is a hypothesis to test, not a claim about a particular existing language model.
Instrument overflow by relevant cohort, input length, task category and device where privacy and sample sizes allow. Report both the numerator of affected assignments and the number of eligible assignments for that group. If 40 of 100 assignments in one group overflow, its rate is 40%, regardless of the global system’s much lower average.
Changes to balancing may improve a disadvantaged group or harm another. A redistribution that reduces overflow can send tokens to experts that have less training on that subgroup. Fair evaluation considers both acceptance of routing and the final user-facing error, not just equal dispatch counts.
Be careful when grouping sensitive information. Collect only what is necessary, protect private content and avoid storing raw inputs merely to label an expert as associated with a demographic category. Many relevant questions can be answered through aggregated metrics and authorised test sets.
A decision to deploy a sparse layer is still subject to safety requirements outside its architecture. Numerical predictions affecting people need appropriate calibration, independent testing and human escalation regardless of how efficiently experts are routed.
Nine engineering errors and how to repair them
Error one is confusing total parameter count with active per-token work. Repair by reporting both. Error two is using an average capacity calculation without inspecting per-expert distribution. Repair by logging actual accepted and overflowed assignments. Error three is changing the capacity factor and calling the model faster without measuring memory, latency and output quality.
Error four is interpreting auxiliary load-balancing loss as a guarantee that no expert will overflow. Repair by retaining a hard capacity policy and measuring its outcome. Error five is comparing token-choice and expert-choice counts as if both guarantee one expert per token. Repair by reporting assignments per expert and unique tokens covered separately.
Error six is assuming a bypassed token disappears entirely from the network. Repair by reading the implementation’s overflow pathway, residual connection and final output semantics. Error seven is treating routing confidence as a calibrated probability that an expert is semantically correct. Repair with task-level outcome evaluation rather than labels invented for internal modules.
Error eight is drawing conclusions from only aggregate expert loads. Repair with time-series and appropriate subgroup analysis, especially for rare or difficult inputs. Error nine is benchmarking on hardware or batch sizes unlike deployment and then promising a universal speedup. Repair with reproducible, matched test conditions.
The repeated lesson is that a router is simultaneously a learned choice mechanism and a constrained scheduling system. Correctness requires understanding both. If a model’s output degrades, determine whether the cause lies in representation learning, routing decisions, capacity management, communication or evaluation before modifying the architecture.
Changed-context exercise: an eight-expert batch with top-2 routing
Transfer the method to a model using top-2 token choice. There are 80 input tokens and eight experts. Each token selects two expert assignments, so the total number of assignments is 80×2=160 rather than 80. Assume an illustrative capacity of 24 assignments per expert. Global slot capacity is 8×24=192, which exceeds 160 by 32 slots. That spare capacity still does not guarantee that every expert stays within its individual budget.
Suppose expert A receives 50 assignments, B receives 30, C through F receive 15 each, G receives 12 and H receives 8. Check the sum: 50+30+4×15+12+8=160. Capacity A can accept at most 24, leaving 26 overflow assignments; B can accept 24, leaving six overflow assignments; the others fit. Total overflow is 26+6=32 assignments, or 32/160=20% of assignments.
The numerator 32 is not necessarily the number of distinct tokens harmed. A token can have two selected experts; both, one or neither assignment might overflow. To quantify unique affected tokens, the evaluator must inspect the assignment map. Reporting 32/80=40% affected tokens without that map would be an unsupported inference.
Now imagine a revised router spreads assignments as eight experts with twenty each. The total remains 8×20=160 and none exceeds capacity 24. But the change could alter combinations of experts for tokens and thus prediction quality. The correct experiment holds test data and settings fixed and compares both task outcomes and routing measurements.
Finally suppose serving uses single-token decoding rather than the 80-token training group. Communication and expert availability may differ; the training capacity arithmetic is not sufficient to predict latency. The transferable principle is to specify the unit of accounting and deployment pathway before applying formulas.
Limits of the mechanism and research evidence
MoE is not a claim that machine learning models contain independent conscious specialists. The experts are trainable functions. Their names are convenient labels, not evidence of deliberate understanding or accountability. No artificial superintelligence is asserted or implied by the title of the surrounding site collection.
The simple capacity formula used here is a transparent teaching convention, not a claim about every MoE software stack. Real systems differ in grouping, rounding, expert replication, shared experts, top-k weighting, priority ordering and treatment of overflow. An implementation must state its actual dispatch rules.
Original Switch and expert-choice publications report experiments on defined architectures, data and hardware. Their speedups and task results are not automatically current benchmarks for an arbitrary new system. Model performance and latency may change under other sequence lengths, memory budgets, routing patterns and deployment settings.
Balancing can improve utilisation while reducing useful specialisation. A large capacity factor can reduce overflow but increase memory cost; too little capacity can create bypass or latency. There is no single optimal setting independently of the intended task and service limits.
Finally, counting correct assignments is not the same as checking external truth. A model may route every token without overflow and still produce a false statement or an unusable answer. Responsible applications need task-specific verification beyond MoE architecture.
Frequently asked questions
What is mixture of experts? It is an architecture with several learned subnetworks and a router that selects some of them for each token or example. Sparse MoE activates only part of the total expert parameter set for each input.
Does each expert correspond to a human field like medicine or mathematics? Not necessarily. Expert roles are learned and can be mixed or uninterpretable. Human-readable names require evidence, not anecdotes.
What does top-1 routing mean? Each token initially selects one expert according to router scores, subject to capacity and execution constraints.
What is expert capacity? A per-expert budget for token assignments in a routing group. Overflow occurs when requests for that expert exceed the budget under a defined policy.
Does overflow mean the input token vanishes? Not necessarily. Some layers skip that expert computation while preserving a residual path; other systems reroute or reschedule. Check the implementation.
Why balance experts? Concentration can overtrain some experts, undertrain others and create memory or latency bottlenecks. Balancing itself can also distort preferred assignments.
What is expert-choice routing? Each expert chooses a fixed number of tokens, producing balanced expert slot counts but potentially a variable number of selected experts per token.
Is MoE always faster than a dense model? No. It may offer capacity efficiency, but communication, batching, memory, routing and target quality determine actual speed.
How do I verify an MoE system? Record routing assignment maps, accepted and overflow counts, per-expert loads, final task quality, latency, compute environment and any subgroup failures.
Reader pathways and original publications
If you study model architecture, begin with a two-expert toy router and enumerate which tokens choose each expert. Add a finite capacity and count overflows. Then change the selection rule to expert choice and compare assignment count with unique token coverage. The arithmetic makes the difference more concrete than a vague specialist-team analogy.
If you engineer production ML, test realistic batch and sequence shapes, overload scenarios, numerical stability and checkpoint migration. Keep overflow handling documented, and validate that a routing configuration from training matches the production graph. Pair infrastructure metrics with independent output correctness checks.
If you evaluate fairness or product usefulness, inspect both model outcomes and routing distribution for relevant subgroups. Do not promise improvement from balancing loss alone. For related topics, explore test-time augmentation for independent inference-time concerns, and How AI Works for the wider model pipeline.
Primary publications: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (JMLR 2022); Mixture-of-Experts with Expert Choice Routing (NeurIPS 2022); ST-MoE: Designing Stable and Transferable Sparse Expert Models; and the authors’ Google Research explanation of expert-choice routing.
All examples in this guide are invented teaching cases with stated assumptions. No model training or benchmark was executed for this article. For the broader educational reasoning approach, read The Gold Standard of Education.