MoE Active Parameters Calculator
Inputs
| Total experts | 8 |
|---|---|
| Active experts (top-k) | 2 |
| Parameters per expert | 7 |
| Shared parameters | 1 |
MoE Active Parameters Calculator
Compute the total and active parameter counts of a Mixture-of-Experts model from the expert count, top-k routing, per-expert size, and shared weights, separating memory footprint from per-token compute.
Inputs
Model
Results
Enter a value to see results.
Details
MoE Active Parameters
A Mixture-of-Experts model carries a striking pair of numbers: a large total parameter count that describes how much it can hold, and a much smaller active count that describes how much work it does per token. The two are often confused, yet they answer different questions — one sets the memory bill, the other sets the compute bill. This calculator separates them from the expert count, the top-k routing, the per-expert size, and the shared weights.
How a Mixture of Experts is built
In a dense transformer, every token passes through every weight. A Mixture of Experts replaces the feed-forward block of a layer with many parallel experts and adds a small router that, for each token, picks only the top-k of them. The rest of the model — attention, the router, the embeddings, and sometimes a shared expert that always runs — stays common to all tokens. So a token's path touches the shared weights and just the handful of experts it was routed to, while the remaining experts sit idle for that token.
This decouples two quantities that move together in a dense model. Capacity grows with the number of experts, because more experts can store more specialized knowledge. Compute grows only with the active experts, because only those do arithmetic on a given token.
The two counts
With total experts, active per token, parameters per expert, and shared parameters, the total and active counts are
TAr=s+E⋅p=s+k⋅p=TAwhere is the total that determines the memory footprint, is the active count that determines per-token compute, and is the fraction of the weights touched by each token.
Worked example
Consider a model with eight experts of seven billion parameters each, routing two per token, on top of one billion shared parameters:
TAr=1+8×7=57 billion=1+2×7=15 billion=5715≈0.263=26.3%The model holds 57 billion parameters but spends compute as if it were a 15-billion-parameter model on each token — roughly a quarter of the weights are active at a time. That is the appeal: the capacity of a 57-billion model at close to the per-token cost of a 15-billion one.
Memory still follows the total
The catch is that the savings are in compute, not memory. Because any expert can be chosen for any token, and a batch of tokens spreads across many experts, all of them must be resident at once — the model cannot predict which experts a future token will need. So the memory footprint scales with the total count , while only the compute scales with the active count . A Mixture-of-Experts model is therefore large to host but cheap to run, the opposite balance from a dense model of equal per-token cost.
The total count that drives the memory side can be built up from architecture with the Transformer Parameter Count Calculator, and the memory needed to actually serve those weights is covered in the LLM Inference VRAM Calculator.
Frequently Asked Questions (FAQ)
Why use a Mixture of Experts?
A Mixture of Experts lets a model grow its parameter count — and therefore its capacity to store knowledge — without a matching rise in the compute spent on each token. The router sends every token to only a small number of experts, so the floating-point work per token tracks the active parameters rather than the total. This makes it possible to train and serve a model with far more parameters than a dense model of the same per-token cost.
What is the difference between total and active parameters?
Total parameters count every weight in the model: the shared layers plus all of the experts. Active parameters count only the weights used on a single token: the shared layers plus the few experts the router selects. The total parameter count determines the memory footprint, because all the experts must be available to be chosen. The active count determines the per-token compute and latency, because only the routed experts perform work.
If only a few experts run per token, why does memory scale with the total?
Any expert can be selected for any token, and over a batch of tokens many different experts are used, so all of them must be resident in memory at once.
The model cannot know in advance which experts a future token will need. As a result the memory footprint scales with the total parameter count, even though the compute for any single token scales only with the active count. This is the central trade-off of the approach: cheap compute, but the full weights still have to be held.
Disclaimer
This is a first-order estimate that assumes uniform expert sizes and treats shared weights as always active. It ignores routing imbalance, the small router network itself, and any duplication from expert parallelism across devices. Use it to compare designs, not as an exact accounting.