What is mixture of experts (MoE)? How it works

What mixture of experts means in AI

Mixture of experts is a model design that splits part of the network into many smaller networks, called experts, and runs only a few of them for each token. A token is a small chunk of text, often a word or part of one. The part that decides which experts handle each token is called the router, or gating network.

Mixture of experts at a glance

Short form	MoE
What it is	A model made of many expert sub-networks plus a router
What runs per token	Only the few experts the router picks
Main benefit	A bigger model without a matching rise in compute
Main cost	Every expert still has to be stored in memory

Many experts, but only a few run per token

In a regular model, every parameter takes part in processing every token. Parameters are the numbers a model learns during training, and their count, in billions, is the "B" in many model names.

An MoE model doesn't work that way. It keeps a large pool of experts, and for each token the router sends the work to a few of them while the rest stay idle. When Noam Shazeer and colleagues introduced their MoE layer in 2017, it held up to thousands of experts, and a trainable gating network chose a sparse combination of them for each example.

A way to grow a model without growing the work

MoE is a form of conditional computation, which means running only the parts of a network that a given input needs. More parameters give a model more capacity to learn, but in a regular model each extra parameter also adds work on every token.

Conditional computation breaks that link. Shazeer's team reported more than 1000x more model capacity with only minor losses in computing efficiency. That trade, a much larger model for roughly the same work per token, is the reason to use MoE.

How a mixture of experts model works

An MoE model looks like a regular language model from the outside, but some of its layers are swapped for a set of experts and a router. Each token passes through the model layer by layer, and at every MoE layer the router decides which experts process it.

What happens to one token in an MoE layer (Mixtral 8x7B)

Step	What happens
Arrive	The token reaches the layer
Score	The router scores all 8 experts
Pick	The 2 highest-scoring experts process the token
Combine	Their outputs are added together
Move on	The result goes to the next layer

A router picks experts for each token

The router is a small network that looks at each token and chooses where it should go. In Mixtral 8x7B from Mistral AI, each MoE layer has 8 experts, and for every token the router picks 2 of them and adds their outputs together.

The choice is made fresh at every layer and for every token. The same word can go to one pair of experts in one layer and a different pair in the next. The router is not set by hand: Mistral trains the experts and the router at the same time, so the router learns which experts handle which tokens well.

Experts replace the feed-forward layers

In today's language models, the experts usually take the place of the feed-forward layers. A transformer is the design behind ChatGPT, Claude and most other chatbots. It alternates attention blocks, which relate each token to the others, with feed-forward blocks, which process each token on its own.

MoE leaves the attention blocks as they are and turns each feed-forward block into a set of experts. So the experts in Mixtral share the same attention blocks, which matters when you count the model's parameters.

Experts don't specialize by topic

The name suggests a "math expert" and a "biology expert," but that is not how the experts end up working. Mistral's researchers checked which experts Mixtral picked for different kinds of documents and found no obvious pattern by topic.

ArXiv papers, biology abstracts and philosophy texts were sent to the experts in very similar proportions. The choices lined up more with the structure of the text than with its subject, especially in the first and last layers. So an "expert" is better thought of as a slice of the model that the router has learned to use for certain kinds of tokens, not a specialist you could call on for a subject.

Total parameters vs active parameters

Every MoE model has two parameter counts: the total it holds and the active share it uses for each token. Launch posts often give both, and reading them correctly tells you how fast a model runs and how much memory it needs.

Two parameter counts in an MoE model

	Total parameters	Active parameters
Also called	Sparse parameter count	Activated parameters
What it counts	Everything the model holds	What runs for one token
Mixtral 8x7B	46.7B	12.9B
What it tells you	How much memory you need	How fast and costly each token is

Why Mixtral 8x7B has 46.7B parameters, not 56B

The name Mixtral 8x7B reads like 8 experts of 7B each, or 56B, but the model has 46.7B parameters in total. The gap comes from the shared parts: the experts only replace the feed-forward layers, and the attention layers are counted once, not eight times.

Of those 46.7B, Mixtral uses 12.9B for each token, about 28% of the model. Mistral says that is why it reads input and writes output at the same speed and cost as a 12.9B model.

What "A22B" in a model name means

Some MoE models put both counts in their name, and the number after "A" is the active count. Qwen3-235B-A22B from the Qwen team has 235 billion parameters in total and 22 billion activated for each token.

Its smaller sibling, Qwen3-30B-A3B, has 30 billion in total, with 3 billion of them active. When you see a name in this pattern, the first number is the size you have to store and the "A" number is roughly the size you are running.

Active parameters set speed, total sets memory

Active parameters decide how much computing each token takes, while total parameters decide how much memory the model fills. The Mixtral paper ties inference compute directly to the active count and memory cost to the total, which it rounds to 47B.

That gives a simple rule for reading a spec sheet. A model listed as 671B total and 37B active will respond roughly like a 37B model once it is loaded, but you still need room for all 671B. If you run models locally, the total usually decides whether the model fits at all.

MoE vs dense models: what you gain and lose

A dense model is the regular kind, where every parameter works on every token. MoE trades some of that simplicity for speed, and pays for it in memory.

MoE vs dense models

	Dense model	MoE model
Parameters used per token	All of them	A small share
Speed for its size	Slower	Faster
Training compute for a given quality	More	Less
Memory needed	Matches its size	Matches its total, not its active size
Complexity	Simpler	Router, balancing, more moving parts

A dense model uses every parameter

A dense model runs the same parameters on every input, so it does its full size's worth of work for every token it reads or writes. It is the design most people picture when they think of a language model.

The Switch Transformer authors put the contrast simply: deep learning models usually reuse the same parameters for all inputs, while MoE picks different parameters for each one. Qwen3 shows both approaches side by side, with two MoE models released alongside six dense ones.

Where MoE wins: speed and training cost

For the same amount of computing, an MoE model can hold far more parameters than a dense one. Shazeer's team reported better results than earlier models at lower computing cost, and later work showed MoE also trains faster.

The Switch Transformer team simplified the routing step and reported up to 7x faster pre-training than a comparable dense model using the same computing resources. At serving time, the gain shows up as speed: a model that runs only its active share answers faster than a dense model of the same total size.

Where MoE loses: memory and complexity

The biggest cost is memory, because every expert has to be loaded even though only a few work at a time. Mixtral needs memory for its 47B total parameters, not the 13B it uses per token.

MoE is also harder to build and run. The Switch Transformer paper names complexity, communication costs and training instability as what held MoE back, and the router has to be kept from overloading a few favorite experts, a problem known as load balancing. Running a very large MoE model on a single PC is still possible with careful setup, as we show in "GLM 5.2 local: run 744B on a 24 GB PC, no GPU."

MoE models from the 2017 paper to today

MoE went from a research idea to the design behind many of today's largest open models in under a decade. The chart below shows how small the active share has become in open-weight models you can download now.

Share of parameters active per token

Active parameters as a share of total (%)

Calculated from each developer's published total and active counts.

28Mixtral 8x7B

9Qwen3-235B-A22B

5.5DeepSeek-V3

01020

The 2017 paper and the Switch Transformer

The 2017 paper by Shazeer and colleagues showed that MoE could work at scale in modern neural networks. Its largest MoE held up to 137 billion parameters and was used for language modeling and translation.

In 2021, the Switch Transformer simplified the routing step and trained models with up to a trillion parameters. It also tackled the training instability that had kept MoE out of wider use.

Mixtral 8x7B brought MoE to open weights

Mixtral 8x7B, released by Mistral AI on December 11, 2023, made MoE something anyone could download and run. Mistral published it as a sparse mixture of experts model with open weights under the Apache 2.0 license.

Mixtral ran at the speed of a 12.9B model while holding 46.7B parameters. Open models such as DeepSeek-V3 and Qwen3 later used the same idea at a much larger scale.

DeepSeek-V3 and Qwen3 run a small share of their weights

Newer MoE models push the active share much lower than Mixtral did. DeepSeek-V3, described in a December 2024 technical report, has 671B total parameters and uses 37B for each token, about 5.5%.

Qwen3, released on April 29, 2025, includes two MoE models, each with 128 experts and 8 active per token; the larger one runs about 9% of its weights per token. Open-weight MoE releases have kept coming since, such as "Reflection Beam is a 501B open-weight MoE model."

MoE runs a few experts per token to save compute

Mixture of experts is a model design that holds many expert sub-networks and runs only a few of them for each token, chosen by a router. The experts sit where a regular model has its feed-forward layers, and they divide the work by the kind of token, not by subject.

When you read a model name or spec sheet, look at two numbers. The active count tells you how fast and costly each token is, and the total count tells you how much memory the model needs. That gap is MoE's main benefit and its main catch.