Moe on lo0 — Tech Blog

Device-native architectures: fine-grained MoE and pre-attention router

Tue, 09 Jun 2026 01:50:00 +0000

This post belongs to the series on inference performance in small models. It is the architectural face of a problem we have already looked at from the compute-regime side (the SLM’s inverted roofline) and from the weight-loading side in From disk to HBM. Here the question is different: what if, instead of adapting a large model to the device, we design the model for the device from the first commit?

TL;DR

The default move to get an LLM onto a laptop, a phone or an edge box is to take a dense model built for cloud and compress it: distillation, pruning, quantisation. It is a gesture of reduction: you start from something big and take things away. SmallThinker (arXiv:2507.20984, SJTU IPADS + Zenergize AI) argues for the reverse gesture, designing from scratch, and articulates it in three pieces. First: fine-grained MoE, many small experts with very few activated per token, so that total parameters N (the capacity) decouple from activated parameters A (the compute cost per token). Second: sparse FFN, ReLU-style activation sparsity inside each block, which adds a second level of sparsity on top of the first. Third: a pre-attention router that predicts which experts will be needed before running the attention block and launches the prefetch of those weights from SSD/flash in parallel with attention compute, hiding the storage latency, which is the real bottleneck when the model does not fit entirely in RAM. The authors report SmallThinker-4B-A0.6B and SmallThinker-21B-A3B exceeding ~20 tok/s on consumer CPUs with Q4_0, using ~1 GB and ~8 GB of RAM. The numbers are interesting and the direction is right; the evaluation methodology and the quality cost of activating so little deserve scepticism, and that is what the last part is for.

The analogy: the librarian who anticipates your request

Picture a huge library with a small reading room. You are sitting in the room at a single desk: only a few books fit on it at a time (that is RAM). The bulk of the collection is in the back room, on long shelves that are slow to walk (that is the SSD/flash). And there is a librarian.

The naive method: you read, you reach a point where you need a specific book, you request it, and then the librarian gets up, walks to the back room, looks for it and comes back. Meanwhile, you wait with the page open, making no progress. Every time you need a new book, you pay the full trip to the back room. The reading room spends most of its time waiting, not reading.

The SmallThinker method: the librarian is clever and gets ahead of you. While you are still reading the chapter’s table of contents, working out what it is about and connecting ideas, which in the model is the attention block, they have already glanced over your shoulder, predicted which three or four books you are going to ask for and gone to the back room to fetch them. By the time you finish the contents and formulate the request, the books are already on your desk. You have not waited: the trip to the back room happened in parallel with your reading of the contents.

The analogy holds on four details:

The small desk is RAM; the slow back room is the SSD/flash.
The books are the MoE’s experts: only a few are on the desk at any moment.
Reading the contents is the attention block; requesting and using the books is the FFN/expert block.
The librarian who predicts and gets ahead is the pre-attention router: the prediction is made first, and the trip to fetch (the prefetch) overlaps with reading the contents (attention).

The quantitative question running through the whole post is: does the librarian get back in time? The wait is only hidden if the trip to the back room takes less than your reading of the contents. That is the condition t_{\text{attention}} \ge t_{\text{prefetch}}, and we will do it with numbers.

Compressing a dense model vs. designing for device

It is worth laying out the two approaches plainly, because they are not degrees of the same thing: they are different philosophies.

Approach A — compress a dense model built for cloud. You start from, say, a dense 7B–14B model trained to run on an RTX 4090 (24 GB, Ada Lovelace) or on a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). To fit it into a device you apply three levers, each with its own post: distillation (you train a small student that imitates the teacher), pruning (you remove weights or whole structures) and aggressive quantisation (you go down to 4 bits or fewer). The resulting model is still dense: all its parameters are activated on every token. You have reduced the number of parameters, but the compute pattern is the cloud’s, only smaller.

Approach B — design for device from scratch. Here the device’s constraints enter the architecture, not a later compression phase. The constraints are three and very concrete:

Weak compute. A laptop CPU or a mobile SoC does orders of magnitude fewer FLOPs than a datacenter GPU. This pushes you to minimise the activated parameters per token, not the total ones.
Little RAM. Tens of GB do not fit. This pushes you to keep resident only what is essential and to stream the rest.
Slow storage. The SSD or flash you are forced to stream from has far less bandwidth than a GPU’s HBM. This turns storage I/O into the bottleneck, and pushes you to hide it.

SmallThinker is approach B taken to the detail: each of those three constraints has an architectural answer. Weak compute is attacked with fine-grained MoE + sparse FFN (minimise A). Scarce RAM is attacked with streaming from SSD (resident ≈ A + cache, not N). Slow storage is attacked with the pre-attention router (hide the I/O behind attention). It is no accident that the three pieces fit together: each one solves a constraint, and together they reinforce each other.

An important nuance, so as not to fall into the hype: approach B is neither free nor universally superior. It requires training a new model (you do not reuse existing weights), and the quality ceiling of a model with a very small A is intrinsically bounded, as we will see. The argument is not “B always wins”, but “for the device regime, B attacks the right bottlenecks, and A only attacks them in passing”.

Two levels of sparsity

The central capacity idea is old and well understood in MoE: separate capacity from compute cost. In a MoE, the model has N total parameters spread across experts, but for each token only A parameters are activated (those of the top-k experts the router picks). Compute cost per token scales with A; knowledge capacity scales with N. SmallThinker applies this idea at two superimposed levels.

Level 1 — fine-grained MoE. “Fine-grained” means many small experts instead of few large ones, with very few activated per token. Instead of, say, 8 experts of which you activate 2, you have dozens of experts of which you activate a handful. With smaller experts, the same A is spread across more possible combinations, which gives the router fine granularity and keeps A very low relative to N. The result is an aggressive N/A ratio: a lot of capacity, very little compute per token.

Level 2 — sparse FFN (ReLU-style activation sparsity). This level is orthogonal and operates inside each FFN. With a ReLU-style non-linearity, a large fraction of the intermediate layer’s neurons produce exactly zero for a given token. A neuron that comes out at zero contributes nothing to the output: its matrix-vector multiplication can be skipped. This is activation sparsity: predictable token by token, and exploitable so as not to load or multiply the weight rows/columns corresponding to inactive neurons. It is the same phenomenon exploited by work such as Deja Vu or PowerInfer; SmallThinker builds it in from the factory by choosing activations that favour it.

The combined effect, in one sentence: large N (capacity), tiny A (compute cost per token ≈ proportional to A), and on top of that, inside that A, a fraction of the multiplications is saved through activation sparsity. It is sparsity on top of sparsity.

The pre-attention router: predict and prefetch

Here is the paper’s specific piece, and the one that gives the post its name. The problem it solves is one of I/O scheduling, not of quality.

When the model does not fit entirely in RAM, the expert weights live in SSD/flash and are loaded on demand. The naive flow of a MoE layer is sequential: you run attention, then the router decides which experts are needed, then you load those experts from SSD (waiting), then you run the FFN of those experts. The loading step is pure waiting: the CPU is blocked waiting for bytes from the SSD. In the device regime, where the SSD is slow, that waiting time dominates the decode step.

The pre-attention router breaks the sequentiality by inverting the order of the decision. The observation is that the router does not need this same layer’s attention output to make a reasonable prediction of which experts will be needed: it can predict it from the state it already has before running attention. So:

Before running the layer’s attention block, the router predicts the experts that will be needed.
It launches the prefetch of those experts from SSD/flash asynchronously.
In parallel, the CPU runs the attention block, which is pure compute and does not need the SSD.
When attention finishes, the prefetched experts are (ideally) already in RAM, and the FFN proceeds without waiting.

Storage I/O has been overlapped with attention compute. It is exactly the librarian who goes to the back room while you read the contents.

The hiding condition is the inequality above: the prefetch is completely hidden if and only if

$$t_{\text{attention}} \;\ge\; t_{\text{prefetch}}.$$

If attention takes longer than loading the experts, the load is free (it was already done). If the experts are too large or the SSD too slow, t_prefetch > t_att and a wait bubble equal to t_prefetch − t_att appears. That is why the design needs A to be small (small experts → fewer bytes to prefetch → low t_prefetch) and the grain to be fine: the two things level 1 of sparsity does are not only about saving FLOPs, they are about making the prefetch fit underneath attention.

The maths that matter

Memory footprint: resident N vs. A + cache

The parameter that decides whether the model fits is how much you have to keep resident in RAM at once.

Everything in RAM. If you require all the experts to be loaded, the footprint is \approx N (all parameters, multiplied by bytes/parameter according to the quantisation). For a 21B this is prohibitive on a device.
Streaming from SSD. If you only keep the active experts resident plus a cache of the recent/likely ones, the footprint drops to \approx A + \text{cache}. The weights that are not in RAM live on SSD and get prefetched when needed. This is where the real saving is: the resident set scales with A, not with N.

The non-expert part of the model (embeddings, attention, router, layernorms) is always resident, but in a fine-grained MoE the bulk of N is in the experts, so the approximation resident ≈ A + cache + dense_part is a good one.

The prefetch calculation, with numbers

Let us put numbers on the analogy. Suppose a consumer SSD at 5 GB/s of sequential read and a quantised expert of size X MB. The time to load one expert is

$$t_{\text{1 expert}} = \frac{X \text{ MB}}{5000 \text{ MB/s}} = \frac{X}{5000}\ \text{s} = \frac{X}{5}\ \text{ms}.$$

Let us make X concrete. In SmallThinker-4B-A0.6B with Q4_0 (~0.5 byte/param effective counting block overhead), a small expert of, say, 4M parameters weighs \approx 4\text{M} \times 0.5 = 2 MB. Loading it costs t_{\text{1 expert}} = 2/5 = 0.4 ms.

Now the scheduling question: if the layer’s attention block takes Y ms, how many experts can I prefetch while attention runs? The number is

$$n_{\text{prefetch}} = \left\lfloor \frac{Y}{t_{\text{1 expert}}} \right\rfloor = \left\lfloor \frac{Y \cdot 5}{X} \right\rfloor.$$

With Y = 2 ms of attention and X = 2 MB per expert: n_{\text{prefetch}} = \lfloor 2 \times 5 / 2 \rfloor = 5 experts. That is, within that layer’s attention window the SSD manages to bring in 5 experts. If the layer’s top-k activates ≤ 5 experts, the prefetch hides them all and t_prefetch ≤ t_att: zero load latency. If the layer needed 8 experts, you would bring 5 for free and pay the load of the remaining 3 as a bubble: (8-5) \times 0.4 = 1.2 ms of waiting per layer. Hence why the design wants fine grain with a small top-k: to fit underneath the attention window.

Two critical observations about this calculation:

The 5 GB/s is idealised sequential read. The experts are scattered across the disk; random 4K reads on a consumer SSD are far slower. The effective bandwidth can be a fraction of the nominal one, which reduces n_{\text{prefetch}}. Any methodology reporting tok/s ought to say whether it measures with experts pre-sorted on disk or with realistic access.
The attention window Y shrinks with short context and at the start of generation. With short prompts, attention is cheap and may not cover the prefetch; the advantage of the overlap grows with longer sequences. Another detail an honest benchmark should break down.

Weight footprint: why they report ~1 GB for a 4B

Let us do the 4B calculation in Q4_0. Quantisation to 4 bits ≈ 0.5 byte/param, plus a small overhead of per-block scales (Q4_0 adds one FP16 scale every 32 weights, ~0.56 effective byte/param). So:

$$4\text{B} \times 0.5\ \text{B/param} \approx 2\ \text{GB}.$$

That is, the full model in Q4_0 takes up ~2 GB on disk. But the authors report ~1 GB of RAM. A contradiction? No, and understanding why is understanding the design:

Not all experts are resident. Only the activated ones (A = 0.6B) plus a cache fit in RAM; the rest lives on SSD and is streamed. 0.6\text{B} \times 0.5 \approx 0.3 GB of active experts, plus the dense part (attention, embeddings, router) and a cache of hot experts.
Sparse FFN reduces the work and the useful resident set. The neurons that come out at zero do not need to be materialised for that token.

Adding up active experts + dense part + a reasonable cache, ~1 GB is plausible. But mind the nuance: ~1 GB is the resident set in RAM, not the total footprint in storage, which is still ~2 GB on SSD. Conflating the two, reporting a bare “1 GB”, is misleading if the reader understands “the model takes up 1 GB”. It takes up 2 GB; it keeps 1 GB in RAM. The distinction matters for a device with 2 GB of free storage: there it does not fit.

Analogously, SmallThinker-21B-A3B: 21\text{B} \times 0.5 \approx 10.5 GB on disk; 3\text{B} \times 0.5 \approx 1.5 GB of active experts, and the ~8 GB of RAM reported includes active experts + a generous cache + the dense part. The large cache is what takes it from 1.5 to ~8 GB: you keep many hot experts resident so as not to hit the SSD constantly.

The quality cost: the necessary scepticism

All the machinery above reduces the compute per token to \approx A. But A = 0.6B activated is very little. This is where the enthusiasm needs a brake:

Bounded reasoning capacity. A model that activates 0.6B parameters per token has, per token, the compute power of a 0.6B model, not of a 4B one. The total capacity N=4B helps to store more knowledge (more specialised experts), but the processing of each token is still limited by A. For tasks requiring composition and intensive multi-step reasoning, this is a real ceiling, not a detail.
The router is a single point of failure for quality. If the fine-grained router picks the wrong experts, and with fine grain there are more decisions to take, quality drops without any speed metric reflecting it. The pre-attention router makes this worse: it predicts the experts before seeing attention, with less information than a post-attention router. The authors ought to report how much quality is lost by predicting early (mismatch between the prefetched expert and the expert a post-attention router would have chosen).
The ~20 tok/s needs small print. On which CPU exactly? With what context and generation length (the advantage of the overlap depends on Y)? Cold start included or steady state? Did the SSD have the experts pre-sorted sequentially? An “exceeds 20 tok/s” without those conditions is a marketing number, not a methodological one.
A fair comparison. The right question is not “is it fast?”, but “at equal quality on an independent benchmark, is it faster or smaller than an equivalent compressed dense model?”. That requires evals the reader can reproduce, not just tok/s on the authors’ machine.

None of this invalidates the direction. Designing for device is, conceptually, the right approach: it attacks the real bottlenecks (compute, RAM, I/O) in the architecture instead of patching them afterwards. But “20 tok/s in ~1 GB” is a claim about efficiency, and efficiency only means something anchored to a level of quality measured honestly. Until that anchor is clear, the right amount of scepticism is high.

Implications for on-premise and edge inference

The SSD becomes part of the inference hierarchy. In cloud, the hierarchy is HBM → RAM. On device, the SSD/flash enters as one more level, and its bandwidth and random-access latency become first-order performance parameters. This connects with From disk to HBM: cold start and weight streaming stop being just a startup problem and become part of the steady state.
The heterogeneous edge box starts to make sense. In a mixed environments pattern, a device-native model like SmallThinker runs on the NUC/edge with CPU and SSD, serving locally, while the heavy work stays on the central cluster. The pre-attention router is what makes a GPU-less edge box viable.
Capacity planning changes axes. As discussed in Inference capacity planning, on device the resource to plan is not VRAM but the triad resident-RAM / SSD-bandwidth / CPU-FLOPs. A model with small A and overlapped prefetch moves the bottleneck from “does it fit in RAM?” to “does the SSD feed the prefetch in time?”.

Conclusion

SmallThinker is, above all, a change of question. Not “how do I shrink this cloud model so it fits on the device?” but “what would the model look like if I designed it for the device from the first parameter?”. The answer, fine-grained MoE to decouple N from A, sparse FFN to save inside A, and a pre-attention router that hides storage I/O underneath attention, attacks the device’s three constraints (compute, RAM, I/O) in the architecture, not in a later compression phase. The key condition, t_att ≥ t_prefetch, explains why the pieces fit: fine grain does not only save FLOPs, it makes the prefetch fit underneath attention. The reported numbers (~20 tok/s, ~1 GB / ~8 GB of RAM) are promising and the direction is solid; the cost of activating so little and the lack of methodological detail on quality call for caution. Designing for device is the right bet; measuring it honestly is the unfinished business.

References

Equipo SmallThinker (SJTU IPADS + Zenergize AI). SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment. arXiv:2507.20984. https://arxiv.org/abs/2507.20984
Repositorio oficial SmallThinker: https://github.com/SJTU-IPADS/SmallThinker
Self-Speculative Decoding for On-device MoE Acceleration. ACM The Web Conference (WWW) 2026. doi:10.1145/3774904.3792218. https://doi.org/10.1145/3774904.3792218
Liu, Z. et al. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML 2023. https://arxiv.org/abs/2310.17157
Song, Y. et al. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (sparse activation + hot/cold experts). SJTU IPADS, 2023. https://arxiv.org/abs/2312.12456

MoE inference: the call centre with 256 specialists and 8 taking each call — fundamentals, expert parallel and the economics of DeepSeek-V3

Sat, 30 May 2026 08:00:00 +0200

This post goes into an architectural decision in the Deploy stage of the six-stage LLMOps pipeline. It complements those on KV cache (which does not change with MoE), FlashAttention (still the attention kernel, also dense in every current MoE), Quantization (FP8/NVFP4 over expert weights is what lets DeepSeek-V3 fit into a modest cluster) and Speculative decoding (MTP in DeepSeek-V3 is native speculative decoding).

You are here: DEPLOY

TL;DR

A Mixture of Experts (MoE) replaces the transformer’s dense FFN layer with N parallel experts plus a router that picks the k experts relevant to each token. The operational consequence is that the model’s total parameters (how many were trained and how many must be loaded into VRAM) decouple from the active parameters per token (how many take part in each forward). DeepSeek-V3 has 671 B total but only 37 B active per token; Qwen3-235B-A22B is 235 B total and 22 B active; Mixtral 8x22B is 141 B and 39 B; Llama 4 Maverick is 400 B and 17 B. Compute per token falls to 2·N_active FLOPs, half that of a dense Llama-3-70B in DeepSeek-V3’s case, but memory stays proportional to N_total: MoE sparsity is compute sparsity, not memory sparsity. Since each expert sees only batch·k/N tokens per step, the memory-bound regime persists at a much higher batch than in dense. Serving DeepSeek-V3 671 B FP8 at low latency requires expert parallel (EP) at scale, EP=32 in prefill and EP=144 in decode in DeepSeek’s production deployment, where the dominant bottleneck stops being HBM and becomes the all-to-all of dispatch and combine between GPUs. As of May 2026, the pieces that make this viable are DeepEP (FP8-native all-to-all CUDA kernels, intra-node NVLink plus inter-node RDMA), EPLB (balancing by replicating hot experts) and Wide-EP on NVL72 (72 GPUs in one coherent NVLink domain). This post takes apart the mechanism, the maths (FLOPs/token, all-to-all bytes), the updated table of MoE models as of May 2026, the parallelisms (TP / EP / DP+EP / Wide-EP), the operational pitfalls (memory wall, cold experts, imbalance) and the real reproducible numbers on H100, B200 and GB200 NVL72.

The analogy: the call centre with 256 specialists

A serious call centre handling complex technical queries runs on specialists, not generalists. You have 256 operators on the payroll, each an expert in their sub-domain: employment law, mortgages, income tax, probate, and so on. But each incoming call only needs the attention of eight specific specialists (top-8 routing): a client’s query about tax relief on a mortgage for an inherited second home involves the income tax person, the mortgage person, the capital gains person, and so on. The receptionist (the router) listens to the first second of the call, decides which eight specialists are relevant, conferences them in and combines their answers with weights.

Three immediate consequences, identical to those of MoE in production:

You pay 256 salaries (every expert is loaded in VRAM) but you only spend the time of 8 (only k experts take part in the forward of each token). Your payroll cost scales with N_total; your cost per minute of call scales with N_active.
If all the operators fit in one office (a single GPU), fine: paper just moves between desks. But if they do not fit, you have to open branches: the conference call between operators in different branches goes over a telephone line between buildings (intra-node NVLink, inter-node InfiniBand) and that line becomes the dominant bottleneck once dozens of branches are coordinating.
If one week every client asks about income tax, the three income tax operators saturate while the probate ones sit idle. You need a system that replicates the hot operators across several branches to absorb peaks, and that pairs hot with cold so that no branch goes idle.

The receptionist is the router. The branches are the GPUs. The conference call between branches is the all-to-all (dispatch plus combine). The system for replicating hot operators is EPLB. The good telephone line is DeepEP. And when someone designs a building where all 72 branches fit together with very fast internal cabling, that is NVL72.

The bare mechanism

A classic MoE layer replaces the dense FFN (y = FFN(x) = down(act(up(x) · gate(x))) in SwiGLU) with:

Router: a linear projection g(x) = W_router · x produces N scores. A softmax over those scores gives affinity probabilities.
Top-k selection: the k experts with the highest score are chosen. Their softmax_topk weights are normalised to sum to 1.
Parallel compute: each of the k selected experts runs its own FFN over x. The N - k not selected sit idle for this token.
Weighted combination: the output is y = Σ_{i ∈ topk} w_i · Expert_i(x).

Variants the market consolidated in 2025-2026:

Shared experts: on top of the k routed ones, there are s experts that always run (1 or 2). They capture general knowledge every token needs. DeepSeek-V3 uses 1 shared + 8 routed; Mixtral uses 0 shared.
Fine-grained experts: more small experts instead of a few large ones (DeepSeek: 256 vs Mixtral: 8). This allows better specialisation because the top-k combination covers more sub-domains.
Auxiliary-loss-free routing (DeepSeek-V3): instead of adding a balancing loss during training, it dynamically adjusts a bias in the routing. It keeps balance without polluting the main loss.

Attention is still dense in every current MoE model (Mixtral, DeepSeek, Qwen3-MoE, Llama 4, Kimi K2). Only the FFN is replaced. That is why FlashAttention v3/v4 is still used as-is; DeepSeek’s MLA is an optimisation orthogonal to MoE.

The relevant MoE models in May 2026

Model	Date	Total	Active	Experts	Top-k	Shared	Notes
Mixtral 8x7B	Dec 2023	47 B	13 B	8	2	0	The one that normalised MoE in open weights
Mixtral 8x22B	Apr 2024	141 B	39 B	8	2	0	—
Grok-1 (open)	Mar 2024	314 B	~86 B	8	2	0	Weights opened by xAI
DeepSeekMoE 16B	Jan 2024	16 B	2.8 B	64+2	6	2	Introduces fine-grained + shared
DeepSeek-V2	May 2024	236 B	21 B	160+2	6	2	+ MLA
Snowflake Arctic	Apr 2024	480 B	17 B	128	2	10 B dense residual	Dense-MoE hybrid
Hunyuan-Large	Nov 2024	389 B	52 B	16+1	1+shared	1	Cross-layer KV cache
DeepSeek-V3	Dec 2024	671 B	37 B	256+1	8	1	MLA + MTP, aux-loss-free, FP8 training
Llama 4 Scout	Apr 2025	109 B	17 B	16	1+shared	1	Designed for 1 H100 node
Llama 4 Maverick	Apr 2025	400 B	17 B	128	1+shared	1	—
Qwen3-235B-A22B	Apr 2025	235 B	22 B	128	8	0	—
Kimi K2 (Moonshot)	2025	1 T	32 B	384	8	(MLA)	Trained with MuonClip
DeepSeek-V3.2-Exp	Sep 2025	671 B	37 B	256+1	8	1	+ DeepSeek Sparse Attention

Three operational observations:

The active/total ratio falls with every generation: Mixtral 8x7B was 28 %, DeepSeek-V3 is 5.5 %, Llama 4 Maverick is 4.3 %. What the industry discovered is that sparsity can be very aggressive without losing quality, provided N is large enough and the routing is well learned. Fine specialisation is worth more than per-expert capacity.
The flagships of the large providers (OpenAI, Anthropic, Google) are almost certainly MoE, but the architectures are not published. The only semi-credible leak (Hotz, Jul 2023) claimed GPT-4 was an 8×~220 B MoE; no official confirmation.
DeepSeek-V3 marks the inflection point: open weights, frontier-class, MoE with aggressive sparsity and compatible with FP8 quantisation. It is the model that forced the ecosystem (vLLM, SGLang, TensorRT-LLM) to optimise wide-EP in 2025.

The maths that matters

Three numbers drive the whole architectural decision with MoE.

Compute per token. A dense transformer spends roughly 2N FLOPs per token in the forward pass (with N = total parameters). A MoE spends 2·N_active FLOPs per token (the N_total - N_active experts do not take part). For DeepSeek-V3: 2 × 37 B = 74 GFLOPs/token. For dense Llama-3-70B: 2 × 70 B = 140 GFLOPs/token. DeepSeek does half the compute per token of Llama-3-70B while offering capacity equivalent to a model almost 10 × larger.

Total memory. But all the experts must be loaded in VRAM at some point in the forward pass. The sparsity is in compute, not memory. Weight memory:

$$\text{Memory}_{\text{weights}} \approx N_{\text{total}} \cdot \text{bytes per param}$$

For DeepSeek-V3 FP8 (1 byte): 685 GB. It does not fit in 8×H100 SXM (640 GB). You need either extra quantisation or 16+ GPUs. For Mixtral 8x22B FP8: 141 GB, comfortably inside 2×H100. For Qwen3-235B-A22B FP8: 235 GB, fits in 4×H100.

All-to-all bytes per MoE layer. When experts are spread across GPUs (EP), each token has to travel to the GPUs where its k experts live. The volume per MoE layer is roughly:

$$\text{bytes a2a/layer} \approx 2 \cdot \text{batch} \cdot \text{seq len} \cdot d_{\text{model}} \cdot k \cdot \text{bytes}$$

(the factor 2 is dispatch plus combine). For DeepSeek-V3 with d_model = 7168, k = 8, batch = 4096, FP8 (1 byte): each MoE layer moves ~470 MB; multiplied by 61 MoE layers in DeepSeek-V3, the complete step adds ~29 GB of inter-GPU traffic in MoE comms alone. That is of the order of a GPU’s HBM bandwidth in a single step. This is why bandwidth and NVLink/InfiniBand topology determine how large EP can be before comms saturate.

Persistent memory-bound regime. Here is the least intuitive consequence. In dense, raising the batch moves the operation from memory-bound to compute-bound: with enough tokens per batch, the weights are amortised over more operations. In MoE, each expert sees only batch · k / N tokens. For Qwen3-235B (k=8, N=128) with batch=128, each expert processes 8 tokens per step. To reach the compute-bound regime of the tensor cores (arithmetic intensity ~100-200 FLOPs/byte) you need batches in the thousands, around 1,600 according to recent estimates for Qwen3 (Memory-Bound MoE Serving, arXiv:2512.09277). The operational consequence: MoE scales throughput with batch more linearly than dense, but it also needs far larger batches to get close to its compute ceiling.

Expert parallel and the all-to-all bottleneck

There are three main ways to parallelise a MoE across GPUs:

Tensor Parallel (TP, Megatron-style). Splits each weight matrix across GPUs by width or height. Every GPU processes all tokens and all layers. Comms per layer: 2 all-reduces. It works well for dense; for MoE it leaves the routing sparsity unexploited (each GPU holds fragments of every expert).

Expert Parallel (EP). Splits whole experts across GPUs: 256 experts / 32 GPUs = 8 experts per GPU. Each token, after routing, travels to the GPUs where its top-k k experts live. Comms per MoE layer: an all-to-all dispatch (tokens towards experts) plus an all-to-all combine (outputs back). The cost grows with EP size and becomes the bottleneck when EP > 8.

DP + EP / Wide-EP. Replicates the model across DP groups; inside each group, EP shards the experts. Wide-EP takes EP to the full dimension of a rack (NVL72, 72 GPUs in a coherent NVLink domain). Each expert receives more tokens per step, so arithmetic intensity in each expert’s GEMMs improves and per-GPU throughput rises. vLLM reports 1.8× throughput per GPU with wide-EP against smaller setups on DeepSeek-V3 (vLLM blog, Dec 2025).

Quick decision table:

Case	Recommendation
Small MoE that fits in 1 node (Mixtral 8x7B, Llama 4 Scout)	TP only or EP=2 intra-node
Medium MoE (Mixtral 8x22B, Qwen3-235B) on 1-2 H100 nodes	EP=8 intra-node, optionally TP=2
Large MoE (DeepSeek-V3) on a multi-node cluster	TP × EP cross-node (e.g. TP=4 × EP=8 on 32 GPUs)
Large MoE with NVL72 available	Wide-EP=72 (decode), EP=32 (prefill)

DeepSeek’s production deployment combines Prefill EP=32 (4 nodes × 8 GPUs) with Decode EP=144 (18 nodes × 8 GPUs). The prefill/decode split fits conceptually with Disaggregated serving: prefill is compute-bound and benefits from moderate EP; decode is memory-bound and benefits from Wide-EP, which raises the tokens per expert per step.

DeepEP, EPLB and Wide-EP: the pieces that unblocked 2025

DeepEP (DeepSeek, opened in February 2025) is a library of all-to-all CUDA kernels optimised specifically for MoE EP. Four important properties:

Native FP8 dispatch: tokens travel in FP8, halving the bandwidth consumed.
Mixing intra-node NVLink with inter-node RDMA: the kernel picks the optimal route according to topology.
CPU bypass: the GPU→GPU dispatch goes over RDMA without touching the host.
Aligned with DeepSeek-V3’s group-limited gating: the routing prefers experts in the same node group where possible, minimising inter-node traffic.

In 2025 Tencent contributed optimisations that added +30 % on the normal kernels. There is a ROCm port for AMD.

EPLB (Expert Parallelism Load Balancer) solves the runtime imbalance problem: even when the model is well balanced in global distribution, a specific batch can heavily activate certain experts (prompts in a single language, a single domain). EPLB replicates the hot experts across several GPUs (redundant experts) and packs them heuristically with cold experts to minimise load variance. It has two modes: hierarchical (for prefill with medium EP) and global (for decode with Wide-EP). It is integrated into SGLang and vLLM.

Wide-EP on NVL72 is the peak setup of 2026: 72 Blackwell GPUs in a coherent NVLink domain (1.8 TB/s bidirectional per GPU, intra-rack). Each expert receives many more tokens per step, so arithmetic intensity improves and so does throughput per GPU. Combined with NVFP4 over expert weights and FP8 over attention, SGLang reported on GB200 NVL72 26,156 tok/s/GPU in prefill and 13,386 tok/s/GPU in decode for DeepSeek-V3 (LMSYS, Sep 2025), a 3.8× prefill and 4.8× decode versus H100.

Operational pitfalls

HBM fragmentation from expert weights. With EP=8 in 1 node, each GPU holds N/8 experts, each a tuple of matrices (gate, up, down). If N=256, that is 32 FFNs per GPU. The allocator can fragment; in some cases you need a more conservative --gpu-memory-utilization than usual.

Cold experts. In real distributions some experts are activated by <1 % of tokens. EPLB compensates by replicating hot ones, but the cold ones sit occupying HBM without taking part, “dead” memory. When real demand diverges a lot from the training distribution, this shows.

Continuous batching. It works with MoE, but the effective batch size per expert is batch_total · k / N. With large N and a moderate batch, each expert sees very few tokens per step. You need a batch_total far larger than in dense to match per-GPU throughput.

Speculative decoding + MoE. They interact well. MoE sparsity keeps the memory-bound regime at high batch, which is exactly the regime where speculative decoding wins most. DeepSeek-V3 integrates MTP (Multi-Token Prediction), speculative decoding native to the model, with no external draft, with acceptance around 85-90 % on the second token (see Speculative decoding for the mechanism).

MLA and FlashMLA are not MoE. DeepSeek-V3 combines MoE with Multi-head Latent Attention (MLA), an attention variant that reduces the KV cache ~10× against standard MHA. MLA and MoE are orthogonal: MLA optimises attention, MoE optimises the FFN. For serving DeepSeek you use the specific FlashMLA kernel (not FA3 directly).

DSA (DeepSeek Sparse Attention) in DeepSeek-V3.2-Exp (Sep 2025) introduces sparsity in attention too. The first time at the frontier that sparsity is applied in both attention and FFN.

Implications for on-premise hardware

On an RTX 4090 (24 GB). Large MoEs are out of reach. What does fit: Mixtral 8x7B AWQ-INT4 (~24 GB of weights, no headroom for KV cache, so it really requires TP=2 across two 4090s); DeepSeekMoE 16B BF16 (~33 GB, does not fit whole; INT4 ~8 GB, fits with room to spare). The interesting case is llama.cpp with --n-cpu-moe: expert weights offloaded to RAM (CPU), attention and shared experts on the GPU. It lets you run DeepSeek-V3 IQ4 (~400 GB) on a workstation with 1-2 GPUs plus 512 GB of RAM at modest throughput, but it opens the door to a low-budget setup.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). These fit comfortably:

Mixtral 8x22B FP8 (~141 GB) with TP=2 or EP=4. Low latency, excellent throughput.
Qwen3-235B-A22B FP8 (~235 GB) with TP=4 or EP=4. Plenty of room.
Llama 4 Scout 109B FP8 (~110 GB) with TP=2 or TP=4. Designed by Meta to serve well on one H100 node.

What does not fit in 4×H100: DeepSeek-V3 FP8 (~685 GB) without aggressive quantisation. Options: (1) scale to 8-16 H100; (2) use AWQ INT4 over routed experts plus FP16 over activated ones (~400 GB), which fits in 8×H100; (3) wait for Blackwell or use AMD MI300X (192 GB/GPU, so DeepSeek-V3 FP8 on 4×MI300X).

The rule of thumb as of May 2026: one 4×H100 node comfortably serves MoEs up to ~200 B total in FP8; for DeepSeek-V3 you have to scale to 2 nodes or wait for B200 NVL72 (where the whole model fits with room for enormous batches).

What we have not covered

MLA and FlashMLA in detail: DeepSeek’s KV cache optimisation, orthogonal to MoE.
DeepSeek Sparse Attention (DSA) introduced in V3.2-Exp: the first production implementation of attention sparsity for frontier models.
MoE during training: load balancing losses, drop policies, auxiliary terms vs aux-loss-free, Kimi K2’s MuonClip.
MoE + LoRA: how adapter fine-tuning is done on a MoE, and what happens to the routing.
LLM-d and other open-source platforms that materialise Wide-EP on Kubernetes.

References

Shazeer, N. et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017. https://arxiv.org/abs/1701.06538
Lepikhin, D. et al. GShard: Scaling Giant Models with Conditional Computation. ICLR 2021. https://arxiv.org/abs/2006.16668
Fedus, W., Zoph, B., Shazeer, N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 2022. https://arxiv.org/abs/2101.03961
Gale, T., Narayanan, D., Young, C., Zaharia, M. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts. MLSys 2023. https://arxiv.org/abs/2211.15841
Mistral AI. Mixtral of Experts. 2024. https://arxiv.org/abs/2401.04088
DeepSeek-AI. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. 2024. https://arxiv.org/abs/2401.06066
DeepSeek-AI. DeepSeek-V2. 2024. https://arxiv.org/abs/2405.04434
DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. https://arxiv.org/abs/2412.19437
Tencent. Hunyuan-Large. 2024. https://arxiv.org/abs/2411.02265
DeepEP repo: https://github.com/deepseek-ai/DeepEP
EPLB repo: https://github.com/deepseek-ai/EPLB
DeepSeek open-infra-index, V3/R1 Inference System Overview: https://github.com/deepseek-ai/open-infra-index
vLLM blog, Large-Scale Serving DeepSeek (Dec 2025): https://blog.vllm.ai/2025/12/17/large-scale-serving.html
vLLM blog, WideEP en GB200 (Feb 2026): https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html
LMSYS blog, DeepSeek 96 H100 PD+EP (May 2025): https://www.lmsys.org/blog/2025-05-05-large-scale-ep/
LMSYS blog, DeepSeek GB200 NVL72 part II (Sep 2025): https://www.lmsys.org/blog/2025-09-25-gb200-part-2/
NVIDIA Dynamo + GB200 NVL72 for MoE: https://developer.nvidia.com/blog/how-nvidia-gb200-nvl72-and-nvidia-dynamo-boost-inference-performance-for-moe-models/
Tensor Economics, MoE Inference Economics from First Principles: https://www.tensoreconomics.com/p/moe-inference-economics-from-first
Cohere, Why MoE models get more from speculative decoding: https://cohere.com/blog/mixture-of-experts-models-get-more-from-speculative-decoding

Moe on lo0 — Tech Blog

Device-native architectures: fine-grained MoE and pre-attention router

TL;DR

The analogy: the librarian who anticipates your request

Compressing a dense model vs. designing for device

Two levels of sparsity

The pre-attention router: predict and prefetch

The maths that matter

Memory footprint: resident N vs. A + cache

The prefetch calculation, with numbers

Weight footprint: why they report ~1 GB for a 4B

The quality cost: the necessary scepticism

Implications for on-premise and edge inference

Conclusion

See also

References

MoE inference: the call centre with 256 specialists and 8 taking each call — fundamentals, expert parallel and the economics of DeepSeek-V3

You are here: DEPLOY

TL;DR

The analogy: the call centre with 256 specialists

The bare mechanism

The relevant MoE models in May 2026

The maths that matters

Expert parallel and the all-to-all bottleneck

DeepEP, EPLB and Wide-EP: the pieces that unblocked 2025

Operational pitfalls

Implications for on-premise hardware

What we have not covered

See also

References