MoE inference: the call centre with 256 specialists and 8 taking each call — fundamentals, expert parallel and the economics of DeepSeek-V3

· By José Manuel Román
Contents

This post goes into an architectural decision in the Deploy stage of the six-stage LLMOps pipeline. It complements those on KV cache (which does not change with MoE), FlashAttention (still the attention kernel, also dense in every current MoE), Quantization (FP8/NVFP4 over expert weights is what lets DeepSeek-V3 fit into a modest cluster) and Speculative decoding (MTP in DeepSeek-V3 is native speculative decoding).

You are here: DEPLOY

You are here: DEPLOY · compute sparsity without memory sparsity1 · Data2 · Tune3 · Eval4 · Deploy5 · Observe6 · Retrain

TL;DR

A Mixture of Experts (MoE) replaces the transformer’s dense FFN layer with N parallel experts plus a router that picks the k experts relevant to each token. The operational consequence is that the model’s total parameters (how many were trained and how many must be loaded into VRAM) decouple from the active parameters per token (how many take part in each forward). DeepSeek-V3 has 671 B total but only 37 B active per token; Qwen3-235B-A22B is 235 B total and 22 B active; Mixtral 8x22B is 141 B and 39 B; Llama 4 Maverick is 400 B and 17 B. Compute per token falls to 2·N_active FLOPs, half that of a dense Llama-3-70B in DeepSeek-V3’s case, but memory stays proportional to N_total: MoE sparsity is compute sparsity, not memory sparsity. Since each expert sees only batch·k/N tokens per step, the memory-bound regime persists at a much higher batch than in dense. Serving DeepSeek-V3 671 B FP8 at low latency requires expert parallel (EP) at scale, EP=32 in prefill and EP=144 in decode in DeepSeek’s production deployment, where the dominant bottleneck stops being HBM and becomes the all-to-all of dispatch and combine between GPUs. As of May 2026, the pieces that make this viable are DeepEP (FP8-native all-to-all CUDA kernels, intra-node NVLink plus inter-node RDMA), EPLB (balancing by replicating hot experts) and Wide-EP on NVL72 (72 GPUs in one coherent NVLink domain). This post takes apart the mechanism, the maths (FLOPs/token, all-to-all bytes), the updated table of MoE models as of May 2026, the parallelisms (TP / EP / DP+EP / Wide-EP), the operational pitfalls (memory wall, cold experts, imbalance) and the real reproducible numbers on H100, B200 and GB200 NVL72.

The analogy: the call centre with 256 specialists

A serious call centre handling complex technical queries runs on specialists, not generalists. You have 256 operators on the payroll, each an expert in their sub-domain: employment law, mortgages, income tax, probate, and so on. But each incoming call only needs the attention of eight specific specialists (top-8 routing): a client’s query about tax relief on a mortgage for an inherited second home involves the income tax person, the mortgage person, the capital gains person, and so on. The receptionist (the router) listens to the first second of the call, decides which eight specialists are relevant, conferences them in and combines their answers with weights.

Three immediate consequences, identical to those of MoE in production:

  1. You pay 256 salaries (every expert is loaded in VRAM) but you only spend the time of 8 (only k experts take part in the forward of each token). Your payroll cost scales with N_total; your cost per minute of call scales with N_active.
  2. If all the operators fit in one office (a single GPU), fine: paper just moves between desks. But if they do not fit, you have to open branches: the conference call between operators in different branches goes over a telephone line between buildings (intra-node NVLink, inter-node InfiniBand) and that line becomes the dominant bottleneck once dozens of branches are coordinating.
  3. If one week every client asks about income tax, the three income tax operators saturate while the probate ones sit idle. You need a system that replicates the hot operators across several branches to absorb peaks, and that pairs hot with cold so that no branch goes idle.

The receptionist is the router. The branches are the GPUs. The conference call between branches is the all-to-all (dispatch plus combine). The system for replicating hot operators is EPLB. The good telephone line is DeepEP. And when someone designs a building where all 72 branches fit together with very fast internal cabling, that is NVL72.

The bare mechanism

A classic MoE layer replaces the dense FFN (y = FFN(x) = down(act(up(x) · gate(x))) in SwiGLU) with:

  1. Router: a linear projection g(x) = W_router · x produces N scores. A softmax over those scores gives affinity probabilities.
  2. Top-k selection: the k experts with the highest score are chosen. Their softmax_topk weights are normalised to sum to 1.
  3. Parallel compute: each of the k selected experts runs its own FFN over x. The N - k not selected sit idle for this token.
  4. Weighted combination: the output is y = Σ_{i ∈ topk} w_i · Expert_i(x).

Variants the market consolidated in 2025-2026:

  • Shared experts: on top of the k routed ones, there are s experts that always run (1 or 2). They capture general knowledge every token needs. DeepSeek-V3 uses 1 shared + 8 routed; Mixtral uses 0 shared.
  • Fine-grained experts: more small experts instead of a few large ones (DeepSeek: 256 vs Mixtral: 8). This allows better specialisation because the top-k combination covers more sub-domains.
  • Auxiliary-loss-free routing (DeepSeek-V3): instead of adding a balancing loss during training, it dynamically adjusts a bias in the routing. It keeps balance without polluting the main loss.

Attention is still dense in every current MoE model (Mixtral, DeepSeek, Qwen3-MoE, Llama 4, Kimi K2). Only the FFN is replaced. That is why FlashAttention v3/v4 is still used as-is; DeepSeek’s MLA is an optimisation orthogonal to MoE.

The relevant MoE models in May 2026

ModelDateTotalActiveExpertsTop-kSharedNotes
Mixtral 8x7BDec 202347 B13 B820The one that normalised MoE in open weights
Mixtral 8x22BApr 2024141 B39 B820—
Grok-1 (open)Mar 2024314 B~86 B820Weights opened by xAI
DeepSeekMoE 16BJan 202416 B2.8 B64+262Introduces fine-grained + shared
DeepSeek-V2May 2024236 B21 B160+262+ MLA
Snowflake ArcticApr 2024480 B17 B128210 B dense residualDense-MoE hybrid
Hunyuan-LargeNov 2024389 B52 B16+11+shared1Cross-layer KV cache
DeepSeek-V3Dec 2024671 B37 B256+181MLA + MTP, aux-loss-free, FP8 training
Llama 4 ScoutApr 2025109 B17 B161+shared1Designed for 1 H100 node
Llama 4 MaverickApr 2025400 B17 B1281+shared1—
Qwen3-235B-A22BApr 2025235 B22 B12880—
Kimi K2 (Moonshot)20251 T32 B3848(MLA)Trained with MuonClip
DeepSeek-V3.2-ExpSep 2025671 B37 B256+181+ DeepSeek Sparse Attention

Three operational observations:

  1. The active/total ratio falls with every generation: Mixtral 8x7B was 28 %, DeepSeek-V3 is 5.5 %, Llama 4 Maverick is 4.3 %. What the industry discovered is that sparsity can be very aggressive without losing quality, provided N is large enough and the routing is well learned. Fine specialisation is worth more than per-expert capacity.
  2. The flagships of the large providers (OpenAI, Anthropic, Google) are almost certainly MoE, but the architectures are not published. The only semi-credible leak (Hotz, Jul 2023) claimed GPT-4 was an 8×~220 B MoE; no official confirmation.
  3. DeepSeek-V3 marks the inflection point: open weights, frontier-class, MoE with aggressive sparsity and compatible with FP8 quantisation. It is the model that forced the ecosystem (vLLM, SGLang, TensorRT-LLM) to optimise wide-EP in 2025.

The maths that matters

Three numbers drive the whole architectural decision with MoE.

Compute per token. A dense transformer spends roughly 2N FLOPs per token in the forward pass (with N = total parameters). A MoE spends 2·N_active FLOPs per token (the N_total - N_active experts do not take part). For DeepSeek-V3: 2 × 37 B = 74 GFLOPs/token. For dense Llama-3-70B: 2 × 70 B = 140 GFLOPs/token. DeepSeek does half the compute per token of Llama-3-70B while offering capacity equivalent to a model almost 10 × larger.

Total memory. But all the experts must be loaded in VRAM at some point in the forward pass. The sparsity is in compute, not memory. Weight memory:

$$\text{Memory}_{\text{weights}} \approx N_{\text{total}} \cdot \text{bytes per param}$$

For DeepSeek-V3 FP8 (1 byte): 685 GB. It does not fit in 8×H100 SXM (640 GB). You need either extra quantisation or 16+ GPUs. For Mixtral 8x22B FP8: 141 GB, comfortably inside 2×H100. For Qwen3-235B-A22B FP8: 235 GB, fits in 4×H100.

All-to-all bytes per MoE layer. When experts are spread across GPUs (EP), each token has to travel to the GPUs where its k experts live. The volume per MoE layer is roughly:

$$\text{bytes a2a/layer} \approx 2 \cdot \text{batch} \cdot \text{seq len} \cdot d_{\text{model}} \cdot k \cdot \text{bytes}$$

(the factor 2 is dispatch plus combine). For DeepSeek-V3 with d_model = 7168, k = 8, batch = 4096, FP8 (1 byte): each MoE layer moves ~470 MB; multiplied by 61 MoE layers in DeepSeek-V3, the complete step adds ~29 GB of inter-GPU traffic in MoE comms alone. That is of the order of a GPU’s HBM bandwidth in a single step. This is why bandwidth and NVLink/InfiniBand topology determine how large EP can be before comms saturate.

Persistent memory-bound regime. Here is the least intuitive consequence. In dense, raising the batch moves the operation from memory-bound to compute-bound: with enough tokens per batch, the weights are amortised over more operations. In MoE, each expert sees only batch · k / N tokens. For Qwen3-235B (k=8, N=128) with batch=128, each expert processes 8 tokens per step. To reach the compute-bound regime of the tensor cores (arithmetic intensity ~100-200 FLOPs/byte) you need batches in the thousands, around 1,600 according to recent estimates for Qwen3 (Memory-Bound MoE Serving, arXiv:2512.09277). The operational consequence: MoE scales throughput with batch more linearly than dense, but it also needs far larger batches to get close to its compute ceiling.

Expert parallel and the all-to-all bottleneck

There are three main ways to parallelise a MoE across GPUs:

Tensor Parallel (TP, Megatron-style). Splits each weight matrix across GPUs by width or height. Every GPU processes all tokens and all layers. Comms per layer: 2 all-reduces. It works well for dense; for MoE it leaves the routing sparsity unexploited (each GPU holds fragments of every expert).

Expert Parallel (EP). Splits whole experts across GPUs: 256 experts / 32 GPUs = 8 experts per GPU. Each token, after routing, travels to the GPUs where its top-k k experts live. Comms per MoE layer: an all-to-all dispatch (tokens towards experts) plus an all-to-all combine (outputs back). The cost grows with EP size and becomes the bottleneck when EP > 8.

DP + EP / Wide-EP. Replicates the model across DP groups; inside each group, EP shards the experts. Wide-EP takes EP to the full dimension of a rack (NVL72, 72 GPUs in a coherent NVLink domain). Each expert receives more tokens per step, so arithmetic intensity in each expert’s GEMMs improves and per-GPU throughput rises. vLLM reports 1.8× throughput per GPU with wide-EP against smaller setups on DeepSeek-V3 (vLLM blog, Dec 2025).

MoE layer with Expert Parallel: EP=4 (4 GPUs, 8 experts/GPU)Router (W_router)softmax → top-k=2per tokentokensall-to-all dispatch (FP8)GPU 0E0E1E2E3experts 0-7(8 experts)GPU 1E8E9experts 8-15GPU 2E16E17experts 16-23GPU 3E24E25experts 24-31all-to-all combine (weighted sum returns to the origin)y = Σ w_i · Expert_i(x)

Quick decision table:

CaseRecommendation
Small MoE that fits in 1 node (Mixtral 8x7B, Llama 4 Scout)TP only or EP=2 intra-node
Medium MoE (Mixtral 8x22B, Qwen3-235B) on 1-2 H100 nodesEP=8 intra-node, optionally TP=2
Large MoE (DeepSeek-V3) on a multi-node clusterTP × EP cross-node (e.g. TP=4 × EP=8 on 32 GPUs)
Large MoE with NVL72 availableWide-EP=72 (decode), EP=32 (prefill)

DeepSeek’s production deployment combines Prefill EP=32 (4 nodes × 8 GPUs) with Decode EP=144 (18 nodes × 8 GPUs). The prefill/decode split fits conceptually with Disaggregated serving: prefill is compute-bound and benefits from moderate EP; decode is memory-bound and benefits from Wide-EP, which raises the tokens per expert per step.

DeepEP, EPLB and Wide-EP: the pieces that unblocked 2025

DeepEP (DeepSeek, opened in February 2025) is a library of all-to-all CUDA kernels optimised specifically for MoE EP. Four important properties:

  • Native FP8 dispatch: tokens travel in FP8, halving the bandwidth consumed.
  • Mixing intra-node NVLink with inter-node RDMA: the kernel picks the optimal route according to topology.
  • CPU bypass: the GPU→GPU dispatch goes over RDMA without touching the host.
  • Aligned with DeepSeek-V3’s group-limited gating: the routing prefers experts in the same node group where possible, minimising inter-node traffic.

In 2025 Tencent contributed optimisations that added +30 % on the normal kernels. There is a ROCm port for AMD.

EPLB (Expert Parallelism Load Balancer) solves the runtime imbalance problem: even when the model is well balanced in global distribution, a specific batch can heavily activate certain experts (prompts in a single language, a single domain). EPLB replicates the hot experts across several GPUs (redundant experts) and packs them heuristically with cold experts to minimise load variance. It has two modes: hierarchical (for prefill with medium EP) and global (for decode with Wide-EP). It is integrated into SGLang and vLLM.

Wide-EP on NVL72 is the peak setup of 2026: 72 Blackwell GPUs in a coherent NVLink domain (1.8 TB/s bidirectional per GPU, intra-rack). Each expert receives many more tokens per step, so arithmetic intensity improves and so does throughput per GPU. Combined with NVFP4 over expert weights and FP8 over attention, SGLang reported on GB200 NVL72 26,156 tok/s/GPU in prefill and 13,386 tok/s/GPU in decode for DeepSeek-V3 (LMSYS, Sep 2025), a 3.8× prefill and 4.8× decode versus H100.

Operational pitfalls

HBM fragmentation from expert weights. With EP=8 in 1 node, each GPU holds N/8 experts, each a tuple of matrices (gate, up, down). If N=256, that is 32 FFNs per GPU. The allocator can fragment; in some cases you need a more conservative --gpu-memory-utilization than usual.

Cold experts. In real distributions some experts are activated by <1 % of tokens. EPLB compensates by replicating hot ones, but the cold ones sit occupying HBM without taking part, “dead” memory. When real demand diverges a lot from the training distribution, this shows.

Continuous batching. It works with MoE, but the effective batch size per expert is batch_total · k / N. With large N and a moderate batch, each expert sees very few tokens per step. You need a batch_total far larger than in dense to match per-GPU throughput.

Speculative decoding + MoE. They interact well. MoE sparsity keeps the memory-bound regime at high batch, which is exactly the regime where speculative decoding wins most. DeepSeek-V3 integrates MTP (Multi-Token Prediction), speculative decoding native to the model, with no external draft, with acceptance around 85-90 % on the second token (see Speculative decoding for the mechanism).

MLA and FlashMLA are not MoE. DeepSeek-V3 combines MoE with Multi-head Latent Attention (MLA), an attention variant that reduces the KV cache ~10× against standard MHA. MLA and MoE are orthogonal: MLA optimises attention, MoE optimises the FFN. For serving DeepSeek you use the specific FlashMLA kernel (not FA3 directly).

DSA (DeepSeek Sparse Attention) in DeepSeek-V3.2-Exp (Sep 2025) introduces sparsity in attention too. The first time at the frontier that sparsity is applied in both attention and FFN.

Implications for on-premise hardware

On an RTX 4090 (24 GB). Large MoEs are out of reach. What does fit: Mixtral 8x7B AWQ-INT4 (~24 GB of weights, no headroom for KV cache, so it really requires TP=2 across two 4090s); DeepSeekMoE 16B BF16 (~33 GB, does not fit whole; INT4 ~8 GB, fits with room to spare). The interesting case is llama.cpp with --n-cpu-moe: expert weights offloaded to RAM (CPU), attention and shared experts on the GPU. It lets you run DeepSeek-V3 IQ4 (~400 GB) on a workstation with 1-2 GPUs plus 512 GB of RAM at modest throughput, but it opens the door to a low-budget setup.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). These fit comfortably:

  • Mixtral 8x22B FP8 (~141 GB) with TP=2 or EP=4. Low latency, excellent throughput.
  • Qwen3-235B-A22B FP8 (~235 GB) with TP=4 or EP=4. Plenty of room.
  • Llama 4 Scout 109B FP8 (~110 GB) with TP=2 or TP=4. Designed by Meta to serve well on one H100 node.

What does not fit in 4×H100: DeepSeek-V3 FP8 (~685 GB) without aggressive quantisation. Options: (1) scale to 8-16 H100; (2) use AWQ INT4 over routed experts plus FP16 over activated ones (~400 GB), which fits in 8×H100; (3) wait for Blackwell or use AMD MI300X (192 GB/GPU, so DeepSeek-V3 FP8 on 4×MI300X).

The rule of thumb as of May 2026: one 4×H100 node comfortably serves MoEs up to ~200 B total in FP8; for DeepSeek-V3 you have to scale to 2 nodes or wait for B200 NVL72 (where the whole model fits with room for enormous batches).

What we have not covered

  • MLA and FlashMLA in detail: DeepSeek’s KV cache optimisation, orthogonal to MoE.
  • DeepSeek Sparse Attention (DSA) introduced in V3.2-Exp: the first production implementation of attention sparsity for frontier models.
  • MoE during training: load balancing losses, drop policies, auxiliary terms vs aux-loss-free, Kimi K2’s MuonClip.
  • MoE + LoRA: how adapter fine-tuning is done on a MoE, and what happens to the routing.
  • LLM-d and other open-source platforms that materialise Wide-EP on Kubernetes.

See also

  • KV cache: working memory — structurally the KV cache is the same as in dense (attention is dense in every current MoE); MLA is the specific optimisation DeepSeek adds on top.
  • PagedAttention deep dive — handling the KV cache in physical blocks stays identical under MoE; only the FFN changes.
  • FlashAttention v1/v2/v3/v4 — the attention kernel is reused as-is; FlashMLA is the variant specific to DeepSeek’s MLA, adding a latent compression step to the FA pattern.
  • Quantization for LLM inference — FP8/NVFP4 over expert weights is what makes DeepSeek-V3 fit into a modest cluster and what lets NVL72 reach its peak numbers.
  • Speculative decoding — MTP in DeepSeek-V3 is native speculative decoding; MoE’s persistent memory-bound regime means speculative wins more in MoE than in dense at medium batch.
  • Disaggregated serving: prefill and decode — DeepSeek’s real deployment separates prefill (EP=32) and decode (EP=144) pools; disaggregation is a prerequisite for Wide-EP.
  • Continuous batching — the other side of the coin. MoE needs far larger batches than dense for the same per-GPU throughput because each expert sees batch · k / N tokens per step; the iterative scheduler is what makes it viable at scale.
  • The six-stage LLMOps pipeline — the master map where Deploy is stage 4.
  • Capacity planning for on-premise LLM inference — MoE sizing differs from dense: the “MoE case” section there quantifies the VRAM cost dominated by total weights and the TPOT dominated by active weights per token.
  • Device-native architectures: fine-grained MoE and pre-attention router — MoE taken to the fine-grained extreme and designed from scratch for device: the pre-attention router prefetches experts from SSD in parallel with attention, hiding the storage I/O.

References