Edge on lo0 — Tech Blog

Device-native architectures: fine-grained MoE and pre-attention router

Tue, 09 Jun 2026 01:50:00 +0000

This post belongs to the series on inference performance in small models. It is the architectural face of a problem we have already looked at from the compute-regime side (the SLM’s inverted roofline) and from the weight-loading side in From disk to HBM. Here the question is different: what if, instead of adapting a large model to the device, we design the model for the device from the first commit?

TL;DR

The default move to get an LLM onto a laptop, a phone or an edge box is to take a dense model built for cloud and compress it: distillation, pruning, quantisation. It is a gesture of reduction: you start from something big and take things away. SmallThinker (arXiv:2507.20984, SJTU IPADS + Zenergize AI) argues for the reverse gesture, designing from scratch, and articulates it in three pieces. First: fine-grained MoE, many small experts with very few activated per token, so that total parameters N (the capacity) decouple from activated parameters A (the compute cost per token). Second: sparse FFN, ReLU-style activation sparsity inside each block, which adds a second level of sparsity on top of the first. Third: a pre-attention router that predicts which experts will be needed before running the attention block and launches the prefetch of those weights from SSD/flash in parallel with attention compute, hiding the storage latency, which is the real bottleneck when the model does not fit entirely in RAM. The authors report SmallThinker-4B-A0.6B and SmallThinker-21B-A3B exceeding ~20 tok/s on consumer CPUs with Q4_0, using ~1 GB and ~8 GB of RAM. The numbers are interesting and the direction is right; the evaluation methodology and the quality cost of activating so little deserve scepticism, and that is what the last part is for.

The analogy: the librarian who anticipates your request

Picture a huge library with a small reading room. You are sitting in the room at a single desk: only a few books fit on it at a time (that is RAM). The bulk of the collection is in the back room, on long shelves that are slow to walk (that is the SSD/flash). And there is a librarian.

The naive method: you read, you reach a point where you need a specific book, you request it, and then the librarian gets up, walks to the back room, looks for it and comes back. Meanwhile, you wait with the page open, making no progress. Every time you need a new book, you pay the full trip to the back room. The reading room spends most of its time waiting, not reading.

The SmallThinker method: the librarian is clever and gets ahead of you. While you are still reading the chapter’s table of contents, working out what it is about and connecting ideas, which in the model is the attention block, they have already glanced over your shoulder, predicted which three or four books you are going to ask for and gone to the back room to fetch them. By the time you finish the contents and formulate the request, the books are already on your desk. You have not waited: the trip to the back room happened in parallel with your reading of the contents.

The analogy holds on four details:

The small desk is RAM; the slow back room is the SSD/flash.
The books are the MoE’s experts: only a few are on the desk at any moment.
Reading the contents is the attention block; requesting and using the books is the FFN/expert block.
The librarian who predicts and gets ahead is the pre-attention router: the prediction is made first, and the trip to fetch (the prefetch) overlaps with reading the contents (attention).

The quantitative question running through the whole post is: does the librarian get back in time? The wait is only hidden if the trip to the back room takes less than your reading of the contents. That is the condition t_{\text{attention}} \ge t_{\text{prefetch}}, and we will do it with numbers.

Compressing a dense model vs. designing for device

It is worth laying out the two approaches plainly, because they are not degrees of the same thing: they are different philosophies.

Approach A — compress a dense model built for cloud. You start from, say, a dense 7B–14B model trained to run on an RTX 4090 (24 GB, Ada Lovelace) or on a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). To fit it into a device you apply three levers, each with its own post: distillation (you train a small student that imitates the teacher), pruning (you remove weights or whole structures) and aggressive quantisation (you go down to 4 bits or fewer). The resulting model is still dense: all its parameters are activated on every token. You have reduced the number of parameters, but the compute pattern is the cloud’s, only smaller.

Approach B — design for device from scratch. Here the device’s constraints enter the architecture, not a later compression phase. The constraints are three and very concrete:

Weak compute. A laptop CPU or a mobile SoC does orders of magnitude fewer FLOPs than a datacenter GPU. This pushes you to minimise the activated parameters per token, not the total ones.
Little RAM. Tens of GB do not fit. This pushes you to keep resident only what is essential and to stream the rest.
Slow storage. The SSD or flash you are forced to stream from has far less bandwidth than a GPU’s HBM. This turns storage I/O into the bottleneck, and pushes you to hide it.

SmallThinker is approach B taken to the detail: each of those three constraints has an architectural answer. Weak compute is attacked with fine-grained MoE + sparse FFN (minimise A). Scarce RAM is attacked with streaming from SSD (resident ≈ A + cache, not N). Slow storage is attacked with the pre-attention router (hide the I/O behind attention). It is no accident that the three pieces fit together: each one solves a constraint, and together they reinforce each other.

An important nuance, so as not to fall into the hype: approach B is neither free nor universally superior. It requires training a new model (you do not reuse existing weights), and the quality ceiling of a model with a very small A is intrinsically bounded, as we will see. The argument is not “B always wins”, but “for the device regime, B attacks the right bottlenecks, and A only attacks them in passing”.

Two levels of sparsity

The central capacity idea is old and well understood in MoE: separate capacity from compute cost. In a MoE, the model has N total parameters spread across experts, but for each token only A parameters are activated (those of the top-k experts the router picks). Compute cost per token scales with A; knowledge capacity scales with N. SmallThinker applies this idea at two superimposed levels.

Level 1 — fine-grained MoE. “Fine-grained” means many small experts instead of few large ones, with very few activated per token. Instead of, say, 8 experts of which you activate 2, you have dozens of experts of which you activate a handful. With smaller experts, the same A is spread across more possible combinations, which gives the router fine granularity and keeps A very low relative to N. The result is an aggressive N/A ratio: a lot of capacity, very little compute per token.

Level 2 — sparse FFN (ReLU-style activation sparsity). This level is orthogonal and operates inside each FFN. With a ReLU-style non-linearity, a large fraction of the intermediate layer’s neurons produce exactly zero for a given token. A neuron that comes out at zero contributes nothing to the output: its matrix-vector multiplication can be skipped. This is activation sparsity: predictable token by token, and exploitable so as not to load or multiply the weight rows/columns corresponding to inactive neurons. It is the same phenomenon exploited by work such as Deja Vu or PowerInfer; SmallThinker builds it in from the factory by choosing activations that favour it.

The combined effect, in one sentence: large N (capacity), tiny A (compute cost per token ≈ proportional to A), and on top of that, inside that A, a fraction of the multiplications is saved through activation sparsity. It is sparsity on top of sparsity.

The pre-attention router: predict and prefetch

Here is the paper’s specific piece, and the one that gives the post its name. The problem it solves is one of I/O scheduling, not of quality.

When the model does not fit entirely in RAM, the expert weights live in SSD/flash and are loaded on demand. The naive flow of a MoE layer is sequential: you run attention, then the router decides which experts are needed, then you load those experts from SSD (waiting), then you run the FFN of those experts. The loading step is pure waiting: the CPU is blocked waiting for bytes from the SSD. In the device regime, where the SSD is slow, that waiting time dominates the decode step.

The pre-attention router breaks the sequentiality by inverting the order of the decision. The observation is that the router does not need this same layer’s attention output to make a reasonable prediction of which experts will be needed: it can predict it from the state it already has before running attention. So:

Before running the layer’s attention block, the router predicts the experts that will be needed.
It launches the prefetch of those experts from SSD/flash asynchronously.
In parallel, the CPU runs the attention block, which is pure compute and does not need the SSD.
When attention finishes, the prefetched experts are (ideally) already in RAM, and the FFN proceeds without waiting.

Storage I/O has been overlapped with attention compute. It is exactly the librarian who goes to the back room while you read the contents.

The hiding condition is the inequality above: the prefetch is completely hidden if and only if

$$t_{\text{attention}} \;\ge\; t_{\text{prefetch}}.$$

If attention takes longer than loading the experts, the load is free (it was already done). If the experts are too large or the SSD too slow, t_prefetch > t_att and a wait bubble equal to t_prefetch − t_att appears. That is why the design needs A to be small (small experts → fewer bytes to prefetch → low t_prefetch) and the grain to be fine: the two things level 1 of sparsity does are not only about saving FLOPs, they are about making the prefetch fit underneath attention.

The maths that matter

Memory footprint: resident N vs. A + cache

The parameter that decides whether the model fits is how much you have to keep resident in RAM at once.

Everything in RAM. If you require all the experts to be loaded, the footprint is \approx N (all parameters, multiplied by bytes/parameter according to the quantisation). For a 21B this is prohibitive on a device.
Streaming from SSD. If you only keep the active experts resident plus a cache of the recent/likely ones, the footprint drops to \approx A + \text{cache}. The weights that are not in RAM live on SSD and get prefetched when needed. This is where the real saving is: the resident set scales with A, not with N.

The non-expert part of the model (embeddings, attention, router, layernorms) is always resident, but in a fine-grained MoE the bulk of N is in the experts, so the approximation resident ≈ A + cache + dense_part is a good one.

The prefetch calculation, with numbers

Let us put numbers on the analogy. Suppose a consumer SSD at 5 GB/s of sequential read and a quantised expert of size X MB. The time to load one expert is

$$t_{\text{1 expert}} = \frac{X \text{ MB}}{5000 \text{ MB/s}} = \frac{X}{5000}\ \text{s} = \frac{X}{5}\ \text{ms}.$$

Let us make X concrete. In SmallThinker-4B-A0.6B with Q4_0 (~0.5 byte/param effective counting block overhead), a small expert of, say, 4M parameters weighs \approx 4\text{M} \times 0.5 = 2 MB. Loading it costs t_{\text{1 expert}} = 2/5 = 0.4 ms.

Now the scheduling question: if the layer’s attention block takes Y ms, how many experts can I prefetch while attention runs? The number is

$$n_{\text{prefetch}} = \left\lfloor \frac{Y}{t_{\text{1 expert}}} \right\rfloor = \left\lfloor \frac{Y \cdot 5}{X} \right\rfloor.$$

With Y = 2 ms of attention and X = 2 MB per expert: n_{\text{prefetch}} = \lfloor 2 \times 5 / 2 \rfloor = 5 experts. That is, within that layer’s attention window the SSD manages to bring in 5 experts. If the layer’s top-k activates ≤ 5 experts, the prefetch hides them all and t_prefetch ≤ t_att: zero load latency. If the layer needed 8 experts, you would bring 5 for free and pay the load of the remaining 3 as a bubble: (8-5) \times 0.4 = 1.2 ms of waiting per layer. Hence why the design wants fine grain with a small top-k: to fit underneath the attention window.

Two critical observations about this calculation:

The 5 GB/s is idealised sequential read. The experts are scattered across the disk; random 4K reads on a consumer SSD are far slower. The effective bandwidth can be a fraction of the nominal one, which reduces n_{\text{prefetch}}. Any methodology reporting tok/s ought to say whether it measures with experts pre-sorted on disk or with realistic access.
The attention window Y shrinks with short context and at the start of generation. With short prompts, attention is cheap and may not cover the prefetch; the advantage of the overlap grows with longer sequences. Another detail an honest benchmark should break down.

Weight footprint: why they report ~1 GB for a 4B

Let us do the 4B calculation in Q4_0. Quantisation to 4 bits ≈ 0.5 byte/param, plus a small overhead of per-block scales (Q4_0 adds one FP16 scale every 32 weights, ~0.56 effective byte/param). So:

$$4\text{B} \times 0.5\ \text{B/param} \approx 2\ \text{GB}.$$

That is, the full model in Q4_0 takes up ~2 GB on disk. But the authors report ~1 GB of RAM. A contradiction? No, and understanding why is understanding the design:

Not all experts are resident. Only the activated ones (A = 0.6B) plus a cache fit in RAM; the rest lives on SSD and is streamed. 0.6\text{B} \times 0.5 \approx 0.3 GB of active experts, plus the dense part (attention, embeddings, router) and a cache of hot experts.
Sparse FFN reduces the work and the useful resident set. The neurons that come out at zero do not need to be materialised for that token.

Adding up active experts + dense part + a reasonable cache, ~1 GB is plausible. But mind the nuance: ~1 GB is the resident set in RAM, not the total footprint in storage, which is still ~2 GB on SSD. Conflating the two, reporting a bare “1 GB”, is misleading if the reader understands “the model takes up 1 GB”. It takes up 2 GB; it keeps 1 GB in RAM. The distinction matters for a device with 2 GB of free storage: there it does not fit.

Analogously, SmallThinker-21B-A3B: 21\text{B} \times 0.5 \approx 10.5 GB on disk; 3\text{B} \times 0.5 \approx 1.5 GB of active experts, and the ~8 GB of RAM reported includes active experts + a generous cache + the dense part. The large cache is what takes it from 1.5 to ~8 GB: you keep many hot experts resident so as not to hit the SSD constantly.

The quality cost: the necessary scepticism

All the machinery above reduces the compute per token to \approx A. But A = 0.6B activated is very little. This is where the enthusiasm needs a brake:

Bounded reasoning capacity. A model that activates 0.6B parameters per token has, per token, the compute power of a 0.6B model, not of a 4B one. The total capacity N=4B helps to store more knowledge (more specialised experts), but the processing of each token is still limited by A. For tasks requiring composition and intensive multi-step reasoning, this is a real ceiling, not a detail.
The router is a single point of failure for quality. If the fine-grained router picks the wrong experts, and with fine grain there are more decisions to take, quality drops without any speed metric reflecting it. The pre-attention router makes this worse: it predicts the experts before seeing attention, with less information than a post-attention router. The authors ought to report how much quality is lost by predicting early (mismatch between the prefetched expert and the expert a post-attention router would have chosen).
The ~20 tok/s needs small print. On which CPU exactly? With what context and generation length (the advantage of the overlap depends on Y)? Cold start included or steady state? Did the SSD have the experts pre-sorted sequentially? An “exceeds 20 tok/s” without those conditions is a marketing number, not a methodological one.
A fair comparison. The right question is not “is it fast?”, but “at equal quality on an independent benchmark, is it faster or smaller than an equivalent compressed dense model?”. That requires evals the reader can reproduce, not just tok/s on the authors’ machine.

None of this invalidates the direction. Designing for device is, conceptually, the right approach: it attacks the real bottlenecks (compute, RAM, I/O) in the architecture instead of patching them afterwards. But “20 tok/s in ~1 GB” is a claim about efficiency, and efficiency only means something anchored to a level of quality measured honestly. Until that anchor is clear, the right amount of scepticism is high.

Implications for on-premise and edge inference

The SSD becomes part of the inference hierarchy. In cloud, the hierarchy is HBM → RAM. On device, the SSD/flash enters as one more level, and its bandwidth and random-access latency become first-order performance parameters. This connects with From disk to HBM: cold start and weight streaming stop being just a startup problem and become part of the steady state.
The heterogeneous edge box starts to make sense. In a mixed environments pattern, a device-native model like SmallThinker runs on the NUC/edge with CPU and SSD, serving locally, while the heavy work stays on the central cluster. The pre-attention router is what makes a GPU-less edge box viable.
Capacity planning changes axes. As discussed in Inference capacity planning, on device the resource to plan is not VRAM but the triad resident-RAM / SSD-bandwidth / CPU-FLOPs. A model with small A and overlapped prefetch moves the bottleneck from “does it fit in RAM?” to “does the SSD feed the prefetch in time?”.

Conclusion

SmallThinker is, above all, a change of question. Not “how do I shrink this cloud model so it fits on the device?” but “what would the model look like if I designed it for the device from the first parameter?”. The answer, fine-grained MoE to decouple N from A, sparse FFN to save inside A, and a pre-attention router that hides storage I/O underneath attention, attacks the device’s three constraints (compute, RAM, I/O) in the architecture, not in a later compression phase. The key condition, t_att ≥ t_prefetch, explains why the pieces fit: fine grain does not only save FLOPs, it makes the prefetch fit underneath attention. The reported numbers (~20 tok/s, ~1 GB / ~8 GB of RAM) are promising and the direction is solid; the cost of activating so little and the lack of methodological detail on quality call for caution. Designing for device is the right bet; measuring it honestly is the unfinished business.

References

Equipo SmallThinker (SJTU IPADS + Zenergize AI). SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment. arXiv:2507.20984. https://arxiv.org/abs/2507.20984
Repositorio oficial SmallThinker: https://github.com/SJTU-IPADS/SmallThinker
Self-Speculative Decoding for On-device MoE Acceleration. ACM The Web Conference (WWW) 2026. doi:10.1145/3774904.3792218. https://doi.org/10.1145/3774904.3792218
Liu, Z. et al. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML 2023. https://arxiv.org/abs/2310.17157
Song, Y. et al. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (sparse activation + hot/cold experts). SJTU IPADS, 2023. https://arxiv.org/abs/2312.12456

Mixed NVIDIA + Intel environments for LLM inference: from the central H100 cluster to the NUC in the branch office

Tue, 02 Jun 2026 04:30:00 +0200

This post complements those on Capacity planning for on-premise LLM inference (which assumed a pure NVIDIA cluster), Seven layers of the stack (which also did not get into hardware heterogeneity) and The LLM inference router (where capability routing makes full sense once there is mixed hardware). It is the missing piece for talking about “hardware sovereignty” without reducing it to “which vendor do I pick”.

TL;DR

A production LLM inference cluster in 2026 can stop being monolithically NVIDIA if it accepts heterogeneity as an architectural decision. The motivation is not theory but three measurable operational advantages. (1) Cost: an Intel Xeon 6 with AMX (Advanced Matrix Extensions) delivers 7B INT4 at ~80 tok/s while serving embeddings and reranking at a fraction of the cost of dedicating an H100 to that task; the capacity planning adds up better with an Intel CPU handling the cheap work and an NVIDIA H100 the big LLM. (2) Sovereignty and supply chain diversification: NVIDIA holds roughly 94 % of the AI accelerator market (November 2025), single-vendor dependency with all its risks; Intel manufactures in Europe (Leixlip operational, Magdeburg planned) against a design-only NVIDIA with TSMC as foundry, which for a Spanish or European organisation subject to ENS / NIS2 / EU AI Act is a real hedging argument. (3) Edge: an Intel NUC with a Lunar Lake CPU (48 TOPS NPU) or Panther Lake (50 TOPS NPU + 120 TOPS Xe3 = 180 TOPS platform) runs 7B INT4 models at usable speed, which opens up the “branch office with local inference + central DC for complex cases” pattern. Relevant Intel hardware in June 2026: Intel Gaudi 3 (128 GB HBM2e, 1,835 TFLOPS BF16/FP8, 3.67 TB/s; a direct competitor to the H100, Intel claims +20 % on Llama 2 70B but Signal65 published H200 9× over Gaudi 3 on Llama 3.1 405B, you have to cite both; Falcon Shores cancelled in January 2025, Jaguar Shores 2026 as the restart bet, Gaudi 4 confirmed as never happening); Intel Xeon 6 with AMX (up to 288 E-cores on Sierra Forest or 86 P-cores on Granite Rapids, 1,024 FLOPS BF16/cycle/core with AMX, Intel claims 2.7× tok/s vs EPYC 9965 on the vLLM CPU backend); Intel Arc Pro B60 (Battlemage, 24 GB GDDR6, 456 GB/s, 197 TOPS INT8, launched September 2025, with a dual-GPU 48 GB variant and a “Battlematrix” rack with 8× = 192 GB VRAM); Intel NUC with NPU (Lunar Lake 48 TOPS, Arrow Lake similar, Panther Lake 50 TOPS at CES 2026; realistic for 7-13B INT4, not for the 30-70B Intel claims in its marketing). Software: OpenVINO 2025.3 with the GenAI API and vLLM-OpenVINO; IPEX-LLM with integrations into llama.cpp, vLLM, HF, LangChain; vLLM CPU backend with AMX; llama.cpp SYCL (better than Vulkan on Arc). Four canonical patterns: embeddings + reranker on Intel alongside the LLM on NVIDIA; guardrails + PII redaction on a near-edge NUC; speculative drafter on a NUC near the user and target on the H100; NUC dev workstations. Unified observability via DCGM + habana-metric-exporter + intel-gpu-exporter + Intel PCM federated in Prometheus. Pitfalls: tokenizer mismatch between engines, edge↔central round-trip latency, FP8 Hopper ≠ INT8 AMX in quality, version synchronisation. Applied to a generic cluster: central DC 4×H100 SXM + Xeon 6 AMX sidecar + 6-12 Intel NUCs in branch offices. Critical disclaimer: as of June 2026 there are no publicly verifiable cases of a mixed NVIDIA + Intel deployment in European banking or government; the pattern is emergent architecture and worth recommending, not established practice with an industrial track record.

You are here: DEPLOY (with heterogeneity as a decision)

The analogy: the factory with several different machines

A serious factory has several machines with different purposes, not one universal machine. A 200-tonne hydraulic press for heavy stamping; a bench lathe for turned parts; a 3D printer for quick prototypes; a pick-and-place robot for SMD. Each machine does what it does better than the others in its niche, and the plant manager sizes the mix according to the real product portfolio, not according to fashion. Buying three hydraulic presses because “they are the most impressive” when 60 % of the work is turned parts wastes capital: the lathe is cheaper, faster in its niche and frees the press for what genuinely needs it.

An LLM inference cluster with an NVIDIA H100 dedicated to computing embeddings for a RAG corpus is using a hydraulic press to drill bolts. The H100 is magnificent for a 70B LLM in BF16 with concurrency of 40+; for embeddings of an 800-token document with bge-m3, what you need is an Intel Xeon 6 with AMX at a fraction of the cost and power draw. A cluster that wants to serve light guardrails (Llama Guard 4 8B) on every request, with a 50 ms budget, does not need that guardrail on an H100 either: an Intel NUC with a 48 TOPS NPU covers the case with room to spare.

The heterogeneous factory is not theoretical elegance: it is maximising the useful utilisation of fixed capital. Nor is the heterogeneous LLM inference cluster.

Three operational reasons for heterogeneity

Reason 1 — cost

An H100 SXM 80 GB running 24/7 draws ~700 W (real measurement at the wall ~697 W with vLLM Llama 3.1 405B batch=4) and represents between 25,000 € and 35,000 € of amortised hardware. An Intel Xeon 6 with AMX (Granite Rapids 86 cores or Sierra Forest 288 E-cores) draws 350-500 W for the socket and costs a fraction. The operational split: the H100 is reserved for the big LLM (Llama 70B BF16 or FP8, where its HBM3 and FP8 tensor cores are worth their weight); the Xeon AMX absorbs embeddings (bge-m3, e5-large), reranking (bge-reranker-v2-m3), small models (Llama 3.2 1B / 3B INT4) and offline batch processing. It is the same logic as the capacity planning taken one step further: instead of budgeting KV cache VRAM only on the H100, budget each workload on the silicon where its arithmetic intensity fits best.

Reason 2 — sovereignty and supply chain diversification

As of November 2025, NVIDIA holds approximately 94 % of the AI accelerator market. That concentration is risk. For an organisation subject to ENS / NIS2 / EU AI Act, depending on a single supplier whose foundry is concentrated in Taiwan (TSMC) introduces supply chain vulnerabilities that recent regulations (NIS2, supply chain provisions) are starting to require you to document and mitigate. Intel diversifies: it has its own fabs in Europe (Leixlip operational in Ireland; Magdeburg planned in Germany, with EU Chips Act funding), which for a Spanish or European public sector client is a real contractual argument, not marketing.

Mandatory disclaimer: the Intel roadmap after Falcon Shores is unstable. Intel cancelled Falcon Shores in January 2025 and relegated Gaudi 4 to “will never exist”; the restart bet is Jaguar Shores in 2026 as a rack-scale platform, still with no confirmed public specs. Diversification is strategically correct, but assuming continuity of the Intel roadmap at NVIDIA’s level in 2026 would be naive. The operational strategy: Intel for workloads where lock-in is lower (CPU for embeddings, NUC for light edge, replaceable by AMD/Apple/SiFive if Intel pivots again), NVIDIA for the big LLM where the maturity of the software stack still has no rival.

Reason 3 — edge

The “everything travels to the central DC” pattern breaks in three cases: latency (branch office 100+ ms away from the DC, unacceptable for chat), data sovereignty (prompts with personal or classified data that must not leave the local perimeter), and offline operation (branch office with intermittent connectivity). The Intel NUC with a modern CPU (Lunar Lake / Arrow Lake / Panther Lake) brings 48-50 TOPS NPU + 100-180 TOPS Xe2/Xe3 iGPU in a box of 0.5-1.5 L volume drawing 30-65 W. 7B INT4 models run at usable speed; with more aggressive quantisation (Q3_K) a Llama 13B fits. For branch offices with RAG over a local corpus + a 7B LLM + guardrails, the NUC is perfect.

Relevant Intel hardware (June 2026)

Part	Memory	Key performance	Launch	Status
Intel Gaudi 3	128 GB HBM2e, 3.67 TB/s	1,835 TFLOPS BF16/FP8; 1,200 GB/s networking	Apr 2024	Active; successor Jaguar Shores 2026 (not Gaudi 4)
Intel Xeon 6 (Granite Rapids)	DDR5 + MRDIMM	86 P-cores, AMX 1,024 FLOPS BF16/cycle/core	2024-2025	Active
Intel Xeon 6 (Sierra Forest)	DDR5	288 E-cores	2024	Active
Intel Arc Pro B60 (Battlemage)	24 GB GDDR6, 456 GB/s	197 TOPS INT8; 12.28 TFLOPS FP32	Sep 2025	Active; dual 48 GB variant, rack 8× = 192 GB
Intel Data Center GPU Max	128 GB HBM	successor to Ponte Vecchio	discontinued	Discontinued Jan 2026
Intel NUC (Lunar Lake)	DDR5x	NPU 48 TOPS + Xe2 67 TOPS = 120 TOPS platform	2024	Active
Intel NUC (Arrow Lake)	DDR5	NPU 13 TOPS + Xe iGPU	2024	Active (less NPU than Lunar)
Intel NUC (Panther Lake)	DDR5x	NPU 50 TOPS + Xe3 120 TOPS = 180 TOPS platform	CES Jan 2026	Rolling out

Intel Gaudi 3 — the critical note on the marketing

Intel publishes that Gaudi 3 delivers +20 % throughput vs H100 on Llama 2 70B inference and 2× price/performance. The figure appears in the official whitepaper and in launch presentations. However, Signal65 (an independent firm) published in 2025 that the H200 beats Gaudi 3 by a factor of 9× on Llama 3.1 405B. The discrepancy matters: both numbers can be true for their specific benchmarks (Llama 2 70B FP16 vs Llama 3.1 405B FP8) but the operational conclusion changes radically depending on which one you keep.

This post’s recommendation: treat Gaudi 3 as a valid option for Llama-class 70B in BF16/FP8 where Intel claims parity or advantage, not for frontier 200B+ models where NVIDIA keeps a clear margin. And consider the roadmap risk: Gaudi 4 will never exist; the line’s successor is Jaguar Shores 2026 with a completely new rack-scale architecture, a discontinuity rather than an evolution.

Intel Xeon 6 with AMX — the CPU workhorse

The Advanced Matrix Extensions (AMX) are the non-obvious piece. Each Granite Rapids P-core executes up to 1,024 BF16 FLOPS per cycle via AMX, which turns a Xeon 6 with 64-86 cores into a respectable matrix accelerator for small and mid-sized models. Real reported figures: Llama 3.2 INT4 at ~57 tok/s with AMX vs 28 tok/s without AMX (a clean 2× factor). For serving 7B INT4 with the vLLM CPU backend + AMX, Intel claims 2.7× tok/s vs EPYC 9965, a figure with Intel bias but qualitatively corroborated by LMSYS in their DeepSeek R1 671B deployment on Xeon 6 + SGLang.

Operational use case: embeddings and reranking on a Xeon 6 sidecar next to the H100 cluster. Models such as bge-m3 (multilingual embedding) or bge-reranker-v2-m3 run at acceptable throughput on CPU AMX; they do not deserve a dedicated H100. Freeing the H100 for the 70B LLM raises the cluster’s effective RPS without buying more GPUs.

Intel Arc Pro B60 and Battlematrix

Launched in September 2025, the Arc Pro B60 (Battlemage) brings 24 GB GDDR6 with 456 GB/s of bandwidth and 197 TOPS INT8 at 200 W. Maxsun has a dual-GPU 48 GB variant. The “Battlematrix” rack configuration combines 8 units = 192 GB of aggregate VRAM. The interesting point: at a cost far below an H100 SXM 80 GB, which makes it a candidate for 30-70B INT4-INT8 LLMs served via OpenVINO or llama.cpp SYCL.

Phoronix verified that on SYCL the Arc Pro B70 reaches parity with the Radeon PRO W7900 (previous AMD generation) on DeepSeek R1 Llama 8B pp512. The Vulkan backend loses heavily (~1/4 of SYCL performance); for Arc Pro, always SYCL.

Intel NUC with NPU — the edge node

Intel NUCs with a Lunar Lake CPU (Core Ultra Series 2) bring NPU 4 with 48 TOPS and a platform total of 120 TOPS adding the Xe2 iGPU and CPU AVX. Panther Lake (CES January 2026) goes up to NPU 5 = 50 TOPS + Xe3 120 TOPS = 180 TOPS platform.

Intel states that Panther Lake “runs 30-70B models locally”. Reality check: it is marketing. A 30-70B INT4 fits in RAM (DDR5x 32-64 GB) but sustained speed with Q4_K_M quantisation on a NUC hovers around 2-8 tok/s; comfortable for occasional use, not for serving traffic. The NUC’s real sweet spot is 7B INT4 at 20-40 tok/s on the iGPU/NPU, perfect for a client branch office with casual queries.

Intel software — the relevant stack

OpenVINO 2025.3 (June 2026) is the central piece. It supports single-command deployment via the OVMS CLI with automatic download from the HF Hub; it integrates OpenVINO GenAI with a C++/Python API for generative pipelines; it exposes a vLLM v1-compatible API (vLLM-OpenVINO). GGUF model support: DeepSeek Distill, Qwen 2/2.5, Llama 3. Optimisations: Sage Attention (first token with long prompts), per-channel KV-cache compression.

Intel Extension for PyTorch (IPEX), XPU version 2.8.10+xpu, adds Intel backends to PyTorch. IPEX-LLM is the subproject that integrates with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM and DeepSpeed. May 2025: it ran DeepSeek V3/R1 671B and Qwen3MoE 235B on 1-2 Arc A770/B580 cards with FlashMoE.

vLLM CPU backend, the CPU branch of vLLM with AMX optimisations. For 7B INT4 on a 4th-gen Xeon with AMX: 12-50 tok/s; with a Xeon Gold 6530 + INT4: ~80 tok/s. Academic figures (arXiv 2410.04466).

llama.cpp SYCL, the recommended backend for Arc; Vulkan works but hovers around 1/4 of SYCL performance on an Arc B580. SYCL reaches parity with the previous AMD generation.

Habana SynapseAI, the Gaudi 3 stack. The PyTorch bridge habana_frameworks.torch registers the hpu device; it integrates with torch.compile. It is not a complete port to oneAPI but a partial integration via oneMKL. That implies the Gaudi ecosystem keeps a certain separation from Intel’s general oneAPI, relevant in view of the hypothetical Jaguar Shores and future unification.

The four canonical patterns

Pattern 1 — embeddings + reranker on Intel

The most mature and the easiest to adopt. In a typical RAG system, each user request invokes:

Query embedding (50 ms on an H100, 80 ms on a Xeon AMX, 30 ms on an Arc Pro B60).
Vector search (Qdrant / Milvus / Chroma; latency ~10-30 ms).
Reranker over the top-k candidates (60 ms on an H100, 100-150 ms on a Xeon AMX).
LLM over the augmented prompt (200-500 ms TTFT, 30-50 ms/token).

Steps 1 and 3 are memory-bound and relatively small (100M-1B models): a Xeon 6 with AMX (the Arc Pro B60 is faster but is already a dedicated GPU) does the job at a hardware cost that is a fraction of a dedicated H100. Step 4 stays on NVIDIA because that is where its tensor architecture + HBM3 + FP8 is worth what it costs.

Operational implication: a Xeon 6 sidecar (~40 cores, ~10-15 k€) serving embeddings + reranking frees the equivalent of 1-2 H100s of “cheap” load, recovering that capacity for the big LLM. Clear ROI in sizing.

Pattern 2 — guardrails + PII redaction on a near-edge NUC

A bank branch, a medical practice or a law office generates prompts with personal or classified data. Sending those prompts to the central DC (even if it is corporate on-premise) can clash with local retention policies or with specific compliance (GDPR, professional privilege).

The pattern: the NUC in the branch office runs two critical steps before the round-trip:

PII redaction with Presidio (CPU-only, fast) or Llama Guard 4 8B on the NUC’s NPU + iGPU. It replaces names, tax IDs and account numbers with placeholders.
Light guardrails (PromptGuard 2 86M, Llama Guard 4 8B) on the NPU + iGPU. It filters prompt injection, jailbreaks and prohibited content.

Only afterwards does the redacted prompt travel to the central DC so the big LLM can answer. The response comes back to the NUC, which rehydrates the placeholders with the real values before showing it to the user. The sensitive data never leaves the branch office.

Costs: a Panther Lake NUC is ~1,500-2,500 € per unit, scalable to dozens of branch offices with no additional central GPU cost. Latency: 50-150 ms for the edge step before the DC’s 200-500 ms round-trip.

Pattern 3 — speculative decoding drafter on a NUC

Speculative decoding uses a small drafter that proposes γ tokens and a large target that accepts or rejects them in a single forward pass. If the drafter is geographically close to the user (a NUC in the branch office) and the target is in the central DC, the client’s perceived latency falls further still.

Setup: drafter Llama 3.2 1B INT4 on a NUC + target Llama 3.1 70B FP8 on the central H100. The NUC generates γ=4 tokens in ~50 ms locally; the target verifies them in one pass (40-80 ms including round-trip); if the acceptance rate is > 60 %, effective TTFT falls ~50 % compared with Llama 70B without speculative decoding.

An important constraint: drafter and target must share a tokenizer. Llama 3.2 1B and Llama 3.1 70B have compatible tokenizers. Mixing a Llama drafter with a Qwen target breaks the pattern.

Pattern 4 — NUC dev workstations

Dev / CI iterating on prompts, evals and retrieval logic does not need a production GPU to validate correctness. A NUC with Llama 3.2 3B INT4 runs the end-to-end functional tests (including embeddings + retrieval + LLM + guardrails) at a tenth of the cost of iterating on an H100. Only the last pre-prod smoke test uses the production cluster.

A mature pattern in organisations with many developers and scarce production GPU. Iteration that is 10× faster and cheaper translates into feature delivery speed.

Unified observability in a heterogeneous cluster

The GPU observability post covered DCGM Exporter for NVIDIA. In a mixed cluster you need more:

Hardware part	Exporter	Key metrics
NVIDIA H100/A100	`nvidia/dcgm-exporter`	DCGM_FI_DEV_* + DCGM_FI_PROF_*
Intel Gaudi 3	`HabanaAI/habana-metric-exporter`	habana_hpu_utilization, habana_hbm_used
Intel Arc Pro	`intel/intel-gpu-exporter` (unofficial; alternatives exist)	xe_engine_utilization, xe_memory_used
Intel Xeon CPU + AMX	`prometheus/node-exporter` + Intel PCM	cpu_amx_utilization (via PCM)
Intel NUC (NPU+iGPU)	`intel/intel-gpu-exporter` + custom NPU exporter	npu_utilization, xe_iGPU

All federated into a single Prometheus + Grafana. The dashboards are organised by hardware family (NVIDIA, Intel server, Intel edge) plus an aggregate “heterogeneous cluster” view with per-tenant SLOs combining all four.

Cardinality: ~1.5-2× that of a pure NVIDIA cluster. Manageable with Thanos / Mimir for long retention.

Capability routing — from the L7 router to the heterogeneous one

The LLM inference router stops being a version selector for the same model and becomes a capability dispatcher:

models:
 - name: "llama-70b-chat"
 endpoint: "vllm-llama70b.inference.svc:8000"
 backend: nvidia-h100
 capabilities: [chat, tool_use, json_mode]
 - name: "embedding-multilingual"
 endpoint: "ipex-bge-m3.inference.svc:8080"
 backend: intel-xeon-amx
 capabilities: [embeddings]
 - name: "reranker-multilingual"
 endpoint: "ipex-bge-reranker.inference.svc:8080"
 backend: intel-xeon-amx
 capabilities: [reranking]
 - name: "guardrail-prompt-injection"
 endpoint: "openvino-llama-guard.edge-suc01.local:8080"
 backend: intel-nuc-edge
 capabilities: [guardrails, redact-pii]
 region: sucursal-01
 - name: "llama-3b-draft"
 endpoint: "openvino-llama-3b.edge-suc01.local:8080"
 backend: intel-nuc-edge
 capabilities: [speculative-drafter]
 region: sucursal-01
 target_model: "llama-70b-chat"

The router resolves model=embedding-multilingual → Intel Xeon; model=llama-70b-chat → H100; model=guardrail-prompt-injection with region=sucursal-01 → local NUC. If the branch office NUC goes down, failover to an equivalent replica in the central DC, accepting the latency cost.

LiteLLM Proxy, NVIDIA Dynamo and Envoy AI Gateway support this capability routing. The non-obvious piece: the router has to know the compatible tokenizer between drafter and target for pattern 3, which is modelled in extra catalogue metadata.

Specific pitfalls

Tokenizer mismatch between engines. OpenVINO with a Llama 3.2 GGUF and vLLM with the same nominal Llama 3.2 can use slightly different tokenizers (chat template, special tokens). Validate token identity with tokenizer.encode("hola") on both sides before assuming interchangeability. For speculative decoding, a single different token breaks the pattern.

Edge ↔ central round-trip latency. Patterns 2 and 3 assume the NUC and the DC are on the same corporate WAN with controlled latency. If the branch office is on 4G/5G with 100-200 ms of jitter, the speculative drafter compensates for nothing; on the contrary, it adds latency. Measure before promising.

FP8 Hopper ≠ INT8 AMX in output quality. The operator assumes a request that runs FP8 on an H100 and INT8 on a Xeon AMX will produce the same output. That is not true: the two quantisations have different degradation profiles. If the system expects idempotence (for example, evals with golden output), validate offline that the Intel version reproduces the expected behaviour within tolerance.

Model version synchronisation between sites. The model in the central DC gets updated, but the branch office NUCs keep the old version of the drafter or the guardrail for weeks. Result: divergent behaviour between branches with no easy diagnosis. Policy: the central model and the edge model advance together or within a documented window; the canary extends to the NUC fleet.

Unstable Intel roadmap. Falcon Shores cancelled, Gaudi 4 will never exist, Jaguar Shores 2026 still with no confirmed public specs. Buying Gaudi 3 today is reasonable if the use case justifies the 18-24 months of amortisation; committing an architecture for 5+ years on an Intel accelerator is a riskier bet than the NVIDIA equivalent, at least until Jaguar Shores materialises with a mature software stack.

Absence of public production deployments. As of June 2026, the confirmed Gaudi 3 deployments are IBM Cloud, Dell AI Factory and a handful of early adopters (Bharti Airtel, Bosch, Naver). There is no publicly verifiable case of a mixed NVIDIA + Intel cluster in European banking or government. This pattern is recommended emergent architecture, not practice with an industrial track record. The first adopter takes on a validation cost that a second adopter avoids.

Applied to a generic on-premise cluster

For an organisation with a generic heterogeneous LLM inference cluster:

Central DC: 4 nodes × 4×H100 SXM 80 GB with intra-node NVLink = 16 H100s. Serves the big LLMs (Llama 70B, Mixtral 8×22B, Qwen 72B) in BF16 or FP8.
Xeon 6 sidecar: 2-4 Xeon 6 servers (Granite Rapids 64-86 cores) with AMX, 512 GB DDR5, in the same rack as the H100 cluster. Serves embeddings (bge-m3), reranking (bge-reranker-v2-m3) and small models (Llama 3.2 1B/3B) on the vLLM CPU backend with AMX.
Arc Pro sidecar (optional): 1-2 servers with 4-8× Arc Pro B60 24 GB each (Battlematrix), for 13-30B INT8 models via OpenVINO. Useful if the cost per mid-sized LLM has to come down from the H100.
Edge NUCs in branch offices: 1-2 Panther Lake NUCs per branch, with a 50 TOPS NPU + Xe3 120 TOPS, serving Llama Guard 4 + Presidio + a Llama 3.2 1B INT4 drafter via OpenVINO. Corporate WAN connectivity with latency < 80 ms to the DC.

Estimated envelope: central cluster ~120 kW of GPU peak + ~10-15 kW of Intel sidecars. Edge: ~50 W per NUC, negligible compared with office costs.

Observability: federated Prometheus in the DC + pull scrape from the NUCs (via corporate VPN). Dashboards “NVIDIA GPU fleet”, “Intel server fleet”, “Intel edge fleet” plus a “consolidated SLO” view.

Router: LiteLLM Proxy or NVIDIA Dynamo in the DC, with an extended model catalogue to include Intel backends and regions (sucursal-01, sucursal-02, …). Documented edge→central failover.

What we have not covered (upcoming posts)

Reproducible benchmarks of Llama 70B on Gaudi 3 vs H100 SXM on equivalent hardware, the missing material for making decisions with your own data rather than Intel’s or Signal65’s.
AMD ROCm in the mix: how MI300X / MI355X fit into this heterogeneous pattern and what changes in the router catalogue.
Apple Silicon as edge: M3/M4 Max with a Neural Engine of ~38 TOPS + a 40-core GPU, hardware equivalent to the Panther Lake NUC but with a different software stack (MLX).
Energy cost optimisation: how nvidia-smi -pl 500W + an Intel TDP cap on Xeon 6 cuts the bill by 25-30 % with a 15-20 % throughput loss.
Model CI/CD for an edge fleet: how a rolling update of a Llama Guard reaches 50 branch office NUCs without any of them losing service.

References

Intel Gaudi 3

Intel — Gaudi 3 AI Accelerator White Paper. https://cdrdv2-public.intel.com/817486/gaudi-3-ai-accelerator-white-paper.pdf
Intel — Hot Chips 2024 Gaudi 3 deep dive. https://hc2024.hotchips.org/assets/program/conference/day1/60_HC2024.Intel.RomanKaplan.Gaudi3-0826.pdf
Signal65 / DataCenterDynamics — NVIDIA H200 outperforms Intel Gaudi 3 by factor of 9× across first Llama 3.1 405B benchmark test. https://www.datacenterdynamics.com/en/news/nvidia-h200-outperforms-intel-gaudi-3-by-factor-of-nine-across-first-llama-31-405b-benchmark-test-exclusive/
IEEE Spectrum — Intel Gaudi 3 review. https://spectrum.ieee.org/intel-gaudi-3
Tom’s Hardware — Intel cancels Falcon Shores GPU; Jaguar Shores to be successor. https://www.tomshardware.com/tech-industry/artificial-intelligence/intel-cancels-falcon-shores-gpu-for-ai-workloads-jaguar-shores-to-be-successor

Intel Xeon 6 + AMX

Intel — Xeon 6 (Granite Rapids) Product Brief. https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2025-02/xeon-6-granite-rapids-product-brief.pdf
OpenMetal — Intel AMX AI Inference Performance. https://openmetal.io/resources/blog/intel-amx-ai-inference-performance/
LMSYS — Intel Xeon 6 + SGLang for DeepSeek R1 671B. https://www.lmsys.org/blog/2025-07-14-intel-xeon-optimization/
arXiv 2410.04466 — CPU-LLM benchmarks with AMX.
Intel community blog — Accelerating vLLM Inference on Intel Xeon 6 Processor.

Intel Arc Pro Battlemage

Intel — Arc Pro B60 Graphics Specifications. https://www.intel.com/content/www/us/en/products/sku/243916/intel-arc-pro-b60-graphics/specifications.html
StorageReview — Intel Arc Pro B60 Battlematrix Preview: 192GB VRAM for On-Premise AI. https://www.storagereview.com/review/intel-arc-pro-b60-battlematrix-preview-192gb-of-vram-for-on-premise-ai
Phoronix — Intel Arc Pro B-series review. https://www.phoronix.com/review/intel-arc-pro-b-series

Intel NUC / NPU

HotHardware — Intel CES 2026 Panther Lake is a Go. https://hothardware.com/news/intel-ces-2026-panther-lake-is-a-go
TechPowerUp — Intel Panther Lake Technical Deep Dive.
arXiv 2412.11053 — NITRO: LLM inference on laptop NPU.
Intel — AI PC brings larger LLM development to your desk.

Software

OpenVINO — Release Notes 2025.3. https://www.intel.com/content/www/us/en/developer/articles/release-notes/openvino/2025-3.html
HuggingFace — Deploy with OpenVINO. https://huggingface.co/blog/deploy-with-openvino
Intel — Intel Extension for PyTorch XPU 2.8.10. https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/releases.html
IPEX-LLM — github.com/intel/ipex-llm.
Habana — SynapseAI PyTorch Theory of Operations. https://docs.habana.ai/en/latest/PyTorch/PyTorch_Gaudi_Theory_of_Operations.html

Market context

MLCommons — MLPerf Inference v6.0 benchmark results. https://www.spheron.network/blog/mlperf-inference-v6-benchmark-results-2026/
Intel newsroom — Gaudi 3 Expanded Availability. https://newsroom.intel.com/artificial-intelligence/intel-gaudi-3-expands-availability-drive-ai-innovation-scale

Sources: the full URLs are linked inline on each reference.

Edge on lo0 — Tech Blog

Device-native architectures: fine-grained MoE and pre-attention router

TL;DR

The analogy: the librarian who anticipates your request

Compressing a dense model vs. designing for device

Two levels of sparsity

The pre-attention router: predict and prefetch

The maths that matter

Memory footprint: resident N vs. A + cache

The prefetch calculation, with numbers

Weight footprint: why they report ~1 GB for a 4B

The quality cost: the necessary scepticism

Implications for on-premise and edge inference

Conclusion

See also

References

Mixed NVIDIA + Intel environments for LLM inference: from the central H100 cluster to the NUC in the branch office

TL;DR

You are here: DEPLOY (with heterogeneity as a decision)

The analogy: the factory with several different machines

Three operational reasons for heterogeneity

Reason 1 — cost

Reason 2 — sovereignty and supply chain diversification

Reason 3 — edge

Relevant Intel hardware (June 2026)

Intel Gaudi 3 — the critical note on the marketing

Intel Xeon 6 with AMX — the CPU workhorse

Intel Arc Pro B60 and Battlematrix

Intel NUC with NPU — the edge node

Intel software — the relevant stack

The four canonical patterns

Pattern 1 — embeddings + reranker on Intel

Pattern 2 — guardrails + PII redaction on a near-edge NUC

Pattern 3 — speculative decoding drafter on a NUC

Pattern 4 — NUC dev workstations

Unified observability in a heterogeneous cluster

Capability routing — from the L7 router to the heterogeneous one

Specific pitfalls

Applied to a generic on-premise cluster

What we have not covered (upcoming posts)

See also

References