Pruning LLM models: cutting without amputating

· By José Manuel Román
Contents

TL;DR

A 7B-parameter model has tens of billions of neural connections. Many of them contribute so little that you could remove them without any reasonable benchmark noticing. That is pruning: identifying the irrelevant weights and deleting them to obtain a model that is smaller, faster or lighter on memory. Modern techniques (SparseGPT, Wanda, 2:4 structured sparsity) do this without retraining, in a few GPU hours, and with less than 1 point of perplexity penalty. It does not replace quantisation; it combines with it.

The analogy

An oak tree with a hundred branches. When winter comes, the tree prunes its weak branches: it redirects resources towards the main trunks. An expert pruner does not cut at random, they look at which branches have little foliage, which are dry, which grow in the wrong direction, and cut only those.

A language model is that tree. Its “branches” are the weights connecting neurons. After training, many of those connections are vestiges of the optimisation process: they existed so the gradient could descend smoothly, but in production they barely change the output. The pruner that removes them precisely is SparseGPT or Wanda. The one that cuts at random is magnitude pruning without calibration. Both give you a smaller tree; only the expert gives you one that still bears the same fruit.

What pruning really is

A transformer language model stores its knowledge in weight matrices. An attention layer has four matrices: $W_Q, W_K, W_V, W_O$. An FFN layer has at least two ($W_{up}, W_{down}$, plus $W_{gate}$ in SwiGLU). For a 7B model with 32 layers, the number of individual parameters exceeds 7 billion.

Pruning is the process of setting a subset of those parameters to zero so that:

  1. The resulting model takes up less memory (if stored in a sparse format) or computes fewer operations.
  2. Answer quality does not drop appreciably.

There are two classification dimensions that matter:

Granularity: which unit gets removed.

  • Unstructured pruning: individual weights, scattered across the whole matrix. High compression, hard to accelerate on conventional hardware.
  • Structured pruning: whole attention heads, entire FFN neurons, or complete layers. Less compression, but the resulting model is dense and compatible with any hardware.
  • Semi-structured N:M: for each group of M consecutive weights, exactly N are zero. The 2:4 case (2 zeros out of every 4) is the one supported by the Tensor Cores in NVIDIA Ampere and later.

Timing: when the removal happens.

  • Post-training (weight PTQ): requires no gradient, the standard in large LLMs.
  • During training (gradual/iterative): more precise, incompatible with 70B+ models on cost grounds.

Why so many redundant weights exist

The answer lies in how the models are trained. Stochastic gradient descent with millions of steps and a decaying learning rate produces networks that are over-parameterised by design: the extra parameters do not represent additional knowledge, but room for manoeuvre so that optimisation converges more easily.

The Lottery Ticket Hypothesis (Frankle & Carlin, ICLR 2019) formalised this intuition: inside any trained dense network there is a subnetwork that, trained from scratch in isolation, reaches the same quality. The original network is that subnetwork wrapped in parametric noise generated by the training process.

For LLMs, the empirical evidence is consistent: 7B–70B models tolerate up to 50% unstructured sparsity with no observable degradation on conversational tasks. In larger models, the tolerance threshold rises.

The maths that matter

Which weights are safe to remove?

Magnitude pruning: the naive criterion

$$\text{importance}(w_{ij}) = |w_{ij}|$$

The weights with the smallest absolute value are removed. Intuitive, but incomplete: a small weight connected to a very large activation still contributes significantly to the output.

Wanda: magnitude × activation

$$\text{importance}(w_{ij}) = |w_{ij}| \cdot \|x_j\|_2$$

Where $x_j$ is the input activation vector corresponding to weight $j$, computed over a calibration dataset of ~128 samples. The product captures both dimensions: a weight is safe to remove only if it is small and its input neuron is barely active.

Numerical example:

  • Weight A: $|w| = 0.001$, $\|x\|_2 = 500$ → importance = 0.5
  • Weight B: $|w| = 0.01$, $\|x\|_2 = 10$ → importance = 0.1

Magnitude pruning would remove A (smaller absolute value). Wanda removes B (lower importance). B is the safer one to delete.

Wanda requires neither gradients nor Hessian matrix inverses. It runs in minutes on a 70B model on a single GPU. On WikiText-2 perplexity benchmarks at 50% unstructured sparsity, Wanda reaches results comparable to SparseGPT at 10–100× less computational cost.

SparseGPT: Hessian compensation

SparseGPT applies the same mathematical framework as GPTQ (layer-by-layer quantisation), but for pruning. When it removes a weight $w_p$, it computes a correction $\delta w$ over the remaining weights in the same row to minimise the change in the layer’s output:

$$\min_{\delta w} \|W x - (W + \delta W) x\|_2^2 \quad \text{s.t.} \quad w_p + \delta w_p = 0$$

The solution uses the inverse of the second-order Hessian matrix $H = X X^T$. The extra cost is justified by the higher precision when target sparsity is high (>70%) or the model is small (<7B, where redundancy is lower).

MethodCriterionCost50% sparsity (7B, WikiText-2 ppl)
Magnitude|w|Instant+2–5 points
Wanda|w| · |x|Minutes~+0.5 points
SparseGPTHessian1–4h GPU~+0.4 points

2:4 Structured Sparsity: NVIDIA’s special case

NVIDIA Ampere (A100) and later (H100, Ada Lovelace) include dedicated hardware for the 2:4 pattern: exactly 2 out of every 4 consecutive weights are zero. This lets the hardware skip the multiplications by zero efficiently, obtaining up to 2× speedup in matmul on models with 2:4 weights.

The constraint is that the sparsity has to be exactly 2:4, not an arbitrary pattern. NVIDIA’s tools (APEX Sparse, cuSPARSELt) and frameworks such as PyTorch 2.x support this natively:

from torch.sparse import to_sparse_semi_structured, SparseSemiStructuredTensor

# Convert dense weights to 2:4 sparse
sparse_weight = to_sparse_semi_structured(dense_weight)
# Forward pass automatically uses sparse tensor cores
output = F.linear(input, sparse_weight)

What to expect in practice with 2:4:

  • RTX 4090 (Ada Lovelace): supports 2:4 sparse tensor cores for FP16/BF16. Theoretical speedup 2×, real 1.3–1.6× depending on batch and sequence size.
  • H100 (Hopper): the same, with additional improvements in FP8 + 2:4 combined.
  • A100: supported, without FP8.
  • Consumer GPUs earlier than Ada (3090, etc.): no hardware support. 2:4 sparsity gives a smaller model on disk but does not accelerate inference.

Structured pruning: removing whole heads and layers

Attention head pruning

A 32-layer transformer with 32 heads per layer has 1,024 attention heads. Systematic studies on Llama-2 and Qwen models show that between 20–40% of the heads have marginal influence on the final output: their output can be set to zero without the benchmark changing within the margin of error.

The most used metric is Taylor importance: the product of the gradient of the loss with respect to the head’s output and the value of that output, summed over a calibration dataset:

$$\text{I}_{head} = \left| \sum_t \frac{\partial \mathcal{L}}{\partial o_t} \cdot o_t \right|$$

The heads with the lowest $I_{head}$ are removed first. After removing 25% of the heads in Llama-3-8B, MMLU degradation is <1% and attention inference time falls ~20% because the attention matmuls are smaller.

Layer dropping: the most aggressive shortcut

Removing a whole transformer layer deletes its attention block and its FFN. The most robust criterion is Block Influence (BI), introduced in ShortGPT (2024):

$$\text{BI}(l) = 1 - \cos(\text{input}_l, \text{output}_l)$$

A layer whose output is almost identical to its input (cosine close to 1, BI close to 0) acts as an identity function: removing it does not change the flow of information. The layers in the middle of the transformer tend to have lower BI than the initial and final layers.

Numerical example on LLaMA-2-70B:

  • Layers 0–5 (early): BI > 0.3 → do not remove
  • Layers 20–45 (mid): BI < 0.05 → candidates for removal
  • Layers 76–80 (final): BI > 0.2 → do not remove

Removing 8 layers out of 80 (10%): the model goes from ~140 GB to ~126 GB in BF16. Inference speedup: ~10% (proportional to the number of layers removed). Degradation on reasoning benchmarks: 1–3%.

Implications for on-premise inference

Unstructured pruning (50% sparsity) produces models with the same number of parameters but half of them at zero. Without specialised sparse kernels, that gives no speedup: the GPU still runs the multiplications, it just multiplies by zero very efficiently. The real benefit is storage and transfer (the model takes up less on disk and in system RAM).

With 2:4 structured sparsity on Ada/Hopper hardware, the speedup is real but moderate (1.3–1.7×) and requires additional tooling (cuSPARSELt or PyTorch sparse).

Structured pruning (heads, layers) does accelerate on any hardware because it reduces the model’s real size. It is the right option if the goal is throughput on hardware without sparse tensor cores.

Combining with quantisation: pruning and quantisation are orthogonal. A 50% sparse model at INT4 takes up roughly an eighth of the original in FP32. It is the end point of many aggressive compression pipelines for edge inference.

Applied to generic on-premise hardware

RTX 4090 (24 GB, Ada Lovelace)

Supports 2:4 sparse tensor cores for FP16/BF16. With Wanda + 2:4 sparsity on a Qwen2.5-14B:

# Pruning pipeline: Wanda 2:4 + INT4 quantisation
# 1. Run Wanda with calibration over 128 samples
python wanda/main.py \
  --model Qwen/Qwen2.5-14B \
  --sparsity_ratio 0.5 \
  --sparsity_type 2:4 \
  --save pruned_model/

# 2. Quantise the pruned model (optional but complementary)
python -m awq.entry --model_path pruned_model/ \
  --w_bit 4 --output_path pruned_awq_model/

Expected result: ~13 GB BF16 → ~6.5 GB after 2:4 pruning in sparse format → ~3.2 GB with AWQ INT4. The 14B model will fit in the RTX 4090 with room for KV cache.

4× H100 SXM (320 GB total, Hopper)

On this hardware structured pruning (layer dropping) makes more sense than 2:4 for high-concurrency inference: you reduce the number of FLOPs per token proportionally, which benefits throughput at large batch where the bottleneck is compute, not memory.

# Apply layer dropping with the ShortGPT BI metric
from shortgpt import compute_block_influence, drop_layers

bi_scores = compute_block_influence(model, calibration_data)
# Remove the 15% of layers with the lowest BI
model = drop_layers(model, bi_scores, drop_ratio=0.15)

A Llama-3-70B pruned by 15% of its layers fits in 3 H100s instead of 4, freeing a GPU for another task.

See also

  • https://blog.lo0.es/en/posts/quantization-llm-inference-fp8-int4-gguf/ — the complementary lever: quantising reduces the precision of the weights pruning has decided to keep; combined they give maximum compression
  • https://blog.lo0.es/en/posts/kv-cache-working-memory-llm-inference/ — pruning reduces the model’s size, but the KV cache keeps growing with context; they are separate costs in VRAM
  • https://blog.lo0.es/en/posts/speculative-decoding-fundamentals-maths-state-of-play/ — speculative decoding drafters are often pruned versions of the base model, not models trained from scratch
  • https://blog.lo0.es/en/posts/optimising-decode-vllm-squeezing-every-token-small-hardware/ — how the pruned model is served in vLLM: throughput parameters change with a structurally smaller model
  • https://blog.lo0.es/en/posts/knowledge-distillation-teaching-small-model-think-like-big-one/ — a conceptual alternative to pruning: instead of removing parts of the large model, train a small one to imitate its behaviour

References