Knowledge Distillation: teaching a small model to think like a big one

· By José Manuel Román
Contents

TL;DR

Knowledge Distillation is the technique of training a small model (student) using the output probabilities of a large model (teacher) as supervision, instead of using only the hard labels from the training dataset. The result is a small model that reasons better than its size suggests, because it learns the teacher’s uncertainty distributions rather than memorising binary answers. It is the reason Phi-4 (14B) beats most 70B models at reasoning, and why the Gemma 3 family models are surprisingly capable for their size. It is not a compression technique for an existing model: it is a training process that produces a smaller model from scratch or from a different starting point.

The analogy

A master surgeon with thirty years of experience and a first-year resident. If the resident only learns from the anatomy manual (binary correct answers: “cut here, not here”) it will take years to develop the master’s clinical judgement. But if they operate alongside them, watching their micro-decisions, their hesitations, the ambiguous cases where the master knows that two options are almost equally valid, they learn something the manual cannot teach: the structure of uncertainty.

Knowledge distillation is exactly that. The “anatomy manual” is the hard labels (the correct answer). The “master surgeon” is the teacher LLM. The probability distributions over the vocabulary are the materialisation of that uncertainty the student absorbs.

What it really is

When an LLM generates text, it does not produce a single word: it produces a probability distribution over its whole vocabulary at each position. For the next token, the model might say:

"Paris":     42%
"Lyon":       8%
"Marseille":  6%
"the city":   5%
...rest of the vocabulary: 39%

This distribution is dense information. It tells you not only what the correct answer is, but also which other answers were plausible and to what degree. A student trained only with the label “Paris” (probability 1.0 on the correct token, 0.0 on the rest) never sees this richness.

Distillation uses the teacher’s full distribution as the student’s training target. The loss function has two terms:

$$\mathcal{L}_{total} = (1 - \alpha) \cdot \mathcal{L}_{CE}(y, \hat{y}_S) + \alpha \cdot \mathcal{L}_{KD}(p_T, p_S, T)$$

Where:

  • $\mathcal{L}_{CE}$ is the standard cross-entropy with the hard labels (classic supervision).
  • $\mathcal{L}_{KD}$ is the KL divergence between the teacher’s and the student’s distributions.
  • $\alpha$ controls the relative weight of each term (typically 0.5–0.9 in favour of KD).
  • $T$ is the temperature, a parameter that softens the distributions to make the KD signal more informative.

The role of temperature

If the teacher assigns 99% to “Paris” and 0.001% to every other word, the distribution is almost as informative as a hard label. A temperature $T > 1$ softens that distribution:

$$p_T(k) = \frac{\exp(z_k / T)}{\sum_j \exp(z_j / T)}$$

With $T = 4$ and the original logits, the distribution that was previously [99%, 0.001%, 0.001%…] becomes something like [42%, 8%, 6%…]. The student sees the teacher’s real probability neighbourhood, not just its point answer.

Numerical example with temperature:

Teacher logits for “The capital of France is _____”:

Paris:    8.5
Lyon:     3.2
Europe:   2.1
a:        1.8

With T=1 (standard softmax):

$$p(\text{Paris}) = \frac{e^{8.5}}{e^{8.5} + e^{3.2} + e^{2.1} + e^{1.8}} \approx 99.3\%$$

With T=4:

$$p(\text{Paris}) = \frac{e^{8.5/4}}{e^{8.5/4} + e^{3.2/4} + e^{2.1/4} + e^{1.8/4}} = \frac{e^{2.125}}{e^{2.125} + e^{0.8} + e^{0.525} + e^{0.45}} \approx 54\%$$

The signal at T=4 is far more informative for the student: it learns that Lyon is more plausible than Europe, that Europe is more plausible than “a”, and so on.

The three modes of distillation

Offline (or “black-box”)

The teacher generates a synthetic dataset of answers before training. The student is trained on that dataset as if it were ordinary hard labels.

teacher → generates 100M (prompt, completion) pairs → dataset
student  → is trained on that dataset

It is the cheapest way to scale: the teacher runs only once, the student is trained on the generated data with conventional hardware. Most open source instruction models (Alpaca, Vicuna, WizardLM in their early versions) used this strategy: GPT-4 as teacher, saved data, Llama-7B as student.

Limitation: the student never sees the teacher’s probability distributions, only its answers. It is distillation of “behaviour”, not of “knowledge” in the strict sense. If the teacher makes a mistake (and GPT-4 does make mistakes), the error is crystallised into the dataset.

Online (or “white-box”)

Teacher and student run together during training. The student processes each batch, the teacher processes the same batch in parallel, and the KD loss is computed in real time with the full probability distributions.

for batch in dataset:
    logits_teacher = teacher(batch)   # teacher forward pass
    logits_student = student(batch)   # student forward pass
    loss = KL(softmax(logits_teacher/T), softmax(logits_student/T))
    loss.backward()                    # only updates student

The teacher has gradients disabled (torch.no_grad()). The learning signal is richer than in offline, but the cost is high: you need to keep the teacher in VRAM throughout training. To distil a 405B teacher into an 8B student, you would need several H100s for the teacher alone.

On-policy

A recent variant (2024–2026) that combines the best of both: the teacher generates answers dynamically during training, but the student evaluates them with its own distribution. The cycle is:

  1. The student generates a proposed answer (rollout).
  2. The teacher scores that proposal with its probability distribution.
  3. The student updates with the teacher’s signal.

This stops the student from learning from distributions outside its own domain (the distribution shift problem in offline). It is the basis of algorithms such as SimCT (2026) that use teachers from different families (Qwen, Phi, Gemma) to generate a cross-tokeniser signal.

Why the best small models use distillation

Phi-4 (Microsoft, 14B), Gemma 3 (Google, 9B/27B), and the compact Qwen3 family models are the clearest examples. Their benchmarks are anomalous with respect to their size: Phi-4-14B beats LLaMA-3-70B on MATH and GPQA-Diamond, two mathematical and scientific reasoning benchmarks where size is usually decisive.

Why? The key is what supervises the training:

  • A model trained on internet data learns the distribution of human text, which includes a lot of low-quality text, errors and ambiguities.
  • A student that learns from a frontier teacher (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro) absorbs a distribution filtered towards high-quality text and correct reasoning.

The student with 14B parameters does not “know more” than an undistilled one of the same size, but it has learned to use them better because its training gradients were never contaminated by low-quality text.

Empirical data point: Phi-4 (14B distilled) vs LLaMA-3-70B (not distilled) on the MATH benchmark (2025):

  • Phi-4: 80.4%
  • LLaMA-3-70B: 68.0%

A model 5× smaller beats the large one because the training signal is better, not because it has more parameters.

Reasoning distillation: the case of thinking models

Reasoning models (DeepSeek-R1, Qwen3-thinking, QwQ) generate internal chains of thought before giving the final answer. Distilling reasoning is more complex because you do not only want to transfer the answer: you want to transfer the way of thinking.

The current strategy (2025–2026) is reasoning trace distillation:

  1. The teacher (a large thinking model) generates answers with its full internal chain of thought.
  2. The dataset includes those chains of thought as part of the output.
  3. The student learns to imitate both the chain and the final answer.

This explains why Qwen3-7B-thinking can reason formally about mathematics while being 10× smaller than the models that preceded it without distillation: it learned the process, not just the result.

When to use distillation vs. the alternatives

TechniqueWhat it doesRequires retrainingResult
QuantisationReduces weight precisionNoSame model, smaller
PruningRemoves irrelevant weightsNo (PTQ)Same model, sparser
DistillationTrains a new modelYesDifferent model, smaller

Distillation does not compress an existing model: it produces a new one. That is why it is complementary, not a substitute: you can distil a 405B into an 8B, and then quantise that 8B to INT4 to reduce its inference cost.

When it is the right option:

  • You need a model 5–10× smaller than the best available one.
  • You have access (API or local) to a quality teacher.
  • You have training data or the ability to generate it.
  • Latency or inference cost is a hard constraint.

When it is not:

  • You want to compress an existing model quickly: use quantisation + pruning.
  • You have no training budget (online distillation takes weeks of GPU).
  • The teacher is not significantly better than the base student: the KD signal will be weak.

Implications for on-premise inference

In a sovereign deployment, the teacher can be a large model running locally (no external API needed). The flow is:

generic 4×H100:
  teacher: Llama-3.3-70B-Instruct (on the 4×H100, full load)
  → generates a dataset of 10M (prompt, completion with logits) pairs
  → 3-4 weeks of generation at batch 32

  After the dataset:
  student: Qwen2.5-7B (fine-tuned with KD loss over the dataset)
  → 2-3 days of training on the same H100s
  → result: a 7B that reasons like the 70B in the specific domain

Production:
  RTX 4090: serves the 7B student quantised to INT4 (4 GB)

The teacher is only needed to generate the data. The student is what goes into production. The investment in training compute pays for itself in months of cheaper inference.

For ENS/NIS2: this flow is 100% on-premise, zero dependency on external APIs, and the resulting model is yours in every sense.

See also

  • https://blog.lo0.es/en/posts/llm-pruning-cutting-without-amputating/ — a technical alternative: instead of training a new model, remove parts of the existing model; distillation and pruning are complementary
  • https://blog.lo0.es/en/posts/quantization-llm-inference-fp8-int4-gguf/ — the next step after distilling: quantise the student for efficient inference
  • https://blog.lo0.es/en/posts/speculative-decoding-fundamentals-maths-state-of-play/ — speculative decoding drafters are frequently students distilled from the base model that learn to predict its distribution
  • https://blog.lo0.es/en/posts/continuous-fine-tuning-production-real-traffic-deployed-adapter/ — distillation as a form of continuous fine-tuning: the teacher is the model in production, the student is the next version
  • https://blog.lo0.es/en/posts/modern-alignment-dpo-kto-orpo-simpo/ — DPO and its variants can be seen as distilling human preferences into the model; the maths of the reference distribution is analogous to the teacher in KD

References