What is knowledge distillation? Smaller AI models

What knowledge distillation means

Knowledge distillation is a training method in which a small model learns to behave like a large one. It also goes by model distillation or AI distillation. The large model is called the teacher, and the small one being trained is the student.

Knowledge distillation at a glance

Also called	Model distillation, AI distillation
Teacher	A large model that is already trained
Student	A smaller model trained to imitate the teacher
Goal	A model that is cheaper and faster to run
Named in	Hinton, Vinyals and Dean, 2015

A small student model learns from a large teacher

The teacher is trained first, and the student is then trained to give the same kind of answers the teacher gives. The student does not need to repeat the teacher's long, expensive training on raw data. It learns from what the teacher has already worked out.

The student has far fewer parameters (the numbers a model adjusts as it learns), so it needs less memory and answers faster. It usually ends up a little less capable than its teacher, because a small model cannot match the teacher's answers exactly.

Hinton and two colleagues named it in 2015

The name comes from a paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, posted to arXiv on March 9, 2015. They described training a large, "cumbersome" model first, then using a different kind of training, which they called distillation, to move what it had learned into a small model better suited to deployment.

The idea was not entirely new. The paper credits earlier work by Rich Caruana and colleagues, who had shown that the knowledge in a group of models could be squeezed into a single model.

Why AI companies distill their models

The biggest models give the best answers, but they are expensive and slow to run at scale. Distillation keeps much of that quality in a model that costs far less to serve.

DistilBERT compared with BERT (BERT = 100)

Model size

DistilBERT is 40% smaller. Source: Sanh et al., 2019.

0255075100

100BERT

60DistilBERT

Language understanding

DistilBERT keeps 97% of BERT's score. Source: Sanh et al., 2019.

0255075100

100BERT

97DistilBERT

Large models cost too much to serve

Training happens once, but a model that serves many users has to answer every request quickly and cheaply. Hinton's paper points out that averaging the answers of many models improves accuracy, yet can take too much computation to serve to a large number of users.

The paper compares this to insects: many have one form as larvae for feeding and another as adults for travel and reproduction. A model can be built the same way: one big form for learning and a small one for answering.

DistilBERT kept 97% of BERT's language understanding

DistilBERT, released in 2019, is 40% smaller than BERT, keeps 97% of its language understanding and runs 60% faster. BERT is an earlier language model, and Sanh and colleagues built DistilBERT by distilling it.

They applied distillation during pre-training, the student's first stage of training, rather than afterward. The cost was a 3% drop in language understanding, in exchange for a much smaller, faster model.

How knowledge distillation works

The student learns from how likely the teacher thinks every possible answer is, not only from which answer it picks. That extra detail is what lets a small model pick up so much from a large one.

Two ways to train a small model

	Trained on labels only	Distilled from a teacher
What it learns from	The one correct answer	The teacher's odds for every answer
Example for a photo of a BMW	"Car"	Very likely a car, a tiny chance of a truck, almost no chance of a carrot
What it picks up	What is right	What is right and what is close

Soft targets teach more than the right answer

When a model looks at an input, it assigns a probability to every possible answer, and distillation uses that whole set as the lesson. Hinton's paper calls these probabilities soft targets.

The paper's example is a photo of a BMW. The teacher may rate the chance that it is a garbage truck as very small, but that is still many times higher than the chance it is a carrot. Those tiny differences in wrong answers show which things the teacher sees as similar, and the student learns that too.

A higher temperature brings out the small probabilities

The smallest probabilities are often too close to zero to teach anything, so distillation raises a setting called temperature to spread them out. A higher temperature makes the teacher's answer less peaked, so the gaps between unlikely answers become large enough for the student to learn from.

The student is trained at that same high temperature. Once training is done, it goes back to the normal temperature of 1 and answers like any other model.

How large language models use distillation

With large language models (LLMs), the AI behind chatbots, distillation often means training a small model on text that a big model has written. DeepSeek's R1 shows what this looks like in practice.

DeepSeek-R1 and its distilled models

Teacher	DeepSeek-R1 (671B parameters in total, 37B used per token)
Students	Six models from 1.5B to 70B, based on Llama and Qwen
Training data	800,000 samples curated with DeepSeek-R1
Standout result	R1-Distill-Qwen-32B beat OpenAI-o1-mini on several benchmarks
License	Allows distillation for training other LLMs

DeepSeek distilled R1 into six smaller models

DeepSeek released six distilled versions of its R1 reasoning model, from 1.5 billion to 70 billion parameters, built on the Llama and Qwen model families. R1 itself has 671 billion parameters, of which 37 billion are used for each token, a design we explain in "What is mixture of experts (MoE)? How it works."

The 32-billion-parameter version, R1-Distill-Qwen-32B, beat OpenAI's o1-mini on several benchmarks. DeepSeek's takeaway was that the reasoning patterns of a big model can be passed down to small ones, and that this works better than training the small model to reason on its own.

DeepSeek's students trained on 800,000 R1 answers

DeepSeek fine-tuned its students on 800,000 samples curated with R1. They learned from R1's written answers, not from the full set of probabilities the 2015 method used.

DeepSeek's license explicitly allows this kind of use. The R1 weights can be downloaded, modified and used to train other LLMs, as with other open models described in "What is an open weight model? AI you can download."

When distillation breaks AI terms of service

Distilling your own model, or an open one whose license allows it, is normal practice; training a rival model on another company's output is not. OpenAI and Anthropic forbid it, and Anthropic calls large covert efforts of this kind distillation attacks.

Where distillation is allowed and where it is not

Case	Allowed?
A lab distills its own large model	Yes, routine practice
Distilling an open model whose license allows it, such as DeepSeek-R1	Yes
Training a competing model on ChatGPT output	No, against OpenAI's Terms of Use
Using Claude to train a competing model	No, against Anthropic's Commercial Terms

OpenAI and Anthropic ban training rivals on their output

OpenAI's Terms of Use, under "What you cannot do," forbid using its output to develop models that compete with OpenAI. Anthropic's Commercial Terms, in section D.4, forbid customers from using its services to build a competing product, including training competing AI models.

Both bans cover using the output to compete with OpenAI or Anthropic, not distillation as such. Using ChatGPT or Claude to draft text for your own work is not affected.

Anthropic says three labs ran distillation attacks on Claude

On February 23, 2026, Anthropic said DeepSeek, Moonshot and MiniMax had generated more than 16 million exchanges with Claude through about 24,000 fraudulent accounts. It called this an effort to extract Claude's capabilities to improve their own models, in breach of its terms of service.

Even so, Anthropic calls distillation itself a widely used, legitimate method and says frontier labs routinely distill their own models into smaller, cheaper versions. The White House made the same distinction in July 2026, as we cover in "Kimi K3 Distillation: What the White House Claims." For research on military uses, see "AI Distillation Attacks in PLA-Linked Research."

Distillation trades some ability for speed and cost

Knowledge distillation trains a small student model to imitate a large teacher model, so the result runs cheaper and faster while keeping most of the teacher's ability. The student learns from the teacher's probabilities for every answer, and DistilBERT and DeepSeek's six distilled R1 models show how much of a big model's ability a small one can keep.

Labs distill their own models all the time, and open licenses such as DeepSeek's allow it. Training a competing model on ChatGPT or Claude output breaks OpenAI's and Anthropic's terms, and Anthropic calls large covert campaigns of this kind distillation attacks.