Model Distillation

Overview

Model distillation compresses knowledge from a large “teacher” model into a smaller “student” model.

In the LLM era, distillation is often paired with:

• synthetic datasets generated by stronger models
• step-by-step rationales used as supervision
• preference optimization

Why it is hyped

Distillation is a path to:

• lower inference cost
• on-device / edge deployment
• faster iteration in production

Research directions

• “Distilling step-by-step” uses LLM rationales as additional supervision to train smaller models.
• “Branch-Merge Distillation” distills into specialized students then merges them.

Related pages

• Synthetic Data
• Small Reasoning Models

Sources

See citations.