Model Distillation

Medium

llms Google NVIDIA Microsoft

How does knowledge distillation work in the context of LLMs?

Multiple Choice

The teacher model's weights are directly copied into a smaller architecture A smaller student model is trained to match the teacher model's output probability distributions The teacher model is pruned by removing unnecessary layers The training data is compressed and the model is retrained from scratch

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations

Model Distillation

Multiple Choice

Related problems

Deep-dive guides