Learning Rate Warmup

Medium

neural-networks Google OpenAI Anthropic

Many transformer training recipes include a learning rate warmup phase where the learning rate linearly increases from near zero. Why is this warmup necessary?

Multiple Choice

It prevents large, unstable updates early in training when adaptive optimizer statistics (like Adam's moment estimates) are poorly initialized It allows the tokenizer to calibrate its vocabulary before the model begins learning It pre-computes the positional encodings so they are ready when real training begins It ensures the batch normalization running statistics converge before the learning rate is increased

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations

Learning Rate Warmup

Multiple Choice

Related problems

Deep-dive guides