Neural Network Optimizers: Interview Questions

Why Optimizer Questions Show Up in Almost Every Deep Learning Interview

Choosing and configuring an optimizer is one of the few deep learning decisions every practitioner makes on every project, which is exactly why interviewers treat it as a reliable signal of hands-on experience. Anyone can name "Adam" as an answer; fewer candidates can explain what its two moving averages are actually doing, or why a learning rate schedule matters as much as the optimizer choice itself.

This guide covers six concepts that come up together in these loops: how Adam actually works, why Adagrad's learning rate decays to nothing, learning rate warmup, cosine annealing, and the two dominant weight initialization schemes. Each section links a practice problem.

Adam: Momentum and Adaptive Learning Rates Combined

"Explain what Adam is actually computing at each step, not just that it's 'the default optimizer.'"

Adam maintains two running averages per parameter. The first moment, m_t = β1 · m_{t-1} + (1 − β1) · g_t, is an exponential moving average of the raw gradient g_t, typically with β1 = 0.9; this is the momentum term, smoothing out noisy gradient directions across steps. The second moment, v_t = β2 · v_{t-1} + (1 − β2) · g_t², is an exponential moving average of the squared gradient, typically with β2 = 0.999; this tracks how large the gradients for each parameter have recently been.

Because both moving averages start at zero, they're biased toward zero in early training, so Adam applies a bias correction: m̂_t = m_t / (1 − β1^t) and v̂_t = v_t / (1 − β2^t). The final update is θ ← θ − lr · m̂_t / (√v̂_t + ε). Dividing by the square root of the second moment is what gives Adam its adaptive-learning-rate behavior — parameters with a history of large gradients get their effective step size shrunk, while parameters with small or infrequent gradients keep a relatively larger effective step size.

The framing interviewers reward: Adam is RMSProp's per-parameter adaptive learning rate combined with momentum, plus bias correction to fix the zero-initialization problem at the start of training. Candidates who can state that decomposition, rather than just "Adam adapts the learning rate," are demonstrating they've actually looked at the update rule. Practice: Adam Optimizer Internals.

Adagrad's Core Limitation: A Learning Rate That Never Recovers

"Adagrad also adapts the learning rate per parameter. Why isn't it used as often as Adam or RMSProp?"

Adagrad divides the learning rate for each parameter by the square root of the sum of all squared gradients that parameter has received so far in training: θ ← θ − (lr / √(G_t + ε)) · g_t, where G_t accumulates g² across every step. This per-parameter adaptation is genuinely useful early on — it naturally gives smaller effective steps to frequently updated parameters and larger steps to rarely updated ones, which made Adagrad attractive for sparse data like text features.

The problem is that G_t is a running sum that only grows and never decays. As training continues, G_t keeps increasing for every parameter, so the effective learning rate keeps shrinking monotonically toward zero. On a long training run this means the optimizer can effectively stop learning well before the model has converged, regardless of how the loss landscape actually looks at that point.

RMSProp and Adam both fix this with the same idea: replace Adagrad's unbounded cumulative sum with an exponential moving average of squared gradients, which lets old gradient information decay over time instead of accumulating forever. That's the single change that keeps the effective learning rate from collapsing, and it's the detail that separates "Adagrad adapts learning rates too" from actually understanding why it fell out of favor for deep learning. Practice: Adagrad Limitations.

Learning Rate Warmup

"Why do so many training recipes ramp the learning rate up before decaying it, instead of just starting at the target learning rate?"

Learning rate warmup starts training with a small learning rate and increases it, often linearly, over an initial number of steps or epochs before switching to the main schedule. Early in training, weights are randomly initialized and the loss surface can be poorly conditioned, so a large learning rate applied immediately risks unstable, divergent updates. This is especially relevant with adaptive optimizers like Adam, where the second-moment estimate is based on very few observed gradients at the start and is therefore noisy and unreliable, which can produce oversized effective steps if the base learning rate is already high.

Warmup is also standard practice with large batch sizes, where each gradient estimate is less noisy and a large learning rate applied too early can push parameters into a bad region before the optimizer has "seen" enough of the loss landscape to correct course. It's become a default in transformer training recipes for exactly this reason.

The interview-ready framing: warmup isn't about the final learning rate value, it's about giving the optimizer's internal statistics and the model's weights a stable starting window before applying the full step size. Practice: Learning Rate Warmup.

Cosine Annealing

"How does cosine annealing differ from step decay, and why might you prefer it?"

Cosine annealing decays the learning rate following one half-cycle of a cosine curve, starting near the initial learning rate and smoothly decreasing to a minimum (often near zero) by the end of training, rather than dropping in discrete steps at fixed epochs the way step decay does. Because the cosine curve decreases slowly at first, more quickly through the middle of training, and then slowly again near the end, the learning rate spends more time near its minimum value late in training, which allows finer, more stable updates as the model approaches convergence.

A common extension is cosine annealing with warm restarts, where the learning rate periodically jumps back up to a high value and anneals down again in repeated cycles. The idea is that a sudden jump back to a higher learning rate can help the optimizer escape a sharp local minimum and settle into a flatter one, which tends to generalize better.

Compared to step decay's abrupt drops, cosine annealing avoids sudden discontinuities in the optimization trajectory, and it's frequently paired with warmup at the start of training — warm up to the target learning rate, then cosine-anneal back down. Practice: Cosine Annealing Schedule.

Xavier and He Initialization: Matching Variance to the Activation Function

"How should you initialize the weights of a deep network, and does the answer depend on the activation function?"

Weight initialization matters because if activations shrink or grow layer by layer, a deep network can suffer vanishing or exploding signals before training even gets going. Xavier (Glorot) initialization sets the variance of each weight based on both the number of input units (fan-in) and output units (fan-out) of a layer, designed to keep the variance of activations roughly constant as they pass forward through the network, and the variance of gradients roughly constant as they pass backward. It was derived assuming a symmetric, zero-centered activation function like tanh or sigmoid, where that balance holds.

He initialization was designed specifically for ReLU and its variants. Because ReLU zeroes out roughly half of its inputs (anything negative), a network using Xavier initialization with ReLU activations would see the variance of its activations roughly halve at every layer, which compounds into vanishing activations in deep networks. He initialization compensates by scaling the weight variance by 2 / fan_in instead of Xavier's smaller factor, correcting specifically for that lost half of the signal.

The one-line answer interviewers want: match the initialization scheme to the activation function you're using — Xavier for tanh or sigmoid, He for ReLU family activations — because each scheme's derivation assumes a specific activation's behavior. Practice: Xavier Weight Initialization and He Initialization for ReLU.

How to Prepare

1. Be able to write Adam's update rule from memory, including both moving averages and the bias correction terms — this single question filters a surprising number of candidates.
2. State Adagrad's limitation as "the accumulator never decays," not just "the learning rate gets too small," since the why is what interviewers are testing.
3. Know which initialization scheme pairs with which activation function, and be ready to explain the ReLU-specific reasoning behind He initialization.

For the rest of the concepts this track covers, see our neural network interview questions, which links practice problems across architecture, optimization, and regularization topics.