AdaGrad Limitations

Hard

neural-networks Google DeepMind NVIDIA

AdaGrad adapts the learning rate for each parameter by dividing by the square root of the sum of all past squared gradients. Why can this become problematic in deep learning, and how does RMSProp fix it?

Multiple Choice

The accumulated squared gradients grow monotonically, causing the effective learning rate to shrink to near zero; RMSProp fixes this by using an exponential moving average that forgets old gradients AdaGrad requires storing the full gradient history, causing memory overflow; RMSProp compresses the history into a fixed-size buffer AdaGrad only works with sparse gradients; RMSProp generalizes it to dense gradient updates AdaGrad's denominator can become zero for inactive parameters; RMSProp adds a large epsilon to prevent division by zero

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations

AdaGrad Limitations

Multiple Choice

Related problems

Deep-dive guides