AdaGrad Limitations

Hard

neural-networks Google DeepMind NVIDIA

Multiple Choice

The accumulated squared gradients grow monotonically, causing the effective learning rate to shrink to near zero; RMSProp fixes this by using an exponential moving average that forgets old gradients AdaGrad requires storing the full gradient history, causing memory overflow; RMSProp compresses the history into a fixed-size buffer AdaGrad only works with sparse gradients; RMSProp generalizes it to dense gradient updates AdaGrad's denominator can become zero for inactive parameters; RMSProp adds a large epsilon to prevent division by zero

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations