Exploding Gradients and Gradient Clipping

Hard

neural-networks Google DeepMind Meta

Exploding gradients are common in deep recurrent networks. What causes this problem, and how does gradient clipping address it without fundamentally changing the optimization?

Multiple Choice

Repeated multiplication by large weight matrices causes exponential gradient growth; clipping rescales the gradient when its norm exceeds a threshold, preserving direction but bounding magnitude Large input values overflow the activation functions; clipping truncates inputs before they enter each layer The loss function has discontinuities that produce infinite gradients; clipping smooths the loss landscape Batch normalization layers amplify gradients at each step; clipping disables normalization when gradients are too large

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations

Exploding Gradients and Gradient Clipping

Multiple Choice

Related problems

Deep-dive guides