GELU and Leaky ReLU Activations

Hard

neural-networks OpenAI Google Anthropic

Both Leaky ReLU and GELU were designed to address a limitation of standard ReLU. What is this limitation, and how do these two activations address it differently?

Multiple Choice

Both address the dying ReLU problem; Leaky ReLU uses a small constant slope for negative inputs, while GELU applies a smooth, input-dependent gating based on the Gaussian CDF Both address ReLU's lack of an upper bound; Leaky ReLU caps outputs at 6, while GELU uses a soft ceiling via the error function Both address ReLU's non-differentiability at zero; Leaky ReLU uses a piecewise linear approximation, while GELU uses polynomial smoothing Both address ReLU's computational cost; Leaky ReLU uses lookup tables, while GELU pre-computes activations during initialization

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations

GELU and Leaky ReLU Activations

Multiple Choice

Related problems

Deep-dive guides