He Initialization for ReLU

Medium

neural-networks DeepMind Google NVIDIA

He initialization uses variance 2/n_in instead of Xavier's 2/(n_in + n_out). Why is this modification necessary for networks using ReLU activations?

Multiple Choice

ReLU zeroes out roughly half the activations, halving the variance, so the factor of 2 compensates for this reduction ReLU outputs are unbounded, so larger initial weights are needed to prevent underflow He initialization accounts for the bias terms that Xavier initialization ignores ReLU networks always have more layers, requiring proportionally larger initial weights

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations

He Initialization for ReLU

Multiple Choice

Related problems

Deep-dive guides