Consider a network with $L$ layers. The output variance after $L$ layers depends on the variance of each layer's weights. If the variance shrinks at each layer, activations vanish. If it grows, activations explode. Initialization must preserve the variance of activations and gradients across layers.

Initialize weights from a standard normal distribution:

Problem: For a layer with 1000 inputs, the output variance is 1000 times the input variance. Activations explode after a few layers, causing numerical overflow and gradient explosion.

Xavier initialization preserves variance for linear and sigmoid/tanh activations. It sets weights from:

or uniform:

where $n_{\text{in}}$ is the number of inputs and $n_{\text{out}}$ is the number of outputs.

He initialization accounts for ReLU, which zeros out approximately half the activations:

The factor of 2 compensates for ReLU zeroing out half the variance. For Leaky ReLU with slope $\alpha$:

For RNNs, orthogonal initialization sets the recurrent weight matrix to an orthogonal matrix:

This preserves gradient magnitude through time because $\|\mathbf{W}_{hh}^t\| = 1$ for all $t$.

LSUV is a data-driven initialization method:

1. Initialize all weights to orthogonal matrices
2. For each layer (in order):
   • Forward pass a batch of data
   • Compute output variance
   • Scale weights: $\mathbf{W} \leftarrow \mathbf{W} / \sqrt{\text{Var}(\text{output})}$
3. Repeat until variance is 1.0

LSUV works for any architecture and doesn't require knowing activation functions.

ResNet uses special initialization for residual blocks:

1. Initialize all residual block weights with He initialization
2. Set the last batch normalization in each residual block to $\gamma = 0$

This ensures that at initialization, each residual block computes zero, making the network equivalent to a shallow network. As training progresses, $\gamma$ learns to be non-zero.