- Normalization moved inside the network: standardize each feature over the minibatch, then let the network undo it if it wants
- : mean and variance of that feature over the batch; divide by the standard deviation, not the variance
- are learned parameters, so normalizing never removes what the layer can represent
- for numerical stability
- why it helps
- trains faster, tolerates higher learning rates and worse initialization
- original paper ( Ioffe & Szegedy 2015 ) said it fixes internal covariate shift; later work ( Santurkar et al. 2018 ) argues the real effect is a smoother loss landscape
- batch noise acts as a mild regularizer
- training vs inference
- training uses the batch statistics; inference uses running averages collected during training (
model.eval()in PyTorch ) - small or non iid batches give bad statistics, the main failure mode
- training uses the batch statistics; inference uses running averages collected during training (
- transformers use Layer Norm / RMSNorm instead, which normalize over features within one example, so no batch dependence
- usually goes between the linear layer and the activation:
Python
Output
