Screenshot 2024-09-17 at 10.22.04 AM.png

  • Normalization moved inside the network: standardize each feature over the minibatch, then let the network undo it if it wants
    • : mean and variance of that feature over the batch; divide by the standard deviation, not the variance
    • are learned parameters, so normalizing never removes what the layer can represent
    • for numerical stability
  • why it helps
    • trains faster, tolerates higher learning rates and worse initialization
    • original paper ( Ioffe & Szegedy 2015 ) said it fixes internal covariate shift; later work ( Santurkar et al. 2018 ) argues the real effect is a smoother loss landscape
    • batch noise acts as a mild regularizer
  • training vs inference
    • training uses the batch statistics; inference uses running averages collected during training ( model.eval() in PyTorch )
    • small or non iid batches give bad statistics, the main failure mode
  • transformers use Layer Norm / RMSNorm instead, which normalize over features within one example, so no batch dependence
  • usually goes between the linear layer and the activation:
Python