- backpropagation: get ∂L/∂w for every weight in one backward sweep, then hand it to Gradient Descent ( or ADAM )
- it’s just the chain rule applied layer by layer, reusing each layer’s result for the one before it
- reverse mode autodiff: costs about as much as one forward pass, no matter how many parameters
- forward pass for layer l: zl=Wlal−1+bl, al=f(zl) ( see Artificial Neuron )
- backward pass: define the error δl=∂L/∂zl
δL=∇aLL⊙f′(zL),δl=((Wl+1)⊤δl+1)⊙f′(zl)
∂Wl∂L=δl(al−1)⊤,∂bl∂L=δl
- the error flows back through the transposed weights; each layer multiplies in its activation derivative
- that product of derivatives is why sigmoid nets vanish and why ReLU and Batch Normalization help
- example: one sigmoid neuron, MSE L=21(a−y)2, a=σ(wx+b)
- ∂L/∂w=(a−y)σ′(z)x: error, times local slope, times input
- for recurrent nets you unroll in time: Backpropagation Through Time
- brains probably can’t do this literally: it needs symmetric feedback weights ( weight transport ) and a separate backward phase
- Welch Labs has the best visual walkthrough: The F=ma of Artificial Intelligence Backpropagation