Home

❯

Rhizomatic

❯

Artificial Intelligence

❯

Backpropagation

Backpropagation

Computing the gradient of the loss with respect to every weight by running the chain rule backward through the network.

12 Sept 2024
Written
12 Sept 2024
Revised
20 Nov 2026
Length
241 words, 2 min
8 Aug 2024today
569

PageRank 569th of 889, 0.3× the mean score

rarely reachedoften reached
  • cs/ai/training/backpropagation
11

Nearby

  1. Neural Networks
  2. Loss Functions
  3. Parameter
  4. Normalization
  5. Activation Functions
  6. Linear and Nonlinear Regression

Similar, not linked

  1. Neural Networks in Evolutionary Robotics83%
  2. Diffusion in Machine Learning81%
  3. Self Attention78%
  4. Normalization, Regularization, and Gradient Descent for Linear Regression78%
  5. Reinforcement Learning77%
  • backpropagation: get ∂L/∂w for every weight in one backward sweep, then hand it to Gradient Descent ( or ADAM )
    • it’s just the chain rule applied layer by layer, reusing each layer’s result for the one before it
    • reverse mode autodiff: costs about as much as one forward pass, no matter how many parameters
  • forward pass for layer l: zl=Wlal−1+bl, al=f(zl) ( see Artificial Neuron )
  • backward pass: define the error δl=∂L/∂zl δL=∇aL​L⊙f′(zL),δl=((Wl+1)⊤δl+1)⊙f′(zl) ∂Wl∂L​=δl(al−1)⊤,∂bl∂L​=δl
    • the error flows back through the transposed weights; each layer multiplies in its activation derivative
    • that product of derivatives is why sigmoid nets vanish and why ReLU and Batch Normalization help
  • example: one sigmoid neuron, MSE L=21​(a−y)2, a=σ(wx+b)
    • ∂L/∂w=(a−y)σ′(z)x: error, times local slope, times input
  • for recurrent nets you unroll in time: Backpropagation Through Time
  • brains probably can’t do this literally: it needs symmetric feedback weights ( weight transport ) and a separate backward phase
    • predictive coding networks approximate it with local Hebbian updates ( Whittington & Bogacz )
    • more in Learning Algorithm Of Biological Networks and Backpropagation through space, time and the brain
  • Welch Labs has the best visual walkthrough: The F=ma of Artificial Intelligence Backpropagation