gradDescent.png

  • gradient descent is the base optimizer under basically all of deep learning: nudge the parameters a little downhill on the loss , repeat
    • intuition: walking down hilly terrain in fog, you only feel the slope under your feet so you step the steepest way down
    • the gradient points uphill, so we subtract it
    • introduced in Evolutionary Robotics
  • example: linear regression with a MSE cost over examples,
- the $\frac{1}{2}$ is only there to cancel the 2 from differentiating the square
  • learning rate is the main knob
    • too small: converges painfully slowly
    • too large: overshoots the minimum, oscillates or diverges
    • normalizing features makes the loss surface less of a long thin valley, so one works in every direction
  • convergence: stop when ( e.g. )
    • only guarantees a local minimum or saddle unless is convex
  • how much data per step
    • batch: the whole dataset per gradient, exact but expensive
    • stochastic (SGD): one example per step, noisy but cheap, and the noise can help escape saddles and sharp minima
    • mini-batch: small random subsets, what everyone actually means by SGD
  • the gradient itself comes from backpropagation ( Automatic Differentiation in practice )
  • fancier optimizers are smarter step rules on top: momentum, RMSprop, ADAM ( per parameter adaptive step sizes )
  • Double Descent is the weird overparameterized regime where test error comes back down