
- gradient descent is the base optimizer under basically all of deep learning: nudge the parameters θ a little downhill on the loss J(θ), repeat
- intuition: walking down hilly terrain in fog, you only feel the slope under your feet so you step the steepest way down
- the gradient ∇θJ points uphill, so we subtract it
- introduced in Evolutionary Robotics
θ←θ−α∇θJ(θ)
- example: linear regression with a MSE cost over m examples, hθ(x)=θ⊤x
J(θ)=2m1i=1∑m(hθ(x(i))−y(i))2,∂θj∂J=m1i=1∑m(hθ(x(i))−y(i))xj(i)
- the $\frac{1}{2}$ is only there to cancel the 2 from differentiating the square
- learning rate α is the main knob
- too small: converges painfully slowly
- too large: overshoots the minimum, oscillates or diverges
- normalizing features makes the loss surface less of a long thin valley, so one α works in every direction
- convergence: stop when ∣J(θnew)−J(θold)∣<ϵ ( e.g. 10−6 )
- only guarantees a local minimum or saddle unless J is convex
- how much data per step
- batch: the whole dataset per gradient, exact but expensive
- stochastic (SGD): one example per step, noisy but cheap, and the noise can help escape saddles and sharp minima
- mini-batch: small random subsets, what everyone actually means by SGD
- the gradient itself comes from backpropagation ( Automatic Differentiation in practice )
- fancier optimizers are smarter step rules on top: momentum, RMSprop, ADAM ( per parameter adaptive step sizes )
- Double Descent is the weird overparameterized regime where test error comes back down