- per parameter Gradient Descent optimization algorithm
- takes the first and second moments for each parameter and tunes the learning rate as such.
- First moment - momentum: trackss exponential moving average for gradients, then smoothing updates + maintaining direction.
- Second moment squared - gradients: tracks exponential moving average for squared gradients, then scaling down updates for params with consistently large gradients.
- hyperparameters
- learning rate η,
0.001 default, overall magninute for param updates
- Beta 1 β1
0.9 momentum parameter control
- Beta 2 β2
0.999 squared gradient control
- Epsilon ϵ
10^-8 for preventing div by zero and numerical stability