Adam Optimizer
Adam combines momentum (first moment) with RMSProp (second moment) for adaptive learning rates and is the most widely used optimizer in deep learning.
Purpose
To serve as the default optimizer for deep learning training, adaptively adjusting the learning rate of each parameter.
Mechanism
It maintains the first moment m (momentum) and the second moment v (RMSProp) of the gradient, then updates after bias correction: θ ← θ - η * m_hat/(√v_hat + ε). Defaults are β1=0.9, β2=0.999. It combines momentum acceleration with adaptive learning rates.
Use cases
- Deep learning training
- Neural network optimization
- Adaptive learning rates