Gradient Descent
Gradient descent iterates along the negative gradient x_{k+1}=x_k - η∇f(x_k) to find a function's minimum; it is the most fundamental optimization algorithm.
Purpose
To provide the basic algorithm for unconstrained optimization and the core of machine learning training.
Mechanism
The negative gradient is the direction of steepest descent. A learning rate η that is too large oscillates and diverges; one too small converges slowly. Variants: momentum, Nesterov acceleration, adaptive learning rates (AdaGrad, Adam). Convergence is O(1/T) for convex functions and O(e^(-T)) for strongly convex ones.
Use cases
- Machine learning training
- Unconstrained optimization
- Deep learning