Stochastic Gradient Descent
SGD approximates the full gradient at each step with a single-sample or mini-batch gradient, cutting per-step cost and adding randomness that helps escape local optima.
Purpose
To provide the core optimization algorithm for large-scale machine learning.
Mechanism
The full gradient is ∇f = (1/N)∑∇f_i; SGD approximates it with ∇f_i. The variance introduces noise, but the expectation equals the true gradient. Mini-batches balance variance and efficiency. Learning-rate decay guarantees convergence.
Use cases
- Deep learning training
- Large-scale optimization
- Online learning