Gradient Boosting
Gradient boosting iteratively trains weak learners to fit the negative gradient of the loss at the previous round, progressively reducing loss; it is a powerful ensemble method.
Purpose
To provide one of the strongest prediction methods on structured data (XGBoost/LightGBM).
Mechanism
Initialize F_0 to a constant. Round m: compute the negative gradients r_i=-∂L/∂F(x_i), train a weak model h_m to fit r, and update F_m=F_{m-1}+ηh_m. XGBoost adds regularization and second-order information.
Use cases
- Structured data prediction
- Search ranking
- Click-through rate prediction