Bias-Variance Tradeoff
Bias-variance tradeoff: model error = bias² + variance + irreducible error; increasing complexity reduces bias but raises variance.
Purpose
To understand how model complexity relates to generalization error and guide model selection.
Mechanism
Bias is the systematic gap between predictions and truth (underfitting); variance is the fluctuation of predictions across training sets (overfitting). Simple models have high bias and low variance; complex models the opposite. Regularization finds the balance point.
Use cases
- Model selection
- Regularization tuning
- Overfitting analysis