KL Divergence
KL divergence D_KL(P||Q)=∑p(x)log(p(x)/q(x)) measures the difference between distributions P and Q; it is asymmetric and non-negative.
Purpose
To measure differences between probability distributions, used in variational inference and model selection.
Mechanism
D_KL≥0 with equality if and only if P=Q. It is asymmetric, hence not a metric. Cross-entropy H(P,Q)=H(P)+D_KL(P||Q). Minimizing cross-entropy in machine learning is equivalent to minimizing KL (since H(P) is fixed).
Use cases
- Variational inference
- Model evaluation
- Information theory