Mutual Information
Mutual information I(X;Y)=H(X)-H(X|Y) measures the dependence between two random variables; it is non-negative and symmetric.
Purpose
To quantify the information shared between variables, used in feature selection and clustering evaluation.
Mechanism
I(X;Y)=D_KL(P(X,Y)||P(X)P(Y)), measuring the divergence between the joint distribution and the independence distribution. I=0 if and only if X and Y are independent. Maximizing mutual information is used in the information bottleneck and feature selection.
Use cases
- Feature selection
- Information bottleneck
- Clustering evaluation