Shannon Entropy
Shannon entropy H(X)=-∑p(x)log p(x) measures the uncertainty of a random variable and is the core quantity of information theory.
Purpose
To quantify information content and uncertainty, guiding encoding and decision-tree splits.
Mechanism
Entropy is the expectation of self-information -log p(x). The uniform distribution maximizes entropy (most uncertain); a degenerate distribution has entropy 0. Decision trees choose splits with maximum information gain (entropy reduction). Channel capacity = max H(X)-H(X|Y).
Use cases
- Information encoding
- Decision-tree splitting
- Cryptographic evaluation
Related terms
Mutual Information
Mutual information I(X;Y)=H(X)-H(X|Y) measures the dependence between two random variables; it is non-negative and symmetric. GLOSSARY
Arithmetic Coding
Arithmetic coding maps an entire message to a single real number in the interval [0,1), represented with finite precision, so the compression ratio can approach the source entropy arbitrarily closely. GLOSSARY