Entropy and Information Theory Reference
Reference for Shannon entropy, min-entropy, entropy estimation, and security implications.
Entropy Definitions
Shannon entropy H(X) = -sum p(x) log p(x) measures the average information content of a random variable. Min-entropy H_inf = -log(max p(x)) is the relevant measure for cryptographic security as it bounds guess probability. The entropy of a uniform n-bit string is n bits; biased distributions have less. Entropy pools accumulate randomness from multiple sources before extraction. The gap between Shannon and min-entropy matters for security proofs.
- Shannon
- Average information
- min-entropy
- Worst-case guess prob
- uniform
- Maximum entropy
H = -sum p log pH_inf = -log max pn bits for n-bitEstimation and Extraction
Entropy estimation is difficult; no statistical test can prove true randomness. The NIST SP 800-90B tests provide standardized entropy assessment for physical sources. Randomness extractors (von Neumann, Toeplitz) concentrate weakly random input into near-uniform output. A universal hash function can serve as a randomness extractor with appropriate parameters. Conservative entropy estimates are essential for CSPRNG security.
- estimation
- Hard, no proof possible
- extractor
- Concentrate weak randomness
- 90B tests
- NIST entropy assessment
statistical testsuniversal hashphysical sources