Encoding Detection Algorithms
Heuristic and statistical methods for detecting text file encodings without explicit declaration.
BOM-based Detection
The Byte Order Mark is a unique prefix that unambiguously identifies UTF-16 and UTF-32 variants. A leading 0xEF 0xBB 0xBF signals UTF-8, though the mark is optional and some producers omit it. BOM detection is fast and reliable but only works for a minority of files. Relying solely on the BOM misses UTF-8 files without it and all legacy single-byte encodings.
- UTF-8 BOM
- EF BB BF prefix
- UTF-16 LE BOM
- FF FE prefix
- UTF-16 BE BOM
- FE FF prefix
- no BOM
- Falls back to heuristic detection
optional in UTF-8Windows defaultJava defaultmost UTF-8 filesStatistical Heuristics
When no BOM is present, detectors analyze byte frequency distributions to guess the encoding. UTF-8 has a strict structural pattern where continuation bytes follow lead bytes with specific ranges, so validating this pattern is highly accurate. For legacy encodings, detectors compare character frequency histograms against language models. The confidence score is usually low for ambiguous inputs, so detectors return a ranked list rather than a single answer.
Common Pitfalls
Short strings rarely contain enough evidence for reliable detection, so results on snippets are unreliable. Some byte sequences are valid in multiple encodings, leading to mojibake when the wrong one is chosen. The ISO-8859-1 and Windows-1252 encodings overlap heavily, causing frequent misidentification. Always prefer an explicit encoding declaration from protocol metadata over heuristic detection when available.