How to Detect Text Encoding

Problem

You received a file without a charset declaration and need to determine whether it is UTF-8, Latin-1, or another encoding before processing it.

Prerequisites

• Understanding of character sets
• Familiarity with byte sequences

Steps

1. 1. Check for a BOM

   Read the first few bytes. A UTF-8 BOM is EF BB BF, UTF-16 LE is FF FE, and UTF-16 BE is FE FF. If a BOM is present, the encoding is unambiguous. Many tools strip the BOM after detection.

   Input

   EF BB BF 23 20

   Output

   UTF-8 with BOM

2. 2. Validate against UTF-8

   UTF-8 has strict byte patterns: continuation bytes start with 10xxxxxx, and lead bytes follow specific ranges. If the stream passes UTF-8 validation without errors, it is very likely UTF-8. If it fails, try Windows-1252 or ISO-8859-1.

   Input

   48 65 6C 6C 6F

   Output

   valid UTF-8 (ASCII subset)

3. 3. Use statistical detection as fallback

   When no BOM exists and UTF-8 validation fails, use a statistical detector that analyses byte frequency and n-gram patterns to guess the most likely encoding. Always treat the result as a guess and let the user confirm.

   Input

   mixed bytes

   Output

   probable: Windows-1252 (confidence 87%)

Expected result

A best-guess encoding label with a confidence level, or a definitive result if a BOM or strict validation succeeds.

Common errors

Trusting statistical detection blindly

Statistical detection can be wrong. Always provide a way for the user to override the detected encoding.

Ignoring the BOM

Always check for a BOM first. It is the only definitive indicator and should be stripped after detection.

Troubleshooting

• If UTF-8 validation fails on a file that should be UTF-8, check for mixed encodings or corruption.
• Windows-1252 is a superset of ISO-8859-1 and is the more likely fallback for Western text.

Security limits

• Encoding detection can be fooled by crafted byte sequences. Do not make security decisions based solely on detected encoding.