エンコーディング検出エラーをトラブルシューティングする方法

问题

バイトストリームに対して encoding-detect を実行したところ、予想外の結果（誤ったエンコーディング、低い信頼度スコア、実際のテキストと矛盾する断定的な結果のいずれか）が返されました。どの検出層が失敗したのかを突き止め、修正する必要があります。

前提条件

• 文字エンコーディング（UTF-8、Latin-1、UTF-16）の基礎理解
• バイトの 16 進表記への習熟
• encoding-detect と hex-encode ツールの使用経験

ステップ

1. 1. Inspect the BOM layer with hex-encode

   The Byte Order Mark is the only definitive encoding indicator. Use hex-encode to dump the first 4 bytes of your input. A UTF-8 BOM is EF BB BF, UTF-16 LE is FF FE, UTF-16 BE is FE FF, and UTF-32 LE is FF FE 00 00. If a BOM exists but encoding-detect reported a different encoding, the tool may have a bug or the BOM was stripped before detection. If no BOM exists, the file relies on out-of-band metadata or statistical detection — proceed to the next layer. Common failure: a file declared UTF-8 actually starts with FF FE (UTF-16 LE), which means it was mislabeled during transfer.

   入力

   First 4 bytes: FF FE 41 00

   输出

   BOM detected: UTF-16 LE — overrides any declared charset

2. 2. Parse the charset header and HTML meta tag

   When no BOM is present, the encoding should be declared in transport metadata (HTTP Content-Type charset parameter) or an in-document HTML meta tag. Look for <meta charset="..."> (HTML5) or <meta http-equiv="Content-Type" content="text/html; charset=..."> (HTML4). A frequent error is a mismatch between the declared charset and the actual bytes — for example, a file labeled charset=iso-8859-1 that actually contains UTF-8 multibyte sequences (C3 A9 for é). Use charset-reference to look up the declared encoding's byte ranges and compare against the actual bytes. If the declaration is wrong, the fix is to correct the declaration or re-save the file in the declared encoding.

   入力

   Declared: charset=iso-8859-1 | Actual byte for é: C3 A9

   输出

   Conflict: C3 A9 is UTF-8, not ISO-8859-1 (where é = E9)

3. 3. Validate against UTF-8 strict rules

   UTF-8 has a strict byte structure: ASCII bytes are 00-7F, lead bytes of 2-byte sequences are C2-DF, 3-byte leads are E0-EF, and 4-byte leads are F0-F4. Continuation bytes are 80-BF. If the stream passes this validation cleanly, it is almost certainly UTF-8. If validation fails at a specific byte offset, that byte is either corrupted or belongs to a different encoding. Record the offset — it tells you exactly where the mixed-encoding boundary is. A file that is valid UTF-8 for the first 1000 bytes then fails is likely a concatenation of two files in different encodings.

   入力

   Stream: 48 65 6C 6C 6F E4 B8 AD (Hello + 中 in UTF-8)

   输出

   Valid UTF-8 — all byte sequences conform to the pattern

4. 4. Evaluate statistical detection confidence and override

   When BOM, header, and meta tag are all absent, statistical detection is the last resort. It analyses byte frequency distributions and n-gram models to guess the encoding. The result always includes a confidence score. If confidence is below 90%, treat the result as a hint, not an answer. Common false positives: Windows-1252 is often guessed for short Western text because it is a superset of ISO-8859-1 and covers the 0x80-0x9F range. To disambiguate, try decoding with the guessed encoding and check if the output contains reasonable characters for your expected language. If the output has replacement characters (U+FFFD) or mojibake, try the next candidate encoding. Always let the user confirm the final choice.

   入力

   Statistical result: Windows-1252 (confidence 72%)

   输出

   Low confidence — try ISO-8859-1 and UTF-8 as alternatives, let user confirm

期待される結果

検出エラーの根本原因の特定。どのレイヤー(BOM、ヘッダー、meta タグ、統計的検出)が誤った結果を出したのか、なぜ失敗したのか、そして hex ダンプの証拠に基づく修正済みエンコーディングがわかります。

よくあるエラー

Trusting a high-confidence statistical result on very short input

Statistical detection needs enough bytes to build a reliable frequency model. For inputs under 100 bytes, confidence scores are unreliable — always require manual confirmation.

Assuming no BOM means UTF-8

Absence of a BOM does not prove UTF-8. Many UTF-8 files omit the BOM by convention, but so do Latin-1 and Windows-1252 files. Run strict UTF-8 validation before concluding.

Ignoring a charset declaration that contradicts the bytes

The declaration may be stale from a previous save. Trust the actual byte structure over the label. Use hex-encode to inspect the bytes and charset-reference to verify the expected ranges.

Treating Windows-1252 and ISO-8859-1 as interchangeable

They differ in the 0x80-0x9F range. Windows-1252 defines printable characters there (smart quotes, em-dash) while ISO-8859-1 has control characters. Pick the one that produces sensible output for your text.

トラブルシューティング指引

• If encoding-detect returns 'unknown' on a valid text file, check whether the input was truncated or contains a null byte (00) that makes the tool treat it as binary.
• If the detected encoding produces mojibake but a different encoding reads correctly, the file was likely saved in one encoding and mislabeled as another during a copy or transfer operation.
• If a UTF-8 file is detected as UTF-16, check for a stray FF FE or FE FF sequence near the start that is not actually a BOM but coincidental data bytes.
• If statistical detection oscillates between two encodings on different runs, the input is too short or too ambiguous — gather more bytes or ask the source system for the declared encoding.

セキュリティ上の制限

• Encoding detection results must never be the sole basis for security decisions. Crafted byte sequences can fool statistical detectors.
• All inspection and detection runs locally — no byte stream is transmitted to any server.