バイトオーダーマーク(BOM)を扱う方法

问题

テキストファイルが不可視の BOM バイトで始まっており、パースエラーの原因になったり、出力に奇妙な文字として現れたりします。

前提条件

• 文字エンコーディングの理解
• hex エディタやバイト確認への習熟

ステップ

1. 1. Identify the BOM bytes

   Inspect the first 2-3 bytes of the file. EF BB BF indicates a UTF-8 BOM, FF FE indicates UTF-16 LE, and FE FF indicates UTF-16 BE. The BOM tells the reader the encoding and byte order.

   入力

   EF BB BF 3C 3F 78 6D 6C

   输出

   UTF-8 BOM + '<?xml'

2. 2. Decide whether to keep or strip it

   Keep the BOM when writing UTF-8 files for Windows applications that rely on it for encoding detection. Strip it for web files (HTML, CSS, JS, JSON) where the BOM can cause parsing errors or appear as a visible character.

   入力

   JSON file with BOM

   输出

   strip BOM before parsing

3. 3. Strip the BOM programmatically

   When reading a file, check for and remove the BOM before processing. Most modern parsers handle this automatically, but some (like JSON.parse in older environments) do not and will throw an error on a BOM.

   入力

   \uFEFF{}

   输出

   strip → {}

期待される結果

BOM が対象コンテキストに応じて正しく処理されたテキストファイル。Windows 互換性のために保持し、Web や API 用には除去します。

よくあるエラー

JSON.parse fails with 'Unexpected token'

The file has a UTF-8 BOM. Strip the first 3 bytes (EF BB BF) or the U+FEFF character before parsing.

BOM appears as  in HTML output

The BOM was not stripped and is rendered as a zero-width no-break space. Remove it from the source file.

トラブルシューティング指引

• If a file works locally but fails on a server, check for a BOM added by a Windows editor.
• Use a hex editor to confirm whether a BOM is present when debugging encoding issues.

セキュリティ上の制限

• A BOM is metadata, not a security feature. It can interfere with signature verification if not accounted for.