How to Handle the Byte Order Mark (BOM)

Problem

Your text file starts with invisible BOM bytes that cause parsing errors or appear as strange characters in output.

Prerequisites

• Understanding of character encodings
• Familiarity with hex editors or byte inspection

Steps

1. 1. Identify the BOM bytes

   Inspect the first 2-3 bytes of the file. EF BB BF indicates a UTF-8 BOM, FF FE indicates UTF-16 LE, and FE FF indicates UTF-16 BE. The BOM tells the reader the encoding and byte order.

   Input

   EF BB BF 3C 3F 78 6D 6C

   Output

   UTF-8 BOM + '<?xml'

2. 2. Decide whether to keep or strip it

   Keep the BOM when writing UTF-8 files for Windows applications that rely on it for encoding detection. Strip it for web files (HTML, CSS, JS, JSON) where the BOM can cause parsing errors or appear as a visible character.

   Input

   JSON file with BOM

   Output

   strip BOM before parsing

3. 3. Strip the BOM programmatically

   When reading a file, check for and remove the BOM before processing. Most modern parsers handle this automatically, but some (like JSON.parse in older environments) do not and will throw an error on a BOM.

   Input

   \uFEFF{}

   Output

   strip → {}

Expected result

Text files with BOMs handled correctly for their target context: kept for Windows compatibility, stripped for web and API use.

Common errors

JSON.parse fails with 'Unexpected token'

The file has a UTF-8 BOM. Strip the first 3 bytes (EF BB BF) or the U+FEFF character before parsing.

BOM appears as  in HTML output

The BOM was not stripped and is rendered as a zero-width no-break space. Remove it from the source file.

Troubleshooting

• If a file works locally but fails on a server, check for a BOM added by a Windows editor.
• Use a hex editor to confirm whether a BOM is present when debugging encoding issues.

Security limits

• A BOM is metadata, not a security feature. It can interfere with signature verification if not accounted for.