Understanding Character Sets

Problem

Your text displays garbled characters (mojibake) and you suspect a mismatch between the character set used to write and read the data.

Prerequisites

• Basic understanding of bytes and text
• Awareness that different systems use different character sets

Steps

1. 1. Distinguish charset from encoding

   A character set (charset) is a mapping between characters and numeric code points. An encoding is the way those code points are stored as bytes. UTF-8 is an encoding of the Unicode character set. ASCII is both a charset and a 7-bit encoding.

   Input

   charset: Unicode | encoding: UTF-8

   Output

   charset: ASCII | encoding: 7-bit

2. 2. Identify the source charset

   Determine which charset the data was originally written in. Common candidates are UTF-8, ISO-8859-1 (Latin-1), and Windows-1252. Tools can detect likely charsets by statistical analysis of byte patterns.

   Input

   bytes: 48 65 6C 6C 6F

   Output

   ASCII / UTF-8: Hello

3. 3. Convert to the target charset

   Decode the bytes using the source charset, then re-encode using the target charset. If the target charset lacks a character, it will be replaced with a placeholder. Prefer UTF-8 as the target to avoid data loss.

   Input

   ISO-8859-1 → UTF-8

   Output

   lossless for Latin characters

Expected result

Text is correctly interpreted with no mojibake, by matching the decoding charset to the one used for encoding.

Common errors

Assuming all text is UTF-8

Legacy systems often use ISO-8859-1 or Windows-1252. Detect the charset first or ask the data source.

Mojibake when reading UTF-8 as Latin-1

Re-decode the bytes as UTF-8. The garbled characters are the symptom of a wrong decoding assumption.

Troubleshooting

• If detection is uncertain, try decoding as UTF-8 first; it is self-checking and will reject invalid byte sequences.
• Check HTTP Content-Type headers and HTML meta charset tags for declared charsets.

Security limits

• Charset confusion can cause subtle security bugs like authentication bypass when string comparison behaves differently across encodings.