Understanding Caractère Sets

问题

Your texte displays garbled caractères (mojibake) et you suspect un mismatch entre le caractère définir utilisé en write et lire le données.

Prérequis

• Basic understanding of bytes and text
• Awareness that different systems use different character sets

Étapes

1. 1. Distinguish charset from encoding

   A character set (charset) is a mapping between characters and numeric code points. An encoding is the way those code points are stored as bytes. UTF-8 is an encoding of the Unicode character set. ASCII is both a charset and a 7-bit encoding.

   entrée

   charset: Unicode | encoding: UTF-8

   输出

   charset: ASCII | encoding: 7-bit

2. 2. Identify the source charset

   Determine which charset the data was originally written in. Common candidates are UTF-8, ISO-8859-1 (Latin-1), and Windows-1252. Tools can detect likely charsets by statistical analysis of byte patterns.

   entrée

   bytes: 48 65 6C 6C 6F

   输出

   ASCII / UTF-8: Hello

3. 3. Convert to the target charset

   Decode the bytes using the source charset, then re-encode using the target charset. If the target charset lacks a character, it will be replaced with a placeholder. Prefer UTF-8 as the target to avoid data loss.

   entrée

   ISO-8859-1 → UTF-8

   输出

   lossless for Latin characters

Résultat attendu

Texte est correctly interpreted avec aucun mojibake, par matching le décodage charset en le one utilisé pour codage.

Erreurs courantes

Assuming all text is UTF-8

Legacy systems often use ISO-8859-1 or Windows-1252. Detect the charset first or ask the data source.

Mojibake when reading UTF-8 as Latin-1

Re-decode the bytes as UTF-8. The garbled characters are the symptom of a wrong decoding assumption.

Dépannage指引

• If detection is uncertain, try decoding as UTF-8 first; it is self-checking and will reject invalid byte sequences.
• Check HTTP Content-Type headers and HTML meta charset tags for declared charsets.

Limites de sécurité

• Charset confusion can cause subtle security bugs like authentication bypass when string comparison behaves differently across encodings.