Understanding Character Sets
Learn 差集之间字符 sets, encodings, 和代码 pages, 和为什么 choosing 右 one matters.
问题
你的文本 displays garbled characters (mojibake) 和你 suspect mismatch 之间字符集合 used 到写入和 读取数据.
前置条件
- 基础 understanding 的 bytes 和文本
- Awareness 该不同 systems use 不同字符 sets
步骤
-
1. Distinguish charset from encoding
A character set (charset) is a mapping between characters and numeric code points. An encoding is the way those code points are stored as bytes. UTF-8 is an encoding of the Unicode character set. ASCII is both a charset and a 7-bit encoding.
输入charset: Unicode | encoding: UTF-8输出charset: ASCII | encoding: 7-bit -
2. Identify the source charset
Determine which charset the data was originally written in. Common candidates are UTF-8, ISO-8859-1 (Latin-1), and Windows-1252. Tools can detect likely charsets by statistical analysis of byte patterns.
输入bytes: 48 65 6C 6C 6F输出ASCII / UTF-8: Hello -
3. Convert to the target charset
Decode the bytes using the source charset, then re-encode using the target charset. If the target charset lacks a character, it will be replaced with a placeholder. Prefer UTF-8 as the target to avoid data loss.
输入ISO-8859-1 → UTF-8输出lossless for Latin characters
预期结果
Text 是正确 interpreted 具有否 mojibake, 通过 matching 解码 charset 到 one used 为编码.
常见错误
- Assuming all text is UTF-8
- Legacy systems often use ISO-8859-1 or Windows-1252. Detect the charset first or ask the data source.
- Mojibake when reading UTF-8 as Latin-1
- Re-decode the bytes as UTF-8. The garbled characters are the symptom of a wrong decoding assumption.
排障指引
- If detection is uncertain, try decoding as UTF-8 first; it is self-checking and will reject invalid byte sequences.
- Check HTTP Content-Type headers and HTML meta charset tags for declared charsets.
安全限制
- Charset confusion can cause subtle security bugs like authentication bypass when string comparison behaves differently across encodings.