文字集合の理解
文字集合、エンコーディング、コードページの違いと、正しく選ぶことが重要な理由を学びます。
问题
テキストが文字化け（mojibake）して表示され、書き込み時と読み取り時の文字集合の不一致が疑われる状況です。
前提条件
- バイトとテキストの基礎理解
- システムごとに異なる文字集合が使われていることへの認識
ステップ
-
1. Distinguish charset from encoding
A character set (charset) is a mapping between characters and numeric code points. An encoding is the way those code points are stored as bytes. UTF-8 is an encoding of the Unicode character set. ASCII is both a charset and a 7-bit encoding.
入力charset: Unicode | encoding: UTF-8输出charset: ASCII | encoding: 7-bit -
2. Identify the source charset
Determine which charset the data was originally written in. Common candidates are UTF-8, ISO-8859-1 (Latin-1), and Windows-1252. Tools can detect likely charsets by statistical analysis of byte patterns.
入力bytes: 48 65 6C 6C 6F输出ASCII / UTF-8: Hello -
3. Convert to the target charset
Decode the bytes using the source charset, then re-encode using the target charset. If the target charset lacks a character, it will be replaced with a placeholder. Prefer UTF-8 as the target to avoid data loss.
入力ISO-8859-1 → UTF-8输出lossless for Latin characters
期待される結果
デコード時の文字セットをエンコード時に使ったものと一致させることで、文字化けなく正しく解釈されたテキスト。
よくあるエラー
- Assuming all text is UTF-8
- Legacy systems often use ISO-8859-1 or Windows-1252. Detect the charset first or ask the data source.
- Mojibake when reading UTF-8 as Latin-1
- Re-decode the bytes as UTF-8. The garbled characters are the symptom of a wrong decoding assumption.
トラブルシューティング指引
- If detection is uncertain, try decoding as UTF-8 first; it is self-checking and will reject invalid byte sequences.
- Check HTTP Content-Type headers and HTML meta charset tags for declared charsets.
セキュリティ上の制限
- Charset confusion can cause subtle security bugs like authentication bypass when string comparison behaves differently across encodings.