Détecter the language of text
Language detection identifies le language de un texte sample utilisant caractère n-grams, arrêter mots, et statistical models. Apprendre commun approaches.
- 1
Language detection methods: (1) Stop word analysis — each language has characteristic stop words (the, and, of for English; le, et, de for French). (2) Character n-grams — each language has characteristic letter sequences (sch for German, ll for Spanish). (3) Statistical models (Naive Bayes, neural networks) trained on labeled text. Libraries: franc, cld3, langdetect.
- 2 Compteur de mots →
Compter words and compare against stop word lists for different languages. If the text contains many English stop words (the, is, at), it is likely English. This is a simple but effective heuristic for common languages.
- 3 Fréquence de caractères →
Analyze character frequency distribution. English has high frequency of e, t, a, o, i, n. German has sch, ä, ö, ü. French has ç, accents. CJK languages have no word boundaries. Character distribution is a strong language signal, especially for short texts.
- 4 Compteur de phrases du texte →
Compter sentences to understand text structure. Some languages use different sentence-ending punctuation (。for CJK, ¿? for Spanish questions). Sentence structure and punctuation patterns are additional language signals.
Questions fréquentes
- How accurate is language detection?
- For long texts (>100 characters) in common languages, 95-99% accuracy. For short texts or similar languages (Spanish vs Portuguese, Norwegian vs Danish), accuracy drops. Mixed-language texts are problematic. Use a library (franc, cld3) for best results — they handle edge cases that simple heuristics miss.
- How do I detect language for CJK text?
- CJK languages (Chinese, Japanese, Korean) share some characters but have distinguishing features: Japanese has hiragana (ひらがな) and katakana (カタカナ), Korean has Hangul (한글), Chinese uses only Hanzi (汉字). Detect by the presence of these unique character sets. Use Unicode block analysis.