Tokenize Text für NLP processing
Tokenization splits Text in tokens (Wörter, sentences, subwords). Lernen Wort, sentence, und subword tokenization für NLP Aufgaben.
- 1
Tokenization splits text into units for processing. Word tokenization splits on whitespace and punctuation. Sentence tokenization splits on sentence boundaries (. ! ?). Subword tokenization (BPE, WordPiece) splits rare words into common subwords — used by LLMs. The choice depends on the NLP task.
- 2 Wortzähler →
Zählen words and get basic word statistics. This gives a rough tokenization by splitting on whitespace. For more accurate tokenization, handle punctuation (don't split "don't" into "don" and "t"), contractions, and hyphenated words.
- 3 Regex-Tester →
Use regex for word tokenization: \b\w+\b matches word boundaries. For sentence tokenization: [^.!?]+[.!?] splits on sentence-ending punctuation. For more accurate tokenization, use an NLP library (NLTK, spaCy) that handles edge cases (abbreviations, ellipses, quotes).
Häufige Fragen
- Word vs subword tokenization?
- Word tokenization splits on whitespace — simple but produces a large vocabulary and cannot handle unknown words. Subword tokenization (BPE, WordPiece) splits words into common subword units (e.g., "playing" → "play" + "ing") — smaller vocabulary, handles unknown words. LLMs use subword tokenization.
- How do I handle punctuation in tokenization?
- For word tokenization, decide whether punctuation is a separate token or attached to the word. "Hello, world!" → ["Hello", ",", "world", "!"] (separate) or ["Hello,", "world!"] (attached). For NLP, separate tokens are standard. Use regex or an NLP library for consistent handling.