Tokenization

Tokenization splits text into a sequence of semantic units (words, subwords, characters); it is a foundational step in NLP and search engines.

Purpose

Turns continuous text into discrete token sequences for downstream processing and analysis.

Mechanism

English is split by whitespace and punctuation (abbreviations and hyphens need special handling). Chinese requires word-segmentation algorithms (maximum matching, statistical models such as HMM/CRF, deep models such as BERT). Subword tokenization (BPE, WordPiece) balances vocabulary size against coverage.

Use cases

Search engine index building
NLP preprocessing (BERT tokenization)
Lexical analysis in code analyzers
Variable parsing in template engines

Examples

Input
```
I love NLP
```
Output
```
["I","love","NLP"]
```
Whitespace tokenization

Common errors

Processing Chinese as raw characters without segmentation: Use segmenters such as jieba/THULAC

Related terms

An N-gram is a sequence of N consecutive units (characters or words) segmented from text, used in language models, similarity, and text analysis.

Stemming is a heuristic process that reduces words to their stems (e.g., running → run), so that inflectional variations do not affect retrieval matching.

Stop words are high-frequency, low-semantics words (the, is, at, on) that are commonly filtered out in retrieval and NLP to reduce noise.

The bag-of-words model represents text as an unordered word-frequency vector, ignoring grammar and word order; it is the foundational representation for text classification and retrieval.

References