Lemmatization

Lemmatization is the linguistic process of reducing a word to its dictionary lemma (e.g., better → good); it is more accurate than stemming.

Purpose

Reduces word forms to canonical lemmas based on dictionaries and part of speech, ensuring linguistic correctness.

Mechanism

Determines the lemma via dictionary lookup plus part-of-speech tagging. For example, better (comparative adjective) → good; running (present participle) → run. Requires POS tagging and dictionary support; slower than stemming but more accurate. Provided by spaCy and NLTK.

Use cases

Precise NLP analysis
Semantic matching in question answering systems
Machine translation preprocessing
Word-sense disambiguation

Examples

Input
```
better → good
```
Output
```
Lemma good
```
Requires part-of-speech tagging

Common errors

Missing part-of-speech tagging leads to wrong lemmas: Use a POS-aware lemmatizer

Related terms

Stemming is a heuristic process that reduces words to their stems (e.g., running → run), so that inflectional variations do not affect retrieval matching.

Tokenization splits text into a sequence of semantic units (words, subwords, characters); it is a foundational step in NLP and search engines.

References