Skip to content
uencoder

Stop Words

Stop words are high-frequency, low-semantics words (the, is, at, on) that are commonly filtered out in retrieval and NLP to reduce noise.

Purpose

Removes function words that contribute little to meaning, improving retrieval precision and efficiency.

Mechanism

A predefined stop-word list (e.g., 179 English words in NLTK). Stop words are skipped during indexing and querying. Note: in phrase searches ("to be or not to be") stop words carry meaning and must be kept. Stop words differ across domains.

Use cases

  • Search engine index compression
  • TF-IDF computation
  • Feature selection for text classification
  • Keyword extraction preprocessing

Examples

  • Input 
    the quick brown fox
    Output 
    quick brown fox

    Removes "the"

Common errors

Filtering stop words breaks phrase semantics
Keep stop words in phrase queries

Related terms

Tokenization
Tokenization splits text into a sequence of semantic units (words, subwords, characters); it is a foundational step in NLP and search engines.
GLOSSARY
TF-IDF
TF-IDF (term frequency-inverse document frequency) measures how important a word is to a document: words frequent in the document but rare in the corpus receive high weights.
GLOSSARY
Stemming
Stemming is a heuristic process that reduces words to their stems (e.g., running → run), so that inflectional variations do not affect retrieval matching.
GLOSSARY

References