TF-IDF

TF-IDF (term frequency-inverse document frequency) measures how important a word is to a document: words frequent in the document but rare in the corpus receive high weights.

Purpose

Distinguishes document content words from generic function words, used for retrieval ranking and feature weighting.

Mechanism

TF = the count of a word in a document; IDF = log(N/DF), where N is the total number of documents and DF is the number of documents containing the word. TF-IDF = TF × IDF. Common words (the) have low IDF; rare words have high IDF.

Use cases

Search engine relevance ranking
Document keyword extraction
Feature weighting for text classification
Content similarity in recommender systems

Examples

Input

The word "the" appears in all documents

Output

IDF≈0, low weight

Common words get low weights

Common errors

Failing to log-transform or normalize TF biases long documents: Use 1+log(TF) and document length normalization

Related terms

The bag-of-words model represents text as an unordered word-frequency vector, ignoring grammar and word order; it is the foundational representation for text classification and retrieval.

Cosine Similarity

Cosine similarity measures the cosine of the angle between two vectors, ignoring vector length; it is widely used for text-vector similarity computation.

Stop words are high-frequency, low-semantics words (the, is, at, on) that are commonly filtered out in retrieval and NLP to reduce noise.

References