Calcular TF-IDF for text importance
TF-IDF (Término Frequency-Inverse Documento Frequency) measures palabra importance en un documento relative a un corpus. Usado en search y texto mining.
- 1
TF-IDF = TF × IDF. TF (term frequency) = count of term in document / total terms in document. IDF (inverse document frequency) = log(total documents / documents containing term). Words that appear in many documents get low IDF (common words). Words unique to a document get high IDF (distinctive words). TF-IDF is high for words that are frequent in one document but rare across the corpus.
- 2 Contador de palabras →
Contar term frequencies in each document. This gives TF (raw count). Normalize by document length for comparable scores across documents of different sizes. Build a vocabulary from all documents.
- 3 Frecuencia de caracteres →
Analyze character distribution to understand the text corpus. This complements word-level analysis — some text patterns are character-level (e.g., code, URLs). Use both word and character frequency for comprehensive text analysis.
Preguntas frecuentes
- What does a high TF-IDF score mean?
- The word is frequent in this document (high TF) but rare across the corpus (high IDF). This means the word is distinctive to this document — it characterizes the document's topic. Search engines use TF-IDF to rank documents: a query term with high TF-IDF in a document means the document is relevant to the query.
- TF-IDF vs BM25?
- BM25 is an improved version of TF-IDF used by Elasticsearch and Lucene. It saturates TF (so a word appearing 100 times is not 100x more relevant than once) and normalizes document length. BM25 generally outperforms raw TF-IDF for search ranking. Use BM25 for production search; TF-IDF for teaching and simple applications.