Understanding TF-IDF (Term Frequency-Inverse Document Frequency)

Last Updated : 19 Jan, 2023

TF-IDF stands for Term Frequency Inverse Document Frequency of records. It can be defined as the calculation of how relevant a word in a series or corpus is to a text. The meaning increases proportionally to the number of times in the text a word appears but is compensated by the word frequency in the corpus (data-set).

Terminologies:

Term Frequency: In document d, the frequency represents the number of instances of a given word t. Therefore, we can see that it becomes more relevant when a word appears in the text, which is rational. Since the ordering of terms is not significant, we can use a vector to describe the text in the bag of term models. For each specific term in the paper, there is an entry with the value being the term frequency.

The weight of a term that occurs in a document is simply proportional to the term frequency.

tf(t,d) = count of t in d / number of words in d

Document Frequency: This tests the meaning of the text, which is very similar to TF, in the whole corpus collection. The only difference is that in document d, TF is the frequency counter for a term t, while df is the number of occurrences in the document set N of the term t. In other words, the number of papers in which the word is present is DF.

df(t) = occurrence of t in documents

Inverse Document Frequency: Mainly, it tests how relevant the word is. The key aim of the search is to locate the appropriate records that fit the demand. Since tf considers all terms equally significant, it is therefore not only possible to use the term frequencies to measure the weight of the term in the paper. First, find the document frequency of a term t by counting the number of documents containing the term:

df(t) = N(t)
where
df(t) = Document frequency of a term t
N(t) = Number of documents containing the term t

Term frequency is the number of instances of a term in a single document only; although the frequency of the document is the number of separate documents in which the term appears, it depends on the entire corpus. Now let’s look at the definition of the frequency of the inverse paper. The IDF of the word is the number of documents in the corpus separated by the frequency of the text.

idf(t) = N/ df(t) = N/N(t)

The more common word is supposed to be considered less significant, but the element (most definite integers) seems too harsh. We then take the logarithm (with base 2) of the inverse frequency of the paper. So the if of the term t becomes:

idf(t) = log(N/ df(t))

Computation: Tf-idf is one of the best metrics to determine how significant a term is to a text in a series or a corpus. tf-idf is a weighting system that assigns a weight to each word in a document based on its term frequency (tf) and the reciprocal document frequency (tf) (idf). The words with higher scores of weight are deemed to be more significant.

Usually, the tf-idf weight consists of two terms-

Normalized Term Frequency (tf)
Inverse Document Frequency (idf)

tf-idf(t, d) = tf(t, d) * idf(t)

In python tf-idf values can be computed using TfidfVectorizer() method in sklearn module.

Syntax:

sklearn.feature_extraction.text.TfidfVectorizer(input)

Parameters:

input: It refers to parameter document passed, it can be a filename, file or content itself.

Attributes:

vocabulary_: It returns a dictionary of terms as keys and values as feature indices.

idf_: It returns the inverse document frequency vector of the document passed as a parameter.

Returns:

fit_transform(): It returns an array of terms along with tf-idf values.

get_feature_names(): It returns a list of feature names.

Step-by-step Approach:

Import modules.

Python3

# import required module
from sklearn.feature_extraction.text import TfidfVectorizer

Collect strings from documents and create a corpus having a collection of strings from the documents d0, d1, and d2.

Python3

# assign documents
d0 = 'Geeks for geeks'
d1 = 'Geeks'
d2 = 'r2j'
 
# merge documents into a single corpus
string = [d0, d1, d2]

Get tf-idf values from fit_transform() method.

Python3

# create object
tfidf = TfidfVectorizer()
 
# get tf-df values
result = tfidf.fit_transform(string)

Display idf values of the words present in the corpus.

Python3

# get idf values
print('\nidf values:')
for ele1, ele2 in zip(tfidf.get_feature_names(), tfidf.idf_):
    print(ele1, ':', ele2)

Output:

Display tf-idf values along with indexing.

Python3

# get indexing
print('\nWord indexes:')
print(tfidf.vocabulary_)
 
# display tf-idf values
print('\ntf-idf value:')
print(result)
 
# in matrix form
print('\ntf-idf values in matrix form:')
print(result.toarray())

Output:

The result variable consists of unique words as well as the tf-if values. It can be elaborated using the below image:

From the above image the below table can be generated:

Document	Word	Document Index	Word Index	tf-idf value
d0	for	0	0	0.549
d0	geeks	0	1	0.8355
d1	geeks	1	1	1.000
d2	r2j	2	2	1.000

Below are some examples which depict how to compute tf-idf values of words from a corpus:

Example 1: Below is the complete program based on the above approach:

Python3

# import required module
from sklearn.feature_extraction.text import TfidfVectorizer
 
# assign documents
d0 = 'Geeks for geeks'
d1 = 'Geeks'
d2 = 'r2j'
 
# merge documents into a single corpus
string = [d0, d1, d2]
 
# create object
tfidf = TfidfVectorizer()
 
# get tf-df values
result = tfidf.fit_transform(string)
 
# get idf values
print('\nidf values:')
for ele1, ele2 in zip(tfidf.get_feature_names(), tfidf.idf_):
    print(ele1, ':', ele2)
 
# get indexing
print('\nWord indexes:')
print(tfidf.vocabulary_)
 
# display tf-idf values
print('\ntf-idf value:')
print(result)
 
# in matrix form
print('\ntf-idf values in matrix form:')
print(result.toarray())

Output:

Example 2: Here, tf-idf values are computed from a corpus having unique values.

Python3

# import required module
from sklearn.feature_extraction.text import TfidfVectorizer
 
# assign documents
d0 = 'geek1'
d1 = 'geek2'
d2 = 'geek3'
d3 = 'geek4'
 
# merge documents into a single corpus
string = [d0, d1, d2, d3]
 
# create object
tfidf = TfidfVectorizer()
 
# get tf-df values
result = tfidf.fit_transform(string)
 
# get indexing
print('\nWord indexes:')
print(tfidf.vocabulary_)
 
# display tf-idf values
print('\ntf-idf values:')
print(result)

Output:

Example 3: In this program, tf-idf values are computed from a corpus having similar documents.

Python3

# import required module
from sklearn.feature_extraction.text import TfidfVectorizer
 
# assign documents
d0 = 'Geeks for geeks!'
d1 = 'Geeks for geeks!'
 
 
# merge documents into a single corpus
string = [d0, d1]
 
# create object
tfidf = TfidfVectorizer()
 
# get tf-df values
result = tfidf.fit_transform(string)
 
# get indexing
print('\nWord indexes:')
print(tfidf.vocabulary_)
 
# display tf-idf values
print('\ntf-idf values:')
print(result)

Output:

Example 4: Below is the program in which we try to calculate tf-idf value of a single word geeks is repeated multiple times in multiple documents.

Python3

# import required module
from sklearn.feature_extraction.text import TfidfVectorizer
 
# assign corpus
string = ['Geeks geeks']*5
 
# create object
tfidf = TfidfVectorizer()
 
# get tf-df values
result = tfidf.fit_transform(string)
 
# get indexing
print('\nWord indexes:')
print(tfidf.vocabulary_)
 
# display tf-idf values
print('\ntf-idf values:')
print(result)

Output:

Bag of word and Frequency count in text using sklearn

riturajsaha

News

Improve

Similar Reads

Understanding TF-IDF (Term Frequency-Inverse Document Frequency)

TF-IDF stands for Term Frequency Inverse Document Frequency of records. It can be defined as the calculation of how relevant a word in a series or corpus is to a text. The meaning increases proportionally to the number of times in the text a word appears but is compensated by the word frequency in t

Bag of word and Frequency count in text using sklearn

Text data is ubiquitous in today's digital world, from emails and social media posts to research articles and customer reviews. To analyze and derive insights from this textual information, it's essential to convert text into a numerical form that machine learning algorithms can process. One of the

Understanding min_df and max_df in scikit CountVectorizer

In natural language processing (NLP), text preprocessing is a critical step that significantly impacts the performance of machine learning models. One common preprocessing step is converting text data into numerical data using techniques like Bag of Words (BoW). The CountVectorizer class in the scik

NLP | Storing Conditional Frequency Distribution in Redis

The nltk.probability.ConditionalFreqDist class is a container for FreqDist instances, with one FreqDist per condition. It is used to count frequencies that are dependent on another condition, such as another word or a class label. It is being used here to create an API-compatible class on top of Red

Classification of text documents using sparse features in Python Scikit Learn

Classification is a type of machine learning algorithm in which the model is trained, so as to categorize or label the given input based on the provided features for example classifying the input image as an image of a dog or a cat (binary classification) or to classify the provided picture of a liv

An Easy Approach to TF-IDF Using R

TF-IDF (Term Frequency-Inverse Document Frequency) is a fundamental technique in natural language processing and information retrieval for assessing the importance of a term within a document relative to a collection of documents. In this article, we'll explore how to implement TF-IDF using R Progra

KNN for Text Classification using TF-IDF scores

S When it comes to classifying text into different categories, one effective approach is using the K-Nearest Neighbors (KNN) algorithm in conjunction with Term Frequency-Inverse Document Frequency (TF-IDF) scores. Here’s a brief overview: TF-IDF scores help weigh the importance of words in a documen

Document Similarity Using LSA in R

Latent Semantic Analysis (LSA) is a powerful technique in natural language processing (NLP) that helps uncover the hidden relationships between words in a set of documents. Unlike basic word-matching techniques, LSA goes beyond the surface level by analyzing the context in which words appear. It red

Maximum repeated frequency of characters in a given string

Given a string S, the task is to find the count of maximum repeated frequency of characters in the given string S.Examples: Input: S = "geeksgeeks" Output: Frequency 2 is repeated 3 times Explanation: Frequency of characters in the given string - {"g": 2, "e": 4, "k": 2, "s": 2} The frequency 2 is r

Using CountVectorizer to Extracting Features from Text

CountVectorizer is a great tool provided by the scikit-learn library in Python. It is used to transform a given text into a vector on the basis of the frequency (count) of each word that occurs in the entire text. This is helpful when we have multiple such texts, and we wish to convert each word in

Frequency Measuring Techniques for Competitive Programming

Measuring the frequency of elements in an array is a really handy skill and is required a lot of competitive coding problems. We, in a lot of problems, are required to measure the frequency of various elements like numbers, alphabets, symbols, etc. as a part of our problem. Examples: Input: arr[] =

Visualizing TF-IDF Scores: A Comprehensive Guide to Plotting a Document TF-IDF 2D Graph

For Natural Language Processing (NLP), Term Frequency-Inverse Document Frequency (TF-IDF) is a crucial technique used to evaluate the importance of a word in a document relative to a collection of documents (corpus). Visualizing TF-IDF scores can provide insightful perspectives on the distribution a

Array range queries for elements with frequency same as value

Given an array of N numbers, the task is to answer Q queries of the following type: query(start, end) = Number of times a number x occurs exactly x times in a subarray from start to end Examples: Input : arr = {1, 2, 2, 3, 3, 3} Query 1: start = 0, end = 1, Query 2: start = 1, end = 1, Query 3: star

Latent Text Analysis (lsa Package) Using Whole Documents in R

Latent Text Analysis (LTA) is a technique used to discover the hidden (latent) structures within a set of documents. This approach is instrumental in natural language processing (NLP) for identifying patterns, topics, and relationships in large text corpora. This article will explore using whole doc

Count of Strings of Array having given prefixes for Q query

Given two arrays of strings containing words[] and queries[] having N and Q strings respectively, the task is to find the number of strings from words[] having queries[i] as the prefix for all the strings in queries[]. Examples: Input: words[] = { "geeks", "geeks", "geeks for geeks", "string", "stro

Create a Term-Document Matrix with a Dictionary of One or Two Words using R

A Term-Document Matrix (TDM), is a significant technique in text mining, used to represent the frequency of words across a collection of documents. Each row in the matrix corresponds to a unique phrase, and each column represents a document. The cell values reflect the number of times a phrase occur

Queries to calculate difference between the frequencies of the most and least occurring characters in specified substring

Given a string str consisting of N lowercase characters and an array Q[][] with each row of the form {l, r} representing a query. For each query, the task is to find the difference between the maximum frequency and minimum frequency of the characters in the substring {str[l], .... str[r]}. Note: Con

Mastering TF-IDF Calculation with Pandas DataFrame in Python

Term Frequency-Inverse Document Frequency (TF-IDF) is a popular technique in Natural Language Processing (NLP) to transform text into numerical features. It measures the importance of a word in a document relative to a collection of documents (corpus). In this article, we will explore how to compute

Find the Frequency of a Particular Word in a Cell in an Excel Table in Python

In this article, we'll look at how to use Python to find the number of times a word appears in a cell of an Excel file. Before we begin with the steps of the solution, the following modules/libraries must be installed. We will use the following sample Excel file to determine the frequency of the inp

Article Tags :

Practice Tags :