Learn how to use coherence and perplexity scores to evaluate your topic models with gensim, a Python library for natural language processing.

When doing topic modelling with LDA; generally coherence increases for a while but then decreases. whereas perplexity keeps decreasing as the number of topics increase.

How do you evaluate the coherence and perplexity of your topic models with gensim?

Topic modeling is a technique to discover the latent themes or topics in a collection of documents. It can help you analyze, explore, and visualize your text data in a meaningful way. But how do you know if your topic models are good enough? How do you measure their quality and performance?

One way to evaluate your topic models is to use coherence and perplexity scores. These are two metrics that can help you compare different models and find the optimal number of topics for your data. In this article, you will learn what coherence and perplexity are, how to calculate them with gensim, and how to interpret them.

1 What is coherence?

Coherence is a measure of how well the words in a topic relate to each other. It reflects the human intuition of what makes a good topic. A high coherence score means that the topic is consistent, clear, and relevant. A low coherence score means that the topic is vague, noisy, or irrelevant.

To calculate coherence, you need to use a reference corpus, such as Wikipedia, that represents the general knowledge of the domain. Gensim provides several coherence models, such as C_V, U_Mass, C_Umi, and C_Npmi, that use different methods to compare the word distributions in your topics and the reference corpus. You can use the CoherenceModel class in gensim to compute the coherence score for your topic model.

Add your perspective

2 What is perplexity?

Perplexity is a measure of how well the topic model predicts new or unseen data. It reflects the generalization ability of the model. A low perplexity score means that the model is confident and accurate in its predictions. A high perplexity score means that the model is uncertain and inaccurate in its predictions.

To calculate perplexity, you need to use a held-out test set, that is, a subset of documents that are not used for training the model. Gensim provides the log_perplexity method for LdaModel and LdaMulticore classes to compute the perplexity score for your topic model. You can also use the get_perplexity method for LdaSeqModel class if you are using dynamic topic modeling.

Add your perspective

3 How to use gensim?

Gensim is a widely-used Python library for natural language processing and topic modeling. It provides a range of algorithms and tools to generate, train, and assess topic models. To use gensim, you need to preprocess your text data, create a dictionary and corpus, select a topic modeling algorithm, train the model with the number of topics, hyperparameters, and iterations specified, and evaluate the model with CoherenceModel and log_perplexity methods to calculate the coherence and perplexity scores.

Add your perspective

4 How to interpret the scores?

The coherence and perplexity scores can help you compare different models and find the optimal number of topics for your data. However, there is no fixed rule or threshold for choosing the best model. You should consider the purpose and context of your analysis, as well as the trade-off between coherence and perplexity. Generally, increasing the number of topics will improve the coherence but worsen the perplexity, so you need to find a balance that fits your needs and preferences. Additionally, it’s important to qualitatively inspect the topics to make sure they make sense and capture the essence of your data.

Add your perspective

Devendra Singh

Senior Data Scientist at NatWest Group
(edited)
Report contribution
When doing topic modelling with LDA; generally coherence increases for a while but then decreases. whereas perplexity keeps decreasing as the number of topics increase.

Like

5 How to improve the scores?

If you’re dissatisfied with the coherence and perplexity scores of your topic models, you can attempt to improve them through various strategies. Preprocess your text data more thoroughly, such as by removing irrelevant words, filtering out extreme frequencies, and using n-grams or phrases. Additionally, tune the hyperparameters, such as alpha, beta, and eta, which control the prior distributions of the topics and words. You can use grid search or other optimization methods to arrive at the optimal values for your data. Lastly, try different coherence models such as C_V, U_Mass, C_Umi, and C_Npmi to compare the word distributions in your topics to the reference corpus. Experiment with different models to determine which one works best for your data.

Add your perspective

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

Add your perspective

How do you evaluate the coherence and perplexity of your topic models with gensim?

1

2

3

4

5

6

1 What is coherence?

2 What is perplexity?

3 How to use gensim?

4 How to interpret the scores?

5 How to improve the scores?

6 Here’s what else to consider

Information Retrieval

Rate this article

Thanks for your feedback

More articles on Information Retrieval

More relevant reading