Sumy is one of the Python libraries for Natural Language Processing tasks. It is mainly used for automatic summarization of paragraphs using different algorithms. We can use different summarizers that are based on various algorithms, such as Luhn, Edmundson, LSA, LexRank, and KL-summarizers. We will learn in-depth about each of these algorithms in the upcoming sections. Sumy requires minimal code to build a summary, and it can be easily integrated with other Natural Language Processing tasks. This library is suitable for summarizing large documents.
In this article, we will first understand the benefits of using this article for our summarization task. Then we will look into the process of installing this library into our systems. Then we will first understand how to build a tokenizer and a stemmer before going into the summarization algorithms. We will then understand and implement each summarizer Sumy provides.
Benefits
- Sumy provides many summarization algorithms, allowing users to choose from a wide range of summarizers based on their preferences.
- This library integrates efficiently with other NLP libraries.
- The library is easy to install and use, requiring minimal setup.
- We can summarize lengthy documents using this library.
- Sumy can be easily customized to fit specific summarization needs.
Installation
Now let’s look at the different ways to install this library on our system.
Via PyPI
To install it via PyPI, then paste the below command in your terminal.
pip install sumy
If you are working in a notebook such as Jupyter Notebook, Kaggle, or Google Colab, then add ‘!’ before the above command.
Via GitHub
There are two ways to install this library from GitHub: the first is to install it directly using the git link, and the second is to install it using pip.
pip install git+git://github.com/miso-belica/sumy.git
python3 setup.py install
Tokenizer
Tokenization is one of the most important task in text preprocessing. In tokenization, we divide a paragraph into sentences and then breakdown those sentences into individual words. By tokenizing the text, Sumy can better understand its structure and meaning, which improves the accuracy and quality of the summaries generated.
Now, let’s see how to build a tokenizer using Sumy lirary. We will first import the Tokenizer module from sumy, then we will download the ‘punkt’ from NLTK. We will then create an object or instance of Tokenizer for English language. We will then convert a sample text into sentences, then we will print the tokenized words for each sentence.
Python
from sumy.nlp.tokenizers import Tokenizer
import nltk
nltk.download('punkt')
tokenizer = Tokenizer("en")
sentences = tokenizer.to_sentences("Hello, this is GeeksForGeeks! We are a computer science portal for geeks, offering a wide range of articles, tutorials, and resources on various topics in computer science and programming. Our mission is to provide quality education and knowledge sharing to help you excel in your career and academic pursuits. Whether you're a beginner looking to learn the basics of coding or an experienced developer seeking advanced concepts, GeeksForGeeks has something for everyone. ")
for sentence in sentences:
print(tokenizer.to_words(sentence))
OUTPUT:
('Hello', 'this', 'is', 'GeeksForGeeks')
('We', 'are', 'a', 'computer', 'science', 'portal', 'for', 'geeks', 'offering', 'a', 'wide', 'range', 'of', 'articles', 'tutorials', 'and', 'resources', 'on', 'various', 'topics', 'in', 'computer', 'science', 'and', 'programming')
('Our', 'mission', 'is', 'to', 'provide', 'quality', 'education', 'and', 'knowledge', 'sharing', 'to', 'help', 'you', 'excel', 'in', 'your', 'career', 'and', 'academic', 'pursuits')
('Whether', 'you', 'a', 'beginner', 'looking', 'to', 'learn', 'the', 'basics', 'of', 'coding', 'or', 'an', 'experienced', 'developer', 'seeking', 'advanced', 'concepts', 'GeeksForGeeks', 'has', 'something', 'for', 'everyone')Stemmer
Stemming is the process of reducing a word to its base or root form. This helps in normalizing words so that different forms of a word are treated as the same term. By doing this, summarization algorithms can more effectively recognize and group similar words, thereby improving the summarization quality. The stemmer is particularly useful when we have large texts that have various forms of the same words.
To create a stemmer using the Sumy library, we will first import the `Stemmer` module from Sumy. Then, we will create an object of `Stemmer` for the English language. Next, we will pass a word to the stemmer to reduce it to its root form. Finally, we will print the stemmed word.
Python
from sumy.nlp.stemmers import Stemmer
stemmer = Stemmer("en")
stem = stemmer("Blogging")
print(stem)
Output:
blog
Summarizers
Luhn Summarizer
The Luhn Summarizer is one of the summarization algorithms provided by the Sumy library. This summarizer is based on the concept of frequency analysis, where the importance of a sentence is determined by the frequency of significant words within it. The algorithm identifies words that are most relevant to the topic of the text by filterin gout some common stop words and then ranks sentences. The Luhn Summarizer is effective for extracting key sentences from a document. Here's how to build the Luhn Summarizer:
Python
from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.luhn import LuhnSummarizer
from sumy.nlp.stemmers import Stemmer
from sumy.utils import get_stop_words
import nltk
nltk.download('punkt')
def summarize_paragraph(paragraph, sentences_count=2):
parser = PlaintextParser.from_string(paragraph, Tokenizer("english"))
summarizer = LuhnSummarizer(Stemmer("english"))
summarizer.stop_words = get_stop_words("english")
summary = summarizer(parser.document, sentences_count)
return summary
if __name__ == "__main__":
paragraph = """Artificial intelligence (AI) is intelligence demonstrated by machines, in contrast
to the natural intelligence displayed by humans and animals. Leading AI textbooks define
the field as the study of "intelligent agents": any device that perceives its environment
and takes actions that maximize its chance of successfully achieving its goals. Colloquially,
the term "artificial intelligence" is often used to describe machines (or computers) that mimic
"cognitive" functions that humans associate with the human mind, such as "learning" and "problem solving"."""
sentences_count = 2
summary = summarize_paragraph(paragraph, sentences_count)
for sentence in summary:
print(sentence)
Output:
Artificial intelligence (AI) is intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans and animals.
Colloquially, the term "artificial intelligence" is often used to describe machines (or computers) that mimic "cognitive" functions that humans associate with the human mind, such as "learning" and "problem solving".
Edmundson Summarizer
The Edmundson Summarizer is another powerful algorithm provided by the Sumy library. Unlike other summarizers that primarily rely on statistical and frequency-based methods, the Edmundson Summarizer allows for a more tailored approach through the use of bonus words, stigma words, and null words. These type of woreds enable the algorithm to emphasize or de-emphasize those words in the summarized text. Here's how to build the Edmundson Summarizer:
Python
from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.edmundson import EdmundsonSummarizer
from sumy.nlp.stemmers import Stemmer
from sumy.utils import get_stop_words
import nltk
nltk.download('punkt')
def summarize_paragraph(paragraph, sentences_count=2, bonus_words=None, stigma_words=None, null_words=None):
parser = PlaintextParser.from_string(paragraph, Tokenizer("english"))
summarizer = EdmundsonSummarizer(Stemmer("english"))
summarizer.stop_words = get_stop_words("english")
if bonus_words:
summarizer.bonus_words = bonus_words
if stigma_words:
summarizer.stigma_words = stigma_words
if null_words:
summarizer.null_words = null_words
summary = summarizer(parser.document, sentences_count)
return summary
if __name__ == "__main__":
paragraph = """Artificial intelligence (AI) is intelligence demonstrated by machines, in contrast
to the natural intelligence displayed by humans and animals. Leading AI textbooks define
the field as the study of "intelligent agents": any device that perceives its environment
and takes actions that maximize its chance of successfully achieving its goals. Colloquially,
the term "artificial intelligence" is often used to describe machines (or computers) that mimic
"cognitive" functions that humans associate with the human mind, such as "learning" and "problem solving"."""
sentences_count = 2
bonus_words = ["intelligence", "AI"]
stigma_words = ["contrast"]
null_words = ["the", "of", "and", "to", "in"]
summary = summarize_paragraph(paragraph, sentences_count, bonus_words, stigma_words, null_words)
for sentence in summary:
print(sentence)
Output:
Artificial intelligence (AI) is intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans and animals.
Leading AI textbooks define the field as the study of "intelligent agents": any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals.
LSA Summarizer
The LSA summarizer is the best one amognst all because it works by identifying patterns and relationships between texts, rather than soley rely on frequency analysis. This LSA summarizer generates more contextually accurate summaries by understanding the meaning and context of the input text. Here's how to build the LSA Summarizer:
Python
from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.lsa import LsaSummarizer
from sumy.nlp.stemmers import Stemmer
from sumy.utils import get_stop_words
import nltk
nltk.download('punkt')
def summarize_paragraph(paragraph, sentences_count=2):
parser = PlaintextParser.from_string(paragraph, Tokenizer("english"))
summarizer = LsaSummarizer(Stemmer("english"))
summarizer.stop_words = get_stop_words("english")
summary = summarizer(parser.document, sentences_count)
return summary
if __name__ == "__main__":
paragraph = """Artificial intelligence (AI) is intelligence demonstrated by machines, in contrast
to the natural intelligence displayed by humans and animals. Leading AI textbooks define
the field as the study of "intelligent agents": any device that perceives its environment
and takes actions that maximize its chance of successfully achieving its goals. Colloquially,
the term "artificial intelligence" is often used to describe machines (or computers) that mimic
"cognitive" functions that humans associate with the human mind, such as "learning" and "problem solving"."""
sentences_count = 2
summary = summarize_paragraph(paragraph, sentences_count)
for sentence in summary:
print(sentence)
Output:
Leading AI textbooks define the field as the study of "intelligent agents": any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals.
Colloquially, the term "artificial intelligence" is often used to describe machines (or computers) that mimic "cognitive" functions that humans associate with the human mind, such as "learning" and "problem solving".
Conclusion
In conclusion, Sumy is one of the best automatic text summarizing libraries available. We can also use this library for tasks like tokenization and stemming. By using different algorithms like Luhn, Edmundson, and LSA, we can generate concise and meaningful summaries based on our specific needs. Although we have used a smaller paragraph for examples, we can summarize lengthy documents using this library in no time.