A crucial aspect of NLP involves addressing "stop words." These are the words that occur frequently in a text but do not often provide significant insights on their own. For example, words such as "the," "and," and "I," while commonplace, typically do not contribute meaningful information about the specific topic of a document, hence the name stopwords. Eliminating these words help us better identify unique and relevant terms.
It's important to emphasize that there is no universally agreed-upon list of stop words in the field of NLP, each framework offers its own list of stop words. In the following, we will explore Natural Language Toolkit (NLTK) stop words list and compare it to that of WordCloud, a popular python library for word cloud plotting.
If you don't have the packages installed in your computer, simply run the following code on your termianl/command window:
pip install nltk wordcloud
NLTK’s Stopword List
In order to access NLTK's stopwords we first need to download the stopwords package:
import nltknltk.download('stopwords')
A graphical user interface (GUI) will appear when you are prompted. Select “All” and then click “Download”. This may take some time, so may want to grab a coffee (or tea) while waiting.
Once, it is downloaded, you can access the package via:
from nltk.corpus import stopwords as sw
To see the list of languages NLTK provides stopword:
At the time of this writing, NLTK supports stopwords in 29 languages.To access the stopwords for a given language we use words module. Lets check the number of stopwords NLTK provides per language:
print("Number of stop words per language:")for i,l inenumerate(sw.fileids()):print(f"{i+1}. {l.title()} - {len(sw.words(l))}")
Number of stop words per language:
1. Arabic - 754
2. Azerbaijani - 165
3. Basque - 326
4. Bengali - 398
5. Catalan - 278
6. Chinese - 841
7. Danish - 94
8. Dutch - 101
9. English - 179
10. Finnish - 235
11. French - 157
12. German - 232
13. Greek - 265
14. Hebrew - 221
15. Hinglish - 1036
16. Hungarian - 199
17. Indonesian - 758
18. Italian - 279
19. Kazakh - 324
20. Nepali - 255
21. Norwegian - 176
22. Portuguese - 207
23. Romanian - 356
24. Russian - 151
25. Slovene - 1784
26. Spanish - 313
27. Swedish - 114
28. Tajik - 163
29. Turkish - 53
English language has 179 stopwords in NLTK at the
The Top3 languages with the highest number of stopwords:
Slovene - 1784
Hinglish (a mix of Hindi and English) - 1036
Arabic - 754
Let's take a closer look at the stopwords in 'English'.
stops = sw.words('english')print(f"Number of stopwords in NLTK :{len(sorted(stops))}")print(sorted(stops))
The output shows us that WordCloud provides us with 192 stopwords in English. Now, it is time to compare both lists!
Stopwords: NLTK vs WordCloud
Though, there are multiple ways to extract the words unique to each library, arguably the easiest way to do it is through Numpy' s setdiff1d function, which finds the set difference of two arrays. Let's begin by importing the numpy library:
import numpy as np
The following code snippet will first create those 'difference' arrays then print the number of unique stopwords and the stopwords themselves, respectively per library:
# stopwords unique to the librarysw_nltk_only = np.setdiff1d(sorted(list(stops)),sorted(list(STOPWORDS)))sw_wc_only = np.setdiff1d(sorted(list(STOPWORDS)),sorted(stops))# print stopwords counts and the stopwordsprint(f"\nStopwords only in NLTK: ({len(sw_nltk_only)})\n{sw_nltk_only}")print(f"\nStopwords only in WordCloud: ({len(sw_wc_only)})\n{sw_wc_only}")
The results show that there are 35 stopwords exist only in NLTK, and 48 stopwords only in Wordcloud libraries. Let's create a combined unique list of stopwords from both packages:
# create a combined list of stopwordssw_cobined = stops +list(sw_wc_only)sw_cobined =sorted(sw_cobined)print(f"\nNumber of stopwords NLTK + WC: {len(sw_cobined)}\n{sw_cobined}")python
For this part of the exercise we will be using NLTK's Brown Corpus, which is a well-known text corpus that contains a collection of texts from various domains/categories, making it a great resource for language analysis.
Before we begin, make sure you have NLTK Brown Corpus downloaded. if you haven't already downloaded, you can do so by using the following
import nltknltk.download('brown')
To access the Brown Corpus, you can use the nltk.corpus module:
from nltk.corpus import brown# List the categories (genres) available in the Brown Corpuscategories = brown.categories()print(f"Number of categories: {len(categories)}\n")print(categories)
Number of categories:15['adventure','belles_lettres','editorial','fiction','government','hobbies','humor','learned','lore','mystery','news','religion','reviews','romance','science_fiction']
We will be working with news cateogry for this exercise
# Calculate the number of words in a categorynum_words =len(brown.words(categories='news'))print(f"Number of words in 'news' category: {num_words}")# Calculate the number of sentences in a categorynum_sentences =len(brown.sents(categories='news'))print(f"Number of sentences in 'news' category: {num_sentences}")# Calculate the average words per sentenceavg_words_per_sentence = num_words / num_sentencesprint(f"Average words per sentence in 'news' category: {avg_words_per_sentence:.2f}")
Number of words in 'news' category: 100554
Number of sentences in 'news' category: 4623
Average words per sentence in 'news' category: 21.75
We will now create a frequency distribution of words in a our selected category to show why stopword removal is important:
from nltk import FreqDist# Create a frequency distribution of words in the 'news' categorynews_words = brown.words(categories='news')fdist =FreqDist(news_words)# Print the most common 15 words and their frequenciescommon_words = fdist.most_common(15)print("Most common words in 'news' category:")i=0for word, freq in common_words: i +=1print(f"{i}) {word}: {freq}")
Most common words in 'news' category:
1) the: 5580
2) ,: 5188
3) .: 4030
4) of: 2849
5) and: 2146
6) to: 2116
7) a: 1993
8) in: 1893
9) for: 943
10) The: 806
11) that: 802
12) ``: 732
13) is: 732
14) was: 717
15) '': 702
Notice that NLTK treats functions as words as well and the output shows us that the most common 15 words in our news category, which consists of 100,554 words, are either so called the stopwords or the punctuations! Using our sw_cobined, we now filter the stopwords from our text:
# filter our stopwordsnews_words_filtered = [w.lower()for w in news_words if w.lower()notin sw_cobined]# print statsprint(f"Number of words prior to stopwords filtering: {len(news_words):,}")print(f"Number of words after stopwords filtering: {len(news_words_filtered):,}")print(f"Difference: {len(news_words)-len(news_words_filtered):,} (words have been removed)")
Number of words prior to stopwords filtering: 100,554
Number of words after stopwords filtering: 62,849
Difference: 37,705 (words have been removed)
We use lower case for each word, w.lower(), to make sure 1) when calculate the frequency of a word we should ignore the case status to have the correct counts, 2) because our combined list only consists of lower case words, we need to make sure that we also convert each word before checking its existence in the stopwords list.
Time to create a frequency distribution without the stopwords and display the most common words after filtering:
# create a frequency distribution of words excluding stopwordsfdist_filtered =FreqDist(news_words_filtered)# Print the most common words and their frequenciescommon_words_filtered = fdist_filtered.most_common(15)print("Most common words in 'news' category:")i=0for word, freq in common_words_filtered: i +=1print(f"{i}) {word}: {freq}")
The output now displays a different set of words. Removal of words like "mr.", "mrs.", or "one" and "two" will depend on the insights we are trying to extract from our analysis, we can either leave them as is or add them to our combined list of stopwords and filter them out. However, unarguably, punctuations cause noise in our dataset and should be removed before furthering in our project. NLTK currently doesn't provide a module for punctions and generally we use python's string module instance of punctuation to obtain all punctions. (You can manually enter the punctuations too)
import string# create a variable for punctuationspuncs = string.punctuation #includes '!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'# extend stopword cobined listsw_cobined_extented = sw_cobined + ['mr.','mrs.']# filter out stop words and punctionsnews_words_filtered_2 = [w.lower()for w in news_words_filtered if w.lower()notin sw_cobined_extented and w notin puncs]# create a frequency distribution of words excluding stopwords and punctionsfdist_filtered_2 =FreqDist(news_words_filtered_2)# Print the most common words and their frequenciescommon_words_filtered_2 = fdist_filtered_2.most_common(15)print("Most common words in 'news' category:")i=0for word, freq in common_words_filtered_2: i +=1print(f"{i}) {word}: {freq}")
Though, punction instance helped us remove most of the punctions, we still see '``', "''", '--' remained in our most common word list. In such situations it is best to do manual filtering:
# filter out stop words and punctionsnews_words_filtered_3 = [w.lower()for w in news_words_filtered_2 if (w.lower()notin sw_cobined_extented) and (w notin puncs) and (w notin ['``',"''",'--'])]# create a frequency distribution of words excluding stopwords and punctionsfdist_filtered_3 =FreqDist(news_words_filtered_3)# print the most common words and their frequenciescommon_words_filtered_3 = fdist_filtered_3.most_common(15)print("Most common words in 'news' category:")i=0for word, freq in common_words_filtered_3: i +=1print(f"{i}) {word}: {freq}")
Now, we have a list that consists of only the most common words along with their frequencies.
In this exercise, we explored common words, called stopwords, that occur often in a text but do not provide us significant insights. We also saw that different libraries like NLTK and WorldCloud have their own stopword lists and that there is no such thing as an ultimate list for stopwords. We then filter out those words from one of the categories in NLTK's Brown corpus. Exploring word frequencies in the dataset showed us that removal of stopwords and the punctuations is an iterative process. This brings us to the gist of this exercise that when we work with text data, we need to be aware of the fact that stopword list we are given is limited and we can always combine multiple lists or even create our own stopword list specific to the needs of our project, and the filtering out stopwords and punctuations can be an iterative process (Check out also Agoda Reviews project for reference). If you like to explore what other frameworks offer in terms of stopwords, you can check the list of other NLP packages in the Natural Language Processing section.
