Are You the Keymaster?

OK, you may not be Vince Clortho, Keymaster of Gozer, but you can still master the art of extracting keywords from texts! (Not nearly as exciting, I know, but still useful!) In this article we’ll talk about statistical, graph-based, algorithmic, and machine learning techniques to identify keywords and keyphrases in texts.

A keyword is a word that represents the most important ideas in a text. Keywords are useful for information retrieval systems, extractive summarization, classification tasks, and more. Keywords also provide a useful means of quickly comparing texts, with the idea that texts with very similar keywords are likely to be similar in their totality. Identifying these words is by no means easy though, and different techniques can produce very different results. Further, what makes a “good keyword” is a subjective question, and the type of text you’re working on makes a difference. For that reason, it’s useful to understand the various techniques, and experiment with which ones are best suited to your needs.

In this article we’ll use three categories of texts: news articles, scientific articles, and books. As you’ll see, the types of keywords we extract from each will vary in usefulness, particularly as texts get longer. For the sake of reproducibility, we’ll use datasets from Huggingface for the news and scientific articles, and books from Project Gutenberg. To keep things interesting, we’ll save details about what these texts are actually about until the end. Let’s get started by downloading our texts.

1
from datasets import load_dataset
2
import requests
3
import re
4

5
def read_text_from_url(url: str, encoding: str = "utf-8") -> str:
6
    """
7
    Reads text content from a URL with specified encoding.
8

9
    Args:
10
        url (str): The URL of the text file to read
11
        encoding (str, optional): The encoding to use when reading the text. Defaults to 'utf-8'.
12

13
    Returns:
14
        str: The text content from the URL
15

16
    Raises:
17
        requests.RequestException: If the request fails
18
    """
19
    try:
20
        response = requests.get(url)
21
        response.raise_for_status()  # Raise an exception for bad status codes
22
        response.encoding = encoding  # Set the encoding
23
        return response.text
24
    except requests.RequestException as e:
25
        print(f"Error fetching text from URL: {e}")
26
        raise
27

28
def read_gutenberg_url(url: str, encoding: str = "utf-8") -> str:
29
    """
30
    Reads text content from a Project Gutenberg URL with specified encoding.
31

32
    Args:
33
        url (str): The URL of the text file to read
34
        encoding (str, optional): The encoding to use when reading the text. Defaults to 'utf-8'.
35

36
    Returns:
37
        str: The text content from the URL
38
    """
39
    try:
40
        text = read_text_from_url(url, encoding)
41
        # Remove Project Gutenberg header and footer
42
        start = re.search(r"\*\*\* START OF THE PROJECT GUTENBERG EBOOK.*?\*\*\*", text, re.DOTALL)
43
        end = re.search(r"\*\*\* END OF THE PROJECT GUTENBERG EBOOK.*?\*\*\*", text, re.DOTALL)
44
        if start and end:
45
            text = text[start.end():end.start()].strip()
46
        return text
47
    except Exception as e:
48
        print(f"Error reading Project Gutenberg URL: {e}")
49
        raise
50

51
# load news and science datasets from Huggingface
52
news = load_dataset("cnn_dailymail", "3.0.0", split="train[:2]")
53
news = news.rename_column("article", "text")
54
news = news.rename_column("highlights", "summary")
55

56
science = load_dataset("ccdv/arxiv-summarization", split="train[:2]")
57
science = science.rename_column('article', 'text')
58
science = science.rename_column('abstract', 'summary')
59

60
# download books from Project Gutenberg
61
books = [
62
    { "url": "https://www.gutenberg.org/ebooks/43.txt.utf-8" },
63
    { "url": "https://www.gutenberg.org/ebooks/5200.txt.utf-8" }
64
]
65

66
for book in books:
67
    book["text"] = read_gutenberg_url(book["url"])
68

69
# combine all texts into a single list
70
texts = []
71
for article in news:
72
    texts.append({"category": "news", "text": article["text"], "summary": article["summary"]})
73
for article in science:
74
    texts.append({"category": "science", "text": article["text"], "summary": article["summary"]})
75
for book in books:
76
    texts.append({"category": "book", "text": book["text"], "summary": "N/A"})
77

78
# print length of each text
79
for idx, text in enumerate(texts):
80
    print(f"Text {idx + 1} ({text['category']}): {len(text['text'])} characters")

Text 1 (news): 2527 characters
Text 2 (news): 4051 characters
Text 3 (science): 26346 characters
Text 4 (science): 17929 characters
Text 5 (book): 141352 characters
Text 6 (book): 120348 characters

Keywords by Frequency

The first, and perhaps most obvious approach to identifying keywords, is to simply identify the most common words in each text. We can do this by building a bag of words for each text, and representing our frequency data as a term-document matrix, where each row is a text and each column shows the frequency of a word in the combined vocabulary. (For more detail on this topic, check out the article [How Does a Computer Learn to Read](/blog/how-does-a-computer-learn-to-read TODO)). For the sake of consistency, we’re going to use spaCy for tokenization, and identifying stopwords and punctuation.

1
import spacy
2
import numpy as np
3
from tqdm import tqdm
4

5
# load the spaCy model
6
nlp = spacy.load('en_core_web_sm', disable=['parser', 'ner', 'lemmatizer', 'tagger'])
7

8
# process each text with the spaCy model
9
for text in tqdm(texts):
10
    text["doc"] = nlp(text["text"])
11

12
print("Text lengths in words:")
13
for idx, text in enumerate(texts):
14
    cat = text["category"]
15
    print(f"Text {idx + 1} ({cat}) length: {len(text['doc'])}")

Text lengths in words:
Text 1 (news) length: 555
Text 2 (news) length: 836
Text 3 (science) length: 4675
Text 4 (science) length: 3477
Text 5 (book) length: 33026
Text 6 (book) length: 26788

Now that we have our spaCy documents, the next step is to extract tokens, or perhaps more accurately “candidates” for possible keywords or keyphrases. To do this, we can use the attributes spaCy adds to each token to filter out punctuation. (We could, and probably should, also filter out stopwords, but for reasons that will become clear, we’ll leave them in for now.)

One thing we should consider right away is the difference between a keyword and a keyphrase. A keyword, as you might expect is just a single word, whereas a keyphrase could be multiple words. In many cases, the keyphrases are actually going to be more informative for a text. For example, if we have a scientific document that repeatedly uses the phrase “quantum entaglement”, knowing that is a keyphrase is likely more useful than seeing the keywords “quantum” and “entaglement” on their own—particularly if our extraction method only returns one of those words.

In order to get keyphrases, we’ll include ngram functionality in our tokenization method. An ngram is a group of adjacent words in a text, so in the sentence “Quantum entaglement is a phenomenon where two particles become linked, regardless of distance” the ngrams would include “quantum entaglement”, “entaglement is”, “is a”, and so on. These examples are bigrams (ngrams with 2 words), but an ngram could be any number of words. Typically, it’s useful to keep the minimum and maximum values of n relatively low, so we’ll use the range of (2, 4) in our examples.

1
def get_tokens(
2
    doc: spacy.tokens.Doc,
3
    with_stopwords: bool = False,
4
    with_punct: bool = False,
5
    min_n: int = 1,
6
    max_n: int = 1
7
) -> list[str]:
8
    """
9
    Get candidate tokens and n-grams from a spaCy Doc based on filtering criteria.
10

11
    Args:
12
        doc (spacy.tokens.Doc): Input spaCy Doc
13
        with_stopwords (bool, optional): Include stopwords. Defaults to False.
14
        with_punct (bool, optional): Include punctuation. Defaults to False.
15
        min_n (int, optional): Minimum n-gram length. Defaults to 1.
16
        max_n (int, optional): Maximum n-gram length. Defaults to 1.
17

18
    Returns:
19
        list[str]: List of candidate tokens and n-grams
20
    """
21
    tokens = []
22
    for token in doc:
23
        if token.is_space:
24
            continue
25
        if not with_stopwords and token.is_stop:
26
            continue
27
        if not with_punct and token.is_punct:
28
            continue
29
        tokens.append(token.text.lower())
30

31
    # generate n-grams
32
    # if range is (1, 1), this will return unigrams (single words)
33
    candidates = []
34
    for n in range(min_n, max_n + 1):
35
        for i in range(len(tokens) - n + 1):
36
            ngram = " ".join(tokens[i:i + n])
37
            candidates.append(ngram)
38

39
    return candidates
40

41
def build_vocab(
42
    token_lists: list[list[str]]
43
) -> list[str]:
44
    """
45
    Build vocabulary from lists of tokens.
46

47
    Args:
48
        token_lists (list[list[str]]): Lists of tokens to build vocab from
49

50
    Returns:
51
        list[str]: Sorted list of unique tokens
52
    """
53
    vocab = set()
54
    for tokens in token_lists:
55
        vocab.update(tokens)
56

57
    return sorted(vocab)
58

59
# tokenize each text and build a joint vocabulary
60
for text in texts:
61
    text["tokens"] = get_tokens(
62
        text["doc"],
63
        with_stopwords=True,
64
        with_punct=False,
65
        min_n=1,
66
        max_n=1
67
    )
68

69
# build joint vocabulary and helper maps between words and indices
70
vocab = build_vocab([text["tokens"] for text in texts])
71
stoi = {word: idx for idx, word in enumerate(vocab)}
72
itos = {idx: word for idx, word in enumerate(vocab)}
73

74
print("Vocabulary sizes in unique words:")
75
print(f"Corpus: {len(vocab)}")
76
for idx, text in enumerate(texts):
77
    cat = text["category"]
78
    vocab_size = len(set(text["tokens"]))
79
    print(f"Text {idx + 1} ({cat}): {vocab_size}")

Vocabulary sizes in unique words:
Corpus: 6374
Text 1 (news): 270
Text 2 (news): 330
Text 3 (science): 978
Text 4 (science): 888
Text 5 (book): 3939
Text 6 (book): 2577

Note that as we tokenize our texts, we also built a joint vocabulary across all six texts. We’ll come back to this shortly, but the reason is that identifying keywords in one document sometimes requires knowing how a word is or is not used in other documents. Now that we have our tokenized texts, and our vocab, we can get the word frequencies from each text, and pull out the keywords as the most common words in the texts.

1
def get_token_frequencies(
2
        tokens: list[str],
3
        vocab: list[str],
4
        vocab_stoi_map: dict[str, int]
5
    ) -> np.ndarray:
6
    """
7
    Computes the frequency of each word in a spaCy Doc.
8

9
    Args:
10
        tokens (list[str]): The list of tokens
11
        vocab (list[str]): The vocabulary list
12
        vocab_stoi_map (dict[str, int]): A dictionary mapping words to
13
            indices in the vocabulary
14

15
    Returns:
16
        np.ndarray: The frequency of each word in the vocabulary
17
    """
18
    frequencies = np.zeros(len(vocab), dtype=int)
19
    for token in tokens:
20
        if token.lower() in vocab_stoi_map:
21
            idx = vocab_stoi_map[token.lower()]
22
            frequencies[idx] += 1
23
    return frequencies
24

25
# build the term-document matrix for the texts
26
tdm = np.array([get_token_frequencies(text["tokens"], vocab, stoi) for text in texts])
27
tdm

array([[  1,   3,   1, ...,   0,   0,   0],
        [  0,   0,   0, ...,   0,   0,   0],
        [ 19,   0,   0, ...,   0,   0,   0],
        [  0,   0,   0, ...,   0,   0,   0],
        [  0,   0,   0, ...,   2, 140,   3],
        [  0,   0,   0, ...,  11, 194,   9]], shape=(6, 6374))

Before pulling out the most common keywords, let’s look at the term-document matrix (TDM) we created. The TDM provides counts of every word in the vocabulary for all the texts. This provides a handy way of comparing word usage across texts. Let’s check how a few words differ in frequency.

1
words = ["apple", "forest", "london", "president"]
2
word_indices = [stoi[word] for word in words]
3

4
print("Word frequencies by document category:")
5
print("-" * 50)
6

7
cat_width = max(len(text["category"]) for text in texts)
8
word_width = max(len(word) for word in words)
9

10
header = f"{'Category':<{cat_width}}"
11
for word in words:
12
    header += f"  {word:>{word_width}}"
13
print(header)
14
print("-" * (cat_width + (word_width + 2) * len(words)))
15

16
for idx, text in enumerate(texts):
17
    cat = text["category"]
18
    frequencies = tdm[idx, word_indices]
19
    row = f"{cat:<{cat_width}}"
20
    for freq in frequencies:
21
        row += f"  {freq:>{word_width}}"
22
    print(row)

Word frequencies by document category:
--------------------------------------------------
Category      apple     forest     london  president
---------------------------------------------------
news             0          0          1          0
news             0          0          0          1
science          0          0          0          0
science          0          0          0          0
book             0          1         12          0
book             5          0          0          0

Checking individual words is useful if you know what you’re looking for, but our real goal in this article is to identify keywords that we don’t know ahead of time. Using the TDM, we can now pull out the most frequent words for each text.

1
def get_top_keywords(
2
    tdm: np.ndarray, vocab: list[str], n_keywords: int = 5
3
) -> list[list[str]]:
4
    """
5
    Extracts the top n keywords from a term-document matrix.
6

7
    Args:
8
        tdm (np.ndarray): A term-document matrix
9
        vocab (list[str]): The vocabulary list
10
        n_keywords (int, optional): The number of keywords to extract. Defaults to 5.
11

12
    Returns:
13
        list[list[str]]: The top n keywords for each document
14
    """
15
    keywords = []
16
    for row in tdm:
17
        indices = np.argsort(row)[::-1][:n_keywords]
18
        keywords.append([vocab[idx] for idx in indices])
19
    return keywords
20

21
def print_keywords_lists(
22
        keyword_lists: list[list[str]], texts: list[dict], bulleted: bool = False
23
    ):
24
    """
25
    Prints a list of keywords for each document.
26

27
    Args:
28
        keyword_lists (list[list[str]]): A list of keyword lists
29
        texts (list[dict]): The list of text dictionaries
30
        bulleted (bool, optional): Print the keywords as a bulleted list. Defaults to False.
31
    """
32
    for idx, keywords in enumerate(keyword_lists):
33
        cat = texts[idx]["category"]
34
        if bulleted:
35
            print(f"Text {idx + 1} ({cat}):")
36
            for keyword in keywords:
37
                print(f"  - {keyword}")
38
            print("")
39
        else:
40
            print(f"Text {idx + 1} ({cat}): {', '.join(keywords)}")
41

42
tdm_keywords = get_top_keywords(tdm, vocab, n_keywords=5)
43

44
print("Top keywords by frequency:")
45
print_keywords_lists(tdm_keywords, texts)

Top keywords by frequency:
Text 1 (news): the, he, to, of, a
Text 2 (news): the, in, to, and, they
Text 3 (science): the, of, is, and, a
Text 4 (science): the, of, and, in, is
Text 5 (book): the, and, of, to, i
Text 6 (book): the, to, and, he, his

Well. That’s disappointing. Every single one of our keywords by frequency are stopwords like “and”, “the”, and “to”. This is of course exactly what we expected (and the reason we could/should have filtered stopwords out before building our vocabulary), but it illustrates an important weakness of frequency-based keyword methods. Perhaps the keyphrases will tell us something more.

1
# calculate (2, 4)-grams for each text``
2
for text in texts:
3
    text["ngrams"] = get_tokens(
4
        text["doc"],
5
        with_stopwords=True,
6
        with_punct=False,
7
        min_n=2,
8
        max_n=4
9
    )
10

11
# build joint ngram vocabulary and helper maps between ngrams and indices
12
ngram_vocab = build_vocab([text["ngrams"] for text in texts])
13
ngram_stoi = {ngram: idx for idx, ngram in enumerate(ngram_vocab)}
14
ngram_itos = {idx: ngram for idx, ngram in enumerate(ngram_vocab)}
15

16
print(f"Corpus ngram vocabulary size (2, 4): {len(ngram_vocab)}\n")
17

18
# build the term-document matrix for the ngrams
19
ngram_tdm = np.array(
20
    [get_token_frequencies(text["ngrams"], ngram_vocab, ngram_stoi)
21
     for text in texts]
22
)
23
ngram_keywords = get_top_keywords(ngram_tdm, ngram_vocab, n_keywords=5)
24

25
print("Top keyphrases by frequency:")
26
print_keywords_lists(ngram_keywords, texts, True)

Corpus ngram vocabulary size (2, 4): 137230

Top keyphrases by frequency:
Text 1 (news):
    - harry potter
    - he told
    - in a
    - of the
    - he has

Text 2 (news):
    - mentally ill
    - it 's
    - leifman says
    - on the
    - forgotten floor

Text 3 (science):
    - of the
    - in the
    - learning rates
    - @xmath159 quantile
    - for the

Text 4 (science):
    - of the
    - in the
    - is the
    - and the
    - et al

Text 5 (book):
    - of the
    - in the
    - it was
    - and the
    - mr. utterson

Text 6 (book):
    - gregor 's
    - of the
    - he had
    - it was
    - in the

The keyphrases do seem to be a bit more informative. The first article has a pretty big give-away with the term “Harry Potter”, suggesting it’s about entertainment. The second article seems to be related to mental health with a phrase like “mentally ill” being so common. The term “learning rates” is certainly interesting in the third text—maybe it’s about machine learning or math? The fourth text has zero useful clues, but the two books appear to reveal the names of characters. But who are Mr. Utterson and Gregor? Surely there has to be a better way!

TF-IDF Keywords

The most common statistical method to select keywords from documents is known as term frequency-inverse document frequency (TF-IDF). The advantage of the TF-IDF model over the bag-of-words approach is that TF-IDF uses a weighting factor to give greater weight to words that appear frequently in one document, but rarely in other documents. As a result, TF-IDF assigns very low weights to stopwords like “the” or “and” because they tend to appear very frequently in all documents, whereas unique words such as scientific terms or character names may only appear in a few documents and thus have greater weight.

As with our TDM, the rows in a TF-IDF matrix are documents and the columns are words. However, in the TDM the values were word frequencies, and in the TF-IDF matrix the values will be the term frequency (TF) multiplied by the inverted document frequency (IDF). We can calculate $TF$ with the below formula, where $t$ is the term, $d$ is the document, and $|d|$ is the total number of terms in the document, and $f_{t,d}$ is the raw count of term $t$ in $d$ .

\text{TF}(t, d) = \frac{f_{t,d}}{|d|}

We then calculate $IDF$ (the logarithm of the inverse fraction of documents that contain a word) where $D$ is the corpus (set of documents), and $|D|$ is the total number of documents.

\text{IDF}(t, D) = \log \left( \frac{|D|}{|\{d \in D: t \in d\}|} \right)

Together, this gives us the final formula for TF-IDF:

\text{TF-IDF}(t, d, D) = \text{TF}(t, d) \cdot \text{IDF}(t, D)

Don’t worry too much about the math, the main point is that words that are common across the whole corpus are down-weighted, and words that only appear in one or a few documents are up-weighted. We can calculate this in code using our existing TDMs, as shown below.

1
def calculate_tfidf(tdm: np.ndarray) -> np.ndarray:
2
    """
3
    Calculate TF-IDF matrix from a term-document matrix.
4

5
    Args:
6
        tdm (np.ndarray): Term-document matrix
7

8
    Returns:
9
        np.ndarray: TF-IDF values
10
    """
11
    n_documents, n_terms = tdm.shape
12
    tfidf = np.zeros_like(tdm, dtype=float)
13

14
    for term_idx in range(n_terms):
15
        docs_with_term = sum(1 for doc_freq in tdm[:, term_idx] if doc_freq > 0)
16
        idf = np.log(n_documents / (docs_with_term + 1))
17
        tfidf[:, term_idx] = tdm[:, term_idx] * idf
18

19
    return tfidf
20

21
tfidf = calculate_tfidf(tdm)
22
tfidf

array([[  0.69314718,   3.29583687,   1.09861229, ...,   0.        ,
            0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
            0.        ,   0.        ],
        [ 13.16979643,   0.        ,   0.        , ...,   0.        ,
            0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   0.        ,
            0.        ,   0.        ],
        [  0.        ,   0.        ,   0.        , ...,   1.38629436,
            97.04060528,   2.07944154],
        [  0.        ,   0.        ,   0.        , ...,   7.62461899,
        134.47055303,   6.23832463]], shape=(6, 6374))

Note that our TF-IDF matrix has the same shape as the TDM: one row per document in our corpus and one column per word in the shared vocabulary. However, unlike the TDM, the values are decimal numbers, with each representing the TF-IDF score for that term/document combination. Let’s see what keywords we get using the TF-IDF matrix.

1
tfidf_keywords = get_top_keywords(tfidf, vocab, n_keywords=5)
2

3
print("Top keywords by TF-IDF:")
4
print_keywords_lists(tfidf_keywords, texts)

Top keywords by TF-IDF:
Text 1 (news): potter, radcliffe, reuters, 'll, harry
Text 2 (news): leifman, mentally, 's, miami, jail
Text 3 (science): additive, kernel, quantile, @xcite, assumption
Text 4 (science): tag, @xcite, mass, st, mev
Text 5 (book): utterson, my, i, hyde, jekyll
Text 6 (book): gregor, she, 's, sister, he

Much more informative! Due to the down-weighting of stopwords and other very common terms, our TF-IDF keywords are significantly more interesting. We now see what appear to be proper nouns in texts 1, 5, and 6, and some unusual words like “jail” in text 2 or “kernel” and “quantile” in text 3. We even get some big give-aways like the terms “hyde” and “jekyll” in one of our books. Let’s see what we get using the same approach with ngrams.

1
ngram_tfidf = calculate_tfidf(ngram_tdm)
2

3
ngram_tfidf_keywords = get_top_keywords(ngram_tfidf, ngram_vocab, n_keywords=5)
4

5
print("Top keyphrases by TF-IDF:")
6
print_keywords_lists(ngram_tfidf_keywords, texts, True)

Top keyphrases by TF-IDF:
Text 1 (news):
    - harry potter
    - this year
    - this year he
    - potter and the order
    - potter and the

Text 2 (news):
    - leifman says
    - mentally ill
    - it 's
    - the forgotten floor
    - the forgotten

Text 3 (science):
    - learning rates
    - @xmath159 quantile
    - quantile regression
    - loss function
    - given by

Text 4 (science):
    - et al
    - the tag
    - the leptonic
    - the st
    - leptonic decay

Text 5 (book):
    - mr. utterson
    - the lawyer
    - of my
    - i was
    - it was

Text 6 (book):
    - gregor 's
    - his sister
    - his father
    - he had
    - he was

Looking at the keywords and keyphrases from TF-IDF, it’s increasingly clear what these documents are about. Text 1 seems to be about Harry Potter and the Order of the Phoenix, text 2 is still a bit unclear but seems to be about mental health, text 3 looks like it’s about machine learning with terms like “loss function” and “learning rates”, text 4 seems to be about physics or astronomy, and for the books we have some solid hints with character names.

Now that we’ve seen a few statistical approaches at work, let’s take a look at something slightly different: a graph-based approach.

Co-occurence Graph

We won’t spend too much time on this one, but it’s worth briefly exploring graph-based approaches to keyword extraction, if only to see that this is a hard problem with many different solutions. Imagine that each unique word in a document is a node on a graph, and the edges of the graph count the number of times that two words “co-occur”—that is, appear within a few words of one another. If we build a graph like this, we can then use a graph algorithm like PageRank (the same algorithm used by Google to rank webpage search results) to see which nodes in the graph are most important to the overall network. The top nodes would be our keywords. Let’s see it in action.

1
import networkx as nx
2
from collections import Counter
3

4
def build_cooccurrence_graph(tokens: list[str], window_size: int = 3) -> nx.Graph:
5
    """
6
    Build a graph of word co-occurrences from a list of tokens.
7

8
    Args:
9
        tokens (list[str]): The list of tokens
10
        window_size (int): The size of the co-occurrence window. Defaults to 3.
11

12
    Returns:
13
        nx.Graph: A graph where nodes are words and edges represent co-occurrences
14
    """
15
    co_occurrences = Counter()
16
    for i in range(len(tokens)):
17
        for j in range(i + 1, min(i + window_size, len(tokens))):
18
            pair = tuple(sorted([tokens[i], tokens[j]]))
19
            co_occurrences[pair] += 1
20

21
    G = nx.Graph()
22
    for (w1, w2), weight in co_occurrences.items():
23
        G.add_edge(w1, w2, weight=weight)
24

25
    return G
26

27
def get_keywords_from_graph(G: nx.Graph, n: int = 5) -> list[str]:
28
    """
29
    Extract keywords from a word co-occurrence graph using PageRank.
30

31
    Args:
32
        G (nx.Graph): The word co-occurrence graph
33
        n (int): The number of top keywords to return. Defaults to 5.
34

35
    Returns:
36
        list[str]: A list of the top keywords
37
    """
38
    pagerank_scores = nx.pagerank(G, weight='weight')
39
    keywords = sorted(pagerank_scores, key=pagerank_scores.get, reverse=True)[:n]
40

41
    return keywords
42

43
graph_keywords = []
44
for text in texts:
45
    g = build_cooccurrence_graph(text["tokens"], window_size=3)
46
    keywords = get_keywords_from_graph(g, n=5)
47
    graph_keywords.append(keywords)
48

49
print("Top keywords by graph-based keyword extraction:")
50
print_keywords_lists(graph_keywords, texts)
51
print("")
52

53
ngram_graph_keywords = []
54
for text in texts:
55
    g = build_cooccurrence_graph(text["ngrams"], window_size=3)
56
    keywords = get_keywords_from_graph(g, n=5)
57
    ngram_graph_keywords.append(keywords)
58

59
print("Top keyphrases by graph-based keyword extraction:")
60
print_keywords_lists(ngram_graph_keywords, texts)

Top keywords by graph-based keyword extraction:
Text 1 (news): the, he, his, a, to
Text 2 (news): the, in, to, and, they
Text 3 (science): the, of, is, and, a
Text 4 (science): the, of, and, is, in
Text 5 (book): the, and, of, a, to
Text 6 (book): the, to, and, he, his

Top keyphrases by graph-based keyword extraction:
Text 1 (news): harry potter, in a, he told, of the, actor says he
Text 2 (news): leifman says, it 's, mentally ill, on the, the forgotten floor
Text 3 (science): of the, in the, learning rates, for the, for some
Text 4 (science): of the, is the, in the, and the, the tag
Text 5 (book): of the, in the, it was, and the, of a
Text 6 (book): gregor 's, of the, he had, it was, in the

As with our TDM, the keywords we get are all stopwords, though the keyphrases give us a bit more information. This is not surprising, given that our rankings are based on edge weight, and as the most frequent words, stopwords are likely to have the most co-occurrences and thus the highest weight. (In fairness, similar to the TDM, we really should remove stopwords in a real example.) However, co-occurrence graphs have a distinct advantage over TDMs, because we can plot them!

1
import matplotlib.pyplot as plt
2

3
def visualize_word_network(
4
    G: nx.Graph, figsize: tuple[int, int] = (20, 16), min_edge_weight: int = 2, seed: int = 42
5
):
6
    """
7
    Visualize a word co-occurrence network.
8

9
    Args:
10
        G (networkx.Graph): Graph of word co-occurrences
11
        figsize (tuple): Figure size (width, height)
12
        min_edge_weight (int): Minimum edge weight to display
13
        seed (int): Random seed for layout initialization
14
    """
15
    # filter edges and nodes based on weight
16
    edges_to_remove = [(u, v) for u, v, w in G.edges(data='weight') if w < min_edge_weight]
17
    G_filtered = G.copy()
18
    G_filtered.remove_edges_from(edges_to_remove)
19
    G_filtered.remove_nodes_from(list(nx.isolates(G_filtered)))
20

21
    if len(G_filtered) == 0:
22
        print("No nodes to display. Adjust the minimum edge weight.")
23
        return
24

25
    # visualize the network
26
    pos = nx.spring_layout(G_filtered, k=2, iterations=50, seed=seed)
27
    plt.figure(figsize=figsize)
28
    nx.draw_networkx_nodes(G_filtered, pos,
29
                          node_color='lightblue',
30
                          node_size=3000,
31
                          alpha=0.7)
32

33
    edge_weights = [G_filtered[u][v]['weight'] for u,v in G_filtered.edges()]
34
    max_weight = max(edge_weights)
35
    scaled_weights = [3*w/max_weight for w in edge_weights]
36

37
    nx.draw_networkx_edges(G_filtered, pos,
38
                          width=scaled_weights,
39
                          alpha=0.4)
40

41
    pos_attrs = {}
42
    for node, coords in pos.items():
43
        pos_attrs[node] = (coords[0], coords[1] + 0.02)
44

45
    nx.draw_networkx_labels(G_filtered, pos_attrs, font_size=14)
46

47
    plt.title("Word Co-occurrence Network", fontsize=20, pad=20)
48
    plt.axis('off')
49
    plt.tight_layout()
50
    plt.show()
51

52
visualize_word_network(
53
    build_cooccurrence_graph(texts[2]["tokens"], window_size=3),
54
    figsize=(16, 12),
55
    min_edge_weight=12
56
)

Graph of keyword co-occurrences.

In the above example, we are plotting all of the nodes from text 3 that have edges with a weight of at least 12. These are the same words that would have appeared if we had set the number of keywords much higher, so we’re cheating a bit here to learn more about text 3 than we did before, but the visualization is still useful to see which words are most common, and how they relate to one another.

Although we’ve made some interesting discoveries thus far, all of our approaches have some significant downsides. The TDM and graph-based approaches are interesting, but without filtering stopwords they don’t actually reveal much. The TF-IDF approach is more revealing, but it relies on processing a full corpus of documents and comparing them, rather that acting on a single document on its own. To improve our keyword extraction, we’ll now explore a few approaches that will hopefully produce more interesting results without requiring filtering or multiple documents.

Entropy to the Rescue!

In a [previous article](/blog/understanding-entropy TODO) we investigated how entropy can help us understand the information content in a particular text. It stands to reason that the words or phrases that contribute the most entropy to a text might also be useful keywords. Even better, this is something we can calculate for words in a single document, using a pre-calculated estimate of the probability distribution of words in a particular language, meaning we don’t have to rely on comparing documents across a full corpus. Fortunately, spaCy has lookup tables that we can use to get the probability of a given token (independent of surrounding tokens), and then use to calculate information content and entropy contributions. Let’s take a look at what keywords this gives us on our texts.

1
from spacy.lookups import load_lookups
2

3
def get_spacy_probability(token: str, nlp: spacy.Language) -> float:
4
    """
5
    Get the probability of a word in the spaCy language model.
6

7
    Args:
8
        token (string): The word to get the probability for
9
        nlp (spacy.Language): The spaCy language model
10

11
    Returns:
12
        The probability of the word (float)
13
    """
14
    _ = nlp.vocab[token]
15
    ln_prob = nlp.vocab[token].prob
16
    prob = np.exp(ln_prob)
17

18
    return prob
19

20
def word_entropy_df(tokens: list[str]) -> dict:
21
    # load the spaCy language model with lookups
22
    nlp = spacy.load('en_core_web_sm', disable=['parser', 'ner', 'lemmatizer', 'tagger'])
23
    lookups = load_lookups("en", ["lexeme_prob"])
24
    nlp.vocab.lookups.add_table("lexeme_prob", lookups.get_table("lexeme_prob"))
25

26
    # calculate entropy measures
27
    measures = []
28
    for token in tokens:
29
        joint_probability = 1
30
        subtokens = token.split(" ")
31

32
        # if token is an ngram, split and use joint probability
33
        # assume independence between subtokens, even though
34
        # this is not accurate in the real world
35
        for subtoken in subtokens:
36
            prob = get_spacy_probability(subtoken, nlp)
37
            joint_probability *= prob
38

39
        information_content = -np.log2(joint_probability)
40
        entropy_contribution = joint_probability * information_content
41
        measures.append({
42
            "token": token,
43
            "joint_probability": joint_probability,
44
            "information_content": information_content,
45
            "entropy_contribution": entropy_contribution
46
        })
47

48
    return pd.DataFrame(measures)
49

50
entropy_keywords = []
51
entropy_keyphrases = []
52

53
for text in tqdm(texts):
54
    word_df = word_entropy_df(text["tokens"])
55

56
    keywords = word_df.sort_values(
57
        "entropy_contribution", ascending=True
58
    )["token"].drop_duplicates().tolist()[:5]
59
    entropy_keywords.append(keywords)
60

61
    ngram_df = word_entropy_df(text["ngrams"])
62
    keyphrases = ngram_df.sort_values(
63
        "entropy_contribution", ascending=True
64
    )["token"].drop_duplicates().tolist()[:5]
65
    entropy_keyphrases.append(keyphrases)
66

67
print("Top keywords by entropy contribution:")
68
print_keywords_lists(entropy_keywords, texts)
69
print("")
70

71
print("Top keyphrases by entropy contribution:")
72
print_keywords_lists(entropy_keyphrases, texts, True)

Top keywords by entropy contribution:
Text 1 (news): rudyard, shaffer, kipling, equus, radcliffe
Text 2 (news): leifman, --charges, soledad, o'brien, pretrial
Text 3 (science): @xmath0, rkhs, @xmath18, @xmath17, @xmath22
Text 4 (science): @xmath55, @xmath2, ikado, @xmath33, lqcd
Text 5 (book): mr., respectors, utterson, blackguardly, amities
Text 6 (book): wyllie, charlottenstrasse, mr., mrs., family”—here

Top keyphrases by entropy contribution:
Text 1 (news):
    - potter star daniel radcliffe
    - london england reuters harry
    - england reuters harry potter
    - peter shaffer 's equus
    - shaffer 's equus meanwhile

Text 2 (news):
    - miami dade pretrial detention
    - dade pretrial detention facility
    - miami dade county jails
    - soledad o'brien takes users
    - judge steven leifman says

Text 3 (science):
    - \mbox{~~and~~ q\bigl([t \infty)\bigr \ge
    - q\bigl([t \infty)\bigr \ge 1-\tau\bigr\}\,.\
    - @xcite @xcite @xcite @xcite
    - \ge \tau \mbox{~~and~~ q\bigl([t
    - @xmath168\bigr \ge \tau \mbox{~~and~~

Text 4 (science):
    - @xcite @xmath156 gev@xmath157 @xmath158
    - @xmath122 @xmath123 @xmath124 @xmath125
    - @xmath121 @xmath122 @xmath123 @xmath124
    - @xmath123 @xmath124 @xmath125 @xmath126
    - @xmath156 gev@xmath157 @xmath158 mev

Text 5 (book):
    - dr. jekyll's mr. utterson
    - m.d. d.c.l. l.l.d. f.r.s.
    - jekyll m.d. d.c.l. l.l.d.
    - carew hastie lanyon henry
    - mr. enfield dr. jekyll

Text 6 (book):
    - mr. samsa let 's
    - tearfully thanking gregor 's
    - collar mr. samsa shouted
    - of excusal mr. samsa
    - mr. and mrs. samsa

Using entropy contribution as our measure for keywords we get some very interesting results, even without having to analyze an entire corpus or filter stopwords! The results are arguably as useful as those from the TF-IDF method, insofar as they reveal proper nouns like names of characters. Amusingly, in the scientific articles, the keywords all appear to be drawn from LaTeX formulas, which isn’t particularly useful, but it tells us at least that these are scientific documents.

So far we have primarily been using statistical measures to identify keywords. But that’s not the only approach. As we continue our exploration, let’s look at the final two approaches: algorithmic, and modeling.

Raking Up the Keywords

When it comes to algorithmic approaches to keywords extraction, there are a ton of methods—far too many to look at here. So, we’ll just take a quick look at one of the more famous methods: Rapid Automatic Keyword Extraction (RAKE). This approach was developed by Rose et al. (2010) in their paper Automatic Keyword Extraction from Individual Documents. The algorithm is fairly complex, and worthy of an entire article on its own, but it boils down to the following basic steps:

Split the text into “candidate phrases” with delimiters at stopwords and punctuation. So, the text "I am studying machine learning and statistics. I think statistics is really interesting." splits at stopwords like "I", "am", "is", and at punctuation like "." to become candidate phrases ['studying machine learning', 'statistics', 'think statistics', 'interesting'].
Build a co-occurrence graph of words in candidate phrases, where words are nodes and edges go between any words that appear in the same candidate phrases. In the above example, this would link “studying” to both “machine” and “learning”.
Compute word scores by calculating the degree of each word (number of connections), frequency (number of appearances in the text), and the degree-frequency ratio.
Compute phrase scores as the sum of the word scores in the phrase.
Rank and extract keywords.

As you might have guessed from the algorithm steps, RAKE is focused on producing keyphrases only, because the weighting scheme requires looking at how many phrases a word appears in. For this reason, we can’t extract keywords alone. Now that we know the algorithm, we can implement it in Python, using spaCy documents to help us identify punctuation and stopwords to use during candidate phrase construction.

1
def get_rake_phrases(doc: spacy.language.Doc) -> list[list[str]]:
2
    """
3
    Extract candidate phrases from a spaCy Doc using the RAKE algorithm.
4

5
    Args:
6
        doc (spacy.language.Doc): The input spaCy Doc
7

8
    Returns:
9
        list[list[str]]: A list of candidate phrases consisting
10
            of lists of tokens per phrase
11
    """
12
    phrases = []
13
    current_phrase = []
14
    for token in doc:
15
        if token.is_stop or token.is_punct or token.is_space:
16
            if current_phrase:
17
                phrases.append(current_phrase)
18
                current_phrase = []
19
        else:
20
            current_phrase.append(token.text.lower())
21

22
    if current_phrase:
23
        phrases.append(current_phrase)
24

25
    return phrases
26

27
def get_rake_word_score(phrases):
28
    """
29
    Calculate word scores using RAKE algorithm.
30

31
    Args:
32
        phrases (list[list[str]]): List of phrases
33

34
    Returns:
35
        dict: Dictionary of word scores
36
    """
37
    word_freq = Counter()
38
    word_degree = {}
39

40
    for phrase in phrases:
41
        for word in phrase:
42
            word_freq[word] += 1
43
            word_degree[word] = word_degree.get(word, 0) + len(phrase)
44

45
    word_scores = {}
46
    for word in word_freq:
47
        word_scores[word] = word_degree[word] / word_freq[word]
48

49
    return word_scores
50

51
def get_rake_phrase_scores(phrases, word_scores):
52
    """
53
    Calculate phrase scores using RAKE algorithm.
54

55
    Args:
56
        phrases (list[list[str]]): List of phrases
57
        word_scores (dict): Dictionary of word scores
58

59
    Returns:
60
        dict: Dictionary of phrase scores
61
    """
62
    phrase_scores = {}
63
    for phrase in phrases:
64
        phrase_scores[" ".join(phrase)] = sum(word_scores[word] for word in phrase)
65

66
    return phrase_scores
67

68
rake_keyphrases = []
69

70
for text in tqdm(texts):
71
    phrases = get_rake_phrases(text["doc"])
72
    word_scores = get_rake_word_score(phrases)
73
    phrase_scores = get_rake_phrase_scores(phrases, word_scores)
74
    top_phrases = sorted(phrase_scores, key=phrase_scores.get, reverse=True)[:5]
75
    rake_keyphrases.append(top_phrases)
76

77
print("Top keyphrases by RAKE:\n")
78
print_keywords_lists(rake_keyphrases, texts, True)

Top keyphrases by RAKE:

Text 1 (news):
    - harry potter star daniel radcliffe gains access
    - massive sports car collection
    - uk box office chart
    - reported £ 20 million
    - kid star goes

Text 2 (news):
    - soledad o'brien takes users inside
    - leifman says 200 years ago people
    - ninth floor severely mentally disturbed
    - heavy wool sleeping bag
    - dade pretrial detention facility

Text 3 (science):
    - uniformly lipschitz continuous satisfying @xmath14
    - shifted loss function @xmath17 introduced
    - integral operator @xmath117 satisfies @xmath242
    - standard gaussian rbf kernel @xmath63
    - reproducing kernel hilbert space @xmath0

Text 4 (science):
    - @xmath40 @xmath41 containing @xmath42 @xmath3 pairs
    - degree chebyshev polynomial background function
    - layer stereo wire drift chamber
    - gaussian signal function plus second
    - rich detector covers approximately @xmath52

Text 5 (book):
    - disordered sensual images running like
    - good qualities seemingly unimpaired
    - fast drawers stood open
    - circulation like mr. enfield
    - haired old woman opened

Text 6 (book):
    - grey earth mingled inseparably
    - electric street lamps shone
    - leaving vile brown flecks
    - mr. samsa twisted round
    - visit gregor relatively soon

As you can see, RAKE produces very different results from our previous keyphrase extractions. In the news articles, we now get a much better sense of the content. For example, the first article doesn’t just seem to be about Harry Potter the franchise, but about the star actor’s income and sports car collection. In the books however, these phrases actually seem to tell us less than we knew before. The phrases are poetic and descriptive, but they don’t really seem to tell us much about the content of the books (though some characters are named). As with any keyphrase detection method, RAKE has its limitations, especially in creative works where the very definition of a “keyphrase” is up for debate.

KeyBERT

In our final section, we’ll take a look at the last major category of keyword/phrase extractors: machine learning models. The advantage of using machine learning models is that it gives us the option of creating and training a model to recognize what we consider to be a keyphrase. Given a sufficient amount of annotated training data, you could create a custom model for keywords.

In our example, we’re going to take a slightly different approach though and leverage the power of [word embeddings](blog/how-does-a-computer-learn-to-read TODO). The basic idea is that we can identify keywords or phrases by creating a word embedding for each candidate and a document embedding for the text as a whole, and then finding the candidates that have the closest cosine similarity to the overall document embedding. The intuition here is that words with the closest similarity to the document are likely to be the most representative of the document.

To implement this approach, we’ll use an adapted version of code developed by Maarten Grootendorst, who also developed the excellent KeyBERT library.

1
from sentence_transformers import SentenceTransformer
2
from typing import Callable, Optional
3
from sklearn.metrics.pairwise import cosine_similarity
4
import itertools
5

6
def top_candidates(
7
    candidates: list[str], similarities: np.ndarray, n: int = 10, repeat: bool = False
8
) -> list[str]:
9
    """
10
    Gets the top n keywords based on similarity scores.
11

12
    Args:
13
        candidates (List[str]): List of candidate words.
14
        similarities (np.ndarray): Cosine similarity scores.
15
        n (int): Number of keywords to return.
16
        repeat (bool): Whether to allow duplicate words.
17

18
    Returns:
19
        List[str]: The top n keywords.
20
    """
21
    top_indices = np.argsort(similarities[0])[::-1]  # Sort in descending order
22
    keywords = []
23

24
    if not repeat:
25
        for i in top_indices:
26
            keyword = candidates[i]
27
            if keyword not in keywords:
28
                keywords.append(keyword)
29
            if len(keywords) == n:
30
                break
31
    else:
32
        keywords = [candidates[i] for i in top_indices[:n]]
33

34
    return keywords
35

36
def max_sum_sim(
37
    doc_embedding: np.ndarray,
38
    word_embeddings: np.ndarray,
39
    words: list[str],
40
    top_n: int,
41
    nr_candidates: int,
42
) -> list[str]:
43
    """
44
    Selects top_n words that are least similar to each other from nr_candidates.
45

46
    Args:
47
        doc_embedding (np.ndarray): Embedding of the full document.
48
        word_embeddings (np.ndarray): Embeddings of the candidate words.
49
        words (List[str]): List of candidate words.
50
        top_n (int): Number of keywords to return.
51
        nr_candidates (int): Number of initial candidates to consider.
52

53
    Returns:
54
        List[str]: The selected keywords.
55
    """
56
    distances = cosine_similarity(doc_embedding, word_embeddings)
57
    distances_candidates = cosine_similarity(word_embeddings, word_embeddings)
58

59
    words_idx = list(distances.argsort()[0][-nr_candidates:])
60
    words_vals = [words[index] for index in words_idx]
61
    distances_candidates = distances_candidates[np.ix_(words_idx, words_idx)]
62

63
    min_sim = np.inf
64
    candidate = None
65
    for combination in itertools.combinations(range(len(words_idx)), top_n):
66
        sim = sum(
67
            distances_candidates[i][j]
68
            for i in combination
69
            for j in combination
70
            if i != j
71
        )
72
        if sim < min_sim:
73
            candidate = combination
74
            min_sim = sim
75

76
    return [words_vals[idx] for idx in candidate]
77

78
def mmr(
79
    doc_embedding: np.ndarray,
80
    word_embeddings: np.ndarray,
81
    words: list[str],
82
    top_n: int,
83
    diversity: float,
84
) -> list[str]:
85
    """
86
    Uses Maximal Marginal Relevance (MMR) to select diverse keywords.
87

88
    Args:
89
        doc_embedding (np.ndarray): Embedding of the full document.
90
        word_embeddings (np.ndarray): Embeddings of the candidate words.
91
        words (List[str]): List of candidate words.
92
        top_n (int): Number of keywords to return.
93
        diversity (float): Controls balance between relevance and diversity.
94

95
    Returns:
96
        List[str]: The selected keywords.
97
    """
98
    word_doc_similarity = cosine_similarity(word_embeddings, doc_embedding)
99
    word_similarity = cosine_similarity(word_embeddings)
100

101
    keywords_idx = [np.argmax(word_doc_similarity)]
102
    candidates_idx = [i for i in range(len(words)) if i != keywords_idx[0]]
103

104
    for _ in range(top_n - 1):
105
        candidate_similarities = word_doc_similarity[candidates_idx, :]
106
        target_similarities = np.max(
107
            word_similarity[candidates_idx][:, keywords_idx], axis=1
108
        )
109

110
        mmr = (1 - diversity) * candidate_similarities - diversity * target_similarities.reshape(
111
            -1, 1
112
        )
113
        mmr_idx = candidates_idx[np.argmax(mmr)]
114

115
        keywords_idx.append(mmr_idx)
116
        candidates_idx.remove(mmr_idx)
117

118
    return [words[idx] for idx in keywords_idx]
119

120
def get_bert_keywords(
121
    model: SentenceTransformer,
122
    text: str,
123
    tokens: list[str],
124
    n: int = 10,
125
    repeat: bool = False,
126
    diversification_fn: Optional[Callable] = None,
127
    diversification_params: Optional[dict] = None,
128
) -> list[str]:
129
    """
130
    Extracts keywords from a text using a BERT sentence embedding model.
131

132
    Args:
133
        model (SentenceTransformer): A SentenceTransformer model.
134
        text (str): The input text.
135
        tokens (list[str]): List of candidate words.
136
        n (int): Number of keywords to return.
137
        repeat (bool): Whether to allow duplicate words.
138
        diversification_fn (Optional[Callable]): A diversification function.
139
        diversification_params (Optional[dict]): Parameters for the diversification function.
140

141
    Returns:
142
        List[str]: The top n keywords.
143
    """
144
    candidates_embeddings = model.encode(tokens)
145
    text_embedding = model.encode([text])
146

147
    similarities = cosine_similarity(text_embedding, candidates_embeddings)
148

149
    if diversification_fn:
150
        if not diversification_params:
151
            raise ValueError("Diversification function requires parameters.")
152
        return diversification_fn(
153
            text_embedding, candidates_embeddings, tokens, n, **diversification_params
154
        )
155

156
    return top_candidates(tokens, similarities, n, repeat)
157

158
model = SentenceTransformer('distilbert-base-nli-mean-tokens')
159

160
bert_keywords = []
161
bert_keyphrases = []
162

163
for text in tqdm(texts):
164
    keywords = get_bert_keywords(
165
        model, text["text"], text["tokens"], n=5, repeat=False
166
    )
167
    bert_keywords.append(keywords)
168

169
    keyphrases = get_bert_keywords(
170
        model, text["text"], text["ngrams"], n=5, repeat=False
171
    )
172
    bert_keyphrases.append(keyphrases)
173

174
print("Top keywords by BERT:")
175
print_keywords_lists(bert_keywords, texts)
176
print("")
177

178
print("Top keyphrases by BERT:")
179
print_keywords_lists(bert_keyphrases, texts, True)

Top keywords by BERT:
Text 1 (news): monday, dvds, fortune, birthday, gossip
Text 2 (news): jails, cnn, inmates, jail, prisoners
Text 3 (science): advantages, popularity, @xmath168\bigr, satisfied, improved
Text 4 (science): physics, neutrinos, quark, fermi, quarks
Text 5 (book): jekyll”—he, jekyll's, murder, satan, funeral
Text 6 (book): sleepiness, sleep, slept, bed, nap

Top keyphrases by BERT:
Text 1 (news):
    - $ 41.1 million fortune
    - harry potter star daniel
    - reuters harry potter star
    - his number one movie
    - first five potter films

Text 2 (news):
    - miami dade pretrial detention
    - county jails are mentally
    - many mentally ill inmates
    - jails are mentally ill
    - miami dade county jails

Text 3 (science):
    - good estimators in additive
    - theorem quantilethm is better
    - advantages of additive models
    - many nice mathematical properties
    - additive models which helps

Text 4 (science):
    - leptonic decay @xmath71 without
    - leptonic decay efficiency
    - leptonic decay efficiency hence
    - decays to lighter leptons
    - the leptonic decay efficiency

Text 5 (book):
    - dr. jekyll looking deathly
    - dr. jekyll 's disappearance
    - of henry jekyll dead
    - henry jekyll dead
    - of dr. jekyll's mr.

Text 6 (book):
    - excessive sleepiness after sleeping
    - his room crawling over
    - pain in bed
    - bed into a horrible
    - excessive sleepiness

The results of the BERT approach are pretty compelling, particularly with the keyphrases. Performance on the books is better—the topic of the first book in particular now seems quite obvious—but as before, selecting keyphrases becomes harder and harder as texts get longer. We do have a small issue though, particularly as seen in text 4 where the keyphrases are all very similar. As Grootendorst points out in his blog post on this topic, we need a way to diversify our selections. For this, he recommends max sum similarity or maximal marginal relevance. For the sake of brevity, I’ll leave that to you to try out, but the code is included above, and Grootendorst does an excellent job of explaining how these diversification functions work.

Final Thoughts

Over the course of this article we’ve seen that keyword and keyphrase extraction is a task with many approaches and each of them has their upsides and downsides. In practice, it’s probably best to try a few different methods to see what works for a particular type of document. In some cases, a simple frequency-based approach might actually be best (I recommend you filter out stopwords though—we didn’t do that here so that we could compare methods more effectively), and in others something more complex may be required.

OK, before we go, let’s print out our results next to each other so you can compare. Which do you think is best?

1
keyword_results = {
2
    "Frequency": tdm_keywords,
3
    "TF-IDF": tfidf_keywords,
4
    "Graph": graph_keywords,
5
    "Entropy": entropy_keywords,
6
    "BERT": bert_keywords,
7
}
8

9
print("Keyword Analysis Results")
10
for idx, text in enumerate(texts):
11
    print(f"\nText {idx + 1} ({text['category']}):")
12
    print("-" * 50)
13

14
    for method, results in keyword_results.items():
15
        keywords = results[idx]
16
        print(f"{method:25}: {', '.join(keywords)}")

Keyword Analysis Results

Text 1 (news):
--------------------------------------------------
Frequency                : the, he, to, of, a
TF-IDF                   : potter, radcliffe, reuters, 'll, harry
Graph                    : the, he, his, a, to
Entropy                  : rudyard, shaffer, kipling, equus, radcliffe
BERT                     : monday, dvds, fortune, birthday, gossip

Text 2 (news):
--------------------------------------------------
Frequency                : the, in, to, and, they
TF-IDF                   : leifman, mentally, 's, miami, jail
Graph                    : the, in, to, and, they
Entropy                  : leifman, --charges, soledad, o'brien, pretrial
BERT                     : jails, cnn, inmates, jail, prisoners

Text 3 (science):
--------------------------------------------------
Frequency                : the, of, is, and, a
TF-IDF                   : additive, kernel, quantile, @xcite, assumption
Graph                    : the, of, is, and, a
Entropy                  : @xmath0, rkhs, @xmath18, @xmath17, @xmath22
BERT                     : advantages, popularity, @xmath168\bigr, satisfied, improved

Text 4 (science):
--------------------------------------------------
Frequency                : the, of, and, in, is
TF-IDF                   : tag, @xcite, mass, st, mev
Graph                    : the, of, and, is, in
Entropy                  : @xmath55, @xmath2, ikado, @xmath33, lqcd
BERT                     : physics, neutrinos, quark, fermi, quarks

Text 5 (book):
--------------------------------------------------
Frequency                : the, and, of, to, i
TF-IDF                   : utterson, my, i, hyde, jekyll
Graph                    : the, and, of, a, to
Entropy                  : mr., respectors, utterson, blackguardly, amities
BERT                     : jekyll”—he, jekyll's, murder, satan, funeral

Text 6 (book):
--------------------------------------------------
Frequency                : the, to, and, he, his
TF-IDF                   : gregor, she, 's, sister, he
Graph                    : the, to, and, he, his
Entropy                  : wyllie, charlottenstrasse, mr., mrs., family”—here
BERT                     : sleepiness, sleep, slept, bed, nap

1
keyphrase_results = {
2
    "Frequency": ngram_keywords,
3
    "TF-IDF": ngram_tfidf_keywords,
4
    "Graph": ngram_graph_keywords,
5
    "Entropy": entropy_keyphrases,
6
    "RAKE": rake_keyphrases,
7
    "BERT": bert_keyphrases,
8
}
9

10
print("Keyphrase Analysis Results")
11
for idx, text in enumerate(texts):
12
    print(f"\nText {idx + 1} ({text['category']}):")
13
    print("-" * 50)
14

15
    for method, results in keyphrase_results.items():
16
        keywords = results[idx]
17
        print(f"{method:25}: {', '.join(keywords)}")

Keyphrase Analysis Results

Text 1 (news):
--------------------------------------------------
Frequency                : harry potter, he told, in a, of the, he has
TF-IDF                   : harry potter, this year, this year he, potter and the order, potter and the
Graph                    : harry potter, in a, he told, of the, actor says he
Entropy                  : potter star daniel radcliffe, london england reuters harry, england reuters harry potter, peter shaffer 's equus, shaffer 's equus meanwhile
RAKE                     : harry potter star daniel radcliffe gains access, massive sports car collection, uk box office chart, reported £ 20 million, kid star goes
BERT                     : $ 41.1 million fortune, harry potter star daniel, reuters harry potter star, his number one movie, first five potter films

Text 2 (news):
--------------------------------------------------
Frequency                : mentally ill, it 's, leifman says, on the, forgotten floor
TF-IDF                   : leifman says, mentally ill, it 's, the forgotten floor, the forgotten
Graph                    : leifman says, it 's, mentally ill, on the, the forgotten floor
Entropy                  : miami dade pretrial detention, dade pretrial detention facility, miami dade county jails, soledad o'brien takes users, judge steven leifman says
RAKE                     : soledad o'brien takes users inside, leifman says 200 years ago people, ninth floor severely mentally disturbed, heavy wool sleeping bag, dade pretrial detention facility
BERT                     : miami dade pretrial detention, county jails are mentally, many mentally ill inmates, jails are mentally ill, miami dade county jails

Text 3 (science):
--------------------------------------------------
Frequency                : of the, in the, learning rates, @xmath159 quantile, for the
TF-IDF                   : learning rates, @xmath159 quantile, quantile regression, loss function, given by
Graph                    : of the, in the, learning rates, for the, for some
Entropy                  : \mbox{~~and~~ q\bigl([t \infty)\bigr \ge, q\bigl([t \infty)\bigr \ge 1-\tau\bigr\}\,.\, @xcite @xcite @xcite @xcite, \ge \tau \mbox{~~and~~ q\bigl([t, @xmath168\bigr \ge \tau \mbox{~~and~~
RAKE                     : uniformly lipschitz continuous satisfying @xmath14, shifted loss function @xmath17 introduced, integral operator @xmath117 satisfies @xmath242, standard gaussian rbf kernel @xmath63, reproducing kernel hilbert space @xmath0
BERT                     : good estimators in additive, theorem quantilethm is better, advantages of additive models, many nice mathematical properties, additive models which helps

Text 4 (science):
--------------------------------------------------
Frequency                : of the, in the, is the, and the, et al
TF-IDF                   : et al, the tag, the leptonic, the st, leptonic decay
Graph                    : of the, is the, in the, and the, the tag
Entropy                  : @xcite @xmath156 gev@xmath157 @xmath158, @xmath122 @xmath123 @xmath124 @xmath125, @xmath121 @xmath122 @xmath123 @xmath124, @xmath123 @xmath124 @xmath125 @xmath126, @xmath156 gev@xmath157 @xmath158 mev
RAKE                     : @xmath40 @xmath41 containing @xmath42 @xmath3 pairs, degree chebyshev polynomial background function, layer stereo wire drift chamber, gaussian signal function plus second, rich detector covers approximately @xmath52
BERT                     : leptonic decay @xmath71 without, leptonic decay efficiency, leptonic decay efficiency hence, decays to lighter leptons, the leptonic decay efficiency

Text 5 (book):
--------------------------------------------------
Frequency                : of the, in the, it was, and the, mr. utterson
TF-IDF                   : mr. utterson, the lawyer, of my, i was, it was
Graph                    : of the, in the, it was, and the, of a
Entropy                  : dr. jekyll's mr. utterson, m.d. d.c.l. l.l.d. f.r.s., jekyll m.d. d.c.l. l.l.d., carew hastie lanyon henry, mr. enfield dr. jekyll
RAKE                     : disordered sensual images running like, good qualities seemingly unimpaired, fast drawers stood open, circulation like mr. enfield, haired old woman opened
BERT                     : dr. jekyll looking deathly, dr. jekyll 's disappearance, of henry jekyll dead, henry jekyll dead, of dr. jekyll's mr.

Text 6 (book):
--------------------------------------------------
Frequency                : gregor 's, of the, he had, it was, in the
TF-IDF                   : gregor 's, his sister, his father, he had, he was
Graph                    : gregor 's, of the, he had, it was, in the
Entropy                  : mr. samsa let 's, tearfully thanking gregor 's, collar mr. samsa shouted, of excusal mr. samsa, mr. and mrs. samsa
RAKE                     : grey earth mingled inseparably, electric street lamps shone, leaving vile brown flecks, mr. samsa twisted round, visit gregor relatively soon
BERT                     : excessive sleepiness after sleeping, his room crawling over, pain in bed, bed into a horrible, excessive sleepiness

Oh, and in case you’re curious, the following are the texts we used! Were you able to guess correctly?

A news article about Harry Potter store Daniel Radcliffe turning 18 and gaining access to a $41.1 million fortune.
A news article with CNN correspondents going behind the scenes of a jail where inmates are mentally ill.
Learning rates for the risk of kernel based quantile regression estimators in additive models (link)
Improved Measurement of Absolute Branching Fraction of Ds to tau nu (link)
The Strange Case of Dr. Jekyll and Mr. Hyde by Robert Louis Stevenson (link)
Metamorphosis by Franz Kafka (link)