Search relevance is a difficult problem in information retrieval. How do you ensure that you get the best results back from searching a collection of documents? Let's explore a few basic strategies, including simple searching, term-frequency searching, and TF-IDF searching.

• information retrieval
• nlp
• search

Term frequency-inverse document frequency is a means of assigning weight to a search term when comparing individual documents within a corpus. It is an improvement on the bag-of-words model in that it considers the relative rarity of a term within a larger corpus.

• information retrieval
• linguistics
• nlp
• reference

The bag-of-words (BoW) model is used to represent a document as a fixed-length vector of frequency counts for each token in the document. BoW models are useful in classification, machine learning, and topic modeling tasks, among others.

• information retrieval
• linguistics
• nlp
• reference

Natural language processing is a complex and evolving field. Part computer science, part linguistics, part statistics--it can be a challenge deciding where to begin. One starting place is to look at the most influential papers in academic literature--if you can master these papers, then you'll be well on the path to becoming an NLP expert.

• academia
• ai
• books
• nlp

One of the great things about using Python for natural language processing (NLP) is the large ecosystem of tools and libraries. From tokenization, to machine learning, to data visualization--Python has something for every NLP task in your workflow. Of course, choosing the right tool isn't always so easy.

• deep learning
• machine learning
• nlp
• python
• tools

In mid-August 2020, the Stanford NLP Group released version 1.1.1 of the Stanza Python library. Let's explore what Stanza can do, and while we're at it, see if we can learn something about the characters in a classic novel.

• literature
• nlp
• python
• stanza
• text analysis

In natural language processing, stemming is the process of reducing a word to its stem form. Typically, stemming is used as part of an NLP pipeline in order to reduce all words in a text to their stems so that they can be analyzed together.

• linguistics
• nlp
• nltk
• reference

Lemmatization is the process of reducing a word to its lemma (canonical form). In natural language processing, a lemmatizer may be used to reduce all words in a given text to their lemmas, which makes comparative analysis possible based on canonical forms.

• gensim
• linguistics
• nlp
• nltk
• reference
• spacy

A lemma is the canonical form of a word, also known as the dictionary form because it is often used as the primary entry in a dictionary.

• linguistics
• nlp
• reference