Term frequency-inverse document frequency is a means of assigning weight to a search term when comparing individual documents within a corpus. It is an improvement on the bag-of-words model in that it considers the relative rarity of a term within a larger corpus.

• information retrieval
• linguistics
• nlp
• reference

The bag-of-words (BoW) model is used to represent a document as a fixed-length vector of frequency counts for each token in the document. BoW models are useful in classification, machine learning, and topic modeling tasks, among others.

• information retrieval
• linguistics
• nlp
• reference

In natural language processing, stemming is the process of reducing a word to its stem form. Typically, stemming is used as part of an NLP pipeline in order to reduce all words in a text to their stems so that they can be analyzed together.

• linguistics
• nlp
• nltk
• reference

A word stem is the form of a word that can accept affixes (suffixes and prefixes) and is thus common to all inflected variants.

• linguistics
• reference

A root word is a word with no prefixes or suffixes, meaning that it is the primary lexical unit of a set of words and represents the principle semantic meaning of the set.

• linguistics
• reference

Lemmatization is the process of reducing a word to its lemma (canonical form). In natural language processing, a lemmatizer may be used to reduce all words in a given text to their lemmas, which makes comparative analysis possible based on canonical forms.

• gensim
• linguistics
• nlp
• nltk
• reference
• spacy

A lemma is the canonical form of a word, also known as the dictionary form because it is often used as the primary entry in a dictionary.

• linguistics
• nlp
• reference