Lemmatize Tokens

Replace tokens with their lemmas. Two engines are available: a dependency-free "lookup" engine that applies a user-supplied lemma map, and a "spacy" engine that derives lemmas with the optional spacyr backend. In both cases lemmas are applied at the token-type level via quanteda::tokens_replace().

Usage

lemmatize_tokens(
  x,
  engine = c("lookup", "spacy"),
  lemma = NULL,
  ncores = 1,
  nchunks = ncores,
  socket = c("PSOCK", "FORK"),
  ...
)

Arguments

x

A quanteda::tokens object.

engine

Character. "lookup" (default) applies the lemma map; "spacy" derives lemmas using spacyr (requires a working spaCy installation and an initialized model; see parse_corpus()).

lemma

For engine = "lookup", the lemma map: either a named character vector (c(token = "lemma")) or a data.frame/data.table with columns token and lemma. Ignored for engine = "spacy".

ncores, nchunks, socket

Passed to parse_corpus() for the "spacy" engine's parsing step, which runs external spaCy worker processes (see parse_corpus() for their meaning). Ignored for engine = "lookup".

...

For engine = "spacy", additional arguments passed to parse_corpus() / spacyr::spacy_parse().

Value

A quanteda::tokens object with lemmatized tokens.

Details

Lemmatization here is type-level: each distinct token form is mapped to a single lemma and substituted with quanteda::tokens_replace(). The "spacy" engine builds that map from a spaCy parse, taking the most frequent lemma per token form, so it approximates fully context-sensitive lemmatization. For richly context-dependent analyses, parse the corpus directly with parse_corpus() and work from the returned annotations.

The parallel arguments (ncores, nchunks, socket) apply only to the "spacy" engine's parsing step.

See also

stem_tokens(), singularize_tokens(), parse_corpus()

Examples

corp <- quanteda::corpus(c(
  doc1 = "mice were running",
  doc2 = "the geese are flying"
))
toks <- tokenize_corpus(corp)
#> 
#> ── Tokenizing corpus ──
#> 
#> ℹ quanteda::tokens() has been called with default parameters
#> ✔ Corpus successfully tokenized

# Dependency-free lookup engine with a small custom map
lemma_map <- data.frame(
  token = c("mice", "were", "running", "geese", "are", "flying"),
  lemma = c("mouse", "be", "run", "goose", "be", "fly")
)
lemmatize_tokens(toks, engine = "lookup", lemma = lemma_map)
#> 
#> ── Lemmatizing tokens ──
#> 
#> ℹ Applying lookup lemma map (6 entries)
#> ✔ Lemmatization complete
#> Tokens consisting of 2 documents.
#> doc1 :
#> [1] "mouse" "be"    "run"  
#> 
#> doc2 :
#> [1] "the"   "goose" "be"    "fly"  
#>