Stem Tokens

Reduce tokens to their stems using the Snowball stemmer. Stemming is applied once per unique token type in a single call to quanteda::char_wordstem() (SnowballC's C implementation) and mapped back onto the tokens with quanteda::tokens_replace(), so the cost scales with vocabulary size, not corpus size. Unlike singularize_tokens() (English plurals only), stemming supports many languages via Snowball.

Usage

stem_tokens(
  x,
  language = "english",
  ncores = NULL,
  nchunks = NULL,
  socket = NULL
)

Arguments

x

A quanteda::tokens object.

language

Character. Snowball stemmer language. Defaults to "english". See SnowballC::getStemLanguages() for the full list.

ncores, nchunks, socket

Deprecated since NLPstudio 1.2.0 and ignored: stemming is vocabulary-level C code, for which worker processes cost far more than they save. Supplying any of them raises a warning of class NLPstudio_deprecated.

Value

A quanteda::tokens object with stemmed tokens.

See also

singularize_tokens(), lemmatize_tokens()

Examples

corp <- quanteda::corpus(c(
  doc1 = "running runners ran easily",
  doc2 = "computational computers compute"
))
toks <- tokenize_corpus(corp)
#> 
#> ── Tokenizing corpus ──
#> 
#> ℹ quanteda::tokens() has been called with default parameters
#> ✔ Corpus successfully tokenized
stem_tokens(toks)
#> 
#> ── Stemming tokens ──
#> 
#> ℹ Extracting vocabulary
#> ℹ Stemming 7 types
#> ✔ Stemming complete
#> Tokens consisting of 2 documents.
#> doc1 :
#> [1] "run"    "runner" "ran"    "easili"
#> 
#> doc2 :
#> [1] "comput" "comput" "comput"
#>