Skip to contents

Construct Token N-Grams

Source: R/ngram_tokens.R
ngram_tokens.Rd

Build n-grams (and optional skip-grams) from a quanteda quanteda::tokens object in a single call to quanteda::tokens_ngrams(), whose n-gram constructor runs in quanteda's multithreaded C++ core. On top of the quanteda core, this wrapper validates its input, logs the call, and scopes quanteda's internal multithreading via threads.

Usage

ngram_tokens(
  x,
  n = 2L,
  skip = 0L,
  concatenator = "_",
  threads = NULL,
  ncores = NULL,
  nchunks = NULL,
  socket = NULL
)

Arguments

x

A quanteda::tokens object.

n

Integer vector. The number(s) of tokens to concatenate. For example n = 2 produces bigrams and n = 1:2 keeps both unigrams and bigrams.

skip

Integer vector. The number of tokens to skip when forming n-grams. skip = 0 (default) yields adjacent n-grams; positive values yield skip-grams. See quanteda::tokens_ngrams().

concatenator

Character used to join tokens within an n-gram. Defaults to "_".

threads

Integer or NULL. Number of threads quanteda's internal (TBB) pool may use for this call. The previous setting is restored on exit. NULL (default) leaves quanteda::quanteda_options("threads") untouched — quanteda itself defaults to all available cores, so the default is already parallel. See the Threading section.

ncores, nchunks, socket

Deprecated since NLPstudio 1.2.0 and ignored: the chunked process-level (PSOCK/FORK) backend has been removed because it duplicated quanteda's internal multithreading while adding cluster startup, serialization, and peak-memory overhead. Supplying any of them raises a warning of class NLPstudio_deprecated; ncores is mapped to threads when threads is not given. These arguments will be removed in a future release.

Value

A quanteda::tokens object containing the requested n-grams, with documents in the same order as the input.

Details

The chunked process-level backend was removed in NLPstudio 1.2.0; see the Threading section of tokenize_corpus() and vignette("performance-and-threading", package = "NLPstudio").

See also

compound_tokens(), detect_collocations()

Examples

corp <- quanteda::corpus(c(
  doc1 = "the quick brown fox",
  doc2 = "a slow green turtle"
))
toks <- tokenize_corpus(corp)
#> 
#> ── Tokenizing corpus ──
#> 
#> ℹ quanteda::tokens() has been called with default parameters
#> ✔ Corpus successfully tokenized
ngram_tokens(toks, n = 2)
#> 
#> ── Building token n-grams ──
#> 
#> ℹ n = 2; skip = 0
#> ✔ N-grams complete
#> Tokens consisting of 2 documents.
#> doc1 :
#> [1] "the_quick"   "quick_brown" "brown_fox"  
#> 
#> doc2 :
#> [1] "a_slow"       "slow_green"   "green_turtle"
#>