Compound Multiword Expressions into Single Tokens

Join sequences of tokens (phrases or detected collocations) into single compound tokens in a single call to quanteda::tokens_compound(), whose matcher runs in quanteda's multithreaded C++ core. Composes directly with detect_collocations(). On top of the quanteda core, this wrapper validates and coerces pattern, logs progress, and scopes quanteda's internal multithreading via threads.

Usage

compound_tokens(
  x,
  pattern,
  concatenator = "_",
  threads = NULL,
  ncores = NULL,
  nchunks = NULL,
  socket = NULL,
  ...
)

Arguments

x

A quanteda::tokens object.

pattern

Phrases to compound. One of: a character vector of phrases (e.g. c("annual report", "cash flow")); a data.frame/data.table with a collocation column, such as the output of detect_collocations(); or a quanteda collocations object.

concatenator

Character used to join compounded tokens. Defaults to "_".

threads

Integer or NULL. Number of threads quanteda's internal (TBB) pool may use for this call. The previous setting is restored on exit. NULL (default) leaves quanteda::quanteda_options("threads") untouched — quanteda itself defaults to all available cores, so the default is already parallel. See the Threading section.

ncores, nchunks, socket

Deprecated since NLPstudio 1.2.0 and ignored: the chunked process-level (PSOCK/FORK) backend has been removed because it duplicated quanteda's internal multithreading while adding cluster startup, serialization, and peak-memory overhead. Supplying any of them raises a warning of class NLPstudio_deprecated; ncores is mapped to threads when threads is not given. These arguments will be removed in a future release.

...

Additional arguments passed to quanteda::tokens_compound().

Value

A quanteda::tokens object with matched phrases compounded, in the same document order as the input.

Details

The chunked process-level backend was removed in NLPstudio 1.2.0: besides duplicating quanteda's own parallelism, it re-serialized the phrase patterns to the workers once per chunk. See the Threading section of tokenize_corpus() and vignette("performance-and-threading", package = "NLPstudio").

See also

detect_collocations(), ngram_tokens()

Examples

corp <- quanteda::corpus(c(
  doc1 = "the annual report described cash flow risk",
  doc2 = "annual report disclosures mention cash flow"
))
toks <- tokenize_corpus(corp)
#> 
#> ── Tokenizing corpus ──
#> 
#> ℹ quanteda::tokens() has been called with default parameters
#> ✔ Corpus successfully tokenized
compound_tokens(toks, pattern = c("annual report", "cash flow"))
#> 
#> ── Compounding multiword expressions ──
#> 
#> ✔ Compounding complete
#> Tokens consisting of 2 documents.
#> doc1 :
#> [1] "the"           "annual_report" "described"     "cash_flow"    
#> [5] "risk"         
#> 
#> doc2 :
#> [1] "annual_report" "disclosures"   "mention"       "cash_flow"    
#>