Skip to contents

Sharded corpus for very large document sets

Source: R/corpus.R
optop_corpus.Rd

Wrap a document-term matrix, or a collection of document shards of one, into the container the OpTop entry points accept wherever a dtm or a weighted dfm is accepted today. Sharding is how OpTop passes the hard container limit of R sparse matrices: a single dgCMatrix (and hence a single quanteda dfm) holds at most \(2^{31} - 1\) nonzero entries, a bound no compiler flag lifts. Every OpTop statistic is a sum of independent per-document terms, so evaluating shard by shard and combining is exact, not an approximation: per-document scores are bit-identical to an unsharded run and concatenate; statistics and degrees of freedom add, agreeing with the unsharded totals up to floating-point summation order; and the calibration bootstrap keys every document's random stream by its global index, so each document draws identical replicates whatever the sharding.

Usage

optop_corpus(x, reader = NULL)

Arguments

x

One of:

  • a single dfm or sparse matrix (the trivial one-shard corpus);

  • a list of dfms or sparse matrices, the shards, in document order;

  • a character vector of file paths, one shard per file, loaded on demand through reader (shards are then held in memory one at a time).

reader

A function of one path returning a dfm or sparse matrix; required when x is a character vector (for example function(p) qs2::qs_read(p) or readRDS()).

Value

An object of class optop_corpus.

Details

All shards must share the same vocabulary in the same order, and their document sets must be disjoint; documents are identified by row names (or quanteda::docid() for dfms). In-memory shards are validated at construction; path shards are validated when first materialized. The functions that consume a corpus stream it one shard at a time, so peak memory is one shard plus the per-document result vectors, whose length is the total number of documents.

Whether the corpus must hold counts or proportions is decided by the consumer, exactly as for a single matrix: optop_select() expects proportions, the partition, baseline, and index family expect counts.

References

Lewis, C. M. and Grossetti, F. (2026). Goodness-of-fit indices and diagnostics for topic models. Working paper.

See also

optop_select(), optop_make_partition(), optop_make_baseline(), optop_index_holdout()

Examples

# shard a small corpus in memory: results match the unsharded matrix
rdirich <- function(n, k) {
  g <- matrix(stats::rgamma(n * k, shape = 1), n, k)
  g / rowSums(g)
}
set.seed(1)
theta <- rdirich(40, 3)
rownames(theta) <- sprintf("d%02d", 1:40)
phi <- rdirich(3, 60)
colnames(phi) <- sprintf("w%02d", 1:60)
counts <- sim_dfm(theta, phi, doc_length = 400, seed = 2)
models <- list(optop_model(theta, phi))

corp <- optop_corpus(list(counts[1:25, ], counts[26:40, ]))
corp
#> An optop_corpus of 2 in-memory shards
#>   vocabulary: 60 features

part_sharded <- optop_make_partition(models, corp, c = 1)
part_plain <- optop_make_partition(models, counts, c = 1)
identical(part_sharded$nonrare_words, part_plain$nonrare_words)
#> [1] TRUE

if (FALSE) { # \dontrun{
# at real scale, point at per-shard files written upstream: only one
# shard is ever in memory
corp <- optop_corpus(list.files("shards", full.names = TRUE),
                     reader = function(p) qs2::qs_read(p))
} # }