Harmonized Rare-word Partition for OpTop indices

Compute the fixed, document-specific rare-word sets $C_j^*$ used to evaluate the OpTop goodness-of-fit indices on a common support across a grid of topic models. Following Lewis and Grossetti (2026), the union that defines the harmonized set includes the no-topics baseline: a word $w$ is rare in document $j$ if $\min\bigl(\hat\pi_{\mathrm{glob}}(w),\, \min_K p^{K}_{jw}\bigr) < \tau_j,$ where $p^{K}_{jw}$ is the fitted probability of word $w$ under the $K$-topic model, $\hat\pi_{\mathrm{glob}}$ is the corpus word distribution of optop_make_baseline(), and $\tau_j = c / L_j$ with $L_j$ the document length.

Usage

optop_make_partition(models, dtm, c = 1, n_threads = 1L, pi_glob = NULL)

Arguments

models: A list of fitted topic models spanning a grid of $K$. Supports every class handled by the internal adapters: topicmodels::LDA (VEM or Gibbs) and topicmodels::CTM fits, the seededlda models and NLPstudio nlp_topic_fit objects.
dtm: A document-term matrix of counts with rows = documents and columns = vocabulary (recommended class Matrix::dgCMatrix), or an optop_corpus() of count shards streamed one at a time. The rows, concatenated in shard order, must align with the models' documents.
c: Positive scalar controlling the per-document threshold $\tau_j = c / L_j$. Larger values classify more words as rare. Default: 1, the value the paper adopts and recommends when the deviance index is the primary measure; c = 5 remains appropriate when the Pearson index is of primary interest, and sensitivity to c should be reported.
n_threads: Integer; number of OpenMP threads used by the compiled kernels (default 1L). Results are identical for any value; only wall time changes. The package website article on very large corpora covers thread and BLAS configuration at scale.
pi_glob: Optional numeric vector of baseline word probabilities used as the null member of the harmonized union and in the Pearson inclusion rule. Default NULL computes the corpus distribution of dtm, the in-sample construction. Supply the training baseline (optop_make_baseline()$pi_glob) when partitioning an evaluation corpus for held-out scoring, where the null model must come from the training sample.

Value

A list (the sparse partition, format = 2L) with:

nonrare_offsets, nonrare_words: the complement of $C_j^*$ in ragged form. Document $j$'s non-rare words are the nonrare_words entries (0-based word indices, ascending) between offsets j and j + 1; every other word is rare. The total size is bounded by $\sum_j L_j / c$, so the structure scales with the token count rather than with $J \times W$.
vocab: the vocabulary the word indices refer to.
L: numeric vector of length J with document lengths $L_j = \sum_w N_{jw}$.
chisq_min_ok: logical vector of length J; TRUE when the collapsed min-bin of document $j$ satisfies the Pearson inclusion rule (see Details).
chisq_min_report: list with the number and share of documents whose min-bin is excluded and the mean observed probability mass excluded with it.
c: the threshold constant used.
format: the partition format tag (2L). Partitions saved by OpTop 0.13.0 to 0.14.3 (dense rare_mask) are upgraded automatically on first use, with an alert; recompute to avoid the conversion.

Details

OpTop indices (SE, Pearson $\chi^2$, and Deviance) are computed on the fixed support $\{w \not\in C_j^*\} \cup \{\min\}$ for each document, where all words in $C_j^*$ are collapsed into a single "min" bin. In the paper's notation the union runs over the augmented grid $\mathcal{K}_0 = \mathcal{K} \cup \{\mathrm{null}\}$, which treats the no-topics baseline as an additional model with fitted probabilities $p^{\mathrm{null}}_{jw} = \hat\pi_{\mathrm{glob}}(w)$. Using a harmonized partition ensures comparability across $K$ and prevents support-driven artifacts in goodness-of-fit curves. Because the baseline enters the union, every non-min expected count satisfies $E^{K}_{jw} \ge c$ and $B_{jw} \ge c$ on the active support, which bounds all Pearson denominators away from zero.

Pearson min-bin inclusion rule. For Pearson-type summaries the collapsed min-bin of document $j$ is included only when $\min\bigl(\min_K E^{K}_{j,\min},\, B_{j,\min}\bigr) \ge c$ and is excluded otherwise, from both the fitted and the null discrepancy and for all models simultaneously. The decision is taken here, once for the whole grid, and exposed as chisq_min_ok; the share of documents affected and the excluded observed mass are stored in chisq_min_report and reported by the index functions. The rule does not affect the deviance and squared-error families.

Grid dependence. $C_j^*$ is a union over the model grid, so the indices computed on this support depend on the pair (grid, c): extending the grid weakly enlarges $C_j^*$ and can change every reported index. Fix the grid before evaluation and compare indices across studies only under a common grid and c.

Computational notes

The construction is candidate-based and exact. Because the null baseline belongs to the augmented union, a cell can be non-rare only if $\hat\pi_{\mathrm{glob}}(w) \ge \tau_j$; sorting the vocabulary once by $\hat\pi_{\mathrm{glob}}$ descending makes each document's candidate set a prefix of at most $L_j / c$ words, and only candidates are checked against the models. Total cost is $O((\sum_j L_j / c) \cdot \sum_K K)$ instead of the $O(J W \sum_K K)$ dense products of earlier versions, and no $J \times W$ object of any kind is materialized. The min-bin masses of the Pearson rule are complements of compensated non-rare sums, exact for documents whose support collapses entirely into the min-bin and accurate to $O(L_j\,\epsilon_{mach})$ otherwise. For large corpora, keep dtm sparse.

References

Lewis, C. M. and Grossetti, F. (2026). Goodness-of-fit indices and diagnostics for topic models. Working paper.

Lewis, C. M. and Grossetti, F. (2022). A statistical approach for optimal topic model identification. Journal of Machine Learning Research, 23(58), 1–20. https://jmlr.org/papers/v23/19-297.html

Examples

rdirich <- function(n, k) {
  g <- matrix(stats::rgamma(n * k, shape = 1), n, k)
  g / rowSums(g)
}
set.seed(1)
theta <- rdirich(40, 3)
rownames(theta) <- sprintf("d%02d", 1:40)
phi <- rdirich(3, 60)
colnames(phi) <- sprintf("w%02d", 1:60)
counts <- sim_dfm(theta, phi, doc_length = 400, seed = 2)
models <- list(optop_model(theta, phi))

part <- optop_make_partition(models, counts, c = 1)
part
#> <optop_partition>: harmonized support, 40 documents x 60 terms (c = 1)
#> non-rare entries: 2299 (57.5 per document on average)
#> Pearson min-bin rule: 18 documents excluded (45.0% of the corpus)

# a stricter threshold marks more words as rare per document
part5 <- optop_make_partition(models, counts, c = 5)
c(entries_c1 = length(part$nonrare_words),
  entries_c5 = length(part5$nonrare_words))
#> entries_c1 entries_c5 
#>       2299       1301