OpTop goodness-of-fit indices for a single topic model

Compute the goodness-of-fit indices $R^2_D(K)$ of Lewis and Grossetti (2026) for one fitted topic model: the proportional reduction in discrepancy achieved by the $K$-topic model relative to the no-topics baseline in which every document follows the corpus word distribution, $$R^2_D(K) = 1 - D(K) / D(\mathrm{null}),$$ for the squared-error, Pearson chi-square, and deviance discrepancy families. Document-level indices are evaluated on the harmonized support of optop_make_partition(); word-level indices are evaluated on the unbinned vocabulary.

Usage

optop_index_se(
  model,
  dtm,
  partition,
  baseline,
  macro = FALSE,
  level = c("document", "word"),
  block_size = NULL,
  n_threads = 1L,
  min_null = NULL
)

optop_index_chisq(
  model,
  dtm,
  partition,
  baseline,
  macro = FALSE,
  level = c("document", "word"),
  block_size = NULL,
  n_threads = 1L,
  min_null = NULL
)

optop_index_deviance(
  model,
  dtm,
  partition,
  baseline,
  macro = FALSE,
  level = c("document", "word"),
  block_size = NULL,
  n_threads = 1L,
  min_null = NULL
)

Arguments

model: A single fitted topicmodels::LDA model (VEM or Gibbs), or a loader function returning one (materialized on demand).
dtm: A counts document-term matrix (documents x terms). Use Matrix::dgCMatrix or a quanteda::dfm that represents raw counts (not weighted), or an optop_corpus() of count shards for corpora past the size of a single matrix (shards stream one at a time; per-document results are bit-identical to an unsharded run).
partition: Result from optop_make_partition(), computed on a DTM that is aligned to the model vocabulary (same features and order).
baseline: Result from optop_make_baseline(), computed on the same aligned counts DTM used for partition.
macro: Logical; if TRUE, also compute the Macro index (equal-weight average of the per-document indices). Default: FALSE.
level: Character; aggregation level for the index. "document" (default) computes document-level indices aggregated across words. "word" computes word-level indices aggregated across documents.
block_size: Integer or NULL; number of vocabulary terms to process at once during word-level computation. Smaller values use less memory but may be slower. If NULL (default), automatically chosen based on corpus size to target ~1.5 GB memory usage. Only used when level = "word".
n_threads: Integer; number of OpenMP threads used by the compiled index engine (default 1L). Results are identical for any value; only wall time changes. The package website article on very large corpora covers thread and BLAS configuration at scale.
min_null: Nonnegative scalar; the null-discrepancy floor of the document-level aggregation. Documents with baseline discrepancy $D_j(\mathrm{null}) < \mathrm{min\_null}$ are excluded from the Micro and Macro aggregations and carry an NA document-level index. The default NULL resolves to the partition threshold constant c; min_null = 0 restores the pre-0.14.1 strict positivity rule. See the Null-discrepancy floor paragraph in Details.

Value

When level = "document", a list with:

r2: scalar Micro index over $J_+$.
r2_macro: scalar Macro index if macro = TRUE, otherwise NULL.
r2_doc: numeric vector (length J) with per-document indices; NA for documents with zero baseline discrepancy.
d_model, d_null: numeric vectors (length J) with the per-document fitted and baseline discrepancies.
n_null_excluded, null_excluded_share: number and share of documents excluded by the null-discrepancy floor.
K: number of topics in model.
metric: one of "se", "chisq", "deviance".

When level = "word", a list with:

r2_word: named numeric vector (length W) of per-word $R^2_{D,w}(K)$; NA for words with zero baseline discrepancy.
r2_micro_word: scalar Micro-Word index (discrepancy-weighted average).
r2_macro_word: scalar Macro-Word index (unweighted average over words with positive baseline discrepancy).
d_model, d_null: numeric vectors (length W) with the per-word fitted and baseline discrepancies.
K: number of topics in model.
metric: one of "se", "chisq", "deviance".

Details

Harmonized support. For each document, rare words (as determined by optop_make_partition(), whose union includes the no-topics baseline) are collapsed into a single "min" bin. Document-level indices are then evaluated on the union of the non-rare terms and the min bin, so that changes in the index across $K$ reflect changes in model fit, not changes in binning. Because the harmonized set is a union over the whole model grid, the reported value at any fixed $K$ depends on the pair (grid, c): compare indices across studies only under a common grid and c.

Interpretation. The indices are descriptive effect sizes anchored by the null baseline (lower reference) and the saturated model (upper reference), not tests of correct specification. The deviance index is bounded above by one; the Pearson and squared-error indices are not confined to $[0, 1]$, and negative values are informative: the fitted model performs worse than the no-topics baseline under that discrepancy. In-sample values computed on the estimation corpus summarize descriptive fit and should not be the sole basis for choosing $K$.

Pearson min-bin rule. For the chi-square family the collapsed min-bin of a document enters only when $\min(\min_K E^K_{j,\min}, B_{j,\min}) \ge c$, decided once for the whole grid by optop_make_partition() and applied to the fitted and the null discrepancy simultaneously; excluded shares are reported.

Alignment requirements. The following must share the same vocabulary and column order: the dtm passed to the index, the DTM used for partition and baseline, and the model's term-topic matrix. If they differ, align with optop_align_dtm_to_models() and recompute partition and baseline.

Counts only. SE/chi-square/deviance indices are defined for multinomial counts. Do not pass weighted matrices (e.g., proportions from quanteda::dfm_weight(scheme = "prop") or tf-idf). If your workflow uses proportions elsewhere, reconstruct counts before calling.

Document-level aggregation (level = "document"). The Micro index pools discrepancies before the ratio and is therefore a discrepancy-weighted average of the document-level fits, with weights proportional to the baseline discrepancy $D_j(\mathrm{null})$; the Macro index is their unweighted average. Both aggregate over the non-degenerate documents $J_+ = \{j : D_j(\mathrm{null}) \ge \mathrm{min\_null}\}$ only, and excluded documents carry an NA document-level index. The Micro-Macro gap is itself diagnostic: it equals the covariance between document-level fit and baseline discrepancy (normalized by the mean baseline discrepancy), so a positive gap indicates that fit concentrates in high-discrepancy (often long or atypical) documents. The returned d_model and d_null vectors support the recommended scatter of document-level fit against baseline discrepancy or length.

Null-discrepancy floor. At partition resolution $\tau_j = c/L_j$ a document whose support is entirely swept into the collapsed min-bin has observed equal to baseline counts by construction, so its baseline discrepancy is zero in every family at once, and a document whose word distribution sits close to the corpus baseline has $D_j(\mathrm{null})$ arbitrarily near zero. In both regimes the document-level index $1 - D_j(K)/D_j(\mathrm{null})$ is unbounded, and the Macro index, an unweighted mean, can be destroyed by a handful of such documents, while the Micro index is far less exposed because it weights documents by $D_j(\mathrm{null})$ itself. The aggregation therefore conditions on $D_j(\mathrm{null}) \ge \mathrm{min\_null}$, with default equal to the partition constant c: the estimand becomes the average fit among documents whose baseline discrepancy is at least the resolution of the evaluation support. Tiny baseline discrepancies signal collapsed supports rather than sampling noise: under the null, a document's binned baseline discrepancy is approximately chi-square with (number of effective bins - 1) degrees of freedom, so values orders of magnitude below one arise only when the partition has degenerated. Exclusions are reported once per call and returned as n_null_excluded; report sensitivity to min_null alongside c when the excluded share is non-negligible.

Word-level aggregation (level = "word"). Word-level indices are computed on the unbinned vocabulary and are descriptive diagnostics: they reveal which words the topic structure captures well and which it mispredicts. The word-level deviance is the Poisson unit deviance $2\sum_j [N_{jw}\log(N_{jw}/E_{jw}) - (N_{jw}-E_{jw})]$, which is nonnegative term by term, so the word deviance index never exceeds one; for the corpus baseline the linear correction vanishes identically. Micro-Word weights words by their baseline discrepancy; Macro-Word averages the word-level indices over the words with positive baseline discrepancy (degenerate words carry NA). Restrict attention to words with adequate support when interpreting single-word values.

Functions

optop_index_se(): Squared-error index $R^2_{SE}$.
optop_index_chisq(): Pearson chi-square index $R^2_{chisq}$, subject to the min-bin inclusion rule of optop_make_partition().
optop_index_deviance(): Deviance index $R^2_{dev}$, the default likelihood-based summary: the grouped multinomial deviance at document level and the Poisson unit deviance at word level. Bounded above by one; under exact maximum likelihood and nesting the in-sample document-level index is also nonnegative up to binning effects, a guarantee that does not extend to approximate inference.

References

Lewis, C. M. and Grossetti, F. (2026). Goodness-of-fit indices and diagnostics for topic models. Working paper.

Lewis, C. M. and Grossetti, F. (2022). A statistical approach for optimal topic model identification. Journal of Machine Learning Research, 23(58), 1–20. https://jmlr.org/papers/v23/19-297.html

Agresti, A. (1996). An Introduction to Categorical Data Analysis. Wiley, New York.

Examples

# a corpus with known truth: the true model against a random competitor
rdirich <- function(n, k) {
  g <- matrix(stats::rgamma(n * k, shape = 1), n, k)
  g / rowSums(g)
}
set.seed(1)
theta <- rdirich(40, 3)
rownames(theta) <- sprintf("d%02d", 1:40)
phi <- rdirich(3, 60)
colnames(phi) <- sprintf("w%02d", 1:60)
counts <- sim_dfm(theta, phi, doc_length = 400, seed = 2)

truth <- optop_model(theta, phi)
rand <- local({
  th <- rdirich(40, 3); rownames(th) <- rownames(theta)
  ph <- rdirich(3, 60); colnames(ph) <- colnames(phi)
  optop_model(th, ph)
})

# the harmonized support is shared by the whole grid
partition <- optop_make_partition(list(truth, rand), counts, c = 1)
baseline <- optop_make_baseline(counts)

res_truth <- optop_index_deviance(truth, counts, partition, baseline,
                                  macro = TRUE)
res_rand <- optop_index_deviance(rand, counts, partition, baseline,
                                 macro = TRUE)
res_truth
#> <optop_index>: deviance discrepancy index, K = 3
#> Micro R^2: 0.4410 | Macro R^2: 0.3996
#> J+ = 40 of 40 documents
c(truth = res_truth$r2, random = res_rand$r2)
#>      truth     random 
#>  0.4409784 -2.4478475 

# the Pearson and squared-error variants share the interface
res_x2 <- optop_index_chisq(truth, counts, partition, baseline)
#> ℹ Pearson min bin excluded for 2 documents (5.0% of the corpus; mean excluded observed mass 0.0000)
res_se <- optop_index_se(truth, counts, partition, baseline)

# word-level diagnostic: which words does the model capture best?
word <- optop_index_deviance(truth, counts, partition, baseline,
                             level = "word")
head(sort(word$r2_word, decreasing = TRUE), 3)
#>       w25       w57       w19 
#> 0.8486706 0.7648520 0.7521713