Global Corpus Baseline for OpTop Indices

Compute the global word distribution \(\pi_{\mathrm{glob}}\) from a corpus of counts. This baseline defines the "no-topics" model used to normalize OpTop indices via \(B_{jw} = L_j \, \pi_{\mathrm{glob}}(w)\) and \(B_{j,\min} = L_j \sum_{w \in C_j^*} \pi_{\mathrm{glob}}(w)\). It is the analogue of the intercept-only model in regression: every document follows the same corpus-level word distribution, scaled by its length.

Usage

optop_make_baseline(dtm, smooth_lambda = 0)

Arguments

dtm: A document-term matrix of counts with rows = documents and columns = vocabulary (recommended class Matrix::dgCMatrix), or an optop_corpus() of count shards, whose counts are pooled.
smooth_lambda: Nonnegative additive smoothing constant: \(\hat\pi^{\lambda}_{\mathrm{glob}}(w) \propto N_{\cdot w} + \lambda\). Default 0 (no smoothing, the in-sample construction). A positive value keeps every baseline probability strictly positive, the smoothing option of the paper's held-out support convention; sensitivity to \(\lambda\) should be reported.

Value

A list with:

pi_glob: numeric vector of length W with \(\pi_{\mathrm{glob}}(w) = N_{\cdot w} / N_{\cdot \cdot}\), i.e., the corpus-level term probabilities (smoothed when smooth_lambda > 0).

Details

The returned pi_glob is used by the exported index functions to form per-document baseline counts on the same evaluation support defined by optop_make_partition(). Keeping this baseline fixed across \(K\) enables interpretable, regression-style \(R^2\) measures: the null and the saturated models are the two interpretative boundaries between which every fitted \(K\)-topic model lies. For held-out evaluation, compute the baseline on the training corpus and pass it to optop_index_holdout(); words absent from the training sample have zero baseline probability unless smooth_lambda > 0.

References

Lewis, C. M. and Grossetti, F. (2026). Goodness-of-fit indices and diagnostics for topic models. Working paper.

Examples

set.seed(1)
counts <- matrix(rpois(40 * 60, 3), 40, 60,
                 dimnames = list(sprintf("d%02d", 1:40),
                                 sprintf("w%02d", 1:60)))
dtm <- methods::as(Matrix::Matrix(counts, sparse = TRUE),
                   "CsparseMatrix")
base <- optop_make_baseline(dtm)
sum(base$pi_glob)  # exactly 1
#> [1] 1

# Laplace-style smoothing pulls the baseline toward uniform
base_s <- optop_make_baseline(dtm, smooth_lambda = 0.5)
range(base_s$pi_glob / base$pi_glob)
#> [1] 0.9993512 1.0010358