Held-out goodness-of-fit indices with conditional inference

Answer the question: does the chosen topic model also describe documents it has never seen? Indices computed on the estimation corpus are descriptive and optimistic, because the same documents shaped the topics being evaluated. This function evaluates the same goodness-of-fit indices on an independent evaluation sample instead, following Section 3.7 of Lewis and Grossetti (2026), and attaches the uncertainty measures that make held-out fit values comparable across topic counts: a confidence interval for the average document's fit and standard errors for the pooled index and the Micro-Macro gap.

Usage

optop_index_holdout(
  models,
  dtm_eval,
  baseline,
  c = 1,
  metrics = c("se", "chisq", "deviance"),
  conf = 0.95,
  stabilize = 0,
  n_threads = 1L,
  min_null = NULL,
  ...
)

Arguments

models: A list of topic models fitted on the training corpus, spanning a grid of \(K\); every class supported by the adapters and by fold-in (topicmodels fits, seededlda fits, NLPstudio nlp_topic_fit with a stored backend, text2vec WarpLDA via optop_warplda()).
dtm_eval: A counts document-term matrix of the evaluation documents, or an optop_corpus() of evaluation shards streamed one at a time. Columns are matched to the training vocabulary by name; evaluation-only words follow the out-of-support convention below.
baseline: The training baseline from optop_make_baseline(), optionally smoothed (smooth_lambda).
c: Positive scalar of the harmonized threshold \(\tau_j = c / L_j\); default 1.
metrics: Character subset of c("se", "chisq", "deviance").
conf: Confidence level of the Macro interval (default 0.95).
stabilize: Nonnegative scalar \(\varepsilon\) of the stabilized index \(1 - D^{ho}_j(K) / \max\{D^{ho}_j(\mathrm{null}), \varepsilon\}\). Default 0 (no stabilization; degenerate documents carry NA and are excluded). A positive value makes every document score finite and changes the estimand to the mean of the stabilized score; fix it before evaluation and report sensitivity.
n_threads: Integer; OpenMP threads of the compiled kernels.
min_null: Nonnegative scalar; the null-discrepancy floor. Evaluation documents with baseline discrepancy \(D^{ho}_j(\mathrm{null}) < \mathrm{min\_null}\) are excluded from every estimate and carry an NA score. The default NULL resolves to the threshold constant c; min_null = 0 restores the pre-0.14.1 strict positivity rule. See the Null-discrepancy floor paragraph in Details.
...: Passed to the engine's fold-in routine (e.g. sampler iterations).

Value

An object of class optop_holdout: a list with

summary: data.table with one row per (metric, K). Reading the columns: macro is the average document's held-out fit; macro_se its standard error; ci_lo, ci_hi the bounds of the conf-level interval for the average fit; micro the pooled index, in which documents with a large baseline discrepancy weigh more; micro_se its delta-method standard error; gap equals micro - macro, positive when fit concentrates in high-discrepancy (often long or atypical) documents; gap_se its standard error; n_docs the number of non-degenerate evaluation documents behind every estimate; n_null_excluded the documents excluded by the null-discrepancy floor.
scores: named list (one element per metric) of numeric matrices, evaluation documents by \(K\), holding the per-document held-out indices (NA on degenerate documents when stabilize = 0). These are the scores optop_gain_table() pairs across topic counts.
d_model, d_null: same shape, the per-document fitted and baseline discrepancies, for fit-versus-baseline diagnostics.
K, metrics, conf, stabilize, c, min_null: the evaluation design.

Details

How it works. The workflow has five steps, all handled internally:

Split upstream. You fit the model grid on a training corpus and keep a separate set of evaluation documents that played no role in the fitting; the training baseline comes from optop_make_baseline(dtm_train).
Fold in. For every model and every evaluation document, the document's topic weights are estimated while the trained topics stay fixed (the fold-in step; see the engine notes below). Combined with the trained topics, the weights give the fitted word probabilities of each evaluation document.
Rebuild the common support. The harmonized rare-word partition is reconstructed on the evaluation corpus, with the union running over the whole model grid plus the training baseline, exactly as in the in-sample analysis, so every topic count is scored on the same bins.
Score. Each evaluation document contributes one fitted and one baseline discrepancy per metric, and its document-level index \(1 - D_j(K)/D_j(\mathrm{null})\): the share of the no-topics discrepancy that the model removes on that document.
Aggregate. The Macro index averages the document scores; the Micro index pools the discrepancies first. Because the evaluation documents are independent of one another once the training fits are fixed, both come with valid standard errors.

Held-out target. The scores implement held-out-document reconstruction: the same evaluation document is used to infer its topic mixture and to score its fit, so the model is judged on how well it can rebuild a document after reading it once. For the stricter held-out-token completion target, split each evaluation document upstream and pass the fold-in tokens through the engine directly.

Out-of-support convention. Evaluation documents may contain words the models were never trained on. Those tokens cannot be scored word by word: they are assigned to the observed min-bin of their document (through a synthetic always-rare column with zero fitted and baseline probability), and document lengths \(L_j\) keep every token. Alternatively, smooth the training baseline (optop_make_baseline(dtm_train, smooth_lambda)) and report sensitivity.

Inference. For each \(K\), the Macro index is the mean of the document scores over the non-degenerate evaluation documents \(J_+\); its standard error uses the \(1/(J_+ - 1)\) sample variance and the interval is \(\mathrm{Macro} \pm z_{1-\alpha/2}\, \hat\sigma / \sqrt{J_+}\) (Proposition 2 of the paper). Read the interval as the range of average held-out fits compatible with the evaluation sample, for this training fit: with a different evaluation sample from the same population, the interval would cover the true average about conf of the time. The Micro index and the Micro-Macro gap are smooth functions of conditional means, and their standard errors follow by the delta method. All inference is conditional on the training sample: the target is the expected per-document fit under the training-fitted global objects, not an unconditional population quantity.

Null-discrepancy floor. A document whose entire support is swept into the collapsed min-bin at resolution \(c/L_j\), or whose word distribution sits very close to the training baseline, has a baseline discrepancy at or near zero in every family at once, and its score \(1 - D_j(K)/D_j(\mathrm{null})\) is then unbounded: a handful of such documents destroys the Macro average and its confidence interval, while the Micro index, which weights documents by \(D_j(\mathrm{null})\), is far less exposed. The aggregation therefore conditions on \(D^{ho}_j(\mathrm{null}) \ge \mathrm{min\_null}\) (default: the partition constant c), sharpening the conditioning event of Proposition 2 from a strict positivity to a floor at the resolution of the evaluation support; the estimand becomes the expected fit among evaluation documents at least that far from the baseline. Tiny values signal collapsed supports rather than sampling noise (under the null the binned baseline discrepancy is approximately chi-square with effective bins minus one degrees of freedom, so values orders of magnitude below one arise only from degenerate partitions). Exclusions are reported once per call and per metric in the n_null_excluded column; report sensitivity to min_null alongside c when the excluded share is non-negligible. With min_null > 0, a positive stabilize transforms only the denominators of the documents that pass the floor.

Documents dropped. Evaluation documents with zero aligned tokens cannot be folded in and are dropped with a warning.

References

Lewis, C. M. and Grossetti, F. (2026). Goodness-of-fit indices and diagnostics for topic models. Working paper.

Examples

# \donttest{
# a small synthetic corpus with 6 known topics
rdir <- function(n, k, a) {
  g <- matrix(stats::rgamma(n * k, shape = a), nrow = n)
  g / rowSums(g)
}
set.seed(42)
corpus <- sim_dfm(DTW = rdir(120, 6, 0.4), TWW = rdir(6, 300, 0.1),
                  doc_length = rep(400, 120), seed = 1)

# split: 90 training documents, 30 evaluation documents
dtm_train <- corpus[1:90, ]
dtm_eval <- corpus[91:120, ]

# fit a small grid on the TRAINING split only
models <- lapply(c(4, 6, 8), function(k) {
  topicmodels::LDA(dtm_train, k = k, control = list(seed = 100 + k))
})

# held-out evaluation with the TRAINING baseline
base_tr <- optop_make_baseline(dtm_train)
ho <- optop_index_holdout(models, dtm_eval, base_tr)
ho$summary
#>      metric     K     macro    macro_se     ci_lo     ci_hi     micro
#>      <char> <num>     <num>       <num>     <num>     <num>     <num>
#> 1:       se     4 0.7757723 0.031931388 0.7131879 0.8383567 0.7741158
#> 2:    chisq     4 0.5853783 0.033104635 0.5204944 0.6502622 0.6269005
#> 3: deviance     4 0.5060213 0.030790666 0.4456727 0.5663699 0.5292494
#> 4:       se     6 0.8532328 0.026450311 0.8013911 0.9050745 0.8851021
#> 5:    chisq     6 0.6651107 0.027475643 0.6112594 0.7189619 0.7054246
#> 6: deviance     6 0.5824050 0.029147858 0.5252763 0.6395338 0.6064994
#> 7:       se     8 0.9420041 0.008880918 0.9245979 0.9594104 0.9638684
#> 8:    chisq     8 0.7824089 0.017803137 0.7475154 0.8173024 0.8204904
#> 9: deviance     8 0.7095263 0.020310216 0.6697190 0.7493335 0.7418311
#>       micro_se          gap      gap_se n_docs n_null_excluded
#>          <num>        <num>       <num>  <int>           <int>
#> 1: 0.037917276 -0.001656546 0.021984958     30               0
#> 2: 0.035058638  0.041522272 0.014557906     30               0
#> 3: 0.030980159  0.023228098 0.009501185     30               0
#> 4: 0.019475179  0.031869311 0.015557593     30               0
#> 5: 0.026816992  0.040313909 0.010144119     30               0
#> 6: 0.029285018  0.024094375 0.008710103     30               0
#> 7: 0.004644954  0.021864221 0.006691206     30               0
#> 8: 0.020063507  0.038081489 0.009459553     30               0
#> 9: 0.022486693  0.032304812 0.008351944     30               0

# when do extra topics stop paying for themselves?
gt <- optop_gain_table(ho, epsilon = 0.01)
gt$gains
#>        K succ_K     n       gain        sd     upper        z         pval
#>    <num>  <num> <int>      <num>     <num>     <num>    <num>        <num>
#> 1:     4      6    30 0.07638376 0.1992782 0.1362286 2.099432 1.788941e-02
#> 2:     6      8    30 0.12712122 0.1266775 0.1651635 5.496410 1.938009e-08
gt$k_hat
#> [1] NA
# }