V-fold cross-fitted discrepancy indices

Cross-fit the held-out discrepancy indices of optop_index_holdout(): split the corpus into V folds, refit the model grid on each fold's complement with a user-supplied fitting function, score the held-out fold, and pool. Every document is scored exactly once, by models trained without it, so the pooled indices are out-of-sample for the whole corpus rather than for one evaluation split; this is the implementation the paper recommends in practice (Appendix B).

Usage

optop_crossfit(
  dtm,
  K,
  fit_fun,
  V = 5,
  metrics = c("se", "chisq", "deviance"),
  c = 1,
  conf = 0.95,
  stabilize = 0,
  min_null = NULL,
  stratify = TRUE,
  seed = NULL,
  n_threads = 1L,
  verbose = FALSE,
  ...
)

Arguments

dtm: A document-term matrix of raw counts (a quanteda::dfm, a dgCMatrix, or a base matrix with dimnames). Sharded optop_corpus() objects are not supported: cross-fitting refits the grid V times, which is engine-scale work outside the sharded evaluation contract.
K: Integer vector of topic counts, strictly increasing, all at least 2. The grid is refit in every fold.
fit_fun: A function (dtm_train, k) returning a fitted model; see the dedicated section.
V: Number of folds, between 2 and floor(J / 2); the default 5 follows the paper's appendix.
metrics: Character subset of c("se", "chisq", "deviance").
c: Positive rare-word threshold constant of the harmonized partition, as in optop_make_partition().
conf: Confidence level of the reported intervals.
stabilize: Nonnegative stabilization constant kappa, as in optop_index_holdout().
min_null: Null-discrepancy floor; NULL (the default) resolves to c, and 0 restores the strict-positivity rule.
stratify: Logical; when TRUE (the default) folds are balanced on document length (documents are ordered by length and each consecutive block of V is dealt across folds at random), so no fold concentrates the short documents. When FALSE the assignment is simple random.
seed: Optional integer seeding the fold assignment (the only randomness this function itself draws).
n_threads: OpenMP threads for the compiled scoring kernels; never changes results.
verbose: Logical; when TRUE, progress alerts per fold.
...: Passed to optop_fold_in() through the per-fold scoring (engine-specific fold-in controls).

Value

An object of class optop_crossfit: a list with

summary: One row per metric and K: pooled macro with fold-replicate macro_se and its interval, pooled micro and micro_se, the Micro-Macro gap and gap_se, n_docs in the pooled J+, and n_null_excluded.
gains: Per metric, the cross-fitted adjacent gains with fold-replicate upper bounds and the selected k_hat at the default tolerance (epsilon = 0.01, alpha = 0.05).
scores: Per metric, the J x length(K) matrix of out-of-fold document indices (NA for floor-excluded documents).
d_model, d_null: The pooled out-of-fold discrepancies.
folds: The fold assignment, named by document.
fold_summary: The per-fold holdout summaries with a fold column, the replicates behind every standard error.
K, V, metrics, conf, stabilize, c, min_null, seed: The configuration.

The fitting function

fit_fun(dtm_train, k) receives the training complement of the fold (rows of dtm in the class you passed) and one topic count, and must return a fitted model OpTop supports: anything optop_as_theta_phi() adapts and optop_fold_in() can project, so fits from topicmodels, seededlda, NLPstudio, and text2vec via optop_warplda() all qualify. A bare optop_model() does not: it has no engine to fold unseen documents in. Seeding the engine is fit_fun's own business; make it depend on k (and, if you want fold-level reproducibility, on the training row names) for reproducible fits.

Pooling and standard errors

Point estimates are computed at the document level on the pooled out-of-fold discrepancies, under the same null-discrepancy floor and stabilization conventions as optop_index_holdout(). Standard errors are fold-replicate (cluster) estimates: the V per-fold values of each quantity (Macro, Micro, the Micro-Macro gap, and the adjacent gains) are treated as exchangeable replicates and the standard error of their mean is sd / sqrt(V). This estimator sees the fold-to-fold variation induced by refitting, which the per-document standard error of a single split ignores; it is the conservative choice the paper's appendix recommends, at the price of only V replicates (a warning is issued when folds are small). The epsilon-adequacy selection of optop_gain_table() is applied to the cross-fitted gains with these fold-replicate bounds; the result is reported per metric and recomputable at other tolerances through plot.optop_crossfit().

Examples

# \donttest{
# simulate a corpus with known truth
rdirich <- function(n, k) {
  g <- matrix(stats::rgamma(n * k, shape = 1), n, k)
  g / rowSums(g)
}
theta <- rdirich(60, 3)
phi <- rdirich(3, 120)
colnames(phi) <- sprintf("w%03d", 1:120)
dtm <- sim_dfm(theta, phi, doc_length = 150, seed = 7)

fit_vem <- function(dtm_train, k) {
  topicmodels::LDA(quanteda::convert(dtm_train, to = "topicmodels"),
                   k = k, method = "VEM",
                   control = list(seed = 1000 + k))
}
cf <- optop_crossfit(dtm, K = 2:4, fit_fun = fit_vem, V = 3,
                     metrics = "deviance", seed = 42)
#> Warning: fewer than 30 documents per fold: the fold Macro means are noisy replicates and the fold-replicate standard errors will be unstable
cf
#> <optop_crossfit>: V-fold cross-fitted discrepancy indices
#> 60 documents in 3 folds, K = {2, 3, 4}, metrics: deviance
#> 95% intervals from fold-replicate standard errors
#> epsilon-adequacy pick (deviance, epsilon = 0.01, alpha = 0.05): K = 2
#>      metric     K        macro     macro_se        ci_lo       ci_hi
#>      <char> <int>        <num>        <num>        <num>       <num>
#> 1: deviance     2 0.0009836828 2.081519e-04 0.0005757127 0.001391653
#> 2: deviance     3 0.0014449503 1.837466e-04 0.0010848137 0.001805087
#> 3: deviance     4 0.0009548638 4.745472e-05 0.0008618542 0.001047873
#>           micro     micro_se           gap       gap_se n_docs n_null_excluded
#>           <num>        <num>         <num>        <num>  <int>           <int>
#> 1: 0.0009719143 1.596900e-04 -1.176853e-05 4.864777e-05     60               0
#> 2: 0.0014532048 1.833317e-04  8.254474e-06 1.439177e-06     60               0
#> 3: 0.0009745113 4.907993e-05  1.964757e-05 3.450178e-06     60               0
plot(cf)

# }