Fold document-topic weights in for new documents

Answer the question: what would the fitted model say about a document it has never seen? A trained topic model consists of global objects (the topic-word distributions) and document-specific topic weights, but the weights exist only for the documents used in training. "Folding in" fills that gap: given the word counts of a new document, the engine estimates the document's topic weights $\hat\theta_j$ while holding the trained topics fixed, so nothing about the model itself changes. The combination of the folded-in weights and the trained topics yields the fitted word probabilities of the new document, which is all the OpTop diagnostics need.

Usage

optop_fold_in(model, newdata, ...)

# S3 method for class 'TopicModel'
optop_fold_in(model, newdata, ...)

# S3 method for class 'textmodel_lda'
optop_fold_in(model, newdata, ...)

# S3 method for class 'optop_warplda'
optop_fold_in(model, newdata, ...)

# S3 method for class 'nlp_topic_fit'
optop_fold_in(model, newdata, ...)

# Default S3 method
optop_fold_in(model, newdata, ...)

# S3 method for class 'optop_theta_phi'
optop_fold_in(model, newdata, ...)

Arguments

model: A fitted topic model supported by optop_as_theta_phi(), trained on the training corpus.
newdata: A counts document-term matrix of the new documents. Its columns must be the training vocabulary, in the same order and with the same names: the model can only interpret words it was trained on (the held-out tools handle the alignment and the out-of-support words before calling this generic).
...: Passed to the engine's fold-in routine, for example sampler iterations.

Value

A numeric matrix of document-topic weights with one row per document of newdata and one column per topic; every row sums to 1.

Details

This internal generic is the engine behind optop_index_holdout() and optop_moment_test(): both call it once per model to score evaluation documents. It has one method per supported engine, each delegating to that engine's own prediction routine:

topicmodels fits run the variational E step of topicmodels::posterior(model, newdata) on the new documents;
seededlda fits are re-estimated on the new documents with the trained model held fixed (textmodel_lda(newdata, model = .)), the package's supported prediction path;
text2vec WarpLDA wrappers call model$transform();
NLPstudio nlp_topic_fit objects delegate to the raw backend fit they store.

Fold-in reuses the new document's counts to estimate its own topic weights, which is why the paper calls the resulting scores a reconstruction target: the model is judged on how well it can rebuild a document after being allowed to read it once. This is less optimistic than in-sample evaluation, where the same documents also shaped the topics themselves, but stricter targets exist (splitting each document into fold-in and scoring tokens). Documents with no tokens on the training vocabulary cannot be folded in; the held-out tools drop them with a warning before reaching this point.

Methods (by class)

optop_fold_in(TopicModel): topicmodels fits: the fold-in E step of topicmodels::posterior() on the new documents.
optop_fold_in(textmodel_lda): seededlda fits: refit the new documents with the trained model held fixed (textmodel_lda(newdata, model = .)), the package's supported prediction path.
optop_fold_in(optop_warplda): WarpLDA wrappers: model$transform() on the new documents.
optop_fold_in(nlp_topic_fit): NLPstudio fits: recurse into the raw backend fit stored in model_object.
optop_fold_in(default): Unsupported classes fail with the list of engines that support fold-in.
optop_fold_in(optop_theta_phi): Bare containers built by optop_model() carry no fitted engine, so new documents cannot be folded in: held-out evaluation needs the engine's own object.

References

Lewis, C. M. and Grossetti, F. (2026). Goodness-of-fit indices and diagnostics for topic models. Working paper.