Simulate a document-feature matrix from an LDA specification

Draw a synthetic corpus of word counts from the standard LDA generative model: each document \(j\) mixes the topic-word distributions with its topic weights, \(p_j = \theta_j^\top \Phi\), and its \(L_j\) tokens are one multinomial draw from \(p_j\). The number of topics, documents, and vocabulary size are inferred from the inputs. The generator is self-contained (no external simulation dependency) and is the quickest way to build small corpora with a known truth, for examples and for exercises like the vignette's known-truth study.

Usage

sim_dfm(DTW, TWW, doc_length, alpha = NULL, seed = NULL)

Arguments

DTW: A matrix or data.frame of document-topic weights (\(J \times k\)), rows summing to 1. Row names, when present, become the document names of the result. Ignored (except for its dimensions) when alpha is supplied.
TWW: A matrix or data.frame of topic-word weights (\(k \times W\)), rows summing to 1. Column names, when present, become the feature names of the result.
doc_length: Numeric vector with the desired length (total token count) of each document; a single value is recycled to all \(J\) documents.
alpha: Optional single positive number or vector of \(k\) positive numbers; when supplied, the document-topic weights are drawn fresh from a \(\mathrm{Dirichlet}(\alpha)\) instead of using DTW (default NULL, use DTW as given).
seed: Optional integer passed to set.seed() for reproducible draws (default NULL).

Value

A quanteda::dfm of simulated word counts with one row per document.

Change in 0.16.0

The generator is now implemented inside OpTop rather than delegated to the LDATS package. The simulated distribution is unchanged (LDA Dirichlet-multinomial sampling), but the random draws for a given seed differ from those of earlier versions.

References

Lewis, C. M. and Grossetti, F. (2022). A statistical approach for optimal topic model identification. Journal of Machine Learning Research, 23(58), 1–20. https://jmlr.org/papers/v23/19-297.html

Examples

# a small corpus with known truth: 20 documents, 3 topics, 50 words
rdirich <- function(n, k) {
  g <- matrix(stats::rgamma(n * k, shape = 1), n, k)
  g / rowSums(g)
}
theta <- rdirich(20, 3)
phi <- rdirich(3, 50)
colnames(phi) <- sprintf("w%02d", 1:50)

sim <- sim_dfm(theta, phi, doc_length = 200, seed = 42)
sim
#> Document-feature matrix of: 20 documents, 50 features (10.20% sparse) and 0 docvars.
#>        features
#> docs    w01 w02 w03 w04 w05 w06 w07 w08 w09 w10
#>   text1   6  17   3   2   1   3   7   2   6   3
#>   text2   4   4   7   1   6   3  11   2   1   2
#>   text3   6  11   9   1   1  10   6   4   4   5
#>   text4   7   5   5   2   3  10   7   2   4   2
#>   text5   4   6   8   2   4   6   9   2   2   3
#>   text6   3   3   5   1   8   4   8   1   3   0
#> [ reached max_ndoc ... 14 more documents, reached max_nfeat ... 40 more features ]
quanteda::ntoken(sim)[1:5]
#> text1 text2 text3 text4 text5 
#>   200   200   200   200   200