Deterministic in Theory, Random in Practice: Numerical Precision and the Reproducibility of LLM Measures in Accounting Research

working

accounting

LLM

NLP

methods

Accounting researchers who use large language models (LLMs) to measure disclosure characteristics typically report the model, the prompt, the temperature, and…

Authors

Claudio Ferrantino

Francesco Grossetti

Tim Martens

Abstract

Accounting researchers who use large language models (LLMs) to measure disclosure characteristics typically report the model, the prompt, the temperature, and the random seed. A commonly held belief is that reporting these parameters makes the measure replicable, as the output is believed to be fully determined by the underlying text. We test this belief in an experiment in which four open-weight LLMs score the tone and uncertainty of, and generate summaries for, 1,024 MD&A sections of Form 10-K filings, while we vary the numerical precision of the computation, the number of concurrent requests, and the prompt. We distinguish within-specification variation, which arises when the identical configuration is simply run again, from across-specification variation, which arises when configurations change. Within specification, the output is not fixed. At the lowest precision (BF16), as few as 1.5% of filings receive the same response in ten identical runs, compared with more than 90% at the highest precision (FP32). Across specifications, precision is the setting that matters. Switching the precision shifts the scores of individual filings by about 2.4% of the variation in tone across filings, compared with about 0.1% for switching the number of concurrent requests. Our results show that both types of variation leave averages essentially unchanged, but they affect inference. The uncertainty score and the summaries show the same pattern. Our results suggest that computing in FP32 and reporting the runtime environment remove most of the problem.

Citation

Ferrantino, C., Grossetti, F. & Martens, T. (2026). Deterministic in Theory, Random in Practice: Numerical Precision and the Reproducibility of LLM Measures in Accounting Research. Working paper.

BibTeX

@unpublished{ferrantino2026deterministic,
  title  = {Deterministic in Theory, Random in Practice: Numerical Precision and the Reproducibility of LLM Measures in Accounting Research},
  author = {Ferrantino, Claudio and Grossetti, Francesco and Martens, Tim},
  note   = {Working paper, preparing for submission},
  year   = {2026}
}