Dutch Books for Language Models

Summary

Evaluates the coherence rather than the accuracy of LLM probabilistic forecasts: forecasts are elicited over logically related events built from stock-return data, then linear programs compute the largest guaranteed Dutch-book profit an arbitrageur could extract against the model's own probabilities, used as a measure of incoherence. The procedure requires no outcome labels, so coherence is measurable on questions that have not resolved; incoherence rises with richer logical relationships among events, and irrelevant contextual detail can raise it by an order of magnitude. For a research-forecasting task where ground truth is mostly unresolved this is the most useful measurement instrument in the window - it turns justification soundness into a label-free, quantitative, economically interpretable quantity comparable across organizational architectures.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a high-priority candidate for the Architecture as Epistemology (ISR) paper (angle: LLM agents in forecasting; epistemic quality of LLM outputs). Verified against: Verified by opening https://arxiv.org/abs/2609.02797 on 2026-09-07: full abstract read verbatim, 2 authors, submitted 2 September 2026, econ.GN primary (also cs.AI, cs.CL, cs.LG), 14 pages.

Critique / open questions

Queued — not yet assessed.

Key quotes

People increasingly use language models to support life decisions. (abstract)

Many such decisions involve a probabilistic forecast: How likely is a major life event, a natural disaster, or an economic outcome? (abstract)