Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles

Summary

Elicits probability estimates from 15 LLMs on 254 binary prediction-market questions and compares classical against learned aggregators, finding a multilayer perceptron and a logistic regression outperform all individual models and all classical pooling rules. Direct evidence that the aggregation rule, not merely the existence of an ensemble, carries the performance gain - the central claim the paper's second factor tests. It also documents training-cutoff contamination as a pervasive confound, the frontier-versus-small-model gap collapsing from 35.8% to 8.9% on questions resolved after all cutoffs, a methodological warning for any research-forecasting task design.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the Architecture as Epistemology (ISR) paper (angle: aggregating LLM judgments; LLM agents in forecasting). Verified against: Opened https://arxiv.org/abs/2607.18269 (arXiv abstract page, v2 22 Jul 2026)

Critique / open questions

Queued — not yet assessed.

Key quotes

The wisdom of crowds -- the finding that aggregating judgments across individuals often outperforms the best individual -- has been extensively studied with human forecasters. (abstract)

Whether the same phenomenon emerges when the ``crowd'' consists of large language models (LLMs) is an open question with both theoretical and practical implications. (abstract)