Correlated Errors in Large Language Models

Summary

Large-scale empirical evaluation of over 350 LLMs showing substantial correlation in model errors - on one leaderboard dataset models agree 60% of the time when both are wrong - driven by shared architectures and providers, and crucially still high for larger, more accurate models even with distinct architectures and providers. The empirical foundation for why the independence assumption behind Condorcet and Sah-Stiglitz polyarchy results fails for populations of LLM agents. They trace downstream consequences for LLM-as-judge evaluation and hiring, framed as algorithmic monoculture.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the Architecture as Epistemology (ISR) paper (angle: correlated errors / homogeneity across LLM agents; aggregating LLM judgments). Verified against: Opened https://arxiv.org/abs/2506.07962 (arXiv abstract page, v1 9 Jun 2025; journal-ref: ICML 2025)

Critique / open questions

Queued — not yet assessed.

Key quotes

Diversity in training data, architecture, and providers is assumed to mitigate homogeneity in LLMs. (abstract)

However, we lack empirical evidence on whether different LLMs differ meaningfully. (abstract)