Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity

Summary

Systematic evaluation of 5 representative multi-agent debate methods across 9 benchmarks and 4 foundation models, finding MAD often fails to outperform simple single-agent baselines such as Chain-of-Thought and Self-Consistency even when consuming significantly more inference-time computation. It identifies model heterogeneity as a universal antidote that consistently improves current MAD frameworks. Serves two purposes: the standard citation for the null result on deliberation, and its heterogeneity finding is the direct empirical bridge to the Hong-Page diversity argument the paper builds on.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the Architecture as Epistemology (ISR) paper (angle: aggregating LLM judgments; MAS failure and cost analyses; correlated errors / homogeneity across LLM agents). Verified against: Opened https://arxiv.org/abs/2502.08788 (arXiv abstract page, v3 21 Jun 2025; title changed from the earlier 'If Multi-Agent Debate is the Answer, What is the Question?')

Critique / open questions

Queued — not yet assessed.

Key quotes

Multi-agent debate (MAD) has gained significant attention as a promising line of research to improve the factual accuracy and reasoning capabilities of large language models (LLMs). (abstract)

Despite its conceptual appeal, current MAD research suffers from critical limitations in evaluation practices, including limited benchmark coverage, weak baseline comparisons, and inconsistent setups. (abstract)