Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate

Summary

Tackles the problem of evaluating reasoning quality in multi-agent LLM systems on open-ended tasks with no reference answer, comparing token-level log-probability confidence proxies against rubric-based LLM-as-judge scores in a debate-based essay-scoring framework. Early-token confidence, within the first few generated tokens, is consistently the strongest predictor of reasoning quality, and there is a systematic asymmetry by agent role, with confidence tracking quality more closely for supportive reasoning than for adversarial critique. The role asymmetry matters directly for any design in which some agents are assigned critic roles.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a medium-priority candidate for the Architecture as Epistemology (ISR) paper (angle: epistemic quality of LLM outputs; aggregating LLM judgments). Verified against: Opened https://arxiv.org/abs/2606.10307 (arXiv abstract page, v1 9 Jun 2026)

Critique / open questions

Queued — not yet assessed.

Key quotes

Evaluating reasoning quality in multi-agent LLM systems is challenging, especially for open-ended tasks without reference answers. (abstract)

We investigate whether intrinsic confidence signals, token-level log-probabilities from decoding, can predict reasoning quality as assessed by LLM-as-judge evaluation. (abstract)