Guiding LLM Peer Reviewers: The Impact of Score Anchors on Review Evidence and Accuracy

Summary

Uses 98 Allied Health Professions outputs with specialist human review reports and adjudicated 1-4 reference scores to test whether supplying an oracle score changes not just an LLM reviewer's accuracy but the evidence its review cites. Oracle guidance improves scoring accuracy without the model simply copying the score, and corrected mismatches shift the review frame; alignment is direction-dependent, with LLMs covering human strength/upgrade points far more reliably than expert downgrade evidence. Useful because it evaluates LLM review at the level of evidence rather than score correlation.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a medium-priority candidate for the RISE knowledge base (angle: AI in peer review). Verified against: Opened https://arxiv.org/abs/2609.01905, read full abstract, 2 authors, 2026-09-01, cs.DL. Dedupe: no 'score anchors' match.

Critique / open questions

Queued — not yet assessed.

Key quotes

Large language models (LLMs) are increasingly used for research quality evaluation, with prior work exploring their scoring accuracy and the plausibility of review rationales. (abstract)

However, less is known about whether external score guidance changes the evidence presented in the generated review as well as the final score. (abstract)