Do Large Language Models Favour Any Research Topics?

Summary

Large-N test for topic bias when LLMs score research quality: 73,489 articles from 15 health and life sciences journals, comparing title/abstract words in high- versus low-scoring articles for two models. GPT-OSS-120B favours viruses, genes and cells and penalises surveys, patients and students; Gemma 3 27B systematically diverges from it (notably scoring machine-learning research higher), which proves at least one of the two carries genuine AI bias rather than tracking quality. Scores also shift between full-text and title+abstract inputs. Direct evidence for the reliability question in AI-assisted research evaluation and reviewing.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a high-priority candidate for the RISE knowledge base (angle: AI in peer review). Verified against: Opened https://arxiv.org/abs/2609.00323 on 2026-09-07; verbatim abstract read (73,489 articles, 15 journals, GPT-OSS-120B vs Gemma 3 27B), single author Mike Thelwall, submitted 31 Aug 2026. Not in dedupe_titles.txt.

Critique / open questions

Queued — not yet assessed.

Key quotes

Large Language Models (LLMs) can estimate the quality of published journal articles, potentially supporting human assessment when evaluations are needed. (abstract)

Whilst there are reasons to believe that LLMs may have biases in this role, there is no statistically strong evidence yet. (abstract)