Automated Risk of Bias Assessment in Systematic Reviews Using Large Language Models: A Multi-Model Single-Agent Benchmark on 227 Randomised Controlled Trials

Abstract

Code and data accompanying a benchmark of eleven large language models on zero-shot Cochrane Risk of Bias 1.0 assessment of 227 randomised controlled trials drawn from 123 Cochrane systematic reviews published in 2024. The deposit has two independent halves. The first is the assessment pipeline: a containerised web application (FastAPI, Celery, Postgres, MinIO) that ingests trial PDFs, extracts their text with MinerU, and asks a language model to judge each of the seven RoB 1.0 domains. The second is the resulting judgements — 22,246 across fourteen runs — together with every script that turns them into the metrics, statistical tests and figures the article reports. Running the analysis reproduces all reported values in about eighty seconds from two CSV files. The 227 trial reports and the Cochrane reviewers' support-for-judgement text are third-party copyrighted material and are not redistributed; see LICENSE and README.md for what is included and what is deliberately absent.Software is licensed under MIT; the two data tables under Creative Commons Attribution 4.0 International. See LICENSE for the split.