Sep 2026· Epidemiology Biostatistics and Public Health· 0 citations
TL;DR
A framework for LLM-supported RoB assessments through the RoB2 tool is anticipated, and the agreement between LLM-based outputs and human reference standard is expected to be non-inferior to assessments performed by two humans using an ID.
Abstract
Introduction
Recent studies showed poor performance of large language models (LLM) for assessing risk of bias (RoB) with the RoB2 tool. This is in line with the low reliability that humans have in assessing RoB. However, the use of an implementation document (ID) prepared by expert reviewers – i.e., a standardised document providing clear instructions on how to answer each signalling question – can increase inter-rater agreement, and may improve the performance of LLM in RoB-related tasks.
Aims
To determine the best framework for using LLMs to support human RoB2 assessments, and to evaluate the value of an ID for LLM-assisted evaluation.
Methods
We will include three outcomes from randomised controlled trials (RCTs) across three different systematic reviews in which RoB2 was manually assessed with the guidance of an ID. The final manual assessments will serve as the reference standard. We will prompt LLMs to assess RoB by providing them with (i) the full RoB2 guidance and (ii) the text and supplementary files of the included RCTs. The model output will consist of quotes from the RCT or generated summaries for each signalling question, along with suggested judgements for each applicable signalling question, each domain and overall. Two human reviewers will assess the quality and usefulness of the LLM outputs in answering signalling questions. We will compare the performance of LLMs with and without additional ID guidance, focusing on information extraction quality, accuracy, and reliability of suggested answers and overall judgements. This study will be conducted using locally run LLMs to ensure data privacy and model stability.
Results
This is an ongoing study. We anticipate developing a framework for LLM-supported RoB assessments through the RoB2 tool. We will also report on the perceived quality and usefulness of using LLMs to support these assessments. We expect the agreement between LLM-based outputs and human reference standard to be non-inferior to assessments performed by two humans using an ID.
Conclusions
This study will inform approaches to use LLMs to support RoB assessment. Future studies should focus on a prospective evaluation of this proposed framework to measure efficiency gains.
Current off-the-shelf LLMs cannot reliably perform ROBINS-I risk of bias assessments, as Claude was internally consistent but poorly aligned with human reviewers and GPT had lower reliability and mixed accuracy.
S. Saadi, Farah Fleti, Omar H. Rajjoub et al.· BMJ digital health & AI· 0 citations
Abstract Background Large language models (LLMs) have the potential to improve the efficiency of evidence synthesis, but their reliability in performing complex tasks such as risk-of-bias (ROB) assessment in randomized controlled trials (RCTs) remains unclear. Objective This study aimed to evaluate whether LLMs can rel...
Yu-Ju Lai, Shen-Hua Lin, Jen-Wei Liu· Journal of Medical Internet...· 0 citations
Background: Large language model (LLM)–based tools are increasingly used to automate risk-of-bias assessment of randomized controlled trials with the revised Cochrane tool (RoB 2.0). However, their reproducibility, their accuracy, and their fidelity to the deterministic algorithm mapping signalling-question responses t...
Jiae Choi, Heather Swan, Min Jung Kim et al.· Journal of Evidence-Based Pr...· 0 citations
BackgroundCochrane Risk of Bias 2 (RoB 2) assessment is methodologically demanding and resource intensive. Large language models may assist this process but can produce variable judgments.
AimTo develop a rule-constrained LLM system for RoB 2 assessment and evaluate its temporal stability and agreement with human revi...
J. Joseph, R. Vishwanath, J. V. Sisirkumar· medRxiv· 0 citations
With targeted methodological refinements - including standardization of QUIPS implementation and validation against expert ratings - automated ROB assessments may meaningfully reduce time and cost of systematic reviews of prognosis studies in neurology and beyond.
S. Kampman, K. Braun, W. Otte· medRxiv· 0 citations
This research conducts a set of experiments using open-source Large Language Models with small parameters hosted on local servers to screen titles and abstracts against pre-defined eligibility criteria respective to each topic, and suggests that AI may indeed aid with initial relevance screening, but more research is n...