Skip to content
Review Open access

Use of Large Language Models to Assist Risk of Bias Assessment Supported by an Implementation Document

Sep 2026 · Epidemiology Biostatistics and Public Health · 0 citations

TL;DR

A framework for LLM-supported RoB assessments through the RoB2 tool is anticipated, and the agreement between LLM-based outputs and human reference standard is expected to be non-inferior to assessments performed by two humans using an ID.

Abstract

Introduction Recent studies showed poor performance of large language models (LLM) for assessing risk of bias (RoB) with the RoB2 tool. This is in line with the low reliability that humans have in assessing RoB. However, the use of an implementation document (ID) prepared by expert reviewers – i.e., a standardised document providing clear instructions on how to answer each signalling question – can increase inter-rater agreement, and may improve the performance of LLM in RoB-related tasks.   Aims To determine the best framework for using LLMs to support human RoB2 assessments, and to evaluate the value of an ID for LLM-assisted evaluation.   Methods We will include three outcomes from randomised controlled trials (RCTs) across three different systematic reviews in which RoB2 was manually assessed with the guidance of an ID. The final manual assessments will serve as the reference standard. We will prompt LLMs to assess RoB by providing them with (i) the full RoB2 guidance and (ii) the text and supplementary files of the included RCTs. The model output will consist of quotes from the RCT or generated summaries for each signalling question, along with suggested judgements for each applicable signalling question, each domain and overall. Two human reviewers will assess the quality and usefulness of the LLM outputs in answering signalling questions. We will compare the performance of LLMs with and without additional ID guidance, focusing on information extraction quality, accuracy, and reliability of suggested answers and overall judgements. This study will be conducted using locally run LLMs to ensure data privacy and model stability.   Results This is an ongoing study. We anticipate developing a framework for LLM-supported RoB assessments through the RoB2 tool. We will also report on the perceived quality and usefulness of using LLMs to support these assessments. We expect the agreement between LLM-based outputs and human reference standard to be non-inferior to assessments performed by two humans using an ID.   Conclusions This study will inform approaches to use LLMs to support RoB assessment. Future studies should focus on a prospective evaluation of this proposed framework to measure efficiency gains.

Read PDF

Similar papers

Review Open access Apr 2026

Comparison of three large language models' ability to assess the risk of bias using ROBINS-I tool

Current off-the-shelf LLMs cannot reliably perform ROBINS-I risk of bias assessments, as Claude was internally consistent but poorly aligned with human reviewers and GPT had lower reliability and mixed accuracy.

S. Saadi, Farah Fleti, Omar H. Rajjoub et al. · 0 citations
Review Open access Sep 2026

Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study

Abstract Background Large language models (LLMs) have the potential to improve the efficiency of evidence synthesis, but their reliability in performing complex tasks such as risk-of-bias (ROB) assessment in randomized controlled trials (RCTs) remains unclear. Objective This study aimed to evaluate whether LLMs can rel...

Yu-Ju Lai, Shen-Hua Lin, Jen-Wei Liu · 0 citations
Review Open access Sep 2026

Variability, algorithm conformance, and accuracy of large language model–based tools for risk-of-bias (RoB 2.0) assessment of randomized trials: a pilot study

Background: Large language model (LLM)–based tools are increasingly used to automate risk-of-bias assessment of randomized controlled trials with the revised Cochrane tool (RoB 2.0). However, their reproducibility, their accuracy, and their fidelity to the deterministic algorithm mapping signalling-question responses t...

Jiae Choi, Heather Swan, Min Jung Kim et al. · 0 citations
Review Open access Sep 2026

RoB2-Evaluator: Development and Technical Evaluation of a Rule-Constrained Large Language Model System for Cochrane Risk of Bias 2 Assessment

BackgroundCochrane Risk of Bias 2 (RoB 2) assessment is methodologically demanding and resource intensive. Large language models may assist this process but can produce variable judgments. AimTo develop a rule-constrained LLM system for RoB 2 assessment and evaluate its temporal stability and agreement with human revi...

J. Joseph, R. Vishwanath, J. V. Sisirkumar · 0 citations
#small language model Review Open access Sep 2026

A large language model for risk-of-bias assessment in systematic reviews of prognosis studies in clinical neurology

With targeted methodological refinements - including standardization of QUIPS implementation and validation against expert ratings - automated ROB assessments may meaningfully reduce time and cost of systematic reviews of prognosis studies in neurology and beyond.

S. Kampman, K. Braun, W. Otte · 0 citations
#explainable ai Book Open access Oct 2026

Evaluating and explaining LLM classification for automatic screening in Systematic Reviews

This research conducts a set of experiments using open-source Large Language Models with small parameters hosted on local servers to screen titles and abstracts against pre-defined eligibility criteria respective to each topic, and suggests that AI may indeed aid with initial relevance screening, but more research is n...

Min-Ji Kang, Bipasha Banerjee, C. Comer · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.