Evaluating and explaining LLM classification for automatic screening in Systematic Reviews
TL;DR
This research conducts a set of experiments using open-source Large Language Models with small parameters hosted on local servers to screen titles and abstracts against pre-defined eligibility criteria respective to each topic, and suggests that AI may indeed aid with initial relevance screening, but more research is needed before full adoption.
Abstract
Evidence synthesis methods, such as systematic reviews (SR), require research teams to find, select, evaluate, and synthesize all available evidence to answer a specific research question in a reproducible and transparent, bias-reduced manner. Early in the process, teams must screen all unique titles and abstracts gathered from a comprehensive search, which can result in thousands of records. Therefore, the process often takes months to years to complete. In this research, we conduct a set of experiments using open-source Large Language Models (LLMs) with small parameters hosted on local servers to screen titles and abstracts against pre-defined eligibility criteria respective to each topic. We use two cases from real SRs for which titles and abstracts screening has already been completed by dual, independent human screeners, which we use as our “ground truth”. The goal of this work is to contribute to developing an AI-enabled workflow to reduce the time and labor required for the relevance screening phase of SRs. Therefore, we start by focusing on metrics such as recall to ensure we don’t miss any relevant results during automatic screening. As the value of these methods depends on a comprehensive corpus and missing relevant records can impair the quality of results, recall is the topmost priority. Our results suggest that AI may indeed aid with initial relevance screening, but more research is needed before full adoption. Repeated runs also demonstrated that models have deterministic results across different prompt variations.
