Evaluation of Elicit as an adjunct search, screening, and data extraction tool for systematic reviews

Abstract

Background: Systematic reviews are critical for informing evidence-based medicine, policy, and future research. However, conducting a systematic review is labour- and cost-intensive and error-prone. Elicit is an AI tool that claims to replicate the systematic review process. Methods: Our study investigated Elicit's ability to replicate a systematic review by comparing its results with those of an existing systematic review recently published by our research group. We calculated accuracy, precision, sensitivity, error rate, and person-time savings at each stage of the systematic review. Results: At search, Elicit had poor overlap, finding 158 of the 5285 studies. For abstract and full-text screening, Elicit had moderate sensitivity and accuracy but poor precision, with a skew toward false positives. Elicit's data extraction tool was promising, with sentence-specific citations, and provided helpful information for 85% of values even when it did not find exact matches. Elicit offered person-time savings, taking 3.16 hours to complete the full systematic review process. Conclusions: Elicit shows promise as an adjunct screening and extraction AI tool, but in its current form, it is not ready as a standalone tool to replace the systematic review process.