Biomedical retrieval-augmented generation for relation classification

Abstract

The rapid expansion of biomedical literature requires automated methods for accurate and efficient information extraction. This study addresses relation classification: given a pair of annotated biomedical entities in a research article title and abstract, assigning the relation that holds between them from a pre-defined set. We evaluate a retrieval-augmented generation (RAG) framework in which semantically similar annotated examples are retrieved from a training pool and supplied to a large language model as in-context demonstrations. Four open-source LLMs are compared against a supervised DeBERTa+CNN baseline on GutBrainIE, and four further corpora (ChemProt, BioRED, its BioCreative VIII release BC8, and DDI-2013) are used to test whether the findings generalize. We report macro- and micro-averaged precision, recall and F1-score, and the rate of predictions that fall outside the defined relation set. On GutBrainIE, Qwen3:14B with 25 retrieved examples reaches 0.877 macro and 0.866 micro F1, against 0.440 and 0.562 for the supervised baseline, while the smaller Llama-3.1-8B and Gemma3:12B models do not exceed the baseline on macro F1. Macro F1 rises from 0.105 without demonstrations to 0.877 with 25, with the steepest gains over the first ten retrieved examples and subsequent variation smaller than the run-to-run standard deviation. Predictions falling outside the defined relation set drop from 39 at zero shots to at most one between 20 and 35, so retrieval improves output validity as well as accuracy. Across the four further corpora these gains do not generalize uniformly: the best configuration exceeds the supervised baseline on ChemProt, improves macro but not micro F1 on BioRED and BC8, and falls below it on both measures on DDI-2013. Since DDI-2013 is the most balanced corpus in the set and, like ChemProt, is annotated within single sentences, that reversal is not explained by class imbalance or annotation scope, but is consistent with what the relation labels encode: whether they name relationships recoverable from biomedical knowledge, or boundaries set by an annotation guideline.