Skip to content
Book Open access

BioFlowBench: A Comprehensive Benchmark for Evaluating Bioinformatics Tool-use Capabilities of LLMs and Agents

Aug 2026 · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · pp. 9059-9070 · 1 citation · 65 references

TL;DR

A significant gap exists between static QA and dynamic execution tasks, with top LLMs perform well on static QA but falter in real-world execution scenario; specialized agents outperform general models in real-world execution through environmental interaction and iterative refinement; and domain knowledge remains the primary bottleneck.

Abstract

The rapid advancement of high-throughput technologies has led to an explosion of biological data and a subsequent surge in bioinformatics analysis tools, thereby creating an urgent demand for automated bioinformatics workflows. Recently Large Language Models (LLMs) and LLM-based agents show great potential in this area. However, existing benchmarks primarily focus on static question-answering (QA) tasks, failing to capture the knowledge-action gap between understanding tool usage and executing complex bioinformatics workflows. Furthermore, current evaluation paradigms often prioritize algorithmic success rates, while neglecting the biological validity. Moreover, the construction of execution benchmarks is challenging due to complex environmental dependencies and the high cost of manual annotation, leading to poor scalability. In this study, we propose BioFlowBench, a comprehensive benchmark designed to shift from static knowledge assessment to dynamic execution evaluation in bioinformatics tool utilization. First, we construct a multi-layered dataset consisting of 5,071 test samples, including Syntax Understanding, Contextual Application and Real-world Execution. Second, we introduce BioGen, an agent-based pipeline designed for the automated generation of executable benchmarks. By creating compact, low-overhead synthetic data, BioGen facilitates low-cost and large-scale testing. Third, we propose a multi-dimensional evaluation framework comprising static knowledge, structural integrity, functional validity, and efficiency metrics. Our experiments reveal that: (1) A significant gap exists between static QA and dynamic execution tasks, with top LLMs perform well on static QA but falter in real-world execution scenario; (2) specialized agents outperform general models in real-world execution through environmental interaction and iterative refinement; and (3) domain knowledge remains the primary bottleneck, often leading to executable but biologically inaccurate outputs. The code is available at: https://github.com/YufeiHouAnne/BioFlowBench and the dataset can be accessed at: https://www.scidb.cn/detail?dataSetId=aee284681d674f53bfc6dae44635e773.

Read PDF

Similar papers

Open access Sep 2026

PromptBio: An Agentic Platform for End-to-End Computational Biomedical Research

Findings show that PromptBio can coordinate complex biomedical analyses under researcher oversight, offering an accessible approach for accelerating scientific iteration and transforming research questions into transparent, reusable computational workflows.

Min-Zhe Zhang, Wen-Hao Gu, Bo-Wei Han et al. · 0 citations
Review Open access Aug 2026

TaxoFlow: a step-by-step tutorial to build a nextflow pipeline for metagenomics taxonomic classification

An open, interactive and web-based tutorial that guides scholars with basic command-line skills through the detailed development of a validated and reproducible Nextflow metagenomics classification pipeline, which aims to lower technical barriers in microbiome bioinformatics and promote best practices in metagenomics d...

Jeferyd Yepes-García, Laurent Falquet · 0 citations
Open access Oct 2026

Second funding phase proposal: NFDI4Microbiota – National Research Data Infrastructure for Microbiota Research

Modern microbiological research generates vast quantities of data, particularly from high-throughput "omics" technologies. These datasets promise unprecedented insights into microbial diversity, function, and interactions, yet they are often deposited in unstructured formats with inadequate metadata, significantly hamp...

K. Förstner, Sina-Victoria Barysch, Anke Becker et al. · 0 citations
Open access Aug 2026

MetaboAnnotate: An AI-powered Multi-agent Framework for Integrating Annotation Tools for Untargeted Metabolomics

Evaluation on the CASMI 2016 and CASMI 2022 benchmarks shows that agreement among independent annotation tools substantially reduces false discovery rates and improves annotation accuracy, and application to a fecal metabolomics dataset further demonstrates the utility of multi-tool consensus for identifying high-confi...

Yan-Zhou Chen, Brandon Mukadziwashe, Frederick Zhang et al. · 0 citations
Open access Sep 2026

Evaluating Large‐Language Models in Bioinformatics Applications

Large language models (LLMs) have significantly revolutionized natural language processing through their strong capabilities in text generation and reasoning. Yet, their applicability to bioinformatics applications remains largely unexplored. Here, we systematically evaluate state‐of‐the‐art LLMs across six represent...

Hengchuang Yin, Zi-Wen Cui, Dong-Xu Li et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.