Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· pp. 9059-9070· 1 citation· 65 references
TL;DR
A significant gap exists between static QA and dynamic execution tasks, with top LLMs perform well on static QA but falter in real-world execution scenario; specialized agents outperform general models in real-world execution through environmental interaction and iterative refinement; and domain knowledge remains the primary bottleneck.
Abstract
The rapid advancement of high-throughput technologies has led to an explosion of biological data and a subsequent surge in bioinformatics analysis tools, thereby creating an urgent demand for automated bioinformatics workflows. Recently Large Language Models (LLMs) and LLM-based agents show great potential in this area. However, existing benchmarks primarily focus on static question-answering (QA) tasks, failing to capture the knowledge-action gap between understanding tool usage and executing complex bioinformatics workflows. Furthermore, current evaluation paradigms often prioritize algorithmic success rates, while neglecting the biological validity. Moreover, the construction of execution benchmarks is challenging due to complex environmental dependencies and the high cost of manual annotation, leading to poor scalability. In this study, we propose BioFlowBench, a comprehensive benchmark designed to shift from static knowledge assessment to dynamic execution evaluation in bioinformatics tool utilization. First, we construct a multi-layered dataset consisting of 5,071 test samples, including Syntax Understanding, Contextual Application and Real-world Execution. Second, we introduce BioGen, an agent-based pipeline designed for the automated generation of executable benchmarks. By creating compact, low-overhead synthetic data, BioGen facilitates low-cost and large-scale testing. Third, we propose a multi-dimensional evaluation framework comprising static knowledge, structural integrity, functional validity, and efficiency metrics. Our experiments reveal that: (1) A significant gap exists between static QA and dynamic execution tasks, with top LLMs perform well on static QA but falter in real-world execution scenario; (2) specialized agents outperform general models in real-world execution through environmental interaction and iterative refinement; and (3) domain knowledge remains the primary bottleneck, often leading to executable but biologically inaccurate outputs. The code is available at: https://github.com/YufeiHouAnne/BioFlowBench and the dataset can be accessed at: https://www.scidb.cn/detail?dataSetId=aee284681d674f53bfc6dae44635e773.
Findings show that PromptBio can coordinate complex biomedical analyses under researcher oversight, offering an accessible approach for accelerating scientific iteration and transforming research questions into transparent, reusable computational workflows.
Min-Zhe Zhang, Wen-Hao Gu, Bo-Wei Han et al.· bioRxiv· 0 citations
An open, interactive and web-based tutorial that guides scholars with basic command-line skills through the detailed development of a validated and reproducible Nextflow metagenomics classification pipeline, which aims to lower technical barriers in microbiome bioinformatics and promote best practices in metagenomics d...
Modern microbiological research generates vast quantities of data, particularly from high-throughput "omics" technologies. These datasets promise unprecedented insights into microbial diversity, function, and interactions, yet they are often deposited in unstructured formats with inadequate metadata, significantly hamp...
K. Förstner, Sina-Victoria Barysch, Anke Becker et al.· Research Ideas and Outcomes· 0 citations
Evaluation on the CASMI 2016 and CASMI 2022 benchmarks shows that agreement among independent annotation tools substantially reduces false discovery rates and improves annotation accuracy, and application to a fecal metabolomics dataset further demonstrates the utility of multi-tool consensus for identifying high-confi...
Yan-Zhou Chen, Brandon Mukadziwashe, Frederick Zhang et al.· Analytical Chemistry· 0 citations
Large language models (LLMs) have significantly revolutionized natural language processing through their strong capabilities in text generation and reasoning. Yet, their applicability to bioinformatics applications remains largely unexplored. Here, we systematically evaluate state‐of‐the‐art LLMs across six represent...
Hengchuang Yin, Zi-Wen Cui, Dong-Xu Li et al.· Advanced Computing· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.