Skip to content

SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents

Sep 2026 · 0 citations · 28 references
Computer Science

TL;DR

SciHorizon-eLab is introduced, an agentic protocol-to-task compiler that formulates scientific embodied task construction as a compilation problem and constructs \BenchName, a ready-to-use benchmark comprising 300 certified tasks across diverse laboratory operations.

Abstract

Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systematic evaluation environments. Existing simulation-based laboratory benchmarks rely heavily on manual task engineering, making it challenging to systematically compile diverse scientific protocols into executable and verifiable embodied tasks at scale. To address this challenge, we introduce SciHorizon-eLab, an agentic protocol-to-task compiler that formulates scientific embodied task construction as a compilation problem. Given a natural-language protocol of scientific experiments, SciHorizon-eLab progressively compiles laboratory protocols into semantic-preserving embodied tasks through semantic grounding, executable task synthesis, and multi-stage simulation-based certification. The system generates semantically grounded environments, executable manipulation programs, and step-level success specifications, while enabling reproducible generation of expert demonstrations and execution traces. Using this pipeline, we further construct \BenchName, a ready-to-use benchmark comprising 300 certified tasks across diverse laboratory operations. It supports HIL task execution, reproducible expert-demonstration generation, and ordered step-level evaluation. Across representative tasks, the strongest policy attains an average success rate of only 49.7%, with further evaluations revealing pronounced weaknesses in human and embodied agent coordination. We publicly release the code, benchmark data, and evaluation toolkit at https://github.com/SciHorizon-elab/SciHorizon-elab.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction

Agentic systems offer a promising way to automate embodied benchmark construction, but existing approaches typically cover isolated stages or remain specialized to predefined environments and task families. More importantly, multi-step construction produces dependent intermediate artifacts that are often passed downstr...

Bao-Yang Jiang, Feng-Chun Zhang, Le-Yuan Wang et al. · 0 citations
Open access 2026

Bohrium + SciMaster: Building the infrastructure and ecosystem for agentic science at scale

AI agents are emerging as a practical way to run multi-step scientific workflows that interleave reasoning, tool use, and verification. Scaling such agentic science remains difficult because workflows are hard to observe and reproduce, many scientific tools and laboratory systems are not agent-ready, and execution trac...

Lin-Feng Zhang, Si-Heng Chen, Yu-Zhu Cai et al. · 0 citations

ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments

The model family shows gains in held-out scientific-code repair and across selected general-purpose benchmarks in code, reasoning, and knowledge, providing evidence of positive transfer from scientific experience to broader capabilities.

He-Jia Geng, Ze-Sen Huang, Hao-Yang Li et al. · 7 citations

MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding

Recently, the rapid development of large language models (LLMs) has reshaped software engineering by enabling autonomous code agents that plan, execute, and utilize external tools iteratively to tackle complex tasks. Beyond achieving functional correctness, these agents must faithfully follow process instructions and c...

Bo-Si Wen, Cun-Xiang Wang, Jia-Yi Gui et al. · 1 citation
Preprint Aug 2026

Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modeling

Pufibara is presented, an agent harness that maintains persistent engineering state across revisions, associates execution and simulation evidence with the candidate that produced it, and makes submission an explicit agent action to evaluate end-to-end Modelica agent workflows.

Zi-Zhe Wang · 0 citations
Preprint Oct 2026

HakiCC: LLM-Driven Multi-Agent Design and Optimization of Concurrency Control Protocols

Large language models (LLMs) have recently been applied in systems research as a tool to reduce human-intensive engineering effort through cost-efficient automation. Decades of research have produced a rich landscape of concurrency control (CC) protocols, each encoding distinct trade-offs in correctness, throughput, an...

Farzad Habibi, Jun-Cheng Fang, Faisal Nawab · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.