Aug 2026· Asia-Pacific Workshop on Networking· 0 citations· 3 references
Computer Science
TL;DR
A Network Experimentation Harness for emulator-backed network experiments, helping LLM agents operate across these stateful workflows by pairing a semantic action interface with reusable experimentation skills to handle sequencing, timing, and verification that a careful experimenter would perform by hand.
Abstract
Agent-driven scientific experimentation is emerging across domains such as chemistry, biology, and materials, yet each tool class imposes its own execution discipline. Network experimentation requires more than one-shot topology or configuration synthesis: an experimenter must plan a task, operate a live and evolving network, interpret feedback, refine intermediate state, and validate the resulting behavior. This poster presents a Network Experimentation Harness for emulator-backed network experiments, helping LLM agents operate across these stateful workflows. The Harness pairs a semantic action interface with reusable experimentation skills to handle sequencing, timing, and verification that a careful experimenter would perform by hand. A preliminary study on GNS3-based network protocol experiments shows that this approach reduces wall-clock time by 47% and 37%, and token use by 81% and 76%, on average versus raw GNS3 access and a Python wrapper (GNS3Fy), respectively.
NetConfArena is presented, an executable benchmark for evaluating LLM agents in closed-loop network configuration, and its findings suggest two future directions: using validated trajectories as supervision signals to improve foundation models, and designing harness mechanisms that make agent execution more reliable and accountable.
Chang Liu, Xiaohui Xie, Xinyi Chen et al.· 0 citations
Comparisons against stronger model and coding-agent competitors further indicate that both domain-specific agent runtime structure and foundation-model strength matter for autonomous data analysis.
Summary Agentic AI platforms enable the engineering of autonomous workflows but are not designed for experimentation and hypothesis testing. ASAREE (Analytical Sandbox for Agentic AI Research, Engineering, and Experimentation), is an open-source platform to address this gap. ASAREE creates agents, connects to MCP servers and tools, and designs factorial experiments through a visual interface or Python SDK. It records a full provenance trace for every run and routes all model calls through a provider-agnostic bridge that supports local deployments, ensuring data privacy. As a use-case, we use ASAREE to evaluate key design choices in a mutli-agent machine learning pipeline. Across a 2 × 2 × 2 factorial design, more advanced models, greater reasoning effort, and critic agent use significantly increased compute time, token use, cost, and feature count without improving predictive performance. The lowest-cost baseline, Claude Sonnet 5 with medium effort and no critic, achieved the highest mean PR AUC while Claude Opus 5 with extra high effort and a critic agent cost 15.5× more (USD) and ran 13.1× longer while performing worse on average. These findings highlight ASAREE as a robust framework for evaluating agentic system performance and resource efficiency. Availability and implementation ASAREE is available on GitHub at: https://github.com/EpistasisLab/ASAREE. Contact jason.moore@csmc.edu Supplementary Information Supplementary information is available at https://github.com/EpistasisLab/ASAREE/tree/main/publications/bioinformatics
Jay Moran, P. Freda, Attri Ghosh et al.· bioRxiv· 0 citations
As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering. To address this, we introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. AgentCompass organizes the evaluation process around three independent components, namely Benchmark, Harness, and Environment, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic. Furthermore, it features a fault-tolerant asynchronous runtime and comprehensive trajectory analysis tools to transparently diagnose nuanced failure modes like reward-hacking. Natively supporting over 20 benchmarks across five capability dimensions, AgentCompass provides the community with a scalable and reproducible infrastructure for advancing agent research.
Zichen Ding, Jiaye Ge, Shufan Jiang et al.· 1 citation
DataClawEval is introduced, the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios, and it comprises 100 rigorous, end-to-end tasks spanning five execution engines.
Debin Meng, Jiaming Yang, Zefang Zong et al.· 0 citations
SyntheticAgentTraceQA is proposed, an execution- first framework for generating scalable supervision data for tool- augmented agents and shows that execution-grounded supervision improves tool execution behavior, reference-trace agreement, and answer-generation performance on the evaluated tasks.
Hafsa Ouajdi, Francesco Giannuzzo, Alaa Boukhary et al.· 1 citation· ⚡1