IEC-Bench is built from the changes observed in real-world use, with chains of dependent calls under the execution paths of 4 widely-used harnesses, and intent-execution correspondence (IEC) is defined as the property that the executed action matches the action the emitted call denotes under the tool contract.
Bo-Yang Yang, Zhen-Hao Li, Zi-Yao Yang et al.· 0 citations
TAILOR, a log parsing framework that improves template inference for rare log groups through template-preserving augmentation, is proposed and experimental results show that TAILOR improves parsing accuracy on rare log groups by 19% over the strongest baseline while maintaining competitive performance on complete datas...
This paper operationalizes developer-oriented NFIs into 92 executable rules and develops a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation, and evaluates state-of-the-art commercial and open-source coding agents.
Pengyu Xue, Hengchen Yuan, Xin Wang et al.· 0 citations
This paper proposes AgentExecutor, a novel multi-agent framework for partial code execution that is Supported by the power of LLM agents who can think, act, and get feedback iteratively, and is able to autonomously explore a richer action space, enabling diverse operations such as creating resource files and resolving...
Junkai Chen, Cheng-Ran Yang, Xing Hu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
