Skip to content
Preprint

AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL

Nov 2026 · 0 citations · 27 references
Computer Science

TL;DR

AgentScript-Eval is introduced, an execution-graded benchmark of 100 real tasks for AgentScript, the declarative DSL that Salesforce uses in Agentforce to build conversational agents.

Abstract

Domain-specific languages (DSLs) are widely used in enterprise software, giving a compact interface to complex tasks, from data access to infrastructure automation. Platforms increasingly extend them to agents, where a developer declares the topics, permitted actions, and guardrails of an agent rather than writing general-purpose code. Large language models (LLMs) write general-purpose code well, but that ability has been measured almost entirely on languages abundant in public pretraining data, and enterprise DSLs are neither abundant nor public. We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript, the declarative DSL that Salesforce uses in Agentforce to build conversational agents. Tasks span authoring an agent from a specification, debugging a broken definition, and optimizing a working one. Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch. We further find that model scale and price are weak predictors of success -- a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost. Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy. Beyond that gain, the wording of domain guidance matters more than its packaging. The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%, our best result, with catalog and tooling untouched. Larger models did not reliably outperform smaller ones; what mattered was the harness surrounding the model and the guidance supplied to it.

View source

Similar papers

Preprint Oct 2026

Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows

This paper presents a comprehensive empirical study of agentic LLM systems across five software engineering tasks: code generation, technical debt identification, code vulnerability detection, log parsing, and log analysis, and translates these findings into design guidelines for sustainable, task-aware LLM-based devel...

Merve Astekin, Y. N. Tun, Arda Goknil et al. · 0 citations
#machine learning Preprint Nov 2026

DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?

DistDebug-Bench, a benchmark for evaluating agents'ability to diagnose complex-system failures at the code level, curates 120 high-quality real-world bugs from 11 widely used distributed systems, spanning 5 programming languages and 10 failure patterns.

Yuxuan Jiang, Cheng Liu, John Rose et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Agent4Spec: An LLM-based Agentic System to Optimize API Specifications through Automated Validation and Market Analysis

Agent4Spec is presented, an autonomous multi-agent framework featuring two distinct pipelines: a validation pipeline that detects and self-corrects OpenAPI syntax errors, and a market analysis pipeline that conducts competitive research, identifies functional gaps against industry leaders, and automatically generates m...

Yong-Jian Tang, Costel Moraru, Doruk Tuncel et al. · 0 citations
Book Open access Oct 2026

Agent-HAnS: Maintaining Feature and Code Traceability in Agentic Development

This work presents Agent-HAnS, an agentic workflow which instructs the LLM to update a feature model and generate embedded feature annotations after each code-generation step, and visualizes the evolving structure on a live dashboard served via a Model Context Protocol server.

Johan Martinson, Jacob Bengtsson, Moltas Hultin et al. · 0 citations
Preprint Oct 2026

Evaluating Local Language Model Agents for Data Engineering: An Empirical Software Engineering Study of Mobility Workflows

Context: Large language model (LLM) agents are increasingly used as software and data-engineering assistants, yet evidence about locally deployable open-weight agents remains limited. Existing evaluations often emphasise textual responses or isolated code rather than complete engineering artifacts. Objectives: We evalu...

Jorge García-Carrasco, Javier Sanchis, A. Reina-Reina et al. · 0 citations
#natural language process... Preprint Oct 2026

Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills

Enterprise customer-support agents increasingly rely on Large Language Models (LLMs) to execute bounded, tool-using workflows. We study this problem through the lens of Skill LLMs: specialized executors that operate within a selected skill context and must complete a narrow, stateful workflow under policy and tool cons...

Md Tahmid Rahman Laskar, Gundeep Singh, Xue-Yong Fu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.