ESF-Bench is introduced, a challenging Enterprise Slot Filling benchmark consisting of 810 multi-turn samples and 6530 slots over 8 unique domains and curated using a taxonomy of the 57 most challenging slot-filling scenarios observed during real-world enterprise deployments.
Abstract
The rapid rise of large language models (LLMs) has driven transformative adoption across enterprises. However, deploying these models in real-world settings presents unique challenges due to complex system constraints and unexpected user behaviors. Among these applications, slot filling is essential for converting unstructured input into structured, actionable data. In this work, we introduce ESF-Bench, a challenging Enterprise Slot Filling benchmark consisting of 810 multi-turn samples and 6530 slots over 8 unique domains. Curated using a taxonomy of the 57 most challenging slot-filling scenarios observed during real-world enterprise deployments, ESF-Bench exposes notable limitations in current state-of-the-art LLMs, with GPT-OSS-120b low successfully extracting slots for only 20.7% of benchmark samples. To support continued research in this area, we publicly release the benchmark dataset, taxonomy, and accompanying evaluation code on GitHub.
A framework for scalable, dataset-scale LLM-based data transformation that processes data row-by-row in batches over lazily evaluated, distributed dataframes, enabling LLM-based transformations on arbitrarily large datasets without loading them into memory.
A. Abraham, F. Rahman, Fadil Rahman et al.· Proceedings of the 15th Inte...· 0 citations
Experimental results demonstrate that D2VBench exhibits high reliability and robustness, effectively reflecting the LLMs'alignment across different value categories and dimensions, and providing a more realistic and fine-grained tool for research on value alignment.
A large-scale benchmark with 60\sim 90× more queries than prior work, built on 3× more databases, an automated pipeline that can execute existing methods without manual intervention, and multi-dimensional, fine-grained evaluation metrics for comprehensive assessment.
Bo Li, Chenzhan Wang, Longkang Lin et al.· Proceedings of the 32nd ACM...· 0 citations
DBLifeBench is introduced, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance, and a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving is proposed.
Shunfan Zheng, Dongsheng Shi, Yue Li et al.· 0 citations
Large Language Models (LLMs) are fundamentally transforming computer system research and development. As we employ LLMs in file system (fs) development, it is essential to understand their capabilities, limitations, and operational efficiency for domain-specific tasks. We present \phi-Bench, an LLM benchmarking framework for fs-specific tasks. To facilitate benchmarking, we develop six types of tasks in \phi-Bench: basic understanding, basic implementation, performance modeling, debugging, optimization, and new feature development. Each type emphasizes different LLM capabilities: instruction following, knowledge recall, reasoning, or coding. To create high-quality tasks while achieving broad coverage with minimal human effort, we develop a new AI-assisted task generation pipeline in addition to expert-written and textbook-adapted tasks. With 505 tasks in \phi-Bench, we conduct an empirical study with both open source (DeepSeek-V4-Flash, GLM-5.1, and MiniMax-M2.7) and proprietary (Claude-Opus-4.7, GPT-5.2, and Gemini-3.1-Pro) LLMs. Our study discloses the model efficiency for different tasks, causes of failed fs tasks, and techniques for mitigating LLM failures. We will open source \phi-Bench to facilitate public research on using LLMs for fs development.
