TL;DR
This work shows how machine learning and GenAI can be used to assist with two specific tasks: First, when reading CSV files, it needs to be decided whether the first row is a header or not, and how machine learning and GenAI can be used to assist with two specific tasks.
Large Language Models (LLMs) are fundamentally transforming computer system research and development. As we employ LLMs in file system (fs) development, it is essential to understand their capabilities, limitations, and operational efficiency for domain-specific tasks. We present \phi-Bench, an LLM benchmarking framework for fs-specific tasks. To facilitate benchmarking, we develop six types of tasks in \phi-Bench: basic understanding, basic implementation, performance modeling, debugging, optimization, and new feature development. Each type emphasizes different LLM capabilities: instruction following, knowledge recall, reasoning, or coding. To create high-quality tasks while achieving broad coverage with minimal human effort, we develop a new AI-assisted task generation pipeline in addition to expert-written and textbook-adapted tasks. With 505 tasks in \phi-Bench, we conduct an empirical study with both open source (DeepSeek-V4-Flash, GLM-5.1, and MiniMax-M2.7) and proprietary (Claude-Opus-4.7, GPT-5.2, and Gemini-3.1-Pro) LLMs. Our study discloses the model efficiency for different tasks, causes of failed fs tasks, and techniques for mitigating LLM failures. We will open source \phi-Bench to facilitate public research on using LLMs for fs development.
Many of the domain-specific languages we use every day are not written to files but typed at an interactive prompt, e.g., database shells, cloud CLIs, and in-house analytics consoles. For these REPL-driven command languages, autocomplete is, we argue, not a polish feature but a core usability requirement --and yet they are precisely the languages whose authors often have the fewest resources to invest in tooling. Designing a new programming language tailored to specific domain challenges can be both powerful and rewarding. However, a major hurdle for adoption among users is, in our experience, the lack of tooling support, particularly features like autocomplete that enhance usability and reduce the learning curve. In this paper, we present a fresh perspective that turns parser errors -- traditionally seen as roadblocks -- into opportunities for generating valid, context-aware autocomplete suggestions. By leveraging the parser's built-in feedback mechanisms, our method offers a lightweight, adaptable, and simple solution, enabling domain-specific languages to provide basic development assistance with minimal overhead. The technique fits the REPL-driven DSL genre by design: inputs are modest-size single statements, the cursor sits at the end of the line, and parser-error recovery is unnecessary. We apply the technique to DPella DSL, a production REPL-driven DSL whose grammar comprises 43 top-level command constructors, 130 reserved keywords, and 22 labelled syntactic categories that drive identifier completion.
Synthetic data is extremely important in areas such as data quality, data cleaning, and machine learning. It enables the analysis of use cases in which real data is insufficient, unavailable, or distorted. However, generating synthetic data also presents challenges: The data must be as realistic as possible, but at the same time cover edge cases. It must be possible to insert controlled errors, and at the same time, an error-free version of the data is usually required. Additionally, it is necessary to consider numerous data formats, such as tabular data, but also NoSQL data models. To this end, we present our data generator GouDa. GouDa precisely meets these requirements - it is suitable for different data formats, enables the controlled insertion of errors, and generates ground truth. A wide range of different generation functions and the option to add your own lists of possible attribute values allow the generation of realistic data that covers many different use cases.
DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example, is proposed and effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.
Although existing instruction data selection methods have introduced various metrics, the inherent complexity of real-world datasets makes it impractical for any single metric to generalize across all scenarios. Developers are thus often forced to manually inspect data and craft heuristic rules for each new application---a tedious and error-prone process. In this paper, we propose a paradigm shift from manual configuration to automated orchestration via the Instruction Data Selection Agent (DataMaster), which interprets user intent and autonomously composes optimal selection strategies. By allowing users to specify data needs through natural language descriptions, DataMaster simplifies data curation and removes the burden of manual strategy design. Extensive experiments across the math, medical, and code domains show that DataMaster outperforms static baselines in most settings and surpasses full-pool training in a substantial number of cases. The implementation of DataMaster and the scripts needed to reproduce the reported pipeline are publicly available at https://github.com/nju-websoft/DataMaster.