Jul 2026· Annual International ACM SIGIR Conference on Research and Development in Information Retrieval· 1 citation· 28 references
Computer Science
TL;DR
RUCACoder is proposed, a closed-loop multi-agent framework with a Retriever for hierarchical repository exploration, a Verifier for reranking and validation, and a Coder for feedback-driven script synthesis that consistently outperforms strong retrieval and generation baselines.
Abstract
Repository-level code generation has attracted growing interest, yet most benchmarks and methods remain maintainer-centric, emphasizing bug fixing and feature implementation. In contrast, a common yet underexplored scenario is repository usage: external users want to build applications by correctly invoking repository-internal APIs, composing them into runnable end-to-end workflows rather than modifying the codebase. To support this setting, we introduce RUCCE, a benchmark for repository-usage code generation built from real-world Python repositories. Each instance pairs a natural-language usage instruction with grounded target APIs and a verified reference script, enabling evaluation of both API retrieval and repository-usage code generation. Building on RUCCE, we propose RUCACoder, a closed-loop multi-agent framework with a Retriever for hierarchical repository exploration, a Verifier for reranking and validation, and a Coder for feedback-driven script synthesis. Experiments across multiple backbone LLMs show that RUCACoder consistently outperforms strong retrieval and generation baselines.
PerfAgent is presented, a profiler-guided, verifier-in-the-loop workflow that gives an off-the-shelf coding agent the feedback needed to find real hotspots, improve beyond the first passing patch, and use profiler evidence rather than timing alone to decide what to optimize next.
Ryan Deng, Yuanzhe Liu, Bastian Lipka et al.· 2 citations
Repo0 is presented, a continuous structural evolution framework for zero-to-all code generation that maintains an explicit architectural state instantiated as a Dual-Directed-Acyclic-Graph (Dual-DAG), consisting of a requirement-level DAG, a component-level DAG, and their alignment relation.
Silin Chen, Haoyi Teng, Xiaodong Gu et al.· 0 citations
Organizations often develop and maintain portfolios of related applications: independently deployable codebases that share substantial domain logic, interface patterns, or operational conventions. As LLM coding agents are increasingly used to generate and maintain such software, a naive application-by-application workflow duplicates shared logic across codebases and allows prolonged agentic maintenance to accumulate verbosity, dead code, and structural erosion. We introduce the Super Library Agent problem, where an agent sequentially generates a portfolio of N related applications while maintaining a shared Super Library of reusable cross-application components. A minimal sequential scaffold can in principle extract shared code and migrate applications to the evolving library, but in practice suffers from low extraction recall and fragile dependency migration. We address these failures with candidate-guided extraction over code chunk summaries, pre-extraction codebase consolidation, and context-aware migration using extraction traces and call-graph information. Across WebGen-Bench and PaperBench, our method preserves application functionality while significantly reducing redundancy and token footprint (verbosity, token length) over zero-shot, and avoiding the structural erosion introduced by naive library construction, with additional reductions in LOC and MDL. Our code is available at https://github.com/sbigstar0310/super-library-agent.
Daegyu Sung, Yukyeong Lee, Geon Park et al.· 0 citations
End-to-end GUI testing is essential for verifying web applications, yet existing evaluations rely on predefined checklists and are confined to the data and frameworks of web generation benchmarks, leaving the bug-discovery ability of vision-language models (VLMs) systematically untested. We introduce \textbf{C}ode-driven \textbf{A}gentic \textbf{T}esting (CAT), a paradigm in which the agent writes Playwright code to drive the browser, gathers feedback, and autonomously explores web applications to uncover bugs. We instantiate CAT with CATJudge, an agentic framework that unifies Browser-Use and Computer-Use tools within a single environment and CATTest, a benchmark of 102 AI-generated web applications with carefully annotated bugs, built through close human-AI collaboration to feature complex interactions and subtle defects. Experiments with mainstream VLMs show that all evaluated models perform poorly, revealing a clear gap between current VLM capabilities and the demands of real-world testing in AI web development. We release our code and data at https://github.com/SleepyWithoutCoffee/CATJudge.
Bin Hong, Zhen-Chao Zhang, Jiyuan He et al.· 0 citations
Large language models are increasingly capable of synthesizing executable frontend projects, yet existing benchmarks still treat web generation as a static evaluation problem. We argue that frontend artifacts demand a different paradigm: they are interactive rather than static, admit diverse yet equally valid implementations, and evolve faster than rigid pipelines can accommodate. To address these gaps, we present LiveEvalBench, an automated framework that reformulates web-generation evaluation as an agentic, adaptive, and extensible process. LiveEvalBench instantiates evaluation as a collaborative review workflow, in which a Build Engineer, a Code Engineer, and a UI Tester collectively gather evidence across the full lifecycle of a frontend project, from deployment and code inspection to browser-based interaction. To handle implementation diversity, an adaptive protocol couples shared rubrics for cross-model comparability with implementation-grounded criteria tailored to each artifact. The framework further supports incremental integration of new evaluator roles and assessment dimensions without pipeline redesign. Experiments across diverse real-world web-generation scenarios show that LiveEvalBench aligns closely with human expert judgment and provides fine-grained insights into frontier models'web generation capabilities. Code is available at https://github.com/wyysteelhead/LiveEvalBench
Yiyao Wang, Zhen Wen, Ying Tang et al.· 0 citations
Vero is introduced, the first benchmark to evaluate joint implementation and proof synthesis at the repository level and an audit mechanism where agents are allowed to formally prove unsatisfiability of provided specification or incorrectness of reference code, which surfaces and corrects latent code and specification errors during curation.
Zhe Ye, Hantao Lou, Yuechun Sun et al.· 0 citations