Revisiting LLMs on New Feature Implementation in Real-World Software Development Practices

Abstract

Implementing new features in existing repositories is a key activity in software development. With the rapid advances in large language models (LLMs), benchmarking their capability in this task has attracted growing attention. However, existing benchmarks deviate from real-world practices in two aspects. First, they provide LLMs with unrealistic information such as target code signatures and file paths, which developers do not have access to in practice. Second, they only evaluate the final generated code using functional tests, without examining intermediate steps like requirement understanding and implementation planning. These limitations make it difficult to faithfully assess LLMs’ actual capabilities and diagnose their weaknesses. To address the above limitations, we conduct a systematic evaluation to revisit the capability of large language models (LLMs) in new feature implementation, with a task setup strictly aligned with real-world software development practices. Our evaluation is built on NFEI, a curated benchmark sourced from merged feature pull requests of active GitHub repositories, where each target feature involves cross-snippet multiple code modifications, and LLMs are only provided with the feature description and the target repository throughout the evaluation. Beyond the realistic input settings, we further annotate fine-grained labels for the intermediate steps of the development workflow, including requirement clarification and implementation planning. This enables our evaluation to support a comprehensive stage-wise assessment across three granularities with seven dedicated metrics. We evaluate seven popular LLMs under two development paradigms and three generation strategies. Results show that new feature implementation remains highly challenging, with an overall average Test Pass Rate of only 29.84%. Our stage-wise analysis reveals that intermediate step quality is closely linked to final code performance, and injecting ground-truth intermediate artifacts yields an average absolute gain of about 18%, yet performance remains limited. Qualitative error analysis further identifies three recurrent failure patterns, suggesting that new feature implementation of LLMs adhering to real-world world software development practices remains challenges.