A rigorous comparative analysis of three popular agentic IDEs in the generation of five full-stack Web applications from scratch shows that Agentic IDEs cannot replace developers but shift their role toward building software by orchestrating LLM-based agents through natural-language instructions and iterative refinement.
Abstract
Agentic IDEs are among the most significant innovations in software engineering, aiming to accelerate application development through LLM-based agents that can assist developers during development. However, their evaluation in end-to-end development tasks involving the generation of complete applications remains limited. To fill this gap, we propose a rigorous comparative analysis of three popular agentic IDEs (Copilot, Cursor, and Windsurf) in the generation of five full-stack Web applications from scratch. Results show high maturity in the generation of established patterns, such as CRUD operations and authentication features. In contrast, the generation of less common distributed architectures, such as a task queue architecture, produces significantly more errors. Overall, results show that Agentic IDEs cannot replace developers but shift their role toward building software by orchestrating LLM-based agents through natural-language instructions and iterative refinement. Yet, each agentic IDE shows its peculiarities, although differences are narrow.
GUI agents have advanced rapidly, producing a growing body of frameworks, benchmarks, and applications. However, this growth has outpaced the maturity of the field. GUI agents remain technically brittle, incompletely engineered, and insufficiently validated for sustained real-world use. They are evolving into closed-loop software systems. Within these systems, model reasoning is coupled with interface perception, execution feedback, recovery, and human oversight. This evolution calls for a software engineering perspective that remains largely absent from existing research. We address this gap by reviewing 336 GUI-agent papers from January 2018 to April 2026. Five research questions examine the research landscape, architectures, evaluation, software lifecycle concerns, and future opportunities. Our findings show that the field has expanded sharply since 2024, while mobile and web settings remain dominant. Architectures increasingly adopt modular perceive-reason-act loops, but recovery, human escalation, safety enforcement, and auditability remain underdeveloped. This architectural imbalance extends to evaluation. Evaluations are becoming more interactive, but they remain centered on task success and are difficult to compare across protocols. More broadly, existing studies provide limited support for testing beyond benchmarks and for maintaining agents after release. Observability, privacy engineering, and systematic human oversight are also underdeveloped. Together, these findings show that capability improvements alone cannot ensure deployment readiness. Future research should connect dependable execution with lifecycle-centered testing and reproducible evaluation. It should also integrate permission and privacy controls with cost-aware, human-centered governance. This integration is necessary to build dependable, maintainable, secure, and deployable GUI-agent systems.
Shengcheng Yu, Yuchen Ling, Junyang Xing et al.· 0 citations
A large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective, providing empirical evidence linking framework design choices to engineering risks and highlighting the need for stronger guidance and support in agent framework development.
Yibo Zhai, Junjun Si, Yan Wang et al.· SIGSOFT FSE Companion· 0 citations
Software engineering is in the middle of a quiet but far-reaching handover. For most of the last decade, artificial
intelligence in the developer's tool chain meant auto complete: a model that finished a line, or occasionally a function, while
a human wrote and reviewed everything around it. That arrangement has started to break down. Coding agents such as
Claude Code, OpenAI's Codex CLI, Google's Jules, Devin, and Open Hands can now read an entire repository, plan a multifile change, run the test suite, and iterate on failures with little moment-to-moment supervision. The developer's job is
shifting from typing code to directing agents that type code a change often summarized as a move from code generation to
code orchestration. This paper reviews recent empirical literature to ask what that shift is actually producing.
P. N. Nesarajan, P. Thenmozhi, Shenbaga Priya et al.· International Journal of Inn...· 0 citations
IDE Agent Mode is changing how the developers work, from writing code to debugging & software management, by embedding powerful AI features right inside popular development setups. This article first conceptualizes IDE Agent Mode as a workflow where AIs are coding partners that not only understand the context of the project but are also able to write code, suggest better solutions, automate routine tasks, and even help with debugging at the same time. With the increase in the complexities of software projects and the trend towards shorter development cycles, AI-driven software development has become an effective way to enhance developers' productivity, minimize human errors, and foster innovation. On the other hand, deciding which AI model should be used for which programming task remains one thorny issue that developers and companies face. This is due to significant differences among models in their performance, speed, reasoning capabilities, cost, management of context, and compatibility with the development tools. This article not only lists the AI models that are most often used in IDE Agent environments but also assesses their performance in various coding assistance tasks such as code completion, bug fixing, documentation generation, and architectural reasoning. The method includes a comparative study, obtaining developer feedback, and assessment of the practical workflow in order to determine the pros and cons of different models in real-life software engineering situations. Results show that there is no 'silver bullet' model that fits all requirements; rather, the choice of model should be tailored to the project's needs, the professional level of the team, and scalability requirements, as well as financial constraints. The research also reveals that small models serve very well for fast coding assistance, whereas the more feature-heavy models are quite apt at doing complex tasks such as debugging and system-level designing.
Madhurima Kommuru· International Journal of Mod...· 0 citations
The rapid advancement of LLMs has opened new opportunities in automated software engineering, driving progress in code understanding, agent-based workflows, and productivity tools. However, existing code intelligence systems have largely sidelined the end-users they aim to serve—the developers themselves. Developers exhibit substantial heterogeneity across multiple dimensions: coding style, toolchain preferences, domain-specific expertise, and problem-solving strategies. Failing to account for these individual differences directly compromises both the effectiveness of code intelligence and the likelihood of its adoption. For example, a senior architect and a junior engineer ask: "Describe the authorization module." Without personalized context, the system produces a uniform response—verbose for the expert, incomprehensible for the novice. This gap motivates a fundamental shift: from one-size-fits-all to one-size-fits-one code intelligence. A developer's dynamic in-IDE behaviors—code authoring patterns, navigation pathways, debugging trajectories—implicitly encode a rich representation of their competencies and habits. If captured and interpreted systematically, these signals can enable Personalized Code Intelligence, formalized as: [EQUATION] where P is the developer persona derived from IDE behaviors, injected alongside code context C and instruction ℐ.
Yuhong Liu, Yu Su, Zhipeng Peng et al.· SIGSOFT FSE Companion· 1 citation
AI agents are becoming a fundamental part of modern software creation, helping developers in generating code, debugging, designing systems, etc. But there is a clear difference between how beginners and experienced software engineers get benefits from these tools. Newbies usually depend on agents for one-time prompts and quick answers, whereas mature users utilize them through well-defined, repeated workflows that raise productivity and consistency. In this article, we discuss this difference and emphasize that getting the full potential does not merely depend on better prompts but on workflows driven by instructions developers create clear and reusable instruction files to direct agent behavior across tasks. When developers stop seeing agents only as chat interfaces but as programmable collaborators, they can produce more reliable and high-quality outputs. We offer in our paper methods like designing modular instructions, narrowing down the context, and iterative refinement loops, as well as a case study illustrating how a team made a code review more efficient and minimized the rework by making agent instructions standard. The results stress that structured forms of interaction rather than sporadic use are the main ways to tap into advanced features. Our paper provides a conceptual model for agent usage at large scale, hands-on advice for the implementation of instruction files in actual settings, and validation that skillful developers can far exceed basic usage by adopting orderly, system-like approaches to agent collaboration.
Madhurima Kommuru, Srujana Pulipaka· International Journal of Mod...· 0 citations
