From noisy feedback to evidence-aware issue specifications: an agent-governed retrieval-augmented generation approach
TL;DR
Results show that AGR achieves strong decision accuracy in triage and evidence verification, and produces more actionable and engineering-useful issue specifications than both raw feedback and a strong LLM baseline, while reducing unsupported details.
Abstract
Post-release user feedback is a major control signal for maintenance and evolution in modern software development, yet it is noisy, fragmented, and difficult to translate into developer-usable issue specifications. Large Language Models (LLMs) can assist this transformation, but they often hallucinate or over-commit when evidence is weak, conflicting, or incomplete, limiting their robustness in automated software engineering workflows. We propose AGR (Agent-Governed Retrieval-Augmented Generation), a framework that regulates evidence acquisition and generation decisions via agentic control. AGR first applies an agentic triage step to filter low-signal or off-topic feedback, then retrieves evidence from a three-category hierarchy comprising official documentation, historical bug reports, and targeted web sources. It further performs confidence-weighted fusion across authoritative categories and uses an agentic decision module to verify relevance and sufficiency, trigger additional retrieval or online search when needed, reuse prior reports via memory, and abstain when evidence-supported grounding cannot be established. We evaluate AGR on two open-source software ecosystems, Firefox and VS Code. Results show that AGR achieves strong decision accuracy in triage and evidence verification, and produces more actionable and engineering-useful issue specifications than both raw feedback and a strong LLM baseline, while reducing unsupported details.