DevEx Metrics Compass is a public, open-source web app built on a structured analysis of that landscape across more than 50 engineering organizations, and what the dataset reveals about how DevEx is measured today and where the gaps lie is shared.
Abstract
As AI-augmented development becomes standard practice, engineering leaders face mounting pressure to demonstrate its impact. Yet most organizations are measuring AI adoption and output while the effect on developer experience (DevEx) remains largely unknown. The right metrics can close that gap, surfacing the everyday friction developers face and monitoring the real impact of initiatives such as AI rollouts. But with more than 120 metrics in circulation across frameworks, companies, and scientific literature, identifying the right, meaningful, actionable ones for a specific context has become a challenge in itself. This article introduces DevEx Metrics Compass, a public, open-source web app built on a structured analysis of that landscape across more than 50 engineering organizations. Alongside the tool, we share what the dataset reveals about how DevEx is measured today and where the gaps lie. Whether you are new to DevEx measurement and looking for a first meaningful set of metrics, or a seasoned practitioner assessing whether an existing collection has the right breadth and depth, Compass helps teams navigate the fragmented landscape and identify measures that actually matter for their context and goals.
The recent meteoric rise of LLMs (Large Language Models) and associated tools was largely unexpected and surprising to most. The rapid ascent of this technology has caught many software developers unawares, leaving them suddenly somewhat ignorant, and arguably under-skilled.
LLMs, whilst still advancing, have recently demonstrated impressive capabilities in their ability to assist software developers in their day-to-day tasks (e.g., coding new features, and locating and fixing issues). However, the use and adoption of LLMs presents many larger challenges for society as a whole; many of which are not in themselves technical concerns.
This paper examines the current and perceived impact of this technology in the context of Open Source. We identify several social, economic, environmental, political, legal, and technical concerns regarding the use of LLMs in Open Source projects.
We contribute guidance around defining an AI Policy for Open Source projects. We further offer an AI Policy Score Card to assist projects in clearly defining and declaring how they wish to work with AI or not.
Adam Retter· Balisage Series on Markup Te...· 0 citations
Software defect prediction historically relies on code and history metrics, largely ignoring developer-level indicators. Yet, because defects typically stem from human error, human-factors theory offers complementary, theoretically grounded predictors. This paper introduces a framework for deriving prediction metrics from human-factors theory and instantiates it with two metrics, alertness and memory_decay, operationalizing developer cognitive state during code editing. We evaluate these metrics on twenty-one large-scale open-source projects under two complementary protocols: a discovery-oriented (in-sample) evaluation and a time-aware generalization evaluation on a subset of eighteen projects. Under discovery evaluation, models using cognitive metrics dominate state-of-the-art baselines on every project (PR-AUC 0.64 vs. 0.43), and have surfaced over thirty real, previously-unreported defects across nine critical infrastructure projects, including two assigned CVEs and accepted upstream fixes in systems such as NASA Trick. Under time-aware cross-project evaluation, cognitive metrics encode the strongest cross-project predictive signal in the benchmark: alertness ranks first by mean SHAP importance and memory_decay exhibits the lowest rank volatility, a stability independently confirmed by chance-corrected selection-stability analysis; the configuration combining cognitive and traditional metrics significantly outperforms the traditional baseline alone (Holm-corrected \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p = 0.018$$\end{document}, large effect). Furthermore, cognitive configurations demonstrate a significant, large-effect reverse generalization gap—performing better under cross-project transfer than within-project evaluation—while substituting traditional edit-frequency counts to produce calibration-neutral models. Consequently, our findings validate cognitive theory for explainable, actionable, and interpretable safety-critical defect prediction, laying empirical groundwork to evaluate analogous issues in LLM-generated code through the behavioral study of AI.
Carlos Andrés Ramírez Cataño, Makoto Itoh· International Conference on...· 0 citations
This paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face, and describes a seven-stage workflow and five process shifts, including a move toward evaluation-driven development.
Yunbo Lyu, David Williams, Jieke Shi et al.· 0 citations
AI-generated C++ code has a distinct quality profile, showing higher rates of interface and coupling burdens, copy and allocation overheads, and a reliance on explicit loops over optimized standard APIs, which translates into tangible downstream costs, including increased review effort and a 5-8% increase in compute resource consumption.
Michael Tran, Fred Lewis, Kun Yang et al.· 1 citation
Background: The rapid advancement of large language models (LLMs) has given rise to AI-native applications, a new paradigm in software engineering that fundamentally redefines how software is designed, developed, and evolved. Despite their growing prominence, AI-native applications still lack a unified engineering definition and architectural blueprint, leaving practitioners without systematic guidance for system design, quality assurance, and technology selection. Objective: This study seeks to establish a comprehensive understanding of AI-native applications by identifying their defining characteristics, key quality attributes, and typical technology stacks, as well as by clarifying the opportunities and challenges they present. Method: We conducted a grey literature review, integrating conceptual perspectives retrieved from targeted Google and Bing searches with practical insights derived from leading open-source projects on GitHub. A structured protocol encompassing source selection, quality assessment, and thematic analysis was applied to synthesize findings across heterogeneous sources. Results: We finally identified 106 studies based on the selection criteria. The analysis reveals that AI-native applications are distinguished by two core pillars: the central role of AI as the system's intelligence paradigm and their inherently probabilistic, non-deterministic nature. Critical quality attributes include reliability, usability, performance efficiency, and AI-specific observability. In addition, a typical technology stack has begun to emerge, comprising LLM orchestration frameworks, vector databases, and AI-native observability platforms. These systems emphasize response quality, cost-effectiveness, and outcome predictability, setting them apart from conventional software systems. Conclusion: This study is the first to propose a dual-layered engineering blueprint...
Lingli Cao, Shanshan Li, Ying Fan et al.· 1 citation
VulnScanner Pro is a self-hosted assessment tool that tries to sit in the gap between the two extremes of manual penetration testing and automated security assessment, and recommends four recommendations for fixing the six missing HTTP security headers.
Athili Laxmi Lavanya, Mycharla Madhavkumar, D. Gangadhar· International Journal for Re...· 0 citations
