Measuring What Matters in the Age of AI: The Research Software Metrics Landscape

Abstract

Research software now serves as a primary instrument of science, but the methods the community uses to judge its quality, sustainability, reuse, and impact remain scattered across software engineering, open science, FAIR software, citation, and research-assessment literature. AI-assisted development sharpens the problem: code can be produced at rates that outpace human review capacity, provenance and attribution are harder to track, scholarly credit systems cannot keep pace, and existing metrics inherit assumptions about how research software gets developed that may no longer apply. To aid in navigating these challenges, we present a focused landscape synthesis of research software metrics based on 49 substantive references, organizing the recurring metrics into eight families described by data source, claimed construct, evidence of validity, automation potential, and susceptibility to misuse. Many widely cited research software metrics serve as partial signals but make weak standalone measures of scientific impact. The AI era opens or widens gaps around provenance, maintainability of generated code, reviewer effort, model and data dependencies, reuse and community-health signal inflation, and credit for human labor that AI does not perform. We argue for responsible, contextual, mixed-method use of research software metrics, and offer decision guidance for Research Software Engineers (RSEs), principal investigators, institutions, and funders selecting metrics that support better software, fairer credit, and more trustworthy science. All findings rest on the 49-reference focused review; a reproducible search-and-triage protocol and a 727-record triaged corpus accompany the paper as a transparency artifact for a future full systematic mapping study.