TL;DR
These findings provide practical guidance for selecting LLM judges, designing role prompts, and employing multi-judge voting strategies in “automated software quality assurance”.
This survey provides a comprehensive overview of recent advances in LLM-based evaluation, covering techniques, applications, and challenges across domains, with future directions emphasizing standardized protocols, uncertainty estimation, and human–AI collaboration.
Difficulty differences across parallel-class programming examinations affect the fairness of course assessment. This study repositions large language models from benchmark evaluation targets to auxiliary evidence sources for interpreting exam difficulty, combining AI evidence with aggregated student performance, item exposure, online-judge process data, and teacher interpretation. First, ten models solved an eight-problem final exam synchronously with 120 students: AI pass rate correlated positively with student pass rate (Spearman rho = 0.866, exact p = 0.0119), and a solving-based composite difficulty index correlated negatively with it (rho = -0.905, exact p = 0.0046). A single structured reviewer was then run via auditable API calls on a third-party OpenAI-compatible endpoint whose model label (gpt-5.6-sol) cannot authenticate an official OpenAI upstream model; call metadata and raw responses are archived. Across 79 problems from 11 parallel-class final exams, AI overall difficulty correlated with problem-level pass rate at rho = -0.871 and with non-attempt rate at rho = 0.800; in a 26-problem longitudinal Data Structures and Algorithms B sample, the correlations were -0.829 and 0.883. A 106-problem introductory-course (CS101) sample marks the boundary: the problem-level correlation weakened to rho = -0.552, and the exam-level correlation across 16 exams was near zero, with cohort composition dominating exam-level outcomes. Exposure-discount (0-0.40) and duplicate-problem perturbation tests did not change these directions. AI evidence can thus serve as an external reference for problem validation, parallel-class fairness discussion, and longitudinal quality tracking, while the model-identity boundary, single-reviewer design, and review-output instability set explicit limits: AI difficulty scales must not be used for individual student evaluation or automatic grade adjustment.
It is shown that prior scores, even when included only as context metadata, anchor judgments and systematically shift ratings toward their values, and effective mitigation must be validated for the intended model and task or domain.
Software code testing remains a critically important but labour-intensive process in software quality assurance. Existing research evaluates large language model (LLM)-based unit test generation using various quality metrics, such as correctness, coverage, mutation score, and test code smells. However, these single metrics do not reflect the trustworthiness of the unit test generation process. Therefore, this research formalises the trustworthiness of LLM-based unit test generation as a multidimensional index comprising reliability, hallucination resistance, maintainability, functional completeness, and human-reference alignment. In this research, we investigate the effect of prompt engineering strategies on the trustworthiness of LLM-generated unit tests and compare them with human-written tests for the same focal methods. Each dimension is fed by a distinct artefact-level measurement and grounded in dependability theory and ISO/IEC 25010:2023. A centralised multi-agent system generates, builds, repairs, and measures the tests, so that all inputs are collected automatically. The index is evaluated on real-world C# focal methods across 18 model × prompt configurations and a paired human-written baseline. The human baseline achieves the highest T-UTG value (0.904), and the best configuration, Combined × Gemini, achieves 0.788. Entropy weighting identifies maintainability and hallucination resistance as the most discriminating dimensions, and a rank-acceptability analysis over the whole weight simplex confirms that this ordering does not depend on the chosen weighting scheme.
Introduction We investigate how expressive inline code comments written in various developer styles, functional to progressively poetic, philosophical, and misleading, affect large language model (LLM) behavior during code optimization. Methods In this pilot study, we used a controlledmerge sort implementation across five stylistic variants and evaluated GPT-5 and Claude Opus 4.1 under standardized console prompts, isolating the effect of embedded comment semiotic variation. Seven expert developers (three senior, four mid-level) scored model outputs against adapted ISO/IEC 25010 criteria and novel LLM suggestibility index (LSI) framework. Results Semiotic character of comments measurably altered code quality, with consensus-score reliability ICC(2, k) = 0.65–0.81 for six of seven dimensions; single-rater Krippendorff's α = 0.232 reflects substantial interpretive variability. Claude exhibited higher interpretive sensitivity (mean behavioral divergence 4.00; SD 1.16), while GPT-5 maintained stronger architectural fidelity (mean divergence 3.58; SD 1.26). Reflective comments (philosophical, conversational) were associated with Claude's highest maintainability scores in our panel (both M = 4.00, ~8% above stock M = 3.71), while the same philosophical comments reduced GPT-5 maintainability (M = 2.86), suggesting asymmetric model responses to expressive context. Conclusions These findings position inline comments as model-sensitive latent semantic prompts, with implications for AI-in-the-loop development and design of comment conventions for AI-assisted maintenance.
LLM-as-a-judge evaluation is usually assessed by agreement and robustness to surface perturbations, but reliability does not establish construct validity. We formalize construct validity for an evaluator as a two-dimensional profile: invariance S, the probability that a verdict is unchanged under construct-preserving edits, and construct sensitivity R, the probability that it changes under minimal construct-changing edits. We show that S and R are independent and that no scalar summary preserves all relevant comparisons. We measure the profile across 7 judges and 4 domains using 7 construct-changing intervention types and 5 register-only controls, with intervention direction determined by human annotators and generation, verification, and judging assigned to disjoint model families. At matched invariance S>= 0.90, judges average S = 0.945 but R = 0.319. Sensitivity also differs between scope and strength edits: R_scope = 0.383 versus R_strength = 0.262, a +0.121 gap with the same sign for all 7 judges. We further audit five public label sets and find that surface-only predictors reproduce 55%-67% of labels in paired mode, including 67.4% of MT-Bench human votes. These results show that high judge agreement can coexist with weak sensitivity to changes in the construct being evaluated, motivating joint reporting of invariance and sensitivity and auditing the validation set itself.
In his latest book, Professor Eugene Fitzgerald examines the forces that turn breakthroughs into value — and why innovation resists simple formulas.
MIT students designed, built, and tested a jet engine with AI copilots, assessing AI’s usefulness in developing high-performance aerospace systems.
Researchers developed an auditing technique to test generative AI models for malicious capabilities, without prompting them for illegal outputs.
A USAF cadet and a Lincoln Laboratory researcher found AI chatbots can help nontechnical service members produce viable software applications for their unique problems.
