TL;DR
The findings suggest that LLM evolution translated to meaningful improvements on structured modeling tasks for some of the families, but that progress is neither universal nor monotonic.
Whether contemporary LLMs can reproduce the research outcomes of a fully documented human study: a 1991 article that identified dermatophytosis (ringworm) in historical fine art was evaluated.
With large language models (LLMs) becoming ever more popular and their usage expanding into various domains, this study explores how effective an LLM like GPT-4 would be in analyzing requirement specification documents and using relevant information from those to create a tradespace matrix. This study compares GPT-4's performance with that of 30 human participants from diverse backgrounds, categorized into three groups: engineering, non-technical and (CS/AI). Each participant and the LLM completed a survey based on materials provided for evaluating a complex system design and populating a trade-space matrix. The analysis of these responses included within-group heatmaps, across-group comparisons, and human-versus-GPT-4 heatmap evaluations. The results show that the CS/AI group had the most closely aligned responses with GPT-4. This study demonstrates how LLMs can augment trade-space exploration and serve as an alternative in cases where employing a team of human experts from multiple backgrounds is not feasible.
Software testing prediction models play a critical role in improving software quality, reducing faults, and optimizing testing. Although past studies show that these models have improved over time, little focus has been given to them. Many studies rely on limited datasets that do not fully capture the complexity of real software systems, which limits how well the models can generalize. There is also insufficient evidence on how these models use existing datasets in practical, real-life settings, since most evaluations are conducted under controlled or experimental conditions. In addition, key aspects such as interpretability, generalizability, and practical usability are still not adequately addressed, which reduces trust and slows down adoption in practice. This study presents a systematic literature review of recent empirical research on predictive approaches in software testing, focusing on machine learning, deep learning, and hybrid techniques. A structured methodology was used, including clearly defined inclusion and exclusion criteria, systematic searches across major academic databases, quality assessment, and data extraction from 22 selected studies. The analysis considered model types, datasets, feature methods, evaluation metrics, and methodological approaches. The findings show a shift from traditional statistical models to advanced artificial intelligence techniques, including graph neural networks, contrastive learning, and deep fuzzy clustering. In addition, optimization and data balancing techniques improve predictive performance, while explainable artificial intelligence enhances model interpretability. However, challenges such as limited cross-project generalization and insufficient industrial validation still exist.
As large language models (LLMs) become increasingly integrated into analytical workflows, an urgent question arises: Can these models replace the trained statistician? This paper presents a controlled experiment to directly test the statistical reasoning ability of LLMs. We employ Monte Carlo simulations to generate datasets with known ground‐truth parameters and pose four canonical statistical questions to five commercially prominent models across three linguistically distinct prompt formulations and five sampling temperature settings, yielding 3000 observations in a full‐factorial design. In order to find an answer to our question, we design prompts that reflect how decision‐makers with varying degrees of statistical knowledge would query AI in the absence of a trained statistician. We find that prompts that portray higher statistical competency can result in higher accuracy for some (but not all) LLMs; we also find that LLMs can fail catastrophically on tasks requiring quantitative precision. We connect these findings to architectural differences among models and to recent literature on epistemic mirroring in LLMs and argue that the observed patterns reveal models are performing linguistic pattern matching on statistically flavoured text rather than genuine statistical reasoning. We conclude that current LLMs cannot replace the statistician, though certain architectures approach useful performance on pattern‐recognition subtasks.
Evaluating generative AI models is a routine, but resource-intensive, process that is conducted over and over again during the course of model development. In this work, we propose Collaborative Evaluation (CollabEval), a simple, effective, and principled method for exploiting dependencies between historical runs of different models on the same tasks to improve statistical efficiency. Specifically, our approach treats model evaluation as a matrix completion problem over an $M \times N$ matrix of evaluation scores, where $M$ is the total number of models and $N$ is the total number of evaluation prompts. We assume that a subset of these $M$ models are targeted for evaluation. For these target models only a small fraction, $p$, of prompts has been annotated with evaluation scores. Leveraging recent results in prediction-powered inference, we build a low-rank approximation of the score matrix, and use the reconstructed values as control variates in a manner that guarantees unbiased estimates of the true evaluation metric mean, in addition to statistically valid confidence intervals. Empirically, across a wide range of datasets, models, and sparsity levels $p$, we find that CollabEval substantially reduces the mean confidence interval size, and the mean squared error of the point estimate, compared to baseline methods at the same annotation budget.
This paper constructs a large-scale dataset of agent applications, tools, and tests, and manually label 2,572 test methods from 240 modules, and derives a taxonomy of 23 testing patterns across test fixtures, data, objectives, and assertions, and characterize tests by level.
