Sep 2026· Zenodo (CERN European Organization for Nuclear Research)
Abstract
Benchmark and evaluation harness accompanying the manuscript "Execution-grounded evaluation reveals hidden failures in language-model calculations for environmental science" (under review). Each problem provides only the problem statement (textbook-supplied formulas are stripped); the model writes a Python solve() function, the harness executes it in an isolated sandbox, and the numerical output is graded against certified ground truth at 5% relative tolerance with unit reconciliation. The archive contains the 436 core problems (4,346 graded instances), the execution/grading pipeline, the analysis code behind every reported statistic, and the documentation. Raw per-run model outputs (~5 GB) are deposited separately; see the README. Model API credentials are not included (see models.example.toml).
The results are packaged in the Greenfield Startup Model (GSM), which explains the priority of startups to release the product as quickly as possible, and the need to shorten time-to-market, by speeding up the development through low-precision engineering activities.
Carmine Giardino, Nicolò Paternoster, M. Unterkalmsteiner et al.· IEEE Transactions on Softwar...· 178 citations· ⚡14
Software startup companies develop innovative, software-intensive products within limited timeframes and with few resources, searching for sustainable and scalable business models.
M. Unterkalmsteiner, P. Abrahamsson, Xiaofeng Wang et al.· e-Informatica Software Engin...· 157 citations· ⚡17
This study conducts a case survey study based on the secondary data of the major pivots happened in 49 software startups, and demonstrates that customer need pivot is the most common among all pivot types.
Sohaib Shahid Bajwa, Xiaofeng Wang, Anh Nguyen-Duc et al.· Empirical Software Engineeri...· 127 citations· ⚡15
The comparison of adopter and non-adopter sample reveals three potential adoption inhibitor, security, data privacy, and portability, which underlines the importance of the technical and security perspectives for research investigating the adoption of technology.
Nattakarn Phaphoom, Xiaofeng Wang, S. Samuel et al.· Journal of Systems and Softw...· 111 citations· ⚡8
The ongoing work building a Raspberry Pi cluster consisting of 300 nodes is presented, with potential use cases being an inexpensive and green test bed for cloud computing research and a robust and mobile data center for operating in adverse environments.
P. Abrahamsson, S. Helmer, Nattakarn Phaphoom et al.· IEEE International Conferenc...· 110 citations· ⚡7
The findings show that speed related agile practices are used to a greater extent in comparison to quality practices, and that software startups who adopt the Lean Startup approach do not sacrifice quality for speed more than other startups do.
Jevgenija Pantiuchina, Marco Mondini, Dron Khanna et al.· International Conference on...· 84 citations· ⚡4
