These findings provide the largest empirical characterization of PBT in Python to date, highlight developers’ difficulties in adopting the technique, and expose limitations of current tool support.
Abstract
Property-Based Testing (PBT) automatically generates test inputs to validate properties of programs, shifting developers’ effort from writing examples to specifying invariants. While the technique has gained popularity in Python through the Hypothesis framework, little is known about how developers adopt and use it in practice. This paper reports on three empirical studies. First, we analyzed 367 PBTs from 244 Python projects, classifying them into nine property categories and quantifying their use of Hypothesis constructs. We found that Test Oracle properties dominate (29.97%), and that PBTs are generally concise (median 14 LOC), relying heavily on built-in strategies (75.20%), but also on external (22.62%) and internal (17.17%) ones. Second, we studied 213 Stack Overflow posts tagged with PBT, revealing that the main challenges developers face concern data generation strategies (36.62%), especially for composite and tabular data (24.36%). Finally, we evaluated Ghostwriter, Hypothesis’s automated test generator, against 203 tests from our dataset; only 18.23% were fully automatable, while most required partial adaptation (30.05%) or were incompatible (51.72%). Together, our findings provide the largest empirical characterization of PBT in Python to date, highlight developers’ difficulties in adopting the technique, and expose limitations of current tool support.
Test4Py is presented, a novel framework that enhances type correctness in automated test generation for Python by leveraging the program’s call graph to capture richer contextual information about parameters, and introducing a behavior-based type inference mechanism that accurately infers parameter types and constructs valid test inputs.
Runlin Liu, Zhe Zhang, Yunge Hu et al.· ACM Transactions on Software...· 0 citations
Runtime verification (RV) now scales for testing in thousands of open-source Java projects, helping find hundreds of bugs by monitoring test executions against formal specifications (specs). The popular Python ecosystem could use such benefits. But, current Python RV tools are limited to a domain or spec logic, or they are slow. We present PyMOP, a generic, extensible, and more efficient Python RV tool. PyMOP supports five logics, implements five monitoring algorithms, ships with 81 specs, and supports three instrumentation strategies. On 48,090 unit tests in 839 GitHub projects, we find mainly that (i) PyMOP is up to 419.23x faster than two recent dynamic analysis tools; (ii) Expensive instrumentation is a main cause of RV's runtime overhead for Python; and (iii) 84 of 156 bugs that PyMOP found were confirmed or fixed by developers. PyMOP's genericness and efficiency position it well as an excellent platform for the next advances on RV for Python. PyMOP is at https://github.com/SoftEngResearch/pymop and a video demo is at https://pymop.zhuohangshen.com/demo.
Zhuohan Shen, Mohammed Yaseen, Kevin Guan et al.· SIGSOFT FSE Companion· 2 citations
This work proposes an approach based on a foundation model oracle that analyzes git-style diffs to identify behavioral changes introduced by Python refactorings and uncovered 13 distinct bugs among the seven refactoring types studied.
Jonhnanthan Oliveira, Rohit Gheyi, Márcio Ribeiro et al.· 0 citations
Validating functional correctness of mobile apps is challenging for manual testing and automated UI testing (limited to simple crashing bugs). Property-based testing is promising to tackle this challenge given functional properties. To this end, we introduce Kea2, a practical property-based testing tool for apps: (1) specifying properties in Python with enough flexibility and expressiveness; and (2) reusing existing GUI fuzzing techniques to support effective property checking. Indeed, Kea2 can find functional (logic) bugs in real-world apps. Kea2 has been open-sourced at https://github.com/ecnusse/Kea2 (a demo video: https://youtu.be/HS4rTCcaSPE), and received positive feedback for its usability and features.
Xixian Liang, Cheng Peng, Bo Ma et al.· SIGSOFT FSE Companion· 0 citations
Future TOG systems should be evaluated not only by whether they predict the correct oracle type, but also by whether their predictions are grounded in meaningful exception-triggering evidence, to challenge the assumption that strong exception-oracle accuracy reflects robust use of exception semantics.
Soneya Binta Hossain, Matthew B. Dwyer, Tasfia Tasnim· 0 citations