In the past years, large language models (LLMs) have demonstrated remarkable progress in code generation. However, their ability to reason about program behavior remains an open challenge—an ability that is relevant for applications including reverse engineering, debugging, secure code generation, test-driven synthesis, input reconstruction, reverse fuzzing, behavioral monitoring, and safe execution modeling. To study this ability, we examine the capacity of LLMs to reason about the semantics of code—specifically, their ability to relate code, its inputs, and its outputs to each other. To this end, we investigate whether and how well LLMs can predict one of these three components given the other two—that is, (1) predict the input given code and output, (2) predict the output given code and input, and (3) predict the code given input and output. This way, we assess how well LLMs can reason about and understand the underlying relationships that govern program execution. We construct four datasets covering string processing, array operations, and coding challenges in JavaScript and Python to evaluate diverse program-understanding capabilities, incorporating various code mutation techniques to increase complexity. In our evaluation on tasks covering string processing, array operations, and coding challenges, we find that closed-weight models achieve the strongest performance across all datasets, including perfect input recovery on deterministic string tasks. Across tasks, output prediction is comparatively stable, whereas code prediction remains the hardest setting and often fails for smaller models. Finally, cross-codebase transfer is feasible, especially for input prediction, but highly sensitive to model capacity and fine-tuning strategy.
Norman Becker, Tural Mammadov, Andreas Zeller· AIware· 1 citation
Even in the age of AI, generating comprehensive test inputs for software systems remains a challenge, particularly for interactive and reactive systems, which require entire interaction sequences to reach the desired states or to cover missing behavior. While modern fuzzers are effective at generating random inputs that test the robustness of input processors, they still struggle to produce or mutate complex inputs and interactions. LLM-based systems, on the other hand, can generate small example inputs, but fail to systematically explore the space of inputs and interactions as would be required for comprehensive testing. In this tutorial, we introduce test generation with fandango, a modern specification-based generator of test inputs and interactions. Based on a spec file that defines the structure and properties of the program input, fandango produces inputs that are syntactically and semantically correct, and systematically cover the input space of the program under test. In three interactive sessions, we (1) introduce language-based testing, (2) demonstrate how to use constraints to define desired properties, and then (3) delve into full-fledged protocol testing, producing entire interaction sequences between fandango and the network components under test. Participants are expected to bring a basic understanding of software testing; knowledge of python is a plus, but not required. At the end of the tutorial, they will (1) be able to specify and test formats for inputs and interactions, (2) produce comprehensive test suites using language-based techniques, and (3) understand the trade-offs and practical applications of language-based testing.
José Antonio Zamudio Amaya, Alexander Liggesmeyer, Marius Smytzek et al.· SIGSOFT FSE Companion· 0 citations
