Jul 2026· 2026 3rd World Conference on Computer and Information Security (WCCIS)· pp. 221-226· 0 citations· 16 references
Abstract
Software safety function test cases for Lineside Electronic Units (LEU) are commonly written as long natural-language texts. They involve safety chains such as default telegram output, communication interruption, untrusted input failure, open/short-circuit monitoring, and recovery after abnormal conditions. Manual review suffers from low efficiency, insufficient consistency, and limited reuse of domain knowledge. To address these problems, this paper proposes a rule-dominant LEU software safety function test case evaluation method enhanced by controlled knowledge graph reasoning. The method takes benchmark evaluation items as the basic units, integrates structured rules, a state-failure-behavior safety chain, and evaluation profiles to perform basic judgment. For boundary items located in the neighborhood of the coverage threshold, relations among standard clauses, test stages, safety functions, functional transitions, and scenario-closure nodes are used to provide bounded gains. A negative guard mechanism is also introduced to prevent graph-based associative evidence from overriding explicit counter-evidence. Experiments are conducted on a frozen blind validation set containing 24 cases and 187 annotated items. The results show that the fusion-enhanced model achieves a Precision of 0.9606, Recall of 0.8243, F1-score of 0.8873, and Accuracy of 0.8342, outperforming both the rule-based model and the controlled KG-enhanced model in recall, F1-score, and overall judgment consistency.
Large language models are increasingly used to reason about software vulnerabilities, but their outputs can silently violate domain knowledge, limiting their reliability in safety-critical settings such as medical devices. Prior work either treats that output as a prediction to be scored or constrains it to walks within a single knowledge graph; neither checks whether reasoning over a binary is consistent with an independent body of domain knowledge. We present EntailLLM, which validates each LLM-proposed analyst path by entailment: the path is a traversal of the binary's function call graph, the domain knowledge is represented in a separate graph, and verification aligns the two under temporal annotated logic. Across three CWE classes, four LLMs, three prompting strategies, and seven binaries varying in size from 405 to 12,696 function call-graph nodes, domain knowledge raises pooled entailment from 78% to 98%, with entailment decreasing in only 3% of the experiments. EntailLLM is deployed end-to-end on real medical-device binaries, reaching 98% pooled entailment without per-device tuning. Our system inherits the formal guarantees of generalized annotated logic, providing logical verification of LLM output that is both explainable and grounded in well-defined semantics.
Kaustuv Mukherji, J. Patil, C. Payne et al.· 0 citations
Experimental results indicate that the dynamic validation mechanism and minimal target repair strategy can reduce invalid generation while improving the executability, assertion effectiveness, and fault-revealing capability of generated tests.
Xiaorong Zhang, Na Wang, Chunyang Su et al.· International Conference on...· 0 citations
This paper addresses automated unit test generation with large language models (LLMs). LLM-based test generation has not yet attained a quality level sufficient for practical use in industry. Although LLMs often reproduce API syntax faithfully, they frequently disregard semantic usage constraints and execution-environment dependencies, leading to assertion failures, mock-related errors, and reference/resolution errors. A prior failure analysis of Java unit test generation using GPT-4o classified 2980 trials into eight failure patterns and identified three root-cause mechanisms: external context ignorance, internal context ignorance, and a syntax–semantics gap. Building on that analysis, this paper proposes a prompt design comprising three strategies: (1) making the execution state explicit in the generated test, (2) stating semantic constraints explicitly, and (3) injecting environment constraints prior to generation. In contrast to generic techniques such as few-shot learning or chain-of-thought prompting, each proposed strategy is tied to a specific root-cause mechanism, yielding a systematic design in which each rule is explicitly justified by its correspondence to a specific root-cause mechanism. Experiments on 298 methods with five models (GPT-4o, GPT-5, GPT-5.1-Codex, Claude Sonnet 4.5, and Gemini 2.5 Pro) show improved test execution success rates for every model, with absolute gains ranging from 1.1 to 21.1 percentage points (pp). Mock-related errors were reduced by 61.9%–99.2% relative to the baseline prompt, demonstrating effectiveness against the targeted failure patterns. Finally, conditions under which the strategies transfer to other code-generation tasks are discussed, along with limitations on their scope.
This work presents an evidence-carrying validation interface: every selected node-shape check returns either a satisfaction trace or failure witness, and shows how programs combine passing and failing evidence to diagnose missing information and guide repair.
User experience (UX) evaluation is central to software quality, yet it remains difficult to integrate into software engineering workflows in a systematic, explainable, and early-stage manner. This paper presents TraceUX, a framework for operationalizing UX heuristics and design guidance into a rule-based static evaluation pipeline that combines machine-interpretable formalization, executability-aware assessment, context-sensitive scoring, and actionable reporting. The framework is instantiated using Apple Human Interface Guidelines, Swift abstract syntax trees, and mobile games, and implemented in a proof-of-concept tool named TraceHIG. Evaluation was conducted in four layers: analysis of the full rule repository, controlled synthetic validation with injected violations, baseline assessment of 12 public Swift game projects, and a case study on one project. The full repository contained 206 rules; after excluding non-iOS yet platform-specific rules, 193 rules were retained for the downstream experiments. In controlled validation, 216 injected violations yielded 99.2% precision, 61.6% recall, and an F1-score of 0.760. In baseline analysis, overall project scores ranged from 41.6 to 88.0, reflecting rule-conformance spread under the instantiated rule base rather than direct measures of UX quality. The case study demonstrated that profile-aware scoring can yield materially different UX assessments for the same codebase under different game configurations, highlighting the importance of app profiling in static UX evaluation. These findings show that a meaningful subset of UX knowledge can be operationalized into explainable, context-aware static analysis that provides structured and actionable decision support while complementing, rather than replacing, manual and empirical UX evaluation.
This paper proposes a validation framework that combines the flexibility of LLMs with the logic reasoning capabilities of Answer Set Programming as a complementary layer to existing guardrail mechanisms and demonstrates the practical implementation through a modular architecture that supports customizable validation components.
András Gergely Deé-Lukács, Bal'azs 'Ad'am Toldi, András Földvári· Acta Universitatis Sapientia...· 0 citations