An Empirical Study of Large Language Models for Risk Verification in Automotive Binary Programs

Abstract

Large Language Models (LLMs) have become a key driver of productivity advancement; however, research on LLMs in the context of automotive conformity assessment remains limited. This paper investigates the application of LLMs to risk verification under automotive conformity assessment. First, through expert manual analysis, a benchmark dataset containing 274 automotive programs was constructed. Based on risk reports generated within automotive conformity assessment, an LLM-based risk verification framework was then proposed, which divides risk verification into three stages and incorporates a system prompt together with prompt templates for the three reasoning stages. Finally, nine popular open-source LLMs were selected for experiments under the proposed risk verification framework. Experimental results show that, for the risk verification task, GLM-5 demonstrates the best overall performance among the evaluated open-source LLMs, achieving an average precision of 88.20%, an average recall of 74.72%, and an F1-score of 0.809 across five experimental runs. Kimi-K2.5 and qwen3-coder-480B-A35B-Instruct also show relatively strong performance, with F1-scores of 0.7477 and 0.7378, respectively. Within the evaluated model set, several MoE models achieved relatively high observed risk-verification performance; however, this pattern is reported as an empirical observation of the selected models rather than evidence of an intrinsic architectural advantage. However, current open-source LLMs still suffer from false negatives, and therefore they are more suitable for use as supporting models in human-assisted and evidence-driven automotive risk verification workflows, rather than as standalone replacements for expert assessment or existing binary analysis tools. We also discuss practical considerations and limitations of applying the proposed framework, and provides empirical evidence for using LLMs as evidence-driven assistants to help automotive OEMs and assessment organizations reduce the manual cost of false-positive verification.