Efficient Retrieval-Augmented Generation for Vulnerability Assessment and Penetration Testing in Automotive Engineering

Abstract

Hundreds of connected components expose vehicle systems to an increasing number of cyber attacks. Vehicle manufacturers must establish the most appropriate procedures for vulnerability assessment and penetration testing using a mix of proprietary solutions and open-source standards. The spread of Large Language Models (LLMs) simplifies the interaction between automotive experts and domain-specific knowledge bases. While proprietary LLM services can be expensive and raise data privacy concerns, open-source LLMs are potentially more cost-effective and better suited to in-house solutions. However, the effectiveness of open-source models in retrieving automotive-related cybersecurity information remains unclear. While adopting open-source LLMs with a few billion parameters, their reasoning and generative capabilities under in-context learning settings are questionable. To bridge this gap, this paper explores efficient solutions for Retrieval-Augmented Generation (RAG) architecture for automotive cybersecurity relying on open-source LLMs. The ultimate goal is to enable cost-effective retrieval and question answering from in-domain knowledge bases, overcoming the privacy and confidentiality issues raised by automotive experts. Using a Graph Knowledge Base designed for a corporate scenario, this paper first defines an expert-curated testing benchmark to evaluate in-domain question-answering performance across multiple aspects. Next, it proposes different RAG system variants based on various retrieval strategies and LLMs, both proprietary and open-source. Finally, it quantitatively evaluates the effectiveness of the content retrieval strategies and compares the pertinence, conciseness, and completeness of generated answers through human validation. Notably, within the scope of the performed analysis, RAGs that rely on open-source models demonstrate promising and competitive performance in some respects compared to the OpenAI GPT model. RAG retrieval performance also surpasses that of state-of-the-art solutions on existing cybersecurity benchmarks (Recall@K above 0.95 vs. 0.65 for state-of-the-art in-domain RAGs).