This work proposes a stateful, multi-agent validation pipeline that eradicated cross-phase hallucinations and proves adversarial auditing enables LLMs to reliably synthesize zero-error MBSE architectures.
Abstract
While Large Language Models (LLMs) can accelerate Model-Based Systems Engineering (MBSE) for software-defined vehicles, their probabilistic nature causes"architectural drift", fabricating interfaces in behavioral views that lack structural foundations. To enforce deterministic interface alignment, we propose a stateful, multi-agent validation pipeline. The framework utilizes a sequential generation matrix (Class->Activity->Sequence) and Vehicle Signal Specification (VSS)-grounded Retrieval-Augmented Generation (RAG). An independent AI Validator Agent dynamically audits outputs against a strict error taxonomy, triggering state-preserving backtracking loops to resolve incompatibilities. Evaluated on an Advanced Driver Assistance System (ADAS) scenario, standard RAG yielded 0% Entity Traceability. Conversely, our multi-agent workflow eradicated cross-phase hallucinations, achieving 97% Entity Traceability, 87% Signal Conservation, and an 85% F1-score. This proves adversarial auditing enables LLMs to reliably synthesize zero-error MBSE architectures.
Multi-agent systems built on large language models (LLMs) are increasingly deployed for complex tasks requiring autonomous planning, tool use, and inter-agent coordination. However, the non-deterministic nature of LLM outputs and the emergent behavior arising from agent interactions render traditional test oracles ineffective, creating a critical gap in quality assurance for agentic AI. This work introduces MORPHAGENT, a framework designed to address the oracle problem in multi-agent LLM systems through trace-based behavioral analysis. Our contributions are threefold: (1) goal-preservation relations that verify consistent goal achievement under input perturbations, (2) coordination-consistency relations that validate inter-agent delegation and communication patterns under agent substitution and reordering, and (3) tool-use integrity relations that ensure semantic equivalence of tool invocation sequences under prompt paraphrasing. MorphAgent instruments agent execution to capture structured traces comprising planning steps, tool calls, message exchanges, and final outputs, then systematically applies metamorphic transformations and checks behavioral invariants without requiring ground-truth oracles. We evaluate the framework on four multi-agent benchmarks spanning code generation, research synthesis, customer service, and data analysis tasks, encompassing 2,840 source-followup execution pairs across three LLM backends. Results show that MORPHAGENT detects 82.0% of seeded behavioral faults, including 90.3% of coordination failures and 81.7% of goal-deviation faults, while maintaining a false positive rate of 6.1%. The framework uncovers 14 previously unreported behavioral anomalies in established multi-agent frameworks, demonstrating its practical utility for assuring agentic AI reliability. These results suggest that trace-based metamorphic testing can serve as a practical foundation for reliable validation of emerging agentic AI systems.
Gopalakrishnan Marimuthu· International Conference on...· 0 citations
Multi-agent large language model(LLM)systems are applied to structural design,yet most use one-shot generation and cannot verify their output,leaving themill-suited to safety-critical tasks.Rather than trusting LLM self-correction,thisframework injects feedback from an external physics-based verier into a closedrepair loop.The framework couples a three-layernite-element verication systemwith a dual-node loop.Node 1 turns code violations into hard repair constraints,Node 2 turns a four-dimensional quality score into safety-rst soft constraints,and a retrieval-augmented code base makes every violation traceable to a clause.Overve structure types and 44 cases,code compliance rises from 56.8%to 98.6%and the composite score from 63.8 to 71.4(p<0.000001),using about 5.8%lessmaterial.Removing either node degrades performance,and compliance does notchange detectably across the two backbone LLMs tested,indicating that it ishere attributed to the external verier rather than the model.The framework,the 44-case benchmark and all experiment scripts are released as open source forreplicability.
Jianbin Luo, Wei-Wei Lin, Yiran Lin et al.· 0 citations
RedAgentBench is introduced, an executable framework for autonomous red-teaming and faithful measurement that shows that executable evaluation can improve safety measurement and identify actionable intervention points.
Zixing Chen, Xingyuan Liu, Jie Zhu et al.· 0 citations
CyberLLM is presented, a multi-agent, LLM-orchestrated framework that autonomously detects vulnerabilities and executes remediations under a formal, runtime safety guard, and indicates that LLM agents can perform useful autonomous cyber-defense when wrapped in a deterministic, auditable safety envelope.
Nenad Petrovic, Oussama Jeddou, Feres Ben Fraj et al.· 0 citations
As the global software infrastructure ages, the modernization of legacy systems—particularly the migration of monolithic Java 8 applications to contemporary Long-Term Support (LTS) versions like Java 17 or 21—has become a critical imperative for security, performance, and maintainability. While Large Language Models (LLMs) have demonstrated exceptional proficiency in isolated code generation tasks, their application to repository-scale migration is severely constrained by "contextual hallucinations"—a phenomenon where models generate syntactically correct but structurally invalid code due to a lack of global state awareness. Recent empirical studies (e.g., Collu-Bench [1]) indicate that determining the hallucination status of generated code remains an unsolved problem, with prediction accuracy often below 33%. In this work, we introduce TriArchitect, a novel shared-state multi-agent framework designed to bridge the gap between probabilistic reasoning and deterministic structural integrity. TriArchitect introduces three key innovations: (1) the Typed Migration Graph (TMG), a formal directed acyclic graph that serves as a persistent semantic memory of the migration state; (2) a specialized role-based multi-agent architecture comprising an Archeologist (analysis), Architect (synthesis), and Validator (verification); and (3) a rigorous Validator-Veto Protocol, which enforces a "compilation-as-consensus" rule to reject hallucinated proposals before they corrupt the codebase. We evaluate TriArchitect on J8-to-J17-Bench, a comprehensive benchmark of 1,000 method-level and class-level migration tasks. Results demonstrate a System Success Rate (SSR) of 68.4%, statistically outperforming significant baselines including OpenRewrite (62.0%), SWE-Agent (63.5%), AutoCodeRover (64.8%), and raw GPT-5.1 (65.2%), while reducing token consumption by 40% through state-aware prompting.
Neeraj Kumar Singh Beshane· International journal of com...· 0 citations
Agent Gym is introduced, a modular, domain-agnostic framework that wraps any existing LLM-based agent in a continuous evaluation-and-evolution loop and introduces the Spec-to-Note Gap, an autoencoder-inspired view of agentic system transparency.
Pouya Ghiasnezhad Omran, Michael Zimmermann, Duncan Cambridge et al.· 0 citations