The evaluation shows that IntelliAudit can support control interpretation, evidence-grounded reasoning, and audit-preparation workflows, while also revealing the importance of human oversight for calibrating sufficiency judgments and correcting overly permissive recommendations.
Abstract
IT audits require auditors to judge whether heterogeneous organizational evidence satisfies semantic security and compliance controls. This judgment is difficult to automate because relevant evidence is distributed across policies, records, spreadsheets, and operational artifacts, and because audit conclusions depend on evidentiary sufficiency rather than keyword matching. We present IntelliAudit, a retrieval-grounded multi-agent system for IT audit evidence evaluation. Given a control and an evidence corpus, IntelliAudit retrieves relevant artifacts, generates an evidence-grounded assessment, challenges adverse findings, adjudicates disagreements, and produces an auditor-facing recommendation with cited evidence, rationale, missing-evidence analysis, and remediation guidance. We instantiate IntelliAudit on ISO/IEC 27001 and evaluate it across multiple simulated organizations using expert auditor review and audit-readiness user feedback. The evaluation shows that IntelliAudit can support control interpretation, evidence-grounded reasoning, and audit-preparation workflows, while also revealing the importance of human oversight for calibrating sufficiency judgments and correcting overly permissive recommendations. These results suggest that retrieval-grounded multi-agent systems can assist audit evidence review, but should remain decision-support tools rather than autonomous certification systems.
V VERA-8B is a new end-to-end audit reasoning system that identifies audit risks before enforcement actions occur, and is the first to unify SFT and GRPO for evidence-grounded audit reasoning under one evidence standard.
As AI governance moves from benchmark scores toward auditable oversight, a central question is how reviewers can tell whether an LLM-generated audit report is actually supported by evidence. This paper studies that question in passage-anchored policy audits, where a report must interpret a given policy passage and cite evidence for its claims. We introduce a controlled evaluation framework that holds the passage, rubric, and auditor model fixed while changing only the evidence interface supplied to the auditor. Across 60 AGORA policy cases, we generate 600 structured reports under ten evidence conditions, including passage-based evidence, internal model evidence, a hybrid package, and a shuffled control that preserves evidence format while breaking case relevance. Five human reviewers evaluate the primary interfaces for correctness, passage grounding, diagnostic usefulness, and evidence misuse. The results show that internal evidence changes how reports cite and reason about evidence, but more internal citations do not by themselves make a report more valid. A white-box diagnostic explains the failure mode: causal localization is narrow, while reports readily reuse broader readable labels and token directions. The hybrid interface is the most useful on average, while the shuffled control exposes a key governance risk: reports can sound substantively plausible while citing irrelevant internal evidence. This study reframes internal model access as an evidence design problem for audit workflows, rather than as a guarantee of transparency.
Human action drives most cybersecurity breaches, yet industry reports rely on vague labels that lack diagnostic utility. This study evaluates whether public incident narratives from the VERIS Community Database provide the context required for systemic intervention to 45 Action.Error records from 2020 to 2021 were analyzed across three error varieties: misdelivery, misconfiguration, and publishing. Using the Human Factors Analysis and Classification System, we applied a strict evidence-based coding strategy to map narrative to systemic levels. Our results reveal a significant diagnostic gap: while narratives consistently support Level 1 (Unsafe Act) coding, evidence for latent preconditions, supervision failures, and organizational influences remains largely absent. Specifically, misdelivery and misconfiguration align cleanly with skill-based and decision errors, but the causal chain stops at the individual. These findings suggest that current public reporting can identify the type of error that occurred, but is limited in explaining why it occurred.
Saroja Roy Grandhi, Jeremiah D. Still· Proceedings of the Human Fac...· 0 citations
: Large language models and retrieval-augmented generation (RAG) systems are increasingly employed to transform evidence into decision-facing briefings, alerts, and recommendations. In these settings, explainability cannot be evaluated merely by fluency, readability, or factual correctness. A briefing may be factually correct while still being unsafe if it cites sources that do not substantiate the claim, suppresses uncertainty, converts correlational evidence into causal language, recommends an unauthorized action, or leaves no auditable path for human review. This review synthesizes 104 sources spanning explainable natural language processing (NLP), faithful explanation, hallucination and factuality evaluation, RAG, citation faithfulness, uncertainty communication, causal language, human–AI interaction, engineering and regulatory decision support, and institutional accountability. It makes four contributions. First, it defines evidence-grounded decision briefings as a distinct NLP setting characterized by identifiable evidence inputs, constrained decision-facing outputs, and minimum accountability requirements. Second, it proposes a five-layer taxonomy encompassing evidence representation, explanation generation, retrieval and source grounding, verification and evaluation, and human accountability. Third, it develops an operational evaluation framework for claims, citations, uncertainty statements, causal wording, action labels, and complete briefing episodes. Fourth, it complements the conceptual synthesis with source-level trend analyses, targeted quantitative comparisons, and representative use cases drawn from generic, engineering, and regulatory contexts. The synthesis reveals that existing surveys provide critical foundations but do not jointly address five interdependent requirements: citation-to-claim entailment (whether the cited evidence supports the exact claim), causal-language discipline, uncertainty preservation, action appropriateness, and human accountability in decision-facing generated text. This review concludes with open challenges for claim segmentation, retrieval adequacy, citation-to-claim
Jihoon Moon· Computers, Materials & C...· 0 citations
Freshness-constrained audit capacity (FCAC) is developed, a decision-support framework that treats automation as an authorization decision constrained by action risk, evidence freshness, and shared review capacity.
Large language models are increasingly used to review, clarify, rewrite, and trace software requirements. These applications create a governance problem that output-quality assessment alone cannot resolve: a fluent proposal may rely on inadmissible evidence, alter stakeholder intent, introduce unsupported specificity, or imply an organizational commitment that the model has no authority to make. Existing work on retrieval-augmented generation, controlled natural language, formal verification, human oversight, and AI governance supplies relevant controls, but it does not specify the procedural status of an individual LLM proposal relative to a controlled requirements artifact. This article develops an artifact-centered, human-in-the-loop framework in which the permitted influence of a proposal is the primary object of governance. The framework combines five governance functions—governed evidence, bounded context construction, controlled LLM analysis, pre-commit verification, and accountable human approval—with four artifact-influence states: A0 advisory observation, A1 evidence-linked candidate, A2 verified recommendation, and A3 approved and committed change. Its central theoretical claim is that output quality, evidential legitimacy, verification status, and authority to commit a change are distinct properties and should not be collapsed into a single confidence judgment. Seven falsifiable hypotheses translate the model into measurable comparisons involving source admissibility, context leakage, unsupported specificity, semantic drift, reviewer agreement, unreviewed changes, governance cost, and organizational maturity. Human review is treated as both a necessary decision boundary and a potential source of automation bias, anchoring, and fatigue. The framework is conceptual rather than empirically validated and provides a basis for controlled experiments, field studies, and longitudinal evaluation.
Chuanjin Zhu· Advances in Engineering Inno...· 0 citations
