Skip to content
Open access

Falsdo: Benchmarking Artifact-Controlled Multimodal Fake News Verification via Failure-Aligned Auditing

Sep 2026 · IJCAI/ECAI · pp. 2474-2482 · 0 citations · 42 references
Computer Science

TL;DR

This work introduces Falsdo, a diagnostic benchmark designed to make robustness and auditability identifiable and proposes DREA, a failure-aligned evidence-auditing baseline that improves instruction-level grounding (Joint-F1) and minimizes degradation under artifact-controlled settings.

Abstract

Recent generative AI renders multimodal misinformation structurally harder to detect, making reliable detection dependent on semantic verification grounded in verifiable evidence. However, current benchmarks often fail to isolate true semantic checking from superficial shortcut exploitation. We introduce Falsdo, a diagnostic benchmark designed to make robustness and auditability identifiable. Constructed via a heterogeneous web-mining pipeline, Falsdo provides instruction-conditioned grounding and formalizes a counterfactual artifact-control protocol. This stress test reveals a critical failure: when low-level generation traces are equalized, detector performance degrades, indicating reliance on artifacts rather than robust verification. To enable reproducible diagnosis, we propose DREA, a failure-aligned evidence-auditing baseline. DREA specifies evidence acquisition with explicit noise control and implements constrained channel-wise auditing with reliability-aware late fusion. Experiments demonstrate that Falsdo reliably exposes shortcut dependence, while DREA improves instruction-level grounding (Joint-F1) and minimizes degradation under artifact-controlled settings.

Read PDF

Similar papers

Open access 2026

Auditable and Robust LLM-Based Phishing Detection via Provenance-Guided Evidence Contracts

: Large language model (LLM)-based phishing detectors can produce decisions and explanations, but adversarial webpages violate the assumption that all input text is trustworthy evidence. Hidden comments, metadata, scripts, Cascading Style Sheets (CSS), and retrieved snippets can act as instruction-injection channels, w...

Yue-Kui Xu, Xiong-Zhang Luo · 1 citation
#artificial intelligence Review Oct 2026

Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

Coding agents can return plausible patches that omit required behavior. These failures are hard to review because long traces and confident summaries often hide what was missed. We ask when a nominally weaker reviewer can reliably decide whether a patch solves its issue. We study 411 execution-labeled traces from three...

Jun-Yu Guo, Shangding Gu, Ming Jin et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Logic Hijacking of RAG: Single-Document Poisoning via Local Optimization

Single-document corpus poisoning under black-box access poses a severe threat to Retrieval-Augmented Generation (RAG). However, existing attacks predominantly rely on explicit factual refutations or imperative instruction overrides. Modern aligned LLMs often arbitrate such conflicts conservatively, rejecting overt cont...

Hang-Yi Zhao, Bo-Nan Zhang, Fu-Sen Guo et al. · 0 citations
#machine learning Preprint Oct 2026

Trusting the Tool: Asymmetric Deference of LLM Vulnerability Auditors to Static-Analysis Evidence

Pipelines that combine large language models (LLMs) with static analysis let the model consume the analyser's output as evidence, but whether the model distinguishes true from false evidence, and whether its scrutiny depends on what the evidence claims, has not been measured. We built an intraprocedural analyser for C/...

Zhi-Hong Liu, Shu-Yao Gao · 1 citation · ⚡1
Open access 2026

Um, Actually: Timely and Calibrated Fact-Checking with DIVER

Fact verification pipelines are often unrealistic: they assume full-input access, operate on isolated atomic claims, and overlook the central question of when a system should act under partial evidence. We introduce DIVER (Deliberative and Iterative fact VERification), an LLM -based framework for calibrated incremental...

Sen-Yi Zhang, Tasnim Kabir, Jordan L. Boyd-Graber · 0 citations
Book Open access Sep 2026

From Answers to Audit Opinions: Cost-Aware Expert Verification of Financial Artifacts from Tool-Using LLM Agents

Tool-using LLM agents increasingly produce financial artifacts such as ratio tables, evidence-backed memos, and spreadsheet-style analyses. Existing evaluations often reduce such outputs to binary correctness or faithfulness labels, overlooking whether the artifact has sufficient evidence, whether errors are material o...

Jia-Wei Li · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.