Sep 2026· IJCAI/ECAI· pp. 2474-2482· 0 citations· 42 references
Computer Science
TL;DR
This work introduces Falsdo, a diagnostic benchmark designed to make robustness and auditability identifiable and proposes DREA, a failure-aligned evidence-auditing baseline that improves instruction-level grounding (Joint-F1) and minimizes degradation under artifact-controlled settings.
Abstract
Recent generative AI renders multimodal misinformation structurally harder to detect, making reliable detection dependent on semantic verification grounded in verifiable evidence. However, current benchmarks often fail to isolate true semantic checking from superficial shortcut exploitation. We introduce Falsdo, a diagnostic benchmark designed to make robustness and auditability identifiable. Constructed via a heterogeneous web-mining pipeline, Falsdo provides instruction-conditioned grounding and formalizes a counterfactual artifact-control protocol. This stress test reveals a critical failure: when low-level generation traces are equalized, detector performance degrades, indicating reliance on artifacts rather than robust verification. To enable reproducible diagnosis, we propose DREA, a failure-aligned evidence-auditing baseline. DREA specifies evidence acquisition with explicit noise control and implements constrained channel-wise auditing with reliability-aware late fusion. Experiments demonstrate that Falsdo reliably exposes shortcut dependence, while DREA improves instruction-level grounding (Joint-F1) and minimizes degradation under artifact-controlled settings.
: Large language model (LLM)-based phishing detectors can produce decisions and explanations, but adversarial webpages violate the assumption that all input text is trustworthy evidence. Hidden comments, metadata, scripts, Cascading Style Sheets (CSS), and retrieved snippets can act as instruction-injection channels, w...
Yue-Kui Xu, Xiong-Zhang Luo· Computers, Materials & C...· 1 citation
Coding agents can return plausible patches that omit required behavior. These failures are hard to review because long traces and confident summaries often hide what was missed. We ask when a nominally weaker reviewer can reliably decide whether a patch solves its issue. We study 411 execution-labeled traces from three...
Jun-Yu Guo, Shangding Gu, Ming Jin et al.· 1 citation
Single-document corpus poisoning under black-box access poses a severe threat to Retrieval-Augmented Generation (RAG). However, existing attacks predominantly rely on explicit factual refutations or imperative instruction overrides. Modern aligned LLMs often arbitrate such conflicts conservatively, rejecting overt cont...
Hang-Yi Zhao, Bo-Nan Zhang, Fu-Sen Guo et al.· 0 citations
Pipelines that combine large language models (LLMs) with static analysis let the model consume the analyser's output as evidence, but whether the model distinguishes true from false evidence, and whether its scrutiny depends on what the evidence claims, has not been measured. We built an intraprocedural analyser for C/...
Fact verification pipelines are often unrealistic: they assume full-input access, operate on isolated atomic claims, and overlook the central question of when a system should act under partial evidence. We introduce DIVER (Deliberative and Iterative fact VERification), an LLM -based framework for calibrated incremental...
Sen-Yi Zhang, Tasnim Kabir, Jordan L. Boyd-Graber· Conference on Empirical Meth...· 0 citations
Tool-using LLM agents increasingly produce financial artifacts such as ratio tables, evidence-backed memos, and spreadsheet-style analyses. Existing evaluations often reduce such outputs to binary correctness or faithfulness labels, overlooking whether the artifact has sufficient evidence, whether errors are material o...
Jia-Wei Li· Proceedings of the 2026 ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.