Skip to content

Model Confidence Under Answer-Preserving Attacks: An Informativeness-Manipulability Frontier

Aug 2026 · arXiv.org · Vol abs/2608.06571 · 0 citations · 23 references
Computer Science

TL;DR

This work studies model-internal confidence readouts under white-box, image-only attacks constrained to preserve the generated answer byte-identically, showing that comparable confidence movement can be induced at the representation level rather than only through adversarial images.

Abstract

Deployed vision-language systems often gate their answers on confidence, making confidence robustness relevant to oversight. We study confidence readouts under white-box, image-only attacks constrained to preserve the generated answer byte-identically. Under a reachability assumption, an unmovable readout cannot outperform the answer-string accuracy prior, whose pooled value is 0.617. Independently of that assumption, a uniform amplitude certificate below a measurable threshold guarantees adversarial discrimination above the same floor. Across four vision-language models, three visual question answering benchmarks, five deployed confidence channels and two defense estimators, direct or surrogate-aimed attacks produce itemwise feasible perturbations that refute this uniform certificate in all 84 estimator-by-cell combinations. Coordinated correctness-label-aware attacks drive adversarial discrimination to or below the answer-string floor in all sixty deployed-channel cells, including all fifty-nine that begin above it. Hidden-state interventions and an open-ended text-model activation-space replication show that comparable confidence movement can be induced at the representation level rather than only through adversarial images. None of four tested defense families establishes a robust alternative under the specific evaluation applied to it. In a confidence-gated simulation, a coordinated token-probability attack transferred to a hidden-state gate causes up to 84.8% of previously rejected wrong answers to become accepted. After reweighting to each benchmark's natural correctness prevalence, accepted accuracy falls below the no-gate baseline in eight of twelve cells under transfer and all twelve under a direct gate-aimed attack. Under the studied threat model and budget, confidence is therefore an integrity-sensitive rather than intrinsically robust oversight signal.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Secure Agentic Coding through Counterexample-Grounded Feedback

falcon{}, a counterexample-grounded verification framework that separates attack exploration from behavioral verification and connects them through execution traces of the program under test, is introduced, suggesting that execution-grounded feedback can mitigate reward overoptimization while preserving useful guidance...

Tue Le, A. N. Hoang, Gabriel Sasseville et al. · 0 citations
Preprint Oct 2026

A Certified Robustness Framework for Backdoor Poisoning Attacks using $f$-Divergences

Machine learning systems trained on untrusted data are vulnerable to data and backdoor poisoning attacks, in which an adversary corrupts training and test samples to manipulate predictions at inference time. Randomized smoothing yields provable defenses against such attacks, but existing applications to train-time robu...

Evan Rose, Alina Oprea · 0 citations
#artificial intelligence Preprint Oct 2026

Mitigating Black-Box Score-Based Query Attacks via Gradient Confusion

Score-based black-box adversarial attacks pose a serious threat to deployed machine learning systems by exploiting repeated model queries to craft adversarial examples, requiring neither access to model parameters nor architecture. Existing defenses predominantly rely on stochastic output perturbations, which fail to p...

Ali Javeri, Javad Hezareh, Radin Shahdaei et al. · 0 citations
Open access 2026

PAIR: Against Inference-Time Attacks on Fine-Tuned LLMs via Path-Level Inference Randomization

PAIR, a dynamic framework that applies path-level randomization within LoRA adapters during inference, indicates that inference-time randomization at the path level can strengthen fine-tuned LLMs against the evaluated security and privacy attacks.

Jia-Yang Li, Tian-Qing Zhu, Da-Yong Ye et al. · 0 citations
Book Open access Nov 2026

INSURE: Inference-Time Defense Framework for Securing Agents against Hijacking

Agents are increasingly susceptible to hijack attacks, where malicious instructions embedded in cross-modal inputs (e.g., adversarial images or retrieved web content) are used to deceive the agent's execution. Existing defenses typically rely on strict filtering or refusal mechanisms, which suffer from a critical limit...

Che Wang, Fu-Yao Zhang, Jiaming Zhang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.