Skip to content
Log in
Preprint

Adversarially Robust Abductive Fusion of Pre-trained Transformer-based Perception Models

Mario A. Leiva Yue Ma Qinru Qiu Gerardo I. Simari Paulo Shakarian
Aug 2026 · 0 citations · 28 references
Computer Science

TL;DR

This work frames the fusion of multiple imperfect ViT-based detectors as a consistency-based abduction problem solved at test time by an exact Integer Program (IP) and a polynomial-time heuristic and shows that this metacognitive layer can be learned without any domain knowledge by exploiting vector-space geometry.

Abstract

Deploying pre-trained perception models in novel environments degrades their accuracy under distributional shift, and assembling them alone does not recover it: combiners such as majority voting trade recall for precision and are brittle to coordinated failures. Prior metacognitive methods learn logical rules that flag a model's errors, but rely on hand-authored domain-knowledge cues (object-size priors, segmentation masks) that do not transfer to genuinely novel scenes. We show that this metacognitive layer can be learned without any domain knowledge by exploiting vector-space geometry: per-model Label Vector Pools (LVP), built from each model's own training embeddings, yield error-detection rules from the geometry of detections relative to training-determined prototypes, reaching parity with domain-knowledge rules to within $0.002$ every F1 on test set. Because the approach remains neurosymbolic, these geometric rules share a single logical framework and can still be complemented by domain knowledge when available. We frame the fusion of multiple imperfect ViT-based detectors as a consistency-based abduction problem solved at test time by an exact Integer Program (IP) and a polynomial-time heuristic. On an aerial-imagery benchmark of 15 weather-shifted test sets and six ViT detectors, our domain-knowledge-free layer matches the strongest majority-vote variant on clean data (within $0.005$ F1) and, unlike every majority-vote baseline, retains its performance under a coordinated label-flipping attack: at a $90\%$ flip rate it averages $0.42$ F1 versus $0.35$ for MV-Plurality (a $22\%$ relative gain) and attains the highest F1 on \emph{every} test set once the flip rate exceeds $0.4$

arXiv Semantic Scholar
View source

Similar papers

Preprint Jul 2026

Robustifying Vision-Language Models via Test-Time Prompt Adaptation

This work proposes RITA, a Robust test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment, and employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment.

Xingyu Zhu, Huanshen Wu, Shuo Wang et al. · 1 citation
Book Open access Aug 2026

VisionDES: Robust and Explainable Dynamic Vision Ensemble

Dynamic Ensemble Selection (DES) is an adaptive ensemble learning paradigm that selects a subset of base classifiers specific to each test input, enabling more flexible predictions than static ensemble methods. Although successful in tabular settings, DES remains largely unexplored in robust vision applications. We introduce VisionDES, a novel DES framework for image classification that uses deep model embeddings to estimate classifier competence. VisionDES leverages pre-trained vision transformer models to embed inputs and employs efficient nearest neighbor search to define a local region of competence for each sample. It then dynamically selects and fuses the most reliable models, using a similarity-weighted combination that down-weights less reliable or adversarially-compromised classifiers. VisionDES is extensively evaluated on various benchmarks and under clean conditions, distribution shifts, and strong adversarial attacks. It consistently outperforms static ensembles and existing uncertainty-based DES methods, improving robust accuracy by up to 20% under strong attacks and 2-3% higher accuracy under distribution shifts, with modest inference overhead. VisionDES offers instance-level interpretability by revealing models' contributions to the final decision.

Firuz Juraev, Mohammad Abuhamad, S. El-Sappagh et al. · 0 citations
Preprint Aug 2026

Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models

Noise-Contrastive GRPO is introduced, which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.

Michael M. Jerge, Joseph Pelczar, J. Downes · 0 citations
Preprint Aug 2026

When does fusing hand-crafted knowledge with learned representations pay? A cost-normalized benchmark of stacking, substitution, and interference

This work benchmarked one fixed hand-crafted knowledge source, a pinned bank of Gabor targets injected only during training at $\sim$2\% overhead, against data-driven alternatives under one frozen recipe with fixed subsets.

Ahmad AlMughrabi, Albert Clop, Benjamin Busam et al. · 0 citations