ActProbe, an internal-state-based framework for detecting and localizing poisoned segments in multi-source LLM inputs, is proposed and remains effective against defense-aware adaptive attacks and can protect black-box APIs through surrogate-based poisoned-segment removal.
Abstract
Modern large language model (LLM) agents often construct prompts by aggregating retrieved passages, user reviews, and documents from multiple external sources. This paradigm exposes them to segment-level poisoning attacks, in which an adversary controlling only a small subset of sources injects malicious content to manipulate model outputs. Existing defenses mainly rely on textual patterns, external embeddings, or auxiliary detectors and may therefore fail against fluent, semantically plausible poisoned segments. They also provide limited support for locating the responsible segments. We observe that successful corrupted-evidence and adversarial-instruction attacks induce structured shifts in the LLM's internal activations, forming a consistent activation-space pattern that we call the poison direction. Based on this observation, we propose ActProbe, an internal-state-based framework for detecting and localizing poisoned segments in multi-source LLM inputs. ActProbe projects MLP activations onto a learned poison direction and uses a lightweight linear SVM trained on a small calibration set to detect contaminated prompts. It then applies BinRoL, which combines recursive replacement ablation, Mahalanobis-distance-based branch pruning, and MAD-based robust leaf detection to locate poisoned segments. ActProbe requires no modification to the backend LLM and reduces localization overhead from O(n) exhaustive probing to O(k log n) forward passes. Across three datasets, two attacks, and four open-weight LLMs, ActProbe achieves a 0.01 false-positive rate, a 0.05 false-negative rate, 0.94 localization recall, and a 0.90 localization F1-score. It remains effective against defense-aware adaptive attacks and can protect black-box APIs through surrogate-based poisoned-segment removal.
ToxicRAG is presented, a one-document-per-target attack that expresses misinformation as a coherent knowledge-update narrative that matches or exceeds the strongest evaluated baseline in every combination of dataset--model combinations.
Hao-Zhe Lu, Jia-Qiang Li, Xin-Yuan Zhu et al.· 1 citation
The results indicate that Micro-Collaborative Poisoning is not driven by a single dominant poisoned passage, but by the accumulation of weak adversarial signals across retrieved sources, which achieves downstream influence while leaving a weaker explicit poisoning signature than direct poisoning.
This work introduces TraceGuard, an adaptive rank-based filtering method that uses agreement among complementary feature rankings to identify suspicious examples and refines the selected set through shared patterns and adapts the removal threshold to each corpus without knowing the attack or poison rate.
Hao-Yang Li, Ya-Xin Xiao, L. Dai et al.· 0 citations
BERM is introduced, a lightweight framework that performs in-situ detection by modeling a host LLM’s internal representations extracted during prefill, adding negligible overhead and reducing incremental inference overhead to near-zero.
Maihao Guo, Chaoyang Zhao, Jin-Qiao Wang· Proceedings of the Thirty-Fi...· 1 citation
An Evaluation Agent, middleware that combines Natural Language Inference factual verification, a five-signal poison detector with relevance-weighted aggregation, and a Trust Index is proposed, which reliably blocks instruction injection of unsafe advice while contradiction and subtle semantic weakening remain hard.
Balkrishna Giri, M. Hasan, Jussi Rasku et al.· 0 citations
An exploratory case study that applies explainable artificial intelligence techniques to analyze how Prompt Guard 2 distinguishes malicious from benign prompts finds that Prompt Guard 2's decisions rely on the cumulative contribution of many tokens rather than a few dominant ones, yet saliency-guided synonym substituti...
Fernando Outeda, Gustavo Betarte, J. Campo et al.· arXiv.org· 0 citations
Related blog posts
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT scientists developed a training method to generate models that simulate the effects of human activities on climate much more quickly than other approaches.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.