Skip to content
Preprint

Origin Is All You Need: Provenance-Aware Transformers for Structural Trust-Boundary Separation

Sep 2026 · 0 citations · 53 references
Computer Science

TL;DR

Provenance-Aware Transformers is proposed, a provenance-aware defense that makes application-supplied source labels actionable inside the model and shows that exposing provenance as a first-class architectural signal can shift LLM safety alignment from brittle pattern matching toward explicit trust separation.

Abstract

Indirect prompt injection (IPI) remains a central safety and security challenge for large language model (LLM) systems because standard transformers lack architectural notion of source authority. Retrieved documents, user inputs, and system instructions are all processed through the same undifferentiated attention mechanism, forcing the model to infer from wording alone what should be obeyed and what should be treated as data. We propose Provenance-Aware Transformers, a provenance-aware defense that makes application-supplied source labels actionable inside the model. Each input token is assigned a ring ID encoding its origin, and the model is augmented with origin embeddings, a learnable origin attention bias, and a learnable origin scale that preserves provenance under normalization. The resulting architecture enforces a structural boundary between authoritative and non-authoritative sources during generation. To instantiate this architecture on released pretrained models, we propose a two-stage fine-tuning pipeline to teach the model origin semantics and task behavior under ring constraints. Evaluation shows that Provenance-Aware Transformers maintain robust resistance to IPI both in-distribution and out-of-distribution while preserving utility comparable to the base pretrained model. More broadly, our work shows that exposing provenance as a first-class architectural signal can shift LLM safety alignment from brittle pattern matching toward explicit trust separation.

View source

Similar papers

Open access 2026

TrustTrace: Provenance-Aware Tracing of Indirect Prompt Injection Propagation in LLM-Assisted Learning Workflows

Indirect prompt injection can persist through retrieval, generation, and revision after the visible payload disappears. Existing defenses primarily detect suspicious inputs or constrain immediate actions, but they provide less evidence about which source influenced which final-artifact span through a temporally valid r...

Jing-Yi Chen, Rong-Qing Lu · 0 citations
Preprint Sep 2026

The Shape of Ownership: Verifying LLM Provenance through Semantic Structures

This work introduces PROSE (Provenance through Relational Organization of Semantic Expression), replacing fixed query sets with a target semantical domain and brittle response keys with semantic structures internalized as domain-conditioned response behavior.

Zhong-Rui Sun, Jia-Hao Chen, Ou-Bo Ma et al. · 0 citations
Preprint Aug 2026

Change-Provenant Supervision: Governing Learned Artifacts Under Policy Change

A recorded dependency graph cannot certify that it contains no omitted edge. For learned artifacts, graph-scoped invalidation therefore cannot by itself justify admission after authority changes, especially when output testing misses a provenance-stale derivation. We separate recorded lineage, which proposes impact sco...

Jesus Salas · 1 citation · ⚡1
Sep 2026

AGATE: Provenance-Based Runtime Defense Against Compositional Attacks on LLM Agents

LLM agents can produce harmful effects through sequences of ordinary operations. Judging such actions requires establishing both the authority that permits them and the origin of the data they carry. We present AGATE, an authorization and data-provenance gate at instrumented agent-harness boundaries. Operator declarati...

Xiao-Ru Zhang, Zhuo-Ran Cheng, Kai-Lin Liu et al. · 5 citations
#artificial intelligence Preprint Oct 2026

PACE: Provenance-Aware Capability Enforcement for Tool-Using LLM Agents

Tool-using large language model (LLM) agents turn generated text into real side effects, so poisoned tool metadata, retrieved pages, memory, and reusable skills can steer the next call. Vetting an artifact before admission does not settle this. A safe variant and a leaking variant can produce the same admission evidenc...

Feng-Peng Li, Qi-Zhou Wang, Yu-Ke Hu et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.