Skip to content
Preprint

AgentTrustCI: Build-Time Policy, Grounding, and Trajectory Tests for Java LLM Agents

Nov 2026 · 1 citation · 13 references
Computer Science

TL;DR

A production-trace validation protocol, optional LLM-judge integration, and maintenance of expectations as workflows evolve are described, a practical pattern for expressing behavioral contracts within an existing Java release pipeline.

Abstract

Infrastructure security does not establish whether an LLM agent selected an allowed tool, grounded its answer in tool output, or followed a required workflow. AgentTrustCI is a Java/JUnit 5 framework that expresses these application-layer expectations as policy, grounding, and trajectory checks over recorded traces. A deterministic replay implementation supports repeatable build-time tests; a judge interface provides an extension point for optional semantic evaluation. On 55 synthetic cases spanning seven enterprise domains, the full pipeline correctly classifies all 30 injected faults and 25 passing controls, including ten non-trivial controls, and misses none of 12 designated critical cases. Each single-layer configuration correctly classifies 35 cases because the corpus deliberately isolates ten faults per layer. A grounded answer obtained through a disallowed sequence illustrates the additional coverage of trajectory checks. Repeated measurements quantify validator execution separately from process startup and reporting. These results establish coverage of the authored regression suite, not production accuracy. We describe a production-trace validation protocol, optional LLM-judge integration, and maintenance of expectations as workflows evolve. The contribution is a practical pattern for expressing behavioral contracts within an existing Java release pipeline.

View source

Similar papers

Book Open access Oct 2026

Formal Runtime Verification for Tool-Using LLM Agents: An Offline Same-Benchmark Study on AgentDojo and STAC

Results show that formal temporal monitoring adds value exactly when the trace exposes trustworthy history, and quantify how far current benchmarks are from that point and propose a twelve-field enforcement-ready trace schema.

Nikolaos Kekatos, Stylianos Basagiannis, Marinelio Chintri et al. · 2 citations
Preprint Nov 2026

EvoX: Rethinking the Agent Harness as a Runtime Policy

Large language model (LLM) agents increasingly interact with tools and environments, yet plausible model decisions do not always translate into successful execution due to tool-call mismatches, interrupted workflows, and unsupported completion claims. We present EvoX, a Rust-based, multi-model agent runtime guided by T...

Haoyang Zhang, Qu-Meng Sun, Xiao Zhang et al. · 0 citations
Preprint Oct 2026

AgentTriage: Replayable Security Diagnosis Across Heterogeneous Agent Runtimes

AgentTriage is presented, an executable contract for replayable security diagnosis across heterogeneous runtimes whose claims cite canonical runtime events and requires abstention when mandatory evidence is missing and an open-source implementation and selected sanitized evidence are released.

Jie Ying, Xu-Tao Zhang, Ning-Yuan Chen et al. · 0 citations
Review Oct 2026

AgentPolicyForge: From Executable Environments to Policy Benchmarks and Adaptive Guards

Customized agent deployments define policies over tools, resources, and execution conditions. Providing these policies in the task agent's prompt does not reliably prevent violations. Training a guard requires examples of tool calls that comply with or violate those policies. We introduce AgentPolicyForge, a framework...

Chao Shang, Wan-Yu Du, Yi Zhang et al. · 0 citations
Preprint Oct 2026

Compromise Is Not Consequence: Evaluating Task-Scoped Authorization in LLM Agents with Paired Replay

A tool-using model can follow a malicious instruction even when its credentials are valid. We study whether task-scoped authorization contains the resulting tool execution. Our paired-replay testbed samples a model request once and submits the same action, resource, and arguments to broad bearer, scoped JWT, sender-con...

Tural Hagverdiyev · 1 citation

TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces

An agent can complete a task while exhibiting undesirable behavior during execution. Developers need tests for the specific behaviors encountered in deployment, beyond fixed benchmark suites. We present TraceDance, an agent system that constructs targeted benchmarks from deployment traces for user-specified undesirable...

De-Hai Min, Dao-An Zhang, Yiming Zeng et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.