Evidence integrity is a systems axis distinct from model capability; mitigation depends on enforcement placement and predicate coverage; mitigation depends on enforcement placement and predicate coverage.
Abstract
Agentic systems act, so a defect in the evidence they retrieve becomes a wrong action with a currency cost. The most dangerous enterprise defects are metadata-borne: a stale price or a superseded record, perfectly well-formed in the payload and betrayed only by freshness, lineage, or provenance. Such a defect never enters the agent's context, and an agent cannot doubt data it cannot see. On a priced replenishment benchmark, a competent agent silently converts an injected metadata-borne defect into a costly action about 60% of the time, with zero data-quality flags and behavioral doubt markers at chance (AUC<= 0.50). Across four model tiers spanning roughly 15x in inference price, the rate stays flat: capability does not buy skepticism. A metadata-aware pre-action gate with downstream-only remediation recovers the loss fully on the signals its predicates cover and not at all on those they miss. A model-free oracle derived from the task's decision geometry tracks the measured rates with MAE 0.015 (Pearson r = 0.876, interval coverage 15/16 cells), giving the flat ladder an analytical form. Evidence integrity is a systems axis distinct from model capability; mitigation depends on enforcement placement and predicate coverage. Code, frozen results, and a deterministic analysis pipeline: https://github.com/besanson/dqSarc
A Python harness mirroring the autonomy axis is released so that future methods can be added directly to the leaderboard, and four patterns emerge: Spider gains transfer unevenly to BIRD and Spider~2.0; autonomy buys robustness at non-trivial cost; reasoning internalization sits between answer-only decoding and externa...
Changruo Zhao, Zujun Peng, Yu Tian et al.· International Conference on...· 1 citation
ChurnBench is presented, an open-source benchmark that generates a four-source enterprise data fabric as a timeline rather than a snapshot, and it is found that when a system refreshes on a schedule, cache age does not predict staleness.
V. Singh, Preeti Priyam, Gautam Bhowmick· 0 citations
This work uses LAST-CQ -- a five-agent, training-free, execution-grounded Text-to-Cypher framework -- as an instrumented testbed, running three counterfactuals over 2,471 live-database queries and six backbones spanning three vendor scale tiers.
I. Prokopiou, Athanasios Aidinis, Panagiotis-Christos Kyrmpatsos et al.· 0 citations
A telemetry-to-episode construction method instantiated as BTS-AgentBench is presented, which normalizes BTS metadata and raw histories into a read-only tool store, compiles static tasks with tool-derived gold answers and evidence, and lifts retained tasks into typed, bounded operator-facing episodes.
ProCTOR is described, a Teacher-Student loop in which a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they cannot apply, and a Teacher grades those mutations under five deterministic guardrails: hermetic sandboxes, capability-disjoint roles, acceptance checks tha...
WarehouseReliabilityBench is introduced, 400 frozen tasks over two synthetic warehouses in which roughly half the correct responses are a clarification, an abstention or a refusal, with pinned denominators and a pre-registered paired bootstrap fixing each claim verb before the numbers existed.
Morris Lee· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.