Observed Recoverable Behavioral Failures in LLM Workflows

Abstract

Standard large-language-model evaluations primarily score terminal outputs: whether a final answer is correct, safe, preferred, or useful. This paper studies a recurring class of workflow failures referred to here as recoverable behavioral failures (RBFs). In the operational usage adopted in this paper, an RBF occurs when a model appears to possess the capability, context, and tool access needed to complete a task, does not exercise that capability on the first pass, and subsequently succeeds after a brief retry, verification, or correction prompt that introduces no new substantive information. These failures arise in practical AI workflows such as research synthesis, source verification, document drafting, code generation, multimodal interpretation, and artifact production. They are not captured well by single-turn benchmark accuracy or aggregate human-preference data because they become visible only across turns, when a second prompt reveals that the underlying capability was already available. The paper makes seven contributions. First, it operationalizes recoverable behavioral failure as an evaluation construct that separates capability from first-pass execution behavior. Second, it develops a layered sixteen-mode taxonomy spanning retrieval, reasoning, multimodal interpretation, infrastructure, artifact production, and editorial boundary control. Third, it reports a retry-probe pilot on a fifteen-item heterogeneous retrieval task in which first-pass resolution was 12/15 and after-retry resolution was 15/15, yielding a 20 percentage-point recoverability gap; Wilson 95% confidence intervals and an exact Fisher test are reported for the small-sample rates. Fourth, it extends the taxonomy with abstracted cross-platform workflow observations covering artifact-readiness, source-verification, audience-calibration, and editorial-boundary modes. Fifth, it reports a small intra-vendor pilot on a ten-task experiment pack across three Anthropic models. Sixth, it specifies a controlled cross-vendor evaluation protocol grounded in a companion repository, with task families, deterministic and human-judged checks, sample-size guidance, annotation guidance, raw-output archiving, scored CSV outputs, and a reproducibility checklist. Seventh, it reports a four-condition retry pilot (first-pass, null-retry, generic-retry, verification-retry), finding that retry framing under a strict deterministic rubric can regress pass rates relative to first-pass framing - a reflexive observation supporting the Retry-Failure Invisibility mode. The pilots are exploratory; the cross-vendor protocol is offered as planned replication rather than as a completed cross-vendor study. Companion repository: https://github.com/vjgits/Research-Papers/tree/main/Observed-Recoverable-Behavioral-Failures-in-LLM-Workflows - containing task schemas, prompt templates, runner scripts, scorer, sample raw outputs, scored CSV artifacts, and analysis materials referenced in the paper.