Hallucinations in Structured Extraction: A Case Study on Prompt-Based Semantic Role Labeling

Abstract

This paper studies hallucinations in structured extraction using prompt-based Semantic Role Labeling (SRL) as a controlled case study. We implement a DSPy-based pipeline in which each prediction includes a generated rationale and a citation to a governing rule, allowing extraction errors to be aggregated by attributed rule. This diagnostic signal is used by a Rationale-Oriented (RO) optimizer to perform targeted, human-readable signature revisions. Experiments use a representative 1000-sentence pool drawn from the CoNLL-2012 test split, with a nested 200-sentence development subset used for optimization and the remaining 800 sentences reserved as an unseen subset. After signature selection, the role-specific signatures are fixed; on the disjoint unseen subset, they improve over their corresponding unoptimized baselines for all nine evaluated 5W + 1H-aligned roles, although the magnitude of improvement and the development-to-held-out gap vary substantially by role, with the sparse ARGM-PRP and ARGM-CAU roles generalizing markedly less well than their development scores suggest. Across the full 1000-sentence pool, the selected-role, gold-predicate aggregate reaches 67.04% strict micro-F1; this is a within-study summary and not a full CoNLL SRL score. In the held-out ARG0 analysis, extraction outcomes are non-uniformly associated with cited rules, while signature perturbations show that rule citations are better interpreted as observable diagnostic proxies than as faithful per-instance causal explanations. Compared against MIPROv2 on the same extraction model, RO reaches a comparably low commission-error operating point without the recall collapse MIPROv2 incurs. Two further ablations locate the source of the effect: a full nine-role cross-model transfer to Phi-4-reasoning shows RO signatures carrying across model families unevenly, with gains concentrated on the weak, sparse roles and negligible-to-slightly-negative where the target model is already competent, and with transferred precision capped by its coarser span-boundary behavior; and a matched same-model self-critique ablation, in which Qwen3-14B supplies its own diagnostic feedback across the same nine roles, tests how much of the refinement effect can be recovered without a stronger external critic. Seen together, these ablations separate cross-model signature portability from teacher-capacity effects and clarify which parts of RO depend on the diagnostic structure itself versus the critic used to revise the signatures. The primary condition retains a design-time dependence on a proprietary teacher, and signature optimization is centered on a single open-weight extraction model, Qwen3-14B.