When to Invoke an LLM in Industrial Natural-Language-to-DSL Conversion: A Taxonomy-Driven, Confidence-Gated Selective Hybrid for Safety-Critical Avionics Test-Script Authoring

Abstract

Converting Korean natural-language avionics test procedures into an executable XML domain-specific language (DSL) is a labor-intensive bottleneck, yet neither automation extreme is acceptable in this safety-critical setting: end-to-end large language model (LLM) generation is unauditable and emits schema-violating values, while purely deterministic extraction leaves externally-grounded slots unsolved. We ask where, and how sparingly, an LLM should be invoked. From a 64-slot schema we derive a data-computed, five-class evidence-grounding slot taxonomy and build a confidence-gated selective hybrid: a deterministic tier resolves every slot whose evidence is lexical or catalog-grounded, and an on-premises retrieval-augmented LLM is invoked only for residual open-grounded slots with low retrieval confidence, under a gate threshold selected on a held-out validation split. On a leakage-controlled clean test split of <inline-formula> <tex-math notation="LaTeX">$1{,}705$ </tex-math></inline-formula> unique procedures (drawn from <inline-formula> <tex-math notation="LaTeX">$213{,}013$ </tex-math></inline-formula> industrial pairs after removing <inline-formula> <tex-math notation="LaTeX">$11.4\times $ </tex-math></inline-formula> duplication and 7.4% near-duplicate leakage), the hybrid is statistically equivalent to the pure on-premises LLM (59.4% instance Exact Match; paired 90% CI ±0.7 pp within a ±2 pp margin) while invoking the LLM on only 37% of instances, keeping 90% of emitted slots deterministic and audit-logged, and eliminating closed-vocabulary schema violations by construction (0 vs. 0.89%). Instantiating the same gate with a stronger BM25 backend nominally exceeds the pure LLM (60.4%; paired 90% CI <inline-formula> <tex-math notation="LaTeX">$[+0.2, +1.7]$ </tex-math></inline-formula> pp, McNemar <inline-formula> <tex-math notation="LaTeX">$p{=}0.044$ </tex-math></inline-formula>) at a 14% call rate. Because strict full automation remains below deployment thresholds, we position the system as an authoring-support tool: on a <inline-formula> <tex-math notation="LaTeX">$1{,}879$ </tex-math></inline-formula>-procedure candidate-ranking analysis of the full test split (an upper-bound characterization, not a clean-split estimate), the correct value appears in the top-five list for 84.5% of grounded slots (mean reciprocal rank 0.953), supporting a human-in-the-loop workflow in which a reviewer controls every emitted action. We conclude that a taxonomy-driven selective hybrid is simultaneously accurate, LLM-frugal, and supportive of DO-330 tool-qualification evidence generation.