Margin- and Vocabulary-Aware Adaptive Retrieval for Industrial Technical Question Answering: A Controlled Pilot

Abstract

Efficient industrial question answering requires reliable evidence retrieval despite document-specific terminology and spelling variation. We study confidence-gated dual retrieval (CGDR), which uses the normalized BM25 top-two margin to decide whether to invoke a complementary character n-gram index. On 80 held-out synthetic questions generated from a published review of domain-adapted industrial language models, the margin-only selector obtains 92.5% Recall@1 with 17.5% complementary-index calls. Yet additional spelling edits reveal substantial brittleness: Recall@1 falls to 38.75% for character deletion or transposition. We therefore introduce an out-of-vocabulary (OOV)-aware gate, combining rank ambiguity with lexical coverage of the indexed corpus. Its rule is fitted on separate development pages and their deterministic perturbations under a 55% development-stage call budget. On held-out deletion and transposition edits, the revised gate increases Recall@1 to 60.0% in each condition, compared with 38.75% for the margin-only gate, while requiring the character index on 60% of these requests. On unmodified questions, it returns 91.25% Recall@1 at a 21.25% call rate. Because this gate was designed after the margin-only analysis, its held-out gains are reported as exploratory. The improvements are measured on 80 underlying held-out queries with dependent synthetic transformations, not on naturally authored questions or a production system. Passage-level evaluation, sentence-extraction checks, matched-budget controls, and source-auditable code characterize the robustness–cost trade-off; no generative language model is evaluated.