Quality-Aware Selection for Retrieval-Augmented Fine-Tuning of Small Language Models
TL;DR
Quality-aware selection is a promising, data-efficient safeguard for synthetic RAFT data—performing comparably to full-pool training at one third of the cost, with growing value as pool quality degrades—and larger-scale external validation remains future work.
Abstract
Retrieval-augmented fine-tuning (RAFT) can improve small language models (sLMs) on retrieval-grounded question answering, but the synthetic training data produced by commercial large language models (LLMs) vary in quality. This paper contributes a quality-aware selection protocol—rather than a new RAFT or QLoRA method—that scores LLM-generated alternatives along four embedding-based dimensions (question relevance, answer faithfulness, QA coherence, and semantic similarity) and selects one alternative per task before parameter-efficient fine-tuning. Under pre-specified paired-bootstrap contrasts with Holm correction, the parameter-free faithfulness-based selector only-AF significantly exceeds random selection on Gemma-2-9B-IT (ΔF1 = +0.106, 95% CI [+0.043, +0.174], Holm-corrected p = 0.019), and its pre-specified weighted companion af-70 (wAF = 0.70) shows the same confirmed pattern (Holm-corrected p = 0.002). Both effects persist under a Korean character-level F1 that removes particles and punctuation (Holm-corrected p = 0.004 and p = 0.042), indicating robustness to the choice of lexical metric. Relative to training on the full 150-row augmented pool, the quality-selected 50-row sets are statistically indistinguishable while using one third of the training data, which we interpret as data efficiency rather than superiority. Across six instruction-tuned models (2B–27B), a significant selector-by-model interaction indicates that the optimal quality axis is model-dependent, and the two smallest models show no benefit from selection. The study’s confirmatory contrasts use a small controlled Korean corpus under a transductive design; two pre-registered validation experiments probe external validity. On an independent five-fold larger corpus with a passage-level train/test split, fine-tuning transfers strongly and the selected one-third subsets show no significant difference from the full pool, while the advantage over random selection is directionally positive but small and not significant; under controlled corruption of 35% of the pool, the metrics detect the damaged rows, and for the score-sum selector the selection-versus-random benefit is significantly larger than on the clean pool (difference-in-differences p = 0.0014; directionally consistent but not significant for the faithfulness selectors). Within this scope, quality-aware selection is a promising, data-efficient safeguard for synthetic RAFT data—performing comparably to full-pool training at one third of the cost, with growing value as pool quality degrades—and larger-scale external validation remains future work.