VSRL: A Unified Reinforcement Learning Framework for Autonomous Search in Vertical Domains

Abstract

: Retrieval-augmented large language models (LLMs) are widely used for knowledge-intensive question answering (QA), yet their adaptation to specialized vertical domains presents a cost–reliability dilemma. Simulated search, exemplified by Self-Search Reinforcement Learning (SSRL), reduces retrieval cost but depends on the model’s parametric knowledge. On the Earthquake Dataset, the base model achieves only 1.62% Pass@1. Real-search training, exemplified by Search-R1, supplies external evidence but incurs substantial latency and exhibits reward collapse in representative Earthquake Dataset and Wildfire Dataset trajectories. We propose Vertical-Domain Search Reinforcement Learning (VSRL), a unified single-stage framework that combines progressive search scheduling, domain prompt engineering, and information relevance reward. Experiments cover the Earthquake Dataset (10,866 QA pairs) and Wildfire Dataset (8503 QA pairs) with Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct. VSRL achieves the best EM in all four domain–model settings, ranging from 0.481 to 0.518. It improves over Search-R1 by 10.3%–13.3% and over SSRL by 80.6%–87.2%. The information relevance reward further improves EM by 2.6%–13.7% across the same domain–model settings. Evaluation on the Earthquake Dataset also characterizes the answering-cost trade-off through policy-token usage, answer time, and search times. On the Earthquake Dataset with Qwen2.5-3B-Instruct, VSRL requires 26.2% less training time than Search-R1 over the first 100 steps and remains stable through the observed 200 steps.