Enhancing Web Search Agents With Self-Play Contrastive Fine-Tuning

Abstract

Large language model (LLM)-based web search agents have demonstrated remarkable capabilities in autonomous information retrieval and multi-step reasoning. However, their robustness under real-world distribution shifts, such as evolving webpage structures, dynamic content layouts, and unseen task environments, remains a critical open challenge. Existing approaches predominantly rely on static supervised fine-tuning with human-annotated trajectories, which not only incurs substantial labeling cost but also lacks the adaptive capacity to handle the inherent stochasticity of live web environments. In this paper, we propose TAPE (Trajectory Alignment and exPerience pool Evolution), a novel self-evolving fine-tuning framework designed to enhance the generalization and robustness of web search agents without requiring large-scale human annotation. TAPE introduces a dual-stream experience pool that continuously accumulates both successful and failed agent trajectories during self-play execution. A trajectory alignment module maps heterogeneous execution paths into a unified semantic space, enabling contrastive learning to identify failure-inducing operations and reward generalizable search strategies. The framework further incorporates an adaptive pool evolution mechanism that filters, reweights, and distills experiences to prevent knowledge stagnation and distributional collapse. Extensive experiments on the GAIA benchmark and BrowseComp dataset demonstrate that TAPE consistently outperforms strong baselines across all three difficulty levels, achieving up to 6.2% absolute improvement in task success rate on GAIA Level-2 tasks (a 134.8% relative gain over the SFT-Only baseline on the GAIA validation split) and 1.3% absolute (217% relative) improvement on BrowseComp hard-tier queries, while exhibiting significantly greater resilience to webpage structure perturbations. These improvements are consistent across three open base models spanning two model families and the 7B–14B scale range. Our analysis further suggests that the contrastive self-play signal can serve as a useful partial surrogate for human preference labeling in agent trajectory optimization, substantially reducing, though not eliminating, reliance on human-annotated trajectories.