CoEvoKG is introduced, a framework that turns a knowledge graph into both a source of verifiable training tasks and a persistent evidence memory for agent evolution, closing the loop between model self evolution and knowledge accumulation.
Abstract
Large language models can improve with reinforcement learning for search agents, yet existing self play agents repeatedly generate tasks while discarding the knowledge gained during successful searches. We introduce CoEvoKG, a framework that turns a knowledge graph into both a source of verifiable training tasks and a persistent evidence memory for agent evolution. CoEvoKG jointly trains a task generator and a search agent: the generator creates multihop questions from entity chains sampled from the knowledge graph, while the agent learns from rewards for answer correctness and search trajectories whose entity paths are supported by graph evidence. When a search succeeds, CoEvoKG verifies and deduplicates the retrieved evidence, then writes it back to the corresponding graph nodes and edges. Future rounds reuse this enriched graph for task generation and reward computation, closing the loop between model self evolution and knowledge accumulation. Experiments on six QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, and Bamboogle) with three backbone models show that CoEvoKG improves macro average accuracy over the corresponding base models by +11.2, +10.1, and +11.6 points on Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Llama-3.1-8B-Instruct, respectively. Under matched training budgets, CoEvoKG further improves over competitive self play baselines and RL baselines for search agents by +2.6 to +3.7 macro average points across the three backbones. Code is available at https://github.com/lazzy1225/CoEvoKG.
EvoGraph-R1 is introduced, a self-evolving GraphRAG framework that reconceptualizes knowledge graphs as dynamic environments shaped through agent interactions, establishing self-evolving knowledge graphs as a fundamental paradigm across modalities.
Jiashi Lin, Changhong Jiang, Xiangru Lin et al.· 1 citation
This work proposes HyperSkill, a hypergraph-based memory framework that jointly improves what to store, how memory is structured and retrieved, and how memory evolves, and represents memory as a hypergraph with two node types, subtask steps and reusable skills.
Evolutionary program search guided by Large Language Models (LLMs) has emerged as a powerful paradigm for automated scientific discovery. However, current approaches are fundamentally constrained by three bottlenecks: structurally blind parent selection, sparse whole-program evaluation rewards, and static mutation operators that fail to adapt during search. We present GAE (Graph-Augmented Evolution), a framework that resolves these limitations through a tightly coupled, three-pillar architecture. First, a relational graph neural network (GNN) parses programs into typed computation graphs, producing structure-aware embeddings. Second, an RL-optimized meta-controller leverages these embeddings to replace blind evolutionary sampling with a directed policy, dynamically selecting optimal parents and mutation directions based on reward history. Third, an online GRPO fine-tuning loop continuously updates the LLM mutation operator at test-time using group-normalized evaluation rewards, directly aligning the model's generation distribution with high-fitness structural edits. We evaluate GAE on a challenging scientific discovery task: symbolic regression for complex nonlinear oscillator systems. By transforming stochastic search into a directed, self-improving trajectory, GAE efficiently discovers closed-form physical equations, consistently matching or outperforming static LLM-driven baselines and achieving state-of-the-art out-of-distribution performance.
Reinforcement learning (RL) search agents commonly model retrieval as free-form natural-language query generation and optimize multi-turn interactions using final-answer rewards. Current studies mainly improve training with denser or more structured credit signals, but rarely examine whether retrieval is properly formulated at the policy-environment interface. We observe pronounced retrieval aliasing during Search-R1 training: rollouts for the same question continue to generate distinct query strings, yet their accumulated evidence sets increasingly overlap. We call this phenomenon retrieval-equivalence collapse; in this regime, trajectories approach utility equivalence with respect to retrieval decisions, leaving within-group returns with little effective retrieval contrast. To address this problem, we propose Harness-G, a graph-structured retrieval framework that redesigns this interface. It reformulates free-form query generation as finite action selection: the policy selects an evidence sentence or entity, or chooses to answer, while the environment constructs the menu, tracks retrieval state, and validates and executes each choice. This interface reduces linguistic aliasing and makes same-state alternatives directly comparable. Building on this interface, we introduce Structured Non-myopic Credit (SNC), which uses a frozen answer scorer to compare the selected action with its alternatives and assigns downstream gains to the earlier actions that enabled them. Across six QA benchmarks, Harness-G achieves the highest average F1 at both evaluated model scales, outperforming the strongest baseline, Graph-R1, by 10.74 points at 1.5B and 3.98 points at 3B.
Yanning Hou, Hao-Yuan Chen, Sihang Zhou et al.· 0 citations
Web browsing—widely used for information retrieval and fact verification—has become a fundamental capability of recently emerged large language model (LLM) agents, which is often elicited by training on complex questions requiring web search. However, this task faces challenges with respect to data and training: existing QA datasets are mostly 1-3 hop over closed corpora (e.g., Wikipedia); meanwhile, outcome-based on-policy RL that used by recent works is inefficient and brittle in long-horizon, tool-heavy browsing environments. To address these challenges, we introduce GraphSynthQA, a knowledge-graph (KG)—guided synthesis framework in an open-web setting. Starting from Wikidata seed entities, GraphSynthQA iteratively retrieves and verifies evidence from the internet to expand a KG, then synthesizes complex, answer-verifiable queries grounded in multi-evidence dependencies. Building on the synthesized data, we train web-browsing agents with a compute-efficient two-stage recipe: (i) cold-start supervised fine-tuning on ReAct-style trajectories, and (ii) step-level Direct Preference Optimization (DPO), where preferences are constructed offline via single-step branched rollouts that contrast candidate actions by their downstream success rates, providing dense process supervision without expensive on-policy exploration. Experiments show that our approach consistently improves performance on challenging web-browsing benchmarks and remains competitive among models of similar size.
Chiwei Zhu, Mingxuan Du, Benfeng Xu et al.· Annual International ACM SIG...· 0 citations
Large language model (LLM)-based web search agents have demonstrated remarkable capabilities in autonomous information retrieval and multi-step reasoning. However, their robustness under real-world distribution shifts, such as evolving webpage structures, dynamic content layouts, and unseen task environments, remains a critical open challenge. Existing approaches predominantly rely on static supervised fine-tuning with human-annotated trajectories, which not only incurs substantial labeling cost but also lacks the adaptive capacity to handle the inherent stochasticity of live web environments. In this paper, we propose TAPE (Trajectory Alignment and exPerience pool Evolution), a novel self-evolving fine-tuning framework designed to enhance the generalization and robustness of web search agents without requiring large-scale human annotation. TAPE introduces a dual-stream experience pool that continuously accumulates both successful and failed agent trajectories during self-play execution. A trajectory alignment module maps heterogeneous execution paths into a unified semantic space, enabling contrastive learning to identify failure-inducing operations and reward generalizable search strategies. The framework further incorporates an adaptive pool evolution mechanism that filters, reweights, and distills experiences to prevent knowledge stagnation and distributional collapse. Extensive experiments on the GAIA benchmark and BrowseComp dataset demonstrate that TAPE consistently outperforms strong baselines across all three difficulty levels, achieving up to 6.2% absolute improvement in task success rate on GAIA Level-2 tasks (a 134.8% relative gain over the SFT-Only baseline on the GAIA validation split) and 1.3% absolute (217% relative) improvement on BrowseComp hard-tier queries, while exhibiting significantly greater resilience to webpage structure perturbations. These improvements are consistent across three open base models spanning two model families and the 7B–14B scale range. Our analysis further suggests that the contrastive self-play signal can serve as a useful partial surrogate for human preference labeling in agent trajectory optimization, substantially reducing, though not eliminating, reliance on human-annotated trajectories.
Minjae Rhee, Jitong Zou, Tianjun Mo et al.· IEEE Access· 0 citations