This work proposes DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience, and introduces DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility.
Abstract
As automated scientific discovery advances, Large Language Models (LLMs) can now generate research ideas at an unprecedented scale, shifting the bottleneck from idea generation to idea evaluation. Existing evaluators mainly rely on parametric LLM knowledge or unstructured retrieval, producing judgments that lack the experience-grounded reasoning used by human instructors. To address this, we propose DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience. DeepInstructor constructs an Experience Graph from 58,607 peer reviews and employs a ReAct-based agent to retrieve dimension-specific evidence for traceable evaluation. We further introduce DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility. Experiments show that DeepInstructor substantially outperforms existing baselines, improving Hit@1 and Hit@2 alignment with human judgments by 24.4% and 29.7%, respectively. Our findings suggest that scientific idea evaluation can be grounded in explicit reasoning over structured scholarly experience
AgentIdeaBench is introduced, a multidisciplinary benchmark that evaluates scientific ideation under two matched settings, static observation and active exploration, and Scientific World Modeling is explored, a generation-time loop that refines a draft hypothesis through structured thought experiments.
Yunxiang Mo, Tianshi ZHENG, Yi-Sen Gao et al.· 1 citation
A framework that orchestrates human and AI agency to produce high-fidelity multilingual Socratic hints at scale is presented, and a component contribution analysis demonstrates that removing any pipeline stage degrades yield from 98% to as low as 38%.
Kamyar Zeinalipour, Amir Sadeghi, Giovanni Angelini et al.· 0 citations
The bottleneck is scientific judgment rather than coding, and genuine discovery remains out of reach, so TruthInsightBench makes this gap a measurable target.
Zhi-Bo Yang, Chen Zhang, Yue-Wei Zhang et al.· 2 citations
It is revealed that current agents can explore novel regions of the solution space but lack the capacity to convert this novelty into improved task performance, and that LLMs exhibit greater H-Creativity than medal-winning humans, yet achieve lower performance.
Shitanshu Bhushan, Yun-Xiang Zhang, Lu Wang· 1 citation· ⚡1
This work presents AgentPanel, a multi-agent forum for human--AI collaboration in scientific exploration, a multi-agent forum for human--AI collaboration in scientific exploration that outperforms a centralized multi-agent debate baseline and shows that users value AgentPanel for perspective diversity and exploration s...
Zhi-Yao Cui, Qianyi Wang, Hao Yan et al.· 1 citation
Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games, offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.
Lorenzo Molfetta, Wai-Chung Kwan, Giacomo Frisoni et al.· arXiv.org· 0 citations