Skip to content

DeepInstructor: An Agentic AI Instructor for Experience-Driven Idea Evaluation

Aug 2026 · 0 citations · 24 references
Computer Science

TL;DR

This work proposes DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience, and introduces DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility.

Abstract

As automated scientific discovery advances, Large Language Models (LLMs) can now generate research ideas at an unprecedented scale, shifting the bottleneck from idea generation to idea evaluation. Existing evaluators mainly rely on parametric LLM knowledge or unstructured retrieval, producing judgments that lack the experience-grounded reasoning used by human instructors. To address this, we propose DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience. DeepInstructor constructs an Experience Graph from 58,607 peer reviews and employs a ReAct-based agent to retrieve dimension-specific evidence for traceable evaluation. We further introduce DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility. Experiments show that DeepInstructor substantially outperforms existing baselines, improving Hit@1 and Hit@2 alignment with human judgments by 24.4% and 29.7%, respectively. Our findings suggest that scientific idea evaluation can be grounded in explicit reasoning over structured scholarly experience

View source

Similar papers

#artificial intelligence Preprint Sep 2026

AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era

AgentIdeaBench is introduced, a multidisciplinary benchmark that evaluates scientific ideation under two matched settings, static observation and active exploration, and Scientific World Modeling is explored, a generation-time loop that refines a draft hypothesis through structured thought experiments.

Yunxiang Mo, Tianshi ZHENG, Yi-Sen Gao et al. · 1 citation

MultiHint-9K: Orchestrating Human and AI Agency for Scalable Pedagogical Hint Generation

A framework that orchestrates human and AI agency to produce high-fidelity multilingual Socratic hints at scale is presented, and a component contribution analysis demonstrates that removing any pipeline stage degrades yield from 98% to as low as 38%.

Kamyar Zeinalipour, Amir Sadeghi, Giovanni Angelini et al. · 0 citations
#artificial intelligence Preprint Aug 2026

Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks

It is revealed that current agents can explore novel regions of the solution space but lack the capacity to convert this novelty into improved task performance, and that LLMs exhibit greater H-Creativity than medal-winning humans, yet achieve lower performance.

Shitanshu Bhushan, Yun-Xiang Zhang, Lu Wang · 1 citation · ⚡1
Preprint Aug 2026

AgentPanel: Toward a New Paradigm for Human--AI Collaboration in Exploring Scientific Questions

This work presents AgentPanel, a multi-agent forum for human--AI collaboration in scientific exploration, a multi-agent forum for human--AI collaboration in scientific exploration that outperforms a centralized multi-agent debate baseline and shows that users value AgentPanel for perspective diversity and exploration s...

Zhi-Yao Cui, Qianyi Wang, Hao Yan et al. · 1 citation

Self-Play Search Distillation for Large Language Model Reasoning

Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games, offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.

Lorenzo Molfetta, Wai-Chung Kwan, Giacomo Frisoni et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.