Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3\% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0\% to 26.3\%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.
Jingjie Ning, Xiaochuan Li, Shan Zhong et al.· 0 citations
Deep research systems represent an emerging class of agentic information retrieval methods that generate comprehensive and well-supported reports to complex queries, and/or answers to hard-to-locate factual questions. However, most existing systems rely on dynamic commercial search APIs, which pose reproducibility and transparency challenges, in addition to high costs. To address these limitations, we introduce DeepResearchGym as a free and open-source search sandbox for reproducible research on deep research systems. The sandbox provides a search API that indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN. It achieves comparable latency to popular commercial APIs while ensuring stable document rankings across runs. We demonstrate the sandbox's utility through two use cases. For training, we synthesize queries grounded in the indexed corpora and show that search agents trained within the sandbox generalize to commercial search at inference time, enabling cost-effective reinforcement learning. For evaluation, we extend the Researchy Questions benchmark with LLM-as-a-judge metrics to measure alignment with users' information needs, retrieval faithfulness, and report quality. Evaluation results show that system rankings remain consistent when switching from commercial APIs to ours.
João Coelho, Jingjie Ning, Jingyuan He et al.· International Conference on...· 2 citations
The hidden phenomenon accuracy-blind answer churn is called and the Snapshot Compatibility Audit is introduced, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement.
