Scientific ideation is driven by curiosity: researchers ask questions that expose knowledge gaps, reveal competing hypotheses, and clarify missing evidence relevant to decision making. Yet, most LLM-based ideation systems optimize the idea text while leaving curiosity under-modeled, resulting in brittle, engine-specific gains. To address this, we propose Curiosity-Driven Questioning (CDQ), a training-free, engine-agnostic method that constructs a compact set of curiosity-driven questions from a topic corpus under black-box LLM access. CDQ scores questions with three corpus-relative indices (GapExposure, DisagrDiversity, ClarityGain) and uses a two-stage scoring with an entropic mirror-ascent reweighting over question meta-policies to concentrate on high-utility question policies. Our results show that prepending CDQ questions improves downstream ideation across single-pass, retrieval-augmented generation, and self-reflection engines with multiple base models under preference-based evaluation. Ablations and a human study further support the complementarity of the indices and the reliability of the evaluation. Our code and dataset are available at: https://github.com/Sikun-Skyler-Guo/CDQ
Sikun Guo, Di Wang, Xiaohan Fan et al.· Proceedings of the 32nd ACM...· 0 citations
Large language models (LLMs) possess extensive latent knowledge yet remain largely static at inference. Once prompted, their generation policy typically cannot evolve, and post-hoc ''self-reflection'' methods provide no explicit principled learning signals. To address this limitation, we formally model iterative research idea optimization as a finite-horizon Markov Decision Process and propose InfRL (Inference-time Reinforcement Learning), a framework designed for effective policy improvement at inference without updating model weights. InfRL coordinates three specialized LLM agents: (i) a State Transition Agent that proposes candidate ideas; (ii) a Policy Update Agent that learns feedback strategies based on idea trajectories; and (iii) a Reward Agent that assigns normalized, comparative rewards, enabling nuanced reinforcement signals. This empowers the Policy Update Agent to dynamically refine feedback strategies, progressively leveraging latent knowledge encoded within the LLM. We evaluate InfRL on five balanced datasets covering 500 recent research papers from health, genetics, environment, neuroscience, and engineering domains. Compared to a single-pass GPT-4o baseline and a strong self-reflection baseline, InfRL improves the novelty of generated ideas by 3.93%--32.7% and the feasibility of generated ideas by 70.1%--448.6% with GPT-4o, exhibiting consistent improvements across ten inference-time iterations. Ablation studies underscore the critical roles of our reward formulation and modular agent design, while human evaluations confirm alignment between reward trends and perceived idea quality. InfRL thus offers a practical and domain-agnostic approach to harness reinforcement learning during inference, bridging the gap between static prompting and computationally intensive parameter-level fine-tuning. The code and the dataset we use are provided at: https://github.com/amir-hassan25/InfRL
Sikun Guo, Amir Hassan Shariatmadari, Jiuqi Wang et al.· Proceedings of the 32nd ACM...· 0 citations
