The rapid proliferation of conversational AI agents necessitates evaluation methods that assess performance across diverse dialogue scenarios, utilizing multiple situational criteria, as appropriate behavior depends on context and encompasses various aspects. Existing dialogue benchmarks typically use static natural co...
Ravil Mussabayev, Saveliy Zavertyaev, Dmitriy Parpulov et al.· 0 citations
Large Language Models (LLMs) encode substantial factual knowledge. Yet, estimating confidence in factual claims reliably remains challenging. We study likelihood-based factual confidence from the complementary perspectives of discrimination and calibration across multilingual factual statements. Specifically, we instan...
Ponhvoan Srey, Jason Lucas, Pittawat Taveekitworachai et al.· 0 citations
Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, t...
Min-hua Lin, Zi-Jun Wang, Ze-Wen Liu et al.· 0 citations
GRPO assigns the same group-relative advantage to equally rewarded responses within a question, leaving their generation uncertainty unused. We introduce MGE-PO, centered on four-quadrant (4Q) advantage shaping that combines verified correctness with rollout generation entropy. Correctness determines the sign of an add...
Ji-Hao Huang, Guang-Jia Chai, Zhi-Rui Chen et al.· 0 citations
Reach audiences
Advertise in front of researchers, engineers, and readers.
Long-context decoding in large language models (LLMs) is increasingly bottlenecked by the memory and computation required to attend over an ever-growing Key-Value (KV) cache at each autoregressive step. Although token-level sparse attention reduces this burden by restricting computation to a small subset of relevant pa...
Zhen-Xu Tian, Ke-Bin Liu, Zheng-Wu Yang et al.· 0 citations
Tree search enhances multi-step reasoning in large language models (LLMs), yet prevailing methods expand every step with a fixed budget, irrespective of the marginal utility of expansion. We first show, across existing search algorithms, that continuations sampled from a shared intermediate state are largely duplicates...
Jianguo Huang, Zhiyi Lyu, Fuxiang Zhang et al.· 0 citations
Chain-of-Thought (CoT) faithfulness matters for monitoring, explaining and safely deploying large language models, yet it has been studied mostly in English. Languages differ in level, the amount of text available to learn from, so models know less in lower-resource languages, which may change how much they rely on the...
Lucas Resck, Jing-Yi Sun, Qianli Wang et al.· 0 citations
JEPA-Prune is proposed, a post-hoc neuron pruning framework guided by a model trained exclusively on PII-free views, which trains a public model on surrogate and masked views of the same records using a joint-embedding predictive architecture (JEPA), capturing domain knowledge without identifiers.
Large language model (LLM) agents have shown promising potential in coding tasks, and their performance is jointly determined by the policy model and the harness that mediates their interaction with the environment. Indeed, harness and policy are mutually reinforcing and evolving only one component is suboptimal---a ha...
Zhepei Wei, Wenlin Yao, Baolin Peng et al.· 0 citations
Large reasoning models solve complex problems through long chains of thought, in which prior work has documented failure modes such as overthinking, underthinking, and unsupported deductions. Existing methods detect these failures from local signals: textual similarity to nearby steps, switch tokens, or an LLM's judgme...
Large language model agents must discover relevant tools from large libraries, e.g., thousands of tools, as new needs arise during task execution. When several candidates could satisfy the same need, execution may be required to identify the relevant one. Investigating candidates in a shared trajectory can leave altern...
Anna Mazhar, Seiji Maekawa, Nikita Bhutani· 0 citations
Large language models (LLMs) are increasingly used in critical domains, where their susceptibility to hallucinations makes reliable and interpretable uncertainty quantification essential. We introduce the concept of mechanistic uncertainty, which characterizes uncertainty through the stability of the internal mechanism...
Aneesh Komanduri, Majid Afshar, Timothy A. Miller et al.· 0 citations
