Preference labels align model training, but the judgments behind them are difficult to inspect or verify. In this paper, we examine how preference construction can be checked when intermediate decisions are recorded, and which of them remain uncertain. We present Symbolic Preference Labeling, a method in which a pretra...
This paper is an exposition of the theorem prover Arend developed at JetBrains Research. Arend language has been designed to facilitate the formalization of mathematics within the framework of univalent foundations and with the focus on constructive mathematics. It is based on a variant of homotopy type theory with cub...
Large language models can detect vulnerable code, yet the form of vulnerability encoding in their representations remains unclear. We analyze 8 code models spanning 6.7B-15B parameters on C programs using three datasets (DELTASECOMMITS, SVEN, and PRECISE-BUGS), comparing two diagnostic tasks on the same representations...
Rui Melo, Andre Catarino, Claudia Mamede et al.· 0 citations
Generative AI can create programming feedback at scale, but validating its alignment with instructors'pedagogical goals and strategies remains challenging. Instructors may struggle to fully specify validation criteria upfront, as these often emerge through inspecting concrete examples and revising misaligned feedback....
Xiaohang Tang, Rehema Abulikemu, Peng Gao et al.· 0 citations
Reach audiences
Advertise in front of researchers, engineers, and readers.
Hallucination stress tests can yield similar average scores across language models even when different prompts elicit hallucinations in each model. We develop a procedure that iteratively edits source-grounded seed prompts to expose hallucinations in a target model. An LLM editor revises each prompt in response to the...
Jiho Lee, Bingquan Zhang, Lin Ting Liang et al.· 0 citations
This work introduces AutoRedClaw, an automated red-teaming framework that generates end-to-end attacks together with the environments needed to execute them, and constructs RedClaw-Bench, a 330-case benchmark that yields substantially higher attack success rates (ASRs) than AgentDojo, AgentDyn, and DTap.
Hao Li, Tong-Yan Hu, Weidi Luo et al.· 0 citations
Medical workflows often require multiple interdependent analyses, but existing specialist and generalist AI models are not designed to coordinate them. We introduce MMedAgent-Plus, a multimodal medical tool-use agent that combines a vision-language planner with 18 leading specialist models spanning 12 tasks and 8 modal...
Ruoxin Wang, Yuanting Pan, Binxu Li et al.· 0 citations
Profile-DP is introduced, a policy-aware local differential privacy framework designed specifically for user-profile release in personalized LLM inference, which provably bounds the release of user-withheld facts while treating public instructions, profile schemas, and authorized facts as public side information.
Mingchen Li, Zi-Fan Peng, Ke-Wei Sha et al.· 0 citations
Deep neural networks have achieved remarkable success on vision benchmarks, yet they continue to struggle with many generalization challenges. Supervised vision training relies on one-hot labels, which provide limited information about semantic structure and shared attributes between classes. This limited supervision c...
Shruthi Gowda, Nikita Jain, Elahe Arani et al.· 0 citations
Web agents are typically evaluated on English-language interfaces, leaving their ability to operate across the world's multilingual websites largely unexplored. This gap is especially significant for low-resource languages, where standardized resources for evaluating web agents remain limited. To address this gap, we i...
Ocean Monjur, Daud Ahmad Nisar, Shahriar Kabir Nahin et al.· 0 citations
Multimodal large language models (LLMs) are increasingly used for structured extraction from financial documents, but their behavior under visual ambiguity is poorly understood. We study CARLAR (Courtesy Amount Recognition and Legal Amount Recognition) influence: a model returns the same amount for both cheque fields e...
Modern language models are increasingly asked to handle several tasks within a single prompt. Doing so requires solving each task correctly while integrating the resulting answers into a coherent response. In this paper, we identify a fundamental limitation in this process, where models make substantially more errors w...
Chenpan Li, Thies Brockmoeller, Stefanie Jegelka et al.· 0 citations