Self-Consistency (SC) is a decoding strategy that samples diverse reasoning paths and selects the most consistent answer, demonstrating strong performance on complex reasoning problems. However, the excessive token consumption incurred by generating multiple reasoning paths has been identified as a major limitation of SC. To improve computational efficiency, several studies have proposed strategies that adjust the number of reasoning paths or allocate resources differentially according to problem difficulty. Nevertheless, most existing methods categorize difficulty into a few fixed levels, failing to fully capture the continuously varying nature of reasoning complexity. In this work, we propose Flexible Self-Consistency (FSC), which estimates problem difficulty as a continuous signal and dynamically adjusts the number of generated reasoning paths accordingly. FSC predicts the output entropy of an input question using a pre-trained probe and leverages it as an indicator of model uncertainty to flexibly control the sampling budget. Experimental results show that, across various models and benchmarks, FSC maintains accuracy comparable to SC while achieving token savings of up to 76%.
Sihyeong Yeom, Geon Park, Geunyeong Jeong et al.· 0 citations
Multimodal Large Language Models (MLLMs) have achieved remarkable progress in multimodal understanding and reasoning by integrating linguistic and visual information, while benchmarks facilitate iterative model improvement by evaluating their performance and analyzing their limitations. However, existing dialogue-based multimodal benchmarks do not fully reflect the characteristics of real-world interactions, as they often construct a single, lengthy user utterance to provide all requirements or treat visual information as static even in multi-turn conversations. To address these limitations, we propose the Multi-turn Multimodal Interactive Dialogue (MMID) Benchmark, where user requirements are incrementally conveyed across turns and images are interleaved with text throughout the conversation to enable dynamic multimodal interaction. With this design, MMID enables comprehensive evaluation of the Perception, Memorization, and Reasoning abilities of MLLMs. Furthermore, while most tasks adopt a multiple-choice question format, each incorrect option is mapped to fine-grained error types, enabling an analysis of model strengths and weaknesses beyond coarse-grained performance comparison. MMID reveals MLLMs perform well with text-based input but degrade with images, requiring improved leverage fine-grained visual cues. Our benchmarks and detailed descriptions are available at https://github.com/KUNLP/MMID.
Seulgi Kim, Juoh Sun, Sumin Kim et al.· Proceedings of the 32nd ACM...· 0 citations
