This work takes a fine-grained look at memorization by decomposing the loss trajectory of memorized sequences over training and model parameters, finding that memorization in both cases is characterized by sequence-level gradient alignment.
Florian Eichin, Philipp Mondorf, Andrei Mircea et al.· 0 citations
An extensive evaluation shows that, on classification-style benchmarks, adding NLEs to few-shot prompts often improves accuracy over few-shot prompting without explanations; among NLE sources, externally generated LLM-NLEs often provide strong downstream utility and remain competitive with human rationales where both a...
Mahdi Dhaini, Adam Dejl, Juraj Vladika et al.· 0 citations
We propose the AIMO Interpretability Challenge, a competition on distinguishing robust from spurious reasoning in frontier mathematical language models based on the models'internal mechanisms. The challenge is motivated by a central limitation of standard reasoning benchmarks: strong final-answer accuracy does not reve...
Michal vStef'anik, Philipp Mondorf, Andreas Waldis et al.· 0 citations
Cultural evaluation of large language models (LLMs) often focuses on high-resource standard languages, leaving regional culture and dialect communities underrepresented. We introduce BavGround, a benchmark for evaluating Bavarian regional cultural grounding and dialect competence across English, German and Bavarian. Ba...
Jophin John, Michael Hoffmann, Jan Fillies et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.