Behavior-Conditioned Symbolic Synthesis is introduced---an approach for automated search of interpretable, deterministic symbolic programs that verifiably reproduce behavior---and applied to model both human and machine behavior across hundreds of challenging reasoning problems, pointing to underlying mechanisms remini...
Solim LeGris, Philipp Mondorf, Jin-Ran Jin et al.· 0 citations
This work takes a fine-grained look at memorization by decomposing the loss trajectory of memorized sequences over training and model parameters, finding that memorization in both cases is characterized by sequence-level gradient alignment.
Florian Eichin, Philipp Mondorf, Andrei Mircea et al.· 0 citations
We propose the AIMO Interpretability Challenge, a competition on distinguishing robust from spurious reasoning in frontier mathematical language models based on the models'internal mechanisms. The challenge is motivated by a central limitation of standard reasoning benchmarks: strong final-answer accuracy does not reve...
Michal vStef'anik, Philipp Mondorf, Andreas Waldis et al.· 0 citations
The Last Translation Benchmark is introduced, a collection of human-authored and peer-reviewed examples that break leading machine translation models and a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and a...
Vilém Zouhar, Niyati Bafna, Mukund Choudhary et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.