This work presents AutoMat, a benchmark for evaluating LLM-based agents'ability to reproduce claims from computational materials science, and curates a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow needed to support (or undermine) such cla...
Ziyang Huang, Yiyun Cao, A. Shargh et al.· arXiv.org· 9 citations
This work introduces VBVR-Pro, a closed-loop testbed that makes native visual reasoning through generation trainable, verifiable, optimizable, and experimentally controllable, and identifies recurring failure modes of the prevalent VLM-as-a-judge paradigm.
Junhua Xu, Rui-Si Wang, Fanyi Pu et al.· 5 citations
The Information Abundance Paradox is supported and it is suggested that scaling toward near-infinite context is not simply a matter of supplying more data, even when high-quality long-context data is abundant.
Arda Uzunouglu, Benjamin Van Durme, Daniel Khashabi· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.