Large language models (LLMs) now show strong reasoning and coding abilities, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them. Existing benchmarks, however, mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and...
Lei-Lei Ding, Shu-Min Wang, Yu-Ting Huang et al.· 1 citation
This work presents $\Phi$-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization.
Lei-Lei Ding, Shu-Min Wang, Yu-Ting Huang et al.· 1 citation
The experimental results validate the effectiveness of the LabDex task design and demonstration data, and show that the benchmark supports the training and systematic evaluation of existing robotic policies across laboratory dexterous manipulation tasks at different levels, providing a foundation for further research a...
Zhipeng Tang, Sihan Chen, Sha Zhang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.