A benchmark built from human-designed, open-source circuits associated with Tiny Tapeout manufacturing shuttles is introduced, exposing distinct visual and design capabilities and, in one family, a policy rather than capability limit.
FDE-Bench evaluates this capability with 136 deployment-configuration tasks spanning Docker images, multi-service Compose stacks, and Kubernetes, in greenfield and diagnose-and-repair modes, finding readiness is the largest failure stage.
Wei-Hang Ding, Jun-Fei Zhan, Yue-Ting Li et al.· 0 citations
This work ran four frontier agents end to end end to end on metered L40S, A100, and H200 GPUs across 8B to 70B open bases, certifying every scenario before scoring, and ran a human FDE arm under the same oracle and compare every agent against it.
Wei-Hang Ding, Jun-Fei Zhan· Proceedings of the 2026 Conf...· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.