NovelAPIBench: Diagnosing How A Code LLM Learns to Use Novel APIs
NovelAPIBench is presented, an automated diagnostic framework that builds benchmarks around the API knowledge gaps of a target model, identifies novel APIs, generates executable coding tasks with separately controllable knowledge components, and classifies failures into six categories.
Jin-Nuo Liu, Yue Peng, Jin-Hang Niu et al.
· 0 citations