Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Lin Shi (Audrey)Haowei Lin (Audrey)Zixuan Zhu (Audrey)Xiaoyue Zhou (Audrey)Xiang Li (Audrey)Xiangning Lin (Audrey)Yaxuan Deng (Audrey)Han Xu (Audrey)Yuangang Li (Audrey)Shanda Li (Audrey)Zizhao Chen (Audrey)Hanwen Xing (Audrey)Harsh Raj (Audrey)Bo Chen (Audrey)Quan Shi (Audrey)Steven Dillmann (Audrey)Yipeng Gao (Audrey)Puneesh Khanna (Audrey)Ruofan Lu (Audrey)Chao Beyond Zhou (Audrey)Michael Yang (Audrey)Robert Zhang (Audrey)Siyuan Chai (Audrey)Jiayu Chang (Audrey)Yizhao Chen (Audrey)Xiaokun Chen (Audrey)Yiwei Dai (Audrey)Wenting Yang (Audrey)Hange Liu (Audrey)Minghao Liu (Audrey)Zihan Wang (Audrey)Adnan El Assadi (Audrey)Benedikt Stroebl (Audrey)E. Kelly Buchanan (Audrey)Han Meng (Audrey)Junwei He (Audrey)Longxuan Yu (Audrey)Radin Shayanfar (Audrey)Yukyung Lee (Audrey)Zhikang Dong (Audrey)Allen G Hart (Audrey)Anjiang Wei (Audrey)Anurag Kashyap (Audrey)Arpandeep Khatua (Audrey)Audrey Jixin Zheng (Audrey)Chengrui Ma (Audrey)David Heineman (Audrey)Dubing Chen (Audrey)Hai-Anh Trinh (Audrey)Haishuo Fang (Audrey)Hefan Zhang (Audrey)Hui Shen (Audrey)Issa Sugiura (Audrey)Jiankai Sun (Audrey)Jiechao Gao (Audrey)Junhong Lin (Audrey)Junnan Li (Audrey)Kai Yang (Audrey)Lei Hsiung (Audrey)Maoyu Wang (Audrey)Mengze Tang (Audrey)Nabil Omi (Audrey)Negin Raoof (Audrey)Nicholas Edwards (Audrey)Octavia Guo (Audrey)Orfeas Menis Mastromichalakis (Audrey)Pengliang Ji (Audrey)Przemys{\l}aw Hejman (Audrey)Qi Qi (Audrey)Qunshu Lin (Audrey)Richard Zhuang (Audrey)Rui Yang (Audrey)Ruichen Zheng (Audrey)Ryan Marten (Audrey)Shaghayegh Fazliani (Audrey)Shizheng Hou (Audrey)Sicong Jiang (Audrey)Sijie Li (Audrey)Song Bian (Audrey)Terry Yue Zhuo (Audrey)Tianqing Wu (Audrey)Tom Tang (Audrey)Wanjia Zhao (Audrey)Weihao Xuan (Audrey)Wenhua Liang (Audrey)Xian Liu (Audrey)Xin Lan (Audrey)Xuan Zhang (Audrey)Xuandong Zhao (Audrey)Yanchuan Tang (Audrey)Yifan Jiang (Audrey)Yijiang Li (Audrey)Yitong Guan (Audrey)Yizhi Li (Audrey)Yonghui Liu (Audrey)Yuheng Tang (Audrey)Yujun (Audrey)MaoYunfei ZhaoYuxin WangYuxuan TangZhenheng TangZhifei LiZiruo WangZiyu SheKaiyuan LiuIheb ChaabaneYuxin TangXiangyi LiAndy KonwinskiBoxuan LiLeon Liangyu ChenAlex DimakisNicholas CarliniSoroush VosoughiDi HeEtash GuhaBenjamin FeuerMike MerrillLudwig SchmidtAlex Shaw
Sep 2026
Artificial IntelligenceNatural Language Processing
Abstract
Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a unified evaluation infrastructure for agentic benchmarks. Our work makes three contributions. First, we develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents, and validate them through rigorous code review and parity experiments. Second, we conduct a large-scale evaluation of 8 models spanning capability tiers across 54 benchmarks; every model is run with Terminus-2 and with one of 3 native harnesses. This enables a broader analysis of agent capabilities and failure modes than was previously possible. Third, we introduce Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks, refined from the adapted suite through difficulty filtering, AI and human audit, and an audit-and-fix loop. Harbor-Index preserves the challenge and breadth of large-scale agentic evaluations while being affordable to run; no evaluated model-harness configuration exceeds 30% pass rate, and the strongest (GPT-5.5 with Codex) reaches 28.0%. We release the adapters, evaluation results, in-depth analysis, and Harbor-Index as open-source artifacts to support more reliable and comprehensive evaluation of language-model agents.
This publication proposes a definition and a classification of agile software development approaches and analyses ten software development methods that can be characterized as being "agile" against the defined criterion.
P. Abrahamsson, O. Salo, Jussi Ronkainen et al.· arXiv.org· 727 citations· ⚡54
Consequences of happiness and unhappiness that are beneficial and detrimental for developers' mental well-being, the software development process, and the produced artifacts are found.
D. Graziotin, Fabian Fagerholm, Xiaofeng Wang et al.· Journal of Systems and Softw...· 236 citations· ⚡13
Mobile phones have been closed environments until recent years. The change brought by open platform technologies such as the Symbian operating system and Java technologies has opened up a significant business opportunity for anyone to develop application software such as games for mobile terminals. However, developing mobile applications is currently a challenging task due to the specific demands and technical constraints of mobile development. Furthermore, at the moment very little is known about the suitability of the different development processes for mobile application development. Due to these issues, we have developed an agile development approach called Mobile-D. The Mobile-D approach is briefly outlined here and the experiences gained from four case studies are discussed.
P. Abrahamsson, Antti Hanhineva, H. Hulkko et al.· Conference on Object-Oriente...· 225 citations· ⚡18
GAOKAO-Bench is introduced, an intuitive benchmark that employs questions from the Chinese GAOKAO examination as test samples, including both subjective and objective questions that contribute a robust evaluation benchmark for future large language models and offers valuable insights into the advantages and limitations of such models.
Xiaotian Zhang, Chun-yan Li, Yi Zong et al.· arXiv.org· 216 citations· ⚡17
A study with 42 participants investigates the relationship between the affective states, creativity, and analytical problem-solving skills of software developers and offers support for the claim that happy developers are indeed better problem solvers in terms of their analytical abilities.
D. Graziotin, Xiaofeng Wang, P. Abrahamsson· PeerJ· 216 citations· ⚡13
This state-of-practice investigation was performed using a literature review followed by a multiple-case study approach and presents how inconsistency between managerial strategies and execution can lead to failure by means of a behavioral framework.
Carmine Giardino, Xiaofeng Wang, P. Abrahamsson· International Conference on...· 175 citations· ⚡19
