UI-Venus-2 is presented, a general-purpose foundation GUI agent designed to operate across mobile, web, and desktop environments through a unified closed-loop reasoning-action framework that integrates safety-aware mechanisms to ensure controlled execution of consequential actions.
Abstract
Multimodal GUI agents have emerged as a promising paradigm for digital task automation, yet transitioning from benchmark-oriented models to dependable real-world applications remains challenging due to limited environment coverage, brittle task construction, and unreliable reward verification. In this work, we present UI-Venus-2, a general-purpose foundation GUI agent designed to operate across mobile, web, and desktop environments through a unified closed-loop reasoning-action framework. To bridge the gap toward practical deployment, we jointly scale three critical dimensions: (1) Environments, expanding coverage to more than 170 multilingual mobile apps and native desktop operating systems; (2) Tasks, employing a deep-research pipeline for function-grounded instruction generation; and (3) Verification, adopting trace-level and sample-level evaluators with visual keypoints and multi-model voting to ensure reliable RL signals for training. Furthermore, we integrate safety-aware mechanisms to ensure controlled execution of consequential actions. By offering a capable, efficient, and open-source foundation, UI-Venus-2 advances the field toward more generalizable, verifiable, and self-reflective agents for real-world applications.
Experiments show that UI-Mate-27B sets a new open-weight state of the art on general computer-use benchmarks, substantially improving long-horizon reliability, and makes three contributions to an environment-grounded training stack with in-context demonstration learning.
Zihan Ding, Long-Xu Dou, Qixiao Gao et al.· 1 citation
Mobile task automation aims to streamline multi-step, cross-app interactions on smartphones and in-vehicle systems. Recent LLM-based GUI agents have advanced rapidly, yet reliability remains limited because modern mobile workflows are increasingly service-composed (app switching, forms, pop-ups/permissions, copy-paste)...
Wen-Tao Zhou, Si-Cong Liu, Zi-Mu Zhou et al.· Proceedings of the ACM on In...· 1 citation
CUA-Universe is introduced, a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments and training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration.
GMA is presented, a benchmark for evaluating general mobile assistants in challenging real-world scenarios, and shows that appropriate harness design can meaningfully improve performance, particularly on demanding workflows, while the effectiveness of specific designs can vary across foundation models.
Yi-Qi Zhu, Feiyu Gao, Jiakang Fan et al.· 0 citations
This work presents ComponentBench, a benchmark and diagnostic pipeline for component-level evaluation of computer-use agents on modern web UIs, and introduces a scalable pipeline for auditing realized structural difficulty after implementation and synthesizing structured failure analyses across tasks and component fami...
Tian-Chen Guan, Xinlei Lin, Royce Cheng-Yue et al.· 0 citations
The results establish harness-level capability composition as a practical route to broad, evolvable Omni systems without changing the host agent's reasoning core.
Yan-Lin Li, Ming-Yang Hao, Sheng-Qiong Wu et al.· 0 citations
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
Exploring how generative AI could make machine vision more accessible to businesses. The post GenEye in a Box: Making Machine Vision Something You Can Just Ask For appeared first on GPT-Lab.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.