Skip to content
Preprint

VABAR: A Variational Automation Benchmark for Agentic Robotics

Nov 2026 · 1 citation · 33 references
Computer Science

TL;DR

A general set of agentic robotics benchmark design criteria that includes accessibility, repeatability, diversity, relevance to agentic robotics and practical applications, physical realizability, resistance to benchmark saturation, and computational efficiency is proposed.

Abstract

For robots to operate reliably in commercial and industrial applications, they must persistently perform specific tasks with known variations in object shape and pose. We call this Variational Automation (VA) to differentiate these settings from fixed automation. Inspired by recent papers on Agentic Robotics, where pretrained LLM and VLM coding agents combine interpretable robot programming with the open-world adaptability of model free policies, we propose a general set of agentic robotics benchmark design criteria that includes accessibility, repeatability, diversity, relevance to agentic robotics and practical applications, physical realizability, resistance to benchmark saturation, and computational efficiency. We instantiate these principles in six manipulation task classes spanning industrial, medical, and infrastructure applications: tool sorting, tool hanging, gear meshing, syringe packing, cable insertion, and cable routing. We introduce the Variational Automation Benchmark for Agentic Robotics (VABAR), an open, easily replicable, simulation-based manipulation benchmark to systematically evaluating agentic robotics systems. Built fully on open-source tools, VABAR features detailed implementations using the Newton physics engine, standardized task variations, difficulty tiers, success metrics, and a library of 33 generalizable skills with human-authored programmatic reference solutions. We provide initial baseline evaluations using an open agentic manipulation harness, Graph-as-Policy (GaP), and Isaac Lab Arena.

View source

Similar papers

Preprint Oct 2026

LABIT: Long-Horizon Robotic Assembly Benchmark for Industrial Tasks

This work proposes LABIT, an industrial robotic assembly benchmark for long-horizon manipulation, which encompasses DIN-norm assembly tasks, including screwing, placing, inserting, and snapping, under both high and low tolerance conditions.

Constantin Schempp, Christian Friedrich, L. Kofler et al. · 1 citation
Preprint Nov 2026

RoboARCade: Benchmarking Agentic Robot Control

Improvements in multi-modal capabilities have enabled agents to act directly as robot policies, a setting we refer to as agentic robot control (ARC). We introduce RoboARCade, a benchmark of long-horizon manipulation tasks across three embodiments designed to test in-context adaptation, memory, and reasoning for precise...

Satvik Sharma, Heng-Yuan Hu, Mihika Kamat et al. · 0 citations

SimEX: Simulation-Integrated Robotics AutoResearch

Coding agents powered by large language models (LLMs) have shown remarkable abilities to autonomously reason about and achieve goals in the digital world. However, bringing this success to the physical world remains challenging. On the one hand, direct generation methods (e.g., Code as Policies) often suffer from the L...

Jiaheng Hu, R. Martin, Peter Stone et al. · 4 citations · ⚡1
Review Nov 2026

LEASH: Stay Safe in Agentic Robotics

Foundation models such as GPT and Claude have shown strong capabilities in spatial understanding and embodied intelligence. These advances make agentic robotics a promising path toward artificial general intelligence in robotics. However, we find that using foundation models to control robots poses serious safety risks...

Qin-Feng Zhu, Weiguang Zhao, Ke Sun et al. · 0 citations
Preprint Nov 2026

On Agentic Manipulation Policies

Recent work on robot manipulation has focused on specialist policies trained on demonstrations. We show that generalist AI models can control manipulators more reliably than specialist policies such as vision-language-action models (VLAs) and world-action models (WAMs). We systematically analyze agentic manipulation po...

Chen Chen, Omid Bagherieh, Naveen Nikamanth Appiah Balaji et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.