Clemson Off-Road Dataset (CORD): A Multimodal Benchmark for Autonomy beyond the Flat-World
TL;DR
“AI LabelMate,” a context-aware semi-automated annotation agent that fuses Vision-Language Models (Florence-2) with SAM2 to generate 6331 pixel-perfect annotated frames using a specialized off-road ontology and a human-in-the-loop pipeline, is introduced.
Abstract
While autonomous perception has matured within the structured confines of urban roadways, it remains brittle when confronting the chaotic, non-rigid terrain of the natural world. This paper introduces the Clemson Off-Road Dataset, a high-fidelity, multimodal dataset engineered to bridge this gap by challenging standard “flat-world” assumptions. Featuring 2.90 TB of sensor data, the dataset captures a diverse spectrum of unstructured environments, ranging from the transitional trails of CU-ICAR and the day/night lighting dynamics of TN3 to the unstructured wilderness of Camp Daniels and the novel coastal scenery of Edisto Island. Distinguishing itself from existing forest-centric benchmarks, the Clemson Dataset provides a first-of-its-kind focus on coastal data, featuring unique adversarial conditions such as extreme solar glare, loose sand, and shifting tide lines. The data is collected aboard a Polaris RZR Pro R 4, a high-performance platform integrated with a sensor suite designed to perceive physics beyond geometry. Alongside 360° HD camera coverage, 3D LiDAR, and Radar, we integrate Cubert Ultris Hyperspectral imaging and Prophesee EVK4 Event-based vision to enable material classification and high-dynamic-range motion tracking. To overcome the bottleneck in ground truth generation, we used our “AI LabelMate,” a context-aware semi-automated annotation agent that fuses Vision-Language Models (Florence-2) with SAM2 to generate 6331 pixel-perfect annotated frames using a specialized off-road ontology and a human-in-the-loop pipeline. We establish performance baselines using Oneformer for semantic segmentation and used SalsaNext for lidar point clouds labelling. Available in both raw ROS2 bag and extracted standard formats, this Dataset serves as a pivotal testing ground for the next generation of robust autonomous systems.The dataset of this paper is available upon request to the Virtual Prototyping of Autonomy-Enabled Ground Systems (VIPR-GS) Center.