Review

DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

Boyan Li Zhuowen Liang Yupeng Xie Xiaotian Lin Tianqi Luo Xinyu Liu Yizhang Zhu Zhangyang Peng Yuan Li Zhengxuan Zhang Jiayi Zhang Nan Tang Guoliang Li Yuyu Luo

Aug 2026 · 1 citation · 56 references

Computer Science

TL;DR

DataSpace, a benchmark in which data agents produce verifiable tabular results from task-local heterogeneous workspaces, is introduced and key challenges for improving data-agent reliability are identified.

Abstract

Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia. Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis, leaving heterogeneous evidence discovery, complete tabular outputs, and deterministic evaluation insufficiently unified. We introduce DataSpace, a benchmark in which data agents produce verifiable tabular results from task-local heterogeneous workspaces. It contains 410 cross-language tasks and 7,439 artifacts totaling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video. DataSpace also served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition. Each agent receives only a question and workspace and returns the complete requested tabular result. We construct DataSpace with DataSpace-Builder, an execution-grounded framework comprising cross-language transformation, constraint-aware relational sampling, modality routing and artifact rendering, and human review and task repair by 11 domain experts. A deterministic evaluator performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison. Across six recently released frontier multimodal models and five widely used agent harnesses, the best accuracy reaches 66.34%, while harness choice creates a 15.36-point spread with the backbone fixed. Multimodal evidence integration and joins consistently reduce accuracy across all six backbones. These results show that DataSpace remains unsaturated and identify key challenges for improving data-agent reliability.

arXiv Semantic Scholar

View source

Similar papers

Preprint Jul 2026

QwenPaw-Data: Bridging Facts, Methodology, and Execution for Autonomous Enterprise Data Analytics

Experiments on public benchmarks and real-world industrial BI workloads show that QwenPaw-Data improves both verifiable data access capability and higher-level analytical quality, offering a practical foundation for reliable, traceable, and continuously improving enterprise data agents.

Tian Zeng, Yuntao Hong, Zhongjun Ding et al. · 1 citation

Preprint Jul 2026

DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness

DataClawEval is introduced, the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios, and it comprises 100 rigorous, end-to-end tasks spanning five execution engines.

Debin Meng, Jiaming Yang, Zefang Zong et al. · 0 citations

Towards a Context Layer for Self-Improving Data Agents

It is argued that the key challenges are bridging the gap between personal and organizational memory, building effective agentic retrieval beyond naive RAG, and enabling organizations to evaluate their agents against their own data through self-serving evals.

Till Döhmen, MotherDuckNetherlands, Jacob Matson et al. · 0 citations

Preprint Aug 2026

Scientific Data Skills: Enabling Agent-Ready Scientific Data Services at Scale

Scientific data are increasingly used by AI agents, yet existing dataset representations provide limited support for autonomous discovery, interpretation, and invocation. This limitation stems from the fragmentation of scientific data across heterogeneous repositories and from dataset representations designed primarily for human use. To address this limitation, we introduce the Scientific Data Skill (SciDSK), an agent-ready representation that packages dataset-specific knowledge and operational guidance as a reusable agent skill. A SciDSK integrates dataset descriptions, scientific context, file organization, usage procedures, quality checks, and provenance information while retaining the underlying data in its original repository. We define a structured SciDSK specification and develop a systematic construction pipeline that grounds each SciDSK in authoritative dataset records and associated supporting materials. We further establish the Scientific Data Skill Bank, a unified platform that publishes SciDSK resources across six scientific disciplines and supports package access, persistent identification, and traceability to source datasets. We evaluate SciDSK through a retrieval benchmark for dataset discovery and controlled cases for dataset interpretation. The results show that SciDSK improves agent-driven dataset discovery and provides more precise and actionable support for dataset interpretation. These findings support the value of organizing dataset-specific knowledge in an agent-ready representation.

Xiaohan Huang, Qingqing Long, Xiaolei Du et al. · 0 citations

Preprint Aug 2026

MAP-Graph: Provenance-Aware Shared Memory for Multi-Agent Workflows

MAP-Graph is introduced, a provenance-aware memory layer that represents agents, sources, memories, claims, and actions in a typed execution graph and supports provenance as an operational control signal, rather than only post-hoc audit metadata, within the evaluated setting.

Yiqi Wang, Zihao Yan, Jiaqi Zhang et al. · 4 citations

DeepEye: A Workflow-Centric Agentic Data System for Steerable Data Analytics

This work presents DeepEye, a workflow-centric agentic data system that turns user intents into transparent and steerable analytical workflows and develops DataMagic as the system’s Video Generator, a declarative multi-agent method that improves data-video quality.

Unknown authors · 0 citations