Skip to content

GraphDecisionBench: Benchmarking System One Models on Graph Tasks

Oct 2026 · 0 citations · 15 references
Computer Science

TL;DR

GraphDecisionBench is introduced, a model-independent benchmark that combines structural task profiles, matched graph-text input contrasts and heuristic-proposal controls to diagnose graph decision performance and illustrates the benchmark's central distinctions.

Abstract

Large language models (LLMs) are increasingly explored for graph understanding and decision-making, while System One models such as Jev select directly from supplied options. However, the capabilities of System One models on graph-related tasks remain unclear. We introduce GraphDecisionBench, a model-independent benchmark that combines structural task profiles, matched graph-text input contrasts and heuristic-proposal controls to diagnose graph decision performance. We evaluate Jev and related choice-based models alongside language-model baselines, covering fourteen model-interface configurations. Jev's results illustrate the benchmark's central distinctions: accurate adjacency recognition does not guarantee broader structural correctness, joint graph-text input does not consistently improve prediction, and feasible construction does not establish high solution quality. Its task contracts, candidate interfaces and scoring rules support comparison across native selectors and language-model adapters. Code and aggregate results are available at https://github.com/VictorYXL/GraphDecisionBench.

View source

Similar papers

Open access Sep 2026

From Compression to Execution: What Helps Large Language Models Digest Urban Graphs for Spatial QA?

These results should be interpreted as a comparison of graph-language interaction designs rather than a fully input-matched ablation, but they suggest that exposing task-relevant structural information is an important factor for urban spatial QA.

Rachid Oucheikh, A. Mansourian · 0 citations
#artificial intelligence Preprint Sep 2026

GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMs

The Graph Theory Agent (GTA), which pairs a preference-trained representation selector with plan-and-decompose scaffolding around a frozen executor LLM, is proposed, which lifts Phi-4 from 53.5% to 69.1% on the benchmark's easy split and from 33.0% to 41.5% on its hard split.

Zi-Xiang Xu, Yan-Bo Wang, Chenxi Wang et al. · 4 citations · ⚡1
#artificial intelligence Preprint Sep 2026

Evaluating and Benchmarking the System One Model Jev

Jev answers MMLU's calculation-heavy questions more accurately than other MMLU questions (94% vs. 91%), whereas both open models, and all three on C-Eval, find them harder.

Tobias Deußer, Rajkumar Ramamurthy, L. Sparrenberg et al. · 33 citations · ⚡7
#machine learning Preprint Sep 2026

NodeGround: A Node Classification Benchmark in the Graph Foundation Model Era

Can a pretrained graph model replace training and tuning a separate predictor for each dataset? Answering this requires evaluating prediction quality alongside computational cost. We present NodeGround, a node classification benchmark that puts graph foundation models (GFMs) and dataset-specific supervised learning und...

Jin-Mo Lee, Dooho Lee, Minho Jeong et al. · 0 citations
Open access Sep 2026

Research on Structured Output of Large Language Models Based on Dual-Dimensional Testing

The results show that JSON, although widely used in practice, is not always the most efficient structured output format, and output formats and constraint methods should be selected according to task shape, latency budget, model capability, and tolerance for downstream repair.

Chao Zhang · 0 citations
#artificial intelligence Conference Open access Oct 2026

GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation

Large language models (LLMs) have demonstrated strong performance on coding and reasoning benchmarks; however, their ability to solve graph-structured machine learning problems remains largely unexplored. In particular, no benchmark currently evaluates whether LLMs can autonomously solve end-to-end Graph Neural Network...

M. Hossen, Tasneem Selim, Gurur Gamgam et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.