Skip to content

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis

Sep 2026 · arXiv.org · Vol abs/2609.37233 · 0 citations · 43 references
Computer Science

TL;DR

DatalogBench is presented, a benchmark of 136 text-to-Datalog synthesis tasks curated from existing Datalog-based artifacts that identifies recursive reasoning and decomposition as open challenges for current LLMs and agents, and offers a reliable, execution-grounded measure of both.

Abstract

Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write. Existing synthesizers automate this task but require users to state their intent as input-output examples. Large language models (LLMs) suggest a more natural route, text-to-Datalog synthesis from a natural-language question, yet how well they do so has not been systematically evaluated. We present DatalogBench, a benchmark of 136 text-to-Datalog synthesis tasks curated from existing Datalog-based artifacts. Synthesized programs are graded by execution on held-out inputs against an oracle validated by mutation analysis. Across six LLMs and four prompting configurations, exact match peaks at 68.4%, and relation descriptions or an input-output example have only modest, model-dependent effects. Under direct prompting, most failures occur at compile time, typically because a model invents auxiliary predicates that it never declares or types consistently. Two coding agents reach up to 83.8% and eliminate nearly all such failures, leaving mostly semantic errors concentrated in recursive tasks. DatalogBench thus identifies recursive reasoning and decomposition as open challenges for current LLMs and agents, and offers a reliable, execution-grounded measure of both.

View source

Similar papers

Preprint Aug 2026

Code as Representation: A Compilable Parsing Paradigm for Academic Documents

Compilable Academic Document Parsing (CADP) is proposed, a paradigm that reconstructs a full page as contextual \LaTeX{} plus executable Python, so that structure-preserving elements and executable chart representations can be reconstructed, recompiled, and directly verified against the source page.

Rihui Jin, Jun Wang, Chen Zhu et al. · 0 citations
Sep 2026

Where the LLM Ends and Reliable Decisions Begin

The ANVIL compiler architecture is proposed, where ANVIL performs exceptionally well on leading benchmarks by using language models purely where they are effective, rather than as a catch-all tool.

Priyadarshan Patil, A. Basu, Vikas Reddy et al. · 0 citations
#artificial intelligence Conference Open access Sep 2026

An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks

This study empirically evaluates whether cost-efficient Large Language Models (LLMs) can be trusted to generate enterprise code to a written specification. Three models (Gemini Flash 3, GPT-5.4 mini and Claude Haiku 4.5) were asked to solve 992 algorithmic problems as Java Spring Boot service methods conforming to a ma...

Chandimal Adikari, N. Herath · 0 citations
Jul 2026

Developing and Benchmarking Verification Algorithms to Improve Text-to-SQL Generation

This work formalizes Text-to-SQL verification as a standalone task: assessing whether a candidate SQL query semantically satisfies a natural language request, without access to ground-truth labels, and proposes and compares two modular verification strategies.

Tarfah Alrashed, Madhup Sukoon, David R Karger et al. · 0 citations

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

SWE-Flux is introduced, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs.

Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband et al. · 1 citation
Open access Oct 2026

ScratchNet: A Multi-modal Benchmark for Evaluating and Advancing LLMs on Scratch Programming Tasks

Large language models (LLMs) have achieved impressive performance on text-based programming tasks, yet they remain unreliable for block-based languages such as Scratch. Scratch programs feature deeply nested, nonlinear structures, event-driven concurrency across multiple sprites, and tight coupling between code and mul...

Yuan Si, Si-Meng Han, Da-Ming Li et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.