Text-to-SQL with Large Language Models: Challenges Revisited and New Dimensions

GradeSQL: Outcome reward models for intelligent Text-to-SQL generation from LLMs

As Large Language Models (LLMs) become foundational to next-generation Intelligent Information Systems, the bridge between natural language interfaces and structured database systems remains a critical bottleneck. While Text-to-SQL generation enables cooperative support for complex query formulation, ensuring the reliability of these generated queries at inference time is a central challenge. Conventional methods rely on coarse execution-based signals, which may limit their ability to capture the nuanced semantic alignment required for high-stakes database environments. In this work, we propose the use of Outcome Reward Models (ORMs) as a fine-grained, probabilistic feedback mechanism for test-time verification in Text-to-SQL tasks. We introduce GradeSQL, a framework for training task-specific ORMs that assign scalar utility scores to candidate SQL queries based on their semantic correctness and alignment with database schema. Our approach is evaluated on the BIRD and Spider benchmarks across multiple open-source LLM families. Experimental results demonstrate that ORM-based verification consistently outperforms traditional execution-based heuristics.

Agentic-SQL Revisited: Autonomy-Based Taxonomy and Empirical Benchmark Analysis for LLM Text-to-SQL

LLM-based Text-to-SQL progress is reported across heterogeneous benchmarks, backbones, and inference protocols, making cross-system comparison fragile. We reframe the field as a leaderboard aggregation: we collect the metrics authors themselves report and organize them along an inference-autonomy axis spanning constrained, in-context, iterative, agentic, and reasoning-internalized generation, with traceable provenance for every cell. To anchor the aggregation empirically, we run a focused case study on Spider, comparing 8B open-source backbones with and without chain-of-thought (CoT) supervision against few-shot DeepSeek~V3 and GLM-4 baselines. Four patterns emerge: Spider gains transfer unevenly to BIRD and Spider~2.0; autonomy buys robustness at non-trivial cost; reasoning internalization sits between answer-only decoding and externally orchestrated agents; and CoT gains concentrate on Hard and Extra-Hard queries. We release a Python harness mirroring the autonomy axis so that future methods can be added directly to the leaderboard.

ABISS: Evaluating Text-to-SQL Systems Through Agent Interaction

A unified taxonomy of 8 categories covering ambiguous and unanswerable questions is addressed, a multi-agent generation pipeline with a two-stage process (NLQ generation followed by SQL grounding) and an explicit Category Conformance validation stage are addressed.

Large Language Model Vulnerabilities

The Poisoned Chalice of LLM Evaluation Report

The first Poisoned Chalice of LLM Evaluation Competition is organized, which frames contamination detection as a white-box membership inference task on source code and provides participants with curated datasets, target models, baseline attacks, and a final evaluation on a held-out model and dataset.

Automating End-to-End Hybrid Query Processing: Benchmark, Solution, and Insights

A large-scale benchmark with 60\sim 90× more queries than prior work, built on 3× more databases, an automated pipeline that can execute existing methods without manual intervention, and multi-dimensional, fine-grained evaluation metrics for comprehensive assessment.