Jul 2026· International Journal of Semantic Computing· 0 citations
TL;DR
Experimental results demonstrate that GPS-Relational constructs executable reference SQL across existing benchmark and complex generated queries, and that LLM execution accuracy degrades significantly as nesting depth increases up to 15 levels.
Abstract
Large language models (LLMs) have improved natural language to SQL translation, but evaluating their outputs remains difficult when queries become complex, deeply nested, or produced at scale. Existing evaluations often rely on standard benchmarks with limited structural complexity or on model-based judgments that may be inconsistent.We present GPS-Relational, an evaluation framework that validates LLM-generated SQL by constructing an executable reference SQL query from the same natural-language intent and database schema. GPS-Relational builds this reference through grammarguided decomposition, schema mapping, semantic validation, and SQL assembly. The reference SQL is executed to produce the expected result, which is then compared with the execution result of the LLM-generated SQL, allowing equivalent SQL forms to be accepted without requiring textual or structural matching.
Additionally, GPS-Relational facilitates large-scale benchmark generation by systematically synthesizing extensive datasets of natural-language and SQL pairs with controlled nesting depth and relational complexity. These generated cases enable a systematic methodology to evaluate LLM performance across varied levels of structural complexity. Experimental results demonstrate that GPS-Relational constructs executable reference SQL across existing benchmark and complex generated queries, and that LLM execution accuracy degrades significantly as nesting depth increases up to 15 levels.
Logs are essential for system monitoring and failure diagnoses in modern software systems, yet querying them through natural language remains an open challenge. Existing approaches either treat logs as plain text, generate queries for schema-light backends, or assume predefined relational schemas, but none addresses a fundamental obstacle: raw logs carry no executable schema over which structured queries can be defined and run. To address these limitations, we present LogNLQ, a framework that formulates natural-language log querying as executable SQL generation over parser-induced and semantically grounded schemas. LogNLQ parses raw logs into template-partitioned relational tables, then applies dual-granularity semantic grounding to annotate both templates and parameter columns with interpretable names and descriptions. At query time, relevant schema candidates are retrieved via semantic search, and a large language model (LLM) generates executable SQL constrained to the retrieved context. To support rigorous evaluation, we introduce LogNLQ-Bench, an execution-verified benchmark of 8,895 queries over four real-world log datasets. Experimental results demonstrate that LogNLQ consistently outperforms all representative baselines by wide margins, with especially pronounced gains on analytically complex scenario queries.
Jue Wang, Jinyang Liu, Zhuangbin Chen et al.· 0 citations
It is suggested that structured prompt engineering provides a practical alternative to model fine-tuning for locally deployed LLMs, offering an effective balance between SQL generation accuracy, computational efficiency, and data privacy.
Nurjayanti Nurjayanti, A. Adiwijaya, A. Romadhony et al.· Jurnal RESTI (Rekayasa Siste...· 0 citations
Schema-Aware Query Translation and Tabular Reasoning for Enterprise Databases aka Inference-from-RDBMS is presented, an open-source framework designed for schema-aware query translation, dynamic context pruning, and execution-guided tabular inference over complex RDBMS structures.
Harshil Lodhiya· International Journal of Res...· 0 citations
This semantic path compilation (SPC) system against direct DDL-to-SQL generation on the ACME insurance benchmark supports an end-to-end systems result, not a causal claim that compilation alone produced the gain, because SPC receives governed semantic artifacts that the DDL baseline does not.
This work presents a training-free framework that formulates SQL correction as a plan-guided, tree-structured debugging process that mitigates error accumulation during iterative refinement and demonstrates the effectiveness and stability of the approach in real-world deployments.
Xinmei Huang, Jie Song, Peng Li et al.· 0 citations
Decisions are being made based on data, and the demands for easy-to-use interfaces that enable users to simply type in a question to a relational database without having to be a Structured Query Language (SQL) expert have increased. Although the development of Large Language Models (LLMs) has seen remarkable advancements in the past few years, and has led to the emergence of LLMs that have significantly improved the generation of SQL from natural language (Text-to-SQL)[1], there is a relative lack of systematic comparisons between the latest commercial LLMs that has been tested against a series of graded SQL complexity levels with both syntactic and execution-based metrics. This research aims to assess and contrast ChatGPT, Google Gemini, DeepSeek, and Microsoft Copilot for their accuracy and efficacy in creating SQL queries from natural language queries. In a quantitative experimental design, 100 natural language questions were given using a purpose-built benchmark, with six types of queries that included simple retrieval, filtering, joins, aggregation, GROUP BY and HAVING, and nested subqueries. The results of the models were assessed using the manually written and validated ground truth query, with the use of normalised Exact Match Accuracy (EMA) and Execution Accuracy (EXA). The best overall results (EMA 97%, EXA 97%) were obtained by ChatGPT. The Copilot scored EMA 56% and EXA 96%, DeepSeek EMA 55% and EXA 93% and Gemini the lowest with a score of EMA 36% and EXA 87%. The large margin between EMA and EXA for the models (+51 points, +40 points, +38 points) demonstrates the significant shortcomings of exact-match scoring when it comes to assessing real-world skills for producing valid but structurally different SQL. The study empirically demonstrates comparative behaviours of the current LLMs, and suggests the use of execution-aware evaluation in the studies and deployment of Text-to-SQL.
Bharath Kumar N, T L Manasa· International journal of com...· 0 citations