How users actually use LLM-powered data systems is examined: how they struggle to specify and validate queries, how they iterate extensively to converge on bespoke operators specific to their documents and domains, and what these patterns imply for the systems and benchmarks their community should be building.
This study provides an initial assessment on the feasibility of using LLMs as a KB for CH, using “Galois”, a recent framework for executing Structured Query Language (SQL) queries over LLMs with logical and physical optimizations tailored to the model’s behavior.
Querying LLMs as digital libraries is feasible, but its effectiveness depends on model strength, deployment conditions, dataset structure, and execution strategy, and Galois remains valuable when relational discipline and controlled query execution are required.
A mixed-method, between-subjects user study comparing SQL-LLM, a GPT-4o-backed NLIDB, with Snowflake, a traditional SQL analytics platform finds that a verification burden survives the switch to natural language, so an NLIDB that hides the generated SQL would remove the very step users relied on to trust the answer.
A large-scale benchmark with 60\sim 90× more queries than prior work, built on 3× more databases, an automated pipeline that can execute existing methods without manual intervention, and multi-dimensional, fine-grained evaluation metrics for comprehensive assessment.
Bo Li, Chenzhan Wang, Longkang Lin et al.· Proceedings of the 32nd ACM...· 0 citations
In today’s data-driven environment, the ability to explore and manipulate datasets is essential, however, many existing data analysis tools require technical expertise in platforms such as Excel, Python, or Power BI, creating accessibility barriers for non-technical users. This project presents Quiksight, a web-based conversational data analysis system powered by Large Language Models (LLMs) that enables users to interact with datasets using natural language. The objective of this work was to develop an intuitive system that simplifies data analysis by allowing users to upload Excel or CSV files and perform operations such as filtering, sorting, summarization, column modification, missing-value handling, and data export without writing code. The system was implemented using Google Gemini as the LLM, FastAPI for backend processing, and HTML, JavaScript, and Tailwind CSS for the frontend interface.Evaluation showed that Quiksight successfully interpreted and executed a wide range of natural language data manipulation queries with high accuracy. Testing recorded an average query response time of approximately 2 seconds, while dataset uploads up to 30MB remained below 20 seconds under suitable network conditions. Compared with existing platforms such as Julius.ai and PowerDrill.ai, Quiksight demonstrated advantages through its lightweight architecture, simplified interaction model, and focus on everyday users. The project demonstrates the feasibility of LLM-powered conversational interfaces for democratizing data analysis. Future improvements include database integration, support for larger datasets, and advanced analytical capabilities such as statistical modelling.
Wumi Ajayi, Ihuanyachi O. OGBONNA, Daniel I. OGHINAN· Global Journal of Engineerin...· 0 citations
This work analyzes query logs from MotherDuck, a cloud analytics platform built on DuckDB, to show that real-world SQL extends far beyond analytical queries, with users frequently employing dialect-specific functionality that LLMs struggle with.
Till Döhmen, Adithya Krishnan, Hamilton Ulmer et al.· The VLDB journal· 0 citations