Semantic query engines evaluate natural-language filters with a large language model, and every evaluation is a costly oracle call. Optimizers that reduce these calls face a dilemma: cascades that trust a cheap proxy miss their precision and recall targets on many queries, methods with finite-sample guarantees save lit...
SAGE achieves the strongest overall SemBench performance and, on a representative factorable join, reduces pairwise model calls by more than two orders of magnitude, yielding a 358-fold measured cost reduction.
Xiangqi Wang, Nhan H. Pham, Oktie Hassanzadeh et al.· 0 citations
This work demonstrates Text2SQL-Eval, an open-source, modular framework for rigorous evaluation of text-to-SQL systems, which provides a comprehensive suite of over twelve metrics, spanning execution accuracy, SQL syntactic equivalence, and LLM-as-judge scoring.
Oktie Hassanzadeh, Yotam Perlitz, Nhan H. Pham et al.· Proceedings of the VLDB Endo...· 0 citations
TEmBed is introduced, the Tabular Embedding Test Bed, a unified benchmark for systematically evaluating tabular embeddings across four representation levels: cell, row, column, and table and shows that which model to use depends on the task and representation level.
Liane Vogel, Kavitha Srinivas, N. D'Souza et al.· arXiv.org· 4 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.