Benchmarking Agentic Text-to-SPARQL for Multi-Hop Scholarly Knowledge Graph Exploration

Abstract

Conversational access to scholarly knowledge graphs promises more intuitive ways of exploring research fields, but current approaches remain limited by two major barriers: the lack of realistic multi-hop benchmarks and the brittleness of Text-to-SPARQL methods over real-world schemas. We address these gaps by introducing a benchmark of challenging, researcher-authored questions grounded in a domain-specific subset of DBLP. The benchmark targets realistic multi-hop scholarly information needs rather than synthetic or template-based queries. We evaluate five large language model-based SPARQL generation pipelines with increasing levels of schema grounding, decomposition, rule guidance, and endpoint feedback. Our results show that baseline and sequential approaches remain brittle, while a rule-augmented pipeline substantially improves execution accuracy and semantic alignment with the intended information need. The study contributes both a new testbed for evaluating conversational access to scholarly knowledge graphs and practical design insights for building more reliable, schema-aware Text-to-SPARQL agents.