Comparison of natural language models for searches in virtual observatories

Abstract

This work presents the development and evaluation of a web platform called Virtual Observatory (VO) Artificial Intelligence Search, enabling natural language queries on the astronomical VOs using state-of-the-art LLMs. The architecture includes an interactive frontend, coordinating backend, database, search engine compatible with IVOA’s SIAP protocols (International Virtual Observatory Alliance, Simple Image Access Protocol), and two LLMs: GPT-4o and DeepSeek-R1. The main goal was to compare their performance in retrieving astronomical images, assessing precision, response time, resource consumption, and sustainability. Twenty-one thematic queries per model were executed, measuring Recall@1, Recall@3, MRR, tokens, cost, and carbon footprint. DeepSeek-R1 achieved higher Recall@3 (86% vs. 71%), lower cost and environmental impact, while GPT-4o was faster and used fewer tokens. The evaluation shows distinct strengths enhancing semantic access to large astronomical datasets.