Evaluating the Efficacy of RAG-Enhanced Local LLMs in Cross-lingual Medical Information Retrieval: A Case Study on Indonesian to English Retrieval

Abstract

Retrieval-Augmented Generation (RAG) with local Large Language Models (LLMs) offers a reliable and secure way to retrieve medical information. However, when querying specific data in a different language (English) with external data like Indonesian pharmaceutical corpus, it introduces problems. This study evaluates an RAG pipeline using Google’s Gemma 3 model as well as Alibaba’s Qwen 3.5 model for medical question-answering and a Gemma 4 model as an automated judge capable of running in a local machine. The evaluated pipeline features a zero-shot hybrid retriever combining BGE-M3 dense vectors, BM25 sparse matching, and reranking. While the integration of RAG improved answer relevancy by 130%, our evaluation found a problem, a context recall of just 0.024. This caused the RAG setup to perform worse in overall factual accuracy than the non-RAG baseline. This paper serves as a warning against deploying zero-shot hybrid search in cross-lingual medical domains. We demonstrate how domain-specific vector misalignment and a lack of token overlap constrain standard retrieval methods, proving that current multilingual embedding models are insufficient for specialized clinical applications without the mandatory integration of pre-retrieval query translation layers.