Building a Multilingual AI Legal Assistant Using Retrieval-Augmented Generation: A Case Study on the Legal System of Kazakhstan

Abstract

This study presents the development of an Artificial Intelligence (AI)-based legal assistant using the Retrieval-Augmented Generation (RAG) architecture to provide legal assistance to citizens of the Republic of Kazakhstan. The proposed solution is designed to generate accurate, evidence-based responses to user queries using the regulatory legal acts of the Republic of Kazakhstan as the primary source of information. A legal corpus comprising 101,000 legislative documents and court decisions, with approximately 77 million tokens in Kazakh and Russian, was constructed to support the retrieval component of the system. To identify the most effective semantic retrieval method, three multilingual embedding models—Multilingual-E5-Large, BGE-M3, and KazEmbed-V5—were evaluated for vector search. The experimental results showed retrieval accuracies of 87.6%, 76.8%, and 83.3%, respectively. The GPT-5.4 and Llama-4-Scout-17B-16E-Instruct large language models were used to generate legal reasoning and responses based on documents retrieved through semantic search. The quality of the generated responses was evaluated using two complementary approaches. First, legal experts assessed the factual correctness and legal validity of the answers. Second, automatic evaluation was performed using word-level F1, BLEU, ROUGE, and BERTScore-F1 metrics. Among all evaluated configurations, GPT-5.4 combined with Multilingual-E5-Large achieved the highest overall accuracy (88.5%), whereas Llama-4-Scout-17B-16E-Instruct combined with KazEmbed-V5 achieved an accuracy of 83.6%. Based on the proposed architecture and the selected semantic retrieval and language models, an AI legal assistant was developed and integrated into the “Adal Azamat” legal services platform providing users in Kazakhstan with practical access to AI-assisted legal consultation.