ADAPTIVE TURKISH FEDERATED RAG ARCHITECTURE FOR LOW-RESOURCE SYSTEMS
TL;DR
This study examines distributed learning approaches aimed at protecting data privacy and QLoRA-based fine-tuning strategies within a mathematical framework and demonstrates that combining RAG-based systems with federated learning and homomorphic encryption creates a secure, decentralized architecture.
Abstract
Today, Large Language Models (LLMs) perform many tasks in the field of natural language processing with high success, from text generation to translation, semantic analysis to code writing. However, these models have some fundamental limitations that make their reliable use challenging. They can produce factual errors known in the literature as hallucinations and cannot directly access developments after their training period. They also sometimes reproduce biases present in the training data. The Re-trieval-Augmented Generation (RAG) approach aims to produce more up-to-date and verifiable outputs by dynamically feeding the model with external information sources, thereby reducing hallucination and temporal limitations. This study examines distributed learning approaches aimed at protecting data privacy and QLoRA-based fine-tuning strategies within a mathematical framework. It also addresses 4-bit NormalFloat (NF4) quantization techniques used to improve system efficiency. Furthermore, it de-tails morphology-aware hybrid access methods and adaptive routing mechanisms that can make deci-sions based on query complexity to achieve results more suitable for Turkish. The study demonstrates that combining RAG-based systems with federated learning and homomorphic encryption creates a secure, decentralized architecture, and that LLM can be efficiently run on low-resource systems using the NF4-QLoRA combination. However, limitations such as encryption latency, erroneous information from untrusted sources, and the inadequacy of standard evaluation methods for Turkish also exist.