Performance of local Medical Large Language Models as clinical decision support systems under connectivity constraints

Abstract

In the context of military health support, medical teams frequently operate under extreme conditions, characterized by severe connectivity restrictions, high cognitive pressure, the need for constant mobility, and severe limitations in computational and logistical resources. In this sense, the objective of this study was to investigate the performance of Large Language Models (LLMs), fine-tuned for medical applications, executed locally (offline), considering their application in operational scenarios with connectivity constraints. The experiment followed the classic validation pipeline of Clinical Decision Support Systems (CDSS). The experiment was executed using Python code on two hardware architectures: with a GPU and with a CPU (without a GPU). Eight models were tested using three reference medical benchmarks. The evaluation metrics were: Clinical Accuracy (%), Average Latency per Question (in seconds), and Tokens per Second (TPS). The results show that quantized models in the 8-billion parameter range, especially LlaMA-3.1-8B-Medical (Q4_K_M), have significant potential to act as force multipliers, achieving accuracies of up to 74% in complex medical benchmarks (such as MMLU_Medical and MedMCQA). However, artificial intelligence tools must be integrated into the battlefield exclusively as cognitive assistants under human supervision.