VitalLM: Compact Language Models with Depth Up-Scaling for Advanced Contextual Understanding

Abstract

Even though Large Language Models (LLMs) can perform numerous tasks in medicine, they have not yet found widespread use in practice because of their complex computational requirements. In order to tackle the abovementioned issues, we introduce VitalLM, a Small Language Model designed to make medical AI more accessible, doubling its use on traditional computers. VitalLM is a simple model consisting of only 50 million parameters which has been trained on solid texts of 764 million biomedical tokens and thus makes it very quick in response time on ordinary machines. However, we also introduce a new tool known as Depth UpScaling (DUS) which allows for duplicating layers to learn novel features without the need to invest in training the model from scratch. VitalLM is a decoder-only transformer model based on SwiGLU feed-forward layers that help swift completion of language-related tasks. Its effective structure allows for fine tuning and achieving good results on conventional computers.