Fine-Tuning Large Language Models for Financial Applications Using Direct Preference Optimization

Abstract

Large language models have changed natural language processing across many fields, but their general-purpose design limits performance in specialized domains like finance. This study examines the adaptation of Meta-Llama-3-8B-Instruct for financial NLP tasks through parameter-efficient fine-tuning methods and direct preference optimization. We use LoRA and QLoRA techniques with the Sujet-Finance-Instruct-177k dataset covering six financial task categories. While parameter-efficient finetuning improves performance on structured tasks like sentiment analysis and topic classification, complex question-answering tasks show limited improvement. To address this limitation, we considered direct preference optimization using synthetically generated preference pairs from GPT-4o. A Task Completion Score was created that uses an LLM as a judge and a GPT-4 model and defines model performance based on instructions in the domain. Results demonstrate that combining parameter-efficient fine-tuning with direct preference optimization increases model performance in challenging question-answering scenarios. Our findings show that preference-based optimization is essential for optimal performance in domainspecific applications requiring both accuracy and alignment with user expectations.