Prediction and evaluation of parameter optimization effect of large language model fine-tuning

Abstract

Large language models have demonstrated outstanding performance in the field of natural language processing, but pretrained models are difficult to adapt to specific tasks. Fine-tuning technology has become a key solution, but its application is limited by problems such as complex parameter selection, high training costs, and uncertain effect evaluation. This study aims to construct a complete framework for optimizing fine-tuning parameters of large language models, to achieve accurate prediction and scientific evaluation of fine-tuning effects, and to solve the problems of overfitting and high-performance adaptation under limited data. By constructing a multi-task mixed dataset, comparative experiments were designed using the controlled variable method. The influence of key hyperparameters such as learning rate and batch size was systematically analyzed, the performance of classic and modern optimization algorithms was compared, and a prediction mechanism was established by combining regression and classification models. A comprehensive evaluation system based on "performance-efficiency-resource" was constructed. The results show that a learning rate of 1E-3 and a batch size of 64 are the optimal configurations. Modern optimization algorithms converge faster, and multinomial regression and neural networks perform best in effect prediction. The performance of small-scale models after fine-tuning can surpass that of un-fine-tuned large models, significantly reducing computational costs. This study provides a scientific basis for optimizing fine-tuning parameters, improves resource utilization efficiency, and promotes the efficient implementation of large language models in vertical fields.