Evaluation of college student tutoring dialogue based on large language model prompting engineering

Abstract

The dialogue evaluation of student counseling in colleges and universities relies on human supervision, which faces the problems of strong subjectivity, high cost and lagging feedback, and the consistency between different supervisors is only medium. Different from clinical counseling, college counseling focuses on developmental issues such as academic pressure and interpersonal relationship. Due to the lack of strategy alignment for counseling micro-skills, the existing large model zero-sample prompt evaluation is prone to score inflation, standard deviation, and is significantly affected by the change of prompt words. To this end, this study constructs a dialogue evaluation dataset labeled by senior consultants according to the competency model for multiple rounds, covering six indicators such as emotional reflection, open inquiry, empowerment reconstruction and security risk. On this basis, a fine-tuning evaluation method is proposed to parameterize the tutoring micro-skill evaluation system into the large model. Based on LAMA-2-7B, a prompt template with embedded evaluation anchors and stepwise reasoning is designed, and QLoRA low-rank adaptation is used for instruction fine-tuning. Contrast samples are constructed for confusion dimensions such as "empathy-compassion" and "leading-preaching", and the preference ranking and supervised contrast loss optimization are combined. The training is completed on a single consumer-level GPU, and the motivation generation is fine-tuned synchronously to enhance the transparency of evaluation. GPT-4, Claude, no fine-tuning base and pure cue engineering baseline were selected for comparison. On real tutoring records, the proposed method significantly outperforms all comparison models in terms of weighted Kappa, fine-grained F1 and cross-dialogue length stability. Ablation studies confirmed that the joint contrast preference optimization made a prominent contribution to suppressing the inflation of confounding dimension scores, and the evaluation illusion and position deviation were also significantly reduced.