Evaluating Vision-Language Models for Emotion Recognition: A Comparative Study of In-Context Learning, LoRA Fine-Tuning, and CNN Baselines on FER-2013
Abstract
Emotion recognition from facial expressions is an important task in computer vision and has recently attracted attention in the context of Vision-Language Models (VLMs). However, most existing studies emphasize outcome-level performance, with limited analysis of how different adaptation paradigms behave on fine-grained affect recognition tasks. This work presents a controlled paradigm-level evaluation of a modern VLM, LLaMA 3.2 11B Vision, on the FER-2013 dataset, focusing on in-context learning (ICL) and parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA). Zero-shot, one-shot, and few-shot prompting are examined alongside LoRA adaptation with frozen and retrained vision towers and compared against a conventional CNN baseline under a unified experimental protocol designed to analyze paradigm-level behavior. The results reveal clear paradigm-dependent trends and limitations. Zero-shot inference achieves moderate accuracy (54.30%), while one-shot and few-shot ICL degrade substantially (18.36% and 16.94%), reflecting the sensitivity of prompt-based adaptation to FER-2013’s low resolution and class imbalance. LoRA fine-tuning improves performance, particularly when retraining the vision tower (58.34%), yet remains below the CNN baseline (63.44%). These findings indicate that although VLMs exhibit strong representational capacity, their adaptation mechanisms remain fragile for low-resolution facial emotion recognition. Overall, this study characterizes the strengths and failure modes of ICL and parameter-efficient fine-tuning in a controlled setting, positioning VLMs relative to established CNN approaches and providing methodological insight into their practical reliability for affective computing. Future work will explore higher-quality datasets (e.g., RAF-DB, ExpW), different VLM scales, improved few-shot support selection, and multimodal extensions incorporating audio, text, or facial landmarks.