Generative vs. Discriminative? An Empirical Study on Code Understanding Classification Tasks

Abstract

Automated code understanding is crucial for software reliability and maintainability. Encoder-only pretrained models excel in code classification tasks such as vulnerability detection, cross-language clone detection, and exception classification due to their bidirectional context awareness. However, the dominant “pre-training then fine-tuning” paradigm suffers from a fundamental objective mismatch: generative masked language modeling during pretraining versus discriminative classification during fine-tuning. This gap hinders direct knowledge transfer, forcing models to reconstruct semantic spaces, a process prone to overfitting in resource-scarce or long-tail scenarios common in code understanding classification tasks. To bridge this gap, we conduct a systematic empirical study of the prompt-based fine-tuning approach for code understanding classification tasks. By constructing prompt templates with task-specific mask tokens, we reformulate heterogeneous downstream tasks into unified cloze-style problems aligned with pretraining objectives. This approach directly activates latent semantic knowledge without extensive parameter reinitialization. Experiments on three datasets show that the prompt-based fine-tuning approach generally outperforms traditional fine-tuning in F1-score under full-data settings and demonstrates strong data efficiency in low-resource scenarios. With only 0.5% of the training data, it improves the F1-score by 30.52 and 13.95 percentage points on cross-language clone detection and exception classification, respectively. When combined with optimization strategies, it also enhances robustness to class imbalance and samples that are hard to classify. This work reveals the potential of prompt learning to unlock encoder model capabilities, offering new perspectives and pathways to overcome data scarcity in intelligent code understanding classification tasks.