Automated Department Classification of Clinical Discharge Summaries Using Ensemble Transformer and Classical Machine Learning Models: A Cross-Validated Study on MIMIC-III

Abstract

Manual routing of clinical discharge summaries to the appropriate hospital department is a time-consuming task that could benefit from automated text classification. Real-world clinical corpora such as MIMIC-III present two compounding difficulties: severe class imbalance across department categories, and the scarcity of reliable ground-truth labels, since department assignment is not natively recorded and must be derived indirectly. This paper presents a systematic comparison of six machine learning approaches – Logistic Regression, Linear Support Vector Machine, Random Forest, Naive Bayes, a domain-adapted transformer (Bio_Discharge_Summary_BERT), and a soft-voting ensemble of the transformer with the best classical model – for classifying MIMIC-III discharge summaries into department categories derived from ICD-9 diagnosis codes. We construct both a fine-grained six-class scheme and a coarser four-class scheme, and evaluate every model under 5-fold stratified cross-validation to obtain statistically defensible mean ± standard deviation performance estimates rather than relying on a single train/test split. On the six-class scheme, the ensemble achieves the highest mean macro-F1 (0.391 ± 0.013), but a paired t-test against the standalone transformer does not reach significance (t=1.30, p=0.264). On the four-class scheme, the same ensemble achieves substantially higher performance (accuracy 0.680 ± 0.012, macro-F1 0.713 ± 0.009), with a statistically significant advantage over the transformer (t=3.58, p=0.023). We support these findings with a survey of 62 works spanning clinical text classification, transformer-based clinical NLP, class imbalance handling, and ensemble learning in healthcare. These results indicate that ensembling a transformer with classical text classifiers provides a measurable, class-count-dependent benefit, and that schemes with severely underrepresented classes should be flagged as unreliable rather than reported without qualification. A controlled ablation study isolating each imbalance countermeasure reveals that keyword-based semi-supervised augmentation is the dominant driver of performance in both schemes, but with opposite sign: it reduces six-class macro-F1 by 0.093 while increasing four-class macro-F1 by 0.105, indicating that augmentation's value is gated by the ratio of genuine to pseudo-labelled examples already available for the target class rather than being intrinsically helpful or harmful.