Comparative Performance Analysis of Vision Transformer (ViT) and Convolutional Neural Network (CNN) Architectures for Semarang Batik Motif Classification

Abstract

Vision Transformers (ViT) capture global image context through self-attention but are data-hungry, typically underperforming Convolutional Neural Networks (CNNs) on the small datasets common in fine-grained tasks such as batik motif recognition. This study investigates whether a ViT, trained via knowledge distillation using the Data-efficient Image Transformer (DeiT), can overcome this limitation and compete with CNNs on a small Semarang Batik dataset. A distilled DeiT-Tiny student learns from a ResNet-50 CNN teacher and is benchmarked against two CNN references: ResNet-50 itself (a substantially larger model) and EfficientNet-B0 (a parameter-matched counterpart). In establishing this comparison, we first uncover a critical dataset integrity issue: the publicly available Semarang Batik Dataset (3,020 images) originates from only 18 unique source photographs, each augmented approximately 167 times prior to publication. This near-duplication causes severe data leakage under conventional random splitting, inflating the test accuracy of all models to a misleading 100% and rendering such evaluation meaningless. We therefore introduce a source-aware splitting strategy that enforces group-level separation between training, test partitions, and evaluate all models across three random seeds for statistical reliability. Under this corrected protocol, the distilled DeiT-Tiny attains the highest mean accuracy (95.18 ± 0.30%) and the lowest variance among the three models, matching both the larger ResNet-50 (94.87%) and the parameter-matched EfficientNet-B0 (94.68%) while using only 5.5M parameters. These results confirm knowledge distillation enables a compact Vision Transformer to compete CNNs on a limited fine-grained dataset, and underscore that verifying sample independence is a prerequisite for trustworthy evaluation on pre-augmented public datasets.