Vision Transformer-Based Recognition of Riau Malay Architectural Features with Cross-Regional Comparison for Digital Heritage Documentation

Abstract

Traditional Riau Malay architecture requires systematic digital documentation for heritage preservation. This study evaluates a Vision Transformer (ViT-B/16) model initialised with ImageNet-1K pretrained weights for recognising Riau Malay architectural features, using Pontianak Malay architecture for cross-regional comparison. The dataset was constructed from 24 architectural videos covering roof shapes, building structures, ornaments, windows, staircases, and full-building views. Using automated spatiotemporal segmentation at five frames per second, 13,230 frames were extracted, resized to 224×224 pixels, normalised, augmented, and divided into 16×16-pixel patches. Evaluation on a balanced, held-out test set of 32 clips yielded an overall accuracy of 84.38%, macro precision of 84.51%, macro recall of 84.38%, and macro F1-score of 84.36%. Distinctive elements, such as roofs, windows, staircases, and full buildings, achieved higher recognition performance when clearly visible. Conversely, partially visible structures and detailed ornaments exhibited variable performance due to lighting, viewpoint, and visual complexity. Given the single hold-out split and the limited number of source videos, these findings are preliminary; high feature-specific accuracies should not imply perfect recognition or generalizability. Nonetheless, the results demonstrate ViT-B/16’s strong potential to support the digital recognition of Malay architectural heritage. Future work should incorporate grouped five-fold cross-validation, independent building-level testing, CNN baseline comparisons, ROC–AUC analysis, and attention map visualisations.