Adaptive multimodal large language model with hierarchical cross modal grounding for real time industrial visual anomaly detection on edge devices

Abstract

The problem of industrial surface anomaly detection is still a disheartening one because there are light defect data, computationally intensive, and interpretable only to a limited extent when applied in real-world systems. In this paper, we present the HCG-MLLMAD - Hierarchical Cross-Modal Grounding framework of Multimodal Large Language Model-based Anomaly Detection - a variant of the original Light-MLLMAD architecture with multi-scale visual feature pyramids, prompt routing, and a hierarchical cross-attention fusion mechanism. The suggested architecture uses a two-branch visual encoder with ViT-Tiny (Vision Transformer-Tiny) and MobileNet-V4 representations, which allows extracting global structural semantics and fine-grained local texture features complementary to each other. A new Adaptive Contrastive Prototype Memory (ACPM) module updates normal-class prototypes dynamically at inference time to maintain high discrimination accuracy in the face of distributional shifts typical of changing manufacturing lines. A Hierarchical Prompt Router (HPR) module with three semantic granularities coarse, moderate and fine is chosen with the help of a lightweight Hierarchical Prompt Router (HPR) to create finer signals of cross-modal alignment. Extensive results on MVTec-AD, VisA, BTAD, and PCB-Defect benchmarks indicate that HCG-MLLMAD is more accurate and high-performing with 99.1% and 0.993 AUC respectively with one-shot supervision, compared to Light-MLLMAD [1] in terms of accuracy and F1-score. This model has a small 18.3 M parameter footprint and 78 ms inference latency on Jetson Orin Nano. Semantically grounded anomaly localization is verified by SHapley Additive exPlanations (SHAP)-based interpretability analysis and Gradient-weighted Class Activation Mapping (Grad-CAM++] visualizations, and is consistent with operator-defined defect vocabularies.