A multi‑modal recognition method for building structural materials integrating structured text description and adaptive decision weighting

Abstract

Accurate identification of building structural materials is a critical step in structural safety assessment and damage detection. Traditional image‑based classification methods are limited by factors such as illumination, occlusion, and surface treatment, making it difficult to effectively distinguish different materials using visual features alone. To address this problem, an intelligent recognition method for building structural materials based on multi‑modal data fusion is proposed. First, a prompting method based on a multi‑modal large language model is designed to generate structured text descriptions tailored to material classification tasks, covering discriminative information such as surface characteristics, geometric details, signs of degradation, and material inference. Second, a multi‑modal neural network named MRNet is proposed. It introduces a lightweight text feature transformer that reshapes the global vector from the Contrastive Language‑Image Pre‑training (CLIP) text encoder into a sequence and adds positional encoding. A lightweight text-feature encoder further refines the CLIP global text embedding through nonlinear transformation and channel mixing, alleviating the information compression caused by the original CLIP global pooling. The model adopts a decision‑level learnable weighted fusion strategy, where the decision results from the image and text modalities are fused through a learnable linear layer, enabling efficient cross‑modal integration without complex feature alignment. Furthermore, Bayesian optimization is introduced to automatically tune key hyperparameters such as learning rate, batch size, and weight decay coefficient. Experimental results on the PEER dataset show that MRNet achieves 96.00% in accuracy, precision, recall, and F1 score, significantly outperforming mainstream image classification models such as AlexNet, MobileNet‑v3, ResNet, and VGG. The proposed method provides a high‑accuracy, high‑efficiency multi‑modal solution for building structural material identification.