Key Technologies and Applications of Multimodal Cognitive Agent Construction in the Vertical Domain of Inspection and Testing

Abstract

 Inspection and testing underpin product safety and regulatory compliance across industries such as manufacturing, healthcare and food and beverage. However, conventional engineering test processes that mainly depend on manual, offline actions, failed to adapt to next generation, automated, digital and environmental-sensitive testing, resulting in very low productivity and high cost for most firms. The automation and digitalization of inspection and testing processes have become a research hotspot in both academia and industry. The natural language processing and computer interpretation of test results have been a focus of AI research as well. Nonetheless, due to the lack of real data, the verification and simulation of real inspection and testing environments are still difficult for researchers. As a solution, we manage to develop a multimodal cognitive test system that fuses the textual regulatory documents and instrument data, through a series of modules and processing flows. Our test system is based on a so-called multimodal cognitive agent, which includes large language model, vision module, knowledge graph and retrieval-augmented generation. We introduce the design, development and application of our test system, which used for rubber heater in a glass factory, and layout the future challenges for the exploration of multimodal agent technology in the test engineering. The project is supported by the Guangxi Key Research and Development Program. This paper was partially presented at the IEEE 2023 International Conference on Intelligent Commerce (ICIC).