MultiModal Deep Learning Framework for Missing Children Detection using Vision-Language Mode
TL;DR
A multimodal deep learning system that combines Vision-Language Transformers like CLIP, Face Re-identification networks, and Graph Neural Networks in order to help in locating and tracking missing children demonstrates a high possibility of combining it with police technology.
Abstract
As report of missing children continue to rise around the world there is an urgent requirement of some smart and scalable systems which can help in timely and accurate identification. In this paper, we propose a multimodal deep learning system that combines Vision-Language Transformers like CLIP, Face Re-identification networks, and Graph Neural Networks (GNNs) in order to help in locating and tracking missing children. The proposed system is unlike conventional facial recognition systems, which can only act on visual stimulus whereas the proposed system handles a multitude of different stimulus e.g. facial image, descriptive text, metadata location, time greater and context of the stimulus e.g. description of clothing. The vision language module enables visual inputs to be correlated to the textual description so that the model is able to interpret and correlate the narrative indicators within the public reports. A face re-identification module with high tolerances to visual variability because of age, lighting, occlusion, etc. and a GNN to learn spatial and temporal links between sightings to make movement pattern predictions. Initial tests on publicly available data show a better matching across the unimodal baselines and a substantial improvement in recall and contextual relevance in the metrics. Uniting the visual, linguistic, and structural data, the model provides more accurate findings in real-life situations and demonstrates a high possibility of combining it with police technology. There should be more work on increasing child-targeted multimodal data sets and evaluating in the real world testing.