Building a Multimodal and Multilingual Chatbot for Enhanced Retrieval Using AI and ML Algorithms

Abstract

A strong student support system plays a crucial role in providing quality education for students. Facilitating the requisite support to students in the academic environment by human facilitators is still an uphill task, as facilitators are inherently subject to changes in knowledge levels and emotional states, which can affect the quality of service expected. In this research article, the authors have designed a retrieval-based multimodal and multilingual chatbot interface using artificial intelligence and machine learning algorithms. The authors have considered both continuous speech and text inputs to interact with the chatbot in the domain of educational institutions. The chatbot can be interacted with by the students in three languages, namely Hindi, Odia, and English. The authors have rigorously validated the proposed artificial intelligence and machine learning-driven chatbot models using various statistical metrics, including F1 score, Precision, Recall, Sensitivity, Specificity, Confusion matrix, BLEU score, Wilcoxon signed-rank test, and 95% Confidence Intervals (CI), etc. Our proposed DNN-based model has yielded promising results compared to other algorithms, including SVM, Naïve Bayes, KNN, Random Forest, and Gradient Boost. BLEU scores for a multimodal, multilingual chatbot in Hindi, Odia, and English are 0.89, 0.92, and 0.91, respectively. Our proposed architecture for the MM-chatbot can accept queries in both voice and text modes. A pronunciation dictionary has been integrated to address pronunciation variations, accents, and speaking styles among stakeholders. MM-chatbot can generate voice responses in accordance with the user’s query language. We have significantly improved the voice output of our proposed MM-chatbot by manipulating pitch in the synthesized speech. However, users are required to explicitly select the query language at the beginning of their conversation with the MM-chatbot. This research aims to design a multimodal and multilingual chatbot that can be useful for retrieving information from student support systems in educational institutions. Further the authors have proposed a chatbot that can operate in both voice and text modes.