MFAB: A Multimodal Fine-Grained Attention Decoder With Boundary-Aware Multi-Task Learning for Backchannel Prediction
Backchannel prediction is a key component of human-like conversational AI, enabling systems to generate timely and contextually appropriate listener responses. Existing approaches primarily rely on encoder-based architectures that fuse audio and text features through simple concatenation, limiting their ability to capture fine-grained multimodal interactions. To address this limitation, we propose Multimodal Fine-grained Attention decoder for Backchannels (MFAB), a decoder-based model that dynamically integrates audio frame and text token representations using self-attention mechanisms. MFAB effectively captures multimodal dependencies and leverages extended conversational history to improve both the timing and type of backchannel responses. In addition, we introduce a boundary-aware multi-task learning framework in which the model jointly predicts backchannels and discourse boundaries, including speech pauses and text chunking points. We obtain boundary labels using two automated methods: pause labels are generated using pause-based heuristics, and chunk boundaries are derived from ChatGPT-based text segmentation. Experiments on the Korean Counseling and SwitchBoard datasets show that incorporating boundary information significantly enhances model performance, achieving state-of-the-art results in backchannel prediction. Furthermore, we compare our approach with alternative multi-task learning methods, including sentiment classification and dialogue act classification, showing that boundary-aware learning is a viable and effective alternative. The consistent improvements across structured and spontaneous conversations highlight the importance of discourse structure in backchannel modeling.