A large-scale multimodal dataset of real-world expert telemedicine conversations in Bengali, collected from nationally broadcast telemedicine programs featuring board-certified physicians, showing that DocTalkBN is a practically useful resource, particularly for clinically grounded reasoning tasks.
Abstract
Reliable medical conversational AI requires authentic expert--patient interaction data, yet such datasets remain scarce, especially for low-resource languages such as Bengali. We present DocTalkBN, a large-scale multimodal dataset of real-world expert telemedicine conversations in Bengali, collected from nationally broadcast telemedicine programs featuring board-certified physicians. DocTalkBN contains 557.63 hours of paired audio and text, 1,515 multi-turn patient calls, 10,274 host--doctor question--answer exchanges, totaling 1.7M tokens, spanning 26 medical specialties. Unlike prior resources derived from medical forums, written health content, or synthetic data, our dataset preserves the spontaneity, contextual richness, and spoken characteristics of authentic medical interactions in a low-resource setting. To support benchmark-driven research, we further construct three downstream tasks from the corpus, medical triage classification, advice safety evaluation, and medical named entity recognition, and benchmark a diverse set of large language models and encoder-based baselines. Our results show that DocTalkBN is a practically useful resource, particularly for clinically grounded reasoning tasks. We release this resource to facilitate future research on reliable medical NLP and safer, more culturally grounded healthcare systems for low-resource languages. Our source codes and dataset are publicly available at https://anonymous.4open.science/r/doctalk.
The increasing demand for accessible healthcare services, coupled with the shortage of medical professionals and geographical barriers, highlights the need for intelligent digital healthcare solutions. Traditional medical chatbots are largely limited to text-based interactions, lacking the ability to process multimodal inputs such as speech and medical images, thereby restricting their effectiveness in real-world scenarios. This paper presents DawAI, a multimodal AI-powered virtual medical assistant designed to simulate real-time doctor– patient interactions. The system integrates advanced technologies including speech-to-text conversion for interpreting spoken symptoms, image-based analysis for visual medical inputs, and large language models for generating contextaware medical responses. Additionally, a text-to-speech module enables the system to deliver responses in a natural, human-like voice, enhancing user accessibility and interaction. DawAI operates through a unified architecture that processes voice and image inputs, performs multimodal reasoning, and generates informative, empathetic responses within seconds. A structured dataset comprising symptom descriptions, severity levels, and precautionary measures supports the system’s reasoning capability, ensuring coherent and medically relevant outputs. Experimental evaluation demonstrates that the system provides consistent and context-sensitive responses while maintaining real-time performance. By addressing the limitations of existing healthcare chatbots, DawAI offers a scalable, accessible, and user-friendly solution for preliminary medical consultation, particularly benefiting users in remote and resource-constrained environments.
Dr. Abdul Khadeer, Mohammed Zubair Ahmed· International Journal of Eng...· 0 citations
A large-scale benchmark for multimodal online medical consultation built from de-identified patient-doctor interactions collected from a nationwide Chinese internet hospital, MedRealMM offers a realistic and reproducible benchmark for evaluating multimodal medical reasoning in real-world online consultation.
Runhan Shi, Quan Zhou, Yuqian Xu et al.· 0 citations
This work introduces IndicMedQA, a novel multimodal AI framework that integrates Indic large language models (LLMs) and visual encoders to analyze patient inquiries using both textual and visual cues, and creates a multilingual multimodal medical corpus spanning seven major Indian languages, translated using a semi-automated approach.
Akash Ghosh, Arkadeep Acharya, M. Muhsin et al.· ACM Transactions on Computin...· 1 citation
Clinical diagnosis is fundamentally interactive and incremental, yet the dominant paradigm for evaluating Large Language Models (LLMs) in medicine remains static QA benchmarks or template-based dialogues. These benchmarks say little about whether a model can serve as a diagnostic agent in a dynamic clinical encounter, with LLMs showing significant accuracy and reliability degradation in multi-turn settings. To address this issue, we present MTDiag, a large multi-turn diagnostic dialogue dataset constructed from three heterogeneous sources: DDXPlus, MIMIC-IV, and published case reports (AJCR), covering common ED presentations as well as long-tail rare and atypical conditions. All cases are normalized into a canonical schema anchored in the most comprehensive and widely-adopted medical knowledge bases (UMLS concept identifiers, with ICD-10 diagnosis codes). We release the schema, a UserLM-8B-based utterance-generation pipeline, and the physician-validated dataset that converts structured clinical evidence into natural-language utterances. Importantly, we introduce and motivate clinical knowledge-grounded metrics for evaluating LLMs as diagnostic agents, beyond diagnostic accuracy, for the task of multi-turn differential diagnosis.
Pia Chouayfati, Alexander M. Fichtl, Miriam Anschütz et al.· SIGDIAL Conferences· 0 citations
Large language models (LLMs) exhibit strong general conversational capabilities, yet their deployment in domain-specific service environments such as hospitality remains limited by the absence of emotionally grounded datasets and validated end-to-end system architectures. This paper presents HOSPIT-LLM, an EU-funded euROBIN Technology Exchange Program pilot, as a complete, integrated pilot pipeline for human-centered conversational AI in hotel reception scenarios. We deploy a multimodal hotel-terminal assistant in a real hotel reception, capturing synchronized dual-camera video and audio to collect authentic guest–staff interactions. Speech is transcribed using Whisper, and emotion is extracted from the corresponding video segments via DeepFace, producing 582 real Greek guest–receptionist exchange examples. The resulting data are classified into eight Standard Operating Procedure (SOP) categories. To address data scarcity, we augment the corpus with 1269 synthetic dialogues generated by eight diverse LLMs through the OpenRouter API, yielding a total of 1851 dialogue records with explicit emotion-token annotation. We fine-tune Qwen3.5-35B-A3B using Low-Rank Adaptation (LoRA) through a two-stage process: supervised fine-tuning (SFT) on an 888-example conversation pool and Simple Preference Optimization (SimPO) on a 1899-pair preference pool, each split 80/10/10 into training, validation, and test. The resulting model is integrated into an interactive hotel-terminal system combining YOLO-based person detection, face-recognition-driven guest personalization, Kokoro neural text-to-speech (TTS), and a multi-service orchestration layer connected to the hotel Property Management System (PMS). Evaluation combines standard text metrics, emotion-aware scoring, and a large-model judge. The results indicate targeted improvements in the rule-based contextual emotion-policy match and staff-emotion policy compliance compared to the base model, while general response-quality gains remain more modest. In particular, the rule-based contextual policy-match score improves from 0.614 to 0.901, while forbidden staff-emotion outputs decrease from 0.142 to 0.018. The deployed pilot demonstrates the practical integration of a personalized, emotion-aware LLM assistant in an interactive hotel-terminal setting; end-to-end latency and fully hotel-side edge deployment were not evaluated and are left for future work. HOSPIT-LLM provides a reproducible framework for multimodal dataset creation, preference-based fine-tuning, and deployment of human-centered AI systems. A mobile robotic embodiment is planned as future work.
Homer Papadopoulos, Antonis Korakis, G. Balaskas· Future Internet· 0 citations
This paper describes TalTech's submissions to the Beyond Transcription Challenge (BeTraC), which requires generating SOAP notes directly from long doctor-patient conversation recordings, without intermediate transcription. After screening open-weight speech LLMs for long-audio robustness, we adapted Voxtral Mini (lightweight track) and Voxtral Small (heavyweight track) with LoRA supervised fine-tuning followed by DAPO reinforcement learning that uses the challenge metric, Open Medical Concept F1, as its reward. Our systems ranked first in both tracks, and an independent LLM-as-a-judge evaluation showed the lowest hallucination rate among all submissions, indicating that reinforcement learning against a concept-matching metric need not compromise factual reliability. We also find that fine-tuning on text transcripts transfers well to speech input and appears to improve robustness on out-of-domain real recordings.