The reason for low performance is analyzed and a covariance corrected Mahalanobis distance is proposed for detecting out-of-domain intents, outperforming competing approaches.
Abstract
Conversational agents like chatbots and voice assistants are trained to understand and respond to user intents. On encountering an utterance with an intent different from the ones they have been trained on, these agents are expected to classify the intent as `unknown'or `out of domain'. This problem is known as out of domain (OOD) intent detection. Podolskiy et al. (2021), showed that Mahalanobis distance can be used effectively for identifying OOD intents, outperforming competing approaches. However, their method fails to outperform the baselines in the practically important few-shot setting. In this paper we analyze the reason for low performance and propose a covariance corrected Mahalanobis distance for detecting out-of-domain intents.
Out-of-distribution (OOD) intent detection remains a critical challenge for deploying conversational agents in realworld scenarios. This paper presents a comparative analysis of three distinct paradigms for OOD detection: embedding-based semantic similarity, zero-shot Large Language Model (LLM) reasoning, and Retrieval...
Ebru Koçdor, Furkan Göz· Automation, Control, and Inf...· 0 citations
This work evaluates four omni LLMs in a zero-shot setting and shows that fine-tuning consistently outperforms zero-shot inference, and explores synthetic data augmentation by using an LLM to generate culturally grounded Tunisian Derja utterances, followed by voice cloning to generate synthetic speech.
Tajwaar Shafiq, Hunzalah Hassan Bhatti, Firoj Alam et al.· 1 citation· ⚡1
Fundamental trade-offs are revealed: domain-invariant features enable zero-shot deployment when no real data exists, while SSL methods achieve higher performance but require real samples and compute.
Jeongmin Lee, Seung Yun, Minkyu Lee et al.· 0 citations
It is found that fine-tuning is a prerequisite for these inference-time techniques: without domain adaptation, prompting and shallow fusion cause severe hallucination on air traffic control speech, suggesting that domain difficulty determines whether prompting can work without fine-tuning.
Currently, chatbots have been widely used in various types of applications, most of them still generate responses that are general in nature and have not been able to reflect the specific communication style of different users. Therefore, this study aims to create a personal chatbot that is able to learn communication...
Adrian Marcello Budiman, Christian Verrell, Hans Ewaldo Kristiawan et al.· International Conferences on...· 0 citations
This work curates a human-annotated topic-utterance judgments dataset and examines the superior performance of lightweight LLM matchers over embedding and regex models when equipped with natural language descriptions.
Saman Rahbar, Xi-Liang Zhu, Irvin Cardoza et al.· 0 citations
