LexTester is presented, an automated model-based testing technique for Amazon Lex chatbots that explores the conversational space of the chatbot under test to generate a Dialog Graph of all possible interactions, from which an executable test suite is generated according to different coverage strategies.
Abstract
Task-based chatbots are nowadays widely adopted software systems, usually integrated into real-world applications and communication channels, designed to assist users in completing tasks through conversational interfaces. Like any other software, even chatbots are prone to bugs. Despite their increasing pervasiveness in everyday activities, existing techniques for assessing their quality still exhibit several limitations, such as the simplicity of generated test scenarios and oracle weaknesses. In this paper, we present LexTester, an automated model-based testing technique for Amazon Lex chatbots. The technique explores the conversational space of the chatbot under test to generate a Dialog Graph of all possible interactions, from which an executable test suite is generated according to different coverage strategies. LexTester was evaluated against the state-of-the-practice chatbot testing tool Botium on five Amazon Lex chatbots, consistently outperforming it in all subjects, generating more tests with nearly double complexity, achieving overall 83-95% coverage of conversational elements, and improving fault detection effectiveness by up to four times at comparable time costs.
A reverse engineering technique is presented for extracting a behavioural model of a deployed chatbot through the automated exploration of its functionality via Large Language Models through the automated exploration of its functionality via Large Language Models.
Iván Sotillo del Horno, Alejandro del Pozzo, Esther Guerra et al.· SN Computer Science· 0 citations
The study successfully validated that the intelligent chatbot efficiently bridges communication barriers, automates repetitive administrative inquiries, and improves service accessibility, confirming that the application is highly usable, practical, and effective as an inclusive, assistive communication tool.
Gil G. Dialogo, Hershey Alburo-Abugadie, Enrico C. Lucero· International journal of com...· 0 citations
These results show that conversational refusal benchmarks can substantially overstate the safety of deployed coding agents and motivate defenses that reason about safety across multi-turn IDE workflows and their generated artifacts, not only individual chat turns.
: Artificial Intelligence (AI)-based code suggestion tools are becoming increasingly relevant in the software development community. While several studies analyse the quality and security of these tools, relatively few systematic studies have investigated how different input text languages can affect the effectiveness of code generation models. Similar research has been conducted with other tools, addressing multiple programming languages and different languages and challenge platforms. However, it is important to expand this knowledge base by considering additional scenarios. In this study, we analyse the performance of ChatGPT 5.2 in generating Python solutions for 70 programming problems extracted from the Beecrowd repository, considering three languages: Portuguese, English and Spanish. These 70 challenges were selected to encompass a range of difficulty levels for mathematical problems. Each question was submitted to the model five times in each language and the accuracy rate was evaluated using the platform’s validation tests. It was observed that Portuguese prompts generally showed a slight advantage in terms of the accuracy rate compared to prompts in other languages. However, the analysis of variance (ANOVA) indicates that this variation was not significant at the 5% level. The aim of this article is to extend current knowledge about the impact of languages on code generation by demonstrating that the language used in the problem description may or may not affect the ability of AI models to provide correct solutions. Finally, this article discusses the influence of potential linguistic biases, as well as opportunities for future research to address other languages, different problem modalities and various AI tools.
Jonathan Cardoso de Sales, S. Oliveira· Proceedings of the 21st Inte...· 0 citations
Findings indicate that clear prompt instructions can improve the reliability of RAG-based academic chatbot responses for academic information services and show the strongest improvement in faithfulness and context recall.
Adi Surya Artayasa, Aniek Suryanti Kusuma, Putu Sugiartawan· IJOEM: Indonesian Journal of...· 0 citations