Risks that may form on the user side during long and repeated interaction are examined to establish safety across a series of conversations, with the central claim that an acceptable response on its own does not establish safety across a series of conversations.
Abstract
Current assessments of conversational AI focus mainly on model outputs, including hallucinations and factual errors. These measures matter, but this paper examines risks that may form on the user side during long and repeated interaction. The study follows one user across nearly four thousand conversations with the same system over twenty months. The user gradually interpreted the system as having memory, care, judgment, and authority, and reorganized part of their thinking around it. A single response may show no clear problem, while long and frequent interaction can still create another layer of risk. This paper calls that layer User-Side Contextual Phenomena (USCP) and examines records from August 2024 to April 2026. The study uses an exploratory single-case longitudinal qualitative design with autoethnographic positioning. A hybrid deductive-reflexive thematic approach organizes the material into three main modes: contextual projection, contextual attachment, and contextual authority transfer. The paper does not estimate prevalence, make diagnoses, or validate an instrument. It offers a non-clinical vocabulary and four evidence roles: inclusion, gray-zone, negative, and protective gray-zone. Its central claim is that an acceptable response on its own does not establish safety across a series of conversations. User-side risk can still form during long-term interaction.
A mixed-methods investigation to characterize and mitigate memory misalignment from user perspectives, which highlights the tension between supervisory agency and interaction overhead, and advocates for friction-aware memories that balance user oversight with conversation smoothness.
Jing-Ruo Chen, Shu-Ning Zhang, Er-Yue Xu et al.· 0 citations
It is argued that HCI researchers and designers should account for users'histories with their chatbots and broader care ecologies when evaluating AI systems over time and introducing updates that may disrupt established sources of support.
Meryl Ye, Briana Vecchione, Livia Garofalo et al.· 0 citations
These findings suggest that, in this setting, LLMs function less as human co-authors and more as adaptive narrative amplifiers that reshape how agency is distributed in collaborative writing.
Halfdan Nordahl Fundal, Yuri Bizzoni, Charlotte Gjørup Bilde et al.· 0 citations
This study proposes a transparent rule-based framework for estimating persistence, delegation-related lexical patterns, and refinement-related lexical markers without inferring psychological dependence in conversational generative models.
Aracely Mera-Navarrete, Solange Revelo, Jefferson Beltrán-Morales et al.· Frontiers in Artificial Inte...· 0 citations
It is hypothesize that existing benchmarks and user satisfaction are tracking different capabilities: benchmarks measure elicited retrieval (recall when asked), while conversation requires natural integration (detecting relevance and naturally weaving prior context into a response).
Ryuichi Sumida, K. Inoue, Tatsuya Kawahara· 0 citations
Retrieval-augmented generation (RAG) is increasingly used across knowledge-intensive enterprise contexts, yet little is known about real-world interaction with such systems. We present a qualitative field study of a RAG-based assistant in B2B sales, analyzing 190 chat sessions and a complementary user survey. Drawing o...
H. Schneider, Androniki Mertsiotaki, Sven Winkelmann· Message Understanding Confer...· 0 citations
Students in MIT’s Concourse program delve deeply into the human condition, debate challenging questions, and learn to develop judgment about issues that can’t be quantified.
Jennifer Neville did not want to go into computer science—but that’s exactly where she landed. Neville discusses the starts and stops that led to her professional sweet spot and her work identifying “surprising failures” making it hard for AI to handle complexity. The post What AI gets wrong and what failure teaches us appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduSep 30, 2026
Able to defeat top-ranked human players and more efficient than other models, the new system could help decision-makers in military maneuvers or business negotiations.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.