Hidden Behavioural States of Chatbot Users: A Hidden Markov Model of Message Length and Pacing in 60,000 Real Conversations

Abstract

Purpose. Interaction logs with AI assistants record how people behave when they use AI, but it is unclear whether this behaviour has a latent structure that an algorithm can recognise and that carries information beyond the most recent action. We fit a hidden Markov model (HMM) to the sequence of user turns in real chatbot conversations, ask what behavioural states it finds, and test whether they predict whether a user continues the conversation. Design and methods. We used the first shard of the WildChat-1M corpus of real user conversations with ChatGPT (59,857 conversations by 13,908 hashed IP addresses, 172,246 user turns, 9 April to 4 May 2023). Each user turn was summarised as one of 30 symbols: message length (5 bins) crossed with the time since the previous assistant reply (5 bins, plus a symbol for the first turn). HMMs with 2 to 8 states were fitted by expectation–maximisation on 80% of the users and evaluated on the held-out 20% of users by the log-likelihood per turn, against independent-symbol and first-order Markov baselines. With a four-state model chosen for interpretability, we studied state profiles, whether the filtered state predicts that the conversation continues (AUC, bootstrap over conversations), and differences in state occupancy by language and model. Findings. Held-out log-likelihood per turn was −2.964 for independent symbols, −2.083 for a first-order Markov chain and −2.168 for the four-state HMM; it improved up to −2.082 with eight states, which equals but does not exceed the Markov chain. The four-state model separated a first-turn state from three highly persistent behavioural states (self-transition probability 0.97–0.96): brief messages, medium messages at a moderate pace, and long messages at a slow pace. Whether a conversation continued was predicted with AUC 0.642 from the turn number alone and 0.666 when the last symbol was added, but the HMM state added nothing (0.641 with the state in place of the symbol; 0.667 with both). Among users with at least five multi-turn conversations, the user explained 46.9–58.8% of the variance of state occupancy, against about 7.2% expected by chance. Occupancy differed strongly by language and model, but message length in characters and the time between assistant replies (which includes the model's response time) are not comparable across scripts and models, so we do not interpret those differences as differences in user behaviour. Originality. The study documents what a latent-state description of real chatbot use does and does not add: it finds persistent, user-specific pacing regimes, but shows that a direct Markov chain over observed symbols fits as well, that the states add nothing to predicting continuation, and which group comparisons are confounded by measurement.