TokenMinds: Pretrained User Tokens and Embeddings for User Understanding in Large Recommender Systems

Abstract

User modeling in industrial recommender systems typically produces dense embeddings, which suffer from representational constraints due to fixed-dimensional vectors. As an emerging alternative for discrete user representation, LLM-generated natural-language user profiles often capture language semantics rather than sequential patterns in user behaviors. Meanwhile, Semantic ID (SID) is effective for item representation, yet existing discrete user representations remain largely confined to quantizing already-pooled user vectors, clustering historical watches, or predicting coarse topic clusters solely for candidate retrieval. We propose TokenMinds, an industrial system that extends the PLUM framework from SID-based item retrieval to generative sequential user modeling. Built on an encoder-decoder architecture adapted from pretrained LLMs, TokenMinds introduces a dual-output design: the encoder outputs a dense user embedding capturing a holistic user state, while the decoder generates discrete SID-based user tokens representing multi-faceted future interests. Grounded in item semantics, these tokens preserve distinct interests that a single pooled vector would average away. Additionally, a shared SID vocabulary across long- and short-form videos unifies cross-scenario behaviors into a single model, cutting upstream training and serving compute by 50% and 31%. Furthermore, leveraging the text vocabulary inherited from the pretrained LLM, TokenMinds interleaves textual search queries directly with watch histories to capture explicit user intent. We validate TokenMinds through offline evaluations and live A/B experiments across YouTube's primary ranking surfaces, serving billions of users asynchronously: SID-based user tokens are effective on their own, yield additive gains alongside dense embeddings, and together outperform the previous-generation, non-LLM user model in production.