Multi-Faceted Learning to Rank With Content-Behavior Fusion for Satisfaction-Diverse Web Search

Abstract

Web search inherently involves multiple facets of user satisfaction, relevance, quality, authority, recency and user behavior, yet traditional learning-to-rank (LTR) models often overlook these dimensions, producing suboptimal results. We tackle this “satisfaction-diverse” challenge by unifying content-based signals (e.g., relevance, quality, authority, and recency) and behavioral signals (e.g., click-through and browsing time) within a pre-trained language model (PLM). Specifically, we address three key hurdles: (1) Signal Diversity: Standard methods often combine diverse signals into a single loss function with multiple terms, thereby neglecting the unique distinctions each signal brings to representation learning. (2) Unpaired Fusion: Only an extremely small fraction of query-document pairs include user behavioral signals, together with content-based ones, resulting in sparse and imbalanced supervision. (3) Mixture of Experiences: While a Mixture-of-Experts (MoE) architecture can handle diverse, sparse, and imbalanced supervision, naive aggregation (e.g., average pooling or concatenation) of multiple expert heads might undermine performance, as these signals reflect orthogonal user experiences. To overcome these challenges, we propose CBRank, a Content-and-Behavior fusion learning to Rank framework. The proposed solution employs a pyramid PLM for query–document representations on top of two MoE-based modules–one integrating core ranking features and another incorporating user behavioral feedback–each with multiple expert heads to handle diverse signals under sparse and imbalanced supervision. A dynamic fusion module is then fine-tuned using high-quality satisfaction-annotated query-document pairs to adaptively balance the learned representations and generate satisfaction-oriented ranking scores. Extensive experiments on a large-scale offline dataset demonstrate that our approach consistently outperforms strong baselines. We further conduct rigorous online evaluations using real-world web traffic from a large-scale commercial search engine.