The Boltzmann-rational model for pairwise comparisons is adopted, which extends the Bradley-Terry-Luce model by incorporating worker competencies, and an EM-based algorithm for learning is derived by introducing Polya-Gamma latent variables to transform the logistic likelihood into a conditionally Gaussian form, enabling tractable optimization and leading to a simplified $Q$ function in the E-step of the algorithm.
Abstract
The problem of learning from pairwise comparisons has been widely studied across many domains such as recommendation systems, social choice, and more recently, fine-tuning large language models. In this problem, the goal is to learn item rewards based on pairwise comparisons between them. In many scenarios, these comparisons are elicited from crowdworkers using platforms such as Amazon Mechanical Turk, Scale AI, etc. However, crowdworkers are often unreliable due to limited domain knowledge or revenue-maximizing (spamming) behavior. In this work, our goal is to understand whether worker reliability (competency) can be learned jointly with item rewards. To this end, we adopt the Boltzmann-rational model for pairwise comparisons, which extends the Bradley-Terry-Luce model by incorporating worker competencies. We derive an EM-based algorithm for learning under this model by introducing Polya-Gamma latent variables to transform the logistic likelihood into a conditionally Gaussian form, enabling tractable optimization and leading to a simplified $Q$ function in the E-step of the algorithm. This technique allows us to reduce our formulation to a matrix sensing problem, using which we establish theoretical convergence guarantees for our algorithm. We conduct extensive experiments on real-world and synthetic datasets. These experiments demonstrate the advantages of using our algorithm over several baselines and confirm its strong robustness to both spammers and adversarial workers, highlighting its practical effectiveness in realistic crowdsourcing and reward learning settings. The code and data is publicly available at https://github.com/KaustubhShejole/BoRa_EM.
In recommendation systems, users interact with only a small fraction of a vast item catalog, producing feedback that is both sparse and noisy. This challenges post-training generative recommenders: reward models trained from logged interactions often fail to generalize, while directly optimizing imperfect rewards can lead to reward over-optimization. We propose Exponential reward-weighted fine-tuning (Exp-RSFT), where each logged interaction is weighted by $\exp(r/\lambda)$, avoids this failure by optimizing directly on the logged rewards, with the temperature $\lambda$ regularizing against their noise. We theoretically show that Exp-RSFT's suboptimality decomposes into two costs: a coverage cost arising from limitations of the logging policy and a noise cost from imperfect feedback. The temperature $\lambda$ balances these competing effects, yielding an optimal tradeoff between exploiting high-reward behavior and robustness to noise. Across three public benchmarks and a large-scale industrial dataset, we verify this theoretical prediction: performance follows an inverted-U trend as a function of $\lambda$, while PPO and DPO often over-optimize unreliable reward models and degrade recommendation quality. Exp-RSFT consistently improves ranking performance without requiring online exploration or preference data.
Cross-domain recommendation is a well-known technique for improving recommendations in a target domain, especially under sparse data or cold-start conditions. A common strategy is to train user embeddings separately in the source and target domains and learn a transfer function between them. In contrast, we propose SetCDR, which constructs more effective user representations in the target domain by directly incorporating each user's source and target history. We additionally introduce a lightweight domain indicator that preserves data–domain relational information. These histories, composed of item–rating pairs, are represented as variable-length sets and processed using a permutation-invariant neural architecture. This differs from conventional neural networks, which do not naturally handle unordered inputs, and is well suited to recommender systems where user history sizes vary greatly among users. The use of a permutation-invariant architecture ensures consistent embeddings regardless of input order, improving robustness to real-world variability and training efficiency. We demonstrate SetCDR in two forms: a simple sum-pooling method and an extended multihead attention–based method that captures more complex dependencies within user histories. Moreover, because SetCDR operates directly on sets of user history records, it provides a natural way to examine how histories influence user representations. Finally, SetCDR adapts immediately to new interactions without additional retraining, enabling on-the-fly performance improvement. Experimental results across multiple cross-domain benchmarks confirm that SetCDR consistently outperforms strong baselines in recommendation quality.
K. Kim, James Geraci· Annual International ACM SIG...· 0 citations
Multi-armed bandit algorithms, especially Thompson sampling, are widely used in online recommendation. Despite their ability to adapt from online feedback, these methods often suffer from cold-start limitations when newly introduced arms have little or no interaction history. In our setting, the candidate arms are user-generated textual comments, whose semantic content can reveal a title's appeal before sufficient interaction feedback is available. We therefore use large language models (LLMs) to extract semantic signals from comment text and convert them into informative Bayesian priors that warm-start Thompson sampling under sparse early-stage feedback. To account for aggregate segment-level differences in response patterns, we maintain and update posteriors separately for each gender-age segment. In a real-world online A/B/C test, we compare a uniform prior with two LLM-based designs: a Gender Prior for demographic-affinity cues and a Content Prior for title-specific identity cues. The results show that LLM-based priors are most beneficial in sparse-feedback regimes -- with the largest gains emerging once a small amount of interaction evidence has accumulated -- and that prior design leads to distinct funnel-level effects. We further analyze prior-reward alignment and demographic heterogeneity, finding that click-oriented alignment is strongest for the Gender Prior and that treatment effects vary substantially across demographic segments. These findings suggest that LLM-derived priors can serve as a practical warm-start mechanism for text-rich bandit recommendation, while also revealing deployment trade-offs.
E. Lee, Oseong Choi, Byungsoo Kang et al.· 0 citations
We consider the problem of learning a mixture of $k$ Plackett-Luce models given multi-way ranking responses from annotators that may represent heterogeneous underlying preferences. This problem has many applications in AI alignment and preference optimization. Prior work has studied mixtures of Bradley-Terry models from pairwise comparisons. However, uncovering mixture models is theoretically unidentifiable when $k$ exceeds $m/2$, where $m$ is the length of a ranking. We propose an efficient implementation to address this limitation, which involves first augmenting the rankings to a larger size by generating new responses from a base language model, followed by a gradient-based estimation to reduce inference cost in the input embedding space. Based on this procedure, we then design an expectation-maximization algorithm with these two steps to fit a mixture of Plackett-Luce models, called MoPLEx. Extensive experiments are conducted to verify this approach. First, we show that the gradient-based approximation estimates true probabilities with less than 5% error on models with up to 34 billion parameters. Second, we show that MoPLEx improves clustering and ranking accuracy by an average of 43.7% and 15.2% over baselines using single ranking and mixtures of Bradley-Terry models, on preference optimization datasets. These results demonstrate the effectiveness of MoPLEx for tackling multi-way rankings from heterogeneous preferences through measuring alignment between gradients.
Dongyue Li, Ziniu Zhang, Lu Wang et al.· 0 citations
Implicit feedback, such as clicks and browsing behaviors, is ubiquitous in recommender systems as a proxy for user preferences. However, these signals are inherently noisy; interactions such as misclicks, unintended views, or unsatisfactory purchases often introduce false positive patterns that mislead model learning. Existing denoising strategies mainly rely on heuristic ''small-loss'' principles to suppress the influence of high-loss samples. However, this approach creates a fundamental trade-off: by indiscriminately penalizing large-value losses, these methods inadvertently weaken the model's ability to learn ''hard'' true positive interactions, thereby compromising robustness and personalization. By conceptualizing this ambiguity as a ''candidate label set'' that encompasses both true and noisy feedback, we are the first to reformulate the recommendation denoising problem into a Partial Label Learning (PLL) task. This novel perspective allows us to address the fundamental challenge of unreliable pseudo-labels by transforming traditional heuristic-based filtering into a principled label disambiguation process. Specifically, we propose PLLD, a Partial Label Learning-inspired Denoising method. Unlike existing methods that rely on indirect signal filtering, PLLD innovatively leverages PLL paradigms to directly resolve ambiguous implicit feedback, effectively recovering clean signals from noisy candidate sets. Experiments on multiple real-world benchmark datasets demonstrate that PLLD consistently improves ranking performance and robustness under substantial noise.
Huilin Chen, Jie Lu, Kezhi Lu et al.· Annual International ACM SIG...· 0 citations
Post-click conversion rate (CVR) prediction is a central task in recommender systems, yet selection bias creates a severe distributional gap between the clicked training samples and the entire inference space. To address selection bias, propensity-based methods such as inverse propensity scoring (IPS) and doubly robust (DR) have been adopted, which aim to estimate the unbiased learning objective from biased training samples. However, these approaches assume strictly positive propensities, implying every user-item pair has a nonzero probability of interaction. In practice, such positivity assumption maybe violated, for example, in food-delivery platforms, some restaurants located more than 10 kilometers away will be blocked for recommendation. In this study, we theoretically show that when such zero-propensity samples, termed extrapolation samples exist, both IPS and DR estimators become biased. To overcome this limitation, we propose ExtraDebias method, which enables debiased recommendation in both non-extrapolation and extrapolation samples. Specifically, we first train a propensity model to identify extrapolation samples with extremely small propensity estimates, then estimate their pseudo-label intervals, and derive an upper bound of the learning objective for extrapolation samples. By minimizing the derived upper bound, debiased learning on extrapolation samples is ensured, while unbiased learning on non-extrapolation samples is achieved by standard IPS. Experiments on four real-world offline datasets and one online A/B test show that ExtraDebias effectively minimizes prediction errors on extrapolation samples and achieves optimal performance.
Yanghao Xiao, Hao Wang, Xiang Li et al.· Annual International ACM SIG...· 0 citations