RosePO: Customized Preference Alignment in LLM-Based Recommendation

Abstract

Recent advancements in Large Language Models (LLMs) have inspired their application in sequential recommendation systems, often through supervised fine-tuning (SFT). However, conventional SFT methods often struggle to capture nuanced comparative relationships between items. While recent approaches utilize Direct Preference Optimization (DPO), they remain constrained by challenges in modeling customized preferences, including capturing fine-grained user preferences and being susceptible to semantic hallucination and popularity bias. To overcome these challenges, we propose RosePO, a framework to refine LLM-based recommendations through pairwise preference optimization with personalized smoothing. We illustrate the concept with three critical preference examples pertinent to LLM-based recommendation. Specifically, we design rejected sampling strategies tailored for each customized preference. To ensure robustness against uncertain labels present in automatically constructed preference data, we incorporate a personalized smoothing factor predicted by a user oracle into the optimization objective. Empirical evaluation on three real-world datasets demonstrates the effectiveness of our method, showcasing not only promising recommendation performance but also mitigation of semantic hallucination and popularity bias. We hope this work paves a way to build helpful and harmless LLM-based recommendation service in the future.