Distribution-Robust Reranking for Conversational Music Recommendation: PoliBaJukebox at the ACM RecSys Challenge 2026

Abstract

We describe the PoliBaJukebox submission to the ACM RecSys Challenge 2026 on conversational music recommendation over the TalkPlayData 2 corpus. Our system implements a modular two-stage pipeline: a first stage retrieves candidates from the full track catalog using ten heterogeneous sources, fuses them with weighted Reciprocal Rank Fusion (wRRF), and reranks with gradient-boosted models; a second stage writes the reply through a two-pass, fact-grounded LLM pipeline. Our central finding is methodological: on this task, offline development metrics can move against the blind leaderboard rather than track it, so components validated offline routinely failed or regressed on the blind split. We trace this to a large, measurable covariate shift between development and blind sessions (an adversarial classifier separates them at AUC 0.937 from input-side descriptors alone); a standard correction for it, reweighting training toward blind-like cases, lowers development nDCG@20 yet is what let the ranker transfer. Under this shift, the offline selection criterion itself must change, not just the model. Our compliant submission achieved a composite score of 0.576 (nDCG@20 0.445), placing PoliBaJukebox in the final top 5. All codes and weights are released for exact reproduction: https://github.com/lopsandrea/music-crs-team2.