Jul 2026· Annual International ACM SIGIR Conference on Research and Development in Information Retrieval· 0 citations· 21 references
Computer Science
TL;DR
This work proposes a fine-grained task vector adaptation method that learns parameter-wise scaling coefficients for the task vector that are optimized using a language modeling objective while keeping all model parameters fixed, enabling effective integration of domain-specific knowledge without degrading reranking capabilities.
Abstract
Large language model (LLM)-based rerankers have demonstrated strong performance in information retrieval tasks, but adapting them to specialized domains remains challenging due to the substantial cost and effort required to construct high-quality domain-specific training datasets. To address this limitation, we leverage task vectors derived from continuous pretraining as a mechanism for transferring domain knowledge. However, existing task vector integration methods are highly sensitive to scaling factors and can lead to unstable performance across domains and model scales. In this work, we propose a fine-grained task vector adaptation method that learns parameter-wise scaling coefficients for the task vector. These coefficients are optimized using a language modeling objective while keeping all model parameters fixed, enabling effective integration of domain-specific knowledge without degrading reranking capabilities. Experiments on a general-domain benchmark and five specialized domains across two model scales demonstrate that our method provides stable improvements across most domains and avoids the degradation observed with fixed task vector scaling.
Two-stage retrieval pipelines, where a fast retriever identifies candidates and then a more powerful reranker scrutinizes them, have become standard across retrieval-based applications, including open-domain question answering and retrieval-augmented generation systems. In specialized domains or low-resource languages, domain adaptation is often required to ensure retrieval quality. In this paper, we explore domain adaptation of rerankers using unlabeled and labeled, real and synthetic datasets across legal, biomedical, and auditing domains. In the domains we study, we find that when pretraining data is limited, task-specific fine-tuning—using only a few hundred labeled examples in the domain—can match or surpass the gains of domain-adaptive pretraining, suggesting that domain-adaptive pretraining may not always be necessary for strong domain performance. We further evaluate this on two industrial Japanese datasets, where we observe practical improvements with purely synthetic training data.
Tuukka Karvonen, Alessio Staffini, Kenichi Maeda et al.· Annual International ACM SIG...· 0 citations
An algorithm framework named DR-EFT (Domain-Representative Experts for Fine-Tuning), which explores and loads the domain-representative experts for subsequent retraining and reincorporation and demonstrates robustness through validations on popular MoE LLMs, including Qwen, DeepSeek, and Ernie.
Zhaomeng Cheng, Zhong Ji, Yan Zhang et al.· Neural Networks· 0 citations
TopoTuner is competitive with full fine-tuning while training only 1-2% of the model parameters, and outperforms LoRA in 7 out of 9 model-dataset settings, which can change up to 39.57% of the projection parameters.
Abdulkadir Erol, Yash Mahajan, Vepaul Hariprashad et al.· 0 citations
Cross-domain recommendation is a well-known technique for improving recommendations in a target domain, especially under sparse data or cold-start conditions. A common strategy is to train user embeddings separately in the source and target domains and learn a transfer function between them. In contrast, we propose SetCDR, which constructs more effective user representations in the target domain by directly incorporating each user's source and target history. We additionally introduce a lightweight domain indicator that preserves data–domain relational information. These histories, composed of item–rating pairs, are represented as variable-length sets and processed using a permutation-invariant neural architecture. This differs from conventional neural networks, which do not naturally handle unordered inputs, and is well suited to recommender systems where user history sizes vary greatly among users. The use of a permutation-invariant architecture ensures consistent embeddings regardless of input order, improving robustness to real-world variability and training efficiency. We demonstrate SetCDR in two forms: a simple sum-pooling method and an extended multihead attention–based method that captures more complex dependencies within user histories. Moreover, because SetCDR operates directly on sets of user history records, it provides a natural way to examine how histories influence user representations. Finally, SetCDR adapts immediately to new interactions without additional retraining, enabling on-the-fly performance improvement. Experimental results across multiple cross-domain benchmarks confirm that SetCDR consistently outperforms strong baselines in recommendation quality.
K. Kim, James Geraci· Annual International ACM SIG...· 0 citations
A simple, budget-preserving pipeline is proposed that realigns the expert pool to the target domain before fine-tuning, and turns this redundant capacity into useful domain capacity and achieves lower training loss, with gains spanning all difficulty levels in downstream evaluation.
As the de facto standard for knowledge-intensive tasks, Retrieval-Augmented Generation (RAG) has significantly enhanced the reliability of Large Language Models by incorporating external non-parametric knowledge. However, adapting general-purpose retrievers to specific vertical domains often triggers catastrophic forgetting, severely degrading performance on open-domain queries. Additionally, existing mitigation strategies, such as linear model fusion, are mathematically constrained within a linear geometric manifold, limiting their ability to effectively rectify complex non-linear semantic drifts caused by domain shifts. To address these limitations, we present an effective approach, SuRe-EM (Subspace-routing & Residual-corrected Expert Model), designed to resolve the issues of domain specialization and Cross-Domain generalization in domain-adaptive retrieval. SuRe-EM enhances Cross-Domain representation by integrating fine-grained subspace routing with non-linear residual correction. Specifically, SuRe-EM first employs subspace routing to dynamically decouple high-dimensional features for maximizing domain specialization, followed by a residual module that generates non-linear semantic compensations. We validate our model on a vertical domain dataset (AHD) and general domain datasets (CMRC, SQuAD). Quantitative results demonstrate the effectiveness of SuRe-EM, which maintains strong In-Domain precision while improving Recall@10 by up to 4.46 points over state-of-the-art linear fusion baselines in Cross-Domain scenarios. Furthermore, comprehensive ablation studies validate the non-redundant synergy of the key design elements within SuRe-EM.
Xifan Liu· Poster Volume 0007 The 2026...· 0 citations