HiVe is proposed, a prompt tuning framework that models prompts at multiple levels and enables input-dependent specialization, and shows that HiVe consistently outperforms strong prompt tuning baselines across diverse tasks.
Abstract
As large language models (LLMs) continue to scale, parameter-efficient fine-tuning (PEFT) has become a practical alternative to full-parameter adaptation. Prompt tuning is effective, but existing approaches either use flat prompt structures or hierarchical structures with fixed prompt composition, limiting adaptive prompt specialization. To address this limitation, we propose HiVe, a prompt tuning framework that models prompts at multiple levels and enables input-dependent specialization. HiVe constructs a prompt hierarchy by leveraging inter-task relationships during training, and employs a vertical mixture-of-experts (V-MoE) mechanism at inference time to compose prompts up to the level of specialization required for each input. Experiments show that HiVe consistently outperforms strong prompt tuning baselines across diverse tasks.
As large language models are increasingly deployed across diverse downstream tasks, efficient task adaptation has emerged as a central challenge. In response, a wide range of task adaptation methods have been proposed, spanning parameter-efficient fine-tuning, in-context learning, and embedding-injection approaches. Ho...
Jungwon Park, Changin Choi, Jimyeong Kim et al.· 0 citations
NSFT (Neural Sub-expert Fine-Tuning), a fine-grained PEFT framework that refines MoE adaptation from experts to sub-experts, is proposed and suggests that sub-expert-level adaptation is a more precise and efficient PEFT paradigm for MoE LLMs.
Zhen-Tao Tan, Chang Liu, Yao Liu et al.· 1 citation
Experiments across three instruction-tuned models show that HiRoute achieves high safety rates across multiple safety benchmarks while preserving safe-response helpfulness, reducing over-refusal, and maintaining competitive performance on general-purpose tasks.
Fang-Zhou Chen, Shiji Zhao, Mengyan Wang et al.· 0 citations
A Cross-graph Tuning-free Prompting Framework (CTP) is introduced, which supports both homogeneous and heterogeneous graphs, can be directly deployed to unseen graphs without further parameter tuning, and thus enables a plug-and-play GNN inference engine.
Yaqi Chen, Shi-Xun Huang, Lei Wang et al.· Neural Networks· 0 citations
The current robustness fine-tuning methods improve over standard fine-tuning and in-context learning, but the best-to-worst prompt gap remains as high as 40-57% of performance, and recent robustness-enhancing methods often fail to outperform the simplest data construction strategy: training on one template per batch.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.