SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning
Supervised fine-tuning (SFT) equips large language models (LLMs) with specialized capabilities, but often comes at the cost of forgetting the general capabilities of their parent models (i.e., the pretrained models before fine-tuning). This trade-off is especially limiting for queries that require both specialized and...
Ke-Nan Tang, An-Dong Hua, Cheng-Xuan Qian et al.
· 0 citations