PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation
PMOPD ( project-based Multi-Teacher On-Policy Distillation), which constructs subspace memories from the cumulative parameter displacements of different tasks and projects both gradients and optimizer updates to remove components that interfere with protected task directions, establishes geometry-aware optimization as...
You Liu, Ruo-Bing Zheng, Bo-Yuan Tong et al.
· 0 citations