LLaVA-CKD: Bottom–Up Cascaded Knowledge Distillation for Vision–Language Models
A bottom-up cascaded knowledge distillation (CKD) framework is proposed, inspired by human formal education systems, and introduces one (potentially, more) additional Teacher(s) of intermediate capacity that gradually bring the Student network to the next level, where the next (higher-capacity) Teacher can take over.
Nikolaos Gkalelis, Vasileios Mezaris
· IEEE Access · 0 citations