Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value cache growing with context length. Research thus spans explicit-memory compression, sparse access, recurrent state construction, structured state dynamics, and heterogene...
Zhen-Tao Tan, Jing-Yi Shen, Yan-Bo Li et al.· 0 citations
It is proposed that intermediate layer tokens from vision encoders should be considered for effective visual token pruning, as semantic focus shifts across layers, with middle-layer tokens capturing more detailed object-centric information that deeper layers may abstract away.
Tian-Xiang Chen, Zhentao Tan, Zi Ye et al.· IEEE Transactions on Pattern...· 0 citations
NSFT (Neural Sub-expert Fine-Tuning), a fine-grained PEFT framework that refines MoE adaptation from experts to sub-experts, is proposed and suggests that sub-expert-level adaptation is a more precise and efficient PEFT paradigm for MoE LLMs.
Zhen-Tao Tan, Chang Liu, Yao Liu et al.· 1 citation
This paper introduces Caduceus, a family of MoE-enhanced foundation models built with a hierarchical pre-training paradigm to jointly integrate biological and natural language, and incorporates a multi-task instruction tuning phase, enabling robust protein parsing and natural language question answering.
Ming-Ze Yin, Yiheng Zhu, Jialu Wu et al.· Proceedings of the 32nd ACM...· 1 citation
This paper introduces Caduceus, a family of MoE-enhanced foundation models built with a hierarchical pre-training paradigm to jointly integrate biological and natural language, and incorporates a multi-task instruction tuning phase, enabling robust protein parsing and natural language question answering.
Mingze Yin, Yiheng Zhu, Jialu Wu et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
