Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value cache growing with context length. Research thus spans explicit-memory compression, sparse access, recurrent state construction, structured state dynamics, and heterogene...
Zhen-Tao Tan, Jing-Yi Shen, Yan-Bo Li et al.· arXiv.org· 0 citations
It is proposed that intermediate layer tokens from vision encoders should be considered for effective visual token pruning, as semantic focus shifts across layers, with middle-layer tokens capturing more detailed object-centric information that deeper layers may abstract away.
Tian-Xiang Chen, Zhentao Tan, Zi Ye et al.· IEEE Transactions on Pattern...· 0 citations
NSFT (Neural Sub-expert Fine-Tuning), a fine-grained PEFT framework that refines MoE adaptation from experts to sub-experts, is proposed and suggests that sub-expert-level adaptation is a more precise and efficient PEFT paradigm for MoE LLMs.
Zhen-Tao Tan, Chang Liu, Yao Liu et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.