Book
Open access
Aug 2026
Semantic Association vs. Position: Which Information Do Attention Heads Prefer?
In large language models (LLMs), attention heads generally encode both semantic association information (SAI) and positional information (PI). This raises a fundamental interpretability question: Which does each attention head relatively prefer, SAI or PI? To address this question, based on theoretical analysis and der...
Meng-Wei Wang, Simin Niu, Sensen Zhang et al.
· Proceedings of the 32nd ACM... · 0 citations