Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
SinkProbe is built, a suite that measures sink mass, massive activation, position resolved recall and the recency gap, and applies it to four small models that differ only in how they mix tokens and depth, and three results follow.
Sara Rizwan, Samaanah Abdus Salam, Mohammed Mudassir Uddin
· 0 citations