Beyond State Capacity: Why Retrieval Breaks in Mamba
This work finds that gates in retrieval-critical layers are not yet aligned with relevance, and finds that Mamba's retrieval bottleneck extends beyond state capacity: even when capacity is sufficient, the delayed emergence of relevance information across depth often prevents successful retrieval.