Do Large Language Models Play Six Degrees of Separation? Measuring Topological Compression in Long-Context Manifolds

Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

This work presents the first comprehensive mechanistic interpretability study of MLA, training a 114M-parameter transformer and analyzing its representations through SVD, attention head taxonomy, linear probing, and a disruption-attribution analysis.

RSF-GLLM: Bridging the Semantic Gap in Multi-Hop Knowledge Graph QA via Recurrent Soft-Flow and Decoupled LLM Generation

Multi-hop Question Answering over Knowledge Graphs faces a critical challenge: traditional retrieve-then-read pipelines break differentiability, preventing the retriever from learning to bridge the semantic gap where intermediate nodes lack lexical overlap with the query. To address this, we propose RSF-GLLM, a framework decoupling differentiable graph reasoning from answer generation. Our Recurrent Soft-Flow (RSF) module employs a GRU-guided query updater to propagate continuous relevance scores, utilizing a dynamic gating mechanism to traverse semantically dissimilar bridge nodes via structural cues. We introduce flow sparsity regularization to theoretically guarantee convergence from soft probabilities to discrete reasoning paths. These paths are extracted and textualized to fine-tune a Large Language Model (LLM), ensuring generation is grounded in factual topology. Experiments on WebQSP and CWQ demonstrate that RSF-GLLM achieves competitive performance with superior inference efficiency compared to LLM based computationally expensive approaches.

Surviving Resource-constraint Compression: Capability Retention under Tensor-train Decomposition for Sub-billion Parameter Language Models

High-dimensional token embeddings underpin generative language models, as they can capture subtle semantic information and significantly enhance the modelling of complex language patterns. However, this high dimensionality also introduces considerable model parameters and prohibitively high model storage and memory requirements, which are particularly unaffordable for low-end devices. Targeting no extra training data and insufficient computation cases, we propose a training-free model compression approach based on the tensor-train decomposition (TTD), whereby each pre-trained token embedding is converted into a lower-dimensional matrix product state (MPS). We investigate what language capabilities are preserved under training-free compression at different compression ratios, providing insights into the distinct redundancy structures captured by tensor-based versus pruning-based compression methods. We then comprehensively investigate the low-rank structures extracted by this approach, in terms of the compression ratio, the language task performance, and latency on a typical low-end device (i.e., Raspberry Pi). Our approach trades increased inference latency (no more than 0.5 ms/token reconstruction overhead on Raspberry Pi) for substantial memory and storage reduction, making it suitable for deployment scenarios where memory and storage are the primary bottleneck. Taking GPT family, OPT models, Qwen (2.5–0.5 B and 3–0.6 B) and MiniCPM4-0.5B as case studies, our approach for the embedding layer compression consistently achieves a compression factor 0.5× – 2.0×. The extension of our approach for dense layers (feed-forward layers and attention layers) compression, can improve the model language task performance in zero-shot reasoning tasks. Our performance analysis of different tasks reveals that tensor decomposition preserves higher-level logical reasoning capabilities (e.g., BoolQ, ARC-Challenge), while pruning-based methods like SliceGPT maintain advantages for tasks requiring broad lower-level lexical feature coverage (e.g., HellaSwag, WinoGrande), demonstrating that different compression approaches preserve complementary linguistic capabilities.

H4G: Unlocking Faithful Inference for Zero-Shot Graph Learning in Hyperbolic Space

Text-attributed graphs are widely used across domains, offering rich opportunities for zero-shot learning via graph-text alignment. However, existing methods struggle with tasks requiring fine-grained pattern recognition, particularly on heterophilic graphs. Through empirical and theoretical analysis, we identify an over-abstraction problem: current approaches operate at excessively large hyperbolic radii, compressing multi-scale structural information into uniform high-level abstractions. This abstraction-induced information loss obscures critical local patterns essential for accurate predictions. By analyzing embeddings in hyperbolic space, we demonstrate that optimal graph learning requires faithful preservation of fine-grained structural details, better retained by representations positioned closer to the origin. To address this, we propose H4G, a framework that systematically reduces embedding radii using learnable block-diagonal scaling matrices and Möbius matrix multiplication. This approach restores access to fine-grained patterns while maintaining global receptive ability with minimal computational overhead. Experiments show H4G achieves state-of-the-art zero-shot performance with 12.8% improvement on heterophilic graphs and 8.4% on homophilic graphs, confirming that radius reduction enables faithful multi-scale representation for advancing zero-shot graph learning.

Beyond a Bag of Features: Set-Level Instability in Sparse Autoencoders

It is found that SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states, but instead track model-internal similarity structure.

Efficient and Exact Global Attention on Latent Summaries for Knowledge Graph Reasoning