Jun 2026· arXiv.org· Vol abs/2606.31166· 1 citation· 20 references
Computer Science
TL;DR
This work proposes method that unifies textual reasoning and graph message passing within a masked diffusion language model, a language model with bidirectional attention and generative decoding that outperforms graph neural networks, graph transformers, and LLM-based baselines on all three TAG benchmarks across two tasks.
Abstract
Text-attributed graphs (TAGs), where each node carries a natural language description, require models to jointly reason over text and graph topology. Existing approaches often handle the two modalities separately: graph neural networks operate on shallow text features, while hybrids of LLMs and graphs use the language model mainly as a text encoder and delegate structure learning to a separate graph module. We propose method that unifies textual reasoning and graph message passing within a masked diffusion language model, a language model with bidirectional attention and generative decoding. For each graph instance, method linearises a sampled local neighbourhood into a token sequence and injects graph structure through a topology attention mask, which realises message passing over the graph. Because the diffusion language model can both interpret and generate text, the method adapts to different tasks simply by changing the prompt, supporting node classification, link prediction, and cross-dataset transfer with no target-specific fine-tuning. Experiments show that method outperforms graph neural networks, graph transformers, and LLM-based baselines on all three TAG benchmarks across two tasks, improving over the strongest baseline by up to 3.9 points.
This work introduces agentic graph token reasoning, which recasts graph tokenization as part of the reasoning process itself and pushes LLM-based graph analysis from static graph-token encoders towards a graph-native agent paradigm.
Dynamic text-attributed graphs (DyTAGs) exhibit coupled textual and structural dynamics, and existing mainstream approaches for DyTAGs extend conventional large language models (LLMs) to capture both dynamics, thereby giving rise to dynamic graph LLMs. However, in DyTAGs, the continuous emergence of new nodes and edges with incoming textual content and interactions drives the joint evolution of graph structural-textual patterns, causing existing methods to struggle with evolving patterns. This motivates a largely unexplored problem of continual learning on DyTAGs, which aims to adapt to constantly evolving graph structural-textual patterns while retaining past knowledge, which imposes two challenges: 1) unlike common graphs, graph structure and textual semantics in emerging DyTAG patterns jointly evolve, requiring dynamic graph LLMs to adapt structure, text, and graph-text fusion simultaneously; and 2) updating dynamic graph LLMs to fit a new pattern may destroy the global graph-text fusion capabilities and bias the model towards recent local dynamics. To address these challenges, we propose a novel Continual Learning Dynamic Graph LLM framework (Continual-GraphLLM) to continually adapt to incoming patterns by routing them to experts specialized in similar past patterns, while mitigating the overwriting of previously learned patterns by assigning new experts to unseen patterns. Specifically, we propose a graph-text factor-based router to adapt to incoming structural-textual joint patterns by utilizing latent factors to adaptively activate suitable experts. Furthermore, we design invariance regularized multi-scale experts that mitigate forgetting by capturing the invariances among learned patterns assigned to the same expert, where each expert progressively integrates structural and textual information from local scale to global scale. Extensive experiments on real-world DyTAGs demonstrate the superiority of our method over competitive baselines, highlighting its effectiveness in adapting to emerging DyTAG patterns.
Tianhang Wan, Xin Wang, Haibo Chen et al.· Proceedings of the 32nd ACM...· 0 citations
Multi-Label Text Classification (MLTC) is a crucial task in Natural Language Processing (NLP) that involves assigning multiple labels to a given text. It has been extensively applied in various domains [1]. Most existing studies focus heavily on the manually annotated labels, while largely overlooking the complex interactions between labels and text. In this paper, a novel Heterogeneous Three-View Graph Convolution Network (HTV-GCN) is proposed, which combines a group-wise smooth contrastive mechanism with three heterogeneous graphs: global, local, and text-lemma. The global label graph serves to enrich the knowledge and conceptual structure of high-frequency labels, and the local label graph focuses on relationships between document-specific and long-tail labels. The text-lemma graph is designed to capture fine-grained word-level information. The proposed three-view framework significantly improves the expressive ability of text-label alignment. In addition, a novel contrastive mechanism is designed to enhance the discriminative strength between global and local label graphs. Comprehensive experiments conducted on benchmark datasets show that the proposed scheme consistently outperforms state-of-the-art baselines under various evaluation metrics, and ablations confirm the effectiveness of its contrastive mechanisms and multi-graph fusion.
Yili Wang, Zhicheng Liu, Chengsheng Yuan· ACM Transactions on Asian an...· 0 citations
Graph neural networks (GNNs) are typically conceptualized as message-passing neural networks, yet it remains unclear why neighborhood aggregation reliably outperforms node-wise multilayer perceptrons (MLPs). Despite its empirical success, this paradigm can be computationally expensive and sensitive to imperfect graph structures. In this work, we present a retrieval-augmented view of GNNs: each layer makes predictions by applying an MLP to a node representation together with a permutation-invariant summary of retrieved graph context. Motivated by this perspective, we propose RTA, a simple MLP-based framework that replaces structural message passing with label-aware retrieval and propagation. We provide theoretical insights that (i) connect retrieval-based aggregation to softmax-attention message passing, and (ii) establish the robustness of retrieved-context supervision to mis-retrieved outliers. Experiments on multiple text-attributed graph benchmarks show that RTA matches or even outperforms strong GNN and graph LLM baselines while improving efficiency and robustness across diverse scenarios.
Jintang Li, Yuhong Chen, Ruo-Fan Wu et al.· 0 citations
Hypergraphs effectively model higher-order groupwise relationships beyond pairwise interactions, while pretrained language models (PLMs) and large language models (LLMs) provide rich semantic understanding from textual attributes. However, research on combining language models with hypergraph learning remains limited due to the lack of public text-attributed hypergraph benchmarks. To address this limitation, we present TAHB (Text-Attributed Hypergraph Benchmark), the first public benchmark integrating hypergraph structures and raw textual attributes. TAHB contains 10 real-world datasets from four domains - e-commerce, academia, movies, and politics networks - enabling systematic evaluation of text-aware hypergraph representation learning. Experimental results show that TAHB preserves key structural properties of real-world hypergraphs and consistently reproduces performance tendencies observed in existing benchmarks. Furthermore, experiments under both LLM-as-Enhancer and LLM-as-Predictor settings demonstrate that LLM-enhanced textual semantics improve hypergraph learning performance, while structural and textual information jointly provide the best setting for LLM-based prediction. Our benchmark provides a foundation for future research at the intersection of hypergraph learning and language models.
D. Y. Kang, JungHyun Kim, Juhyun Jeon et al.· 0 citations
Graph neural networks (GNNs) are widely used to represent complex interactions and relationships among entities. We investigate a multimodal model that combines two complementary ideas: a self-supervised method that enables a GNN encoder pretrained on one dataset to operate directly on another dataset with a different node-feature dimensionality, without rebuilding the model or realigning the data; and an alternating optimization method that updates a language-model module in an E-step and a GNN module in an M-step, rather than jointly training a large language model and a GNN end to end on a large graph. Despite expectations, the combined model did not sufficiently improve predictive performance. We identify six factors: (1) an external anchor in the E-step has a strength-safety trade-off: a weak anchor has little effect, whereas an overly strong anchor can damage the graph representation; (2) the knowledge of the E-step teacher is not injected directly into the GCN embedding Z; (3) the representation space constructed in the M-step is not optimized for the same objective as the E-step teacher space, resulting in a compromise representation for target classification; (4) GCN propagation averages a node's own textual information with information from its neighbors; (5) cosine alignment does not guarantee axes that are discriminative for classification, so stronger geometric alignment with the E-step text anchor need not sufficiently improve the target decision boundary or classification performance; and (6) the force that preserves the source-side self-supervised geometry in the M-step conflicts with the force that moves the representation toward the E-step teacher. We support these observations through a staged set of experiments that varies the influence of the E-step.
Fumiaki Kimino, Ryoma Sato Sokendai, National Institute of Informatics· 0 citations