Temporal-spatial reasoning over hypergraph knowledge structures for multimodal retrieval-augmented generation

Abstract

Existing graph-based retrieval-augmented generation (RAG) systems represent knowledge with binary relations and rely primarily on semantic similarity for retrieval. This design struggles with multimodal queries requiring temporal constraints, spatial relationships, or higher-order interactions among entities. We present HyperTSRAG, a multimodal RAG retrieval algorithm that models knowledge as a hypergraph and performs explicit temporal-spatial reasoning during traversal. HyperTSRAG performs a bipartite-alternating best-first traversal that alternates between entity nodes and hyperedge nodes to capture higher-order connectivity and ranks candidate evidence using a scoring function applied within traversal that integrates semantic similarity, temporal coherence, spatial overlap, and structural importance. We evaluate HyperTSRAG on a benchmark corpus of 1,000 multimodal documents (text, images, audio, and video) with 500 queries spanning simple lookups through complex multi-hop reasoning. For complex queries, HyperTSRAG achieves 78.3% Recall@10, improving by 12.3% over GraphRAG and 18.3% over LightRAG, while maintaining a 95th-percentile latency of 1.83 s. On temporal-spatial subsets, HyperTSRAG attains 85.2% accuracy on temporal queries and 81.6% on spatial queries. Ablation studies show that individual scoring components contribute 2–5% gains. Relative to a matched Projected-Binary-RAG control derived from the same extracted hyperedges, HyperTSRAG improves Recall@10 by 9.1%, while a separate binary-graph traversal ablation yields a 13.4% decrease, underscoring the value of native n-ary structure. These results support hypergraph-native traversal with temporal-spatial-aware scoring as an effective retrieval paradigm for the evaluated multimodal RAG setting, particularly for constraint-driven queries that semantic-only retrieval cannot address.