Skip to content

Context Window Failures in Relational Foundation Models

Aug 2026 · 0 citations · 5 references
Computer Science

Abstract

Recent Relational Deep Learning architectures have been proposed as foundation models for multi-table relational data, yet they impose constrained neighborhood budgets that force row truncation when an entity has many related records. We introduce Animus, a synthetic financial dataset in which predicting customer income requires aggregating up to tens of thousands of transactions. On the raw representation, three recently proposed models (RT, Griffin, RelGT) achieve $R^2 \le 0.18$; a single, routine, temporal pre-aggregation step recovers $R^2$ up to $0.65$. This questions whether current relational foundation models are ready for high-cardinality real-world data.

View source

Similar papers

Preprint Aug 2026

MINT: A Universal Zero-Shot Predictor for Transaction Data

The Multimodal Instruction Network for Transactions (MINT), a framework that connects a pretrained transaction sequence encoder to a decoder-only LLM through lightweight embedding injection, transaction-language alignment, and instruction tuning, achieves state-of-the-art predictive question-answering performance in bo...

Parameswaran Kamalaruban, Viktor Drobnyi, Maeve Madigan et al. · 0 citations
#machine learning Preprint Aug 2026

Can LLMs Use Relational Transformer Embeddings?

It is argued that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.

Francisco Galuppo Azevedo, Clarissa Lima Loures · 0 citations
Preprint Aug 2026

Incremental Evaluation and Training in Relational Deep Learning

Relational Deep Learning (RDL) models multi-tabular databases as temporal heterogeneous graphs to enable end-to-end representation learning. However, prevailing RDL evaluation practices rely on static, single-episode dataset snapshots, overlooking the continuous, time-evolving nature of real-world databases. Consequent...

Jakub Peleška, Gustav Šír · 0 citations
#artificial intelligence Preprint Sep 2026

ReLaG: A Scalable Framework Generalizing Random Splits to Data with Latent Relations

Random splitting can yield non-independent train--test subsets when a dataset contains related samples, as is common in certain applications such as biochemical studies. This leads to overly optimistic generalization estimates. Here, we introduce ReLaG, a modality-agnostic framework that models sample relatedness throu...

Anthony Lavertu, Jacob A. Cote, S. Gobeil et al. · 0 citations
#machine learning Preprint Sep 2026

Temporal Heterogeneous Graph Pretraining for Relational Deep Learning

This framework combines Multi-scale Time Encoding, which captures record age using learnable time scales and type-specific projections, with Rotary Time Encoding, which represents signed inter-record intervals through rotary transformations during graph propagation.

Yi-Xin Peng, Er Jin, Diego Collarana et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.