Stanford Team Releases RT-J: Relational Foundation Model Cuts Required Examples by 10x for Predictions
A Stanford University team introduced RT-J, an 85-million-parameter relational foundation model that reportedly requires 10 times fewer labeled examples than existing strong baselines to achieve comparable predictive accuracy on RelBench benchmarks.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
Researchers at Stanford University have released RT-J, an 85-million-parameter relational foundation model (RFM). The model aims to address the challenge where traditional machine learning models typically require tens of thousands of labeled examples to adapt to new databases.
According to the paper's abstract, on in-context regression tasks from the RelBench benchmark, RT-J requires an average of 10 times fewer examples to match the performance of strong feature engineering pipelines using LLM-agent or RDBLearn featurizers combined with TabICLv2 or LightGBM predictors.
To achieve this efficiency, the team assembled "THE JOIN," a large dataset comprising 639 web-crawled databases and 6,023 LLM-generated forecasting tasks. RT-J utilizes long-context pretraining, a novel lightweight random-walk retriever, and test-time compute scaling via context tuning and ensembling.
These results are based on author-run benchmarks and have not yet been independently reproduced.