Databricks Open-Source Solution: Eliminating Train-Serve Skew in Real-Time Recommendations with Unified Feature Definitions
Databricks has released an open-source solution accelerator demonstrating how declarative feature definitions via Feature Views can simultaneously serve offline training and online inference. In a retail banking scenario, this approach achieves a p95 ranking latency of 34 milliseconds. The article provides capacity planning formulas and SawtoothWindow long-window streaming feature techniques, but all performance data is derived from vendor self-testing using synthetic data.
Real-time recommendation systems face a classic engineering challenge: the features used during training and those used during online inference are often calculated by two independent codebases. Even when the logic is identical, differences in implementation details can drift over time, causing models to perform well on training sets but degrade in effectiveness after deployment—this is known as train/serve skew. In a solution accelerator published on November 16 by Tian Tan, Sixuan He, Nick Joung, and Julia Powell, Databricks proposes eliminating this problem at its root using declarative definitions in Feature Views: feature logic is written only once, shared between training and inference.
The specific approach divides features into two categories. Batch features (such as 30-day average balance, 90-day total spending, and loyalty tier) are declared using SlidingWindow, TumblingWindow, or ColumnSelection; the system automatically materializes them to offline Delta tables for training use while simultaneously writing them to online storage for inference retrieval. Streaming features (such as category views in the last 10 minutes and 30-day category interest) point to Kafka topic declarations; the system continuously aggregates events into online storage, while historical values are reconstructed from source events by timestamp for training. Both feature types share the same definition, retrieving values from online storage by customer_id during inference, and performing point-in-time joins via the create_training_set API during training to prevent future data leakage.