Meta F3 Lightning: After Decoupling Release and Execution, Why CPU Usage Remains High
Meta’s advertising ML feature framework, F3, compiles all features into a single merged graph to share computation, but this makes releases an all-or-nothing proposition. F3 Lightning decouples release granularity so each feature can be published independently, yet early prototypes saw significantly higher CPU usage. The team restored efficiency through four optimizations: stable identifiers, a single execution plan, shared memory buffers, and dense index mapping. The most counterintuitive finding was that even after deduplication and eliminating redundant calculations, differences in memory layout kept CPU usage above traditional F3 levels—at scale with over 20,000 features, where results are stored and how many copies exist matter just as much as how many times they are computed.
Meta’s ad ranking models require extracting more than 20,000 features per request. These features are not computationally independent: a seven-day click count and a seven-day impression count may read from the same event source and apply the same time-window filter, while a click-through rate feature reuses the results of both. Facebook Feature Framework (F3) compiles all features into a single directed acyclic graph (DAG), merging duplicate nodes so that shared computation runs only once. This drastically reduces execution costs, but the trade-off is that the merged graph also serves as the release unit. If one feature definition contains an error, the entire graph cannot go live, forcing all other features to wait.
As the number of features grew from a few hundred initially to 20,000, this problem became increasingly severe. The F3 team implemented several rounds of improvements to restore release speed, but each gain was eroded by newly deployed features because all features still shipped together in the merged graph. The goal of F3 Lightning was clear: enable independent release for each feature without sacrificing F3’s computational efficiency.
The most direct approach was to decouple only the release process. Each feature would compile into its own independent configuration, with separate validation, deployment, and rollback capabilities. However, early prototypes revealed unexpected costs: when independent configurations arrived at the serving process and executed separately, the repeated reads, filters, and aggregations eliminated by F3’s merged graph returned, causing significant CPU spikes. The issue was not release granularity but execution granularity—decoupling releases simultaneously decoupled shared computation.
The core idea behind F3 Lightning is to bring back shared execution without bringing back the merged graph. Specifically, at service startup, it consolidates the shared work across all currently active feature configurations into a single execution order, rather than merging all features into one graph at compile time. This is achieved in four steps.
Step one is stable identifiers. To recognize that two independently compiled configurations contain the same computation, the runtime must be able to match them. Node numbering fails here because numbers are local to each feature’s graph; the same seven-day filter might be node 2 in one configuration and node 7 in another, and inserting a single node causes all subsequent nodes to be renumbered. F3 Lightning has the compiler derive identifiers from the computation itself: the operation type, parameters, and identifiers of all inputs. Because input identifiers already reflect all upstream work, this identifier captures the entire chain of computation that produced the result. Features defining the same operation on the same inputs receive the same identifier, even if compiled and released weeks apart.
Step two is a single execution plan. At service startup, F3 Lightning performs a topological sort across all active features, placing every node after its dependencies. When multiple features contain nodes with the same identifier, the plan retains only one copy, with all consumers pointing to it. This ensures each computation runs only once. However, the plan remains a graph structure requiring pointer-by-pointer traversal by the executor. A flattening pass during startup converts it into an ordered list, turning each request into a single forward scan along the list instead of traversing a graph.
At this stage, computations no longer repeat, yet CPU usage remains higher than in traditional F3. The team identified that the remaining cost stems from memory layout: each configuration still carries its own request-time data container, scattering intermediate results across separate memory allocations. Executing the plan sequentially means jumping between different containers, with each jump incurring a cache miss.
Step three is shared memory buffering. F3 Lightning replaces per-feature independent containers with a single buffer per request. At startup, references for each feature are rebound to slots within that buffer. This yields three improvements: adjacent plan steps now touch adjacent memory, restoring the sequential access pattern the merged graph always possessed; a single readiness vector replaces per-feature state tracking, allowing the executor to check if a value is ready without reading the value itself; and consumers no longer receive their own copies—the shared slot holds a reference to the producer’s result, so a computation used by 20 features is stored only once. Scalar values are still copied because copying small values is cheaper than following pointers.
Step four is dense index mapping. Although values are now centralized, locating them remains expensive: every access requires resolving an identifier—hashing a key, probing buckets, and following pointers. At Meta’s serving volume, such lookups are inefficient. The final fix maps identifiers to consecutive integer slots at initialization and rewrites all references in every configuration to point to these slots. Reading a result becomes a direct array index into the shared buffer, eliminating hashing and lookup tables from the request path.
After completing these four steps, F3 Lightning achieves computational efficiency comparable to F3 while allowing each feature to be released independently. The team reports that extraction features reach production approximately 3.6 times faster than with traditional F3, with no increase in infrastructure compute costs.
However, the most counterintuitive conclusion of this project is not the acceleration metrics, but the discovery made in step three. The team had eliminated all redundant calculations, yet CPU usage remained high. The cause was not how many times computations ran, but where results were stored and how many copies existed. Two designs can execute identical operations in the same order, but one scatters results across multiple independent containers while the other concentrates them in a single continuous buffer—the latter behaves completely differently regarding cache performance. On modern CPUs, the physical location of data and the number of copies matter as much as the number of computations. This insight is more transferable than any single-step optimization: it implies that deduplication is merely one layer of efficiency engineering, while memory layout is another, often revealing itself only after deduplication is complete.