HOPE: Incorporating Expert Pairwise Interactions into Quadratic Programming to Maintain MoE Cooperative Groups at 50% Pruning Rate
Amazon researcher Alex M. Tseng proposes the HOPE method, which constructs an expert pairwise interaction matrix and solves a quadratic program to select the pruning set. This approach maintains the cooperative structure among experts at a 50% pruning rate, keeping SWE-Bench Pro scores on par with unpruned models, whereas the independently scored REAP method saw a 13% decline.
The vast majority of parameters in Mixture-of-Experts (MoE) models remain idle—of the approximately 250 GB of VRAM occupied by Qwen3.5-122B, 95% is taken up by the MLPs of experts across all layers, yet only 8 experts are actually activated per token. Directly deleting inactive experts is the most concise path to compressing this memory bottleneck: it requires no training, no dynamic loading, and can be combined with quantization. The difficulty lies in deciding whom to delete. Existing pruning methods—such as REAP, MAN, and EAN—assign a scalar importance score to each expert independently and then remove those with the lowest scores. Author Alex M. Tseng (Amazon) points out that this approach assumes each expert's contribution is independent of others, but experimental observations indicate that experts in MoEs form stable cooperative groups that are consistently co-activated. Independent scoring completely ignores this: removing expert A may seem harmless on its own, but if A frequently cooperates with high-importance expert B, a single deletion could dismantle an entire cooperative group.
The core of HOPE is an E×E interaction matrix F. The diagonal of the matrix measures the independent importance of individual experts—which correlates highly with REAP rankings, with a Spearman correlation coefficient of 0.988—while the off-diagonal elements capture the average gate-weighted output product of experts i and j on tokens where they are co-activated. With this matrix, selecting the pruning set becomes a quadratic programming problem: minimizing the total interaction strength among pruned experts. The solution employs continuous relaxation followed by rounding, taking 1 to 2 seconds per layer, which is negligible compared to the total overhead of pre-calibration forward propagation.