Miscalibrated benchmarks may have understated deep learning perturbation models

Nature Biotechnology published a peer-reviewed paper on October 1 finding that common benchmarking metrics are miscalibrated, and that deep-learning genetic perturbation models can beat uninformative baselines once metrics are calibrated.

Earlier benchmarks found a simple mean baseline matched or beat deep models on MAE, MSE and related metrics, casting doubt on in silico screening. This paper introduces positive and negative controls across 14 datasets and 18 metrics, showing the common metrics are often insensitive to real perturbation signal.

The authors also propose a calibration measure called dynamic range fraction (DRF) and released code. The result concerns relative performance under calibrated metrics, not whether the models yet reach usable predictive accuracy.