Why Fine-Tuning Takes Much Longer Than FLOP Estimates: Evaluation Frequency and Checkpoint Writes Are the Dominant Factors
Data from 272 production fine-tuning tasks by the Dharma-AI team demonstrates that the industry-standard FLOP formula has very low explanatory power for actual end-to-end duration (R²=0.31, mean absolute error 12.41 hours), whereas a CatBoost model using only configuration information available before task launch achieves R²=0.98 and a mean absolute error of 1.71 hours. The most important predictive factor is not parameter count or compute volume, but the total number of evaluation samples—i.e., how often you evaluate and how much data is evaluated each time.
When scheduling fine-tuning jobs on shared GPU clusters, almost every team uses the same formula to estimate duration: six times the parameter count multiplied by the number of training tokens, divided by the number of GPUs and peak throughput. This formula describes only the optimization loop itself, but a real fine-tuning job involves far more than just the optimization loop. Before the job starts, environment setup, model downloading and loading, and data tokenization are required; during training, periodic evaluations and checkpoint uploads occur; at the end, the final model must be written to object storage. On a multi-tenant cluster, all these operations compete with other jobs for network and storage bandwidth. In a blog post preview released on November 18, 2026, the Dharma-AI team quantified exactly how large these ignored overheads are, using operational logs from 272 supervised fine-tuning tasks in their production environment.
They used the FLOP formula as a baseline, testing across a range where Model FLOPs Utilization (MFU) varied from 0.25 to 0.65. The best result (MFU=0.30) still yielded a mean absolute error of 12.41 hours, with an R² of only 0.31. In other words, the average duration of these tasks was 23.43 hours, while the theoretical formula's prediction deviated by nearly half. As a second analytical baseline, they also tested Calculon (Isaev et al., SC '23), an analytical performance model that accounts for compute, memory traffic, and inter-device communication, extended here to cover LoRA, grouped-query attention, fused attention, bfloat16, and SwiGLU. The median of Calculon's estimates accounted for only 11% of the actual wall-clock time. This indicates that the vast majority of time in production jobs is spent outside the optimization steps.