Why Fine-Tuning Takes Much Longer Than FLOP Estimates: Evaluation Frequency and Checkpoint Writes Are the Dominant Factors
When scheduling fine-tuning jobs on shared GPU clusters, almost every team uses the same formula to estimate duration: six times the parameter count multiplied by the number of training tokens, divided by the number of GPUs and peak throughput. This formula describes only the optimization loop itself, but a real fine-tuning job involves far more than just the optimization loop. Before the job starts, environment setup, model downloading and loading, and data tokenization are required; during training, periodic evaluations and checkpoint uploads occur; at the end, the final model must be written to object storage. On a multi-tenant cluster, all these operations compete with other jobs for network and storage bandwidth. In a blog post preview released on November 18, 2026, the Dharma-AI team quantified exactly how large these ignored overheads are, using operational logs from 272 supervised fine-tuning tasks in their production environment.
They used the FLOP formula as a baseline, testing across a range where Model FLOPs Utilization (MFU) varied from 0.25 to 0.65. The best result (MFU=0.30) still yielded a mean absolute error of 12.41 hours, with an R² of only 0.31. In other words, the average duration of these tasks was 23.43 hours, while the theoretical formula's prediction deviated by nearly half. As a second analytical baseline, they also tested Calculon (Isaev et al., SC '23), an analytical performance model that accounts for compute, memory traffic, and inter-device communication, extended here to cover LoRA, grouped-query attention, fused attention, bfloat16, and SwiGLU. The median of Calculon's estimates accounted for only 11% of the actual wall-clock time. This indicates that the vast majority of time in production jobs is spent outside the optimization steps.
In contrast, the CatBoost regressor trained by Dharma-AI used only information readable from the configuration before the job launched—hyperparameters, dataset size, model identifier, and number of allocated GPUs—and achieved an R² of 0.98, a mean absolute error of 1.71 hours, and a median error of 0.51 hours in 5-fold cross-validation. The source of this gap lies in two types of signals: first, evaluation frequency, checkpoint policy, per-GPU load, and fine-tuning regime, which the FLOP formula does not include at all; and second, the nonlinear interactions between these variables, which can only be learned from historical runs.
Among the 24 features in the CatBoost model, the top-ranked feature is neither parameter count nor GPU count, but num_eval_data—the total number of samples processed in evaluation over the entire lifecycle of the job. Following closely are trainable parameter count, evaluation batch size percentage, total parameter count, and per-GPU training batch percentage, with saved checkpoint parameters also ranking high. SHAP value analysis shows that high evaluation volumes consistently push predicted time upward, while the impact of trainable parameter count becomes significant only for the largest models. This means: how often you run evaluations, how much data is evaluated each time, combined with model size, determines when the job finishes. These are often choices made by habit when writing the training configuration.
Sources:huggingface.co