ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
TopicDisconfirmAI EditorialUpdated 2026-11-21 16:47

Dharma-AI Fine-Tuning Time Prediction

2 steps1 pieces4 Sources

Dharma-AI, a Brazilian AI lab specializing in Small Specialized Language Models (SSLMs), has released findings from an internal study on predicting the end-to-end duration of supervised fine-tuning (SFT) jobs. The team analyzed 272 production runs on NVIDIA H200 GPUs, totaling approximately 7,382 GPU-hours. Their results indicate that the industry-standard FLOP-based calculation formula is a poor predictor of actual wall-clock time, achieving an R² of only 0.31 and a mean absolute error (MAE) of 12.41 hours. In contrast, a CatBoost machine learning model trained exclusively on configuration data available before job dispatch achieved an R² of 0.98 and an MAE of 1.71 hours.

The core finding is that computational volume (parameters and tokens) is not the primary driver of fine-tuning duration; rather, operational overheads such as evaluation frequency, checkpoint writing, and shared cluster contention are dominant factors. The most significant predictive feature identified was num_eval_data (total samples processed during evaluations). This suggests that for teams managing shared GPU clusters, scheduling accuracy depends more on I/O and evaluation cadence settings than on raw compute estimates.

However, these results are currently issuer-reported and based on a single organization's logs using specific hardware (H200s). The full paper, titled "Predicting End-to-End Fine-Tuning Time for Generative Language Models," is forthcoming on arXiv, but no code or dataset has been released yet. The model's transferability to other GPU generations or workloads like RLHF remains unverified.

How it got here

  1. 2026-11-18

    November 18, 2026: Dharma-AI publishes a blog post previewing their upcoming paper, detailing the analysis of 272 SFT jobs on H200 GPUs.

    huggingface.co ↗
  2. 2026-11-18

    November 18, 2026: Secondary outlets including Aiholics, Quanta AI Labs, and DataPhoenix report on the findings, highlighting the disparity between FLOP estimates and learned predictors.

Sign in / Sign up
ReadingResearchRadarInvestment framework
aiholics.com ↗

What is still unsettled

  • When will the full paper be published on arXiv with reproducible methodology?
  • Will the model maintain its high accuracy when transferred to different GPU architectures (e.g., A100, H100) or non-H200 clusters?
  • Does the improved prediction accuracy translate into measurable gains in GPU utilization or reduced idle time in real-world scheduling?

Sources

  • huggingface.coHugging Face Blog (Dharma-AI)↗
  • aiholics.comAiholics↗
  • quantaailabs.comQuanta AI Labs↗
  • dataphoenix.infoDataPhoenix↗

Coverage in this topic

  • Why Fine-Tuning Takes Much Longer Than FLOP Estimates: Evaluation Frequency and Checkpoint Writes Are the Dominant Factors2026-11-19News

    Data from 272 production fine-tuning tasks by the Dharma-AI team demonstrates that the industry-standard FLOP formula has very low explanatory power for actual end-to-end duration (R²=0.31, mean absolute error 12.41 hours), whereas a CatBoost model using only configuration information available before task launch achieves R²=0.98 and a mean absolute error of 1.71 hours. The most important predictive factor is not parameter count or compute volume, but the total number of evaluation samples—i.e., how often you evaluate and how much data is evaluated each time.