NVIDIA Technical Blog: How to Fine-Tune Speech Recognition Models for Saudi Arabic Dialects While Retaining Multilingual Capabilities
NVIDIA engineers detail a workflow for fine-tuning the Nemotron 3.5 ASR model to recognize Saudi Najdi and Hijazi dialects using the NeMo framework. By employing minimal data curation, weighted replay mixing, duration-based bucketing, and partial encoder unfreezing, the model's word error rate on target dialects dropped from 55.05% to 29.96%, while English performance slightly improved. The article also explores the impact of decoding configurations on latency, offering a reproducible engineering reference for low-resource dialect adaptation.
Automatic Speech Recognition (ASR) models often perform well on mainstream languages and standard pronunciations but struggle when deployed in specific regional dialects or noisy environments. On October 1, 2026, the NVIDIA Technical Blog published a detailed tutorial demonstrating how to use the NeMo framework to fine-tune the Nemotron 3.5 ASR model to accurately recognize Saudi Najdi and Hijazi dialects without sacrificing existing English and other Arabic capabilities.
The core of this workflow addresses "catastrophic forgetting," where a model loses old knowledge while learning new dialects. The authors propose a weighted replay mixing strategy: combining 90% target dialect data with 10% previously learned data (7% English FLEURS and 3% Arabic FLEURS). This explicit declaration of proportions prevents issues where sliding-window shuffling might delay the effect of appended replay data until late in training. Experiments showed that just 7% English replay data was sufficient to slightly improve English Word Error Rate (WER) from 11.04% to 10.42%, proving the method effectively retains multilingual capabilities.
Regarding data processing, the article emphasizes "minimal curation." Instead of blindly applying default quality score thresholds, the authors inspected distributions before setting cutoffs. For instance, a default UTMOS quality threshold of 3.0 would have rejected almost all samples, whereas the actual threshold used was 1.25. By removing inaudible annotations (like "غيرواضح") and severely misaligned clips, they retained 103,559 valid utterances (82.5% of the original set). Additionally, duration-based bucketing reduced padding waste, making training more efficient for streaming encoders.
On model adaptation depth, the authors compared freezing the encoder, unfreezing the top N layers, and unfreezing all 24 layers. Results showed that updating all 24 encoder layers yielded the lowest error rates (WER 29.96%) but required training 230.4 million parameters. In contrast, unfreezing only the top 8 layers saved compute resources but resulted in a higher WER of 32.32%. This indicates that with approximately 134 hours of target speech data, full fine-tuning is feasible; however, if data is scarcer or memory is constrained, partial freezing is a more economical choice.
Beyond training, inference-time decoding configurations significantly impact performance. The authors found that expanding the attention context from the default [56, 3] frames to [56, 13] frames, combined with MALSD beam-8 decoding, further reduced WER by 2.71 percentage points to 27.25% without retraining. However, this improvement came at the cost of approximately 800 milliseconds of additional latency. Therefore, smaller context windows are more suitable for real-time captioning scenarios, while batch transcription tasks may justify the latency cost for higher accuracy.
The tutorial concludes that this methodology can be transferred to other languages or domains, though metadata, normalization rules, and evaluation metrics must be adjusted for specific corpora. For languages without whitespace word boundaries, WER may be misleading, suggesting character-level or morpheme-level metrics instead. Overall, this blog provides a complete engineering perspective—from data cleaning and mixing strategies to architecture adjustment and decoding optimization—helping developers achieve high-quality dialect adaptation under limited resources.