Hugging Face Catches Regression on CPU with Shrunk Models: Lessons from a UMT5 Attention Vulnerability Lurking for Nearly Two Months
On July 30, 2026, the Transformers library enabled PyTorch's scaled dot-product attention (SDPA) for T5 models. While this change did not introduce errors itself, it had a side effect: UMT5's decoder could now attend to future tokens. During training, the model could peek at the answers it was supposed to predict. The program did not crash, no tests failed, and the bug quietly lurked in the main branch for nearly two months.
The issue was not a lack of tests. Transformers supports over 750 model types, and its test suite is extensive, including more than 88,000 CPU tests and over 200,000 GPU tests covering pipelines, training, distributed execution, quantization, and attention kernels. However, integration tests require loading real checkpoints and running them on real hardware; executing all of these on every PR would keep many GPUs busy all day, which is impractical. By the time scheduled tasks detected a regression, the code might already be merged into main.
The Hugging Face team's response was to shrink the models themselves. They created something called a "fixture": starting from the configuration of a real checkpoint, they reduced the number of layers and width while preserving every type of layer present in the model, along with unchanged tokenizer and processor files. This shrunk model is published once and then reused. The team runs "fast integration tests" against it on inexpensive CPUs. The UMT5 fixture weights are only 66 MB, compared to the source checkpoint's 1,227 MB.
This approach is packaged as the "Model Integration Contract" (MIC), consisting of three parts: a fixture (the shrunk model), a usage path (a runnable example, such as generating text or performing a training step), and baselines (outputs recorded when running that example, reviewed by humans). The contract holds if running the usage path on the fixture still produces results consistent with the baselines. The contract covers architecture and usage paths rather than individual checkpoints—compatible checkpoints can share a fixture but retain their own expected results.
Shrinking a model is not simply about reducing size. The initial recipe merely reduced dimensions and layer counts, which was insufficient for mixed architectures: if the retained two layers both used sliding attention and the full-attention layers disappeared, the test lost meaning. The current recipe has four steps: shrink the configuration (reduce hidden sizes, attention heads, and depth while retaining different layer types, including mixtures of sliding and full attention, and dense and MoE layers); keep preprocessing unchanged (preserve vocabulary, special tokens, and input dimensions, copying tokenizer and processor files byte-for-byte); build and validate weights (try seeded random weights, change inputs to check for model response, train if necessary, reload to verify outputs, loss, and gradients are finite, and reject failed builds); freeze and record (save accepted weights, measure repeatable outputs on a known-good Transformers version, review baselines for each platform and precision, and publish a pinned snapshot).
Fontes:huggingface.co