On July 30, 2026, Transformers enabled PyTorch's scaled dot-product attention (SDPA) for T5. As a side effect, UMT5's decoder could attend to future tokens. During training, it could peek at the answer it was supposed to predict. Nothing crashed, and the bug stayed in main for almost two months.
When we later replayed those commits, a small UMT5 model running on CPU flagged it on every one of them. This post is about that model and the tests built around it.
Transformers supports over 750 model types, and its test suite is big: 88,000+ CPU tests on main, plus 200,000+ GPU tests in a daily run. Model tests are only part of it: we also test pipelines, training, distributed execution, quantization and attention kernels.
