What NVIDIA measured
Tung, Huang, Saxena, Shirvani, Hukerikar, Jain, Tyagi and Gongalore ran a gate-level stuck-at fault injection campaign on a production-class data-center GPU, over three million simulator hours across 63 CUDA micro-benchmarks, and published it in the DSN 2026 Industry Track. Three of their findings change how a tester should plant faults:
- NaN and infinities are only 1.01% of silent corruptions. Half of all corrupted values, 50.68%, are simply set to zero.
- Single-bit flips are under 40% of bit-flip corruptions. Most flip several bits, and low bits more often than high ones.
- Corrupted outputs repeat at warp-aligned periods of 2, 4, 8 or 16, so one fault hits a group of values.
What the tester now does
Since version 0.4.2, the self-tests of the compute, transformer kernel and data pattern probes plant faults shaped that way, alternating with single random bit flips. The paper gives its flip-count histogram only as a chart, so two parts are an approximation, stated openly: 35% of bit-flip outcomes are single-bit, under the 40% ceiling it reports, and the rest flip two to eight bits.
What it found on a real GPU
NVIDIA's faults are permanent: a broken unit gives the same wrong answer on every run, so comparing runs cannot catch them, and the reference check has to. On 15 October 2026 the compute probe ran on a rented RTX 4090 with PyTorch 2.8.0 and only the NVIDIA-shaped faults, 49 in each of six precisions at 1024×1024×1024. The reference check alone flagged all 294. A smoke run of every probe on the same card caught all 153 planted faults, 55 of them NVIDIA-shaped.