The method behind every verdict
Every probe follows the same idea: run real AI maths many times, check each answer two ways, and prove the checks work by planting faults on purpose. This page walks through each probe, how a verdict is read, the self-test, and what a PASS can and cannot tell you.
- The probes, and what each one exercises
- The two checks behind every verdict: a proven error bound and a bit-for-bit repeat
- How the self-test shows a clean result means something
- What a PASS means, and its limits
How a chip is tested here
Six probes so far, numbered as on the roadmap. Probe 1 makes the chip multiply matrices and judges every output element. Probe 2 fills the chip's memory with known values and reads every one back. Probe 3 checks the other kernels inside a transformer the same way as probe 1. Probe 6 has the chip compute checksums inside the same multiply, so a wrong element can be caught and located without any reference. Probe 9 feeds the matrix multiply inputs built to reach particular bits. Probe 11 reads the chip's own error counters around every run and asks whether the hardware noticed. The compute, kernel and pattern probes run two independent checks on every output, and the checksum probe adds a third, so a fault has to hide from all of them.
Probe 1: compute
The chip multiplies two fixed matrices, A and B, fifty times at each of three shapes and four precisions, plus FP8. Two checks run on every output.
The reference check, in plain words
Floating point arithmetic rounds at every step, so two correct computers can legitimately differ in the last digit. The question is how different they are allowed to be. Numerical analysis has a proven answer: for a dot product of length K, the rounding error can never exceed a bound that depends only on K, the precision and how the hardware rounds. Every element of the chip's result is compared with a float64 reference and must sit inside that bound. For int8 the arithmetic is exact, so the bound is zero.
acc = g·(|A|@|B|)
g = n·u / (1 − n·u)
fp32: n = K, u = 2⁻²⁴
fp16, bf16: n = 2K, u = 2⁻²³
u_out: 2⁻²⁴ fp32, 2⁻¹¹ fp16, 2⁻⁸ bf16
int8, FP8: exact, bound 0
fp32 runs on the FP32 fused multiply-add units with TF32 off. fp16 and bf16 run on tensor cores, which round by truncation rather than to nearest and add in blocks, as Fasi, Higham, Mikaitis and Pranesh measured, so they get the wider allowance. Every fp16 and bf16 row measured with v0.2.0 also passed a bound four times tighter than this one. An element outside the bound is a wrong answer, not a difference of opinion about tolerance.
The repeat check, in plain words
The same inputs through the same chip should give the same bits every time. Run 0 is kept, and every later run is compared bit for bit. This catches an intermittent fault far too small to leave the bound, because the two runs simply no longer match. On the chips measured so far every run matched, which is itself a useful fact: the matrix kernels on these chips are deterministic.
Probe 2: memory
The matrix multiply touches a few megabytes. Most of an 80 GB chip is memory, and a bad row or a stuck bit there corrupts whatever lands on it. The memory probe fills most of the device memory, writes a known value into every 32-bit word, waits, and reads every word back.
Probe 3: transformer kernels
Matrix multiply uses the multiply-accumulate units. A transformer block also runs exponentials, divisions, square roots, error functions, sines and cosines, which live on different silicon, and the research says faults are data-dependent: a unit that multiplies correctly can still get a softmax wrong. The kernel probe runs five of them with the device's own kernels, including its attention kernel, and judges every output element the same two ways.
Softmax
Rows of logits. Exercises exp, a sum reduction and a divide. The bound follows the error of each exponential through the sum and the quotient.
Layer norm
Mean, variance, reciprocal square root, scale and shift. The variance allowance is sized for the two-pass, Welford and E[x²]−μ² methods, so a correct implementation of any of them passes.
GELU
The erf form, elementwise. The bound comes from the argument scaling, the erf itself and three roundings.
Rotary embeddings
How most current language models encode token position. The device takes sin and cos of real angles, up to 4,095 radians at the default size, then rotates each pair of values. Large angles test the argument reduction inside sin and cos.
Attention
softmax(QKT/√d)V through the device's fused attention kernel. The bound carries the tensor-core error of QKT through the softmax, allows for the running rescale fused kernels apply block by block, and for P being held in fp16 or bf16. That is why it is the widest.
Every bound on this site follows from one written arithmetic model, recorded in every result file: FP32 arithmetic rounds to nearest, tensor cores accumulate faithfully in FP32, exp, divide and square roots stay within eight ulps, and erf, sin and cos within 2−20 absolute, which admits the shortcut erf that PyTorch uses on ARM processors. Each step also measures the device's own exp, erf, sin and cos on its inputs, so a loose function library can be told apart from a faulty unit. CUDA's own math library documents errors well inside that. A device that breaks the model, or that substitutes an approximate GELU, shows as outside-error-bound on every run, and the message says so. Below the smallest normal value of the output type the bound becomes an absolute floor, because fp16 cannot represent a probability of 10−8 and no chip should be blamed for that.
Probe 6: checksum-protected matrix multiply
Huang and Abraham showed in 1984 that a multiply can check itself. Append to A a row holding its column sums, append to B a column holding its row sums, and one multiply returns C together with C's own row sums and column sums, computed by the same hardware in the same pass. A wrong element breaks exactly one row check and one column check, and where they cross is the element. No reference answer is needed, so the check can run inside real work for the cost of one extra row and column, and it is the design that later becomes a block in silicon.
B' = [ B , 2⁻ˢ·(row sums of B) ]
A'·B' = [ C , 2⁻ˢ·(row sums of C) ;
2⁻ˢ·(column sums of C) , … ]
In floating point, rounding makes a correct row sum differ slightly from its checksum, so everything depends on the threshold. Here each threshold is derived element by element from the same arithmetic model as every other bound, and the host removes the known rounding of the checksums exactly, so a correct chip can never cross it. The power of two 2⁻ˢ keeps the checksums from overflowing near the top of the range. The probe also measures its own blind spot: a rectangle of four errors that cancel in every row and column sum is invisible to checksums, so any run the reference check fails while the checksums hold is counted, never passed.
Probe 9: data patterns
Some faults only fire on particular bit patterns, and uniform random inputs almost never produce them. The data pattern probe runs the same matrix multiply, with the same two checks and the same proven or exact reference, on inputs built to reach specific parts of the arithmetic. Every input set is scaled by a power of two where needed, so no partial sum in any order can overflow.
FP8 keeps its inputs in {−1, 0, 1} so its answers stay exact, so it runs the alternate and sparse patterns only.
Probe 11: error counters
A silent error is one the chip does not notice. So every run of the other probes reads the chip's own error counters at its start and its end: ECC counts, remapped memory rows and retired pages, with temperature, clock and power beside them. Then it asks whether the hardware noticed what the probe noticed. Wrong answers with every counter quiet are a silent error in the strict sense, the case ECC and the error logs cannot see. The counters come from NVIDIA's management library, so on other chips the probe records that none are available. On a card with ECC switched off, as on the RTX 4090 and RTX A5000 measured here, the counters cannot record memory errors at all, and the probe says so instead of calling them quiet.
Reading a verdict
Every step ends in one of these words. They are chosen so a reader can tell a chip fault from a software quirk without opening the file.
Compute, kernel, checksum and data pattern probes
Error counters, every run
Memory probe
Self-test
The self-test
A clean table is only worth something if the tester can show it would have seen a fault. So every row is paired with an injection run.
Compute self-test
In five of the fifty runs a fault is planted before the checks run, and all five must be caught. Since v0.4.2 the five alternate between two kinds. Three flip one random bit of one random output element, which can be anywhere: sign, exponent or the lowest mantissa bit. Two follow NVIDIA's gate-level fault injection study of a data-center GPU (Tung et al., DSN 2026): one warp-aligned group of 32 outputs is hit at a regular spacing, and each value hit is zeroed, has several bits flipped, or, rarely, turns into NaN or infinity, in the proportions that study measured. On the chips measured so far, with the v0.2.0 bound, the reference check alone caught 46 of every 60 single-bit flips and the repeat check caught all 60. That is why there are two checks.
Memory self-test
After each pattern is written, five random bits in five random words of real device memory are flipped. The sweep must locate all of them by address. Because the flips happen in device memory, this exercises the actual read-back path, not only the comparison code.
NVIDIA's fault patterns on a real GPU
The faults in NVIDIA's study are permanent: a broken unit gives the same wrong answer on every run, so comparing runs cannot catch them and the reference check has to. On 15 October 2026 the compute probe ran on a rented RTX 4090 with only the NVIDIA-shaped faults, 49 in each of six precisions (fp32, fp16, bf16, int8, FP8 and fast-accumulation FP8) at 1024×1024×1024. The reference check alone flagged all 294. Because it judges each run on its own, the same faults would be flagged if they repeated on every run. Result file
What a PASS means, and does not
- It means: no silent error appeared in these matrix multiplies, these checksums, these data patterns, these transformer kernels and this memory sweep, in these runs, on this device, on this day.
- It does not mean: the chip is healthy. Faults that depend on heat or on hours of load are not yet exercised, and the data patterns so far cover the matrix multiply only. The next probes add them.
- Arithmetic model: every bound assumes FP32 arithmetic that rounds to nearest, tensor cores that accumulate faithfully in FP32, exp, divide and square roots within eight ulps, and erf, sin and cos within 2−20 absolute. A device that breaks the model shows as outside-error-bound on every run, and the message says so.
- One device per run: links between chips are not yet tested.
- Self-test scope: the compute injection corrupts the output after it leaves the device, so it proves the checker, not the chip. The memory injection corrupts device memory itself.