LEARN

Silent errors in AI chips, from the beginning

Start here if the subject is new. This page explains how a chip can return a wrong number without raising any error, why that matters for anyone training or serving AI models, and the words used everywhere else on this site.

ON THIS PAGE YOU WILL LEARN

What a silent computation error is, with a worked example
Why it matters at the scale AI now runs at
The terms used in the method and the results

LEARN, PART 1 OF 5

What a silent error is

When a chip fails, there are three ways it can tell you. Only two of them are loud.

It crashes

The job stops. Annoying, but you know. Nothing is quietly wrong.

It raises an error

Memory with ECC, for example, notices a flipped bit, corrects it or reports it. Monitoring sees a counter go up.

It gives a wrong number

The job finishes. Every status light is green. One number in the output is wrong, and nothing on the chip knows. That is a silent error.

Where silent errors come from

A modern AI chip has tens of thousands of arithmetic units and billions of memory cells, made on a process so fine that a few atoms decide whether a transistor switches on time. Most units are perfect. A few are marginal: they work at one temperature and fail at another, work on most inputs and fail on a specific bit pattern, or degrade as the chip ages. When a marginal unit produces the wrong bit during a multiply, nothing compares the answer to the truth, so the wrong number simply flows onward.

The difference between a healthy and a marginal unit can be one bit: far below what anyone notices in a single number, and enough to change a model's output after a billion more multiplies.

Why the usual tests miss them

Vendor diagnostics run a fixed stress pattern for a short time and look for crashes and error counters. The research on real fleets says that is not enough. ByteDance's team reported that screening GPUs with synthetic microbenchmarks missed over 60 percent of the units that later corrupted real training runs, because the faults depend on the data being processed, show up with age, and are specific to individual units. A test that exercises one pattern for a few seconds can pass a chip that fails on the next pattern an hour later.

TRY IT

Flip a bit and watch the number change

Inside the chip every number is a row of bits. Click any bit below to flip it, the way a faulty transistor would, and see what happens to the value.

Starting value Format

signexponentmantissa, the digits of the number

Value now

Change

Relative change

Click a bit to flip it.

TRY IT

How much rounding a long sum is allowed

Adding up many products in floating point always rounds a little, and the rounding allowed grows with the length of the sum. Slide the length and see which mantissa bits could flip and still hide inside that allowance.

Terms in the sum

This uses the textbook bound from Higham's book for a sum of positive terms added in FP32: the error can be as large as γ_K = Ku/(1 − Ku) of the sum, where K is the number of terms and u = 2⁻²⁴. A flip of mantissa bit b changes a value by about 2^b−23 of itself. The tester's own bound also allows for the rounding of the output and each device's order of adding, so its exact numbers differ, but the picture is the same: a low-bit flip can hide inside the allowance, which is why every run is also compared with the first one bit for bit.

LEARN, PART 2 OF 5

Why it matters

One wrong number is harmless. The problem is scale: thousands of chips, billions of operations a second, and nothing that checks the answers.

Training runs go quietly wrong

In large language model training, a corrupted gradient from one faulty GPU spreads to every other GPU at the next synchronisation. The loss curve wobbles, days of compute are lost, and the cause stays invisible until someone finds the one unit. The second OSDI 2026 study below counted 18 such incidents and 13 faulty GPUs across 35 million GPU hours.

Inference answers drift

A serving fleet with one marginal chip produces slightly different answers from that chip, forever. No crash, no alert, just a share of users getting worse results.

The used GPU market has no condition test

Second-hand accelerators are bought and sold on model name and hours of use. Age is the biggest risk factor for silent faults, and there is no open, shared way to test a used card before paying for it.

Lenders cannot inspect the collateral

Loans are secured against GPUs. A jet has maintenance records and an inspection standard; a GPU has a serial number. An open integrity test is the first step towards a condition grade anyone can verify.

LEARN, PART 5 OF 5

Glossary

silent error, silent data corruption, SDC: A wrong result from a chip with no crash, no error flag and no log entry. The chip does not know it happened.
bit flip: One binary digit changing value. In a floating point number a flipped high bit can change the value by orders of magnitude; a flipped low bit changes it by less than one part in a million.
rotary embedding (RoPE): The way most current language models tell the model where a token sits: each pair of values in an attention head is rotated by an angle proportional to the position, computed with sin and cos.
ECC: Error-correcting code. Extra bits stored with memory that let the hardware detect and usually correct a single flipped bit. ECC covers memory, not the arithmetic units, and not every chip has it on all memory.
HBM: High bandwidth memory, the stacked memory next to a data centre GPU die. An H100 has 80 GB of it.
tensor core: A unit on NVIDIA GPUs that multiplies small matrices in one step. fp16, bf16, FP8 and int8 matrix multiplies run through tensor cores; fp32 runs through the regular floating point units.
fp32, fp16, bf16, FP8, int8: Number formats. fp32 keeps 24 bits of precision, fp16 keeps 11, bf16 keeps 8 with a wider range, FP8 (E4M3) keeps 4, int8 is an exact integer. Models are served in the smaller formats because they are faster, which is why the tester checks all five.
accumulation: The running sum inside a dot product. Even when the inputs are fp16, the sum is normally kept in fp32, which is what the reference bound assumes and what the tester enforces on CUDA.
deterministic kernel: A GPU routine that gives bit-identical output every time for the same input. Not all do, which is why non-determinism is reported as its own verdict instead of as a fault.
reference bound: The largest error that correct rounding can produce for a given computation and precision. Anything larger is a wrong answer. The bound used here is the standard componentwise bound for inner products from Higham.
fault injection: Deliberately corrupting a value to confirm the checks notice. Used here as a self-test on every published row.
dwell: The pause between writing memory and reading it back. Weak cells lose their value over time, so a longer dwell catches more of them.
OCP Test and Validation: The Open Compute Project's open format for hardware diagnostic output, used by cloud operators so that any test's results can be read by any fleet tool. All six probes write it.