When a test cannot fail, it manufactures confidence

Engineering

The auditor that examined nothing

We built a script to audit colour contrast against the running interface. Palette-level unit tests already existed, but they can only prove that two token values are compatible — they cannot see which token a given screen actually reaches for, or that a translucent label is sitting on a tinted surface three levels up the tree. Every real failure we found was of that second kind.

The script ran clean. Screen after screen, zero findings. We took that as evidence the interface was in good shape and moved on.

It was skipping every string that contained a letter.

The cause was one line. To avoid auditing emoji and symbol glyphs — where the CSS colour property does not govern rendering anyway — the filter excluded a set of characters. It was written as a character range rather than a set, and one endpoint of that range sat above the Latin block. Read literally, the span covered every ordinary letter. So the filter did exactly what it was told and discarded almost all of the text on the page.

Why it survived

Because it agreed with us. A tool that reports problems gets scrutinised the moment it reports one you disagree with. A tool that reports nothing gets believed, because the result it produces is the result you were hoping for. There is no friction to make you look closer.

We only caught it because a unit test disagreed with the live audit. Two checks looking at overlapping ground, returning different answers, is what surfaced it — not review, and not the tool itself.

The second instance, one week later

We split a brand colour in two, because one value cannot do two jobs on a dark ground: a foreground version light enough to read on the page, and a darker surface version that white text can sit on and still clear AA. Then we swept the codebase to move every filled button onto the darker one.

The sweep was line-based. It matched the background property and the colour token on the same line. Three components wrote them across two lines — a ternary, a wrapped style object — and kept the foreground value under white text at 3.35:1. Every existing test passed, because none of them could see which token a component had reached for. The live audit missed them too; it can only inspect states it navigates into, and one of the three only appears mid-interaction.

The replacement is a static test that reads the source, matches across line breaks, allows the two indicator cases that legitimately use the foreground value as a background — each with a written reason rather than just a filename — and asserts that the pattern still matches a known violation. That last assertion is the whole lesson from the auditor, encoded so it cannot be forgotten.

What to take from this

Ask of any check you rely on: what would make this fail? If you cannot answer concretely, you do not know whether it is working. And a zero from a tool you have never seen produce a non-zero is not a result — it is an untested code path that happens to run in production every day.

← Back to Resources