The rule that looks true

Flip a coin a hundred times. You will probably get a run of six identical flips, and a run of seven is not unlikely. If you saw that run in your company's data — six consecutive months of a metric rising, six cases where a model got it right — you would look for a cause. And you would usually find one, because looking for a cause of something that already happened is a process that always succeeds.

What actually converges

The common idea about a coin is that the flips “even out” in the long run. That is true in exactly one sense and false in every other. What converges is the observed ratio: heads divided by flips, closing on 0.5. The probability of any single flip was 0.5 from the start, did not change, and does not change after ten heads in a row.

More to the point: the absolute gap between heads and tails never shrinks. It grows. Its average is about 0.8 times the square root of the number of flips — a thousand flips gives a gap around 25, a million gives a gap around 800. Every hundredfold increase in flips grows the gap tenfold while shrinking the ratio's error tenfold. Nothing “corrects” an earlier run; the gap widens, and a bigger denominator drowns it.

That square root is not a coincidence. The central limit theorem says the count of heads after many flips is normally distributed — the bell curve — around half, with a spread proportional to the square root of the count. The law of large numbers gives you the centre; the central limit theorem gives you the size of the wobble around it. The first on its own is reassuring. The second is the one that tells you how unconfident to be.

Where the rule breaks

The bell curve appears when many small independent effects are added: human heights, measurement errors, a survey's margin of error. It is absent — dangerously absent — in four cases. When events are correlated rather than independent: the 2008 models assumed mortgage defaults were independent, and they all failed together. When variance is infinite, so more samples improve nothing. When one term dominates the rest. And when factors multiply rather than add, which gives a log-normal distribution, not a normal one.

Which is why wealth, city sizes, book sales and asset prices are not read through their averages. And why a figure like “1% of cases carry 55% of the outcome” can be true in one unit and flatly false in another — something we hit in our own record: the figure is true in percent and false in the unit the ledger actually keeps its accounts in. The fat tail was an artefact of the measure, not a property of the result.

Now, AI systems

All of the above applies literally to evaluating the systems we build today. A model classifying documents, a quality gate judging outputs, a language model scored against a test set — each is measured on a finite number of cases, and each can look excellent by chance alone. The difference is that here the number of attempts is far larger: you change the prompt, change the threshold, run it again. With enough attempts you will always find a configuration that looks like it works.

The most dangerous form of this failure is not a check that gets it wrong; it is a blind check that says PASS to everything. A check that passes whatever you show it is indistinguishable from one that works, so long as you only ever show it good work. The only way to expose it is to ask it a question whose answer you already know: give it material with nothing in it to find. In our crypto study no classifier output is read until the model has first been run on randomly shuffled labels; if it finds an “edge” in the random, the pipeline is broken and no number that came out of it means anything.

Three things that make a result believable

First: write down what you expect before you run, together with the condition that would refute it. Without that, any result gets interpreted after the fact, and there is no result that cannot be interpreted. Register the hypothesis as an opposed pair wherever you can, because a single arm that looks good cannot be falsified at all.

Second, the least common and most important: run the system on noise. Shuffle the labels, or feed it scrambled data, and see what it says. A pipeline that finds an edge in noise is broken, and that has to be discovered before it costs money rather than after. A check whose failure mode you do not know is not a check — it is a ritual.

Third: hold out a sample you do not touch. Not in development, not in tuning, not for “a quick look”. Every look spends it, and a sample that has been looked at once has become part of development whether or not you admit it.

None of these three makes your system better. They make your claim about it truer — which is, in the end, the only difference between a system that works and a system that looks like it works.