Noise

Picture four target-shooting teams, five shots each. The first team hits the bullseye every time. The second clusters its shots tightly together, but off in a corner, far from the center. The third scatters its shots around the center with no grouping at all. The fourth sprays shots all over the target with no rhyme or reason.

The second team has bias: it misses in the same direction, consistently. The third and fourth have noise, which is the scatter of the shots. This is the image that opens the book by Kahneman, Sibony and Sunstein, and it applies to any human judgment, from an essay grade to a prison sentence.

There is one curious detail. If you look at the target from behind, without seeing where the center is, you can still spot the mess of shots. That is noise: it can be seen even when nobody knows what the right answer is. In real life, almost nobody turns the target around to look.

The book proposes a simple equation. The total error of a set of judgments combines bias, which is the average tendency to miss in the same direction, and noise, which is the unwanted variation among judgments that should be identical. In the mean squared error formula, the two carry similar weight.

But public conversation talks almost only about bias: prejudice, slant, partiality. That is understandable, because bias has a culprit and a story to tell. Noise is quiet. Nobody feels wronged by a lottery they did not even know they were entered in.

The authors argue that an organization can cut error substantially by working on noise, even without being able to fix bias. That is why they ask us to stop staring only at the famous side of the problem and start looking at the silent side too.

The most striking case comes from the courts. In the 1970s, the American judge Marvin Frankel criticized the sentencing system, which he described as law without order, because similar cases received very different punishments depending on which judge happened to draw the case. Later studies tested this with fictional cases handed to multiple judges.

The results showed large differences, both between harsher and more lenient judges and in how each one reacts to the details of each case. Sentencing, which is the calculation of punishment within the limits the law allows, leaves room for this kind of variation in any country.

The problem, however, is not limited to courtrooms. It applies to doctors giving different diagnoses for the same test, to banks rating the same client differently, and to teachers grading the same exam with different yardsticks. Whoever is being evaluated pays the price of the lottery without knowing it exists.