The Bombers That Never Came Home: Survivorship Bias

In 1943, a team of statisticians did war work at 401 West 118th Street in New York, next door to Columbia University. The Statistical Research Group included Milton Friedman, George Stigler, and a mathematician named Abraham Wald. Wald was a refugee, barred from a university post in Vienna because he was Jewish.

The military wanted to know how vulnerable its bombers were. It had detailed damage data, but only from the planes that made it home. That is survivorship bias in one sentence: studying only the things that survived, and forgetting the ones that did not. The planes that were shot down took their data with them.

Wald's answer was not a slogan. It was eight dense memoranda, "A Method of Estimating Plane Vulnerability Based on Damage of Survivors".

He used three numbers: planes sent out, planes that returned, and hits on each returning plane. From these he estimated the chance a plane survives one hit, two hits, three. In effect, he reconstructed on paper the damage on the planes nobody could see. He was famously careful with his assumptions: planes are lost to enemy fire, and no plane goes down with zero hits.

The myth is a survivor too

You have probably seen the famous version. A bomber diagram covered in red dots, and the punchline about putting armour where the holes aren't. Enjoy it, but know this.

The punchline is Jordan Ellenberg's phrasing from 2014. In 2016, Bill Casselman traced the primary sources for the American Mathematical Society. He found no diagram, no armour quote, and no armour recommendation anywhere in Wald's memoranda.

So the legend of survivorship bias is itself a survivor of selective retelling. The vivid version outlived the true one. Keep that in mind the next time a case study sounds too tidy.

Survivorship bias in the plant room

You benchmark your chiller plant against "similar buildings". But that dataset only contains buildings still running well enough to share their numbers. The ones that fell apart dropped out years ago. Your benchmark is a class photo of the survivors.

Energy case studies have the same shape. Nobody publishes the retrofit that saved nothing. So the published average is not the real average. It is the average of the wins.

Then there is the classic: "We've never had a chiller failure, so why pay for maintenance?" Ask around. Often the compressor that failed was swapped during a shutdown five years ago, and nobody wrote it up. The survivors talk; the failures got replaced quietly.

Vendor ROI works the same way. "Average payback across our projects" usually means the projects still running and still returning the vendor's calls. Finance proved this formally in 1992: Brown and colleagues showed that trimming a sample to its survivors makes performance look predictable all by itself. Replace "fund returns" with "kW saved" and the arithmetic does not change.

Count missions flown, not planes returned

Wald's real move was not cleverness. It was refusing to let the missing data stay missing. He insisted on the full denominator: every plane sent out, not just every plane that landed.

The plant-room equivalent is instrumentation. Log the failures and the near-misses, not just the wins. And watch the buildings that are quietly getting worse, because they will never raise their hands.

Keep your baseline too. Under IPMVP, the standard protocol for verifying savings, an energy saving is an absence: energy you did not use. You can only measure it against a recorded baseline.

This is the boring case for continuous monitoring, and it is where a platform like CobiNeural earns its keep. It keeps the denominator: every meter, every alert, every anomaly, the bad weeks logged next to the good ones. It turns that record into ISO 50001- and EECA-aligned baselines you can defend later. No magic, just not losing the data.

Where the metaphor breaks

Let's be honest about the limits. Wald had a closed problem. He knew exactly how many planes went out, and losses had one dominant cause: enemy fire. His model could calculate what he could not see.

A facility has none of that. Nobody keeps a registry of missions flown. The causes are tangled together: weather, occupancy, tariffs, deferred maintenance. And buildings are not shot down.

They degrade slowly while staying in the sample, more like planes that came home flying badly and were never inspected. That makes the bias harder to spot, not easier. You cannot repeat Wald's math with plant data. You can only go and collect the data he had to reconstruct.

One more thing worth copying: Wald never told the military where to put the armour. His group answered the statistical question and left the decision to the soldiers. Restraint is part of the method.

Wald died in December 1950, on a lecture tour of India, when his plane went down in the Nilgiri hills. The man who reconstructed the planes that never came home did not come home himself.

If your savings live in a slide deck and your failures live in nobody's memory, the fix starts with keeping the denominator. Book a demo and we will show you what that looks like on your own meters.