What it is
Imagine ten famous companies that made the same bold move and did well. What about those that tried it and failed? You never hear about them, so you never learn from them. Learning only from the winners you can still see is called survivorship bias.
To correct it, count everyone who tried, as if the results were not yet in. Then you learn two things. One is an honest base rate: how many of those who tried succeeded. The other is whether the failures had the winners' famous trait too. If so, it cannot explain the success.
Where it comes from
The idea has no single author, and nobody knows when the name was first used. The statistician Abraham Wald (1902–1950) was the first to study it carefully. In the Second World War he worked in the Statistical Research Group at Columbia University. In 1943 he wrote "A Method of Estimating Plane Vulnerability Based on Damage of Survivors". It was secret then; the Center for Naval Analyses reprinted it in 1980.
Wald could count damage only on the planes that came home. From them, he estimated the damage across every plane that flew. His reasoning: a plane hit in its weakest spot is the least likely to come home. So if returning planes show few hits in one place, hits there were bringing planes down.
In the popular version of the story, Wald tells the air force to armour the places with no bullet holes. That version came later. The paper itself is a set of equations for making estimates.
The term became common in finance in the 1990s. In 1992 four researchers, Brown, Goetzmann, Ibbotson and Ross, showed what goes wrong when data keeps only the investment funds still in business. In such data, fund results can look predictable when they are not.
What it corrects
Say you are deciding whether to copy another company's move: a pricing method, a hiring standard, a new product line. Your evidence is the companies that made the move and still exist. You study these winners carefully, find what they share and treat those traits as causes.
The mistake is hard to see because the list looks complete. You cannot tell from it that companies were left out because they failed. And success or failure is the very thing you want to explain. Working harder makes this worse. The closer you study the survivors, the more their shared trait stands out. You grow surer, but your answer is no more accurate. You can fix it only with evidence from outside the group of winners.
How it works
- Turn the claim into a rate: of everyone who tried this, how many got the result?
- Count everyone who tried, as if the results were not yet in.
- Name why companies go missing: bankruptcy, a quiet sale, leaving the stock exchange, a cancelled project, or a story too dull to write about.
- Find what you can about the missing group, even roughly: company registers, yearly counts of new firms, an investor's whole portfolio.
- Check whether the winners' trait was also common among the failures.
- Keep only the conclusions that pass that check, and say what you could not find.
Worked example
In October 1982 two McKinsey consultants, Tom Peters and Robert Waterman, published In Search of Excellence. They asked partners at their firm and other business people who was doing impressive work, and got 62 names. After testing each company's results, they kept 43. General Electric was on the first list but was dropped: only results counted. The book's eight principles, from "a bias for action" to "stick to the knitting", came from what the 43 had in common.
The authors chose the companies by their results. Any company that followed the principles but had a bad decade failed the tests and was left out. So nobody could tell from the study whether the principles caused those results. Several of the 43 later struggled. One was Wang Laboratories: at its peak in the 1980s, its yearly sales were almost $3 billion. In August 1992 it filed for bankruptcy protection.
That does not make the book worthless. In 2002 Forbes found that after the book came out, shares in the 43 still earned about 14.1% a year on average. Shares in Dow Jones index companies earned 11.3%. But nobody could prove from the book that the principles caused the success: the other half of the evidence was missing.
In a Business Case Weekly case
In the Y Combinator case, you hear first about a few famous startups that Y Combinator funded. In all, it funded 5,668 companies. It put about $1.5 billion into them, and together they are worth roughly $600 billion. One question is whether bigger batches, the groups of startups that go through the programme together, weaken Y Combinator's model.
To correct for survivorship bias, argue from success rates across all 5,668 companies, not from the few graduates everyone can name. Also ask what the median company got from the programme: the one in the middle when all 5,668 are ranked. Then you know which evidence would settle the question. The answer is still yours.
In your answer
- "These cases were picked for how they ended: the list was made after the results were known."
- "About N companies tried the same move. The ones missing from the story are …"
- "The same trait shows up in the failures I can find, so it cannot explain the success. At most, it is a requirement."
- "I cannot rebuild the missing group here, so I am unsure of this conclusion. I would test it by …"
Common misuse
A fake fix is to name the bias and carry on. Someone writes "there is obviously survivorship bias in these examples". Then they draw the same conclusion anyway. The opposite mistake is to refuse every lesson because of the bias, as if a successful company proved nothing. Ask yourself: did your base rate change, and by how much? If no number or conclusion changed, you have not done the work.
References
- Abraham Wald, "A Method of Estimating Plane Vulnerability Based on Damage of Survivors" (1943), reprinted by the Center for Naval Analyses, 1980. The original paper; hard reading.
- Stephen Brown, William Goetzmann, Roger Ibbotson and Stephen Ross, "Survivorship Bias in Performance Studies", Review of Financial Studies 5(4), 1992, 553–580.
- Mark Carhart, Jennifer Carpenter, Anthony Lynch and David Musto, "Mutual Fund Survivorship", Review of Financial Studies 15(5), 2002, 1439–1463. Measures the bias: 0.07% a year in one-year samples, about 1% a year in samples longer than fifteen years.
- Jordan Ellenberg, How Not to Be Wrong: The Power of Mathematical Thinking, Penguin Press, 2014. The book opens with the Wald story.