Eliminate survivorship bias to make objective assessments

Atanu Biswas

Professor, Indian Statistical Institute, Kolkata

MORE than two millennia ago, Roman philosopher and politician Cicero told the story of Diagoras of Melos, the Greek poet and sophist of the 5th century BC. Diagoras, an atheist, was shown a series of painted tablets that portrayed a group of sailors who had prayed during a storm and survived a shipwreck. Diagoras, however, said: “I see those who were saved, but where are those painted who were shipwrecked?” They may constitute a much larger group, Diagoras, perhaps, thought.

Regardless of the perspective, as Cicero characterised it, Diagoras brought attention to a common shortcoming in human reasoning and decision-making: our tendency to firmly establish our foundation on the basis of what is visible while methodically ignoring the invisible. This is referred to as ‘survivorship bias’ since we tend to emphasise the experiences of the survivors and downplay the experiences of the unsuccessful. It makes sense that success stories tend to be more captivating than tales of failure in any sphere of life, which is why failures are rarely discussed, reported or analysed. However, in the process, we alarmingly overestimate the odds of actual success.

Survivorship bias is a significant bias in data that may lead to an entirely false estimate of the likelihood that an event will succeed, while, in reality, the likelihood may be quite low. It can influence all kinds of work and personal methods, including career decisions, putting ideas into practice and planning. Acknowledging it, as one navigates obstacles and problems, can help one avoid disappointment and wasting a lot of time. Consider instances from our daily lives, such as some coaching centres promoting how many of their students cracked difficult exams and some hospitals highlighting the number of successful cases of a specific therapy. Without having the total number of students or patients, these are examples of incomplete data that could cause erroneous judgements and misunderstandings among common people. Technically speaking, ‘survivorship bias’ or ‘survival bias’ refers to the fallacy of focusing only on entities that made it through a selection process and ignoring those that didn’t.

A well-known instance of survivorship bias is the archetypical narrative of renowned statistician Abraham Wald during World War II. To fortify vulnerable spots, the military intended to armour the aircraft. Unfortunately, armour would be too hefty to be placed everywhere. The planes that returned, the ‘survivors’ in this case, were examined for gunshot holes, and the military reinforced those places where the returning planes were hit the hardest.

Wald’s research group at Columbia University was asked to improve aircraft-damage prevention. Wald recognised a survivorship bias in the data: even after being struck in the noted places, the surviving planes managed to return. Therefore, instead of fortifying these sites, it’s imperative to deduce the missing information regarding the impact location of the non-returning aircraft, Wald thought. He realised that since the aircraft that were struck in those regions didn't return, they needed to bolster the areas on the returning planes that were unharmed. His analysis led the military to fortify the engine and other vulnerable parts, greatly enhancing crew safety in combat and saving many lives.

There are many other real-life instances of survivorship bias: we may prefer to concentrate on the amazing tale of a paani-puri seller turning into a sports star, tending to overlook the many instances where such a background resulted in nothing significant at all. In his book The Black Swan: The Impact of the Highly Improbable, Nassim Nicholas Taleb referred to the data masked by survivorship bias as ‘silent evidence’.

Record companies turned down four young men and told them that “guitar groups are on the way out”. Those young men would go on to become the Beatles, the most successful band in history! Steve Jobs, Mark Zuckerberg and Bill Gates are examples of college dropouts who went on to become incredibly successful entrepreneurs and eventually became some of the richest people on this planet. Does this imply that earning a college degree isn’t worthwhile? Although they made it through, their stories don't reveal how many college dropouts were unable to achieve anything meaningful in life. In a 2012 piece titled ‘What Happened to the Facebook Killer? It’s Complicated’, Alec Liu wrote that “for every Mark Zuckerberg, there are thousands of also-rans, who had parties no one ever attended, obsolete before we ever knew they existed.” Most turned-down bands wouldn’t become a public sensation, either.

In actuality, most start-ups fail. Most people never become well-known or wealthy. It does suggest that we should be reasonable in our perception of reality, not that we shouldn’t attempt, though. However, we celebrate the rule-breakers who defy conventional wisdom and nonetheless build successful empires. Overall, the world has a tendency to overlook survivorship bias.

Essentially, survival bias is a type of ‘selection bias’ whereby failures are mostly disregarded, perhaps resulting in unduly optimistic judgements. It’s the tale of forgotten failures. It occurs when we fail to take past failures into sufficient consideration and when we believe that success reveals the complete story. In his other book, Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets, Taleb wrote about our propensity to overlook failures: "We favour the visible, the embedded, the personal, the narrated, and the tangible; we scorn the abstract.”

Additionally, survivorship bias takes advantage of our propensity to conflate cause and effect. It generates an imprecise sample, leading us to draw erroneous inferences. The victorious person is frequently the focus of a captivating narrative. “Beware advice from the successful,” cautioned Google engineer Barnaby James.

How can one eliminate survivorship bias and make, if at all possible, a reliable assessment of the chances of success in various scenarios? Well, instead of focusing only on a successful subgroup, we might endeavour to discover methods for creating a representative sample from the population. Never simple, though. That procedure may require additional expense and effort as well as greater caution and an objective scientific mindset. However, if it is successful, the ‘silent evidence’ might help us comprehend the likelihood of occurrences around us much better.

Join Whatsapp Channel of The Tribune for latest updates.