Most often when faced with a decision, we want to gather as much data as possible in order to reach the best conclusions in our decision-making process. This much is perhaps rather obvious. But what do we do when there is missing data? It will then be tempting to discount whatever is missing as simply unavailable for the decision-making process, but perhaps we should be asking why the data is missing, and what we can learn from that fact. As Rob May writes at businesspundit,
During World War II, statistician Abraham Wald tried to determine where to add extra armor to airplanes. Based on the patterns of bullet holes in returning airplanes, he suggested that the parts not hit should be protected with extra armor. Why?
Think about it a minute and see if you get there. May explains how Wald was looking at something called “dead evidence,” and from that missing data he was drawing a counterintuitive conclusion, but one which is a wise and solid interpretation of what he observed. Essentially in this example, he reasoned that airplanes which were hit where the returning ones most often were hit could still fly… so the ones that did not return must have been hit somewhere else.
In observing all of the data you can gather in order to make a critical decision, stop and ask what you’re not being told. Ask why, and make an effort to determine what the absence of data might indicate. There may be indeed be nothing of value or nothing from which to draw any sound inferences, but if there is, you may be keeping yourself from being blindsided.