Thomas Bayes (Credit: Wikipedia)
I recently read Nate Silver’s The Signal And The Noise (2012). Silver, the founder of statistical analysis website FiveThirtyEight, is arguably the most prominent statistician and forecaster in the United States today.
In this book, Silver seeks to answer a pressing question: Why do so many predictions fail? More specifically, why is it so hard to distinguish the signal (what Silver describes as “the truth”, or what is really important to observe) from the noise (“what distracts us from the truth”)? Silver also offers a solution: a probability-focused, statistical approach to thinking, specifically by applying a formula developed by mathematician Thomas Bayes, to better shape and inform our thinking, in many areas of life.
Silver cites a range of examples, involving both individual and collective judgment, where these issues play out. Most people aren’t very good at playing poker, or betting on athletic events. Intelligence analysts failed to predict either the Pearl Harbor or 9/11 attacks. Very few investors (professional or others) successfully beat stock market indexes, over the long run. These finance professionals share the company of economists, who largely failed to anticipate the 2008 financial crisis (both in terms of likelihood and severity). In fact, economists have done rather poorly, in terms of forecasting economic growth and recessions , over the past few decades as a whole. A projected projected outbreak of the H1N1 virus in 2009, never materialized. Political prognostication by (largely partisan) pundits, generally produces mediocre results. Silver even dives into the results of efforts to predict natural phenomena, such as earthquakes and inclement weather, exploring why these undertakings often don’t succeed.
What’s going on here? Do we lack the mental hardware, to see into the future? Not neccessarily. Let’s consider some of the reasons why predictions fall short.
During the 2008 financial crisis, homeowners overlooked the possibility of a large drop in home prices, since that hadn’t happened in the recent past, while economists underestimated the impact of housing prices on the financial system. Both these predictions proved incorrect, in large part because prior data was out of sample and thus not applicable: housing prices had never risen quite this much, in such a short time, while the financial system had never been quite this leveraged, with such large amounts of debt.
Political forecasters frequently fall short as well. The panelists on The McLaughlin Group fared no better than a coin flip, in terms of the accuracy of predictions offered on the show. Silver cites to the work of Philip Tetlock, who found that most professional politician scientists, commenting on global events, didn’t fare much better: “....about 15 percent of events that they claimed had no chance of occurring in fact happened, while about 25 percent of events that they said were absolutely sure things in fact failed to occur.” Much of this was because most forecasters were rather ideological “hedgehogs”, type A personalities with strongly defined worldviews, through which they offered predictions. The minority who were “foxes” (less ideological, more nuanced, and open to incorporating work from various disciplines) were actually much more effective in their predictions.
Poker players seem to lose as much money as they win (largely due to overconfidence in their own abilities; as professional player Tom Dwan put it “People can have some pretty deluded views on poker.”), while local weather forecasters offer innacurate predictions more often than they should, as compared to those offered by the National Weather Service (in part due to a tendency by many local news weather reports, to focus more on television ratings, than accurate weather predictions).
So what’s the solution? Silver offers a framework (really, a mental model), through which we might improve our understanding of the world: Bayes’ Theorem.
Thomas Bayes was an English statistician and minister, who became most famous for his eponymous Bayes’ Theorem, which was published after his death. As Silver explains, Bayes’ theorem is mainly concerned with “the probability that a theory or hypothesis is true if some event has happened.”
Bayes’ Theorem is a simple equation, which makes use of three variables: x (our initial estimate of the probability of some event), y (a new event occurs, which is conditional on x being true), and z (a new event occurs, but x is false). Bayes Theorem uses the formula (xy)/(xy + z(1-x)), to calculate probabilities. (Note that while Silver used the variables x, y and z, in his description of the formula, many professional statisticians use a different notation, to refer to the same variables).
Rather than consider it in the abstract, let’s use a highly relevant (and tragic) example from Silver’s work: the 9/11 terrorist attacks, specifically, the two planes that hit the World Trade Center.
Silver first asks us to examine the prior probability of someone flying a plan (as part of a terrorist attack) into the WTC. Assign that probability to the variable x (Silver estimates 0.005%). Next, we consider the occurrence of a new event, which is the first plane hitting a tower of the World Trade Center. Silver considers the probability of this plane hitting the World Trade Center, if terrorists are in fact attacking Manhattan skyscrapers (he assigns this probability, denoted by variable y, a value of 100%). Lastly, Silver asks us to consider the probability of a plane hitting the World Trade Center, if terrorists are actually not attacking Manhattan skyscrapers (i.e. an accident). He assigns this variable a probability z, of 0.008%. Silver then applies the formula for Bayes’ theorem, which returns a result of 38%. This means that when the first tower of the WTC was hit by a plane, there was a 38% chance that the WTC was under attack.
However, we aren’t done just yet. Next, we must revise our probabilities of a terrorist attack, to reflect the fact that a plane has already hit one of the towers of the World Trade Center. Here, x will reflect a revised probability of an attack on the World Trade Center, given that the first plane already hit a tower of the WTC. As stated above, that number is 38%. For y, the probability remains at 100%, while z remains at 0.008%. Applying Bayes’ theorem again, we find a result of 99.99%. This means that if a first plane already hit the World Trade Center, the probability that a second plane hitting the towers, indicates a terror attack, is 99.99%, a “near-certainty.”
As Silver explains, Bayes’ Theorem is so powerful, because it allows us to take account of uncertainty, that is “the limits of our knowledge”, despite our mental biases, and an overflow of information (much of it noise), in the era of Big Data. It forces us to take a stand, and either revise or strengthen this initial position, depending on additional information we receive. He touches upon the findings of medical researcher John Ioannidis, who published an influential paper arguing that the majority of research findings, published in scientific and medical journals, are in fact false. Ioannidis tells Silver that we can now measure “millions and millions of potentially interesting variables” (i.e. more data) but “most are not really contributing much to generating knowledge.” Silver frames Bayes’ Theorem as a means of cutting through the fog, and offering a more rigorous framework to our analysis of various questions, in a variety of fields.
Silver details how a successful sports bettor applies the principles of Bayes’ Theorem to better test and revise his beliefs about a team’s future performance, while computer chess programs (most notably IBM’s Deep Blue), use a Bayesian approach, to explore “the more promising lines of attack.” Google’s culture of product experimentation is grounded in Bayes’ work, as it allows them to quickly experiment with new ideas, with their users, make a hypothesis as to what will work, test predictions, and act on the results (while Google’s self-driving cars make use of Bayesian calculations). Professional poker, like sports betting, offers considerable upside to those who bring a Bayesian lens to anticipating another player’s hand and strategies (revising probabilities as the game progresses, and a player learns more about an opponent’s actual hand). Those who forecast climate change (which has proven to be a political minefield), also benefit from a Bayesian approach.
The thing is, each of us might think more effectively, through real-world application of Bayes’ theorem. In one of his book’s more humorous moments, Silver explains how to use Bayes’ Theorem, to figure out whether your partner is cheating on you, or there’s a different explanation for their unusual behavior.
Computer science professor Allen Downey offers a fascinating example of how he assessed the chances of whether a carbon monoxide alarm going off in his house, was in fact accurate, based on his knowledge of false alarms, factors in his house that might set off an alarm, and other environmental factors. As Downey puts it “Think like a Bayesian. As you get more information, update your probabilities, and change your decisions accordingly.”
Interviewed by Gizmodo, mathematician Spencer Greenberg explains how someone might gain a sense of whether a new workout regimen has in fact increase how much extra energy they have have, by looking at prior energy levels, how these have changed, and any other plausible explanations. As Greenberg sees it, Bayesian thinking in everyday life, isn’t really about calculating Bayes’ formula, so much as it is a means of testing and updating our beliefs, by considering the strength of various pieces of evidence, correcting “glitches” in our thinking, and avoiding “absolute certainty”, which often leads us astray.
We live in times of rapid change, and considerable uncertainty. The trajectory of the global economy remains unclear, while political/policy matters, across the globe, remain mired in doubt. Artificial intelligence is changing the world at a brisk pace, and will transform both the nature and availability of work. The lifespan of the largest and most profitable companies is growing shorter and shorter. Revolutionary technologies like CRISPR are transforming genetic engineering, at a breakneck pace. In 2016 and 2015 alone, more data was created, than in the previous 5000 years of humanity.
In this climate, it’s more important than ever, that we have some useful frameworks, to better understand the world around us. Bayes’ Theorem offers an effective way forward, one that we can all make use, in ways large and small, in our personal lives.
No comments:
Post a Comment