Book Review: Superforecasting: The Art and Science of Prediction

Original Biopolitics and Bionews post: September 2, 2016

 In Superforecasting: The Art & Science of Prediction, social scientist Philip E. Tetlock and journalist Dan Gardner (Crown Publishers, September 2015) observe that at its heart, political policy is usually about predicting the future. The exercise boils down to finding and implementing policies that will do best for the public interest (general welfare or common good), regardless of how one defines the concept. What most accurately describes the essence of intelligent, objective, public service-oriented politics? Is it primarily an honest competition among the dominant ideologies of our times, defense of one’s social identity, a self-interested quest for money, influence or power, or some combination thereof? Does it boil down to understanding the biological functioning of the human mind and how it sees and thinks about the world? Is it something else entirely? 

 Subject to caveats, Superforecasting comes down on the side of getting brain biology or cognition right. Everything else is subordinate. Superforecasting describes Tetlock's research into asking what factors, if any, can be identified that contribute to a person’s ability to predict the future. Tetlock asks how well intellectually engaged but otherwise non-professional people can do. The performance of volunteers is compared against experts, including professional national security analysts with access to classified information.

  The conscious-unconscious balance: What Tetlock and his team found was that interplay between dominant, unconscious, fact- and common sense-distorting intuitive human cognitive thinking (“System 1” or the “elephant” as described before) and our far less-powerful but conscious, rational thinking (“System 2” or the “rider”) was a key factor in how well people predicted future events. The imbalance of power or bandwidth between conscious thinking and unconsciousness thinking is estimated to be at least 100,000-fold in favor of unconsciousness. The trick to optimal performance appears to be found in people who are able to strike a balance between the two modes of thinking, with the conscious mind constantly self-analyzing to reduce fact distortions and logic biases or flaws that the unconscious mind constantly generates. Tetlock observes that a “defining feature of intuitive judgment is its insensitivity to the quality of the evidence on which the judgment is based. It has to be that way. System 1 can only do its job of delivering strong conclusions at lightning speed if it never pauses to wonder whether the evidence at hand is flawed or inadequate, or if there is better evidence elsewhere. . . . . we are creative confabulators hardwired to invent stories that impose coherence on the world.” Coherence can arise even when there's insufficient information. In essence, the human mind evolved an ‘allergy’ to ambiguity, contradictions and concepts that are threatening to personal morals, identity and/or self-interest. To deal with that, we rapidly and unconsciously rationalize those uncomfortable things by denying or distorting them.

It turns out, that with some training and the right mind set, a few people, “superforecasters”, routinely trounce professional experts at predicting future events. Based on a 4-year study, Tetlock’s “Good Judgment Project”, funded by the DoD’s Intelligence Advanced Research Projects Agency, about 2,800 volunteers made over a million predictions on topics that ranged from potential conflicts between countries to currency and commodity, e.g., oil price fluctuations. The predictions had to be precise enough to be analyzed and scored. About 1% of the 2,800 volunteers turned out to be superforecasters who beat national security analysts by about 30% at the end of the first year. One even beat commodities futures markets by 40%.

The superforecaster volunteers did whatever they could to get information, but they nonetheless beat professional analysts who were backed by computers and programmers, spies, spy satellites, drones, informants, databases, newspapers, books and whatever else that professionals with security clearances have access to. As Tetlock put it, “. . . . these superforecasters are amateurs forecasting global events in their spare time with whatever information they can dig up. Yet they somehow managed to set the performance bar high enough that even the professionals have struggled to get over it, let alone clear it with enough room to justify their offices, salaries and pensions.”

  What makes superforecasters so good?: The top 1-2% of volunteers were analyzed for personal traits. In general, superforecasters tended to be people who were open-minded about collecting information, their world view and opposing opinions. They were also able to step outside of themselves and look at problems from an “outside view.” To do that they searched out and integrated other opinions into their own thinking. Those traits go counter to the standard human tendency to seek out information that confirms what we already know or want to believe. That bias is called confirmation bias.

The open minded trait also tended to reduce unconscious System 1 distortion of problems and potential outcomes by other unconscious cognitive biases such as the powerful but subtle “what you see is all there is” bias, hindsight bias and scope insensitivity, i.e., not giving proper weight to the scope of a problem. Superforecasters tended to break complex questions down into component parts so that relevant factors could be considered separately. That tends to reduce unconscious bias-induced fact and logic distortions. In general, superforecaster susceptibility to unconscious biases was lower than for other volunteers in the GJP. That appeared to be due mostly to their capacity to use conscious (System 2) thinking to recognize and then reduce unconscious (System 1) biases.

Analysis revealed that superforecasters tended to share 15 traits including (i) cautiousness based on an innate knowledge that little or nothing was certain, (ii) being reflective, i.e., introspective and self-critical, (iii) being comfortable with numbers and probabilities, (iv) being pragmatic and not wedded to any particular agenda or ideology, and, most importantly, (v) intelligence, and (vi) being comfortable with (a) updating personal beliefs or opinions and (b) belief in self-improvement (having a growth mindset). Tetlock refers to that mind set as being in “perpeutal beta” mode. 

 Unlike political ideologues, superforecasters tended to be pragmatic, i.e., they generally did not try to “squeeze complex problems into the preferred cause-effect templates [or treat] what did not fit as irrelevant distractions.” Compare that with politicians who promise to govern as proud progressives or patriotic conservatives and the voters who respond to those appeals. What the best forecasters knew about a topic and their political ideology was less important than how they thought about problems, gathered information and then updated thinking and changed their minds based on new information.

The best engaged in an endless process of information and perspective gathering, weighing information relevance and questioning and updating their own judgments when it made sense, i.e., they were in “perpetual beta” mode. Doing that required effort and discipline. Political ideological rigor such as conservatism or liberalism was generally detrimental. Regarding common superforecaster traits, Tetlock observed that “a brilliant puzzle solver may have the raw material for forecasting, but if he also doesn’t have an appetite for questioning basic, emotionally-charged beliefs he will often be at a disadvantage relative to a less intelligent person who has a greater capacity for self-critical thinking.”

Superforecasters have a real capacity for self-critical thinking. Political, economic and religious ideology is mostly beside the point. Instead, they are actively open-minded, e.g., “beliefs are hypotheses to be tested, not treasures to be protected.” Tetlock asserts that politicians and partisan pundits opining on all sorts of things routinely fall prey to (i) not checking their assumptions against reality, (ii) making predictions that can’t be measured for success or failure, and/or (iii) knowingly lying to advance their agendas. Politicians, partisan pundits and experts are usually wrong because of their blinding ideological rigidity and/or self- or group-interest and the intellectual dishonesty that accompanies those mind sets. Given the nature of political rhetoric that dominates the two-party system and the biology of human cognition, it is reasonable to argue that most of what is said or written about politics is more spin (meaningless rhetoric or lies-deceit) than not.

  Questions: Is Tetlock’s finding of superforecasters real, and if so, does that point to a meaningful (teachable) human potential to at least partially rationalize politics for individuals, groups, societies or nations? Why or why not? After the WMD disaster in Iraq, US Intelligence agencies funded Tetlock’s research and then adopted his techniques to assess intelligence analysts: Will “better” intelligence assessments reduce the frequency or magnitude of mistakes that rigid liberal or conservative politicians tend to make? Would it be any different if the politicians calling the shots are centrists, moderates, socialists, libertarians, anarchists or cognitive science-based, pragmatists focused on problem solving with little or no regard to political ideology or economic theory?