Hate to Say We Told You So…

Scenarios trump forecasts (apologies)

A week ago, something happened that no one was prepared for – and I mean NO ONE.

Donald Trump was elected president.

When I say “no one,” I include Donald Trump and his inner circle. The proof of this was his visit to the White House last Friday, when either he or one of his advisors asked Obama or an Obama staffer how many of the West Wing staff would be staying on. The obvious answer, to anyone who knew the first thing about how the White House functions, was “none,” since every administration is responsible for entirely staffing the White House. (Almost entirely. Food service and cleaning staff normally stay on from administration to administration.)

As Election Day dawned, the New York Times‘ prognostication effort, The Upshot, had posted Hillary Clinton as an 85% probability to be elected president. Their predecessor at the Times and the head of Five Thirty Eight, Nate Silver, hedged his bets, giving Hillary a 71% chance of winning. The prize for hubris must go to the Princeton Election Consortium, which gave Hillary a greater than 99% chance of winning.

That their predicted candidate did not win does not necessarily mean they are charlatans. As I said in my previous blog piece, even tiny, less-than-1% chances come in every day. A 29% underdog winning, as Nate Silver has pointed out, should provoke something like a yawn.

But as one who had the real-time probability meter from the New York Times playing on my computer the evening of November 8, watching the needle swivel back and forth like a drunk navigating a heaving ship deck, starting off toward that 85% number far to the left (where it had essentially been for weeks), ending up horizontal to the right by early morning, I began to wonder what precise function this prognosticatory effort served. Even Nate Silver’s outfit was still showing Hillary as a favorite long after it had become obvious that there was something very big going on nationwide that was causing state after state to show far stronger results for Trump than their models anticipated.

And this highlights the problem with forecasting. By itself, it’s not a sufficient way to prepare for the future. In some cases, it’s not even necessary or useful. Getting a percentage estimate of the probability of the weather, in anything less certain than a 99% situation, is at best a nice-to-know. Preparing for multiple types of weather would be far more useful.

Yet still we prefer the prediction game. So we keep doing it, and we don’t prepare seriously, and as a result, we have little idea what effect that just-elected <1%-, or15%- or 29%-probability-become-100%-probability president might have on our economy, security, and society.

And that’s a shame. Because we really could have gotten to grips with a lot of the ramifications a while ago, if we had tried.

And it’s not just the Nates, Cohn and Silver, who are mesmerizing us with this forecasting parlor trick. Philip Tetlock has a bestseller out this year, co-authored with Dan Gardner, called Superforecasting: The Art and Science of Prediction. In it, he details his efforts over the decades to find out what makes some human beings good predictors. Tetlock gathers hundreds of people, most not experts in any particular field, and tracks their performance on predicting the percentage chances of certain pre-established questions turning out a certain way. Some of these people got very good at predicting these predetermined questions, in studies sponsored by the likes of the U.S. intelligence community.

But the ability of these people to improve their prognostication was limited, and depended upon the precise types of questions asked. One example was “Will two top Russian leaders trade jobs in the next ten years?” (The specific nature of this question was presumably the result of Vladimir Putin and Dmitri Medvedev swapping jobs after Putin’s term as Prime Minister ended, due to a statutory term limit.) Participants would predict the percentage chance of some unknown leading Russian nabobs trading desks, along with dozens more similarly precise questions. They were allowed to alter their percentage chance right up to the end of the prediction period, based on new data, or even just their own hunches – like that New York Times needle, swiveling day to day and moment to moment. The best of the best participants (who were almost always those who altered their predictions most often right up to the end) won the title “superforecaster” from the study.

But are these the kinds of questions you need answered in order to be prepared for the future? No, they are not. But as Tetlock says, they are the kinds of questions that can be assessed after the fact for accuracy. Either two Russian nabobs change desks, or they do not. Of course this question didn’t matter a damn, because Putin simply used his power to eliminate the term limits that had hampered him, so no Moscow bigwigs need ever pack up their staplers and inboxes again.

This is the first of two big problems I see with with prediction. Tetlock has made progress since his first major study, which resulted in the famous bumper-sticker (somewhat distorted-by-media) result of “Dart-Throwing Monkeys Are Better at Prediction Than Pundits.” The nature of that “progress,” however, has been a move, from “experts” baselessly asserting things with impunity and ridiculously bad accuracy, to large numbers of random people making somewhat less atrocious percentage guesses on questions that are so precise that they are almost doomed to irrelevance.

Towards the end of the book, Tetlock and Gardner address this issue. They admit that many people look at the questions being predicted and ask, “Are these questions of any importance at all? Are they the right ones? Are they at a sufficient level between specificity and generality that they are both concrete and also have relevance to our decisions?”

And their basic response, in my opinion, is wrong. They assert that ALL questions of importance can be decomposed from a high, general, difficult-to-judge level into individual sub-questions that are concrete and objective, and that percentage predictions for these sub-questions can be recomposed into a useful prediction of the whole.

My question to them is this: How could people have used this approach to predict the Trump phenomenon, say, two years ago? They could not. The big question “Could Donald Trump be elected president?” would never have occurred to anyone, so the decomposition of that question into sub-questions would never have been undertaken.

But after a complete failure of the prognosticators even to formulate the right question, we end up with a big question that actually IS concrete, AND which has impact on our lives. But even then, is prediction doing anything for us? No.

This is the second big problem with prediction. A President Trump was always a plausible outcome with high impact. That was all we needed to know. Yet the vast majority of us continued taking the temperature of this one concrete dualistic prediction, and not doing the hard thinking about what a President Trump (or for that matter, even a President Hillary Clinton) might mean for us, or what we ought to be doing about it.

But it looks like we are just going to get more and more prediction, and therefore more and more avoidable societal, business, and organizational failure. Tetlock, and now The Big Short and Moneyball author Michael Lewis, in his forthcoming The Undoing Project, are about to make Amos Tversky and Daniel Kahnemann household names. Kahnemann and Tversky were (the former still lives, the latter sadly died far too young) a duo whose monumental work on human judgment is reflected in Kahenmann’s book Thinking: Fast and Slow. Their main thesis is that there are essentially two modes of human decisionmaking: instinctive and fast, and thoughtful and slow. Tetlock’s Superforecasters excel at the latter, overcoming fallacies of judgment through rigorous methodical consistent self-questioning prognostication. Pundits are the worst form of “fast” thinkers, facilely applying their own prejudices and never re-examining them or “marking them to market” as new data comes in.

But Tetlock’s, and Kahnemann’s, and now Lewis’, “thinking” is not the only kind there is, and a perhaps uniquely American overemphasis on this kind of numerical judgment just keeps leaving us all surprised and dismayed over and over and over.

What was lacking in this election period was not prediction – it was IMAGINATION. Swiveling probability meters mesmerized us and captured our imaginations. They made us think that only this one question mattered. In all the breathless endless temperature-taking as to the latest polls, America stopped thinking about what the election of EITHER candidate might actually MEAN, beyond shuddering by his/her opponents and invocation of the End Times.

So, here are some quick lessons from a week ago.

1. The best prediction is based on data. But data is ALWAYS about the past. Past data will predict the future – until it suddenly doesn’t.

2. Prediction kills thought. There is a groove in the human mind into which prophecy – pronouncements seeming to promise certainty – fits perfectly. Once the mind is told that something is “probably going to happen,” its default mode is to assume it WILL happen.

3. No outcome assigned a low probability will ever receive adequate thought – especially when that outcome is difficult to think about, either because it presents imponderable challenges or because it poses a serious threat.

4. Probabilities are a goofy thing, when you think about them. In the end they all go to 0 or 1, yet we think “29%” means something – and usually we think it means “not gonna happen.” (It CAN mean something in financial markets, where risk can be monetized and reduced or increased. It cannot, however, be useful in politics for ordinary citizens, who cannot lay off risk in that way.)

5. Probabilities can only apply to simple, objective, “yes/no”-type situations. In January 2015, there was a great hunger on the part of the media for a Bush-Clinton contest, because they were “the favorites” and it was easy to think about this dualistic matchup between familiar, well-understood actors. No one used their imagination to think of the broader universe of possibilities, because they could not be formulated into a Tetlock-style simple yes-no question.

6. We all would have been better off not getting fixated on prediction markets, Nate Silver, etc. Politics is like the weather. Especially if humans are involved, it’s much better to bound the universe of all plausible outcomes and examine EVERY ONE for its possible impacts on our worlds of work, lives, society, national security, etc., regardless of “probability.” (Because all of those probabilities but one, in this case, was going to 0, and the remaining one to 100%.) We don’t watch a weather report that says there is 29% chance of rain (much less a hurricane) and decide we don’t have to obtain foul weather gear. But instead of buying foul-weather gear, we all simply watched the weather reports on all the stations with mouths agape, assuming that numbers would magically prevent or guarantee certain outcomes. And so when the storm hit, we were all exposed – even the “winners.”

7. The way we choose leaders encourages macho single-point forecasts and discourages holistic thought. Smart humans in positions of urgent responsibility often got where they are by thinking fast and leaping to a simplistic concrete formulation of whatever urgent issue they face. They don’t do what Edward de Bono called “lateral thinking,” the difficult up-front holistic stepping-back preliminary attempt to understand the entire strategic uncertainty space. It’s too airy-fairy and non-quantitative. So they fail. They drill dry holes, and instead of stepping back and thinking, “Hey, there’s a whole universe of potential places to drill, maybe I should sample a large number of areas,” they decide they have to drill deeper in the same place. They stick with that simplistic concrete formulation of the issue, instead of stepping back and saying, “Hey, maybe I should think up MORE simplistic concrete formulations, and test them all.” Coming up with those alternative formulations is the hard part for them, and they don’t want to abandon one once one occurs to them in a blinding flash of insight. And they probably have become identified with it within their organization, so they are emotionally invested in seeing it proven to be true.  It takes courage to admit that one’s own model might be wrong and to invite other models. But it’s the only real way forward.

Nate Silver and all the best of the prognosticators admit this last point and try to improve constantly. But as long as they are doing prediction, they are not really doing the right thing; they are doing an amusing but essentially unserious thing marginally better.

The right way to approach this election, and all elections, indeed all situations of uncertainty, is to start by by forcing ourselves to use that lateral thinking, to imagine things that never have been (or at least that have not been thought of before). The particular way we do that with clients is through scenario planning. 

Scenarios, as we do them, don’t muck around with assigning probabilities at all. They simply ask what affects your life and world of work, develop a set of alternative future “worlds” that are spread as far out as is plausible along the main dimensions that define what matters to you (and then maybe a bit farther out), and then reason back from those futures to figure out how they COULD happen. This forces people to use their imaginations to think of combinations of circumstances that no (past) data-based model will ever entertain, because they haven’t happened before.

In the early summer, we at FSG held a three-day offsite to begin construction of a set of platform scenarios as told from a decade or so hence, with a particular focus on consumer attitudes, lifestyles, and spending habits across those futures – but easily applicable to other spheres as well.

Work continues on these four scenarios, which will be available for client customization and strategy workshop use in the very near term.  They will, of course, contain detailed and varying outcomes for this new and unexpected administration.

But they will NOT contain a single point forecast for that administration, or any probabilities assigned to any aspect of them. All important variables must be systematically varied and then their implications derived and taken seriously, so the client is prepared and has stress-tested the future before it arrives.

As prognosticators – and more broadly America, whether pro-Trump, pro-Clinton, or otherwise – certainly did not this time around.

Go out and buy the Lewis and Tetlock books, and enjoy the Nates. They are interesting and entertaining. But just remember, they are not the answer.