Better Answers to "What If" Scenarios

​TRADITIONAL decision making works well when the decision makers are presented with a predictable scenario. When all of the facts of a situation are clear, then a conclusion can be deduced from the readily available information. Risk and return can be easily calculated. But when the situation has not been seen before, and outcomes and details are fuzzy, decision making often turns into a game of “what if,” says Stephen Popper, who is a senior economist at the Rand Corporation. This is inductive reasoning, which is reasoning from the bottom up. And Popper says there are far fewer tools that can be used to quantify problems and help with decision making when dealing with such complicated uncertainties.

Researchers advocate an approach called Robust Decision Making (RDM). Rand recently published a look at work in this field; Popper was a coauthor of the work.

RDM aims to tell decision makers what sort of conditions can lead to success. According to the report: “Instead of asking ‘What will the future bring?’ it might be better to ask what steps could one take today to most assuredly shape the future to our liking?”

Rand researchers have taken those “what if” conversations and figured out ways to place them into computer models. RDM does not predict the future but rather starts with the various scenarios and plans under consideration. Instead of looking at a few cases to use as models of what might happen, RDM looks at thousands of possibilities, without knowing how probable each possibility is. “What ultimately we’re doing is we’re getting to a point where we’re characterizing uncertainties not in terms of probabilities—because again, if it’s a sufficiently uncertain situation, the probability itself is a guess—[but] in terms of what effect different types of uncertainty would have on different courses of action that you might pick,” says Popper.

“You make a big database of these model results, and then, you can use that to ask questions that help you make better decisions,” says study coauthor and senior Rand scientist Robert Lempert.

Popper and Lempert created a computer program that can do this through an outside company called Evolving Logic.

RDM is particularly helpful when there are numerous deep uncertainties with a complex question, and when there are many possible courses of action. One of the examples the authors used in the recent paper was the decision on whether federal terrorism insurance would cost or save money for taxpayers. Many insurance companies dropped commercial real estate terrorism insurance after 9-11. In response, the government passed the Terrorism Risk Insurance Act of 2002 (TRIA) to compensate insurance companies for losses in major terror attacks in exchange for the companies providing insurance for attacks. But the report states that when TRIA was up for reauthorization in 2007, policymakers found it was unclear whether TRIA would actually save or cost taxpayers money, particularly when considering how uncertain the risk of an attack was.

“With TRIA, more property owners are insured, and in the case of a small attack, their losses are paid by their insurers. Thus for small attacks, TRIA reduces the number of uninsured property owners that Congress might choose to compensate with taxpayer funds. However, with TRIA the government is also obligated to pay for losses (above some large deductible) in the case of a large terrorist attack,” says Lempert.

The researchers put together a model that “projected the costs to taxpayers, the insurance industry, and commercial property owners given various assumptions about the size and type of any future terrorist attacks, the behavior of the insurance industry and its customers, and the willingness of future Congresses to compensate property owners without insurance.” There was much uncertainty in the modeling; for example, the researchers had to guess that Congress would compensate anywhere between 0 and 100 percent of terrorism losses because they had no idea how a future Congress would act. The RDM analysis discovered the two dominant characteristics of the situation’s outcome were whether Congress would compensate property owners post-attack and the probability of a large-scale attack (much larger than 9-11, which is considered a “small attack”). According to the report and Lempert, the RDM analysis found more situations where taxpayer costs were lower with a reauthorized TRIA than without it. Says Lempert: “the net result is that TRIA saves the taxpayer money if any attack is small and if Congress would compensate the uninsured after that attack. TRIA costs the taxpayer money in the case of large attacks, in particular if the Congress would not compensate the uninsured after the attack. Whether on balance TRIA would save or cost the taxpayer money depends on how likely the large attacks are and Congress' tendency to compensate uninsured property owners after an attack.”

These findings were the exact opposite of what the Congressional Budget Office (CBO) and Department of Treasury had found using more traditional “predict-then-act” analyses. Because those agencies had no idea how to predict how Congress would act, they had chosen only the scenario where Congress would do nothing for the uninsured after an attack, which ended up with taxpayers losing money on TRIA. The report concludes, “had these agencies assumed compensation at almost any other level, they would have come to a different answer.” And given the real-life examples of ad hoc compensation, that’s probably a more realistic assumption. Congress decided to reauthorize the law despite the CBO estimates, but the RDM finding suggests that was likely the more fiscally responsible outcome.

RDM does require modeling and simulation capabilities and is more computer intensive than traditional decision making, but the other major challenge to the approach is that it’s so different from the decision making that most people are used to. Popper says it’s been easier to convince people of RDM’s worth in the years since 9-11, because there have been so many low-probability high-impact events.