Improving the Contribution of Climate Model Information to Decision Making: the Value and Demands of Robust Decision Frameworks

Home , Climate change scenario

Advanced Review Improving the contribution of climate model information to decision making: the value and demands of robust decision frameworks

Christopher P. Weaver,1∗ Robert J. Lempert,2 Casey Brown,3 John A. Hall,4 David Revell5 and Daniel Sarewitz6

In this paper, we review the need for, use of, and demands on climate modeling to support so-called ‘robust’ decision frameworks, in the context of improving the contribution of climate information to effective decision making. Such frameworks seek to identify policy vulnerabilities under deep uncertainty about the future and propose strategies for minimizing regret in the event of broken assumptions. We argue that currently there is a severe underutilization of climate models as tools for supporting decision making, and that this is slowing progress in developing informed adaptation and mitigation responses to climate change. This underutilization stems from two root causes, about which there is a growing body of literature: one, a widespread, but limiting, conception that the usefulness of climate models in planning begins and ends with regional-scale predictions of multidecadal climate change; two, the general failure so far to incorporate learning from the decision and social sciences into climate-related decision support in key sectors. We further argue that addressing these root causes will require expanding the conception of climate models; not simply as prediction machines within ‘predict-then-act’ decision frameworks, but as scenario generators, sources of insight into complex system behavior, and aids to critical thinking within robust decision frameworks. Such a shift, however, would have implications for how users perceive and use information from climate models and, ultimately, the types of information they will demand from these models—and thus for the types of simulations and numerical experiments that will have the most value for informing decision making. © 2012 John Wiley & Sons, Ltd.

How to cite this article: WIREs Clim Change 2013, 4:39–60. doi: 10.1002/wcc.202

3Department of Civil and Environmental Engineering, University of Massachusetts, Amherst, MA, USA This article is a U.S. Government work, and as such, is in the public domain in the United States of America. 4Strategic Environmental Research and Development Pro- ∗ gram/Environmental Security Technology Certiﬁcation Program, Correspondence to: [email protected] U.S. Department of Defense, Alexandria, VA, USA 1National Center for Environmental Assessment, Ofﬁce of 5Environmental Science Associates—Environmental Hydrology, Research and Development, U.S. Environmental Protection Agency, San Francisco, CA, USA Washington, DC, USA 6Consortium for Science, Policy, and Outcomes, Arizona State 2The RAND Corporation, Santa Monica, CA, USA University, Tempe, AZ, USA

INTRODUCTION this broader perspective on the value of climate models for supporting decision making, have the potential ssessment of environmental and societal risks to simultaneously and effectively address the challenge Afrom climate change, and the design and above. evaluation of strategies to build readiness and Such a shift, however, would have implications resilience in the face of these risks, is a generational for the ways in which users perceive and use scientific challenge. How should society steer climate information from climate models and, ultimately, the research to best meet this challenge? How should types of information they will demand from these we attempt to use insights and information from models—and thus for the types of simulations and climate science to inform decision making about numerical experiments that will have the most value responses to climate change? These questions are for societal applications. In this context, we review the particularly relevant now, as national governments use of, and demands on, climate modeling to support and international scientific bodies are discussing these frameworks, in the context of improving uptake how best to organize research efforts and to of climate information into decision making. deliver knowledge, data, tools, and advice—climate services—to a potentially huge universe of clients.1–3 There has been some self-organization around LONG-TERM CLIMATE PREDICTION IS two strategic framings of the problem. The first DIFFICULT ... (dominant) paradigm assumes the need to improve our ability to predict, using physically based computer As noted by various commentators,4,5 one of models, multidecadal, regional climate change as the most oft-repeated statements in the climate a prerequisite for effective planning; the second impacts literature goes something like this: ‘Accurate, to instead improve our understanding of regional high-resolution predictions of future climate are and sectoral climate-related vulnerabilities, and the a prerequisite for developing effective responses cognitive, social, and institutional contexts within to climate change impacts at regional scales.’ A which these will be managed, in light of the deep number of recent papers and editorials have called uncertainties associated with climate change and its for increased efforts to deliver these improved possible impacts. These two conversations have to predictions.6–12 At the same time, everyone seems date largely occurred in parallel. to broadly agree that we are a long way from fulfilling The purpose of this paper is to discuss potential this need. For example: synergies between climate modeling for decision support as currently practiced, and the more bottom- The climate science community now faces a major up approaches to climate change vulnerability and new challenge of providing society with reliable impacts assessment, as a pathway to more effectively regional climate predictions. Adapting to climate informing climate-related decisions. We argue that change while pursuing sustainable development will require accurate and reliable predictions of changes in currently there is a severe underutilization of climate regional weather systems, especially extremes ...Yet, models as tools to support decision making. This current climate models have serious limitations in underutilization stems from a double challenge noted simulating regional weather variations and therefore by a growing number of scholars: a widespread, but in generating the requisite information about regional limiting, conception that the potential usefulness of changes with a level of confidence required by climate models in planning begins and ends with ever- society.12 finer regional-scale predictions of multidecadal climate 8–10,13,14 change; and the general failure so far to incorporate Andsoon(seealsoRefs and many learning from the decision and social sciences into others) (Box 1). climate-related decision support in key sectors such as As the number of legal and political mandates water resources, coastal protection, agriculture, and for incorporating climate change information into public health. decision making increase, the demands on climate In this paper, we argue that alleviating this prob- science and modeling to deliver so-called ‘actionable’ lem will require expanding the conception of climate information in direct support of planning will also models, not simply as prediction machines within continue to increase—as will scrutiny of their ability ‘predict-then-act’ decision frameworks, but as aids to do so. to exploratory modeling—for scenario generation, It is well understood that current limits on sci- insight into complex system behavior, and supports entific understanding and computing power result for critical thinking—within so-called ‘robust’ deci- in deficiencies in simulating impact-relevant cli- sion frameworks. Such frameworks, combined with mate system elements—clouds, precipitation, winds,

BOX 1 employ them as shorthand for a continuum of decreasingly confident statements about the NOTES ON LANGUAGE: PREDICTION, future, from prediction, through projection, to PROJECTION, AND SCENARIO scenario, using them to manage expectations created by the word prediction.16–22 In addition, AccordingtotheIPCC15: ‘A climate prediction the IPCC definition of projection above still or climate forecast is the result of an attempt implies that at least the climate portion is to produce an estimate of the actual evolution deterministic, and it is only the future emissions of the climate in the future, for example, at pathway that is non-deterministic and thus must seasonal, interannual, or long-term time scales. be captured via a set of scenarios. Since the future evolution of the climate system Rather than using projection and scenario may be highly sensitive to initial conditions, such as if they are simply a kind of less-accurate predictions are usually probabilistic in nature.’ prediction, it would seem to make more sense Here, the term ‘prediction’ refers to attempts to to adopt functional definitions that relate to answer questions posed as ‘What will happen (in how the different objects are actually used the future)?’ This definition also suggests that to support decision making. How would one various factors will prevent us from achieving envision using a climate prediction to support an 100% accuracy, and that an important part actual planning decision, and how would that of any climate prediction is our degree of use differ from how one would use a climate belief that it will come true. The IPCC also scenario? Attention to how best to judge the distinguishes prediction from two other common ‘success’ of a scenario23 is useful here. One terms, ‘projection’ and ‘scenario’: potential casualty of such a functional approach Climate Projection: ‘A projection of the to these definitions is the word ‘projection’, response of the climate system to emis- which seems to be uneasily perched between sion or concentration scenarios of green- these two different uses. house gases and aerosols, or radiative forcing scenarios, often based on simulations by climate models. Climate projections are distin- guished from climate predictions in order to the diurnal cycle, and the atmospheric moisture 24–27 emphasize that climate projections depend on balance —as well as major weather patterns 28 the emission/concentration/radiative forcing sce- such as the Indian Monsoon, the mid-latitude nario used, which are based on assumptions storm tracks,29 and the El Nino-Southern Oscillation concerning, for example, future socioeconomic (ENSO).30 and technological developments that may or Spatio-temporal resolution is clearly part of the may not be realized and are therefore subject story. High resolution is crucial for representing to substantial uncertainty’. key processes with realism, but is also extremely Climate Scenario: ‘A plausible and often computationally intensive. And the most relevant simplified representation of the future climate, spatial and temporal scales for resource management based on an internally consistent set of or ecosystem processes are those for which the climatological relationships that has been obstacles to prediction are the greatest.31 Various constructed for explicit use in investigating downscaling techniques, while important for many the potential consequences of anthropogenic applications, offer no free way out.32,33 Other climate change, often serving as input to impact dimensions that compete for computing resources models. Climate projections often serve as the include model complexity (biogeochemistry, aerosols, raw material for constructing climate scenarios, land-use change, ice sheet dynamics, etc.), integration but climate scenarios usually require additional length, ensemble size, and specification of initial information such as about the observed current climate. A climate change scenario is the conditions. difference between a climate scenario and the Progress in research and technology over time current climate’. will presumably improve this situation somewhat, but All three terms are entrenched in the the challenge is great: climate literature, but with some semantic confusion surrounding their use. A number of The past 40 years of climate simulation have made it authors suggest that many, instead of using apparent that no shortcut is possible; every process can them to describe distinct objects, essentially and ultimately does affect climate and its variability and change. It is not possible to ignore some

component or some aspects without paying the price large literature about how decisions are actually made of a gross loss of realism.34 explains the factors that determine whether and how scientific information becomes part of that process. Other issues make long-term climate system This literature makes clear that scientific knowledge, 35 prediction an even more difficult problem. Once including products such as predictions, is only one we start considering timescales beyond a decade or part of a much broader system of decision-making two, the changing human influence on climate cannot practice.4,19,65 be ignored. Societies will likely react in complex ways This starts with the acknowledgement that infor- to climate change, ways that will alter the trajectory of mation may be scientifically relevant without being subsequent changes (e.g., in greenhouse gas emissions, decision-relevant. Scientific information can be effec- land use, and geoengineering), with further feedbacks tive in influencing decision making, but only if this on human behavior, and these changes will likely be 36–38 information is perceived as ‘credible, salient, and legit- unpredictable. imate’: i.e., scientifically and technically accurate in its Furthermore, detailed predictive skill beyond a 39–41 evidence and arguments, but also relevant to the needs few seasons is not yet a reality. Because of the long of the decision makers and having been produced in a timescales involved, it is impractical to directly assess way that is unbiased and respectful of their divergent the skill of multidecadal climate forecasts (in contrast perspectives and values.66,67 These characteristics tend with short-term weather forecasting) leaving us to rely 42,43 to result only from a sustained process of close inter- on indirect methods. However, we do not yet have action among knowledge producers and users, often the scientific understanding to convincingly measure facilitated by so-called ‘boundary organizations’.68 climate model skill this way. Because future climate Because different actors perceive the usefulness of will be a not-yet-experienced state of the Earth system, scientific information differently,19 decision support it may ultimately be impossible to aprioricalibrate a characterized by one-way communication and a focus given climate model in such a way as to ensure that it on ‘products’ (e.g., reports, predictions) as opposed to will produce a meaningful long-term forecast.43–49 We ‘process’ (of communication, mediation, translation, lack a generally accepted set of metrics for evaluating feedback, and trust building) has been demonstrated climate model performance, either in general or in a to be ineffective.66,67,69 sector-specific50 way. The aspects of observed climate This highlights long-understood truths of the that must be simulated correctly to ensure reliable decision sciences that no simple relationship exists future forecasts are not known; agreement across a between more information and better decisions, infor- collection of models does not provide a rigorous basis mation does not necessarily precede decision making, for assessing how much we should believe a future important basic questions are often raised in the prediction either.14,42,43,51–56 (It is important to note course of making decisions, and ‘reducing uncer- that the obstacles to skillful prediction on shorter tainty’ will likely have different meanings in the timescales—say, a decade—are likely not nearly as context of scientific understanding versus that of deci- great, as discussed extensively elsewhere37,57 but these sion outcomes.70–73 Scientific assessments to support efforts are still in their infancy.) decision making should be viewed, fundamentally, as Finally, despite strenuous efforts, it is clear that social processes embedded within particular institu- current models and simulations capture only a subset tional structures.69 With climate predictions in par- of our scientific uncertainty about how the climate sys- ticular, a suite of technical, cognitive, institutional, tem works (and how to encapsulate our understanding and structural factors determine the capacity of deci- in a model).22,37,58–63 So far, the more model variants sion makers to make use of them. Lemos and Rood19 we create, the more complexity we add, and the more write of the ‘uncertainty fallacy,’ in their words ‘a simulations we carry out, the greater the range of belief that the systematic reduction of uncertainty in scientifically justifiable future climates (and the corre- climate projections is required in order for the pro- sponding range of societal outcomes64) that result. jections to be used by decision makers’. Experience in attempting to use forecasts of seasonal-to-interannual ...AND SO IS CLIMATE-RELATED climate variability (e.g., as driven by ENSO cycles) DECISION SUPPORT for planning in sectors such as water resources and agriculture clearly illustrates these issues. There is Meeting the scientific and technical challenges of widespread agreement that such forecast informa- climate prediction would of course not be the only pre- tion, while having the potential to support improved requisite for achieving better climate-related decision decision making, is almost universally underutilized making. It may not even be the most difficult. A in these sectors.19,74–82

The use of scientific information for building readiness to much longer-term climate change will Demand: Can user benefit from research? certainly require at least a similar level of effort, and at minimum face similar obstacles, as discussed in a number of recent publications.65,83–87 That climate- YES NO related decisions are made at all organizational levels, from individuals to international bodies, Research agendas Research agendas and precluding the development of ‘generic’ decision may be user needs poorly support, compounds the challenge. Combine this inappropriate matched; users may be with the previously mentioned difficulties in modeling NO the climate system over these timescales (and thus disenfranchized. the poor prospects for forecasts that will likely not have even the credibility of the seasonal forecasts currently available) and we are faced with a significant Empowered users Unsophisticated or challenge. Is there a path forward? taking advantage of marginalized users,

HARNESSING CLIMATE MODELING well-deployed institutional TO THE MACHINERY OF EFFECTIVE YES research constraints, or other

DECISION SUPPORT Study: Is relevant information produced? capabilities. obstacles prevent What are the implications of the above twofold challenge for supporting decision making about information use. adapting to long-term climate change? Social sciences research clearly shows that simply overcoming the | extremely difficult scientific and technical obstacles FIGURE 1 The missed opportunity matrix. (Reprinted with permission from Ref. 73 Copyright 2007 Elsevier Ltd.) to climate prediction will not suffice. Climate-related decision support is hard, not only because long-term climate prediction is hard, but also because creating for understanding the value of climate information decision-relevant processes for the production and in decision contexts. Furthermore, in light of the uptake of climate information is hard. We suggest challenges discussed in the previous two sections, that a kind of cognitive dissonance infuses discussions we suggest that reconciliation of supply and demand of the social value of climate modeling. On one could usefully occur around a class of decision hand, we recognize that the climate system is almost frameworks that rely on bottom-up approaches to unimaginably complex and the challenges of modeling assessing risks and seek to develop robust actions it are enormous. On the other hand, we tell ourselves that hedge effectively against these risks over the that we urgently need the predictions that only models many significant uncertainties in the climate change can supply. We wrestle with the question: ‘If climate problem. science is not delivering actionable predictions, then what good is it?’ Sarewitz and Pielke73 have introduced the idea Decision-Making Paradigms of ‘supply of and demand for science’—the importance The decision sciences recognize multiple paradigms of simultaneously reconciling the capabilities and for choosing which actions to take. Here we aspirations of both knowledge producers and contrast what is sometimes termed a ‘predict-then-act’ knowledge users. In the terminology of their ‘Missed approach—i.e., where the best available prediction of Opportunity Matrix’ (Figure 1) this reconciliation system behavior drives decision making at a given results in ‘Empowered users taking advantage of well- moment in time—with approaches rooted in different deployed research capabilities’. underlying principles. One major factor in the match of supply and In the context of the challenges we have been demand is the decision framework within which the discussing—of climate prediction and of climate- scientific information must operate, and by which related decision support—we argue that prediction- its relevance is judged. For the case of climate based frameworks place unrealistic demands on modeling to support decision making, we argue that climate science and modeling and artificially limit supply and demand are out of balance because of their use for supporting real decisions. Such thinking the predominance of a prediction-based paradigm reinforces a ‘decision support as information product’

TABLE 1 Two Paradigms surprises that may be highly relevant to the policy 90 Paradigm 1: ‘Predict-then-Act’ question under consideration, thereby resulting in 10,94 Figure out your best-guess future and design the best policy decisions that may be insufficiently protective. you can for that future They also exacerbate the difficulty of getting diverse Conceptual framework: Maximize expected utility stakeholders to agree in advance on the predictions (and probabilities) that are the prerequisites for the Question: ‘What is most likely to happen?’ decision, while incentivizing particular stakeholders Paradigm 2: ‘Seek Robust Solutions’ to focus on those predictions and uncertainties most Identify greatest vulnerabilities across full range of futures consistent with their interests and world views. and identify the suite of policies that perform reasonably However, there exist other decision paradigms well across this range (Table 1) that may be better able to embrace the Conceptual framework: Minimize regret uncertainties of climate change (and better align with Question: ‘How does my system work and when might my the messy realities of decision support). One particular policies fail?’ class of such approaches, with variants such as ‘Robust Decision Making (RDM),’ ‘Decision Scaling (DS),’ ‘Assess Risk of Policy,’ ‘Info-gap,’ and ‘Context-First,’ frame, contrary to the ‘decision support as process’ provides an opportunity to overcome the deficiencies perspective that more accurately describes real-world of prediction-based frameworks in deciding how decision making. In addition, if we are committed to respond to long-term climate change.90,92,95–107 to prediction-based frameworks, not only are we These share a number of core ideas, beginning with stuck having to deliver climate predictions with well- defining a proposed policy or policies, identifying characterized probability distributions, but ultimately vulnerabilities of these policies under multiple views also a set of much more difficult ecological and of the future, seeking in particular those futures socioeconomic predictions,8,85–89 including the deeply under which the policies fail to meet their goals, uncertain behavior of future decision makers,90 to identifying potential responses to these vulnerabilities, provide meaningful predictions of impacts. and organizing scenarios to help decision makers The predict-than-act paradigm (Table 1) is determine the circumstances under which they would closely linked with the classical decision-theoretic adopt these responses. Interest in applying these kinds concept of maximizing expected utility: of approaches has increased recently, due to growing Most traditional decision-analytic methods for risk recognition of factors such as the importance of and decision analysis91 are designed to identify extreme and unanticipated events and the need to optimal strategies contingent on a characterization develop decision support processes that can engage of uncertainty that obeys the axioms of probability multiple stakeholders with diverse perspectives.108 theory. These approaches begin with a single best- Before discussing robust decision frameworks estimate description of the relevant system, consisting and the demands they place on (and opportunities they of a system model that generates outcomes of interest create for) climate science and modeling, we briefly given a choice of strategy and a single set of probability review a number of key concepts from the domain of distributions over the model’s input parameters to ‘exploratory modeling’, to provide a broader context characterize the uncertainties.92 for the integration of climate modeling and decision Although decision approaches that maximize analysis. expected utility clearly work best under many conditions, they tend to be difficult to apply when the strategy they suggest is highly sensitive to assump- Exploratory Modeling tions about which future is most likely.22,93 Because Argument over the appropriate uses of models has they require a quantitative characterization of the a long history. Many authors have emphasized that problem’s uncertainty as a prior input to the decision, models’ usefulness rests in their ability to help us think these approaches lose value as they become increas- about a system—as heuristic tools to help us under- ingly sensitive to this characterization; in offering no stand what we can actually observe or estimate,109 systematic way to choose among several ‘best’ strate- or as a way to challenge existing formulations and gies (each dependent on different, but all plausible, intuitions.70,110 Models allow one to keep track of choices of priors and assumptions). These issues cre- more variables than is possible in one’s head, as ate incentives for analysts to focus on the parts of well as processes like feedbacks, stocks and flows, the problem for which the uncertainties are well char- time delays, and nonlinearities that are non-intuitive acterized and ignore deep uncertainties and possible for most people.111–113 In the history of science, the

44 © 2012 John Wiley & Sons, Ltd. Volume 4, January/February 2013 WIREs Climate Change The value and demands of robust decision frameworks twin ideas of explaining the known and predicting BOX 2 the unknown have evolved and interacted, alternat- ing dominance.114 To be useful, a simulation does not have to be a prediction, or testable as such, TAXONOMY OF EXPLORATORY but simply to provide a useful description of the MODELING ARGUMENTS115 world. 115 Hypothesis Generation and Existence Proofs: Bankes, with criticisms and insights that Show that an ensemble of model simulations remain highly relevant for our discussion, articu- has a member exhibiting some property of the lated such arguments as being rooted in the tension world and/or suggest a particular class of models between ‘consolidative’ and ‘exploratory’ modeling. as a possible explanation of that property. A Traditionally, models (computer simulations or sta- plausible model displaying unexpected behavior tistical estimations) have been seen as consolidative, is useful in that it can reorient an analysis meaning that they bring all known facts together into to confront an expanded range of possibility. a single package which, once validated, can be used as This argument encompasses identification of a surrogate for the real world (e.g., prediction). This is bounding cases, surprises, and unknown worst the situation in which much of climate modeling finds cases. itself today. Reasoning from Special Cases: Computer However, in many cases models cannot be search can discover special cases that suggest readily validated, due to missing data, inadequate arguments relevant to a decision. For example, theory, or a future with elements that are unavoidably previously unexpected disaster scenarios can unpredictable. In contrast to consolidative modeling, suggest precautionary measures; unexpected exploratory modeling aspires to construct valid opportunities can suggest policies to exploit arguments by carrying out experiments with these opportunities; a fortiori arguments can ensembles of models that capture sets of assumptions strengthen policy choices; extreme cases where about the world, and to explore the implications of the uncertainties are all one-sided can simplify choice. This argument encompasses the identifi- these varying assumptions (Box 2). cation of hedging strategies to avoid worst-case Bankes emphasizes that, in exploratory model- outcomes when risk aversion is prudent. ing, models are used not to generate predictions or Assessing Properties of the Entire Ensem- explicit answers to policy makers’ initial questions ble: If all cases examined have property X, assert but instead to generate new information helpful in X true for the entire ensemble (for example deci- formulating informed decisions. And he reminds us sion A dominates decision B everywhere in the that even simple models produce surprises: parameter space of the problem). This argument encompasses the search for cases that distinguish By throwing light on treacherous assumptions between alternative decision options using sat- or revealing unrealized implications of existing isficing criteria such as robustness over a wide information, computer modeling can perform an range of views of the future. important service. When used for exploratory modeling, the computer functions as a prosthesis for the intellect, supporting the discovery of implications futures as possible, generated using models (perhaps of a priori knowledge, novel explanations of known combined with other sources of information). facts, or unrealized properties of conjectures.115

The three main categories in the above taxonomy suggest different roles for exploratory modeling at Exploratory Modeling for Robust Decision different stages of decision analysis, often requiring Frameworks different models or systems of linked models at each How do robust decision frameworks connect model- stage. For example, we might (1) use exploratory ing to decision making in a way that addresses the modeling to identify the possibility of important principal challenges we have recognized here: i.e., special cases in system behavior, such as extremes, the current and foreseeable limits on climate predic- thresholds, and surprises. We might then (2) use this tion and the need to incorporate scientific knowledge information to search for policy options capable of about climate change into a highly contextualized and satisfactory performance in the face of such special participatory process of decision support? cases, thereby expanding the space of choices for the First, modeling in robust decision frameworks policy-maker. And, finally, we might (3) evaluate these is explicitly exploratory rather than consolidative, fit- (and other proposed) options over as wide a range of ting neatly into the threefold exploratory modeling

Volume 4, January/February 2013 © 2012 John Wiley & Sons, Ltd. 45 Advanced Review wires.wiley.com/climatechange taxonomy of describing the range of possible system volumes of information from sophisticated models behavior, identifying policy options capable of rel- into decision processes, but also a heuristic frame- atively better performance under important extreme work for planning and project design. Without using cases, and evaluating all proposed options across pos- simulation models, decision makers can think through sible futures to clarify the tradeoffs associated with the combinations of climatic and socioeconomic fac- each. Rather than suggest a single, best-guess path tors that would cause one or more proposed policies to an optimal decision, robust frameworks are asked to fail to achieve their goals (see Refs 120,121 for to assess the implications of various assumptions, examples related to road design and forest manage- uncover unanticipated vulnerabilities, characterize the ment). In addition, relatively simple models are often few deep uncertainties most important to the prob- at the core of such analyses. For example, in the lem, and generally orient the analysis so as to most sea-level rise case study we discuss later, the cost–ben- efficiently guide the search for the most useful policy efit model used to evaluate the robustness of the alternatives.100 different policy alternatives was essentially a simple Faced with deep uncertainty about the future, spreadsheet. This relates to the feasibility, particularly robust decision frameworks suggest that it is most for resource-limited, local projects, of constructing effective to work backwards from the decision con- large numbers of futures for assessing the viability of text to the technical and information requirements of policy options over as comprehensive a set of assump- the problem, relying heavily on participatory pro- tions as possible. As detailed in the case studies below, cesses that bring together the analysis team and often the conversion of an existing, computation- the various parties to the decision to structure the ally inexpensive management model with which the problem appropriately for meeting stakeholder needs. users are already familiar (e.g., to run overnight in This premise is shared generally with all ‘bottom-up’, ‘batch’ mode) suffices for quickly and simply testing ‘risk-based’, or ‘vulnerability-based’ climate change assumptions over many inputs. impacts frameworks.84,116–119 One example is RDM, The idea of using multiple views of the future which has recently been applied in a number of to explore policy options is not unique to robust climate-related decision problems. In RDM, prob- decision frameworks, but also informs traditional sce- lems are structured according to three major con- nario planning methods. Such methods rest on the cepts: multiple views of the future, a ‘robustness’ presumption that a small collection of stories about criterion, and an iterative vulnerability-and-response- different futures can help planners better prepare option analysis (Figure 2). for surprises.122 Scenarios can be either normative, The use of robustness as a satisficing criterion exploratory, or have aspects of both. Normative sce- provides a major point of departure from predict- narios are prescriptive and explicitly values-based, in then-act approaches. Decision analytic frameworks that they describe a future that may be realized only that attempt to maximize expected utility will tend to through specific policy actions (e.g., a greenhouse rank alternative policy options contingent on the best- gas stabilization scenario). In contrast, exploratory estimate probability distributions to suggest a single scenarios describe the future according to known pro- best option. Robust frameworks instead suggest a set cesses of change and pose ‘what if?’ questions.20 How- of choices that perform reasonably well compared to ever, traditional scenario methods struggle to address the alternatives across a wide range of future scenarios, two questions: which futures to highlight, and how often sacrificing some performance in order to reduce to inform real decisions. Robust decision frameworks sensitivity to broken assumptions and thus minimize address these issues, in part by viewing scenarios as regret under particular futures. cases where a candidate strategy fails to meet decision RDM and related approaches offer not only makers’ goals and standards, as evaluated against a a quantitative methodology for incorporating large large number of future states of the world.123

Identify and assess Choose options for Structure candidate Characterize ameliorating problem strategy vulnerabilities vulnerabilities

To explore these concepts further, specifically in address the water demands of the region’s projected the context of using climate modeling in decision growth. As required by the state, IEUA’s plans support, we turn to a few cases of real-world must demonstrate that the agency can successfully applications. These examples highlight the following: accommodate future demand under historical climate conditions, but, like the others developed previously, (1) Advantages of robust decision frameworks the 2005 plan does not consider how it would perform when compared to traditional prediction-based under altered future climate. IEUA assembled a team approaches, in particular for the types of of researchers and analysts to use RDM to assess the information typically available from climate vulnerabilities of their 2005 plan in the face of climate models today. change, as well as help identify options for addressing (2) The importance of considering climate models, and managing these vulnerabilities. not as a standalone element of the analysis, but There were a number of factors that led to as part of linked systems of multiple models IEUA’s management agreeing to participate in this (and multiple lines of scientific evidence) with RDM demonstration. The first was that the agency different roles at different stages of the analysis. regards itself as a thought-leader in the water sector; in fact, based in part on the results of this effort, (3) Areas where useful developments in climate other agencies (e.g., Metropolitan Water District of modeling research and practice could occur Southern California, U.S. Bureau of Reclamation, that would better enhance the value of climate Denver Water, California Department of Water model-derived information in robust decision Resources, and the World Bank) have subsequently frameworks. begun to explore the use of similar RDM analyses for their planning. IEUA’s leaders also recognized the Illustrative Case Studies need for improved ways to include their community Managing for Uncertainty at a California in discussions of climate uncertainties and long-range Water Agency planning. They guessed (correctly, as it turned out) Water agencies have always considered hydrologic that the analysis would support the policy directions uncertainty in their planning, but typically only in they had already been advocating to deal with this year-to-year conditions about a historically stationary challenge. mean and not in long-term trends. Most water The first phase of an RDM analysis structures agencies develop a single best estimate of how the problem by (1) defining the key performance water needs will evolve in the long term under metrics an organization should use to judge the suc- projected future demand and historical hydrologic cess of alternative strategies, (2) articulating a set of conditions, along with associated schedules of such strategies that might achieve these objectives, capital improvements and program implementation (3) identifying potentially significant uncertainties to meet performance goals. Under climate change, about the future that would impact the performance the assumption that the hydrologic conditions of the of these strategies, and (4) adapting or developing a past will be a good guide to the future will likely be computer simulation model(s) that can compute strat- even less valid, compromising well-established water egy performance contingent on these uncertainties. planning concepts such as ‘reliability’.96,124 RDM-based processes are, by definition, participa- Here we describe an application of RDM for tory. Here, the analysis team carried out these steps in Southern California’s Inland Empire Utilities Agency collaboration with IEUA planners and presented the (IEUA), a wholesale water and wastewater provider results to the agency’s stakeholders and decision mak- in Southern California’s Riverside County.104 The ers via a series of workshops at IEUA headquarters. IEUA region is in the midst of rapid urban growth A critical aspect of this participatory process was and socioeconomic transformation, with a regional the solicitation of information about key uncertainties population that is anticipated to grow from 800,000 in the management system. Through interviews with in the mid-2000s to approximately 1.2 million by IEUA planners and stakeholders, the analysis team 2025, placing new demands on the water supply and identified six uncertain factors potentially important wastewater treatment system, a challenge typical for to the ability of IEUA to achieve its objectives, southern California and the American Southwest. along with a plausible range of uncertainty for each Also typical for the region, IEUA’s 2005 Urban (Table 2). To represent a range of climate futures, the Water Management Plan125 was a static 25-year plan analysis team developed a large ensemble of synthetic that emphasized improved groundwater management monthly temperature and precipitation sequences that and large increases in wastewater recycling to reflect both local weather variability and the range

TABLE 2 Uncertainties Addressed in the IEUA RDM Analysis (Reprinted with permission from Ref. 104 Copyright 2010 Elsevier Ltd.) Uncertain Key Factors Representation Within WEAP Model Future climate key factors 450 Sequences of monthly temperature and precipitation reflecting trends in ◦ ◦ temperature −0.4 Cto+2.75 C and precipitation from −27% to +19% over 30 years Future water demand Water intensity of new development ranging from 0% to 30% more efficient than existing housing stock Impact of climate change on imported supplies Range of maximum climate change-induced declines in imported supplies between 30% and 50% Response of groundwater basin to urbanization Change in percentage of precipitation that percolates into the groundwater and changes in precipitation patterns basin (e.g., runoff) from 0% to 10% Achievement of management strategies Delay in recycling program achievement between 0 and 10 years; achievement in groundwater replenishment goals between 80% and 100% Future costs Annual cost increase in imported supplies between 2.5% and 8%; annual cost increases in efficiency achievement between 2.5% and 8%

WEAP, Water Evaluation and Planning. of regional trends simulated by a variety of leading parameters considered, a specific combination of climate models.126 three of them—large declines in precipitation, larger- The analysis team then developed a simulation than-expected impacts of climate change on the model of the IEUA water management system within availability of imported supplies, and reductions the Water Evaluation and Planning (WEAP) modeling in percolation of precipitation into the region’s environment127 that incorporated representations groundwater basin—would cause the agency to suffer of these uncertainties. An initial application to high costs. This finding motivated IEUA to identify the 2005 plan revealed significant future climate- a further suite of modifications to the 2005 plan related vulnerabilities, in conjunction with the other that reduced these vulnerabilities without significantly uncertain factors. The analysis team then worked increasing costs (or other measures of agency effort) with IEUA planners to identify a set of potential and begin implementing a more adaptive strategy modifications to the plan that might address these consistent with the analysis findings. This new strategy vulnerabilities. By running WEAP over 450 different calls for IEUA to accelerate efforts to expand the parameter combinations drawn from the ranges of agency’s dry-year-yield program and to implement the six uncertain factors, the team was able to its recycling program, making additional investments evaluate the performance of each of these water in efficiency and storm-water capture if monitoring management strategies throughout this space of future suggests approaching shortages. IEUA has committed uncertainties, using metrics such as the costs to the itself to this new plan, often citing the RDM analysis agency of incurring water shortages. in support of their decision. One of the challenges faced in this and other The above summarizes the main points of this RDM analyses is in representing the richness of these case study. What of the lessons about climate modeling multiple-criteria evaluations of a variety of manage- to support this type of analysis? This particular ment plans over such a large number of futures in application of RDM did not use a large, sophisticated, ways that communicate clearly and simply the impor- and customized set of climate simulation data. Rather, tant nuances of the problem. RDM uses a systematic, it relied on basic information developed for other computer-aided process of ‘scenario discovery’128 to purposes and accessible to a variety of users. In this identify a small number of combinations of uncertain context, this example provides two important lessons parameter values that best represent those future states about the use of, and demands on, climate models to where a given management plan performs poorly. This support planning processes designed to account for allows the analysis team to provide a concise descrip- future climate change. tion of representative cases that best summarize such First, and most fundamentally, the story of this poorly performing strategies with a small number of case is about stakeholder acceptance of climate model scenarios meaningful to decision makers. outputs as useful inputs to planning. On the basis of For IEUA and its candidate plans, this scenario various stakeholder interactions carried out as part discovery process suggested that, of the six uncertain of the analysis process, including formal evaluations

48 © 2012 John Wiley & Sons, Ltd. Volume 4, January/February 2013 WIREs Climate Change The value and demands of robust decision frameworks of the impact of the RDM analysis,104 it was clear for natural systems, human communities, and that many of the stakeholders would have distrusted socioeconomic sectors.130–133 A key challenge for the coarse-scale climate model outputs, spanning a planning is to incorporate the possibility of such wide range of values for key variables, if they were abrupt and impactful changes for which uncertainty presented as predictions. In other words, the supply of is both deep and poorly characterized. climate predictions would not have met the demand The Port of Los Angeles (PoLA) is one of the for confidence in those predictions. Under the RDM largest container shipping facilities in the world, and paradigm, however, the available supply of model- it faces the challenges of planning for the uncertain derived climate information was sufficient to meet the probability but potentially large impact of extreme demands of an analysis of vulnerability and robust sea level rise over the coming century. Such extremes responses. Here, the stakeholders were only asked to can affect infrastructure investments but are difficult consider how much the climate would have to change to incorporate into decision processes because of their before the current IEUA plan would begin to gener- deep uncertainties. Complicating matters, available ate shortages, and then to believe that such changes information about these factors may span a wide were sufficiently plausible to warrant consideration of range of uncertainties, from well characterized to potential responses. The users were able to perceive the deep. Both annual means and short-term extremes same climate model output with greater salience, cred- of future sea level are expected to differ in the future, ibility, and legitimacy. Even without agreement across due to effects such as thermal expansion, changes in parties as to the level and meaning of the uncertainty oceanic structure, melting of land-based ice, shifts in in the climate futures, trust in the process allowed for oceanic and atmospheric circulation, and changes in 134 trust in the management plan eventually arrived at. the terrestrial water balance. While some of these Second, this example highlights how RDM processes are relatively well understood, others remain is structured to consider adaptive management deeply uncertain, such as rapid flows of land-based ice approaches explicitly within a quantitative decision and changes in the frequency and intensity of future 135–137 support framework. This is a particular advantage, extreme wave events and storm surges. as adaptive strategies are obviously useful for plan- This application is therefore amenable to ning under conditions of deep uncertainty but often a robust decision approach. An interdisciplinary prove difficult for public agencies to implement, for analysis team conducted an RDM analysis for PoLA a variety of reasons. In addition, theoretical trade- to help them assess one particular decision—whether offs between adaptability and prediction uncertainty PoLA should harden their container ship terminals have been noted.129 The RDM-based decision sup- against future sea level rise during the next major 105 port provided to IEUA created the opportunity for upgrades of those terminals (Figure 3). At some that agency to identify and evaluate an adaptive man- time in the future, the Port will upgrade its terminals, agement alternative to its current plan, as well as the and at that time it can decide to spend an additional justification to ultimately adopt this alternative. In sum to protect against higher future sea levels during this context, RDM specifically allows for the quan- the terminal lifetime, thereby suffering no further costs tification of how adaptive a policy need be and how through the next upgrade time. If PoLA decides not valuable would be attempts to acquire new scien- to harden, the terminal may prove vulnerable during tific knowledge about the observed state of the given its lifetime to sea level rise and storm surges, with the system (monitoring and assessment) and its possi- potential for significant extra cost. ble range of behaviors and likely future evolution The RDM analysis attempted to answer two (process modeling and prediction).103 This quantifi- questions: cation can help decision makers weigh investments in, for example, new and improved climate modeling, (1) Under what future conditions would PoLA find against the costs of acquiring other types of infor- it advantageous to have hardened its terminal mation about the system or making specific decisions at the next upgrade? that would obviate the need for that information. We (2) Does current science and other available build on both of these lessons in our next case study. information suggest these conditions are sufficiently likely to justify a decision to harden Rapid Sea Level Rise and the Port of Los at the next upgrade? Angeles One critical reason for turning to robust frameworks The parameters identified as relevant to PoLA’s is the growing recognition of the potential for decision fell into two categories: eight describing abrupt climate changes with important consequences future sea level and six describing the terminal and its

= C1 Charden Harden at upgrade

Probability of flooding exceeds Pcrit t < L − t at time L C (t) = C e−dt Do not 2 upgrade L harden at upgrade

FIGURE 3| Simplified representation of the Port of Los Angeles’ decision regarding No flooding whether or not to harden the terminal at its = begins before C3 0 next upgrade and the costs resulting from its next upgrade choices. future management (see table 1 in105). Some of these This assessment suggested that the requisite combina- parameters have known values, some are relatively tion of extreme sea level rise, storm surge increase, well understood and can be represented with well- and longer-than-expected terminal lifetime were suf- characterized probability distributions, and some are ficiently unlikely, given current knowledge, to war- deeply uncertain. The analysis team constructed an rant hardening at only one of the four facilities analytical cost–benefit model for terminal hardening analyzed. that incorporated representations of these different Building on our first case study, this example uncertainties. highlights a number of important elements of RDM, The team carried out 500 runs of this including how to employ climate information with cost–benefit calculation over this space and performed different levels of uncertainty in the same analysis and scenario discovery on the resulting database of outputs combining this uncertain climate information with to succinctly characterize the conditions where an uncertain information about relevant socioeconomic early hardening would meet a cost–benefit test. This factors. Combining the different types of information phase of the analysis identified a number of futures, considered in this case, e.g., statistical fits to observed characterized by a near-term, rapid increase in mean data, probabilistic estimates, and model-based process sea level, a significant increase in sea level variability studies, provides a concrete illustration of how one (e.g., due to storm surges), and a long terminal lifetime, does not have to ‘build the big model’ (i.e., do where PoLA might regret a decision not to harden at consolidative modeling) to provide decision-relevant the next upgrade. In particular, if the probability of information about the behavior of a complex system. occurrence of these conditions were at least 7%, it In addition, similarly to the IEUA example, the would make sense to invest in early hardening. supply of information about factors such as rapid sea This led to the second analysis question: how level rise due to ice sheet disintegration or increased likely do the PoLA decision makers judge these futures storminess would not have been sufficient to meet to be? Answering this question entails evaluating the the demand for high confidence under a prediction available scientific evidence to determine whether this paradigm but was, however, sufficient to support the scenario is sufficiently likely to justify a decision to RDM analysis. At the same time, this case suggested harden. Here is where we confront the limits of today’s ways in which climate science and modeling could scientific knowledge. Current climate models do not progress to improve the bounding analyses for these provide the right kind of information to evaluate either deeply uncertain factors. It provided a quantification extreme sea level rise or changes in the frequency of of the value of such new knowledge by specifying the storm surges.135,138,139 The team therefore assembled degree of change in the range estimate that would various lines of observational and theoretical evi- alter the choice of decision. dence, as well as insights from other types of models More generally, the imprecise probability inter- (e.g., Earth system models of intermediate complex- val for a particular threshold is decision-relevant but ity), drawing from a number of recent studies.140–142 not, in general, what the climate modeling community

50 © 2012 John Wiley & Sons, Ltd. Volume 4, January/February 2013 WIREs Climate Change The value and demands of robust decision frameworks would currently attempt to provide. Even significantly freshwater body in the world, under the auspices of improved scientific knowledge of climate-induced the International Upper Great Lakes Study (IUGLS), threshold crossing in human and natural systems an independent study board comprised of U.S. is unlikely to do much to improve predictions of and Canadian members established in 2007 by the impacts, being more likely to suggest scientific plausi- Great Lakes International Joint Commission. The bility and the presence of risk rather than information charge to IUGLS is to review the operating rules such as precise timing. RDM, on the other hand, has and management criteria for the governance of the proven to be a useful framework for incorporating lake system and recommend changes to the lake’s these uncertain thresholds (see also Ref 106). This regulation plan.98,143 suggests that an increase in the use of RDM and IUGLS has already rejected the traditional pro- related decision frameworks would increase demand cess of selecting an optimal plan based on a ‘most for improved scientific knowledge about the potential likely’ future scenario, because of the uncertainties for exceeding key thresholds, even if that information associated with future climate change, and hence were not sufficient to support decision making under lake levels, on top of large uncertainties associ- prediction-based frameworks. ated with ecosystem responses to future change and the various socioeconomic drivers of lake use and DS for Lake Superior conditions. Instead, they have chosen a more bottom- Our final case provides an example of DS. Like RDM, up, vulnerability-based approach that will explicitly DS is a member of the family of robust decision account for these uncertainties in identifying a regula- frameworks that we have been discussing, and that, tion plan design that performs well over a very broad with RDM, shares a number of the key features of range of possible futures (Figure 4), using DS to inte- such frameworks.97 grate local and user-based perspectives on significant A team of researchers and managers are vulnerabilities of key lake management goals with currently working to develop a long-term management information on future climate derived from climate strategy for Lake Superior, the largest managed models.

Traditional approach Decision scaling

1. Downscale multiple model projections 3. Determine plausibility of climate conditions/ vulnerability 2. Generate a few water supply cases

3. Find whether system is vulnerable for 2. Link to climate these series conditions

Vulnerability domain

1. Determine the vulnerability domain

FIGURE 4| Decision scaling begins with a bottom-up analysis to identify the climate states that impact a decision and then uses climate information to provide insight to the decision. (Reprinted with permission from Ref. 98 Copyright 2007 Wiley Blackwell.)

The strategy rests on using the insights from analysis, developed for assessing flood risk in the a vulnerability analysis to inform the selection and UK,144 where sensitivity analyses of watershed processing of the climate model information—to tailor responses to climate in a river flow model were used the choice and use of model outputs to maximize their to construct a response surface that was then used to credibility and utility in the assessment. Two key answer questions about policy vulnerabilities in the elements of the DS decision-analytic process in this face of uncertain future climate change: e.g., what example make possible this linking of the bottom-up fraction of members of a given ensemble of climate and top-down information sources: scenarios would be accommodated by a given design safety margin? (1) Holding a series of technical meetings with As with the RDM cases already discussed, DS stakeholder experts to define three ‘coping increases the credibility and salience of the climate zones’—i.e., lake levels that were deemed A model information for use in the decision process (acceptable), B (significant negative impacts by changing the requirements for the information. but survivable), or C (intolerable without For Lake Superior, the synthesis of stakeholder- and policy changes)—for meeting use goals and system-specific decision information in coping zones limiting impacts in areas such as ecosystems, and climate response functions then facilitates precise hydropower, shipping, water systems, coastal evaluation of the decision options with respect to any systems, and recreation and tourism set of climate futures: (2) Developing a set of ‘climate response functions’ that relate the occurrence of climate-driven in top-down approaches to climate change impact changes in Net Basin Supplies (i.e., the assessments, the emphasis is often on attempting to sum of precipitation, runoff, releases, inflows estimate the future f(x), that is, the future distribution of climate or hydrologic variables. In our approach, and diversions, and evaporation) to the the initial emphasis is on C(x), the response of the consequences for and of a particular lake system to all the possible values of x, possible future management decision climates, without regard to the probability associated with those values.98 The coping zones provide the bottom-up context for identifying the regional climate states that would This study emphasizes the general importance tip the lake system into a new regime—i.e., over the of multi-model ensembles, in an exploratory mod- thresholds from acceptable to survivable to intoler- eling context, for decision-analytic approaches to able. The process of defining these coping zones for climate change, citing their usefulness for estimat- each sector systematizes the participation of stake- ing the frequency of occurrence of climate conditions holders in the analysis. It also subsequently allows the that may lead to threshold crossing in the system analysis team to ask tailored questions of the climate of interest. Although these frequencies will not, in models and, in the context of this climate informa- general, represent true probabilities of future out- tion, evaluate plan performance in a comparable way comes, they can be viewed as providing a lower across impact sectors. bound on the maximum range of uncertainty, and The complementary concept of a climate thus a ‘non-discountable’ envelope of futures.45 Here, response function serves as the quantitative link one uses simulation models to make this enve- between the coping zones and this tailoring of cli- lope as wide as possible, ensuring that the decision mate information. Given the identification of climate options incorporate, and are evaluated against, the conditions that are critical to a decision, processing range of possible system behaviors (including extreme of information about future climate change derived cases). from models can be focused on those key aspects, as Though this effort is ongoing, this DS analysis opposed to beginning with a pre-determined set of of Lake Superior has so far facilitated a number of scenarios as input to the analysis. For example, the significant outcomes. For example, the lake regula- choice of climate models, spatial and temporal scale, tion strategy now incorporates what is being termed and associated process models need only be made ‘robust adaptation’, a hierarchical approach for man- after the specific information needed for a particular aging uncertainty and to facilitate adaptation to decision context is identified. This choice can then climate change (among other unanticipated changes). be made strategically to improve the relevance of the This involves the development of a dynamic regulation climate information in the decision process. plan that selects from a portfolio of plans developed The climate response function concept in DS for a wide range of climate conditions, coupled with is very similar to the concept of ‘scenario neutral’ an adaptive management process for continuously

52 © 2012 John Wiley & Sons, Ltd. Volume 4, January/February 2013 WIREs Climate Change The value and demands of robust decision frameworks reviewing performance (via monitoring) and recom- however, suggests new emphases in climate modeling mending improvements as they become necessary.98 that could increase both the richness of our understanding of climate system behavior and the decision support value of this richer understanding. We sug- SYNTHESIS: OPPORTUNITIES AND gest that these new emphases fall into the following IMPLICATIONS OF ROBUST DECISION categories: FRAMEWORKS FOR CLIMATE MODELING (1) Model applications (2) Model structures and complexity Decisions made with support of climate modeling can follow multiple pathways, from a focus on resilience (3) Technical and technological considerations and supporting adaptive management within the full range of what the models have to say, to The view from the perspective of both a more rigid path that places great reliance on exploratory modeling and robust decision frameworks producing an accurate forecast of the future. We have implies at least two distinct roles for climate modeling: argued that a prediction-based paradigm is poorly to help structure the problem by exploring the limits of suited for overcoming the twin physical and social possible system behaviors; and in the ‘scenario gener- sciences challenges of (1) long-term, regional-scale ator’ role, to help explore the performance of decision climate prediction and (2) providing decision-relevant options over as wide a range of futures as possi- information about potential future climate change ble. The first suggests an increased focus on climate within the kinds of stakeholder-focused, participatory model applications like the development of bounding processes that lead to better decision making. In and extreme cases, identification of potential warn- contrast, the class of robust decision frameworks ing signs, and the assessment of relative likelihoods offers an opportunity to address these challenges in a and probable contingencies for future events, includ- way that more fully harnesses knowledge and insights ing the potential for rapid or accelerating change and produced by climate science and modeling. threshold crossing in natural and human systems. The In our case studies, we have seen how the use second suggests complementary, redoubled efforts to of robust decision frameworks can confer greater expand the space of climate model formulations and credibility, salience, and legitimacy on the types of structures, as well as to sample this space more com- information typically provided from climate models prehensively. today. By beginning with bottom-up articulations of In addition, for most applications we are con- management contexts and system vulnerabilities, such cerned with coupled systems, such as linked climate processes guide the search for tailored climate infor- and hydrologic systems. We will often not be able to mation and provide specific questions that can then explore the full richness of interactions and behaviors be asked of models. This promotes greater stake- in such systems, nor generate appropriately diverse holder acceptance of model-derived information. In model formulations (and hence scenarios), without addition, such frameworks allow flexibility in defin- paying attention to issues of model structure and ing relevant climate futures based on the different risk model complexity. For example, the development of tolerances that different decision-making entities may the climate response functions for our Lake Superior have. These examples have also illustrated how the case imply some degree of interoperability between use of such frameworks can suggest avenues for new climate models and hydrologic models to properly research to improve the quality of climate informa- capture important interactions between the two sys- tion—and specifically to quantify the value of this new tems. In general, improving climate and impacts model knowledge for a particular decision. interoperability and integration will help alleviate the What our examples have also indirectly high- problem of developing transfer functions that may lighted, however, is that current applications of fail to represent key system behaviors. Extending to robust decision frameworks have not been particularly better integration with management models (such as demanding of climate models. Existing archives of WEAP) will make it easier to address questions of basic model outputs such as monthly temperature and the full, coupled human-natural system, such as those precipitation, potentially enhanced with some spatial related to unintended consequences of policies and and temporal downscaling and other kinds of post- other possible feedbacks from decisions taken, as well processing, have provided sufficient raw material to as explore non-climatic pressures. meet the needs of these initial forays into robust deci- As we make our modeling systems more com- sion frameworks. Extrapolating from these examples, plex, however, we run the risk of succumbing to the

TABLE 3 Newly Initiated Efforts in Alternative Frameworks for Climate-Related Decision Support U.S. Department of Defense Strategic Environmental Research and Development Program (SERDP) Pilot Projects: Understanding Data Needs for Vulnerability Assessment and Decision Making to Manage Vulnerability of DoD Installations to Climate Change (PI: R. Moss, Batelle, Pacific Northwest Division) Assessing Climate Change Impacts for DoD Installations in the Southwest United States during the Warm Season (PI: C. Castro, University of Arizona) Decision Scaling: A Decision Framework for DoD Climate Risk Assessment and Adaptation Planning (PI: C. Brown, University of Massachusetts) Climate Change Impacts and Adaptation on Southwestern DoD Facilities (PI: R. Sagarin, University of Arizona) Climate Change Impacts to Department of Defense Installations (PI: V. Kotmarthi, Argonne National Laboratory) U.S. EPA: Exploring the Use of Robust Decision Making Methods for EPA’s National Water Program’s Climate Change Responses (PI: R. Lempert, RAND Corporation) State of California: A Methodology for Predicting Future Coastal Hazards due to Sea-Level Rise on the California Coast (PI: D. Revell, Philip Williams and Associates) pitfalls of consolidative modeling, with models which Finally, all of the above suggests a need for are difficult to validate and interpret. There exists a new tools and technologies to be used within these tension between trying to simulate by capturing as processes and institutional arrangements to systemat- much of the dynamics as we can in comprehensive ically generate large ensembles of model runs, select numerical models, and, on the other hand, trying to the appropriate ensemble members for the given prob- understand by simplifying and capturing the essence of lem, archive the right outputs from them, and inform a phenomenon in an idealized or conceptual model.145 the next set of model runs. This will mean, among We gain understanding of a complex system in part by other things, building from existing efforts to find relating its behavior to those of a hierarchy of progres- new ways to construct diverse models and gener- 148 sively simpler systems (e.g., much simpler biological ate bigger ensembles of model runs and discover systems to improve understanding of more complex the families of scenarios of greatest relevance to the 104,105 organisms like human beings, as per Ref 145). Greater problem at hand. The computational demands attention to the systematic development and applica- associated with attempting to widen the range of cli- tion of the less-complex members of the modeling mate impact uncertainty (as opposed to attempting to hierarchy will be essential for extracting the insights narrow it) are clearly large and may themselves justify we need from exploratory climate and Earth system investments in computational capacity to rival those associated with improving climate predictions. modeling. Determining the aspects of model development and application that are important should not occur CONCLUSION in a vacuum but rather in the context of the kinds of As we have argued in this paper, climate-related deci- participatory, bottom-up processes that underlie our sion support is hard, not only because, as often stated, case studies. Such processes will likely require new multidecadal, regional-scale climate prediction is hard, institutional arrangements, including new boundary but also because creating decision-relevant processes organizations, knowledge networks, and communities for the production and uptake of climate informa- of practice, to accommodate them. One example is the tion is hard. This has resulted in climate models U.S. National Climate Assessment, newly envisioned significantly falling short of their potential as tools 3 as a sustained process over time, and explicitly for supporting decision making, thereby limiting the structured around participatory sub-processes (e.g., available options for developing informed adaptation 146 for scenario development ). In addition, one and mitigation responses to climate change. outcome of the symposium that originally inspired We have proposed addressing the problem, in this review (see Acknowledgments), at least in part, part, by expanding the conception of climate models: was a set of new efforts supported (individually) by the not simply as prediction machines, but as scenario U.S. Department of Defense, the U.S. Environmental generators, sources of insight into complex system Protection Agency, and the State of California,147 behavior, and aids to critical thinking within robust designed to explore some of these modeling questions decision frameworks. Such frameworks are perfectly and the value of alternative decision frameworks for suited for a large-scale integration with climate supporting climate-related decisions (Table 3). modeling (and associated region- or sector-specific

54 © 2012 John Wiley & Sons, Ltd. Volume 4, January/February 2013 WIREs Climate Change The value and demands of robust decision frameworks impacts models), with the adjustment that climate and we certainly believe that models that can rep- modeling is viewed from an exploratory rather than resent the climate system with increasing resolution a consolidative perspective. Under this paradigm, and sophistication will have greater potential value increasing the quality, complexity, diversity, and for supporting decision making about responding amount of climate information generated from cli- to climate change. As we highlight in our review, mate models has the potential to greatly increase the however, likely even more important for success- richness of decision-relevant information. ful outcomes than the technical attributes of the We have illustrated these arguments with a num- models are the decision-analytic and institutional con- ber of examples. These have shown how use of texts within which they are developed and applied. robust decision frameworks leads to greater stake- And ﬁnally, the success of robust decision frame- holder acceptance of climate model results. They have works (and the management approaches they enable) also suggested new emphases in climate modeling, in the areas of model applications, model development, depends on other critical factors not addressed here: and technological advances, that could increase both e.g., continuous, well-designed, and well-maintained the richness of our understanding of the climate sys- monitoring networks to support adaptive manage- tem and the value of information derived from climate ment and learning; good governance and ﬂexible, models in robust decision frameworks. inclusive institutions; and the technical skills, support, The focus of this paper has been climate models, and information products for managers to implement and their use within different decision frameworks, these new approaches into their day-to-day work.143

ACKNOWLEDGMENTS The origins of this paper are in a session of the 2009 U.S. Department of Defense Strategic Environmental Research and Development Program (SERDP)/Environmental Security Technology Certification Program (ESTCP) Partners in Environmental Technology Symposium, titled, ‘Challenges Associated with Regional Predictions of Climate Change Impacts’, of which the authors of this paper were the organizers or speakers. The Symposium was held in Washington, DC in December 2009. Many thanks to J. Shukla for participating in the symposium as well and for offering his contrasting perspectives on the paper and the issues that it attempts to address. C.P.W. would like to thank A. Grambsch and K. Hychka, who made insightful comments on earlier drafts of this article, as well as EPA’s Global Change Research Staff in the Office of Research and Development’s National Center for Environmental Assessment for many useful discussions that helped frame a number of the ideas and perspectives presented in this paper. Finally, we would like to thank three anonymous reviewers whose thoughtful comments have helped us improve our paper significantly. The views expressed here do not necessarily represent the views or policies of the EPA, the DoD, or of any of the other institutions with which the authors are affiliated.

REFERENCES 1. National Academy of Public Administration (NAPA). at http://library.globalchange.gov/u-s-global-change- Building Strong for Tomorrow: NOAA Climate Ser- research-program-strategic-plan-2012-2021. vice. Washington, DC, USA: National Academy of 4. Dessai S, Hulme M, Lempert R, Pielke R Jr. Climate Public Administration; 2010, 116. prediction: a limit to adaptation? In: Adger WN, 2. World Meteorological Organization (WMO). Climate Lorenzoni I, O’Brien KL, eds. Adapting to Climate Knowledge for Action: A Global Framework for Cli- Change: Thresholds, Values, Governance. Cambridge, mate Services—Empowering the Most Vulnerable. UK: University Press; 2009. Geneva, Switzerland: World Meteorological Organi- 5. Meyer R. The public values failures of climate science zation; 2011, 248. in the US. Minerva 2011, 49:47–70. doi:10.1007/ 3. U.S. Global Change Research Program (USGCRP). s11024-011-9164-4. The National Global Change Research Plan: 6. Shapiro M, Shukla J, Brunet G, Nobre C, Beland M, 2012–2021. A Report by the U.S. Global Change Dole R, Trenberth K, Anthes R, Asrar G, Barrie L, Research Program and the Subcommittee on Global et al. An earth-system prediction initiative for the Change Research. Washington, DC, USA: U.S. twenty-ﬁrst century. Bull Am Meteorol Soc 2010, 91: Global Change Research Program; 2012. Available 1377–1388.

7. Piao S., Ciais P., Huang Y., Shen Z., Peng S., Li J., 20. Rosentrater LD. Representing and using scenarios for Zhou L., Liu H., Ma Y., Ding Y., et al. The impact of responding to climate change. WIREs Clim Change climate change on water resources and agriculture in 2010, 1:253–259. China. Nature 2010, 467:43–51. 21. Rubino A. What will a new generation of world climate 8. Barron EJ. Beyond climate science. Science 2009, research and computing facilities bring to climate long- 326:643. term predictions? Theor Appl Climatol 2011, 106: 9. Collins M. Ensembles and probabilities: a new era in 473–479. doi:10.1007/s00704-011-0448-2. the prediction of climate change. Phil Trans R Soc A 22. Smith LA, Stern N. Uncertainty in science and its role 2007, 365:1957–1970. in climate policy. Phil Trans R Soc A 2011, 369: 10. Goddard L, Baethgen W, Kirtman B, Meehl G. The 4818–4841. urgent need for improved climate models and predic- 23. Hulme M, Dessai S. Predicting, deciding, learning: can tions. EOS 2009, 90:39. one evaluate the ‘success’ of national climate scenar- 11. Doherty SJ, Bojinski S, Henderson-Sellers A, Noone ios? Environ Res Lett 2008, 3:045013. K, Goodrich D, Bindoff NL, Church JA, Hibbard KA, 24. Randall DA, Wood RA, Bony S, Colman R, Fichefet Karl TR, Kajfez-Bogataj L, et al. Lessons learned from T, Fyfe J, Kattsov V, Pitman A, Shukla J, Srinivasan IPCC AR4: Scientific developments needed to under- J, et al. 2007: Cilmate models and their evaluation. stand, predict, and respond to climate change. Bull In: Solomon S, Qin D, Manning M, Chen Z, Marquis Am Meteorol Soc 2009, 90:497–513. M, Averyt KB, Tignor M, Miller HL, eds. Climate Change 2007: The Physical Science Basis. Contribu- 12. Shukla J, Hagedorn R, Hoskins B, Kinter J, Marotzke tion of Working Group I to the Fourth Assessment J, Miller M, Palmer TN, Slingo J. Revolution in climate Report of the Intergovernmental Panel on Climate prediction is both necessary and possible: A decla- Change. Cambridge, UK and New York, NY: Cam- ration at the World Modelling Summit for Climate bridge University Press; 2007. Prediction. Bull Am Meteorol Soc 2009, 90:175–178. 25. Randall D, Khairoutdinov M, Arakawa A, Grabowski 13. Cox P, Stephenson D. A changing climate for predic- W. Breaking the cloud parameterization deadlock. Bull tion. Science 2007, 317:207–208. Am Meteorol Soc 2003, 84:1547–1564. 14. Hurrell J, Meehl GA, Bader D, Delworth TL, Kirt- 26. Stephens GL, L’Ecuyer T, Forbes R, Gettlemen A, man B, Wielicki B. A unified approach to climate Golaz JC, Bodas-Salcedo A, Suzuki K, Gabriel P, system prediction. Bull Am Meteorol Soc 2009, Haynes J. Dreary state of precipitation in global mod- 90:1819–1832. els. J Geophys Res 2010, 115:D24211. doi:10.1029/ 15. Intergovernmental Panel on Climate Change (IPCC). 2010JD014532. 2007: Climate change. In: Solomon S, Qin D, Man- 27. Liepert BG, Previdi M. Inter-model variability and ning M, Chen Z, Marquis M, Averyt KB, Tignor M, biases of the global water cycle in CMIP3 coupled Miller HL, eds. Climate Change 2007: The Physical climate models. Environ Res Lett 2012, 7: doi:10. Science Basis: Contribution of Working Group I to 1088/1748-9326/7/1/014006. the Fourth Assessment Report of the Intergovernmen- tal Panel on Climate Change. Cambridge, UK and 28. Shukla J. Monsoon mysteries. Science 2007, New York: Cambridge University Press; 2007, 996. 318:204–205. 16. Parson E, Burkett V, Fisher-Vanden K, Keith D, 29. Kidston J, Gerber EP. Intermodel variability of the Mearns L, Pitcher H, Rosenzweig C, Webster M. poleward shift of the austral jet stream in the Global Change Scenarios: Their Development and CMIP3 integrations linked to biases in 20th cen- Use. Sub-report 2.1B of Synthesis and Assessment tury climatology. Geophys Res Lett 2010, 37:L09708. Product 2.1 by the U.S. Climate Change Sci- doi:10.1029/2010GL042873. ence Program and the Subcommittee on Global 30. Neale R, Richter JH, Jochum M. The impact of con- Change Research. Washington, DC, USA: Depart- vection on ENSO: from a delayed oscillator to a series ment of Energy, Office of Biological & Environmental of events. J Clim 2008, 21:5904–5924. Research; 2007, 106. 31. Hirsch RM. A perspective on nonstationarity and 17. Morgan MG, Keith DW. Improving the way we think water management. J Am Water Resour Assoc 2011, about projecting future energy use and emissions of 47:436–446. carbon dioxide. Clim Change 2008, 90:189–215. 32. Castro CL, Pielke RA Sr, Leoncini G. Dynami- 18. Bray D, von Storch H. ‘‘Prediction’’ or ‘‘projection’’? cal downscaling: assessment of value retained and The nomenclature of climate science. Sci Commun added using the Regional Atmospheric Modeling Sys- 2009, 30:534–543. doi:10.1177/1075547009333698 tem (RAMS). J Geophys Res 2005, 110:D05108. 19. Lemos MC, Rood RB. Climate projections and their doi:10.1029/2004/D004721. impact on policy and practice. WIREs Clim Change 33. Pielke RA Sr, Wilby RL. Regional climate downscal- 2010, 1:670–682. ing: what’s the point? EOS 2012, 93:52–53.

34. Navarra A, Kinter JL III, Tribbia J. Crucial experi- 51. Hagedorn R, Doblas-Reyes FJ, Palmer TN. The ratio- ments in climate science. Bull Am Meteorol Soc 2010, nale behind the success of multi-model ensembles in 91:343–352. seasonal forecasting—I. Basic concept. Tellus 2005, 35. Giorgi F. Climate change prediction. Clim Change 57A:219–233. 2005, 73:239–265. 52. Gleckler PJ, Taylor KE, Doutriaux C. Performance 36. Dessai S, Hulme M. Does climate adaptation policy metrics for climate models. J Geophys Res 2008, need probabilities? Clim Policy 2004, 4:107–128. 113:D06104. doi:10.1029/2007JD008972. 37. Hawkins E, Sutton R. The potential to narrow uncer- 53. Weigel AP, Knutti R, Liniger MA, Appenzeller C. Risks tainty in regional climate predictions. Bull Am Meteo- of model weighting in multimodel climate projections. rol Soc 2009, 90:1095–1107. J Clim 2010, 23:4175–4191. 38. Pielke RA Sr, Pitman A, Niyogi D, Mahmood R, 54. Schaller N, Mahlstein I, Cernak J, Knutti R. Analyzing McAlpine C, Hossain F, Klein K, Golewijk U, Nair R, precipitation projections: a comparison of different Betts S, et al. Land use/land cover changes and climate: approaches to climate model evaluation. J Geophys modeling analysis and observational evidence. WIREs Res 2011, 116:D10118. doi:10.1029/2010JD014963. Clim Change 2011, 2:828–850. 55. Pirtle Z, Meyer R, Hamilton A. What does it 39. Palmer TN, Alessandri A, Andersen U, Cante- mean when climate models agree? A case for assess- laube P, Davey M, Delecluse P, Deque M, Diez ing independence among general circulation models. E, Doblas-Reyes FJ, Feddersen H, et al. Develop- Environ Sci Policy 2010, 13:351–361. doi:10.1016/ ment of a European multimodel ensemble system for j.envsci.2010.04.004. seasonal-to-interannual prediction (DEMETER). Bull 56. Knutti R, Abramowitz G, Collins M, Eyring V, Gleck- Am Meteorol Soc 2004, 85:853–872. ler PJ, Hewitson B, Mearns L. Good practice guidance 40. Boer GJ, Lambert SJ. Multi-model decadal poten- paper on assessing and combining multi model cli- tial predictability of precipitation and temperature. mate projections. In: Stocker TF, Qin D, Plattner GK, Geophys Res Lett 2008, 35:L05706. doi:10.1029/ Tignor M, Midgley PM, eds. Meeting Report of the 2008GL033234. Intergovernmental Panel on Climate Change Expert 41. Hargreaves JC. Skill and uncertainty in climate models. Meeting on Assessing and Combining Multi Model WIREs Clim Change 2010, 1:556–564. Climate Projections. Bern, Switzerland: IPCC Work- ing Group I Technical Support Unit, University of 42. Raisanen J. How reliable are climate models? Tellus Bern; 2010. 2007, 59A:2–29.. 57. Keenlyside NS, Ba J. Prospects for decadal climate 43. Knutti R. The end of model democracy? Clim Change prediction. WIREs Clim Change 2010, 1:627–635. 2010. doi:10.1007/s10584-010-9800-2. 58. Murphy JM, Sexton DMH, Barnett DN, Jones GS, 44. Stainforth DA, Allen MR, Tredger ER, Smith LA. Con- Webb MJ, Collins M, Stainforth DA. Quantiﬁcation ﬁdence, uncertainty, and decision-support relevance in of modeling uncertainties in a large ensemble of climate climate predictions. Phil Trans R Soc A 2007a, 365: change simulations. Nature 2004, 430:768–772. 2145–2161. 59. Stainforth DA, Aina T, Christensen C, Collins M, Faull 45. Stainforth DA, Downing TE, Washington R, Lopez A, N, Frame DJ, Kettleborough JA, Knight S, Martin A, New M. Issues in the interpretation of climate model Murphy JM, et al. Uncertainty in predictions of the ensembles to inform decisions. Phil Trans R Soc A climate response to rising levels of greenhouse gases. 2007b, 365:2163–2177. Nature 2005, 433:403–406. 46. Rastetter EB. Validating models of ecosystem response 60. Stott PA Forest CE. Ensemble climate predictions using to global change. BioScience 1996, 46:190–198. climate models and observational constraints. Phil 47. Boer GJ. Changes in interannual variability and Trans R Soc Lond 2007, 365A:2029–2052. decadal potential predictability under global warming. J Clim 2009, 22:3098–3109. 61. Roe GH, Baker MB. Why is climate sensitivity so unpredictable? Science 2007, 318:629–632. 48. Reifen C, Toumi R. Climate projections: past performance no guarantee of future skill. Geophys Res Lett 62. Lemoine DM. Climate sensitivity distributions depen- 2009, 36:L13704. doi:10.1029/2009GL038082. dence on the possibility that models share biases J Clim 2010, 23:4395–4415. 49. Macadam I, Pitman AJ, Whetton PH, Abramowitz G. Ranking climate models by performance using 63. Pennell C, Reichler T. 2011 On the effective number actual values and anomalies: Implications for climate of climate models J Clim 24: 2358–2367. change impact assessments. Geophys Res Lett 2010, 64. Weitzman ML. On modeling and interpreting the eco- 37:L16704. doi:10.1029/2010GL043877. nomics of catastrophic climate change Rev Econ Stat 50. Wilby RL. Evaluating climate model outputs for 2009, 91:1–19. hydrological applications—opinion. Hydrol Sci J 65. National Research Council (NRC). Informing Deci- 2010, 55:1090–1093. sions in a Changing Climate. Panel on Strategies and

Methods for Climate-Related Decision Support, Com- 80. Baethgen WE, Carriquiry M, Ropelewski C. Tilting mittee on the Human Dimensions of Global Change. the odds in maize yields. Bull Am Meteorol Soc 2009, Division of Behavioral and Social Sciences and Educa- 90:179–183. tion. Washington, DC: The National Academies Press; 81. Ziervogel G, Johnston P, Matthew M, Mukheibir 2009, 188. P. Using climate information for supporting climate 66. Cash DW, Borck JC, Patt AG. Countering the loading- change adaptation in water resource management in dock approach to linking science and decision making: South Africa. Clim Change 2010, 103:537–554. comparative analysis of El Nino/Southern Oscillation 82. Buizer J, Jacobs K, Cash D. Making short-term climate (ENSO) forecasting systems Sci Tech Hum Val 2006, forecasts useful: linking science and action. Proc Natl 31:465–494. Acad Sci USA 2010. doi:10.1073/pnas.0900518107. 67. Cash DW, Clark WC, Alcock F, Dickson NM, Eckley 83. Moser SC, Luers AL. Managing climate risks in Cali- N, Guston DH, Jager J, Mitchell RB. Knowledge sys- fornia: the need to engage resource managers for suc- tems for sustainable development Proc Natl Acad Sci cessful adaptation to change. Clim Change 2008, 87: USA 2003, 100:8086–8091. S309–S322. 68. Guston DH. Stabilizing the boundary between politics 84. Johnson TE, Weaver CP. A framework for assess- and science: the role of the office of technology trans- ing climate change impacts on water and watershed fer as a boundary organization Social Stud Sci 1999, systems. Environ Manag 2009, 43:118–134. 1:87–111. 85. Matthews JH, Wickel AJ. Embracing uncertainty in 69. Farrell A, VanDeveer SD, Jager J. Environmen- freshwater climate change adaptation: a natural his- tal assessments: four under-appreciated elements of tory approach. Clim Dev 2009, 1:269–279. design. Glob Environ Change 2001, 11:311–333. 86. Gober P, Kirkwood CW, Balling RC Jr, Ellis AW, 70. Fischhoff B. What forecasts (seem to) mean. Int Deitrick S. Water planning under climatic uncertainty J Forecast 1994, 10:387–403. in Phoenix: Why we need a new paradigm. Ann Assoc 71. In: Sarewitz D, Pielke RA Jr. Byerly R Jr, eds. Pre- Am Geogr 2010, 100:356–372. diction: Science, Decision Making, and the Future of 87. White. DD, Wutich A, Larson KL, Gober P, Lant T, Nature. Washington, DC: Island Press; 2000, 405. pp. Senneville C. Credibility, salience, and legitimacy of 72. Pielke RA Jr, Conant RT. Best practices in prediction boundary objects: water managers’ assessment of a for decision-making: lessons from the atmospheric and simulation model in an immersive decision theater. Sci earth sciences. Ecology 2003, 84:1351–1358. Pub Policy 2010, 34:219–232. 73. Sarewitz D, Pielke RA Jr. The neglected heart of sci- 88. Reid WV, Chen D, Goldfarb L, Hackmann H, Lee ence policy: Reconciling supply of and demand for YT, Mokhele K, Ostrom E, Raivio K, Rockstrom J, Schnellnhuber HJ, et al. Earth system science for science. Environ Sci Policy 2007, 10:5–16. global sustainability: grand challenges. Science 2010, 74. Changnon SA, Kunkel KE. Rapidly expanding uses of 330:916–917. climate data and information in agriculture and water 89. New M, Cuellar M, Lopez A. Probabilistic regional resources: causes and characteristics of new applica- and local climate projections: False dawn for impacts tions. Bull Am Meteorol Soc 1999, 80:821–830. assessment and adaptation. In: IPCC TGICA Expert 75. Jacobs K, Garfin G, Lenart M. More than just talk: Meeting ‘Integrating Analysis of Regional Climate connecting science and decision-making. Environment Change and Response Options’. Meeting Report. 2005, 47:8–21. Intergovernmental Panel on Climate Change; 2010. 76. Rayner S, Lach D, Ingram H. Weather forecasts are 90. Lempert R, Nakicenovic N, Sarewitz D, Schlesinger for wimps: why water resource managers do not use M. Characterizing climate-change uncertainties for climate forecasts. Clim Change 2005, 69:197–227. decision-makers. Clim Change 2004, 65:1–9. 77. Miles EL, Snover AK, Whitely Binder LC, Sarachik 91. Morgan MG, Henrion M. Uncertainty: A Guide to ES, Mote PW, Mantua N. An approach to designing Dealing with Uncertainty in Quantitative Risk and a national climate service. Proc Natl Acad Sci USA Policy Analysis. Cambridge, UK: Cambridge Univer- 2006, 103:19616–19623. sity Press; 1990, 346. 78. Vogel C, O’Brien K. Who can eat information? Exam- 92. Lempert RJ, Groves DG, Popper SW, Bankes SC. A ining the effectiveness of seasonal climate forecasts general, analytic method for generating robust strate- and regional climate risk-management strategies. Clim gies and narrative scenarios. Manage Sci 2006, 52: Res 2006, 33:111–122. 514–528. 79. Lemos MC. What influences innovation adoption by 93. Polasky S, Carpenter SR, Folke C, Keeler B. Decision- water managers? Climate information use in Brazil making under great uncertainty: environmental man- and the US. J Am Water Resour Assoc 2008, 44: agement in an era of global change. Trends Ecol Evol 1388–1396. 2011, 26:398–404.

94. Hall J. Probabilistic climate scenarios may misrepre- 108. US Climate Change Program (CCSP). Best practice sent uncertainty and lead to bad adaptation decisions. approaches for characterizing, communicating, and Hydrol Proc 2007, 21:1127–1129. incorporating scientific uncertainty in decisionmaking. 95. Ranger N, Millner A, Dietz S, Fankhauser S, Lopez In: Morgan M, Dowlatabadi H, Henrion M, Keith D, A, Ruta G. Adaptation in the UK: A Decision-Making Lempert R, McBride S, Small M, Wilbanks T, eds. U.S. Process. Policy brief, Centre for Climate Change Eco- Climate Change Science Program and the Subcommit- nomics and Policy, Leeds and London, UK; 2010. tee for Global Change Research, Washington, DC. 96. Brown C. The end of reliability. ASCE J Wat Res 109. Ravetz JR. Models as metaphors. In: Kasemir B, Jager Plann Manage 2010, 2:146–148. J, Jaeger CC, Gardner MT, eds. Public Participation 97. Brown C, Ghile Y, Laverty M, Li K. Decision scaling: in Sustainability Science. Cambridge, UK: Cambridge Linking bottom-up vulnerability analysis with climate University Press; 2003, 62–78. projections in water sector. Water Resour Res 48:1–12 110. Oreskes N, Shrader-Frechette K, Belitz K. Verification, doi:10.1029/ 2011WR011212. validation, and confirmation of numerical models in 98. Brown C, Werick W, Leger W, Fay D. A decision- the earth sciences. Science 1994, 263:641–646. analytic approach to managing climate risks: applica- 111. Demeritt D. The construction of global warming and tion to the Upper Great Lakes. J Am Water Resour the politics of science. AnnAssocAmGeogr2001, 91: Assoc 2011, 47:524–534. 307–337. 99. Carter TR, Jones RN, Lu SBX, Conde C, Mearns 112. Sterman JD. Risk communication on climate: men- LO, O’Neill BC, Rounsevell MDA, Zurek MB. New tal models and mass balance. Science 2008, 322: assessment methods and the characterisation of future 532–533. conditions. In: Parry ML, Canziani OF, Palutikof JP, 113. Sterman JD. All models are wrong: Reflections on Linden P. J. v. d., Hanson CE, eds. Climate Change becoming a systems scientist. Syst Dyn Rev 2002, 18: 2007: Impacts, Adaptation and Vulnerability. Contri- 501–531. bution of Working Group II to the Fourth Assessment Report of the Intergovernmental Panel on Climate 114. Oreskes N. Why predict? Historical perspectives on Change. Cambridge, UK: Cambridge University Press; prediction in earth science. In: Sarewitz D, Pielke RA 2007, 33–171. Jr, Byerly R Jr, eds. Prediction: Science, Decision Mak- ing, and the Future of Nature. Washington, DC: Island 100. Dessai S, Hulme M. Assessing the robustness of adap- Press; 2000, 405. tation decisions to climate change uncertainties: a case study on water resources management in the East of 115. Bankes S. Exploratory modeling for policy analysis. England. Glob Environ Change 2007, 17:59–72. Oper Res 1993, 41:435–449. 101. Wilby RL, Dessai S. Robust adaptation to climate 116. Jones RN. An environmental risk assessment/ change. Weather 2010, 65:180–185. management framework for climate change impact 102. Groves DG, Lempert RJ. A new analytic method assessments. Nat Hazards 2001, 23:197–230. for finding policy-relevant scenarios. Glob Environ 117. Brekke LD, Maurer EP, Anderson JD, Dettinger MD, Change 2007, 17:73–85. Townsley ES, Harrison A, Pruitt T. Assessing reservoir 103. Lempert RJ, Collins MT. Managing the risk of uncer- operations risk under change. Water Resour Res 2009, tain threshold responses: Comparison of robust, opti- 45:W04411. doi:10.1029/2008WR006941. mum, and precautionary approaches. Risk Anal 2007, 118. Pielke RA Sr. Ben K, Brasseur G, Calvert J, Chahine 27:1009–1026. M, Dickerson RR, Entekhabi D, Foufoula-Georgiou 104. Lempert RJ, Groves DG. Identifying and evaluating E, Gupta H, Krajewski W, et al. Climate change: the robust adaptive policy responses to climate change need to consider human forcings besides greenhouse for water management agencies in the American west. gases. EOS 2009, 90:413. Technol Forecast Soc Change 2010, 77:960–974. 119. West JM, Julius SH, Kareiva P, Enquist C, Lawler 105. Lempert RJ, Sriver RL, Keller K. Characterizing Uncer- JJ, Petersen B, Johnson AE, Shaw MR. US nat- tain Sea Level Rise Projections to Support Investment ural resources and climate change: concepts and Decisions. Prepared for California Energy Commis- approaches for management adaptation. Environ sion Public Interest Energy Research Program; 2012. Manag 2009, 44:1001–1021. 106. Hall J, Lempert RJ, Keller K, Hackbarth A, Mijere C, 120. Lempert R, Kalra N. Managing climate risks in devel- McInerney DJ. Robust climate policies under uncer- oping countries with Robust Decision Making. World tainty: a comparison of robust decision making and Resources Report. Washington, DC, 2012. Available Info-gap methods. Risk Anal 2012, 32:1657–1672. at: http://www.worldresourcesreport.org. doi:10.1111/j1539-6924.2012.01802.x. 121. McDaniels T, Mills T, Gregory R, Ohlson D. Using 107. Ben-Haim Y. Information-Gap Decision Theory: expert judgments to explore robust alternatives for Decisions under Severe Uncertainty. New York, NY, forest management under climate change. Risk Anal USA: Academic Press; 2001. 2012. doi:10.1111/j1539-6924.2012.01822.

122. Schwartz P. The Art of the Long View—Planning for sea level extremes along the California coast. Clim the Future in an Uncertain World. New York, NY: Change 2008, 87:S57–S73. Currency-Doubleday; 1996. 136. Pollard D. A retrospective look at coupled ice sheet- 123. Lempert RJ. Scenarios that illuminate vulnerabilities climate modeling. Clim Change 2010, 100:173–194. and robust responses. Clim Change 2012. In press. 137. Rosenzweig C. Climate Change Adaptation in New 124. Milly PCD, Betancourt J, Falkenmark M, Hirsch RM, York City: Building a Risk Management Response. Kundzewicz ZW, Lettenmaier DP, Stoffer RJ. Station- New York: 2010. arity is dead: Whither water management? Science 138. Oppenheimer M, O’Neill BC, Webster M, Agrawala 2008, 319:573–574. S. The limits of consensus. Science 2007, 317: 125. Inland Empire Utilities Agency (IEUA). Regional 1505–1506. Urban Water Management Plan. Chino, CA.: Inland 139. Rahmstorf S. A new view on sea level rise. Nat Rep Empire Utilities Agency; 2005. Clim Change 2010, 4:44–45. 126. Groves DG, Yates D, Tebaldi C. Developing and 140. Pfeffer WT, Harper JT, O’Neel S. Kinematic con- applying uncertain global climate change projec- straints on glacier contributions to 21st-century sea- tions for regional water management planning. level rise. Science 2008, 321:1340–1343. Water Resour Res 2008, 44:W12413. doi:10.1029/ 2008WR006964. 141. Co-CAT. State of California Sea-Level Rise Interim Report, edited, Coastal and Ocean Working Group of 127. Yates D, Purkey D, Sieber J, Huber-Lee A, Galbraith the California Action Team, 2010. Available at: www. H. WEAP21—a demand-, priority-, and preference- ... driven water planning model: part 1. Water Int 2005, opc.ca.gov/webmaster/ftp/project /SLR_Guidance_ 30:487–500. Document.pdf. 128. Bryant BP, Lempert RJ. Thinking inside the box: 142. Rahmstorf S. A semi-empirical approach to projecting a participatory, computer-assisted approach to sce- future sea-level rise. Science 2007, 315:368–370. nario discovery. Technol Forecast Soc Change 2010, 143. Wilby RL. Wells of wisdom. Nat Clim Change 2011, 77:34–49. 1:302–303. 129. Millner A. Climate prediction for adaptation: who 144. Prudhomme C, Wilby RL, Crooks S, Kay AL, Rey- needs what?. Clim Change 2012, 110:143–167. nard NS. Scenario-neutral approach to climate change 130. National Research Council (NRC). Abrupt Climate impact studies: application to ﬂood risk. JHydrol Change: Inevitable Surprises. Washington, DC, USA: 2011, 390:198–209. National Academies Press; 2002. 145. Held IM. The gap between simulation and understand- 131. Schneider SH, Semenov S, Patwardhan A, Burton I, ing in climate modeling. Bull Am Meteorol Soc 2005, Magadza CHD, Oppenheimer M, Pittock AB, Rahman 86:1609–1614. A, Smith JB, Suarez A, et al. Assessing key vulnerabil- 146. U.S. Global Change Research Program (USGCRP). ities and the risk from climate change. In: Parry ML, Scenarios for Research and Assessment of Our Canziani OF, Palutikof JP, van der Linden PJ, Hanson Climate Future: Issues and Methodological Per- CE, eds. Climate Change 2007: Impacts, Adaptation spectives for the U.S. National Climate Assessment. and Vulnerability. Contribution of Working Group National Climate Assessment Report Series: Vol- II to the Fourth Assessment Report of the Intergov- ume 6. U.S. Global Change Research Program, ernmental Panel on Climate Change. Cambridge, UK: Washington, DC, USA, 2011. Available at: http:// Cambridge University Press; 2007, 779–810. library.globalchange.gov/national-climate-assessment- 132. Keller K, Schlesigner M, Yohe G. Managing the risk scenarios-for-research-and-assessment-of-our-climate- of climate thresholds: uncertainties and information future needs. Clim Change 2008, 91:5–10. 147. Revell DL, Battalio R, Spear B, Ruggiero P, Vandever 133. Lenton TM, Held H, Kriegler E, Hall JW, Lucht W, J. A methodology for predicting future coastal hazards Rahmstorf S, Schellnhuber HJ. Tipping elements in the due to sea-level rise on the California Coast. Clim Earth’s climate system. Proc Natl Acad Sci USA 2008, Change 2011, 109:S251–S276. doi:10.1007/s10584- 105:1786–1793. 011-0315-2. 134. Milne GA, Gehrels WR, Hughes CW, Tamisiea ME. 148. Murphy JM, Booth BBB, Collins M, Harris GR, Identifying the causes of sea-level change. Nat Geosci Sexton DMH, Webb MJ. A methodology for prob- 2009, 2:471–478. abilistic predictions of regional climate change from 135. Cayan DR, Bromirski PD, Hayhoe K, Tyree M, Det- perturbed physics ensembles. Phil Trans R Soc A 2007, tinger MD, Flick RE. Climate change projections of 365:1993–2028.