Ranking Biodiversity Risk Factors Using Expert Groups–Treating Linguistic

MURDOCH RESEARCH REPOSITORY

This is the author’s final version of the work, as accepted for publication following peer review but without the publisher’s layout or pagination. The definitive version is available at http://dx.doi.org/10.1016/j.biocon.2013.03.005

Metcalf, S.J. and Wallace, K.J. (2013) Ranking biodiversity risk factors using expert groups – Treating linguistic uncertainty and documenting epistemic uncertainty. Biological Conservation, 162 . pp. 1-8.

http://researchrepository.murdoch.edu.au/17256/

It is posted here for your personal use. No further distribution is permitted.

1 Ranking biodiversity risk factors using expert groups – treating linguistic 2 uncertainty and documenting epistemic uncertainty

3 Metcalf, S.J.a1 and Wallace, K.J.ab

4 a Natural Resources Branch, Department of Environment and Conservation, Locked Bag 104, 5 Bentley Delivery Centre, WA 6983, Australia , [email protected], 6 [email protected]

7 b Future Farm Industries Cooperative Research Centre, The University of Western Australia, 35 8 Stirling Hwy, Crawley, WA 6009, Australia

9 Corresponding author: S.J. Metcalf, School of Management and Governance, Murdoch 10 University, 90 South Street, Murdoch, Perth, Western Australia, Australia 6150. Ph. +61 8 9360 11 6344, Fax. +61 8 9360 6966, [email protected]

1Present address: Murdoch Business School, Murdoch University, 90 South Street, Murdoch, Perth, WA, Australia 6150.

13 Abstract

14 Sound planning is vital to ensure effective management of biodiversity, particularly where

15 there is a high risk that management goals may not be achieved. This is the case at Toolibin

16 Lake, an internationally recognised wetland, where changed hydrology as a result of

17 agricultural development has detrimentally affected the quality and quantity of water entering

18 the lake. Although management actions have slowed or halted degradation of the lake’s

19 biological assets, goals have not been fully achieved and management is under review. To rank

20 the hydrological risk factors threatening the lake’s biota as a foundation for more detailed

21 planning, a structured elicitation process was used with an expert, cross-disciplinary group.

22 Techniques used were explicitly aimed at minimising and documenting uncertainty. These

23 included calibration questions to assess the accuracy of experts, and tightly specified goals and

24 terms to minimise vagueness, ambiguity and redundancy. Surface water salinity, groundwater

25 salinity and drought were the only factors identified as having a high probability of causing

26 goal failure. Importantly, the majority of risk factors were evaluated as having a low probability

27 of causing goal failure, enabling these factors to be rapidly eliminated from short-term

28 consideration. The experts, acting anonymously when estimating probabilities, varied

29 considerably in the assessment of one risk. This underlines the importance of rigorous

30 processes to identify knowledge gaps and assess the likelihood that proposed management

31 will be successful. The elicitation process provided low cost, rapid assessment of large

32 numbers of risk factors with explicit assessment of uncertainty.

34 Keywords: Elicitation, expert, Ramsar, hydrology, risk assessment, uncertainty.

39 1. Introduction

40 Resources must be wisely allocated through effective planning and priority setting to achieve

41 goals for biodiversity management (Joseph et al., 2009). Sound planning is particularly crucial

42 where important biological assets are highly threatened, resources are limited, and goal failure

43 is thus a distinct possibility. Here, we define a goal as “the desired outcome of management

44 constrained in both space and time” (Wallace, 2012, page 4). Biological management goals are

45 frequently stated in terms of biotic composition – for example, conserving the existing biota or

46 biodiversity of a specific area for a stated planning period, as in this case study for Toolibin

47 Lake. Thus, failure to achieve a goal stated in specific terms within a planning period

48 constitutes goal failure.

50 Conservation goals are often difficult to achieve in the south-west of Western Australia where

51 extensive replacement of native perennial vegetation with annual crops and pastures has

52 significantly altered hydrological processes. Processes associated with altered hydrology that

53 threaten native taxa include rising saline groundwater, secondary salinisation of soils and

54 surface waters, nutrient enrichment, and altered acidity/alkalinity concentrations (Horwitz et

55 al., 2008). Under these circumstances, and given both the high cost of managing salinity

56 (Sparks et al., 2006) and lack of knowledge concerning catchment processes and biota (Wallace

57 et al., 2011), management goals that aim to retain the existing composition of biota in valley

58 floors and associated wetlands are ambitious and the potential for goal failure is high. This is

59 underlined by the continued decline of wetland biota in the south-west of Western Australia

60 (Clarke et al., 2011). Under such demanding circumstances it is essential to assess the

61 likelihood of achieving management goals – there is generally little point expending resources

62 on management where success is highly unlikely. Rather, alternatives such as changing the

63 goal of management, re-allocating resources, or undertaking further research should be

64 considered (Wallace, 2012).

66 Although there are cases, such as the Everglades in Florida, where management planning is

67 based on detailed science and associated numerical models (see, for example, The Everglades

68 Foundation, 2011), in many places, including the agricultural landscapes of southern Australia,

69 there is much less information and management is based on some combination of expert

70 opinion and existing data (Walshe and Massenbauer, 2008; Jones et al., 2009; Pannell et al.,

71 2012). Burgman (2005) has noted that where decisions concerning operational management

72 must be made in the context of inadequate knowledge and quantitative modelling is

73 impractical, the use of expert analysis is warranted. However, in relation to using expert

74 elicitation, Martin et al. (2012, page 35) state that “formal methods for eliciting and combining

75 judgments only recently have been adopted and tested for application to conservation science

76 and practice.” One aim of this paper is to provide an applied case study of expert elicitation in

77 planning the operational management of biological assets.

79 The work described below is consistent with the biodiversity planning framework described by

80 Wallace (2012). This framework explicitly links the human values driving planning and

81 management with goal formulation, description of biological assets, evaluation of key

82 ecosystem processes (risk factors), and assessment of management feasibility. In this paper we

83 focus on the first of three points required to determine management feasibility once goals and

84 biological assets of management interest have been determined. These points are:

85 • determine the key ecosystem processes (or key risk factors) that should be managed to

86 ensure a high probability of goal achievement (in the case of Toolibin Lake, the assessment

87 assumed that current management practices were maintained, and that no additional

88 management was implemented);

89 • for key processes or factors, undertake investigations and numeric modelling sufficient to

90 describe additional management actions necessary for success; and

91 • combine information from the first two points with other data to complete a feasibility

92 analysis and decide whether to proceed with management.

94 To address the first point when planning the management of Toolibin Lake, we applied an

95 expert elicitation process to generate a risk analysis based on subjective judgement. Experts

96 from different scientific disciplines, operational managers and planners were involved to lay

97 the foundation for a fully cohesive process involving all technical experts. Work reported in

98 this paper will provide the basis for further analyses addressing the second and third dot

99 points above.

100

101 When using experts, the uncertainty associated with expert opinion should be explicitly

102 evaluated (Burgman et al., 1999) so that appropriate levels of confidence, which may differ

103 according to the goals of each study, are attached to analyses used in decision-making. In this

104 case study, one aim was to reduce and explicitly document uncertainty through the use of the

105 Threat Elicitation process developed by the Australian Centre for Excellence in Risk Analysis

106 (ACERA, 2010), which involves discussion and knowledge exchange amongst experts.

107

108 Such threat elicitation processes are largely aimed at reducing and documenting epistemic

109 (knowledge) uncertainty, which relates to incomplete knowledge concerning the state of a

110 system (Regan et al., 2002; Burgman, 2005). For example, the sharing of information and ideas

111 amongst a cross-disciplinary group of experts can be expected to increase overall knowledge

112 of natural variation and improve understanding of interacting processes, thus reducing

113 uncertainty. Increased knowledge of a system can also enhance subjective judgement, or at

114 least ensure the related uncertainties are well documented.

115

116 It is also important to deal with linguistic uncertainty, which arises because language is inexact

117 leading to issues such as vagueness, context dependence, ambiguity and underspecificity

118 (Regan et al., 2002; Burgman, 2005). For example, where definitions of terms are unclear, or

119 there is ambiguity concerning the allocation of factors within classification systems, one or

120 more types of linguistic uncertainty will occur. A number of techniques may be used to

121 minimise linguistic uncertainty (Speirs-Bridge et al., 2010; Patrick and Damon-Randall, 2008).

122 In this case study, the planning approach outlined by Wallace (2012) was applied to reduce

123 linguistic uncertainty by ensuring that goals are fully specifed, terms clearly defined, and

124 ambiguity and redundancy in classification of risks avoided.

125

126 Using an expert analysis of the risks to biota at Toolibin Lake arising from hydrological

127 processes, the objectives of this paper are to: evaluate key risk factors; qualitatively assess the

128 usefulness of methods applied to minimise linguistic uncertainty; and evaluate the

129 contribution of expert elicitation, including measures of uncertainty, to management planning.

130 The results of this study will directly contribute to a revised management plan for Toolibin Lake

131 and its biota, and provide a case study of applied elicitation.

132

133 2. Methods

134 2.1 Study system

135 Toolibin Lake (approximately 300 ha) is an ephemeral, fresh to brackish wetland in south-west

136 Western Australia. It is approximately 200 km south-east of Perth within the inland agricultural

137 region and is the last significant representative of a once common wetland community. The

138 lake bed has large areas covered by two tree species, most commonly Casuarina obesa, and to

139 a lesser extent Melaleuca strobophylla, nationally listed as a Threatened Ecological

140 Community. Fifty species of waterbirds have been recorded visiting the lake and, of these, 25

141 species have been observed breeding. The lake is listed as a Wetland of International

142 Importance under the Ramsar Convention.

143

144 The biological assets of the lake are severely threatened by altered hydrology, particularly

145 increasingly saline surface water inflows and rising, highly saline groundwater. Inflow

146 diversion, groundwater pumping and catchment works are being implemented to halt

147 degradation of the biological assets. A new plan is being drafted to reassess the actions

148 required to achieve the goal, and to evaluate management feasibility of proposed actions. This

149 risk analysis was undertaken as part of this planning process. Ultimately, all risk factors will be

150 analysed for Toolibin Lake. However, as altered hydrology is considered most likely to result in

151 goal failure, it was selected for detailed work as reported here.

152

153 2.2 Reducing linguistic uncertainty

154 To reduce linguistic uncertainty when working with the expert group, the assessment task was

155 tightly specified by the authors in consultation with some of the experts involved in the

156 elicitation. The risk analysis was linked to a spatially and temporally defined management goal,

157 as these bounds are critical to effectively specify the evaluation task (Wallace, 2006). Also, any

158 vague or ambiguous terms were either rigorously defined or removed to minimise uncertainty.

159 The risk analysis was couched in terms of probability of goal failure arising from each risk

160 factor. Given that risk of goal failure cannot be entirely avoided, this approach emphasises to

161 decision-makers the importance of considering the level of risk they are willing to accept.

162

163 For Toolibin Lake, the management goal is: “to maintain the taxonomic composition of biota

164 identified in current species lists from Toolibin Lake over the next 25 years”. This specifies

165 what needs to be conserved and the temporal period, but, as noted, the spatial scale is also

166 important. In discussion with workshop participants, Toolibin Lake was defined as the lake bed

167 in addition to the shoreline occupied by Casuarina obesa and Melaleuca species. These

168 riparian trees were included in the assessment as they occur on the lake bed, and their spatial

169 distribution is tightly coupled with hydrological processes in the area. This definition ensured

170 that the physical boundary of the analysis was unambiguous. The temporal scale was selected

171 as it was considered short enough for experts to comprehend the likely consequences of

172 change, and long enough for change to be possible and management actions to have an

173 impact.

174

175 The number and range of biological assets encompassed by the goal are complex. Therefore,

176 to ensure experts considered the same set of assets when making their assessements,

177 indicator species were identified including plants and invertebrates (Table 1). Indicators were

178 obtained from several experts prior to the workshop, and were sent to all participants for

179 comment and formal agreement at the workshop. Indicators were selected on the basis that

180 they would be the most sensitive to the factors considered and thus indicative of the

181 persistence of other species within the ecosystem. To further reduce any ambiguity concerning

182 the meaning of goal failure, the question addressed by experts for Toolibin was interpreted as:

183 What is the probability of any one species becoming extinct at the Lake within 25 years? The

184 scope of estimation was simplified by confining the goal to the recorded species richness,

185 rather than some minimum bound on abundance for each species present at the lake.

186

187 There are a range of issues relating to clarity in threat classifications (e.g., Balmford et al.,

188 2009; Salafsky et al., 2009). Given that unclear boundaries in classifications may lead to

189 double-counting and related issues (Wallace, 2012), the analysis was couched in terms of risk

190 factors that directly affect biota and thus the achievement of the management goal. Thus, the

191 risk factors evaluated were the stresses that directly impinge on the capacity of native

192 organisms to survive and reproduce (e.g., predation, salinity, drought, disease).

193

194 Prior to the risk analysis workshop, a list of all potential risk factors that could arise from

195 altered hydrology was developed by the facilitators based on Wallace (2012, see Table 4). This

196 list was assessed by two experts and missing items or suggested alterations were incorporated

197 (Table 2). As these experts were also participants in the risk analysis workshop, the only

198 additional information they received prior to the workshop was the spatial and temporal scale

199 of the study and the list of potential key factors associated with altered hydrology.

200

201 2.3 Risk analysis

202 Ten experts in the biology, hydrology and operational management of Toolibin Lake attended

203 the elicitation workshop. Presentations regarding management goals for Toolibin Lake, as

204 defined above, as well as definitions of spatial and temporal scales and an explanation of risk

205 analysis were undertaken prior to the elicitation.

206

207 The 4-step ‘Threat Elicitation’ used for the risk analysis was developed at ACERA and has been

208 shown to reduce overconfidence in expert estimates (Speirs-Bridge et al., 2010), while allowing

209 the estimation of uncertainty. Calibration questions were provided prior to the elicitation to

210 allow the calculation of weights for different subjects and groups of participants. These

211 questions were based on plants, birds, invertebrates, groundwater, rainfall and transpiration,

212 and were specifically selected because the answers were unlikely to be precisely known (i.e. an

213 estimate with higher and lower bounds was required). For example, the first calibration

214 question was: What proportion of the total number of plant species, including introduced

215 plants, in Toolibin lake and reserves (Dulbining + Toolibin) do plants of the Family Myrtaceae

216 comprise?

217  What is your lowest estimate of the proportion?

218  What is your highest estimate of the proportion?

219  What is your best estimate?

220  What is your level of confidence that the true value lies in the range you have

221 provided? (0-100%)?

222 (See Appendix for full calibration question list)

223

224 Risk analysis questions were constructed based on key factors (Table 2) and all questions were

225 of the same format to ensure comparability of results for the prioritisation of risks. Elicitation

226 questions were of the structure: “With current management strategies in place, what is your

227 lowest estimate of the probability that [key factor] will cause species loss (i.e. goal failure) at

228 Toolibin Lake over the next 25 years?” (Step 1). This question was repeated separately for

229 highest (Step 2) and best estimates (Step 3). Participants were also asked “What is your level of

230 confidence that the true value lies in the range you have provided?” (Step 4) following the

231 probability questions. This final question was modified from the original documented in Speirs-

232 Bridge et al. (2010) as experts found the original wording difficult to understand. Initial

233 anonymous responses were collected by the authors and compiled in graphical form for

234 display to the group. This was undertaken to encourage and stimulate discussion regarding

235 similarities or differences. Each participant could then choose to provide modified (second)

236 estimates for all four steps. The final estimates were also made anonymously.

237

238 2.4 Data analysis

239 2.4.1 Calibration: Confidence, hit rates and accuracy

240 All intervals estimated by workshop participants were arcsine transformed which shifted

241 extreme estimates further away from the mean to aid statistical analysis using a normal

242 distribution (Speirs-Bridge et al., 2010). This type of transformation is generally used when the

243 distribution of data points is skewed and bounded by zero and one, as is often the case with

244 proportions and probabilities, and when there are a large number of values close to one or

245 close to zero (Sokal and Rohlf, 1981), as in this study. Following this step, 83% confidence

246 intervals, whereby the true value should lie within the interval in five of six estimates provided

247 by an expert, were derived using the methods of Speirs-Bridge et al. (2010). Hit rates were

248 assessed, where a ‘hit’ was an interval containing the true value. Percent over- and under-

249 confidence was determined using the number of calibration questions for which a ‘hit’ was

250 obtained. The derived 83% confidence intervals (calculated from participants’ own confidence

251 levels) were used in this comparison and a perfectly calibrated participant would have

252 provided intervals containing the true value in five from six questions. While a larger number

253 of calibration questions would have been preferable, the number was limited to six due to

254 time constraints for undertaking the analysis.

255

256 Weighting expert opinion according to calibration performance has been commonly reported

257 in the literature (Cooke, 1991; Goosens et al., 1996). In this study, weights for probabilities

258 provided during the elicitation were based on calibration question hit rates and over- or under-

259 confidence for each participant. To normalise the calibration weights, the raw weight was

260 divided by the average from all participants. Estimated probabilities were weighted to ensure

261 responses from better calibrated individuals had greater influence than those from less well-

262 calibrated individuals.

263

264 2.4.2 Elicitation

265 For each factor (Table 2), experts were first asked whether the probability of species loss was

266 very low (<0.05, a definition accepted by the experts present during the evaluation). If there

267 was consensus that the probability was very low, the full risk analysis was not undertaken for

268 that factor. A probability of 0.05 was selected as the cut-off for inclusion in the risk analysis as

269 management intervention would be unlikely to be implemented for these factors in

270 comparison to any higher risk factors (> 0.05). Following the determination of a very low

271 probability of goal failure, an estimate for the actual probability of species loss was determined

272 by the group as a whole and the reasoning recorded. If consensus was not achieved, the full

273 elicitation was undertaken.

274 During the elicitation, some participants chose not to provide second estimates following

275 discussion. No participants provided second estimates for surface water salinity. When second

276 estimates were provided, a qualitative assessment of differences between the first and second

277 unweighted estimates and variances was undertaken. The ranking of the three factors with the

278 highest probability of causing goal failure was also compared using weighted and unweighted

279 estimates.

280 In the one case of divergent views amongst two groups of participants, the average weight

281 associated with participants in each group was used to identify the view with the highest

282 weight. These analyses were not undertaken for key factors where there was no clear

283 indication of divergent views, as opposed to simple differences between individuals.

284

285 3. Results

286 3.1 Calibration: Confidence, hit rates and accuracy

287 The average hit-and-miss calibration using the 80% derived confidence intervals was 65% (i.e.

288 average 15% overconfidence). Four of the ten participants provided intervals that were

289 perfectly calibrated to the 83% confidence interval (83% hit rate) with five out of six intervals

290 containing the true value. One participant had a hit rate of four from six and were 17%

291 overconfident, while two participants had a hit rate of 3 from 6 (33% overconfidence) and a

292 further two participants had a low hit rate of 1 from 6 (66% overconfidence). Only one

293 participant provided intervals that contained the true value for all six questions. This

294 participant was underconfident (17%) in their responses. Normalising the hit rates from the

295 calibration questions provided the final weights for each participant (Table 3).

296

297 3.2 Elicitation results

298 The majority of key factors assessed during the workshop were determined to be unlikely to

299 cause any species loss (Table 4). Most were allocated expert probabilities of causing goal

300 failure of less than 0.05 during the workshop. The remainder were analysed using the full

301 elicitation process.

302

303 Using expert estimates weighted according to calibration performance, the risk analysis

304 identified surface water salinity as the key factor with the highest probability of causing

305 species loss at Toolibin Lake (Best estimate: 0.411, 83% CI: 0.235 – 0.619). The second highest

306 probability was obtained for a lack of water (Best estimate: 0.336, 83% CI: 0.154 – 0.559)

307 followed by ground water salinity (Best estimate: 0.282, 83% CI: 0.133 – 0.454). The same

308 ranking was found when using weighted or unweighted estimates.

309 There was a clear separation of participants into two groups for the probability of goal failure

310 due to a lack of water (Table 4, Fig. 1). The average best estimates for the different groups

311 were 0.548 (Group A) and 0.202 (Group B). The lower estimate of Group B arose due to the

312 possibility (identified during the workshop) that species are able to cope with a lack of water,

313 as observed in the severe drought of 2010 (Table 4). Linking each participant with their

314 assigned weight was undertaken with the aim of determining the more probable stream of

315 thought, however, the average weight of participants was 0.099 for Group A and 0.102 for

316 Group B. As the average weight of both groups was almost equal, the average best estimate

317 from all participants as reported previously was deemed to be appropriate to allow a ‘middle

318 of the road’ approach by operational managers.

319

320 When second estimates were provided, the average best estimates were reduced. For

321 example, the weighted mean best estimate for the probability of species loss due to ground

322 water salinity was initially 0.342 and was reduced to 0.282 following discussion. The weighted

323 mean best estimate for a lack of water (0.419) was also reduced following discussion (0.336).

324 Variance between unweighted best estimates was reduced in the second estimates for

325 groundwater salinity (initial: 0.025, secondary: 0.021) and lack of water (initial: 0.046,

326 secondary: 0.038).

327

328 4. Discussion

329 The study showed that the elicitation technique is a low-cost means of assessing risks to goal

330 achievement. The expert group dealt with multiple factors efficiently and, crucially,

331 determined that the majority of factors have a very low probability of causing goal failure. This

332 is a critical step in early planning, particularly where management resources are limited.

333 Following this assessment, planners and managers could be confident that, given current

334 knowledge, low risk factors could be ignored in the short-term allowing a focus on the key

335 factors considered to have the greatest probability of causing goal failure. It must be noted,

336 however, that managers and planners should regularly re-assess all potential risk factors to

337 minimise unexpected impacts and account for changes in knowledge arising from new

338 information.

339

340 The risk analysis identified surface water and groundwater salinity as two of the three key

341 factors with a high probability of causing species loss, and thus goal failure, at Toolibin Lake.

342 Increasingly saline surface water or rising, saline groundwaters may result in mortality and

343 ultimately the loss of species. For example, whether from groundwater or surface waters, the

344 infiltration of highly saline water into the root zones of plants may cause widespread mortality.

345 A lack of water (drought) was also identified as having a high probability of causing goal failure

346 and there is evidence that the survival of mature trees and seedlings is increasingly dependent,

347 given decreasing inflow events, on a declining incident, annual rainfall (Ogden and Froend,

348 2002; Pitman et al., 2004). The results concerning drought revealed substantially different

349 views of this risk. One view was that current management (diversion of highly saline water)

350 plus increased frequency of dry years was likely to cause the loss of species from Toolibin Lake.

351 The other view was that after the extreme drought of 2010, species loss in the next 25 years is

352 unlikely given the biota’s apparent capacity to survive. The generation of risk scores from a

353 discussion aimed at achieving consensus or by immediately averaging scores, as used in the

354 past, would have obscured this important result. The difference in views and the wide ranging

355 estimates indicate there is significant uncertainty concerning the likelihood of goal failure

356 arising from these three key hydrological factors. This underlines that further investigation of

357 key risk factors is essential before a full feasibility analysis may be completed.

358 Apart from the knowledge outlined above, the elicitation technique provided a number of

359 other benefits. For example, averaging estimates amongst different people, as well as between

360 two responses from the same person, can be beneficial because the estimates are likely to

361 differ in their systematic and random errors and the errors tend to cancel out across all

362 participants (or estimates for a single participant), improving the overall estimate (Herzog and

363 Hertwig, 2009). Also, the use of structured group discussion ensured all participants were privy

364 to the same information, thereby reducing both linguistic and epistemic uncertainty. Group

365 discussion centred on new information and on those aspects where there was greatest

366 uncertainty and divergent views. This included cross-disciplinary sharing of knowledge which,

367 for example, allowed autoecological knowledge of biota to be combined with what is

368 hydrologically and operationally practical to implement, thus facilitating a more effective

369 assessment. In this context, the identification and consideration of opposing views has been

370 found to improve second estimates (Herzog and Hertwig, 2009) because no two individuals are

371 likely to have the same knowledge and the provision of additional knowledge or counter-

372 arguments allows for the consideration of disconfirming evidence by each participant.

373 Structuring the mode of questioning into four distinct steps is important here as it should

374 involve the reconsideration of all evidence at each step (question) and therefore brings semi-

375 independent information into the forum. Without semi-independent questioning,

376 overconfidence is generally increased as a result of confirmation bias (Soll and Klayman, 2004).

377 Finally, steps taken to minimise linguistic uncertainty prior to the elicitation were also

378 particularly valuable, including the specification of the goal for biodiversity in terms of explicit

379 outcomes within spatial and temporal bounds. Similarly, the reduction of assessments to well

380 defined factors that form a logically coherent set that impinge directly on organisms was

381 readily understood by experts and focussed attention on direct stressors.

382

383 However, biases may also arise from elicitation processes. Discussion amongst participants

384 during the elicitation process was observed to influence second estimates, and this could be

385 attributed to a ‘halo effect’ (Nisbett and Wilson, 1977). This effect occurs when experts make

386 global assumptions about the knowledge of one expert and adjust their own estimate

387 accordingly. Other biases may arise by attributing higher confidence to the beliefs of confident

388 people regardless of whether they are knowledgeable, or by adjusting beliefs to reflect those

389 of people that can influence career progression (ACERA, 2010). However, the elicitation

390 process was specifically designed to reduce the effects of these potential biases, for example,

391 through the use of anonymous estimation of probabilities. In addition, the willingness of

392 experts to participate in the discussion may have reduced the impact of a halo effect by

393 facilitating reasoned arguments and identifying alternative points of view. The authors made

394 every attempt to ensure participants understood that all opinions were valid and should be

395 considered equally by the group. Consequently, it is likely that modifications during the second

396 round of estimates reflected improved certainty of knowledge arising from the discussion. This

397 was illustrated by the significant reduction in estimates produced following discussion of the

398 probability of species loss due to a lack of water (drought). A follow-up elicitation would be

399 valuable to clarify this point and determine if, similar to groundwater salinity and a lack of

400 water, the estimate for surface water salinity declined with second estimates.

401

402 Focussing on how the elicitation process might be improved in future, one issue is the use of

403 calibration questions. These are rarely used in environmental assessments utilising expert

404 opinion, and this has been attributed to the lack of evidence showing their benefit (Walker et

405 al., 2003). Calibration questions may also be problematic if the questions are not specifically

406 structured to reflect the experts’ knowledge , thus resulting in higher weight being attributed

407 to participants with a lesser knowledge of the system or with skewed beliefs (Burgman et al.,

408 2011). However, Burgman et al. (2011) also suggested that facilitators have a mandate to use

409 calibration questions to eliminate sources of uncertainty while ensuring that the scope of

410 questions relate directly to the knowledge being elicited. The weights calculated here from the

411 calibration questions were assumed to be appropriate because, although general, they directly

412 related to knowledge required for effective elicitation. Although there was no difference

413 between the ranking of weighted and unweighted key factors, the use of weights did reduce

414 the variance for estimates in two of the three key factors identified, thereby providing greater

415 overall precision. The divergence of views highlighted for a lack of water did indicate the need

416 for additional research as noted above. Overall, the use of calibration questions at least

417 provides a measure of whether experts are sufficiently knowledgeable to assess risk, and their

418 responses may then be used as a basis for comparison with other expert groups.

419 The elicitation method could be improved in a number of ways. Discussion of the trade-off

420 between accuracy and bias (Yaniv and Foster, 1997) may have resulted in fewer extreme

421 intervals and improved participant weightings. Also, better explanation and use of practice

422 questions may have increased the understanding of the process, thus preventing the exclusion

423 of participants from the final analysis. One participant suggested that, despite group

424 agreement concerning the goal being assessed, they were subconsciously considering the

425 magnitude of expected impacts and not solely the probability of goal failure as defined. This

426 type of response was identified by Slovic (1987) where the perception of risk was influenced by

427 the control people felt they had in managing the contributing processes, as well as the

428 desirability of goal achievement. Knowledge of such factors are crucial for interpreting expert

429 data and should be carefully managed in future elicitations. To some extent, this issue would

430 be addressed by defining outcomes in terms of types of taxa, their number and distribution.

431 However, this would require a much more detailed analysis and discussion by experts, and

432 needs to be balanced against the time available for the elicitation process. Time limitations

433 restricted the number of calibration questions used, and while the six questions posed to

434 participants were useful, a longer workshop with more time for calibration questions would be

435 beneficial.

436 Returning to the objectives set for the paper, the risk analysis has provided a useful evaluation

437 of key hydrological risk factors at Toolibin Lake and is an important step towards a feasibility

438 assessment for practical management in this data-limited situation. The use of expert opinion

439 is a valuable contribution to assessing risk, particularly where there is inadequate knowledge.

440 In this case study, expert analysis quantified uncertainty and provided a firm platform for

441 planning future investigations and modelling. The methods used to reduce linguistic

442 uncertainty, including a robust planning framework with rigorous goal definition and coherent

443 classification sets, were effective based on the qualitative assessment of the facilitators and

444 comments from experts. Despite these steps, this case study highlighted that for some risk

445 factors there may be substantial disagreement regarding the ability to achieve the

446 management goal. Further investigations and effective quantitative models are critical for

447 robust management decisions, particularly where, as at Toolibin Lake, substantial resources

448 and liabilities are involved with operational actions. Finally, important ancillary benefits from

449 the elicitation process included the sharing of knowledge and increased collective

450 understanding of the issues for achieving management goals for an important biological asset.

451

452

453 Acknowledgments

454 The authors would like to thank 10 experts involved in the elicitation workshop for their

455 valuable opinions. We would also like to thank Paul Drake, Michael Smith, Ryan Vogwill and

456 three anonymous referees for their useful comments on this manuscript, Adrian Pinder and

457 Mike Lyons for their assistance with indicator species, and Margaret Smith, Darren Farmer and

458 Jennifer Higbid for feedback on the ‘trial-run’ elicitation. This work was supported by the

459 Department of Environment and Conservation.

460

461

462 References

463 Australian Centre for Excellence in Risk Analysis (ACERA), 2010. Process manual: Elicitation

464 tool. University of Melbourne, 40pp.

465 www. acera.unimelb.edu.au/materials/endorsed/0611-process-manual.pdf

466 Balmford, A., P. Carey, Kapos, V., Manica, A., Rodrigues, A.S.L., Scharlemann, J.P.W., Green,

467 R.E., 2009. Capturing the many dimensions of threat: Comment on

468 Salafsky et al. Conserv. Biol. 23(2), 482-487.

469 Bell, D. T., 1999. Australian trees for the rehabilitation of waterlogged and salinity-damaged

470 landscapes. Aust. J. Bot. 47, 697-716.

471 Burgman, M. A., Keith, D. A., Walshe, T.V., 1999. Uncertainty in comparative risk analysis for

472 threatened Australian plant species. Risk Anal. 19(4), 585-598.

473 Burgman, M., 2005. Risks and Decisions for Conservation and Environmental Management.

474 Cambridge University Press, Cambridge.

475 Burgman, M. A., McBride, M. Ashton, R., Speirs-Bridge, A., Flander, L., Wintle, B., Fidler, F.,

476 Rumpff, L., Twardy, C., 2011. Expert status and performance. PLoS ONE 6(7), e22998.

477 Clarke, A., Lane, J., Jaensch, R., 2011. Surveys of Waterbirds in Selected Wetlands of South-

478 western Australia in Spring-Summer 2009-10 with Further Assessment of Changes in

479 Habitat and Waterbird Usage Over 2-3 Decades. Department of Environment and

480 Conservation, Western Australia.

481 Cooke, R.M., 1991. Experts in Uncertainty. Oxford University Press, Oxford.

482 Doupe, R. G., Horwitz, P., 1995. The value of macroinvertebrate assemblages for

483 determining priorities in wetland rehabilitation: A case study from Lake Toolibin,

484 Western Australia. J. R. Soc. WA 78, 33-38.

485 El-Lakany, M. H., Luard, E. J., 1983. Comparative salt tolerance of selected Casuarina

486 species [Australia; Egypt]. A. J. Forest Res. 13, 11-20.

487 Goossens, L. H. J., Cooke, R. M., Kraan, B. C. P., 1996. Evaluation of weighting schemes for

488 expert judgement studies. Report Prepared for the European Commission, Directorate

489 General for Science. Delft University of Technology, Delft, the Netherlands.

490 Halse, S. A., Pearson, G. P., McRae, J. M., Shiel, R. J., 2000. Monitoring aquatic invertebrates

491 and waterbirds at Toolibin and Walbyring Lakes in the Western Australian Wheatbelt.

492 J. R. Soc. W. Aust. 83, 17-28.

493 Herzog, S.M, Hertwig, R., 2009. The wisdom of many in one mind: Improving individual

494 judgements with dialectical bootstrapping. Psychol. Sci. 20(2), 231-237.

495 Horwitz, P., Bradshaw, D., Hopper, S., Davies, P., Froend, R., Bradshaw, F., 2008. Hydrological

496 Change escalates risk of ecosystem stress in Australia’s threatened biodiversity

497 hotspot. J. R. Soc. W. Aust. 91, 1-11.

498 Jones, S., Lacey, P., Walshe, T., 2009. A dynamic hydrological Monte Carlo simulation model to

499 inform decision-making at Lake Toolibin, Western Australia. J. Env. Mngt. 90, 1761-

500 1769.

501 Joseph, L.N., R.F. Maloney, and Possingham, H.P., 2009. Optimal allocation of resources among

502 threatened species: a project prioritization protocol. Conserv. Biol. 23, 328–338.

503 Martin, T.G., Burgman, M.A., Fidler, F., Kuhnert, P.M., Low-Choy, S., McBride, M., Mengersen,

504 K., 2012. Eliciting Expert Knowledge in Conservation Science. Conserv. Biol. 26, 29-38.

505 Nisbett, R.E., Wilson, T.D., 1977. The halo effect: Evidence for unconscious alteration of

506 judgments. J. Personality Soc. Psych. 35(4), 250-256.

507 Ogden, G., Froend, R.H., 2002. Vegetation Monitoring of Toolibin Lake and Surrounding

508 Reserves, 2002, report prepared by Edith Cowan University for the Department of

509 Conservation and Land Management.

510 Pannell, D.J., Roberts, A.M., Park, G., Alexander, J., Curatolo, A., Marsh, S.P., 2012. Integrated

511 assessment of public investment in land-use change to protect environmental assets in

512 Australia. Land Use Policy 29, 377-387.

513 Patrick, W.S., Damon-Randall, K., 2008. Using a five-factored structured decision analysis to

514 evaluate the extinction risk of Atlantic sturgeon (Acipenser oxyrinchus). Biol. Conserv.

515 141, 2906-2911.

516 Pitman, A.J., Narisma, G.T., Pielke Sr., R.A., Holbrook, N.J., 2004. Impact of land cover change

517 on the climate of southwest Western Australia. J. Geophys. Res. 109, D18109.

518 Regan, H.M., Colyvan, M., Burgman, M.A., 2002. A taxonomy and treatment of uncertainty for

519 ecology and conservation biology. Ecol. Appl. 12, 618-628.

520 Salafsky, N., Butchart, S. H. M., Salxer, D., Stattersfield, A.J., Neugarten, R., Hilton-Taylor, C.,

521 Collen, B., Master, L.L., O'Connor, S., Wilkie, D., 2009. Pragmatism and practice in

522 classifying threats: Reply to Balmford et al. Conserv. Biol. 23(2), 488-493.

523 Slovic, P., 1987. Perception of risk. Sci. 236, 280-285.

524 Sokal, R. R., Rohlf, F. J., 1981. Biometry: The Principles and Practices of Statistics in

525 Biological Research, 2nd ed., W. H. Freeman and Co., San Francisco.

526 Soll, J. B., Klayman.J, 2004. Overconfidence in interval estimates. J. Exp.Psych. 30, 299–

527 314.

528 Sparks, T., George, R., Wallace, K., Pannell, D., Burnside, D., Stelfox, L., 2006. Salinity

529 Investment Framework Phase II, Western Australia Department of Water, Salinity and

530 Land Use Impacts Series, Report No. SLUI 34, 86p.

531 http://www.water.wa.gov.au/PublicationStore/first/46153.pdf.

532 Speirs-Bridge, A., Fidler, F., McBride, M., Flander, L., Cumming, G., Burgman, M.A., 2010.

533 Reducing overconfidence in the interval judgments of experts. Risk Anal. 30(3), 512-

534 523.

535 The Everglades Foundation, 2011. The SERES Project, Review of Everglades Science, tools and

536 Needs Related to key Science Management Questions. The Everglades Foundation,

537 Florida.

538 http://resighin.ipower.com/Documents/SERES_Review_of_Everglades_Science_March

539 _2011_.pdf

540 Walker, K.D., Catalano, P., Hammitt, J.K., Evans, J.S., 2003. Use of expert judgment in exposure

541 assessment: Part 2. Calibration of expert judgments about personal exposures to

542 benzene. J. Expos. Anal. Env. Epidem. 13, 1-16.

543 Wallace, K. J., 2006. A decision framework for natural resource management: a case study

544 using plant introductions. Aust. J. Exp. Agric. 46, 1397-1405.

545 Wallace, K., Connell, K., Vogwill, R., Edgely, M., Hearn, R., Huston, R., Lacey, P., Massenbauer,

546 T., Mullan, G., Nicholson, N., 2011. Review: Natural diversity recovery catchment

547 program. Department of Envrionment and Conservation, Government of Western

548 Australia, Perth, 61pp.

549 Wallace, K. J., 2012. Values: drivers for planning biodiversity management. Env. Sci. Pol. 17, 1-

550 11.

551 Walshe, T., Massenbauer, T., 2008. Decision-making under climatic uncertianty: a case study

552 involving an Australian Ramsar-listed wetland. Ecol. Mngt. Rest. 9, 202-208.

553 Yaniv, I., Foster, D. P., 1997. Precision and accuracy of judgmental estimation. J.

554 Behav. Decision Making 10(1), 21-32.

555

556

557 Table 1. Categories and indicator species used in the elicitation process as identified by experts. The estimation of probability of goal failure was

558 undertaken for each category of species with these indicator species in mind. Note that some of the environmental limits are expert assessments only and

559 require further scientific confirmation.

Category Indicator species Reason for inclusion

Plants Casuarina obesa Sensitive to saline groundwater in root zone and soil salinity. Provides habitat for other species. Groundwater must be maintained at 4m below ground level. Flooding of the lake must be limited to 100 days and dry periods >8yr necessary for recruitment. Surface water salinity of <1500 mg/l required for survival (El-Lakany and Luard 1983).

Melaleuca strobophylla Sensitive to saline groundwater in root zone and soil salinity. Provides habitat for other species. Groundwater must be maintained at 8m below ground level. Flooding of the lake must be limited to 100 days and dry periods >8yr necessary for recruitment. Surface water salinity of <1500 mg/l required for survival (Bell 1999).

Invertebrates Glossophoniidae (leeches) Highly salt sensitive and recorded at Toolibin Lake in salinities of 2400 mg/L (Doupe and Horwitz 1995) but not at 9400 mg/L (Halse et al. 2000).

Conchostracans (Cyzicus spp. Very limited salt tolerance, with most species rarely occurring in salinities >1000-2000 mg/L. Cyzicus concostracans - clam shrimp) have been recorded upstream from Toolibin Lake, at salinities of 3500 mg/L (Doupe and Horwitz 1995) but there are no confirmed records of conchostracans in Toolibin Lake. Conchostracans are present in nearby Lake Bryde when it is fresh, but not when the lake is saline, which suggests that they could return to Toolibin.

Bennelongia ‘australis’ Salt-sensitive and common in temporary freshwater wetlands. 93% (from 294 records) of Bennelongia records (and (ostracod) 90% of Bennelongia ‘australis’ records) are from salinities <5000 mg/L. Present at Toolibin in 1992 at 2400 mg/L and in 1994 at 10000 mg/L but absent in 1998 at 9400 mg/L.

560 Table 2. Risk factors that may be affected by altered hydrology at Toolibin Lake over the next 25 years. Examples of each risk factor that

561 may arise from altered hydrology are provided.

Category of factor Key factor Generic example (associated with altered hydrology)

Resources • Food • Mortality following waterlogging and death of trees that provide food

• Oxygen • Juvenile plant death following deprivation of O2 during inundation and waterlogging. • Light (photosynthesis) • Adult or juvenile plant death following deprivation of light from turbid water

• Water • Inadequate fresh water availability (drought) leads to death

Disease/competition etc. • Disease • Surface water transports diseased plants into the system causing plant death

• Toxic species • Death of animal through consumption of toxins

• Predation • Death of birds due to predation following reduced availability of roosting habitat (due to death of trees)

Physical and Chemical factors • Salinity • Evaporation, saline inflow, and rising groundwater cause salt toxicity in - Surface water macroinvertebrates. - Groundwater

• Contaminants • Increased contaminants through catchment run-off into lake causes death of - Acidity/alkalinity organisms - Pesticides - Heavy metals

• Nutrients (N, P) • Nutrient input through run-off causing toxicity

• Physical damage from flooding (water • Damage to trees causing death velocity)

Reproduction • Lack of mates • Reduced availability of mates due to death or movement to more favourable areas • Genetic diversity • Reduced genetic diversity following death or movement resulting in lower

reproductive success and survival • Lack of nesting habitat • Reduced availability of nesting habitat due to inundation

562

563 Table 3. Level of over- or under-confidence, raw and normalised weights by participant code.

Participant code Over- or under- Raw weight Normalised weight confidence

A 17% underconfident 0.83 0.1084

B, D, E, J NA, Perfectly calibrated 1.00 0.1309

C 17% overconfident 0.83 0.1084

F, H 33% overconfidence 0.67 0.0877

G, I 66% overconfident 0.33 0.0445

564

565

566 Table 4. Workhop participant estimates of the probability that altered hydrology will, through impacts on key factors, cause goal failure (species 567 extinction from the wetland). Participants’ assumptions and reasons underlying their estimates were also recorded. High probability factors were assessed 568 in more detail using the full risk analysis process.

Key factor Probability of Reason species loss

Pesticides/herbicides < 0.05 Application levels on farmland may occasionally be high but entry through run-off predicted to be low.

Acidity/alkalinity < 0.05 Some carbonates exist in groundwater but levels not expected to cause significant change in acidity/alkalinity.

Heavy metals < 0.05 Heavy metal presence and toxicity is highly correlated with acidity/alkalinity. Therefore, risk rated at the same level.

Nitrogen toxicity < 0.01 There are no substantial inputs from external sources.

Phosphorus toxicity < 0.01 There are no substantial inputs from external sources.

Physical damage < 0.01 Current management practices determine inflow levels under usual rainfall levels within the catchment ensuring that physical damage should not occur. Significant flood events that may cause physical damage leading to species loss have not been recorded.

Food < 0.01 Extreme circumstances are necessary for hydrology to alter food resources causing species loss.

Oxygen < 0.05 Prolonged flooding is unlikely to cause the complete removal of a plant species due to oxygen deprivation. Although Melaleuca strobophylla is relatively susceptible to anoxic conditions, species loss would require years of continuous flooding.

Light < 0.01 Inundation by turbid water is unlikely to occur for periods long enough to cause species loss.

Disease < 0.05 Most watercourses in WA have been found to have at least one type of Phytophthora within them but some are simply involved in nutrient cycling (i.e. do not cause plant death). As yet no Phytophthora spp have been found at Toolibin Lake so species loss is thought to be unlikely at this stage.

Toxins < 0.02 Only one cyanobacteria/Botulinum spp event poisoned substantial numbers of waterbirds at Toolibin Lake in the last 30 years, however, there have been no records of species loss there or elsewhere in the south-west of Western Australia.

Predation/grazing < 0.01 Some predation (e.g. by foxes and cats) may increase if inundation of the lake does not occur for a long enough. These predators cannot reach the fledgling birds when water levels are high but shorter hydroperiods mean less time for fledgling birds to learn to fly (i.e., escape from predators). Species loss, as a result of predation is thought to be unlikely.

Lack of mates < 0.05 Sexually reproducing species generally have access to mates over a large spatial scale so lack of mates unlikely to cause species loss.

Lack of genetic < 0.05 As above diversity

Lack of nesting habitat < 0.05 Bird species unlikely to stop visiting Toolibin Lake simply due to a lack of nesting habitat (J. Lane, pers. comm.).

83% CI: 0.154 – Two divergent views were clearly observable: 1) Current management (water diversion) plus increased frequency of dry Lack of water 0.559 (best years is likely to cause the loss of species from Toolibin Lake. 2) After the extreme drought of 2010, species loss in the next estimate 0.336) 25 years is unlikely given their apparent capacity to survive and ability to survive in microhabitats.

Ground water salinity 83% CI: 0.133 – Rising highly saline groundwaters over an extended period may cause death of trees. However, the species may persist in 0.454 (best microhabitats. estimate 0.282) Surface water salinity 83% CI: 0.235 – Evapoconcentration of salts from surface water can cause elevated soil salinity and reduced water availability due to the 0.619 (best osmotic effects in the root zone. Little change would be necessary to see some species loss, however, with current estimate 0.411) management practices there would need to be a large highly saline flood event for this to occur.

569

570

571 Figure 1. Estimates for the highest, lowest and best probability of change according to expert

572 opinion for second estimates for goal failure due to a lack of water (drought) at Toolibin Lake. A clear

573 separation between the two divergent views (Group A - triangles and Group B - squares) is shown.

574

575

576