Options for reducing uncertainty in impact classification for alien 1 1 1 2 DAVID A. CLARKE , DAVID J. PALMER, CHRIS MCGRANNACHAN, TREENA I. BURGESS, 1 1 3,4 5 STEVEN L. CHOWN , ROHAN H. CLARKE, SABRINA KUMSCHICK, LORI LACH , 6,7 8 9,10 11 ANDREW M. LIEBHOLD , HELEN E. ROY, MANU E. SAUNDERS , DAVID K. YEATES, 12 1,13, MYRON P. Z ALUCKI, AND MELODIE A. MCGEOCH

1School of Biological Sciences, Monash University, Clayton, Victoria 3800 2Centre for Climate Impacted Terrestrial Ecosystems, Harry Butler Institute, Murdoch University, 90 South Street, Murdoch 6150 Australia 3Centre for Invasion Biology, Department of Botany & Zoology, Stellenbosch University, Matieland, South Africa 4Cape Town Office, South African National Biodiversity Institute, Claremont, South Africa 5College of Science and Engineering, James Cook University, PO Box 6811, Cairns, Queensland 4870 Australia 6USDA Forest Service Northern Research Station, Morgantown, West Virginia 26505 USA 7Faculty of Forestry and Wood Sciences, Czech University of Life Sciences, Praha 6 - Suchdol, CZ 165 21 Czech Republic 8UK Centre for Ecology & Hydrology, Wallingford OX10 8BB UK 9School of Environmental and Rural Science, University of New England, Armidale, New South Wales 2351 Australia 10UNE Business School, University of New England, Armidale, New South Wales 2351 Australia 11CSIRO Australian National Collection, PO Box 1700, Canberra, Australian Capital Territory 2601 Australia 12School of Biological Sciences, University of Queensland, Brisbane, Queensland 4072 Australia 13Department of Ecology, Environment and Evolution, La Trobe University, Bundoora, Melbourne, Victoria 30186 Australia

Citation: Clarke, D. A., D. J. Palmer, C. McGrannachan, T. I. Burgess, S. L. Chown, R. H. Clarke, S. Kumschick, L. Lach, A. M. Liebhold, H. E. Roy, M. E. Saunders, D. K. Yeates, M. P. Zalucki, and M. A. McGeoch. 2021. Options for reducing uncertainty in impact classification for alien species. Ecosphere 12(4):e03461. 10.1002/ecs2.3461

Abstract. Impact assessment is an important and cost-effective tool for assisting in the identification and prioritization of invasive alien species. With the number of alien and invasive alien species expected to increase, reliance on impact assessment tools for the identification of species that pose the greatest threats will continue to grow. Given the importance of such assessments for management and resource allocation, it is critical to understand the uncertainty involved and what effect this may have on the outcome. Using an uncertainty typology and as a model taxon, we identified and classified the causes and types of uncertainty when performing impact assessments on alien species. We assessed 100 alien insect species across two rounds of assessments with each species independently assessed by two assessors. Agreement between assessors was relatively low for all three impact classification components (mechanism, severity, and confidence) after the first round of assessments. For the second round, we revised guidelines and gave assessors access to each other’s assessments which improved agreement by between 20% and 30% for impact mechanism, severity, and confidence. Of the 12 potential reasons for assessment discrepancies iden- tified a priori, 11 were found to occur. The most frequent causes (and types) of uncertainty (i.e., differences between assessment outcomes for the same species) were as follows: incomplete information searches (sys- tematic error), unclear mechanism and/or extent of impact (subjective judgment due to a lack of knowl- edge), and limitations of the assessment framework (context dependence). In response to these findings, we identify actions that may reduce uncertainty in the impact assessment process, particularly for assess- ing speciose taxa with diverse life histories such as Insects. Evidence of environmental impact was avail- able for most insect species, and (of the non-random original subset of species assessed) 14 of those with evidence were identified as high impact species (with either major or massive impact). Although uncer- tainty in risk assessment, including impact assessments, can never be eliminated, identifying, and commu- nicating its cause and variety is a first step toward its reduction and a more reliable assessment outcome, regardless of the taxa being assessed.

v www.esajournals.org 1 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

Key words: EICAT; impact assessment; invasive alien species; mechanisms of impact; reducing uncertainty; risk communication.

Received 23 September 2020; revised 30 November 2020; accepted 10 December 2020; final version received 27 January 2021. Corresponding Editor: Debra P. C. Peters. Copyright: © 2021 The Authors. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. E-mail: [email protected]

INTRODUCTION assigning relative impact severities to alien spe- cies (Turbe et al. 2017, Gonzalez-Moreno et al. With a changing climate and growing interna- 2019, Vila et al. 2019), producing evidence-based tional trade, the range expansion of alien and data that can then be used in risk assessment and invasive alien species (i.e., invasive alien species prioritization. Semi-quantitative protocols are a being those that have a negative impact within valuable and necessary tool in contexts, such as their recipient environment) is predicted to con- invasion biology, where decisions must consider tinue (Seebens et al. 2017). Ongoing arrival and multiple lines of evidence, be made transparently establishment of alien species necessitates a and where available evidence is incomplete (Gre- triage approach to their management, where the gory et al. 2012). One such tool, Environmental most damaging, or those most likely to cause Impact Classification for Alien Taxa (EICAT), is a damage, is allocated high priority for surveil- formal protocol for using peer-reviewed litera- lance and control (McGeoch et al. 2016). To this ture to assess any environmental impacts (i.e., end, research on the impacts of alien species has negative effects on the native environment) that been growing steadily (Crystal-Ornelas and an alien species has inflicted and with which an Lockwood 2020), accompanied by the develop- impact severity and mechanism is assigned ment of various assessment frameworks for clas- (Hawkins et al. 2015). However, few studies have sifying their impacts to assist in risk assessments yet explored the potential types and causes of (Roy et al. 2018, Gonzalez-Moreno et al. 2019, uncertainty associated with semi-quantitative Vila et al. 2019). tool outcomes for alien species impacts, and how Risk assessments are generally performed using these could affect the final classification outcome multiple data types that vary in reliability, and (but see Kumschick et al. 2017a, who compared therefore, acknowledging, accounting, and com- results from two different applications of EICAT municating associated uncertainty is an essential for alien amphibians). Better understanding of component of the process (Harwood and Stokes the causes of uncertainty in impact classification 2003). For example, assessments by the Intergov- is an essential step in testing these protocols and ernmental Panel on Climate Change (IPCC), that identifying general solutions to strengthen their use multiple lines of evidence, are conducted reliability and value to risk analysis (Milner-Gul- using a formal and agreed upon treatment of land and Shea 2017, Rueda-Cediel et al. 2018, uncertainty. This method assesses the type, Latombe et al. 2019). amount, quality, and consistency of evidence, While uncertainty cannot be removed entirely along with the level of agreement, to assign confi- from semi-quantitative methods such as EICAT, dence in the form of a likelihood scale (Mastran- identifying which causes of uncertainty are redu- drea et al. 2011). Impact assessments of alien cible (and acknowledging those that are practi- species, which contribute to full risk assessments, cally irreducible; Regan et al. 2002) is an essential likewise should be accompanied by estimates of step in improving the reliability and value of the uncertainty (Roy et al. 2018), although this does information they generate. Given the inherent not always occur (Caton et al. 2018). nature of uncertainty in ecology more broadly, Semi-quantitative decision protocols are now a and the need to account for it, frameworks for widely used and accepted approach for identifying and classifying various types of

v www.esajournals.org 2 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL. uncertainty have been found to be useful (Mil- recommendations for how best to communicate ner-Gulland and Shea 2017). For example, the and mitigate general types of uncertainty associ- uncertainty typology developed by Regan et al. ated with semi-quantitative methods for classify- (2002) has been used to assess uncertainty in con- ing the severity of alien species impacts. servation decision making in the face of climate change (Kujala et al. 2013) and to assess uncer- METHODS tainty in the alien species listing process and what effect it may have on estimating the iden- Several tools and methods exist in support of tity and number of such species in countries risk assessment to ensure that the process and its (McGeoch et al. 2012). In both cases, application outcomes are as transparent and as evidence of the typology helped to identify tactics for based as possible. One of these tools is Environ- improving the transparency and repeatability of mental Impact Classification of Alien Taxa environmental decisions. Regan et al. (2002) (EICAT). EICAT and other impact assessment identify two broad types of uncertainty, with methods for IAS are not risk assessments them- multiple categories under each, that is, (1) epis- selves, but rather tools used in support of evi- temic uncertainty, or that brought about by dence-based RA’s that encompass a range of uncertainty in determinate facts, and (2) linguis- other socio-economic and context-specific infor- tic uncertainty, or that caused by the inherent mation (Hawkins et al. 2015, Gonzalez-Moreno variation in our use of language (Regan et al. et al. 2019). Our intention here is to evaluate the 2002, Burgman 2005, Latombe et al. 2019). Lin- types of uncertainty associated with classifying guistic uncertainty has been a significant hin- species based on evidence of their impact. This drance to progress in invasion biology to date exercise is therefore intended as a contribution to (Roy et al. 2018, Vila et al. 2019). The uncertainty improve the robustness of impact assessment framework of Regan et al. (2002) provides a tools. We do so by (1) selecting 100 insect species widely relevant framework for evaluating uncer- to assess, (2) independently applying the EICAT tainty (McGeoch et al. 2012, Kujala et al. 2013). protocol to each species twice, (3) describing the Here, we use this framework to identify and differences found between assessments for each classify the types of uncertainty and their causes species and the reasons for these, (4) using that may occur when assessing the environmen- Regan’s et al. (2002) uncertainty framework to tal impacts of alien species. That is, we provide a assign each difference to an uncertainty type and practical, rather than theoretical, approach to analyze their relative frequency, and (5) discuss addressing uncertainty in impact assessments. ways in which each uncertainty type could be Using insects as a case study, we apply the pub- reduced by refining structured decision protocols lished protocol, EICAT, for classifying the for alien species impacts. impacts of alien species (Blackburn et al. 2014, Hawkins et al. 2015). Insects were selected as a Species pool and those assessed model taxon for multiple reasons: (1) Alien inva- The subset of alien insect species assessed sive insect species pose significant risks to the was deliberately selected to be likely to encom- environment and economy and require impact pass as many species as possible with adequate risk assessments to prioritize management (Brad- peer-reviewed evidence of impact, given the shaw et al. 2016, Lovett et al. 2016, Suckling et al. demonstrated paucity of impact evidence for 2019); (2) they are one of the most species-rich, alien species (Crystal-Ornelas and Lockwood abundant, functionally diverse taxonomic groups 2020). The initial set (~2800 species) consisted (Stork et al. 2015, Noriega et al. 2018), and (3) of all insect species present in the Global Regis- they are a group to which EICAT has not ter for Introduced and previously been applied. We use differences (GRIIS), which provides verified species by found between independent assessments of 100 country occurrence records outside their native insect species to identify, interpret, and classify range at country scale and an attribute for causes and types of uncertainty. Incorporating local evidence of impact (Pagad et al. 2018; as the lessons learnt from performing impact classi- at June 2017). In addition, alien insects known fication on alien insects, we conclude with to impact the environment were extracted from

v www.esajournals.org 3 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL. relevant reviews (Kenis et al. 2009, Vaes-Petig- Database (Blackburn et al. 2014, Hawkins et al. nat and Nentwig 2014, McGeoch et al. 2015, 2015). Cameron et al. 2016, Bertelsmeier et al. 2017, EICAT assessments have, to date, been pub- Evans et al. 2017). Species were then ranked by lished for alien birds (Evans et al. 2016), amphib- the number of times they were referred to ians (Kumschick et al. 2017a), selected across all sources, that is, multiple sources des- gastropods (Kesner and Kumschick 2018), mam- ignated them as invasive or having a negative mals (Hagen and Kumschick 2018) and bamboo environmental impact. The top 100 species species (Canavan et al. 2019). Here, we used the were selected for assessment in this way prior EICAT protocol as published (Hawkins et al. to the literature search for each species. Given 2015), although these guidelines have subse- the assumed positive relationship between pest quently been revised by the IUCN via an online severity and available literature on the species, consultation process and the involvement of one we consider this suite of 100 species to incor- of the co-authors (SK; IUCN 2020a). Our interest porate a best-case scenario for evidence avail- here was to identify forms of uncertainty in the able on which to implement EICAT and impact assessment process, rather than to evalu- interpret the results accordingly. ate EICAT per se, which is similar to a range of other semi-quantitative impact assessment Structured decision protocol protocols (Kumschick et al. 2017b). We adopted EICAT is a protocol with published guidelines previously suggested modifications to the mech- for its implementation to ensure that it is applied anisms used to assess the environmental impacts in a consistent, transparent, and comparable way of alien insects. Specifically, (1) the mechanism across taxa and between assessors (Blackburn grazing/herbivory/browsing was altered to her- et al. 2014, Hawkins et al. 2015). EICAT has now bivory as grazing and browsing are not relevant been adopted by the IUCN as a standard for the for insects, and (2) the addition of a relevant purpose of assigning severity of impact cate- mechanism, facilitation of native species gories and impact mechanisms to IAS, with (whereby an invasive species negatively affects strong parallels to the method used for the IUCN one native species by positively affecting another Red List of Threatened Species. The EICAT sever- native species) was considered important for ity of negative impacts is classified as minimal capturing this frequently encountered form of concern (MC), minor (MN), moderate (MO), indirect impact of invasive insects on biodiver- major (MR), or massive (MV; Hawkins et al. sity (McGeoch et al. 2015). 2015). The final severity attributed to an alien species at the end of the assessment process is its Implementation of EICAT maximum realized impact anywhere within its The general steps involved following the pre- introduced range; that is, EICAT is only con- assessment and assessment steps of the EICAT cerned with the most severe realized (as opposed protocol as a research exercise, using the detailed to potential) impact for a species. Two other cate- guidelines, decision charts, and definitions pub- gories are possible, data deficient (DD) for cases lished in Hawkins et al. (2015). We discussed and where there is insufficient evidence to perform conducted the research (including the assess- an assessment, or not alien (NA) for when a ments unless otherwise specified below) and the given species does not in fact have an alien distri- first and senior authors analyzed and reported bution. Mechanisms of impact are ascribed for the results back to the authorship group for dis- each assessment where impact evidence was cussion and interpretation. found. For example, a given species may have a To measure the level of congruence between negative impact by preying upon native species assessment outcomes and to identify reasons for and would, therefore, be ascribed the mechanism the differences that occurred, each of the 100 of predation for that specific piece of evidence. alien insect species was assessed independently, These mechanisms, of which there were 13 using peer-reviewed literature, for environmen- (Blackburn et al. 2014, Hawkins et al. 2015), tal impacts by two people. The ECAT guidelines expand on those identified by Kumschick et al. specify that assessments may be conducted by (2012) to align with the Global Invasive Species individuals or groups, in person or via email

v www.esajournals.org 4 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

(Hawkins et al. 2015). Our rationale for individu- departing from these only where it was neces- ally conducting the assessments was that by sary to clarify, elaborate, or modify details for the doing so we were able to use differences found purpose of reducing uncertainty. The purpose of between independent assessments to identify revisions to the guidelines was to clarify where and why uncertainty arises in the impact particular protocol points that were identified as assessment process. Comparing individual potentially resulting in differences between assessments rather than group assessments also assessments, particularly to reduce the more allowed us to complete assessments for many common and readily addressed misunderstand- more species than would otherwise have been ings that arose in the first-round assessments possible. Here, the rationale was that including (such as the need to first confirm evidence of the many species, representing a broad range of life existence of alien populations of each species). histories and available information, would likely Fourth, the clarified guidelines, along with the yield a broader array of potential reasons for dif- initial results from both assessors for each ferences found between assessments of the same species, were then used by the assessors in the species. By controlling the assessments and their second assessment round. Differences can occur revisions using the process outlined below, we for one or more outcome of each assessment, that were able to exclude as far as possible potential is, differences in allocated impact mechanism, influence of one assessor over the judgment of severity, confidence level, or any combination of another, as well as maximize the comparability/ the three. Fifth, assessors could then either refine uniformity of the process across each of the spe- their initial assessment considering this new cies assessed. This suited the purpose of this information or leave their assessment as origi- study, which was to identify types and sources of nally provided (Burgman et al. 2011). Either deci- uncertainty. sion required justification. The sixth and final The entire process can be generalized by the step required assessors to provide possible rea- following steps. First, assessors discussed the sons for why, in their view, initial assessments protocol and performed a series of pilot assess- resulted in differences. ments as a group to familiarize everyone with the EICAT process and protocol guidelines Application of the uncertainty framework (Hawkins et al. 2015). Second, impact assess- We used the information generated from the ments were performed independently by two above process to understand and classify the assessors for each species. Literature searches assessment uncertainty, by treating the differ- were performed first by using ISI Web of Science ences in assessment outcomes across assessors as (webofknowledge.com) using the species name types of uncertainty and classifying them using (scientific including synonyms) as the search the uncertainty typology of Regan et al. (2002). terms, after which assessors used supplementary Using the detailed definitions of types of uncer- searches and sources as needed. Each species tainty provided by Regan et al. (2002), causes of assessment resulted in three main outputs with uncertainty were first broadly classified as either supporting literature evidence: (1) the mecha- epistemic or linguistic in origin. Epistemic uncer- nisms of impact (i.e., how the alien species is tainty is that associated with the knowledge of a having a negative effect, e.g., via competition), system’s state and consists of six main types (2) the size of the impact (the EICAT magnitude (measurement error, systematic error, natural of impact severity category), and (3) the confi- variation, inherent randomness, model uncer- dence in the evidence supporting the chosen tainty, subjective judgment; Regan et al. 2002). mechanism and impact severity (Hawkins et al. There are five main types of linguistic uncer- 2015). Third, based on the outcome of the first tainty associated with the omnipresent variation round where differences in outcomes occurred in our use of language (vagueness, context between assessors, the guidelines were clarified dependence, ambiguity, theoretical indetermi- where points of misunderstanding or differences nacy, under-specificity; Regan et al. 2002). Other in understanding arose. Our intention here was studies of uncertainty using this typology have to apply EICAT critically, adhering to the guideli- modified it by, for example, including a third nes as far as possible and elaborating on or broad type of uncertainty (human decision

v www.esajournals.org 5 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL. uncertainty) which contain causes of uncertainty 100 Assessment such as subjective judgment (Kujala et al. 2013). round One We chose to follow McGeoch et al. (2012) who Two used the original typology with minor adjust- 75 ments for application to listing of invasive alien species, distinguishing between subjective judg- 50 ment per se and subjective judgment due to a lack of knowledge, and between systematic error per se and systematic error due to a lack of 25 knowledge. The rationale for this is akin to the Assessor agreement (%) dual pathway of forming a subjective probability ’ 0 (Burgman 2005), where people s subjectivity Mechanism Severity Confidence stems from either a lack of, or despite, available Component knowledge. Instances of both epistemic and lin- guistic uncertainty were in some cases simultane- Fig. 1. Agreement in environmental impact assess- ously possible as the reason for differences ment outcomes between two independent assessors between assessment outcomes (McGeoch et al. across two rounds of assessment. Agreement increased 2012). Therefore, for each species where there across all assessment components in the second round. was a difference between assessors regarding The assessment of severity of impact improved most mechanism and/or severity of impact attribution, and mechanism of impact least across the two assess- the reasons and their associated uncertainty type ment rounds. were identified using the definitions in Regan et al. (2002) and McGeoch et al. (2012). We did not assess the uncertainty associated with differ- most frequently identified (Table 1), and agree- ences in the chosen confidence level due to its ment on herbivory as the primary mechanism of strong dependence on the chosen mechanism impact occurred in 38% (n = 14) of instances. and severity. We did, however, assess the correla- The remaining eight mechanisms were less fre- tion between the amount of disagreement in quently attributed (Table 1). For example, para- impact severity and the mean level of confidence sitism was attributed to six species, five of which for a given species (Appendix S1: Fig. S1). There were cases of agreement. Facilitation of native was no correlation between the two variables in species was also attributed to six species; how- the first or second round (Appendix S1: Fig. S1). ever, all six were cases that differed between An assignment of low confidence does not neces- assessors. Multiple (n = 14) instances occurred sarily lead to disagreement between assessors where the assessors agreed on a mechanism for a and therefore was not used as a proxy for asses- species, but one assessor also assigned additional sor disagreement. mechanisms. The assignment of multiple mecha- nisms in EICAT occurs when the same level of RESULTS impact severity is assigned to more than a single mechanism by an assessor for a given species. First-round assessments led to agreement Taking a conservative approach here, these cases levels of between 32% and 44% for each of the were considered differences in assessment out- three EICAT assessment components, that is, come. After the second round of assessments, mechanism, severity of impact, and confidence agreement on the mechanism of impact (Fig. 1). Mechanism of impact had the highest increased from 44% to 65% (Fig. 1). Little change level of agreement, with 44% of species attribu- between rounds occurred for the allocation of ted with the same primary mechanism of impact mechanisms of impact for most mechanisms by both independent assessors. The level of except for herbivory (37–53%), transmission of agreement, however, varied across mechanisms, disease (33–100%), and other (6–40%; Table 1). as did the frequency with which each mechanism Categories of impact severity varied in their was attributed (Table 1; Appendix S1: Fig. S2). levels of agreement (Fig. 2A). Data deficient Herbivory, competition, and predation were (DD) was the impact severity classification with

v www.esajournals.org 6 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

Table 1. Instances of agreement and disagreement between assessors for mechanism of impact for both assess- ment rounds.

Round 1 Round 2 Mechanisms of impact Agree Disagree Total Agree Disagree Total

Herbivory 14 23 37 17 15 32 Competition 22 21 43 17 19 36 Predation 11 13 24 9 12 21 Other 1 14 15 6 9 15 Transmission of disease 3 6 9 7 0 7 Parasitism 5 1 6 5 0 5 Interaction with other aliens 2 8 10 1 6 7 Facilitation of natives 0 6 6 0 6 6 Hybridization 2 0 2 2 0 2 Chemical/physical/structural ecosystem impacts 0 1 1 0 1 1 None 5 14 19 5 7 9 Note: As more than one mechanism can be attributed to a species, the instance totals within and between assessment rounds do not sum to 100.

A B

MO MR MR MO

MV MV NA NA

DD DD MN

MN MC

MC

Fig. 2. Chord diagrams showing the variation in agreement or disagreement in the assessment of severity of impact between independent assessors in each assessment round (A, B). The segment span width of a given impact severity category (MV, massive; MR, major; MO, moderate; MN, minor; MC, minimal concern; DD, data deficient; and NA, not alien) is proportional to the number of species for which that severity was assigned by at least one assessor. The width of the links connecting segments reflects the number of species for which there was difference between assessors involving those specific impact severities. The low density of links in B compared with A reflects the increase in agreement across assessments, revealing also moderate followed by major and data deficient as the most frequently assigned categories. highest agreement (see Appendix S1: Table S1 Assessor agreement on impact severity increased for full details of impact severity assessments). from 34% to 70% across rounds. As in the first Agreement on impact severity in round 1 round of assessments, variation in levels of occurred for 34% of species; that is, a species was agreement among severity of impact categories assigned the same severity of impact by both remained (Appendix S1: Table S1). Similarly, the assessors (assessment outcomes were as likely to distribution of agreement across the severity of agree as to differ, v2 = 1.37, df = 1, P = 0.24). impact categories differed little between

v www.esajournals.org 7 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

Anoplolepsis gracilipes Pheidole megacephala Apis mellifera Adelges tsugae Xyleborus glabratus Aphis gossypi Bombus terrestris longicornis Wasmannia auropunctata Linepithema humile Lymantria dispar Lysiphlebus testaceipes Merizodus soledadinus Myrmica rubra Pachycondyla chinensis Solenopsis invicta vulgaris quercuscalicis Anoplophora chinensis Anoplophora glabripennis Aulacaspis yasumatsui Cactoblastis cactorum Diaphorina citri Homalodisca vitripennis Orthotomicus erosus Paratrechina fulva Solenopsis richteri Technomyrmex albipes Tetropium fuscum Apis cerana Quadrastichus erythrinae Bessa remota Apis mellifera scutellata Calliphora vicina Cicindelidia trifasciata Cinara cupressi Danaus plexippus Severity Harmonia axyridis Icerya purchasi MV Larinus planus Lasius neglectus Ochlerotatus japonicus MR Pieris rapae Pteromalus puparum MO Pterostichus melanarius Solenopsis geminata MN melanocephalum Torymus sinensis MC Trechisibus antarcticus Trechus obtusus DD Trichopoda pilipes Urophora affinis Urophora quadrifasciata NA Vespa velutina Species Vespula pensylvanica labradus Bactrocera tryoni Chilo partellus Confidence dominula Scyphophorus acupunctatus NA Thaumetopoea processionea Trissolcus basalis Low Aedes albopictus Anopheles quadrimaculatus Medium Forficula auricularia Microctonus aethiopoides High Monomorium destructor Monomorium pharaonis ascalonicus Solenopsis papuana compactus Polistes chinensis Rhyzopertha dominica Acanthoscelides obtectus Bruchus rufimanus Drosophila subobscura Rhopalosiphum maidis Sitophilus oryzae Sitotroga cerealella Bruchus pisorum Leptinotarsa decemlineata Liriomyza huidobrensis Megachile rotundata Trogoderma granarium Andricus corruptrix Andricus lignicola Anthonomus grandis Aspidiotus nerii Ceratitis capitata Stictocephala bisonia Heteronychus arator Chromaphis juglandicola Callosobruchus chinensis Propylea quatuordecimpunctata Saissetia oleae Megachile apicalis NA MV MR MO MN MC DD NA Severity

Fig. 3. Final assessment results of impact severity and the associated confidence rating for all 100 insect

v www.esajournals.org 8 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

(Fig. 3. Continued) species. Species for which there remained difference after both rounds of assessment are shown with two differ- ent colored tiles in the row associated with them. Most instances of difference occurred at the interface between major and moderate, and between minor or minimal concern and data deficient. Category abbreviations are mas- sive, MV; major, MR; moderate, MO; minor, MN; minimal concern, MC; data deficient, DD; not alien, NA. For additional information on species identities, see Appendix S2.

assessment rounds (Appendix S1: Table S1). uncertainty type that occurred most frequently, Assessors were more likely to agree than dis- followed by subjective judgment as a result of agree (v2 = 83.64, df = 1, P < 0.001), and most lack of knowledge and context dependence differences (22/30) between impact severity cate- (Fig. 4). gories in the second round involved categories Final assessments (after the completion of both adjacent on the severity scale, as opposed to the rounds) revealed evidence of environmental first round which had much less structure impact was inadequate or unavailable for 14 spe- (Fig. 2B, Appendix S1: Fig. S2). For example, in cies (Fig. 3, Appendix S2: Table S1), that is, the second round, if there was difference in a where both assessors agreed on the species being Moderate impact, it involved one assessor data deficient. Including the additional 10 spe- assigning either a Minor (one category lower) or cies where assessors disagreed on data availabil- Major (one category higher) impact (Fig. 2B, ity equates to approximately 25% of alien insects Fig. 3). Exceptions to this occurred for two spe- assessed as data deficient by at least one assessor. cies (Diaphora citri, Tetropium fuscum) in which These species were distributed across five orders: one assessor assessed severity as Minor and the Coleoptera (n = 9), Diptera (n = 2), other assessed it as Major, as well as for six spe- (n = 5), (n = 7), and cies (Orthotomicus erosus, Solenopsis richteri, Chae- (n = 1). Of those species for which environmen- tosiphon fragaefolii, Polistes chinensis, Rhyzopertha tal impact information was available and asses- dominica, and Thaumetopoea processionea) in which sors agreed on impact severity (n = 56), 25 were one assessor assigned an impact severity and the Hymenoptera. This order was most frequently other thought there was insufficient data to do so represented in each impact severity category, or that there was no evidence of alien popula- except for Minimal Concern where there were no tions in the case of T. processionea (Fig. 3). Hymenoptera. It was also the only order for Eleven possible causes of uncertainty (Table 2) which there was agreement on a Massive impact, identified a priori occurred across the 100 species that is, for both the yellow crazy (Anoplolepsis assessed during the implementation of the proto- gracilipes) and the big-headed ant (Pheidole mega- col (Table 2). Most frequently, different use of ref- cephala). Few species were attributed this highest erence material was part of, if not the entire level of impact, and only five species were cause of difference in a species assessment, clo- assigned this category by at least one assessor sely followed by limitations of the assessment (Fig. 3). Considering only those species for which framework and the mechanism or extent of there was agreement on severity and excluding impact being unclear in the literature (Fig. 4). those that were data deficient, the number of spe- However, the frequency of these causes of uncer- cies across impact severity categories approxi- tainty differed depending on the assessment mates a symmetric distribution with Moderate component (impact mechanism versus severity). (n = 26) the most attributed category For example, although extrapolation of evidence (Appendix S1: Table S1). Including instances of and deviation from assessment protocol occurred difference alters the distribution of impact severi- similar numbers of times (14 and 12 times, ties, which reflects the uneven frequency with respectively), all but one of the former cases were which each category was associated with assess- related to differences in severity, while all ment differences. Agreement on impact mecha- instances of the latter were related to differences nisms was most common for herbivory (n = 17) in mechanisms (Fig. 4). Systematic error was the and competition (n = 17; Table 1). Hemiptera

v www.esajournals.org 9 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

Table 2. Causes and types of uncertainty that may lead to differences between independent assessors when per- forming environmental impact assessments.

Cause of uncertainty (description used in Figures) Explanation Uncertainty type Species example

1. Human error Performing impact assessments using Measurement error (i) (E) Xylosandrus incorrect information, for example, compactus using the wrong species (e.g., (Coleoptera: Xyleborus glabratus) in an ) assessment can result in misinformation 2. Incomplete information Non-comprehensive literature Systematic error (ii) (E) Drosophila searches (incomplete search) searches can lead to different subobscura assessments for the same species. (Diptera: Drosophilidae) 3. Documented data and Evidence is not uniform in its Systematic error as a result Cinara cupressi knowledge not readily or widely availability to all assessors. of lack of knowledge (Hemiptera: accessible (knowledge Language barriers can also prevent (ii_a) (E) ) inaccessible) the use of some evidence. This can result in either relevant pieces of evidence being overlooked and/or different evidence used between assessors 4. Species identification The dynamic nature of can Systematic error (ii)† (E) Apis mellifera lead to errors of commission and scutellata omission. Accepted species names (Hymenoptera: may change over time leading to Apidae) multiple synonyms for a species causing evidence to be missed. Additionally, insufficient taxonomic resolution can lead to the erroneous inclusion of evidence; specifically, organisms with many sub-species 5. Information regarding a) Inclusion of impact evidence based Systematic error as a result N/A indigenous and/or alien range is on course resolution knowledge of of lack of knowledge Chaetosiphon insufficient (range uncertain) alien range may lead to (ii_a) (E) fragaefolii overestimation of impacts. Subjective judgment as a (Hemiptera: b) Information on alien range is result of lack of Aphididae) scarce or variable leading to a knowledge (iii_a) (E) subjective decision whether to conclude impact evidence is within an alien range 6. Limitations of assessment Lack of clarification in the assessment Context dependence (iv) Orthotomicus erosus framework (assessment limits) framework as to what constitutes (L) (Coleoptera: the inclusion of certain evidence Curculionidae) suitable for assessing environmental impacts 7. Species designation as invasive Definitions for what constitutes an Vagueness (v) (L) Danaus plexippus (invasiveness criteria) alien or invasive species can be (Lepidoptera: vague and vary widely in the Nymphalidae) literature, and not consistent with the definition used in the assessment. This can result in differences in opinion as to whether a species requires an impact assessment or not 8. Unclear mechanism and/or Information obtained from available Subjective judgment as a Monomorium extent of impact (mechanism/ evidence can be unclear regarding result of lack of destructor impact unclear) how a species is having an knowledge (iii_a) (E) (Hymenoptera: environmental impact, leading to an Formicidae) assessor’s subjective interpretation of the evidence available 9. Extrapolation of evidence Based on their expert knowledge, Subjective judgment (iii) Adelges tsugae assessors may reach conclusions (E) (Hemiptera: beyond what the available evidence Adelgidae) states

v www.esajournals.org 10 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

(Table 2. Continued.)

Cause of uncertainty (description used in Figures) Explanation Uncertainty type Species example

10. Deviation from assessment Ascribing impact mechanisms to a Measurement error (i) (E) Aspidiotus nerii protocol (deviation from species despite there being (Hemiptera: protocol) insufficient evidence to carry out an Diaspididae) assessment 11. No apparent cause Difference between assessments has Natural variation (vi) (E) Anoplolepsis no apparent cause other than a gracilipes natural variation in evidence (Hymenoptera: interpretation Formicidae) Notes: Causes of uncertainty were identified from McGeoch et al. (2012), checked for relevance to impact classification, and assigned to the relevant uncertainty type following Regan et al. (2002). Species examples provided are those encountered dur- ing the implementation of impact classification for insects and are discussed further in the text. Uncertainty types are catego- rized as epistemic (E) or linguistic (L). † Although classified as systematic error, this cause of uncertainty would be more accurately considered a form of theoreti- cal indeterminacy as per Regan et al. (2002). were somewhat of an exception, and here, trans- independently conducted assessments, with the mission of disease occurred as often as herbivory. reasons for these differences associated with 11 Although these two mechanisms were most fre- different causes and six different types of uncer- quent, all orders except one (Dermaptera) were tainty. However, levels of agreement increased associated with at least four mechanisms of substantially following discussion between impact (Appendix S1: Fig. S3). This declined assessors of reasons for the uncertainty involved. slightly when considering only instances of This process enabled us to identify several practi- agreement, with Lepidoptera decreasing to two cable recommendations for reducing the uncer- attributed mechanisms. tainty associated with assessing and classifying the impact of alien species. DISCUSSION Alien invasive insects have negative impacts within and across multiple environmental set- With alien species introduction events tings (Kenis et al. 2009, McGeoch et al. 2015). expected to continue (Seebens et al. 2017), the This can add uncertainty to decisions on which prioritization of species of most concern for man- evidence to include (context dependence, agement authorities is likely to become increas- Table 2, Cause 6, Type iv). For example, the ingly important. Information obtained from alien Mediterranean pine has been unintention- species impact classifications can assist in both ally introduced to multiple countries, most likely prioritizing those species to manage, and those via the timber trade (Brockerhoff et al. 2006, in which to invest research. However, uncer- Haack 2006). It is known for causing damage to tainty that arises in the process of conducting conifers, particularly pines, often leading to tree impact assessments for alien species means that mortality (Sanguesa-Barreda€ et al. 2015). How- the results may not be reproducible and, as we ever, all such evidence of a negative impact show here, independent assessments can arrive inside its alien range takes place within managed at different conclusions for the same species, and forestry plantations (Stephens and Wagner 2007). single assessments may produce biased out- As such, both assessors for O. erosus, who dif- comes. Neither of these alternatives is desirable fered in opinion as to which evidence to include, when decision makers rely on such information came to entirely different conclusions, even after to guide action and investment. Here, we the second round of assessments (data deficient showed how examining alternative outcomes for vs. massive). Similar discrepancies occurred in individual species provides insight on the type the assessments of other species (citrus long- and extent of uncertainty involved, and potential horned beetle (Anoplophora chinensis), Asian solutions for reducing it. We found that follow- long-horned beetle (A. glabripennis)). Similar ing the protocol guidelines resulted in relatively arguments can be made for agricultural crops, high levels of disagreement between depending on the extent to which native

v www.esajournals.org 11 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

Mechanism Severity

Incomplete search (2)

Mechanism/impact unclear (8)

Assessment limits (6)

Extrapolation of evidence (9)

Deviation from protocol (10)

Human error (1)

No apparent cause (11)

Invasiveness criteria (7) Cause of uncertainty

Knowledge inaccessible (3)

Range uncertain (5b)

Species identification (4)

Range uncertain (5a)

0 5 10 15 20 0 5 10 15 20 Frequency

Mechanism Severity

Systematic error (ii)

Subjective judgment (iii_a)

Context dependence (iv)

Measurement error (i)

Subjective judgment (iii) Type of uncertainty Type

Natural variation (vi)

Vagueness (v)

Systematic error (ii_a)

0 5 10 15 20 250 5 10 15 20 25 Frequency

Fig. 4. Frequency with which (A) causes of uncertainty and (B) types of uncertainty varied within- and between-impact assessment components (mechanism and severity of impact). Bar lengths highlight the dominant causes (A) or types (B) of uncertainty. The number and Roman numerals alongside the causes and types of uncer- tainty are those used in Table 2, and ii_a and iii_a are the sub-types related to lack of knowledge.

v www.esajournals.org 12 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL. biodiversity or native relatives within them are Uncertainty associated with severity of impact impacted by the alien insect. Such impacts can Agreement on severity of impact increased thus pose a potential risk to native biodiversity from 34% to 70% following the second round of more widely. Without formal direction and clari- assessments, suggesting that this assessment fication at the outset of the assessment process, component is quite amenable to a reduction in differences based on subjective interpretations of uncertainty. Incomplete information searches systems suitable for inclusion are inevitable, (systematic error, Table 2, Cause 2, Type ii) were albeit avoidable. Inclusion of an alien species in a common cause of difference, and the most fre- impact classification should, we argue, be based quent reason for why independent assessors on what is being negatively affected more so assigned different impact severities. There are than where; that is, the context within which the multiple possible explanations for why different impact is occurring is less relevant in some cases. pieces of evidence were selected for use, not least A different situation arises for species that of all are different interpretations of what consti- may arguably never negatively affect the native tutes evidence appropriate for inclusion in the environment, and this assumption led to some impact assessment (i.e., limitations of assessment species being assigned minimal concern despite framework, Table 2, Cause 6, Type iv). the absence of evidence (subjective judgment, Even with definitions for each severity cate- Table 2, Cause 9, Type iii). Two groups of alien gory (Hawkins et al. 2015), assigning impact insects, given their life histories, may lead an severity appears to be a decision open to greater assessor to assign minimal concern without evi- interpretation than assigning impact mecha- dence (i.e., rather than data deficient). The first nisms. One explanation may be the naturally group are those species that are primarily stored unbounded nature of the relationship between product pests. For example, the bean weevil successively more severe impact categories, (Acanthoscelides obtectus) is an economically where even with definitions for each impact important pest of legumes (Soares et al. 2015). severity category there is inevitably scope for While damaging, its impact almost entirely subjective interpretations of degree. For example, occurs on stored products, and it is a species pri- an alien species with a minor impact is one that marily of economic concern with a very low like- negatively affects the fitness (or performance, see lihood of having an environmental impact. The Fig. 5) of a native species (Hawkins et al. 2015). second group are monophagous insects that only However, a decline in fitness of individuals is attack and exist in agricultural crop environ- likely to lead to a decline in population size, ments and/or have a long history of no evidence which constitutes the next most severe impact of environmental impact. The broad bean beetle category (moderate). The difference between a (Bruchus rufimanus) is a widespread crop pest, negative impact on the fitness of individuals in a specifically targeting the faba bean (Vicia faba L.; population and a population-level effect is a mat- Clement et al. 2002). Given the long history of ter of degree (Ricciardi et al. 2013, Blackburn evidence for this species only being of concern et al. 2014). Thus, if we assume a given impact for specific crop types, one may conclude that it severity begets the next most severe category, is of minimal environmental concern, even then there are two alternative pathways to arrive though the lack of evidence would otherwise at a given severity category, that is, either via evi- lead to a data-deficient classification. Such con- dence of no impact or via no evidence of an clusions are an instance of using absence of evi- impact at the next most severe category (the deci- dence as evidence of absence, generally sion tree in Fig. 5 enables these alternatives to be undesirable but logically valid under certain cir- distinguished). For example, an assessor may cumstances (Sober 2009). The decision to classify conclude that a given species is having a minor a species as minimal concern for environmental impact either because (1) there is evidence, it is impact without evidence, however, should occur not causing population declines, or (2) because outside of the impact assessment process in the there is no evidence, it is causing population form of additional information that may comple- declines. For example, Hadley and Betts (2012) ment the final assessment outcome, as suggested found that a proposed lack of negative conse- for accommodating expert input. quences of habitat fragmentation on pollination

v www.esajournals.org 13 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

Decision tree for assigning an impact severity category to species of environmental concern

: No reliable evidence that it has or had Focal species individuals existing in a wild state in a region beyond its native geographic range.

Does the species have alien populations? : Best available evidence indicates that a given taxon has individuals existing in a wild state outside of its native range but there is inadequate information to classify an impact or insufficient time has NA NA elapsed since introductions have Does the species have an become apparent. environmental impact? : Unlikely to have caused deleterious impacts on the native biota or abiotic environment DD MC Is the species negatively affecting : Causing reductions in the fitness* of native species? the fitness of individuals in the native biota, but not declines in native population sizes

: Causing declines in Is the species negatively affecting the population size of native species, the population of native species? but not changes to the structure of communities or abiotic/biotic ecosystem composition

MN MN : Causing the local population extinction of at least one Is the species causing local extinction native species, and leads to of native species leading to community reversible changes in the structure of level effects? communities and the abiotic/biotic ecosystem composition.

: Causing the MO MO replacement and local exitinction of native species, and produces irreversible changes in the structure Are the community level effects of communities and abiotic/biotic irreversible? ecosystem composition.

Information is unavailable MR MR Evidence indicates yes MV Evidence indicates no

Fig. 5. Decision tree for improving the transparency and repeatability of assigning EICAT impact severity

v www.esajournals.org 14 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

(Fig. 5. Continued) categories. This framework is based on the assumption that the different categories (MC–MV) are hierarchically related; that is, a negative effect at one level necessitates a negative effect at the level below it (dashed arrows) and may mean existence of an impact at a higher impact severity level, although no evidence may yet be avail- able to support designation of this higher level of impact. Under this assumption, there can often be two path- ways by which an impact category can be assigned, with the followed path dependent on information availability. For example, in the case of the two pathways for assigning a moderate (MO) impact; if there is infor- mation to support the conclusion that there is a negative effect on the population size of one or more native spe- cies, but no effect on community structure, then the appropriate category to assign is MO. Alternatively, there may be information on native population-level negative effects but no information (lack of evidence) available on the effects at the community level, in which case the category assigned would also be MO. In general terms, these two alternative pathways represent evidence of no impact (right) and no evidence of impact (left). ÃIn the most recent version of the EICAT implementation guidelines (IUCN 2020a, b), the term fitness has been replaced by performance.

dynamics was a result of the absence of evidence, other predation, or one assessor provided both and not evidence of an absence of such conse- competition and predation because based on the quences. Hawkins et al. (2015) state that an evidence, they were not able to distinguish impact severity category is assigned if there is no between them. For example, the evidence of evidence for the next most severe category, environmental impact for the harlequin lady- implying the absence of evidence case (2 above). beetle (Harmonia axyridis; Roy et al. 2012, Masetti However, explicitly distinguishing between these et al. 2018) and the Argentine ant (Linepithema two scenarios is important (Altman and Bland humile; Menke et al. 2018) often inferred the 1995, Alderson 2004), even if the former occurs impact mechanism could be competition or pre- rarely by comparison, because it informs the dation. In situations such as this where the evi- strength of evidence available and confidence dence itself was sometimes unclear on the placed in it. Specifying such reasoning during mechanism of impact, it is difficult to avoid dif- assessments by making the decision process fering subjective interpretations by independent explicit (as shown in Fig. 5) could reduce uncer- assessors (i.e., subjective judgment as a result of tainty, as well as make the process more readily lack of knowledge, Table 2, Cause 8, Type iii_a). repeatable by specifying the basis for the final Another common reason for differences decision, that is, that there was, or was not, evi- between assessors on impact mechanisms was dence for the next most severe category (Fig. 5). the use of different lines of evidence (systematic error, Table 2, Cause 2, Type ii). Multiple reasons Uncertainty associated with mechanism of impact for this are possible. First, for many insects the One of the more common causes of difference quantity of literature is large, and a literature in assigning a mechanism of impact was a lack of search can return thousands of studies for a sin- clarification, on the part of the authors of the gle species (e.g., some species assessed had studies used as evidence, as to how an alien spe- search returns in the order of 14,000). This cies was negatively affecting the native environ- increases significantly the time needed to per- ment. In such cases, assessors were required to form an assessment, and the goal is to be thor- make a subjective interpretation as to what ough. Nonetheless, simple oversight can result in mechanism was most likely, based on the context mistakenly missing, misinterpreting, or includ- of the study. Similarly, it was often difficult to dif- ing a certain piece of evidence (increasing the ferentiate between two mechanisms, for exam- chance of human error and incomplete informa- ple, competition and predation (Sandvik et al. tion searches as the cause of uncertainty; Table 2, 2004), where the evidence suggested that either Cause 1, Type ii). For example, differences in the choice was valid. This led to situations where first round of assessments for the shot-hole borer either one assessor chose competition and the (Xylosandrus compactus) occurred because of the

v www.esajournals.org 15 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL. accidental use of evidence pertaining to a differ- new mechanism of impact. This could subse- ent species (Xyleborus glabratus; Table 2). By con- quently reduce uncertainty associated with trast, the Mediterranean pine beetle impact mechanism decisions. (Orthotomicus erosus) assessments differed as a result of different interpretations of the EICAT Consequences of uncertainty in impact guidelines (Table 2). Both examples become classification for IAS causes of uncertainty for reasons other than sim- Not all causes of uncertainty have equally sig- ply incomplete information searches. This shows nificant consequences for the outcome of an how uncertainty can compound during the assessment. For example, deviation from assess- assessment process, starting with one uncer- ment protocol (measurement error, Table 2, tainty type (in this case measurement error) that Cause 10, Type i) in some instances was simply leads to another (context dependence). However, the situation where one of the assessors, this also implies that by addressing some of the although assessing a species as data deficient more manageable causes of assessor difference, did actually provide a mechanism of impact, uncertainty propagation could feasibly be rather than assigning none (e.g., oleander scale, reduced (see recommendations below on embed- Aspidiotus nerii; Table 2). This tended to occur ding formal systematic review protocols into when the assessor was familiar with the species environmental impact assessment protocols to and providing an educated judgment as to the facilitate the transparency, efficiency, and most likely mechanism of impact. For example, repeatability of literature screening and inclusion Lepidoptera are likely to affect via herbivory (Siddaway et al. 2019, Haddaway et al. 2020)). and carabids by predation (or competition). Dis- The complexity of assessing environmental crepancies of this kind are unlikely to have any impacts extends beyond the broad range of eco- serious consequences for species prioritization, logical systems that species invade to the range and by extension resource allocation, and can and complexity of mechanisms of impact inva- be readily avoided. Extrapolating evidence to sive alien species can have. One of the most fre- assign an impact severity beyond that provided quent mechanism categories was other; that is, by the available evidence is, however, more the evidence suggested a mechanism not cur- problematic (i.e., subjective judgment, Table 2, rently listed under the EICAT protocol. For Cause 9, Type iii). This was one of the more fre- example, the () quent causes of uncertainty associated with was assessed as having an impact by altering the impact severity. Subjective judgment of this sex ratio of native species (Schonrogge€ et al. form stemmed primarily from the expert knowl- 2000), a mechanism that does not easily fit within edge of those performing assessments on spe- any of the other mechanism categories. Another cies with which they are particularly familiar. example is the Asian honeybee (Apis cerana) that For example, impact severity for the hemlock robs food stores of native species (Hyatt 2012) woolly adelgid (Adelges tsugae) differed as a and was noted as possibly fitting under multiple result of this cause of uncertainty (Table 2). mechanism categories. Differences in mechanism Extended familiarity with a given species may choice between assessors involving the use of reduce objectivity leading to a more liberal, but other often also involved the mechanism “inter- not necessarily wrong, interpretation of impact action with other alien species” or “facilitation of severity. This subjectivity can result in an assess- native species.” This may imply that other is ment at odds with an assessor who is less famil- often used to denote some form of indirect inter- iar with the same species, but who therefore action that is associated with a negative environ- bases their assessment on a more objective inter- mental impact. If the outcome of an assessment pretation of the evidence. We suggest that this is the assignment of other for the mechanism of cause of uncertainty could be reduced by explic- impact, then the assessor should provide a con- itly incorporating, within the assessment frame- cise description of what they interpret the mech- work, opportunity for expert opinion to be anism to be. Such information would be useful provided in addition to and alongside the not only for the assessment itself, but also for assessment based on published evidence alone potentially updating the framework to include a (Morgan 2014; see recommendations below).

v www.esajournals.org 16 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

Most types of uncertainty encountered were the outputs of tools such as EICAT. The epistemic in nature, a similar result to other stud- IUCN Red List is used as a communication ies on the presence and effect of assessment uncer- tool in many different contexts, and EICAT tainty (McGeoch et al. 2012, Kujala et al. 2013). could benefit from exploring the most effec- The disproportionate occurrence of epistemic tive approaches employed. Particular atten- causes of uncertainty, however, does not down- tion should be given to transparent play the importance of linguistic uncertainty communication of uncertainty to increase (Table 2, Cause 7, Type v). Historically over- understanding of risk by user groups. looked, this pervasive type of uncertainty can be 2. For an impact classification to convey the influential, problematic, and difficult to reduce most accurate representation of the evi- (Regan et al. 2002, Burgman 2005, Carey and dence, assessments should be performed as Burgman 2008). This is particularly true in inva- objectively as possible. Although perhaps sion biology where variation in definitions and self-evident, it is reasonable to assume that their use have hindered progress in the field (Ver- some assessors may use expert knowledge brugge et al. 2016, Courchamp et al. 2017). Cryp- of a certain species to extrapolate beyond ticity, an aspect of biological invasions that can available evidence. To accommodate this lead to underestimation of impacts, is also in part valuable information, we suggest that proto- affected by linguistic uncertainty due to the cols include the opportunity for assessors to dynamic nature of taxonomy (Jaric et al. 2019). provide their expert judgment, in addition Regan et al. (2002) characterized this as a form of to but separate from the objective results of theoretical indeterminacy where a term, or in this the assessment itself. This could allow for case a species name, will not necessarily retain its what is analogous to uncertainty bounds meaning into the future. Advances in molecular around the final assessment outcome (e.g., biology may unveil previously unknown species as portrayed in Fig. 3). complexes or reveal that specific sub-species that 3. Many assessment differences involving the are in fact alien to a given region (Jaric et al. 2019). use of other also involved mechanisms indicative of indirect or higher order interac- Recommendations for reducing uncertainty in tions. As we found here, indirect interac- impact classification for IAS tions can involve facilitation of native Certain causes of uncertainty are unavoidable species, as well as facilitation by native spe- (albeit theoretically reducible, e.g., human error), cies in the establishment of introduced and their presence and potential impact on organisms (Northfield et al. 2018). The use assessment outcomes can only be acknowledged of any attribution of other as mechanism of (Humair et al. 2014), whereas other causes can impact, as was frequent in this study, should potentially be minimized. Based on a structured always be accompanied by a description of process for identifying causes of uncertainty, we the impact mechanism. This information recommend the following six steps to reduce would not only enrich the impact assess- uncertainty in impact classification for alien spe- ment but would also assist in future devel- cies of environmental concern. opments of the assessment framework to include new mechanisms of impact should 1. The dissemination of information that similar descriptions be repeated over time. informs risk analysis, including impact clas- 4. An unambiguous definition for what quali- sification, is often overlooked (Vanderho- fies as data deficient for environmental even et al. 2017). Not only is it important to impact should be established to reduce mis- communicate the conclusions of impact interpretation about what evidence to assessments, such as EICAT, but also to include in an assessment. Invasive species ensure the degree of uncertainty is transpar- may have a negative environmental impact ently and effectively shared with end-users. in several different settings, including agri- There is a need to develop and implement culture, forestry, and urban environments. effective risk communication methods that Emphasis should be placed on what part of are appropriate to the target audience using the environment is being negatively affected

v www.esajournals.org 17 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

in addition to where the impact is taking biosecurity actions and investment in pest man- place. In other words, the environmental agement, being able to reduce the uncertainty context of the evidence should be specified associated with the outcomes of impact assess- and used as part of the rationale for includ- ments is important. Incorporating the above rec- ing or excluding a given piece of evidence. ommendations is likely to increase the accuracy, 5. The use of a detailed decision tree to harmo- transparency, and reliability of the alien impact nize and capture the decision-making pro- categorizations. Regardless of the type (epistemic cess (such as Fig. 5) would improve the or linguistic), being aware of the potential causes transparency and repeatability of assess- of uncertainty in alien species impact assess- ments, and formally distinguish between ments, and to a greater extent assessment pro- evidence of no impact and no evidence of cesses in general, is important if we are to reduce impact, and therefore improve the rigor of uncertainty as much as possible and base our assigning confidence to each decision. management and prioritization decisions on the 6. Systematic literature review protocols most accurate and reliable information available should be explicitly integrated into the (Hamel and Bryant 2017, Latombe et al. 2019). impact assessment guidelines and literature This is particularly relevant given the current searches performed using established best Intergovernmental Science-Policy Platform on practice and formal protocols such as the Biodiversity and Ecosystem Services (IPBES) Guidelines and Standards for Evidence Syn- assessment on invasive alien species and their thesis in Environmental Management (CEE control (IPBES 2018). 2018, Siddaway et al. 2019, Haddaway et al. 2020). One of the more frequent causes of ACKNOWLEDGMENTS uncertainty was the use of different litera- ture by assessors. Differences in evidence We acknowledge financial support from the Inva- accrual procedures have also been identified sive Species Council, the Ian Potter Foundation, Mon- to affect comparability of various impact ash University, the Australian Department of assessment frameworks (Strubbe et al. Agriculture, and the Queensland Department of Envi- 2019). Following such protocols would ronment. We thank Andrew Cox, Carol Booth, and Rebecca O’Connor, for their comments. MAM allow transparency and repeatability of the acknowledges support from the Australian Research literature search process and should reduce Council (DP200101680). DAC acknowledges support the occurrence of different literature use from an Australian Government Research Training between assessors for the same species. In Program (RTP) scholarship. SLC is supported by Aus- addition, given the pervasive uncertainty tralian Antarctic Science Grant 4482. SK thanks the associated with taxonomy (Pysek et al. South African National Department of Environment, 2013), all known synonyms of the accepted Forestry and Fisheries (DEFF) and the DSI-NRF Centre scientific name of a species should be of Excellence for Invasion Biology (CIB) at Stellen- included in the search terms (Haddaway bosch University. HER was partly supported by the et al. 2020), thus ensuring the inclusion of Natural Environment Research Council award number historical information. This is also why NE/R016429/1 as part of the UK-SCAPE program delivering National Capability. AML received support assessments ultimately will benefit from from the National Science Foundation Macrosystems multiple people independently assessing a Biology Program (grant numbers 1241932, 1638702) given species. This minimizes the chance of and grant EVA4.0, No. CZ.02.1.01/0.0/0.0/16_019/ potentially missing evidence of impact and 0000803 financed by OP RDE. bolsters confidence in the final decision (if fi there is consistent agreement) or identi es LITERATURE CITED problematic species (if there is consistent disagreement). Alderson, P. 2004. Absence of evidence is not evidence of absence. BMJ 328:476–477. With processes to prioritize alien species, Altman, D. G., and J. M. Bland. 1995. Statistics notes: including information obtained from impact Absence of evidence is not evidence of absence. assessments, becoming important to guide BMJ 311:485.

v www.esajournals.org 18 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

Bertelsmeier, C., S. Ollier, A. Liebhold, and L. Keller. possible solutions. Trends in Ecology & Evolution 2017. Recent human history governs global ant 32:13–22. invasion dynamics. Nature Ecology & Evolution Crystal-Ornelas, R., and J. L. Lockwood. 2020. The 1:0184. ‘known unknowns’ of invasive species impact mea- Blackburn, T. M., et al. 2014. A unified classification of surement. Biological Invasions 22:1513–1525. alien species based on the magnitude of their envi- Evans, J., A. Brazenor, B. Hennecke, and S. Parsons. ronmental impacts. PLoS Biology 12:e1001850. 2017. Exotic invasive species: identification of spe- Bradshaw, C. J., B. Leroy, C. Bellard, D. Roiz, C. Albert, cies with environmental impacts. ABARES Techni- A. Fournier, M. Barbet-Massin, J.-M. Salles, F. cal report 17.3, Canberra. Simard, and F. Courchamp. 2016. Massive yet Evans, T., S. Kumschick, and T. M. Blackburn. 2016. grossly underestimated global costs of invasive Application of the environmental impact classifica- insects. Nature Communications 7:12986. tion for alien taxa (EICAT) to a global assessment Brockerhoff, E. G., J. Bain, M. Kimberley, and M. of alien bird impacts. Diversity and Distributions Knızek. 2006. Interception frequency of exotic bark 22:919–931. and ambrosia (Coleoptera: Scolytinae) and Gonzalez-Moreno, P., et al. 2019. Consistency of relationship with establishment in New Zealand impact assessment protocols for non-native spe- and worldwide. Canadian Journal of Forest cies. NeoBiota 44:1–25. Research 36:289–298. Gregory, R., L. Failing, M. Harstone, G. Long, T. Burgman, M. 2005. Kinds of uncertainty. In Pages 26– McDaniels, and D. Ohlson. 2012. Structured deci- 41. Risks and decisions for conservation and envi- sion making: a practical guide to environmental ronmental management. Cambridge University management choices. Wiley-Blackwell, Oxford, Press, Cambridge, UK. UK. Burgman, M. A., M. McBride, R. Ashton, A. Speirs- Haack, R. A. 2006. Exotic bark-and wood-boring Bridge, L. Flander, B. Wintle, F. Fidler, L. Rumpff, Coleoptera in the United States: recent establish- and C. Twardy. 2011. Expert status and perfor- ments and interceptions. Canadian Journal of For- mance. PLOS ONE 6:e22998. est Research 36:269–288. Cameron, E. K., M. Vila, and M. Cabeza. 2016. Global Haddaway, N. R., A. Bethel, L. V. Dicks, J. Koricheva, meta-analysis of the impacts of terrestrial inverte- B. Macura, G. Petrokofsky, A. S. Pullin, S. Savi- brate invaders on species, communities and laakso, and G. B. Stewart. 2020. Eight problems ecosystems. Global Ecology and Biogeography with literature reviews and how to fix them. Nat- 25:596–606. ure Ecology & Evolution 4:1582–1589. Canavan, S., S. Kumschick, J. J. Le Roux, D. M. Hadley, A. S., and M. G. Betts. 2012. The effects of Richardson, and J. R. Wilson. 2019. Does origin landscape fragmentation on pollination dynamics: determine environmental impacts? Not for bam- absence of evidence not evidence of absence. Bio- boos. Plants, People, Planet 1:119–128. logical Reviews 87:526–544. Carey, J. M., and M. A. Burgman. 2008. Linguistic Hagen, B. L., and S. Kumschick. 2018. The relevance of uncertainty in qualitative risk analysis and how to using various scoring schemes revealed by an minimize it. Annals of the New York Academy of impact assessment of feral mammals. NeoBiota Sciences 1128:13–17. 38:37. Caton, B. P., A. L. Koop, L. Fowler, L. Newton, and L. Hamel, P., and B. P. Bryant. 2017. Uncertainty assess- Kohl. 2018. Quantitative uncertainty analysis for a ment in ecosystem services analyses: seven chal- weed risk assessment system. Risk Analysis lenges and practical responses. Ecosystem Services 38:1972–1987. 24:1–15. Clement, S., D. Hardie, and L. Elberson. 2002. Varia- Harwood, J., and K. Stokes. 2003. Coping with uncer- tion among accessions of Pisum fulvum for resis- tainty in ecological advice: lessons from fisheries. tance to pea weevil. Crop Science 42:2167–2173. Trends in Ecology & Evolution 18:617–622. Collaboration for Environmental Evidence. 2018. Hawkins, C. L., et al. 2015. Framework and guidelines Guidelines and Standards for Evidence synthesis for implementing the proposed IUCN Environ- in Environmental Management. Version 5.0 (A. S. mental Impact Classification for Alien Taxa Pullin, G. K. Frampton, B. Livoreil, and G. (EICAT). Diversity and Distributions 21:1360– Petrokofsky, editors) www.environmentalevidence. 1363. org/information-for-authors/ Humair, F., P. J. Edwards, M. Siegrist, and C. Kueffer. Courchamp, F., A. Fournier, C. Bellard, C. Ber- 2014. Understanding misunderstandings in inva- telsmeier, E. Bonnaud, J. M. Jeschke, and J. C. Rus- sion science: Why experts don’t agree on common sell. 2017. Invasion biology: specific problems and concepts and risk assessments. NeoBiota 20:1.

v www.esajournals.org 19 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

Hyatt, S. 2012. Asian (Apis cerana javana)in Latombe, G., et al. 2019. A four-component classifica- Cairns, Far North Queensland: foraging, nesting tion of uncertainties in biological invasions: impli- and swarming behaviour. State of Queensland, cations for management. Ecosphere 10:e02669. Department of Employment, Economic Develop- Lovett, G. M., et al. 2016. Nonnative forest insects ment and Innovation, Brisbane, Queensland, Aus- and pathogens in the United States: impacts and tralia. policy options. Ecological Applications 26:1437– IPBES. 2018. Deliverable 3(b)(ii): thematic assessment 1455. on invasive alien species and their control. 6th Ple- Masetti, A., S. Magagnoli, F. Lami, A. Lanzoni, and G. nary of the Intergovernmental Science-Policy Plat- Burgio. 2018. Long term changes in the communi- form on Biodiversity and Ecosystem Services. Page ties of native ladybirds in Northern Italy: impact of 6. the invasive species Harmonia axyridis (Pallas). IUCN. 2020a. IUCN EICAT Categories and Criteria. BioControl 63:665–675. The Environmental Impact Classification for Alien Mastrandrea, M. D., K. J. Mach, G.-K. Plattner, O. Taxa (EICAT). First edition. IUCN, Gland, Switzer- Edenhofer, T. F. Stocker, C. B. Field, K. L. Ebi, and land and Cambridge, UK. http://dx.doi.org/10. P. R. Matschoss. 2011. The IPCC AR5 guidance 2305/IUCN.CH.2020.05.en note on consistent treatment of uncertainties: a IUCN. 2020b. Guidelines for using the IUCN Environ- common approach across the working groups. Cli- mental Impact Classification for Alien Taxa matic Change 108:675. (EICAT) Categories and Criteria. Version 1.1. McGeoch, M. A., P. Genovesi, P. J. Bellingham, M. J. IUCN, Gland Switzerland, Cambridge, UK. https:// Costello, C. McGrannachan, and A. Sheppard. www.iucn.org/sites/dev/files/content/documents/ 2016. Prioritizing species, pathways, and sites to eicat_guidelines_v1.1.pdf achieve conservation targets for biological inva- Jaric, I., T. Heger, F. C. Monzon, J. M. Jeschke, I. sion. Biological Invasions 18:299–314. Kowarik, K. R. McConkey, P. Pysek, A. Sagouis, McGeoch, M. A., M. J. Lythe, M. V. Henriksen, and C. and F. Essl. 2019. Crypticity in biological invasions. M. McGrannachan. 2015. Environmental impact Trends in Ecology & Evolution 34:291–302. classification for alien insects: a review of mecha- Kenis, M., M.-A. Auger-Rozenberg, A. Roques, L. nisms and their biodiversity outcomes. Current Timms, C. Pere, M. J. Cock, J. Settele, S. Augustin, Opinion in Insect Science 12:46–53. and C. Lopez-Vaamonde. 2009. Ecological effects McGeoch, M. A., D. Spear, E. J. Kleynhans, and E. Mar- of invasive alien insects. Biological Invasions ais. 2012. Uncertainty in invasive alien species list- 11:21–45. ing. Ecological Applications 22:959–971. Kesner, D., and S. Kumschick. 2018. Gastropods alien Menke, S. B., P. S. Ward, and D. A. Holway. 2018. to South Africa cause severe environmental harm Long-term record of Argentine ant invasions in their global alien ranges across habitats. Ecology reveals enduring ecological impacts. Ecology and Evolution 8:8273–8285. 99:1194–1202. Kujala, H., M. A. Burgman, and A. Moilanen. 2013. Milner-Gulland, E., and K. Shea. 2017. Embracing Treatment of uncertainty in conservation under cli- uncertainty in applied ecology. Journal of Applied mate change. Conservation Letters 6:73–85. Ecology 54:2063–2068. Kumschick, S., S. Bacher, W. Dawson, J. Heikkila,€ A. Morgan, M. G. 2014. Use (and abuse) of expert elicita- Sendek, T. Pluess, T. Robinson, and I. Kuhn.€ 2012. tion in support of decision making for public pol- A conceptual framework for prioritization of inva- icy. Proceedings of the National Academy of sive alien species for management according to Sciences of the United States of America 111:7176– their impact. NeoBiota 15:69–100. 7184. Kumschick, S., G. J. Measey, G. Vimercati, F. A. De Vil- Noriega, J. A., et al. 2018. Research trends in ecosystem liers, M. M. Mokhatla, S. J. Davies, C. J. Thorp, A. services provided by insects. Basic and Applied D. Rebelo, T. M. Blackburn, and F. Kraus. 2017a. Ecology 26:8–23. How repeatable is the Environmental Impact Clas- Northfield, T. D., S. G. Laurance, M. M. Mayfield, D. sification of Alien Taxa (EICAT)? Comparing inde- R. Paini, W. E. Snyder, D. B. Stouffer, J.T. Wright, pendent global impact assessments of amphibians. and L. Lach. 2018. Native turncoats and indirect Ecology and Evolution 7:2661–2670. facilitation of species invasions. Proceedings of the Kumschick, S., G. Vimercati, F. A. De Villiers, M. M. Royal Society B: Biological Sciences 285:20171936. Mokhatla, S. J. Davies, C. J. Thorp, A. D. Rebelo, Pagad, S., P. Genovesi, L. Carnevali, D. Schigel, and M. and G. J. Measey. 2017b. Impact assessment with A. McGeoch. 2018. Introducing the global register different scoring tools: How well do alien amphib- of introduced and invasive species. Scientific Data ian assessments match? NeoBiota 33:53. 5:170202.

v www.esajournals.org 20 April 2021 v Volume 12(4) v Article e03461 CLARKE ET AL.

Pysek, P., et al. 2013. Hitting the right target: taxo- development and feeding behavior of Acan- nomic challenges for, and of, plant invasions. AoB thoscelides obtectus (Chrysomelidae: Bruchinae) on Plants 5:plt042. dry bean (Phaseolus vulgaris L.). Journal of Stored Regan, H. M., M. Colyvan, and M. A. Burgman. 2002. Products Research 61:90–96. A taxonomy and treatment of uncertainty for ecol- Sober, E. 2009. Absence of evidence and evidence of ogy and conservation biology. Ecological Applica- absence: evidential transitivity in connection with tions 12:618–628. fossils, fishing, fine-tuning, and firing squads. Ricciardi, A., M. F. Hoopes, M. P. Marchetti, and J. L. Philosophical Studies 143:63–90. Lockwood. 2013. Progress toward understanding Stephens, S. S., and M. R. Wagner. 2007. Forest planta- the ecological impacts of nonnative species. Eco- tions and biodiversity: a fresh perspective. Journal logical Monographs 83:263–282. of Forestry 105:307–313. Roy, H. E., et al. 2012. Invasive alien predator causes Stork, N. E., J. McBroom, C. Gely, and A. J. Hamilton. rapid declines of native European ladybirds. Diver- 2015. New approaches narrow global species esti- sity and Distributions 18:717–725. mates for beetles, insects, and terrestrial arthro- Roy, H. E., et al. 2018. Developing a framework of min- pods. Proceedings of the National Academy of imum standards for the risk assessment of alien Sciences of the United States of America 112:7519– species. Journal of Applied Ecology 55:526–538. 7523. Rueda-Cediel, P., K. E. Anderson, T. J. Regan, and H. Strubbe, D., R. White, P. Edelaar, C. Rahbek, and A. M. Regan. 2018. Effects of uncertainty and variabil- Shwartz. 2019. Advancing impact assessments of ity on population declines and IUCN Red List clas- non-native species: strategies for strengthening the sifications. Conservation Biology 32:916–925. evidence-base. NeoBiota 51:41. Sandvik, G., C. M. Jessup, K. L. Seip, and B. J. Bohan- Suckling, D. M., L. D. Stringer, D. B. Baird, and J. M. nan. 2004. Using the angle frequency method to Kean. 2019. Will growing invasive biodi- detect signals of competition and predation in versity outpace our ability for eradication? Ecologi- experimental time series. Ecology Letters 7:640– cal Applications 29:e01992. 652. Turbe, A., et al. 2017. Assessing the assessments: evalu- Sanguesa-Barreda,€ G., J. C. Linares, and J. J. Camarero. ation of four impact assessment protocols for inva- 2015. Reduced growth sensitivity to climate in sive alien species. Diversity and Distributions bark-beetle infested Aleppo pines: connecting cli- 23:297–307. matic and biotic drivers of forest dieback. Forest Vaes-Petignat, S., and W. Nentwig. 2014. Environmen- Ecology and Management 357:126–137. tal and economic impact of alien terrestrial arthro- Schonrogge,€ K., P. Walker, and M. Crawley. 2000. Para- pods in Europe. NeoBiota 22:23. sitoid and inquiline attack in the of four alien, Vanderhoeven, S., E. Branquart, J. Casaer, B. D’hondt, cynipid gall : host switches and the effect on P. E. Hulme, A. Shwartz, D. Strubbe, A. Turbe, H. parasitoid sex ratios. Ecological Entomology Verreycken, T. Adriaens. 2017. Beyond protocols: 25:208–219. improving the reliability of expert-based risk anal- Seebens, H., et al. 2017. No saturation in the accumula- ysis underpinning invasive species policies. Biolog- tion of alien species worldwide. Nature Communi- ical Invasions 19:2507–2517. cations 8:14435. Verbrugge, L. N., R. S. Leuven, and H. A. Zwart. 2016. Siddaway, A. P., A. M. Wood, and L. V. Hedges. 2019. Metaphors in invasion biology: implications for How to do a systematic review: a best practice risk assessment and management of non-native guide for conducting and reporting narrative species. Ethics, Policy & Environment 19:273–284. reviews, meta-analyses, and meta-syntheses. Vila, M., B. Gallardo, C. Preda, E. Garcıa-Berthou, F. Annual Review of Psychology 70:747–770. Essl, M. Kenis, H. E. Roy, and P. Gonzalez-Moreno. Soares, M. A., E. D. Quintela, G. M. Mascarin, and S. P. 2019. A review of impact assessment protocols of Arthurs. 2015. Effect of temperature on the non-native plants. Biological Invasions 21:709–723.

SUPPORTING INFORMATION

Additional Supporting Information may be found online at: http://onlinelibrary.wiley.com/doi/10.1002/ecs2. 3461/full

v www.esajournals.org 21 April 2021 v Volume 12(4) v Article e03461