Identifying Intervention Strategies for Preventing the Mental Health Consequences of Childhood Adversity: a Modified Delphi Study

Development and Psychopathology (2021), 33, 748–765 doi:10.1017/S0954579420002059

Special Issue Article

Identifying intervention strategies for preventing the mental health consequences of childhood adversity: A modified Delphi study

Leslie R. Rith-Najarian1 , Noah S. Triplett2, John R. Weisz1 and Katie A. McLaughlin1 1Department of Psychology, Harvard University, Cambridge, MA, USA and 2Department of Psychology, University of Washington, Seattle, WA, USA

Abstract Exposure to childhood adversity is a powerful risk factor for psychopathology. Despite extensive efforts, we have not yet identified effective or scalable interventions that prevent the emergence of mental health problems in children who have experienced adversity. In this modified Delphi study, we identified intervention strategies for effectively targeting both the neurodevelopmental mechanisms linking childhood adversity and psychopathology – including heightened emotional reactivity, difficulties with emotion regulation, blunted reward processing, and social information processing biases, as well as a range of psychopathology symptoms. We iteratively synthesized information from experts in the field and relevant meta-analyses through three surveys, first with experts in intervention development, prevention, and childhood adversity (n = 32), and then within our study team (n = 8). The results produced increasing stability and good consensus on intervention strategy recommendations for specific neurodevelopmental mechanisms and symptom presentations and on strength of evidence ratings of intervention strategies targeting youth and parents. More broadly, our findings highlight how intervention decision making can be informed by meta-analyses, enhanced by aggregate group feedback, saturated before consensus, and persistently subjective or even contradictory. Ultimately, the results converged on several promising intervention strategies for prevention programming with adversity-exposed youth, which will be tested in an upcoming clinical trial. Keywords: childhood adversity, Delphi study, intervention development, prevention, psychopathology (Received 29 October 2020; accepted 31 October 2020)

Introduction Miller, 2016; Felitti et al., 1998; Wegman & Stetler, 2009). A substantial body of work has characterized the developmental mech- Exposure to adversity is common in children, both in the USA anisms through which experiences of adversity influence risk for and internationally. Approximately half of all US children will psychopathology and chronic diseases (Cicchetti & Toth, 1995; experience at least one form of adversity – such as physical, sex- Danese & McEwen, 2012; McLaughlin, 2016; McLaughlin, ual, or emotional abuse, neglect, exposure to interpersonal vio- Colich, Rodman, & Weissman, 2020; McLaughlin & Lambert, lence, or chronic poverty – by the time they reach adulthood 2017). Although knowledge of these mechanisms has grown dra- (Benjet et al., 2009; Green et al., 2010; Kessler et al., 2010; matically, little progress has been made in translating those mech- McLaughlin et al., 2012). Evidence from population-based and anisms into interventions to prevent the onset of psychopathology longitudinal studies demonstrates that experiences of adversity on subsequent consequences (McLaughlin, DeCross, Jovanovic, & are associated with the onset of internalizing and externalizing Tottenham, 2019). Through our current work, we hope to carry psychopathology transdiagnostically, including anxiety, mood, on Edward Zigler’s legacy of “be[ing] totally committed to the disruptive behavior, posttraumatic stress, and substance use disor- optimal development of each child” by developing such an inter- ders (Cohen, Brown, & Smailes, 2001; Green et al., 2010; Kessler vention for children who have screened positive for adversity. In et al., 2010; MacMillan et al., 2001; McLaughlin, Conron, Koenen, contrast to the scarcity of research available prior to the develop- & Gilman, 2010; McLaughlin et al., 2012; Weich, Patterson, Shaw, ment of Head Start, considered “the prototype of effective early & Stewart-Brown, 2009). Exposure to childhood adversity childhood intervention,” we are fortunate to now be able to explains about a third of mental disorder onsets in the USA draw on the substantial evidence base supporting early childhood (Green et al., 2010; McLaughlin et al., 2012) and is associated intervention (Zigler & Styfco, 2001). with increased risk for a wide range of chronic diseases and In theory, screening and early intervention could help early mortality (Brown et al., 2009; Chen, Turiano, Mroczek, & adversity-exposed children by preventing future incidences of adversity along with its physical and mental health consequences. Author for Correspondence: Leslie Rith-Najarian, Department of Psychology, We agree with the belief that children who begin life healthy Harvard University, Room 1006, William James Hall, 33 Kirkland St, Cambridge, MA “grow to become contributing members of the society, [and] 02138, USA; E-mail: [email protected] the small investment made in their early years will have com- Cite this article: Rith-Najarian LR, Triplett NS, Weisz JR, McLaughlin KA (2021). pounded to reap a handsome dividend” (Zigler, 1996, p. 47). Identifying intervention strategies for preventing the mental health consequences of childhood adversity: A modified Delphi study. Development and Psychopathology 33, A 10% reduction in childhood adversity prevalence in the USA 748–765. https://doi.org/10.1017/S0954579420002059 has been estimated to save $105 billion annually (Bellis et al.,

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. Development and Psychopathology 749

2019). The broader the scale of screening and prevention efforts, PTSD but also with transdiagnostic psychopathology, including the greater the potential impact becomes. Pediatric primary care is a wide range of internalizing and externalizing problems (Green a promising setting for wide-reaching adversity screening and et al., 2010; Keyes et al., 2012; McLaughlin et al., 2012). prevention efforts, as the majority of US children see a primary Furthermore, such existing evidence-based interventions were care provider annually (Chevarley, 2003). Moreover, addressing designed as treatments for children who have already developed child mental health concerns in a medical setting has myriad ben- symptoms of psychopathology following traumatic events and efits, including reduced stigma, cost, and family discomfort, as thus primarily focus on symptom reduction rather than prevent- well as improved outcomes (Asarnow, Rozenman, Wiblin, & ing the emergence of psychopathology. This distinction is impor- Zeltzer, 2015; Godoy et al., 2017; Kolko & Perrin, 2014). tant as the mechanisms that underlie symptom maintenance may Primary care settings are already increasingly integrating youth differ from those that contribute to the emergence of psychopa- mental health care (Aupont et al., 2013), with approximately thology following experiences of adversity. As such, prevention half of all visits now involving some type of behavioral or mental practices that target the developmental pathways linking adversity health concern (Martini et al., 2012). Simultaneously, there has to the onset of psychopathology – particularly mechanisms that been a movement encouraging pediatricians to screen for adver- are associated with transdiagnostic risk – may ultimately hold sity within routine primary care visits (Garner et al., 2012; the most promise. Pardee, Kuzma, Dahlem, Boucher, & Darling-Fisher, 2017). Many have argued that greater attention to the neurodevelop- One encouraging example in primary care settings is the Safe mental pathways linking childhood adversity with psychopathol- Environment for Every Kid (SEEK) program (Dubowitz, 2014; ogy is needed in order to improve our ability to effectively Dubowitz, Feigelman, Lane, & Kim, 2009), which includes a intervene with children who have encountered adversity validated short adversity screening tool and a brief intervention (Bentovim et al., 2020; Cicchetti & Toth, 2009; Manly, Kim, (5–15 min) of motivational interviewing and community resource Rogosch, & Cicchetti, 2001; Masten & Cicchetti, 2010; referrals for families who endorse adversity experiences McLaughlin et al., 2019; van IJzendoorn et al., 2020). Over the (Dubowitz et al., 2007; Dubowitz, Prescott, Feigelman, Lane, & past decade, a substantial body of evidence has emerged on Kim, 2008; Feigelman et al., 2009; Lane et al., 2007). these mechanisms. As noted in recent reviews of this literature, Randomized controlled trials of SEEK have shown reduced rates developmental pathways that have garnered the most consistent of future child maltreatment (Dubowitz et al., 2009; Dubowitz, evidence for being influenced by experiences of adversity and Lane, Semiatin, & Magder, 2012), and feasibility studies have that, in turn, are associated with the later onset of psychopathol- shown high acceptability and satisfaction among providers and ogy include threat-related social information processing biases caregivers in primary care settings (e.g., Eismann, Theuerling, (e.g., hostile attribution bias), heightened emotional reactivity, dif- Maguire, Hente, & Shapiro, 2019). However, despite all this pro- ficulties with emotion regulation, and blunted reward processing gress, one concern about adversity screening is the dearth of inter- (McLaughlin, 2016; McLaughlin et al., 2019). Threat-related social ventions that can prevent the mental health outcomes associated information processing biases have been observed most consis- with adversity exposure (Campbell, 2020). This scarcity is prob- tently among children exposed to trauma and forms of adversity lematic, as it has been recommended that youth receive evidence- involving threat (e.g., violence exposure, abuse), whereas blunted based prevention strategies that are matched to their screened reward processing has been observed most consistently in chil- risks for psychopathology (Hankin, 2020). There thus remains a dren who have experienced forms of adversity involving depriva- major challenge for adversity screening efforts in primary care tion, including neglect, caregiver separation, and food insecurity. or elsewhere: we must respond to those youth identified as Heightened emotional reactivity and difficulties with emotion adversity-exposed and yet we lack effective strategies for prevent- regulation have been observed in children exposed to many ing the associated psychopathology. forms of adversity (for a review, see McLaughlin et al., 2019). The absence of effective preventive interventions for children Correspondingly, McLaughlin et al. (2019) posit that these mech- exposed to adversity does not reflect a lack of research attention anisms could serve as important behavioral targets for interven- to this issue. Dozens of psychosocial prevention and early inter- tions aimed at preventing psychopathology transdiagnostically. vention programs for maltreated children have been developed Zigler (2000) also suggested that research on brain development and tested, but meta-analyses examining them have not produced had important implications for early childhood interventions – substantive evidence of intervention effectiveness (Churchill, specifically, that such research has made us more certain than 2016; van IJzendoorn, Bakermans-Kranenburg, Coughlan, & ever that adverse environments alter processes such as learning Reijman, 2020). There are some effective interventions for chil- and emotion regulation. Regardless of such suggestions, we dren experiencing psychopathology after exposure to trauma know of no existing prevention program designed to specifically (e.g., abuse, exposure to violence), such as trauma-focused cogni- target the developmental processes that are influenced by expo- tive behavioral therapy (TF-CBT; Cohen, Deblinger, Mannarino, sure to adversity and, in turn, predict the later emergence of & Steer, 2004; Cohen, Mannarino, & Iyengar, 2011; Dorsey psychopathology. et al., 2017). However, these interventions are treatment level, and youth psychological treatments typically involve a large num- Current Research ber of sessions (averaging 16.5 in a recent meta-analysis; Weisz, Kuppens, et al., 2017), which is less practical for typical behavioral In this article we describe a series of studies investigating the health session limits. More feasible for integration into primary development and testing of an intervention program through care would be, for example, a brief (four-session) skills-based pediatric primary care settings to prevent the mental health con- intervention that has demonstrated effective prevention of post- sequences of exposure to childhood adversity. Our planned inter- traumatic stress disorder (PTSD) symptoms in children who vention will incorporate the evidence-based SEEK screening and have experienced trauma (Berkowitz, Stover, & Marans, 2011). case management approach (Dubowitz, 2014; Dubowitz et al., Nevertheless, adversity experiences are associated not only with 2009). We will then extend services by offering a brief

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. 750 L. R. Rith‐Najarian et al.

Table 1. Specific intervention strategies within the FIRST principles

Trying the Feeling calm Increasing motivation Repairing thoughts Solving problems opposite

▪ Guided relaxation ▪ Positive attending and ▪ Identifying unhelpful, ▪ Identifying problems, ▪ Exposure ▪ Self-calming praise distorted thoughts conflicts, dilemmas ▪ Behavioral ▪ Progressive muscle ▪ Differential attention ▪ Cognitive restructuring and ▪ Generating a list of possible activation relaxation ▪ Effective instructions and thought challenging solutions ▪ Anger control ▪ Deep breathing house rules ▪ Cognitive distraction ▪ Weighing pros and cons to skills ▪ Guided imagery ▪ Reinforcement with ▪ Attribution retraining select best option ▪ Role playing tangible rewards ▪ Time-out and response cost ▪ Self-reinforcement

intervention (four to six sessions) to families with children Benchmarking trials of FIRST have shown rapid reductions in who screen positive for adversity exposure. As Zigler and col- child problems and symptoms – assessed via standardized mea- leagues have argued, parents and caregivers play a central role sures of internalizing and externalizing problems and via idio- in children’s behavioral development (Zigler, Pfannenstiel, & graphic measures of the “top problems” most important to Seitz, 2008) and, correspondingly, their response to behavioral children and their caregivers – and with trajectories of improve- health interventions (Sun, Rith-Najarian, Williamson, & ment similar to those seen with markedly more complicated treat- Chorpita, 2019). Accordingly, our brief intervention will intro- ments that require substantially more training (Cho, Bearman, duce coping strategies for the children and adolescents, and par- Woo, Weisz, & Hawley, 2020; Weisz, Bearman, Santucci, & enting strategies for caregivers. Jensen-Doss, 2017). Though we ought not simply repackage treatment strategies for preventive purposes, we unfortunately lack strongly supported PCIT (parent–child interaction therapy) prevention strategies for adversity-exposed youth. In contrast, the PCIT is an intervention focused on improving parent–child inter- development of new intervention strategies for youth mental actions that has demonstrated large and sustained effects for treat- health treatments seems to have reached saturation (Okamura ing youth with disruptive behavior problems (Brestan & Eyberg, et al., 2020). As such, the vast array of intervention components 1998; McNeil & Hembree-Kigin, 2010; Thomas & from youth treatment programs may still provide some candidates Zimmer-Gembeck, 2007). Critical for our purposes, PCIT has for our prevention purposes, though other prevention strategies also been shown to reduce the rate of future maltreatment when likely remain to be discovered through empirical research. delivered to parents (or caregivers) who had abused their children Given that experiences of adversity are associated with psychopa- (Chaffin et al., 2004; Timmer, Urquiza, Zebell, & McGrath, 2005). thology transdiagnostically (e.g., Keyes et al., 2012), it is essential PCIT includes several intervention elements that are taught through that intervention strategies are flexible enough to target a range of behavioral observations of parent–child interactions and live coach- internalizing and externalizing problems and the variable needs of ing of these interactions by a therapist. PCIT includes two primary children and families. Accordingly, we selected two evidence- intervention components: child-directed interaction and based interventions – each of which includes multiple interven- parent-directed interaction. In child-directed interaction, parents tion elements – from which to pull candidate intervention strate- learn a variety of techniques designed to enhance the parent– gies: FIRST (Weisz & Bearman, 2020) and PCIT-ED (Luby, child relationship. These include techniques praise, reflecting, imi- Barch, Whalen, Tillman, & Freedland, 2018). tating, and describing the child’s behavior, and providing contin- gent attention (i.e., attending to positive behaviors and ignoring negative behaviors). Parent-directed interaction provides parents Selection of candidate intervention strategies with training in effective discipline strategies designed to increase FIRST (Feeling calm, Increasing motivation, Repairing thoughts, child compliance and decrease disruptive behaviors. PCIT has Solving problems, Trying the opposite) been expanded to include additional elements designed to target FIRST is a good fit to our goals for several reasons. First, it is internalizing problems (Carpenter, Puliafico, Kurtz, Pincus, & transdiagnostic, encompassing intervention procedures for an Comer, 2014). For our purposes, we focus on PCIT plus emotional array of problems and concerns, including posttraumatic stress, development (PCIT-ED), which includes an additional module anxiety, depressed mood, and conduct problems. Second, it is focused on parenting techniques to help children identify and reg- designed for personalization – for tailoring to fit the specific com- ulate their emotional experiences (Lenze, Pautsch, & Luby, 2011). bination of needs of each individual child and family. Third, it is PCIT-ED has been shown to significantly reduce depression symp- streamlined, efficient and accessible to clinicians, organizing toms in both children and their caregivers (Luby et al., 2018; Luby, many specific evidence-based procedures within five broad prin- Lenze, & Tillman, 2012). More importantly for prevention, another ciples of change. These five principles are Feeling calm, Increasing trial showed that, relative to a waitlist, PCIT-ED produced greater motivation, Repairing thoughts, Solving problems, and Trying the changes in reward processing, which were in turn associated with opposite (see Table 1 for details). Clinicians using FIRST are greater improvements in depression symptoms (Barch et al., encouraged to identify which of these principles may be most rel- 2019). Thus, this research suggests that some interventions evant to the child they are working with, and then apply that prin- designed as treatments for youth psychopathology may also influ- ciple in their work with the child, drawing from the specific ence key developmental mechanisms involved in the link between procedures grouped beneath that principle, as shown in Table 1. adversity and psychopathology.

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. Development and Psychopathology 751

Study aims our intervention development and, later, their actual effects on mechanisms and symptoms empirically tested in future clinical The success of prevention and early intervention programs trials of the prevention program. An exploratory additional aim depends in large part on their design, and incorporated elements was to gain insight into how evidence-informed decision making should be preventive as opposed to purely remedial (Zigler, 1996). in intervention development unfolds. On the one hand, many children and adolescents who screen positive for adversity exposure may already be symptomatic. Thus, these youth will likely be well served by intervention strategies Method from FIRST and PCIT-ED. On the other hand, many other youth who screen positive may not yet be experiencing clinically All procedures were reviewed and approved by the Institutional significant mental health symptoms. Therefore, we still need to Review Board of Harvard University. identify which candidate intervention strategies are more likely to target not only symptoms but also the neurodevelopmental Overview of the Delphi approach mechanisms that put a youth at risk of eventual symptom onset (i.e., threat-related social information processing biases, height- Traditional Delphi studies aim to gather expert input iteratively ened emotional reactivity, difficulties with emotion regulation, over multiple rounds until consensus is achieved, with the first and blunted reward processing; McLaughlin et al., 2019). Such round generating qualitative data that then informs the subse- an approach would be consistent with the American quent items (Hsu & Sandford, 2007). However, given that our Psychological Association (APA) prevention guidelines to develop aim was not necessarily to reach consensus but to assess for con- programs that reduce risks and promote human strengths, as sensus, we conducted a modified Delphi study. First, we invited opposed to strategies that focus only on symptom reduction experts to recommend candidate intervention strategies based (American Psychological Association, 2014). on the five broad principles from FIRST (Weisz & Bearman, To this end, we conducted a modified Delphi study to gather 2020) and three parenting practices from PCIT-ED (Luby et al., feedback on the intervention strategies that we selected as candi- 2018). Second, we invited a subset of experts – members of our dates for our prevention program for adversity-exposed youth. A broader study team – to identify relevant systematic reviews and lack of expert consensus about which youth mental health inter- then consider the strength of evidence for intervention strategies ventions are considered “evidence based” has been a longstanding based on meta-analytic support for practice elements. Finally, our issue related to the science-to-practice gap (Weisz, Sandler, team reviewed anonymous and objective summaries of the rec- Durlak, & Anton, 2005), and that issue is even more salient ommendations from experts and systematic reviews together. when developing a novel intervention. Consensus methods can This approach was designed to produce more informed conclu- enhance decision making in health research by systematically sions than we could reach as individuals, while addressing some assessing expert recommendations for topics that have a dearth limitations of group decision making (e.g., influence of relative of scientific agreement or evidence (Trevelyan & Robinson, authority figures, tendency towards group conformity or polariza- 2015). It has been recommended to combine the existing scientific tion) (Hsu & Sandford, 2007; Surowiecki, 2004). evidence base with the current widespread professional norms (i.e., expert opinions) to maximize effective clinical care (Campbell, Survey 1 Roland, & Buetow, 2000). It has also been noted that, when using Delphi methods in mental health research, the quality of Participants input from experts depends on the quality of input provided to We identified experts whose research focuses on child and/or ado- them (e.g., systematic reviews, personal experience) (Jorm, 2015). lescent mental health in at least one of the following categories of Systematic reviews are a gold standard for assessing the current relevance: developmental psychopathology research focused on state of effectiveness for youth treatments and prevention programs mechanisms linking adversity with psychopathology; prevention (Weisz et al., 2005) and have been specifically recommended as a science; intervention development or treatment-focused research; strong form of evidence for identifying effective components of and dissemination and implementation science. Potential partici- child maltreatment interventions (Powell et al., 2015). We thus pants were identified through four sources: (a) the principal planned to synthesize feedback across survey rounds as informed investigator (PI), coinvestigators (CIs), and established consul- by two sources of knowledge – experts and meta-analyses. tants of the intervention research project; (b) scholars nominated Our primary aim was to evaluate consensus on recommenda- by these members of the research team as experts in the domains tions and strength of evidence ratings for the candidate interven- of interest; (c) members of relevant professional task forces and tion strategies. We recognize that reaching consensus on executive boards; and (d) PIs of relevant preventive intervention recommendations is not necessarily proof that those interventions trials currently active on clinicaltrials.gov. Participants were eligi- are the “best” or “correct” recommendations (Trevelyan & ble to participate if the majority of their work was in one of the Robinson, 2015). Still, even if there is not always a single best sol- categories of interest (as determined by their professional website ution, it is helpful to prioritize options from a list of candidates or CV) and they were still active in their career (e.g., not retired, (Hall, 2009). We also believe that a more extensive approach to had published research in one of the categories of interest within planning intervention design is consistent with the cautionary the past 2 years). We focused on identifying experts across a range notes of Zigler regarding “hasty” prevention development and of populations served (e.g., racial and ethnic minorities, rural and implementation (Zigler, 1996; Zigler & Styfco, 2001). The find- urban populations, and so on). A list of potential participants was ings of the current study could provide some foundational evi- iteratively compiled and revised to limit redundancy and broaden dence to more strongly hypothesize which intervention representative coverage of expertise areas. Recruitment was con- strategies might be mechanism-targeting prevention approaches. ducted by the PI/CIs through email invitations that explained The results of skills recommendations can be directly applied to the purpose of the study, the rationale for expert selection, and

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. 752 L. R. Rith‐Najarian et al.

Table 2. Demographics of participants in Survey 1 intervention recommendations were provided for a total of eight targets – four about developmental mechanisms linking adversity N % and psychopathology, and four about mental health symptoms. In Age each subsection, the survey provided examples of adversity experiences that have been commonly associated with each develop- 26–35 years 3 9.4 mental mechanism or symptom type, and included examples of 36–45 years 9 28.1 how each developmental mechanism or symptom type might pre- 46–55 years 9 28.1 sent clinically or subclinically (e.g., heightened emotional reactivity may present as rapid mood changes or being quickly Over 55 years 11 34.4 consumed by intense emotions; depressive symptoms may present Gender identity as sad mood or loss of pleasure in activities that used to be enjoy- Female 18 56.3 able). Examples of each developmental mechanism were taken from standardized measures or published instruments, including: Male 14 43.8 the Behavioral Inhibition System/Behavioral Activation System Other 0 0.0 scales for blunted reward processing (Carver & White, 1994); Racea the Emotion Reactivity Scale (Nock, Wedig, Holmberg, & Hooley, 2008) for heightened emotional reactivity; the Asian 3 9.3 Children’s Response Styles Questionnaire (Abela, Vanderbilt, & Black or African American 3 9.3 Rochon, 2004) and Zeman and Garber’s(2016) emotion regula- Native American or Alaska Native 1 3.1 tion vignette coding system for emotion regulation difficulties; and the Social Information Processing Application (Kupersmidt, White 24 75.0 Stelter, & Dodge, 2011) for social information processing Middle Eastern 2 6.3 biases. Examples for each of the mental health symptoms were Multiple racial groups 1 3.1 adapted from the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (American Psychiatric Hispanic or Latinx Association, 2013) to remove “clinical-level” qualifying language. Yes 2 6.3 Anxiety symptom examples were informed by the symptom crite- Highest education completeda ria in the generalized anxiety disorder, panic disorder, social anxiety disorder, specific phobia, and separation anxiety disorder Doctoral degree (e.g., PhD, PsyD) 28 88.0 sections of the DSM-5. Depression symptom examples were Doctor of medicine (e.g., MD, DO) 5 15.6 informed by the symptom criteria in the major depressive disor- Master’s degree (e.g., MS, MSW) alone 1 3.1 der and persistent depressive disorder sections. Posttraumatic stress symptom examples were informed by the symptom criteria a Self-reported expertise area in the PTSD section. Disruptive behavior examples were informed Dissemination and implementation science 14 43.8 by the symptom criteria in the conduct disorder and oppositional Prevention science 14 43.8 defiant disorder sections. The survey instructed participants to recommend specific Mechanisms-focused research 18 56.3 intervention strategies that could be used for children and adoles- Research on childhood adversity 18 56.3 cents who have experienced adversity and were exhibiting either a Treatment-focused research 17 53.1 specific developmental mechanism or symptom pattern. Participants were told that all youth would have received case a Note: Respondents could select multiples, so percentages may exceed 100%. management prior to intervention and that the goal was to develop a preventive intervention rather than treatment. Participants were told to select a single answer they felt best anonymity of participation. The study PI/CIs and previously addressed the specific presentation and were reminded that hired consultants did not receive financial compensation, while there were no correct answers. For all recommendations, partici- all remaining invited experts were offered a payment of $100. pants were asked to select their primary recommendations and In total, 46 experts, including the four PI/CIs, were invited to secondary recommendations (optional) for the most relevant participate in the expert recommendations survey, of which 32 intervention principles from FIRST. Participants were provided (69.6%) provided consent and completed the survey. with a written description and a visual of FIRST and its five Demographic data for the participating sample (n = 32) are pro- core treatment principles – Feeling calm, Increasing motivation, vided in Table 2. Repairing thoughts, Solving problems, and Trying the opposite. Participants were asked to make separate recommendations for Procedure and measures younger (i.e., children aged 8–12 years) and older (i.e., adolescents The participants responded to an online survey that assessed aged 13–15 years) presentations. Participants were also asked if demographic information and asked them to recommend inter- they would make additional recommendations beyond the vention strategies that would be most effective in targeting our FIRST principles and if their recommendations would change four developmental mechanisms of interest. The survey also for children and adolescents of certain demographics. Finally, asked participants to recommend intervention strategies for four participants were asked to separately rank the three parenting types of psychopathology symptoms that commonly occur in chil- practices – Increasing positive reinforcement, Decreasing care- dren exposed to adversity, including symptoms of disruptive giver criticism and hostility, and Coaching of child emotional behavior problems, depression, anxiety, and PTSD. As such, competence – in order of importance for a brief preventive

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. Development and Psychopathology 753

intervention for adversity-exposed children and adolescents. a list of which age groups, settings, and clinical targets were Although training in specific discipline strategies is also a core ele- reviewed, as well as key tables and figures. For each meta-analysis, ment of PCIT, we did not include this in our list of parenting participants were asked to rate the clinical practice elements/inter- practices as these techniques are subsumed within the vention components (e.g., psychoeducation, contingency manage- Increasing motivation principle of FIRST. Throughout the survey, ment, relaxation) that had the strongest support indicated for participants were allowed to provide qualitative feedback or expla- treatment and prevention programs. Participants were also nations for their responses. asked to indicate which of the five FIRST principles (Feeling calm, Increasing motivation, Repairing thoughts, Solving problems, and Trying the opposite) and three parenting practices Survey 2 (Increasing positive reinforcement, Decreasing caregiver criticism Participants and hostility, and Coaching of child emotional competence) were Members of our research team (i.e., PI/CIs, staff, graduate stu- supported by each meta-analysis. After responding to questions dents) were invited to participate via an email sent from the for each individual meta-analysis, participants were provided study lead (LRN). The recruitment email emphasized that partic- with a summary of their responses and a spreadsheet that aggre- ipation was voluntary and no compensation was provided for gated all the effect sizes reported across the meta-analyses. completing the survey. Of the eight individuals who were invited Participants were asked to rank the top practice elements/inter- to participate, six provided consent and completed the full survey. vention components that they felt had the “strongest evidence” This survey intentionally took place after Survey 1 had closed to (a) across all targets and treatment/prevention programs, (b) ensure that the study investigators provided their prior responses across all prevention programs only, and (c) and that were rele- before systematically reviewing the relevant meta-analyses that vant for a “brief (two to six sessions) prevention-level service were incorporated in Survey 2. offered to adversity-exposed youth (age 8–15) and their families who were screened in via primary care.” Finally, participants Literature search for relevant systematic reviews were instructed to rate their perceptions of support for the Members of the study team identified meta-analyses through an FIRST principles and parenting practices on a 5-point informal literature search and article nomination process (i.e., Likert-type scale, ranging from “extremely effective” to “not effec- no specified databases or mandatory key terms). The search tive at all.” For all questions there was an open-text field in which aimed to identify recently published systematic reviews and meta- participants could write notes elaborating on their decision mak- analyses on “common elements,”“practice elements,” or “disman- ing. Those with a bachelor’s degree were both currently enrolled tling studies” of clinical trials, with a particular focus on identify- in a clinical psychology doctoral program. ing articles that examined prevention programs, a range of internalizing/externalizing problems, a range of ages (i.e., early Survey 3 childhood through adolescence), and (when possible) samples of adversity-exposed youth, or interventions conducted through Participants primary care gatekeeping. Included studies met the following cri- The same eight individuals invited for Study 2 were invited in the teria: (a) they examined the effects of prevention programs (at the same way for Study 3, all of whom provided consent and com- universal, selective, or indicated level) and (b) they reported effect pleted the full survey. These individuals were selected to partici- sizes by common elements or intervention strategies. pate because of their knowledge of the ultimate goal of the Meta-analyses that examined both prevention and intervention project – to develop a brief intervention for adversity-exposed programs were included if they reported effect sizes for prevention youth. All participants had completed one or both of the prior programs separately from treatment programs. surveys. This sample was 75% female, 88% white with one respondent identifying as biracial, and the highest educational degree Round 1: Meta-analysis selection levels were doctoral (50%), master’s (25%), and bachelor’s The informal literature search yielded 15 potential meta-analyses. (25%) degrees. Those with a bachelor’s degree were both currently The study team members reviewed the identified articles and enrolled in a clinical psychology doctoral program. voted to select the final group of meta-analyses. They were instructed to vote for the meta-analyses that they felt were Summarizing the findings from Survey 1 and Survey 2 “most useful in informing” the design of a preventive interven- The study lead (LRN) consulted with KM and JW, then compiled tion. They were told to consider relevance (i.e., prevention pro- a summary of the results and qualitative feedback from Surveys 1 grams, special samples, age ranges), breadth of targets covered and 2. Based on findings from the prior two rounds, some recom- within and across meta-analyses, and the quality of analysis mendation results were collapsed across age groups rather than (e.g., meta-regression was considered highest quality). presented separately. The study lead distributed the results sum- Meta-analyses were included if more than half of the reviewers mary and instructed participants to focus their review on the (n = 6) voted for their inclusion, resulting in a final sample of quantitative results summaries from both surveys as well as overall seven meta-analyses (Caldwell et al., 2019; Filene, Kaminski, qualitative feedback from Surveys 1 and 2. Participants were Valle, & Cachat, 2013; Leijten et al., 2019; Sanchez et al., 2018; encouraged to read additional qualitative feedback on each target Schleider & Weisz, 2017; Singla et al., 2020; van der Put, from Survey 1 as desired or needed. Assink, Gubbels, & Boekhout van Solinge, 2018). Data collection Round 2: Survey data collection Survey 3 instructed participants to rate the FIRST principles and Participants were provided with links and were instructed to read parenting practices in a similar manner to Survey 2. Based on the the final set of meta-analyses. In addition, the survey instructions findings from Survey 1 that suggested recommendations did not summarized key information from each meta-analysis, including differ significantly by age, participants provided recommendations

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. 754 L. R. Rith‐Najarian et al.

irrespective of child age. Based on qualitative data from Survey 2, frequencies of skills recommendations by target are presented participants were also instructed to provide their recommendations graphically in Figure 1. in three different ways: first, by using responses from the aggregated evidence and recommendations from the results of the two prior Stability of FIRST skills recommendations surveys; second, by also considering their own knowledge and expe- The overall Krippendorff’s α for primary skills recommendations riences; and third, by ranking the skills they would prioritize was .27, suggesting little stability across the intervention skills rec- including in the planned intervention. Participants were also ommended across all developmental mechanisms and symptom invited to provide open-ended responses explaining any changes types. There was a similar α value for recommendations on in their recommendations for this final survey. developmental mechanisms overall (α = .27) and symptom types overall (α = .25). Examining α values for skill recommendations Data analyses for individual targets (e.g., heightened emotion reactivity, anxiety), no stability was observed regardless of collapsing or separating We used a similar statistical approach for analyzing the data from all α − three surveys. To evaluate recommendations, rankings, and ratings recommendation by age groups ( values from .03 to .00). The α values varied slightly between participant groups, with the high- across respondents, it is important to differentiate between stability, α agreement, and consensus (Gisev, Bell, & Chen, 2013; Trevelyan & est stability of responses by study investigators (n =4, = .36), followed by external experts (n = 25, α = .26), and then study Robinson, 2015). For Delphi studies, the stability of responses can α α be assessed by measurements of inter-rater reliability, such as intra- consultants (n =3, = .23). The values also varied slightly by self-reported expertise, with the highest stability of responses by class correlation coefficients (ICCs) (Trevelyan & Robinson, 2015), α whereas consensus of responses can be assessed with measurements dissemination and implementation researchers (n = 14, = .29), followed by treatment researchers (n = 17, α = .28), researchers of variation, such as interquartile range (Trevelyan & Robinson, α 2015), or measurements such as Kendall’s coefficient of concor- focused on developmental mechanisms (n = 18, = .22), childhood adversity researchers (n = 18, α = .22), and prevention dance (Gisev et al., 2013). As such, we first assessed the stability α of responses by calculating Krippendorff’s alphas (α) for nominal researchers (n = 14, = .18). data and ICCs for ordinal data. We selected Krippendorff’s α,as opposed to Fleiss’ K, because they yield similar results but α is Consensus of FIRST skills recommendations more robust to missing data (Zapf, Castell, Morawietz, & Karch, The overall variation ratio (VR) from primary skills recommenda- 2016). We selected ICC(3,k) to test two-way mixed effects of average tions was .39, indicating that 39% of recommendations fell outside measures for multiple raters (Koo & Li, 2016) for assessing consis- each respective modal recommendation. For reference, the “at tency (<.4 = poor,.4=fair,.6=good, .75 = excellent; Cicchetti, chance” VR for items in this survey would be .80, with lower val- 1994). Next, for assessing consensus of responses, we calculated var- ues reflecting higher consensus. There was slightly better consen- iation ratios for nominal data and Kendall’s coefficient of concor- sus for primary skills recommendations for developmental dance for ordinal data. The variation ratio represents spread of mechanisms (overall VR = .36) than for symptom types (overall responses (VR =1− proportion of cases in the mode; Weisberg, VR = .42). 1992)), with scores from 0 to 1 indicating less to more dispersion. There was quite a range in VR values when examining primary Kendall’s coefficient of concordance (W,orWt when corrected skill recommendations by individual targets, with the highest con- for ties) can assess the relative strength of agreement (.3 = low,.5 sensus for disruptive behavior (collapsed age VR = .19) and the = fair,.7=high,.9=very high) by 3+ raters for rank order data lowest consensus for PTSD symptoms (collapsed age VR = .58) (Schmidt, 1997). Given the number of participants and questions (see Figure 1 for recommendations for each FIRST skill). asked in Survey 1, multiple values of VR and α were calculated for Compared with the primary skills recommendations, secondary assessing the recommendations overall, by subtype (e.g., primary skill recommendations had worse consensus overall (VR = .72) vs. secondary), and by participant type (e.g., self-reported exper- and for individual targets (VR range from .67 to .75). As such, tise). In addition, for Survey 1, we only ran chi-squared analyses we discontinued the evaluation of secondary skills recommenda- to determine if the final presentation of recommendations should tions. The VR values varied slightly between participant groups, be collapsed or separated by age group. Finally, for all data types, with the highest consensus of responses by study investigators we calculated frequencies to assess which responses were most com- (n =4, VR = .27), followed by study consultants (n =3, VR monly endorsed. All VRs were calculated in Microsoft Excel and all = .33), and then external experts (n = 25, VR = .26). The VR values remaining analyses were conducted with RStudio version 3.6.0 (R also varied slightly by self-reported expertise, with the highest Core Team, 2019) using the irr package (Gamer, Lemon, Fellows, consensus of responses by dissemination and implementation & Singh, 2019). researchers (n = 14, VR = .36), followed by treatment researchers (n = 17, VR = .39), childhood adversity researchers (n = 18, VR Results = .40), developmental mechanisms-focused researchers (n = 18, VR = .41), and prevention researchers (n = 14, VR = .43). Survey 1 Frequency of FIRST skills recommendations Ranking of parenting practices Examining recommendations for the eight targets individually, Examining the ranking of recommended parenting practices, chi-squared analyses (or Fisher’s exact tests in instances of low stability was excellent both for children (ICC(C,29) = .83) and cell count) compared the frequency of skills recommendations adolescents (ICC(C,29) = .90). However, consensus was very low for children (age 8–12 years) versus adolescents (age 13–15 both for children (Wt = .17) and adolescents (Wt = .27). A years). These analyses showed that none of the recommendations chi-squared analysis revealed that the first ranked parenting prac- significantly differed by age ( p values from .28 to .99), so we col- tice did not differ by age group ( p = .4); the rankings of parenting lapsed the recommendations across both age groups. The practices for each age group are presented in Figure 2.

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. Development and Psychopathology 755

Figure 1. Frequencies of recommended FIRST principles by targets (Survey 1).

Survey 2 Survey 3 Intervention skills broadly FIRST skills ratings and rankings From a compiled list of 33 intervention components/practice ele- When respondents (n = 8) were instructed to rate each FIRST skill ments that were reviewed in the seven screened meta-analyses, according to the evidence they reviewed from the results of Survey respondents were asked which they would select as the top five 1 (expert recommendations) and Survey 2 (meta-analysis review), with “strongest evidence” for different purposes. Stability across stability was excellent (ICC(C,8) = .94) and consensus was high respondents (n = 6) was low for top five skills selected overall (α (Wt = .75) (see Figure 5 for average ratings). When respondents values from .29 to .37). However, consensus was very good for were instructed that they could also consider knowledge from the selection of the top five skills overall (VR = .11), considering their own experiences, stability was still excellent (ICC(C,8) prevention programs only (VR = .11), and considering programs = .85) but consensus was moderate (Wt = .60). Relative to the rat- that were most similar to our planned early adversity intervention ings presented in Figure 5, these second ratings based also on (VR = .09). For reference, the “at chance” VR for these cases respondents’ individual knowledge produced the following would be .83. Moreover, about 20 options were never ranked by changes (as measure by standardized mean difference; SMD): any respondent, showing some collective discrimination of ele- Feeling calm had a small decrease (SMD = −0.43); Increasing ments that were clearly not in the top five. Figure 3 shows motivation had a small increase (SMD = 0.23); Repairing those skills ranked as the top five by respondents for our preven- thoughts, Solving problems, and Trying the opposite had a mod- tion program purposes. erate increase (SMD = 0.57 to 0.66). Finally, when respondents rank-ordered the skills to prioritize for our planned intervention, stability was still excellent (ICC(C,8) = .97) and consensus was FIRST skills and parenting practices high (Wt = .82), with the rank order of FIRST skills exactly follow- Examining respondents’ strength of evidence ratings for the five ing the strength of evidence ratings order as per Figure 5. FIRST skills, stability was excellent (ICC(C,6) = .88) and consensus was high (Wt = .85). The five intervention strategies ranked as Parenting practice ratings and rankings having the strongest evidence were discipline/behavior manage- When respondents were instructed to rate each parenting practice ment, parenting skills for positive reinforcement, improving par- based on evidence from Surveys 1 and 2, stability was excellent ent–child relationship/interactions, problem-solving for parents, (ICC(C,8) = .98) and consensus was very high (Wt = .97) (see and relaxation skills (see Figure 4). Examining respondents’ Figure 5 for average ratings). When respondents were instructed strength of evidence ratings for the three parenting practices of to also consider their own past knowledge, stability was again interest, stability was excellent (ICC(C,6) = .96) and consensus excellent (ICC(C,8) = .97), while consensus decreased but was was high (Wt = .81). still high (Wt = .88), with the following changes: Increasing

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. 756 L. R. Rith‐Najarian et al.

Figure 2. Ranking of parenting practices for younger (ages 8–12) and older (ages 13–15) presentations (Survey 1).

positive reinforcement stayed at the maximum possible rating, recommending Trying the opposite. The α value was −.12 and Decreasing caregiver criticism and hostility had a small increase the VR was .38 for the remaining three targets of emotion regu- (SMD = 0.40), and Coaching of child emotional competence had lation difficulties, anxiety symptoms, and posttraumatic stress a large increase (SMD = 0.86). Finally, when respondents symptoms (see Figure 6 for details). rank-ordered the parenting practices, stability was still excellent (ICC(C,8) = .98) and consensus was still high (Wt = .89), with the rank order again exactly following the strength of evidence rat- Qualitative feedback across surveys ings order as per Figure 5. Although a comprehensive qualitative analysis is beyond the pur- view of this paper, selected responses may be helpful to contextu- Final stability and consensus of FIRST skills recommendations alize quantitative findings. We have provided some exemplar α The overall value for skills recommendations was .60, suggesting quotes from each survey in Table 3 to demonstrate how respon- moderate stability. There was also moderate stability for recom- dents explained their decision making process or elaborated on α mendations by developmental mechanisms overall ( = .58) and rationales, qualifiers, or additions to their recommendations. symptom types overall (α = .52). As for consensus, the overall VR was .22, an improvement over the consensus of Survey 1. The “at chance” VR for these items is .80. The consensus was Discussion identical for skills recommendations for developmental mechanisms (overall VR = .22) and symptom types (overall VR = .22). In the service of developing an intervention to prevent the mental Regarding skill recommendations for individual targets, we health consequences of childhood adversity, we conducted a mod- reached absolute stability (ICC(C,8) = 1.00) and consensus ified Delphi study to assess intervention strategy recommenda- (VR = .00) for four targets – Feeling calm was recommended by tions and ratings based on expert feedback and review of the all participants for targeting heightened emotional reactivity; existing intervention evidence base. Over three survey rounds, Increasing motivation for targeting disruptive behavior; we found increasing consensus on intervention strategies regard- Repairing thoughts for targeting social information processing ing recommendations for individual targets, strength of evidence biases; and Trying the opposite for targeting depressive symp- ratings, and ranking order. Certain intervention strategies for toms. For blunted reward processing, consensus was split (VR youth skills and parenting practices emerged as stronger candi- = .5) and thus stability was very low (α = -.12), with 50% (n =4) dates, some with consensus for targeting specific developmental of respondents recommending Increasing motivation and 50% mechanisms and symptoms. In addition, our findings produced

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. Development and Psychopathology 757

Figure 3. Skills ranked as top five based on review of meta-analyses (Survey 2).

useful information about the process of evidence-informed inter- improved for recommendations of both youth skills and parenting vention decision making. practices. The final most frequently recommended skills were First, we were interested in investigating which candidate inter- again Feeling calm, Increasing motivation, and Trying the oppo- vention strategies – specifically, youth skills drawn from FIRST site; Solving problems was never recommended. Youth skills rec- (Weisz & Bearman, 2020) and parenting practices drawn from ommendations reached 100% consensus in Survey 3 for four of PCIT-ED (Luby et al., 2018) – were recommended for the devel- the eight targets – Feeling calm for heightened emotional reactiv- opmental mechanisms and symptoms we intend to target. To do ity, Increasing motivation for disruptive behavior symptoms, so, we first asked a group of experts (Survey 1). Consensus in Repairing thoughts for social information processing biases, and these recommendations can be interpreted from the lower VR val- Trying the opposite for depressive symptoms – and was split ues and higher W values within the survey rounds, and by relative 50/50 between Increasing motivation and Trying the opposite decreases or increases between rounds (Gisev et al., 2013; Holey, for blunted reward processing. For the final parenting practice Feeley, Dixon, & Whittaker, 2007). For the youth skills recom- recommendations in Survey 3, Increasing positive reinforcement mendations in Survey 1, we found very low stability and low con- emerged as the unanimous first parenting practice to prioritize sensus overall, though one or two modal recommendations and Coaching of child emotional competence remained the emerged for all eight targets. Overall, the most recommended last. youth skills in Survey 1 were Feeling calm, Increasing motivation, We were also interested in investigating which intervention and Trying the opposite; Solving problems was rarely recom- strategies were most strongly supported by existing empirical evi- mended. Regarding parenting practices recommended for our dence. To do so, we identified and reviewed relevant meta-analyses intervention overall, the experts in Survey 1 had excellent stability and rated which intervention elements had the strongest evidence of rankings, but very low consensus. Low consensus reflects that related to our intended prevention program. In Survey 2, although respondents are divergent from each other, whereas high stability there were more than 30 practice elements and intervention strat- reflects that they are diverging in a consistent way. The parenting egies presented across the reviewed meta-analyses, the respondents practice Coaching of child emotional competence was clearly showed good consensus for identifying the intervention elements ranked lowest of the three. Of note, consensus of the recommen- that had the strongest effects. When asked which intervention strat- dations in Survey 1 did not appear to depend on youth age, egies had the most evidence for our prevention program, at least respondent’s affiliation with our study team, or self-reported two thirds of our study team selected the following within their expertise type. In Survey 3, we invited only study team members top five: discipline skills/behavior management; parenting skills to complete these recommendations a second time after they had for positive reinforcement; improving parent–child relationships/ reviewed summaries of the recommendations from the broader interactions; problem-solving skills for parents; and relaxation group of experts. This time, stability and consensus substantially strategies. Respondents were then asked to translate their

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. 758 L. R. Rith‐Najarian et al.

Figure 4. Strength of evidence ratings based on review of meta-analyses (Survey 2).

selections into strength of evidence for our candidate FIRST youth opposite (from FIRST), and Increasing positive reinforcement skills and parenting practices, and the resultant ratings had excel- (from PCIT) receiving the most frequent recommendations, lent stability and high consensus. According to Survey 2, strength higher rankings, and stronger strength of evidence ratings. The of evidence ratings were “extremely effective” for Increasing moti- fact that these same intervention strategies were the most highly vation (a skill that targets both youth and parent behaviors; see recommended/rated across all three surveys suggests that consen- Table 1) and Increasing positive reinforcement (parenting prac- sus is generalizable beyond our study team. These intervention tice), “moderately effective” for Feeling calm (youth skill), strategies are also conceptually similar to the practice elements Solving problems (youth skill), and Decreasing caregiver criticism with strong effects in relevant meta-analyses. As examples, relax- and hostility (parenting practice), and “slightly effective” for ation (which was included in two of the meta-analyses) is the cen- Trying the opposite (youth skill), Repairing thoughts (youth tral intervention strategy of the Feeling calm principle in FIRST, skill), and Coaching of child emotional competence (parenting and positive attending, improving parent–child relationships, pos- practice). Strength of evidence ratings in Survey 3 again had itive reinforcement strategies, and other behavior management excellent stability, while consensus for youth skills was still high skills (at least one of which was included in all seven meta- but somewhat decreased, and consensus for parenting practices analyses) are core skills in the Increasing motivation principle was very high, a relative increase. According to Survey 3, the of FIRST and the Increasing positive reinforcement element of final strength of evidence ratings stayed the same for Increasing PCIT. Although it is important to highlight that intervention motivation, Increasing positive reinforcement, Repairing thoughts, strategies with high consensus do not guarantee that those strate- and Coaching of child emotional competence. The changes in gies will necessarily be most effective (Trevelyan & Robinson, strength of evidence ratings from Survey 2 to Survey 3 were an 2015), the current findings certainly can help us prioritize our increase to “very effective” for Feeling calm and Decreasing care- best candidates, given the currently existing evidence. In particu- giver criticism and hostility, an increase to “moderately effective” lar, given that our planned prevention program will be brief (up to for Trying the opposite, and a decrease to “slightly effective” for six sessions), it may be useful to focus on Feeling calm, Increasing Solving problems. motivation, and Trying the opposite as the most relevant intervention elements from FIRST, while deprioritizing Repairing thoughts and Solving problems. Similarly, Increasing positive Implications for preventing mental health consequences of reinforcement is the parenting practice that could be prioritized childhood adversity above both Decreasing caregiver criticism and hostility and Our primary aim in investigating these research questions was to Coaching of child emotional competence. evaluate intervention strategy consensus so that findings could One particularly encouraging aspect of our findings was reach- inform the development of our prevention program for youth ing consensus on intervention strategies for some of our target with varying presentations. As stability and consensus improved developmental mechanisms. Many psychosocial interventions across surveys, there was some convergence with Feeling calm are designed for targeting symptoms, which necessarily involves (from FIRST), Increasing motivation (from FIRST), Trying the more remedial intervention strategies rather than preventive

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. Development and Psychopathology 759

Figure 5. Strength of evidence ratings based on respondents’ review of prior rounds (Survey 3).

strategies that reduce risks and promote strengths (American possible for mental health professionals to apply their interven- Psychological Association, 2014). Moreover, many clinical psy- tion decision making to planning a mechanisms-based preventive chologists engage in intervention decision making in a symptom- intervention. driven way (Bakker, 2019) – part of the motivation for developing the APA prevention guidelines was that, despite the increased Insights into evidence-informed clinical decision making focus on prevention within clinical psychology, there is little understanding of the best preventive practices (American We had an additional exploratory aim, which was to gain insight Psychological Association, 2014). Considering how little guidance into how evidence-informed decision making in intervention and experience clinical psychologists have regarding prevention development unfolds. Our findings illuminate understanding of strategies, it was notable that we achieved 100% or 50/50 consen- intervention decision making processes in four main ways. sus for skill recommendations for three of our four developmental First, the use of meta-analysis to inform intervention decisions mechanisms (i.e., heightened emotional reactivity, social is frequently recommended (Powell et al., 2015; Weisz et al., 2005) information processing biases, and blunted reward processing). yet rarely studied as a practice. Although meta-analyses and sys- Moreover, even before we reached consensus, these mechanisms tematic reviews can, in theory, provide meaningful summaries already had modal skill recommendations in Survey 1, which of intervention evidence, there is little guidance on how to actu- indicates that the experts were not making recommendations ran- ally use them in practice when designing interventions. Our prac- domly. It is not the case that each mechanism was simply treated tice element ranking findings from Survey 2 demonstrate that as a proxy for some perceived parallel symptom (e.g., reward even experts reviewing the same meta-analyses come to different processing vs. depression); otherwise, the same four recommen- conclusions about which intervention strategies emerged with the dations for symptom targets would have emerged for develop- strongest evidence. Relatedly, respondents noted the difficulty of mental mechanism targets. Qualitative data also supported cognitively condensing and interpreting results across meta- this idea, as exemplified by one respondent describing how analyses, especially for those with less familiarity with meta-analysis their intervention recommendation for depressive symptoms statistical methods. The difficulty in translating meta-analysis find- was informed by both the meta-analysis review (which covered ings into intervention selections might explain some variation in the depression targets) and recommendations “for blunted reward recommendations. For example, one respondent rated Solving processing by experts – a mechanism closely related to depres- problems as “moderately effective” based on some meta-analytic sion.” Conceptually, the skills with strongest strength of evidence support of problem solving for caregivers, while another respon- ratings largely involved risk reduction strategies (e.g., addressing dent acknowledged that same evidence but concluded low support positive parent–child relationship through Increasing positive for Solving problems because of other meta-analytic null results reinforcement and Increasing motivation) and strengths of problem solving for youth. Thus, although it is common practice promotion (e.g., Feeling calm as a coping strategy), as opposed to point to meta-analyses to decide best clinical practices, inter- to remedial skills that require some symptoms to be already pre- pretation of meta-analytic findings is not actually a clear-cut sent (e.g., exposure, cognitive restructuring). Thus, it appears task, even for a study team working closely together.

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. 760 L. R. Rith‐Najarian et al.

Figure 6. Survey 3: Frequencies of recommended FIRST principles by target.

Second, our findings provide evidence that a Delphi study can about intervention selection while decreasing bias in group help compensate for biases in decisions made by individuals alone decisions. or by groups without structured consensus building. The use of a Third, our findings are a reminder that aiming for consensus Delphi study is intended to enhance group decision making by does not necessitate achieving consensus. Any Delphi study can mitigating issues of bias and group dynamics (Hsu & Sandford, be continuously iterative, so there must be a determination by 2007; Surowiecki, 2004). Moreover, much research has found researchers about when more rounds would no longer be useful that clinical decisions by individuals are affected by cognitive (Hsu & Sandford, 2007). Compared with more traditional errors such as the availability heuristic, overconfidence, and hind- Delphi methods, our modified Delphi study used a larger initial sight bias (Lilienfeld & Lynn, 2014; Witteman, 2020). To over- sample and then had only our smaller study team continue. come these limitations, it has been recommended that we Through this approach, our Survey 3 had the highest consensus develop debiasing methods, or at least ways to systematically com- levels and we reached maximum consensus for recommendations pensate for them (Lilienfeld, 2019; Lilienfeld & Lynn, 2014), on half of our targets. However, our consensus on strength of evi- which we believe could include the Delphi approach. The fact dence ratings slightly decreased from Survey 2 to Survey 3, along that the study investigators already had relatively high consensus with the growing amount of information for respondents to on specific intervention recommendations in Survey 1, even review and consider. Given that we still achieved high consensus before reviewing the meta-analyses in Survey 2, might indicate in Survey 3, it seemed that we may have reached a ceiling. This that some groupthink had already been occurring. The improved finding highlights how a consensus process may reach saturation, consensus reached in Survey 3 demonstrates how reviewing objec- at which point perpetually synthesizing group feedback may pro- tive summaries of evidence and others’ ideas from prior survey vide diminishing clarity. Finally, consensus is not always the goal rounds ultimately influenced the final decisions. One respondent when there may not a single answer (Hall, 2009), with a good confirmed this experience, saying “the aggregated results very example being our 50/50 split on intervention recommendations much informed my thinking,” even allowing their review of for blunted reward processing. prior results to override personal opinions when caught between Fourth, despite attempts at debiasing our clinical decisions, two recommendations. Finally, when respondents provided their objective decision making is quite challenging. Qualitative feed- strength of evidence ratings in Survey 3, again based also on back across surveys emphasizes this fact. Even though respon- their own knowledge and experiences (i.e., beyond the informa- dents were capable of providing intervention recommendations tion aggregated from experts and the meta-analyses), there was for the prevention mechanism targets, they still had some diffi- a decrease in consensus for both youth skills and parenting prac- culty avoiding symptom-driven thinking. Some comments tices. This decrease was accompanied by a trend of respondents revealed that respondents were still considering clinical experi- giving higher evidence ratings across intervention strategies, ences or citing evidence related to treatment programs. For exam- which makes sense given that our team selected these candidate ple, one respondent in Survey 2 admitted “biased interpretation skills in the first place. Together, our findings increase our confi- from my own clinical understanding of working with [children]” dence that a Delphi study can increase consensus in decisions after reviewing meta-analyses on prevention programs. Even by

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. Development and Psychopathology 761

Table 3. Selected qualitative data across surveys

Survey 1 After recommending Trying the opposite for PTSD symptoms: “If there were not enough time, I would prioritize feeling calm interventions. Also, if we are talking about nonclinical symptoms of PTSD then I would probably focus on feeling calm strategies and cognitive strategies to think about the traumatic event in a way that is more helpful and less distressing.” After recommending Trying the opposite for anxiety symptoms: “[I] want to clarify that I see relaxation skills and exposure being part of the same set here (used jointly).” “My judgment of how to rank order the parenting skills would actually depend on the kind of behavior the child was showing, not on what adversity the child had experienced, because similar patterns of diversity can be followed by very diverse patterns of child behavior – each requiring an appropriate intervention.” “I think there are important considerations about caregiver preparation to enact these skills and dosing considerations are important given different caregiver baseline characteristics which may covary with nature of family adversity as well as cultural and educational background.” Survey 2 “This was a lot of information to absorb, and my stats background is not that strong. I’m also probably biased from my own clinical understanding of working with early/middle childhood… I’m not very confident in my responses.” “Problem-solving skills for parents was a second rated skill, and I rated Solving problems as moderately effective.” “For ‘Solving problems,’ child problem solving was tested in 3 of 7 meta-analyses, and none showed it being significant. The only time I saw problem solving with support it was as a caregiver skill, so I’m not counting that. Moreover, caregiver problem solving was also only significant in 1 of 2 meta-analyses that tested it.” “It’s a bit difficult to do this in a broad, generic way, since the skills most needed and likely to be helpful may differ widely depending on what concerns and problems the parents want to have addressed – e.g., very different skills if dealing with child conduct problems than if dealing with child anxiety or depression.” Survey 3 “The aggregated results very much informed my thinking. I had my own opinions going in which informed my recommendations somewhat, but I probably weighed the aggregated results more highly than my independent opinions… in a few places where there were two strategies that were close in my mind, there were situations where reviewing the results of the two surveys pushed my top choice from one strategy to another.” “The aggregated results did lead me to lower ratings on problem solving than my experience in our intervention trials would suggest, so I did raise that particular rating when invited to use my own knowledge and experience.” After recommending Feeling calm for emotion regulation difficulties: “[I would say] repairing thoughts for older youth maybe, but my main hesitation after reading the first-round feedback is feasibility of teaching higher level skills without all the time necessary for psychoeducation, rapport building, etc.” After recommending Trying the opposite for depressive symptoms: “Increasing motivation would be my second choice, particularly given the weight of evidence for this strategy in the meta-analysis review, and that it was most highly rated for blunted reward processing by experts – a mechanism closely related to depression.” “I rated Trying the opposite as the 3rd most effective skill for a range of reasons. First, in the [meta-analysis of brief interventions], which is similar to our approach, behavioral interventions targeting the child were most effective. Second, Trying the opposite was [recommended first or second] for depression, anxiety, and PTSD by experts. I am also including my personal experiences here of the remarkably large (and fast) effects that behavioral activation and exposure can have for children with mood and anxiety problems.” When asked about PTSD symptoms: “Ideally, we would want to provide something like TF-CBT, to help kids really address their PTS and their avoidance. However, in six sessions – and for many families, fewer – and with clinicians who don’t have TF-CBT training, we may be safer focusing on calming strategies.”

Survey 3, respondents were still referring to treatment recommen- Kwaadsteniet & Hagmayer, 2018). In summary, consensus does dations (e.g., TF-CBT) and effective interventions for symptoms not solve all decision making problems and updating new knowl- (e.g., “behavioral activation and exposure [for] mood and anxiety edge into our pre-existing – and sometimes entrenched – frame- problems”) without making the distinction between treatment works remains a challenge for evidence-informed intervention and prevention programs. Finally, the most notable example decision making. These findings are somewhat reminiscent of sen- comes from some conflicting quantitative findings in Survey timentsfromZiglerthatitisdifficult“to bring together politics, 3. On the one hand, we had 100% consensus for the recommen- knowledge [..], and common sense” when developing standards dation of Repairing thoughts for the social information processing for prevention content, and even “the best standards in the world biases, despite also having high consensus in rating this skill with are meaningless if they are not enforced” (Zigler, 1996,p.38; the lowest strength of evidence overall. On the other hand, none Zigler & Styfco, 2001,p.10) of the respondents recommended Solving problems for a single target, despite having rated Solving problems as “moderately effec- Limitations tive” on average. In other words, within the same survey, we sometimes recommended intervention strategies for our prevention Our study was subject to some limitations. First, interpretation of targets that contradicted our own rating of their strength of evidence the findings was constrained by which intervention strategies we for our overall prevention program. Though such recommendations considered. As an example, we did not query respondents about may still hold some merit, these findings highlight inconsistence in computerized attention bias modification strategies, which are how we apply clinical knowledge. Other research has found that that less commonly used in clinical settings, yet theoretically might clinician intervention recommendations can be more reflective of be the most effective for threat-related social information process- personally held theories rather than current scientific evidence (de ing biases (e.g., attention bias to threat; Bar-Haim, 2010). The

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. 762 L. R. Rith‐Najarian et al.

selected meta-analyses added to this constraint, as we likely behavioral exposures have shown stronger effectiveness than missed some relevant reviews through our informal search proce- relaxation skills in the treatment of youth anxiety (Whiteside dures; furthermore, various intervention strategies are not equally et al., 2020). However, exposures are essentially meaningless in represented in the published literature. the absence of impairing behavioral avoidance, whereas relaxation Second, the five FIRST principles, which served as our candi- strategies, in contrast, can be used by anyone. Our findings thus date intervention strategies for youth, vary in the breadth of skills give credence to that idea that strong intervention candidates contained within each module. Thus, while a respondent’s recom- for prevention programs will likely differ from those of treatment mendation of Feeling calm clearly involves relaxation strategies, a programs, in particular by including strategies that are conceptu- recommendation of Increasing motivation could be focused on ally more focused on risk reduction and strength promotion. one, some, or all of the encapsulated parent management skills Finally, our findings provided initial support for some candidate (e.g., reinforcement strategies to promote positive behaviors, spe- intervention strategies that could hypothetically target these cific discipline strategies). developmental mechanisms. By developing our preventive pro- The third limitation was that the specific respondents varied gram accordingly, we will now be able to empirically test the across the three samples. However, we decided on this methodo- actual effects of the intervention strategies recommended through logical approach to balance (a) representation – compared with the current study. In the words of Zigler, once we “know what most Delphi studies, we invited a larger group with greater diver- quality components are necessary to build successful interven- sity of research expertise and target populations (Cairns, Yap, tions” we must then “put this wisdom to use” (Zigler, 1996, p. 42). Reavley, & Jorm, 2015), (b) feasibility – it seemed unrealistic to expect a group of experts to commit time to multiple surveys Supplementary Material. The supplementary material for this article can (Cairns et al., 2015; Hsu & Sandford, 2007, and (c) quality of be found at https://doi.org/10.1017/S0954579420002059. – responses our study team members had intrinsic motivation Acknowledgments. We would like to thank our study team members – to thoughtfully review materials and could more readily provide Shannon Dorsey, Arianna Riccio, Olivia Fitzpatrick, and Katherine recommendations with our specific intervention purposes in Venturo-Conerly – for their contributions in the preparation of this study. mind. We also thank Joan Luby for her ongoing consultation for this intervention Fourth, not only does consensus on intervention recommenda- development. Finally, we thank Beth Erickson and Deneen Vojta for their sup- tions from experts differ from effectiveness (as previously dis- port of the ongoing implementation studies associated with this larger inter- cussed), it also does not ensure acceptability. Client preferences vention research project. predict their satisfaction with, completion of, and clinical Financial Statement. The presented research was supported by the Harvard response to intervention (Lindhiem, Bennett, Trentacosta, & University Dean’s Competitive Fund for Promising Scholarship (KM). The McLear, 2014). Accordingly, it ultimately does not matter how content of this article is solely the responsibility of the authors and does not evidence-based a preventive intervention is if primary care clinics necessarily represent the official views of the funding sponsor. and families who have experienced adversity do not want to participate. We are thus currently collecting stakeholder feedback Conflicts of Interest. John Weisz is a coauthor of a treatment manual cited from pediatric clinic providers and staff as well as families who in this paper that will be used in the research described herein (Principle-guided psychotherapy for children and adolescents: The FIRST treat- have encountered adversity. Relatedly, to increase multicultural ment program for behavioral and emotional problems, published in 2020 by appropriateness as per APA prevention guidelines (American Guilford Press (Weisz & Bearman, 2020)). He receives royalties for its sales. Psychological Association, 2014), we will (a) more thoroughly review expert qualitative feedback regarding multicultural recommendations and (b) specifically ask stakeholder families in our References feasibility trial about cultural acceptability of materials and lan- Abela, J. R. Z., Vanderbilt, E., & Rochon, A. (2004). A test of the integration of guage adaptations. the response styles and social support theories of depression in third and seventh grade children. Journal of Social and Clinical Psychology, 23, 653–674. doi:10.1521/jscp.23.5.653.50752 Conclusion American Psychiatric Association. (2013). Diagnostic and statistical manual of Although many interventions have been developed to ameliorate mental disorders (5th ed.). Arlington, VA: Author. American Psychological Association. (2014). Guidelines for prevention in psy- the mental health consequences of childhood adversity, evidence – of effectiveness for preventive interventions is scarce – particularly chology. American Psychologist, 69, 285 296. doi:10.1037/a0034569 Asarnow, J. R., Rozenman, M., Wiblin, J., & Zeltzer, L. (2015). Integrated those that are brief and could be feasibly integrated into pediatric medical-behavioral care compared with usual primary care for child and primary care. As such, we have yet to discover which prevention adolescent behavioral health: A metaanalysis. JAMA Pediatrics, 169, 929– strategies are most effective for families with adversity-exposed 937. doi:10.1001/jamapediatrics.2015.1141 youth. Simply borrowing and repackaging the most common Aupont, O., Doerfler, L., Connor, D. F., Stille, C., Tisminetzky, M., & treatment intervention strategies seems unlikely to advance effec- McLaughlin, T. J. (2013). A collaborative care model to improve access to pedi- tiveness. Instead, our study team systematically considered evi- atric mental health services. Administration and Policy in Mental Health and dence from experts and meta-analyses to determine which Mental Health Services Research, 40, 264–273. doi:10.1007/s10488-012-0413-0 intervention strategies were the most relevant and most potent Bakker, G. M. (2019). A new conception and subsequent taxonomy of clinical – for our prevention purposes. Moreover, we invited recommenda- psychological problems. BMC Psychology, 31, 653 673. doi:10.1186/s40359- tions on targeting four mechanisms related to the neurodevelop- 019-0318-8 Bar-Haim, Y. (2010). Research review: Attention bias modification (ABM): A novel mental adaptations of childhood adversity exposure that may treatment for anxiety disorders. Journal of Child Psychology and Psychiatry and place youth at risk of later psychopathology. Given that not all Allied Disciplines, 51, 859–870. doi:10.1111/j.1469-7610.2010.02251.x adversity-exposed youth will be symptomatic at the time of Barch, D. M., Whalen, D., Gilbert, K., Kelly, D., Kappenman, E. S., Hajcak, G., screening, it is crucial that our prevention approach includes skills … Luby, J. L. (2019). Neural indicators of anhedonia: Predictors and that are not dependent on symptom presentations. For example, mechanisms of treatment change in a randomized clinical trial in early

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. Development and Psychopathology 763

childhood depression. Biological Psychiatry, 85, 863–871. doi:10.1016/ Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating j.biopsych.2018.11.021 normed and standardized assessment instruments in psychology. Bellis, M. A., Hughes, K., Ford, K., Ramos Rodriguez, G., Sethi, D., & Psychological Assessment, 6, 284–290. doi:10.1037//1040-3590.6.4.284 Passmore, J. (2019). Life course health consequences and associated annual Cicchetti, D., & Toth, S. L. (1995). A developmental psychopathology perspec- costs of adverse childhood experiences across Europe and North America: a tive on child abuse and neglect. Journal of the American Academy of Child systematic review and meta-analysis. The Lancet Public Health, 4, E517– & Adolescent Psychiatry, 34, 541–565. doi:10.1097/00004583-199505000-00008 E528. doi:10.1016/S2468-2667(19)30145-8 Cicchetti, D., & Toth, S. L. (2009). A developmental psychopathology perspec- Benjet, C., Borges, G., Medina-Mora, M. E., Zambrano, J., Cruz, C., & Méndez, tive on adolescent depression. In S. Nolen-Hoeksema, & L. M. Hilt (Eds.), E. (2009). Descriptive epidemiology of chronic childhood adversity in Handbook of Depression in Adolescents (pp. 3–32). Routledge/Taylor & Mexican adolescents. Journal of Adolescent Health, 45, 483–489. Francis Group. doi:10.1016/j.jadohealth.2009.03.002 Cohen, P., Brown, J., & Smailes, E. (2001). Child abuse and neglect and the Bentovim, A., Chorpita, B. F., Daleiden, E. L., Gray, J., Pizzey, S., & Vizard, E. development of mental disorders in the general population. Development (2020). The value of a modular, multi-focal, therapeutic approach to and Psychopathology, 13, 981–999. doi:10.1017/s0954579401004126 addressing child maltreatment: Hope for children and families intervention Cohen, J. A., Deblinger, E., Mannarino, A. P., & Steer, R. A. (2004). A multi- resources – a discussion article. Child Abuse & Neglect,1–13. Advance site, randomized controlled trial for children with sexual abuse-related online publication. doi:10.1016/j.chiabu.2020.104703 PTSD symptoms. Journal of the American Academy of Child & Adolescent Berkowitz, S. J., Stover, C. S., & Marans, S. R. (2011). The child and family Psychiatry, 42, 393–402. doi:10.1097/00004583-200404000-00005 traumatic stress intervention: Secondary prevention for youth at risk Cohen, J. A., Mannarino, A. P., & Iyengar, S. (2011). Community treatment of of developing PTSD. Journal of Child Psychology and Psychiatry, 52(6), posttraumatic stress disorder for children exposed to intimate partner vio- 676–685. doi:10.1111/j.1469-7610.2010.02321.x lence: A randomized controlled trial. Archives of Pediatrics and Adolescent Brestan, E. V., & Eyberg, S. M. (1998). Effective psychosocial treatments of Medicine, 165,16–21. doi:10.1001/archpediatrics.2010.247 conduct-disordered children and adolescents: 29 years, 82 studies, and Danese, A., & McEwen, B. S. (2012). Adverse childhood experiences, allostasis, 5,272 kids. Journal of Clinical Child Psychology, 27, 180–189. doi:10.1207/ allostatic load, and age-related disease. Physiology and Behavior, 106,29–39. s15374424jccp2702_5 doi:10.1016/j.physbeh.2011.08.019 Brown, D. W., Anda, R. F., Tiemeier, H., Felitti, V. J., Edwards, V. J., Croft, J. de Kwaadsteniet, L., & Hagmayer, Y. (2018). Clinicians’ personal theories of B., … Giles, W. H. (2009). Adverse childhood experiences and the risk of developmental disorders explain their judgments of effectiveness of inter- premature mortality. American Journal of Preventive Medicine, 37, 389– ventions. Clinical Psychological Science, 6, 228–242. doi:10.1177/ 396. doi:10.1016/j.amepre.2009.06.021 2167702617712270 Cairns, K. E., Yap, M. B. H., Reavley, N. J., & Jorm, A. F. (2015). Identifying Dorsey, S., McLaughlin, K. A., Kerns, S. E. U., Harrison, J. P., Lambert, H. K., prevention strategies for adolescents to reduce their risk of depression: A Briggs, E. C., … Amaya-Jackson, L. (2017). Evidence base update for psy- Delphi consensus study. Journal of Affective Disorders, 183, 229–238. chosocial treatments for children and adolescents exposed to traumatic doi:10.1016/j.jad.2015.05.019 events. Journal of Clinical Child & Adolescent Psychology, 46, 303–330. Caldwell, D. M., Davies, S. R., Hetrick, S. E., Palmer, J. C., Caro, P., Dubowitz, H. (2014). The Safe Environment for Every Kid model: promotion López-López, J. A., … Welton, N. J. (2019). School-based interventions to of children’s health, development, and safety, and prevention of child prevent anxiety and depression in children and young people: a systematic neglect. Pediatric Annals, 43, e271–e277. doi:10.3928/00904481-20141022-11 review and network meta-analysis. The Lancet Psychiatry, 6, 1011–1020. Dubowitz, H., Feigelman, S., Lane, W., & Kim, J. (2009). Pediatric primary care doi:10.1016/S2215-0366(19)30403-1 to help prevent child maltreatment: The Safe Environment for Every Kid Campbell, T. L. (2020). Screening for adverse childhood experiences (ACEs) in (SEEK) model. Pediatrics, 123, 858–864. doi:10.1542/peds.2008-1376 primary care: A cautionary note. JAMA, 323, 2379–2380. Dubowitz, H., Feigelman, S., Lane, W., Prescott, L., Blackman, K., Grube, L., … Campbell, S. M., Roland, M. O., & Buetow, S. A. (2000). Defining quality of Tracy, J. K. (2007). Screening for depression in an urban pediatric primary care. Social Science and Medicine, 51, 1611–1625. doi:10.1016/S0277-9536 care clinic. Pediatrics, 119, 435–443. doi:10.1542/peds.2006-2010 (00)00057-5 Dubowitz, H., Lane, W. G., Semiatin, J. N., & Magder, L. S. (2012). The seek Carpenter, A. L., Puliafico, A. C., Kurtz, S. M. S., Pincus, D. B., & Comer, J. S. model of pediatric primary care: Can child maltreatment be prevented in a (2014). Extending parent–child interaction therapy for early childhood low-risk population? Academic Pediatrics, 12, 259–268. doi:10.1016/ internalizing problems: New advances for an overlooked population. j.acap.2012.03.005 Clinical Child and Family Psychology Review, 17, 340–356. doi:10.1007/ Dubowitz, H., Prescott, L., Feigelman, S., Lane, W., & Kim, J. (2008). Screening s10567-014-0172-4 for intimate partner violence in a pediatric primary care clinic. Pediatrics, Carver, C. S., & White, T. L. (1994). Behavioral inhibition, behavioral activa- 121, e85–e91. doi:10.1542/peds.2007-0904 tion, and affective responses to impending reward and punishment: The Eismann, E. A., Theuerling, J., Maguire, S., Hente, E. A., & Shapiro, R. A. (2019). BIS/BAS scales. Journal of Personality and Social Psychology, 67, 319–333. Integration of the Safe Environment for Every Kid (SEEK) model across pri- doi:10.1037/0022-3514.67.2.319 mary care settings. Clinical Pediatrics, 58,166–176. doi:10.1177/ Chaffin,M.,Silovsky,J.F.,Funderburk,B.,Valle,L.A.,Brestan,E.V.,Balachova, 0009922818809481 T., … Bonner, B. L. (2004). Parent-child interaction therapy with physically abu- Feigelman, S., Dubowitz, H., Lane, W., Prescott, L., Meyer, W., Tracy, J. K., … sive parents: Efficacy for reducing future abuse reports. Journal of Consulting Kim, J. (2009). Screening for harsh punishment in a pediatric primary care and Clinical Psychology, 72, 500–510. doi:10.1037/0022-006X.72.3.500 clinic. Child Abuse & Neglect, 33, 269–277. doi:10.1016/j.chiabu.2008.09.011 Chen, E., Turiano, N. A., Mroczek, D. K., & Miller, G. E. (2016). Association of Felitti, V. J., Anda, R. F., Nordenberg, D., Williamson, D. F., Spitz, A. M., reports of childhood abuse and all-cause mortality rates in women. JAMA Edwards, V., … Marks, J. S. (1998). Relationship of childhood abuse and Psychiatry, 73, 920–927. doi:10.1001/jamapsychiatry.2016.1786 household dysfunction to many of the leading causes of death in adults: Chevarley, F. M. (2003). Statistical brief #12: Children’s access to necessary The Adverse Childhood Experiences (ACE) study. American Journal of health care, fall 2001. Retrieved from http://meps.ahrq.gov/mepsweb/data_ Preventive Medicine, 14, 245–258. doi:10.1016/S0749-3797(98)00017-8 files/publications/st12/stat12.shtml Filene, J. H., Kaminski, J. W., Valle, L. A., & Cachat, P. (2013). Components Cho, E., Bearman, S. K., Woo, R., Weisz, J. R., & Hawley, K. M. (2020). A sec- associated with home visiting program outcomes: A meta-analysis. ond and third look at FIRST: Testing adaptations of a principle-guided Pediatrics. doi:132), S100–S109. doi:10.1542/peds.2013-1021H youth psychotherapy. Journal of Clinical Child & Adolescent Psychology, Gamer, M., Lemon, J., Fellows, I., & Singh, P. (2019). Package ‘irr’: Various 1–14. Advance online publication. doi:10.1080/15374416.2020.1796678 coefficients of interrater reliability and agreement author. Retrieved from Churchill, R. C. (2016). The effectiveness, acceptability and cost-effectiveness https://cran.r-project.org/package=irr of psychosocial interventions for maltreated children and adolescents: an Garner, A. S., Shonkoff, J. P., Siegel, B. S., Dobbins, M. I., Earls, M. F., evidence synthesis. Health Technology Assessment, 20,1–508. McGuinn, L., … Wegner, L. M. (2012). Early childhood adversity, toxic

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. 764 L. R. Rith‐Najarian et al.

stress, and the role of the pediatrician: translating developmental science (pp. 1–9). New York, NY: John Wiley & Sons, Inc. doi:10.1002/ into lifelong health. Pediatrics, 129, e224–e231. doi:10.1542/peds.2011-2662 9781118625392.wbecp567 Gisev, N., Bell, J. S., & Chen, T. F. (2013). Interrater agreement and interrater Lindhiem, O., Bennett, C. B., Trentacosta, C. J., & McLear, C. (2014). Client reliability: Key concepts, approaches, and applications. Research in Social & preferences affect treatment satisfaction, completion, and clinical outcome: Administrative Pharmacy, 9, 330–338. A meta-analysis. Clinical Psychology Review, 34, 506–517. doi:10.1016/ Godoy, L., Long, M., Marschall, D., Hodgkinson, S., Bokor, B., Rhodes, H., … j.cpr.2014.06.002 Beers, L. (2017). Behavioral health integration in health care settings: Luby, J. L., Barch, D. M., Whalen, D., Tillman, R., & Freedland, K. E. (2018). A Lessons learned from a pediatric hospital primary care system. Journal of randomized controlled trial of parent-child psychotherapy targeting emo- Clinical Psychology in Medical Settings, 24, 245–258. doi:10.1007/ tion development for early childhood depression. American Journal of s10880-017-9509-8 Psychiatry, 175, 1102–1110. doi:10.1176/appi.ajp.2018.18030321 Green, J. G., McLaughlin, K. A., Berglund, P. A., Gruber, M. J., Sampson, N. Luby, J. L., Lenze, S., & Tillman, R. (2012). A novel early intervention for pre- A., Zaslavsky, A. M., … Kessler, R. C. (2010). Childhood adversities and school depression: Findings from a pilot randomized controlled trial. adult psychiatric disorders in the national comorbidity survey replication Journal of Child Psychology and Psychiatry and Allied Disciplines, 53, I: Associations with first onset of DSM-IV disorders. Archives of General 313–322. doi:10.1111/j.1469-7610.2011.02483.x Psychiatry, 67, 113–123. doi:10.1001/archgenpsychiatry.2009.186 MacMillan, H. L., Fleming, J. E., Streiner, D. L., Lin, E., Boyle, M. H., Jamieson, Hall, E. (2009). The Delphi primer: Doing real-world or academic research E., … Beardslee, W. R. (2001). Childhood abuse and lifetime psychopathol- using a mixed-method approach. In C. A. Lentz (Ed.), The refractive ogy in a community sample. American Journal of Psychiatry, 158, 1878– thinker: Vol. 2: Research methodology (pp. 3–27). Las Vegas, NV: 1883. doi:10.1176/appi.ajp.158.11.1878 Refractive Thinker Press. Manly, J. T., Kim, J. E., Rogosch, F. A., & Cicchetti, D. (2001). Dimensions of Hankin, B. L. (2020). Screening for and personalizing prevention of adolescent child maltreatment and children’s adjustment: Contributions of develop- depression. Current Directions in Psychological Science, 29, 327–332. mental timing and subtype. Development and Psychopathology, 13(4), doi:10.1177/0963721420920231 759–782. doi:10.1017/S0954579401004023 Holey, E. A., Feeley, J. L., Dixon, J., & Whittaker, V. J. (2007). An exploration Martini, R., Hilt, R., Marx, L., Chenven, M., Naylor, M., Sarvet, B., … of the use of simple statistics to measure consensus and stability in Delphi stud- Ptakowski, K. K. (2012). Best principles for integration of child psychiatry ies. BMC Medical Research Methodology, 7, 52. doi:10.1186/1471-2288-7-52 into the pediatric health home. Washington, DC: American Academy of Hsu, C. C., & Sandford, B. A. (2007). The Delphi technique: Making sense of Child & Adolescent Psychiatry. consensus. Practical Assessment, Research & Evaluation, 12, 10. doi:10.7275/ Masten, A. S., & Cicchetti, D. (2010). Developmental cascades. Development and PDZ9-TH90 Psychopathology, 22(3), 491–495. doi:10.1017/S0954579410000222 Jorm, A. F. (2015). Using the Delphi expert consensus method in mental McLaughlin, K. A. (2016). Future Directions in Childhood Adversity and health research. Australian and New Zealand Journal of Psychiatry, 49, Youth Psychopathology. Journal of Clinical Child & Adolescent 887–897. doi:10.1177/0004867415600891 Psychology, 45, 361–382. doi:10.1080/15374416.2015.1110823 Kessler, R. C., McLaughlin, K. A., Green, J. G., Gruber, M. J., Sampson, N. A., McLaughlin, K. A., Colich, N. L., Rodman, A. M., & Weissman, D. G. (2020). Zaslavsky, A. M., … Williams, D. R. (2010). Childhood adversities and Mechanisms linking childhood trauma exposure and psychopathology: A adult psychopathology in the WHO world mental health surveys. British transdiagnostic model of risk and resilience. BMC Medicine, 18, 96. Journal of Psychiatry, 197, 378–385. doi:10.1192/bjp.bp.110.080499 doi:10.1186/s12916-020-01561-6 Keyes, K. M., Eaton, N. R., Krueger, R. F., McLaughlin, K. A., Wall, M. M., McLaughlin, K. A., Conron, K. J., Koenen, K. C., & Gilman, S. E. (2010). Grant, B. F., … Hasin, D. S. (2012). Childhood maltreatment and the struc- Childhood adversity, adult stressful life events, and risk of past-year ture of common psychiatric disorders. British Journal of Psychiatry, 200, psychiatric disorder: A test of the stress sensitization hypothesis in a 107–115. doi:10.1192/bjp.bp.111.093062 population-based sample of adults. Psychological Medicine, 40, 1647– Kolko, D. J., & Perrin, E. (2014). The Integration of Behavioral Health 1658. doi:10.1017/S0033291709992121 Interventions in Children’s Health Care: Services, Science, and Suggestions. McLaughlin, K. A., DeCross, S. N., Jovanovic, T., & Tottenham, N. (2019). Journal of Clinical Child & Adolescent Psychology, 43,216–228. doi:10.1080/ Mechanisms linking childhood adversity with psychopathology: Learning 15374416.2013.862804 as an intervention target. Behaviour Research and Therapy, 118, 101–109. Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass doi:10.1016/j.brat.2019.04.008 correlation coefficients for reliability research. Journal of Chiropractic McLaughlin, K. A., Green, J. G., Gruber, M. J., Sampson, N. A., Zaslavsky, A. Medicine, 15, 155–163. doi:10.1016/j.jcm.2016.02.012 M., & Kessler, R. C. (2012). Childhood adversities and first onset of psychi- Kupersmidt, J. B., Stelter, R., & Dodge, K. A. (2011). Development and valida- atric disorders in a national sample of US adolescents. Archives of General tion of the social information processing application: A web-based measure Psychiatry, 69, 1151–1160. doi:10.1001/archgenpsychiatry.2011.2277 of social information processing patterns in elementary school-age boys. McLaughlin, K. A., & Lambert, H. K. (2017). Child trauma exposure and psy- Psychological Assessment, 23, 834–847. doi:10.1037/a0023621 chopathology: mechanisms of risk and resilience. Current Opinion in Lane, W. G., Dubowitz, H., Feigelman, S., Kim, J., Prescott, L., Meyer, W., … Psychology, 14,29–34. doi:10.1016/j.copsyc.2016.10.004 Tracy, J. K. (2007). Screening for parental substance abuse in pediatric McNeil, C. B., & Hembree-Kigin, T. L. (2010). Parent-child interaction therapy. primary care. Ambulatory Pediatrics, 7, 458–462. doi:10.1016/j.ambp.2007. New York, NY: Springer Science+Business Media. 07.007 Nock, M. K., Wedig, M. M., Holmberg, E. B., & Hooley, J. M. (2008). The Leijten, P., Gardner, F., Melendez-Torres, G. J., van Aar, J., Hutchings, J., Emotion Reactivity Scale: Development, evaluation, and relation to self- Schulz, S., … Overbeek, G. (2019). Meta-analyses: Key parenting program injurious thoughts and behaviors. Behavior Therapy, 39, 107–116. components for disruptive child behavior. Journal of the American doi:10.1016/j.beth.2007.05.005 Academy of Child & Adolescent Psychiatry, 58, 180–190. doi:10.1016/ Okamura, K. H., Orimoto, T. E., Nakamura, B. J., Chang, B., Chorpita, B. F., & j.jaac.2018.07.900 Beidas, R. S. (2020). A history of child and adolescent treatment through a Lenze, S. N., Pautsch, J., & Luby, J. L. (2011). Parent-child interaction therapy distillation lens: Looking back to move forward. Journal of Behavioral emotion development: A novel treatment for depression in preschool chil- Health Services & Research, 47,70–85. doi:10.1007/s11414-019-09659-3 dren. Depression & Anxiety, 28, 153–159. doi:10.1002/da.20770 Pardee, M., Kuzma, E., Dahlem, C. H. G. Y., Boucher, N., & Darling-Fisher, C. S. Lilienfeld, S. O. (2019). Reflections on clinical judgment and the dimensional- (2017). Current state of screening high-ACE youth and emerging adults in categorical distinction in the study of personality disorders: Comment on primary care. Journal of the American Association of Nurse Practitioners, Bornstein (2019). Personality Disorders: Theory, Research, and Treatment, 29,716–724. doi:10.1002/2327-6924.12531 10, 210–214. doi:10.1037/per0000318 Powell, B. J., Bosk, E. A., Wilen, J. S., Danko, C. M., Van Scoyoc, A., & Lilienfeld, S. O., & Lynn, S. J. (2014). Errors/biases in clinical decision making. In Banman, A. (2015). Evidence-based programs in “real world” settings: R.L. Cautin, & S.O. Lilienfeld (Eds.), The Encyclopedia of Clinical Psychology Finding the best fit. In D. Daro, A. C. Donnelly, L. A. Huang, & B. J.

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use. Development and Psychopathology 765

Powell (Eds.), Advances in child abuse prevention knowledge (pp. 145–177). Weich, S., Patterson, J., Shaw, R., & Stewart-Brown, S. (2009). Family relation- Cham, Switzerland: Springer. ships in childhood and common psychiatric disorders in later life: systematic R Core Team. (2019). R: A language and environment for statistical computing review of prospective studies. British Journal of Psychiatry, 194,392–398. Version 3.5.3. Vienna: R Foundation for Statistical Computing. doi:10.1192/bjp.bp.107.042515 Sanchez, A. L., Cornacchio, D., Poznanski, B., Golik, A. M., Chou, T., & Weisberg, H. F. (1992). Measures of Spread. In H. F. Weisberg (Ed.), Comer, J. S. (2018). The effectiveness of school-based mental health services Quantitative applications in the social sciences: Central tendency and vari- for elementary-aged children: A meta-analysis. Journal of the American ability (pp. 46–75). Thousand Oaks, CA: Sage. Academy of Child &Adolescent Psychiatry, 57, 153–165. doi:10.1016/ Weisz, J. R., & Bearman, S. (2020). Principle-guided psychotherapy for children j.jaac.2017.11.022 and adolescents: The FIRST treatment program for behavioral and emotional Schleider, J. L., & Weisz, J. R. (2017). Little treatments, promising effects? problems. New York, NY: Guilford Press. Meta-analysis of single-session interventions for youth psychiatric prob- Weisz, J., Bearman, S. K., Santucci, L. C., & Jensen-Doss, A. (2017). Initial test lems. Journal of the American Academy of Child & Adolescent Psychiatry, of a principle-guided approach to transdiagnostic psychotherapy with chil- 56, 107–115. doi:10.1016/j.jaac.2016.11.007 dren and adolescents. Journal of Clinical Child & Adolescent Psychology, 46, Schmidt, R. C. (1997). Managing Delphi surveys using nonparametric statisti- 44–58. doi:10.1080/15374416.2016.1163708 cal techniques. Decision Sciences, 28, 763–774. doi:10.1111/j.1540-5915. Weisz, J. R., Kuppens, S., Ng, M. Y., Eckshtain, D., Ugueto, A. M., 1997.tb01330.x Vaughn-Coaxum, R., … Fordwood, S. R. (2017). What five decades of Singla, D. R., Waqas, A., Hamdani, S. U., Suleman, N., Zafar, S. W., research tells us about the effects of youth psychological therapy: A multi- Zill-e-Huma, ., … Rahman, A. (2020). Implementation and effectiveness level meta-analysis and implications for science and practice. American of adolescent life skills programs in low- and middle-income countries: A Psychologist, 72,79–117. critical review and meta-analysis. Behaviour Research and Therapy, 130, Weisz, J. R., Sandler, I. N., Durlak, J. A., & Anton, B. S. (2005). Promoting and 103402. doi:10.1016/j.brat.2019.04.010 protecting youth mental health through evidence-based prevention and treat- Sun, M., Rith-Najarian, L. R., Williamson, T. J., & Chorpita, B. F. (2019). Treatment ment. American Psychologist, 60,628–648. doi:10.1037/0003-066X. 60.6.628 features associated with youth cognitive behavioral therapy follow-up effects for Whiteside, S. P. H., Sim, L. A., Morrow, A. S., Farah, W. H., Hilliker, D. R., internalizing disorders: A meta-analysis. Journal of Clinical Child & Adolescent Murad, M. H., … Wang, Z. (2020). A Meta-analysis to guide the enhance- Psychology, 48, S269– S283. doi:10.1080/15374416.2018.1443459 ment of CBT for childhood anxiety: Exposure over anxiety management. Surowiecki, J. (2004). The Wisdom of Crowds: Why the Many are Smarter Clinical Child and Family Psychology Review, 23, 102–121. doi:10.1007/ Than the Few and how Collective Wisdom Shapes Business, Economies, s10567-019-00303-2 Societies, and Nations. New York, NY: Doubleday. Witteman, C. L. M. (2018). Concluding commentary: Clinical decision making. Thomas, R., & Zimmer-Gembeck, M. J. (2007). Behavioral outcomes of Clinical Psychological Science, 6,266–270. doi:10.1177/2167702617745642 parent-child interaction therapy and triple P-positive parenting program: Zapf, A., Castell, S., Morawietz, L., & Karch, A. (2016). Measuring inter-rater A review and meta-analysis. Journal of Abnormal Child Psychology, 35, reliability for nominal data: Which coefficients and confidence intervals are 475–495. doi:10.1007/s10802-007-9104-9 appropriate? BMC Medical Research Methodology, 16, 93. doi:10.1186/ Timmer, S. G., Urquiza, A. J., Zebell, N. M., & McGrath, J. M. (2005). Parent- s12874-016-0200-9 child interaction therapy: Application to maltreating parent-child dyads. Zeman, J., & Garber, J. (1996). Display rules for anger, sadness, and pain: It Child Abuse & Neglect, 29, 825–842. doi:10.1016/j.chiabu.2005.01.003 depends on who is watching. Child Development, 67, 957–973. Trevelyan, E. G., & Robinson, N. (2015). Delphi methodology in health doi:10.1111/j.1467-8624.1996.tb01776.x research: How to do it? European Journal of Integrative Medicine, 7,423–428. Zigler, E. (1994). Reshaping early childhood intervention to be a more effective doi:10.1016/j.eujim.2015.07.002 weapon against poverty. American Journal of Community Psychology, 22, van der Put, C. E., Assink, M., Gubbels, J., & Boekhout van Solinge, N. F. 37–47. doi:10.1007/BF02506816 (2018). Identifying effective components of child maltreatment interven- Zigler, E. (2000). Foreword. In J. P. Shonkoff & S. J. Meisels (Eds.), Handbook tions: A meta-analysis. Clinical Child and Family Psychology Review, 21, of early childhood intervention (2nd ed., pp. xi–xv). Cambridge: Cambridge 171–202. doi:10.1007/s10567-017-0250-5 University Press. van IJzendoorn, M. H., Bakermans-Kranenburg, M. J., Coughlan, B., & Zigler, E., Pfannenstiel, J. C., & Seitz, V. (2008). The parents as teachers pro- Reijman, S. (2020). Annual research review: Umbrella synthesis of gram and school success: A replication and extension. Journal of Primary meta-analyses on child maltreatment antecedents and interventions: Prevention, 29, 103–120. doi:10.1007/s10935-008-0132-1 differential susceptibility perspective on risk and resilience. Journal of Zigler, E., & Styfco, S. J. (2001). Extended childhood intervention prepares Child Psychology and Psychiatry and Allied Disciplines, 61, 272–290. children for school and beyond. Journal of the American Medical doi:10.1111/jcpp.13147 Association, 285, 2378–2380. doi:10.1001/jama.285.18.2378 Wegman, H. L., & Stetler, C. (2009). A meta-analytic review of the effects of Zigler, E., & Styfco, S. J. (2002). A life lived at the crossroads of knowledge and childhood abuse on medical outcomes in adulthood. Psychosomatic children’s policy. New Directions for Child and Adolescent Development, Medicine, 71, 805–812. doi:10.1097/PSY.0b013e3181bb2b46 98,5–16.

Downloaded from https://www.cambridge.org/core. 09 May 2021 at 21:57:36, subject to the Cambridge Core terms of use.