Extending the Boundaries of Systematic Case Study Research: 235 Conceptual and Methodological Issues R. Elliott, S. Stephen, & A. Robinson Pragmatic Case Studies in Psychotherapy, http://pcsp.libraries.rutgers.edu Volume 17, Module 2, Article 4, pp. 235-254, 08-04-21 [copyright by authors] Commentary—Extending the Boundaries of Systematic Case Study Research: Conceptual and Methodological Issues a,b a a ROBERT ELLIOTT, SUSAN STEPHEN, ANNA ROBINSON a University of Strathclyde b Please address correspondence regarding this article to: Robert Elliott, School of Psychological Sciences & Health, University of Strathclyde, 40 George Street, Glasgow G1 1 QE United Kingdom Email: [email protected] ABSTRACT In this commentary we discuss the two examples of systematic case study research in this issue: Miller et al., (2021), who continue the development of the quasi-judicial Panels of Psychological Inquiry method by applying it to a child client with an autistic spectrum condition; and Bohart et al. (2021), who apply their research jury approach to a video recorded case of Emotionally- Focused Therapy for couples. We open by briefly summarizing the main issues addressed in our previous commentary (Stephen & Elliott, 2011), which involved the same authors; we also note some key developments in systematic case study research over the past ten years. The rest of our commentary is divided into three parts. First, we look at more general conceptual issues in systematic case study research, including situations in which systematic case studies are likely to be most useful; the problem of overly broad research questions; the definition and assessment of outcome; and the thorny issue of causality. In the second part, we turn our attention to methodological issues raised by the two articles, returning to the questions of what counts as evidence in systematic case study research (here the use of observational methods for assessing client change and change processes), but also to the processes by which research judges or jurors make decisions about knowledge claims and methods for generalizing from one case to other cases. In the final main section, we offer more substantive commentary on Miller et al. (2021), from the point of view of autism research. We start by putting the DIR/Floortime intervention in context before raising key diagnostic issues that we think circumscribe the case and spelling out uncertainties about the nature of the intervention used. We round off this section with a set of proposals for future systematic single case research on interventions for autism. We close our commentary with a brief set of recommendations. Keywords: systematic case study research; autism; emotionally focused therapy for couples; qualitative research; case studies; clinical case studies 1. INTRODUCTION: THE STORY SO FAR In our commentary (Stephen & Elliott, 2011) on an earlier set of articles that include the same first authors (Miller, 2011; Bohart et al., 2011), we began with a brief introduction to the Extending the Boundaries of Systematic Case Study Research: 236 Conceptual and Methodological Issues R. Elliott, S. Stephen, & A. Robinson Pragmatic Case Studies in Psychotherapy, http://pcsp.libraries.rutgers.edu Volume 17, Module 2, Article 4, pp. 235-254, 08-04-21 [copyright by authors] Hermeneutic Single Case Efficacy Design (HSCED) method, which had in turn been inspired by and built on the work of Bohart and colleagues in particular (e.g., Bohart & Boyd, 1997; Bohart & Humphreys, 2000). We presented the main procedures used in HSCED studies, including the collection of both direct or confirming evidence linking therapy process to outcome and indirect or disconfirming evidence (Elliott, 2002). We described how HSCED came to incorporate adjudication (the systematic use of affirmative and skeptic positions and judges) into its basic procedures (e.g., Elliott et at., 2009; others). We then turned our attention to the studies at hand, first exploring the sources of evidence employed (i.e., client, therapist, observers), an issue that will be important in our commentary here. We advocated for the use of Rich Case Records that include a range of information from different perspectives and obtained by different methods (qualitative and quantitative). Also relevant in our discussion was the use of witnesses, including therapist and supervisor. Second, we explored the issue of how evidence is tested in the various systematic case study methods. For example, the cross examination of evidence, generally used metaphorically, is taken literally in Miller’s (2011) Panel of Psychological Inquiry (PPI) method, as the PPI Advocate and Critic interrogated therapist and supervisor in front of the panel of judges. We also noted the difficulty of enacting the role of Critic or Skeptic in both the PPI method and HSCED studies and steps that have been taken to help overcome the reluctance of researchers in this role. Another evidence-related issue was whether and how to provide judges with criteria for evaluating evidence. A third set of key issues we looked at involved proof, including knowledge claims, burden of proof, and standards of proof. We noted that Miller’s PPI method typically considers a broad range of knowledge claims, whereas HSCED emphasizes (a) whether or how much the client changed and (b) whether/how much therapy was responsible for client change. In contrast, the PPI method also tests knowledge claims about client diagnosis, treatment integrity, and generalizability. (The first two of these are background claims in HSCED research but are not a main focus, while HSCED studies handle the generalizability claim in a quite different manner, which we will discuss here again in more detail.) We then highlighted similarities between burden of proof in PPI and in HSCED (it belongs to the party putting forward the knowledge claim, usually the advocate or affirmative side. Next, we discussed standard of proof, the third proof-related issue, in terms of the subjective probability levels associated with different standards and argued in favor of intermediate standard (“clear and convincing evidence”, roughly >80% probability) between “preponderance of the evidence” (>50% probability) and “beyond a reasonable doubt” (>95% probability). The fourth set of issues we considered had to do with the adjudication process itself, beginning with the lack of current knowledge about the process of judging PPI or HSCED cases but also including the number and selection of judges. Extending the Boundaries of Systematic Case Study Research: 237 Conceptual and Methodological Issues R. Elliott, S. Stephen, & A. Robinson Pragmatic Case Studies in Psychotherapy, http://pcsp.libraries.rutgers.edu Volume 17, Module 2, Article 4, pp. 235-254, 08-04-21 [copyright by authors] We concluded with a discussion of recommended future developments, specifically calling for the following: (a) further clarification of the advocate/affirmative and critic/skeptic roles; (b) careful consideration of how best to respect the experience of the client and therapist whose encounter is the subject of the adjudicated case study; (c) development and clarification of explicit and appropriate standards of proof; and (d) further research into the processes by which jurors/judges evaluate the evidence. (Stephen & Elliott, 2011, pp. 239) In this follow-up commentary, we will revisit the above issues in relation to the two studies presented here in this special section. According to Google Scholar, our previous commentary has been cited 27 times. We are pleased to note that the great majority (15) of these citations were used to support systematic case studies (and more particularly HSCED studies; e.g., Wall et al., 2014). Citations were most commonly made by UK-based researchers (10), but also included researchers from Canada (9), Chile (4), Portugal (2), the Czech Republic (2) and Finland (1). Our commentary was most often cited by researchers representing nonmainstream psychotherapies (more on this later), including Lifespan Integration therapy (9), transactional analysis (4) and cognitive analytic therapy (3). Since our previous commentary we see two key developments. First, Benelli et al. (2015) reported an extensive, detailed systematic review of thirteen HSCED studies, resulting in a checklist of 28 evaluative criteria. Second, Wall et al. (2017) and Stephen et al. (2021) have extended HSCED to include multiple case designs in which collections of 3 or more cases are presented together in the same report, including comparing good and poor outcome cases. For example, in Stephen et al., for the poor outcome cases, the affirmative case sought to prove that the change experienced by the client at the end of therapy was deterioration, rather than improvement. Then, the data generated by the eight HSCED studies (affirmative and skeptic cases, adjudication panel decisions) were synthesized and compared using qualitative metasynthesis in order to develop an understanding of similarities and differences in the therapeutic processes that might explain the contrasting outcomes. 2. GENERAL CONCEPTUAL ISSUES IN SYSTEMATIC CASE STUDY RESEARCH In this section we look at a set of more general issues in systematic case study research such as the two approaches used in this special section and in HSCED. It is our view that these issues have been somewhat neglected in previous writing about systematic case study methods, so it is our goal here to raise awareness about them in order to stimulate further discussion.