CP 951 Research on Individual Interventions Fall 2015

270-951 Research on Individual Interventions Initial Version: 2015-09-08 Fall 2015

Instructor: Bill Hoyt Class meeting: Tuesdays 8-10:45 335D Education Education 327 [email protected] 608-262-0462 Office hours: By appointment

Course Learning Goals  Understand research on individual interventions (counseling and psychotherapy).  Understand principles of scientific inference; develop ability to critique scientific claims and arguments based on evidence.  Recognize strengths and limitations of different forms of evidence adduced to support claims regarding intervention effectiveness.  Apply scientifically informed principles to psychological practice and training.

Required Readings Wampold, B. E., & Imel, Z. E. (2015). The great psychotherapy debate: The evidence for what makes psychotherapy work (2nd ed.). New York: Routledge.

Norcross, J. C., Beutler, L. E., & Levant, R. F. (Eds.) (2006). Evidence-based practices in mental health: Debate and dialogue on the fundamental questions. Washington, DC: American Psychological Association.

Readings on LearnUW.

Recommended Readings/Resources Heppner, P. P., Kivlighan, D. M., Jr., & Wampold, B. E. (1999). Research design in counseling (2nd ed.). Belmont, CA: Brooks/Cole.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi- experimental designs for generalized causal inference. Boston: Houghton Mifflin.

Accommodations I will make every effort to accommodate students with disabilities in this class. If you have need of some accommodation in class, in exams, or in assignments, please let me know as soon as possible. To the extent possible, I will keep our communications confidential.

1 CP 951 Research on Individual Interventions Fall 2015

Course Outline for CP 951

DATE TOPIC READINGS 9-8 Introduction and overview Worthington et al. (2015)

9-15 History Wampold & Bhati (2004) GPD, Ch. 1-2 Evidence and inference APA Task Force on EBP (2006) Epstein (1997) NBL Ch. 1, 2, 9

9-22 Sampling error and meta- Rosenthal et al. (2006) analysis Schmidt (1992) GPD Ch. 3 Threats to validity presentations: Statistical Conclusion Validity

9-29 Absolute efficacy GPD Ch. 4 Chambless & Hollon (1998) Threats to validity Westen et al. (2004) presentations: Internal Validity

10-6 Relative efficacy GPD Ch. 5 NBL Ch. 3, 7 Threats to validity Wampold et al. (1997) + presentations: Construct resps Validity

10-13 Therapist effects: Evidence GPD Ch. 6 for consistent differences in Baldwin & Imel (2013) outcomes between chapter therapists McKay, Imel, & Wampold (2005) BEA Competencies document

10-20 Placebo effects; allegiance GPD Ch. 7 effects Wampold et al. (2005) Kaptchuk (2008) Strength of evidence coding: process-outcome studies

10-27 Adherence, alliance, and NBL Ch. 6 other general effects GPD Ch. 7 Baldwin, Wampold, & Imel General effects: Additional (2007) controversies

Strength of evidence coding: field studies

11-3 Specific effects: Evidence GPD Ch. 8-9 regarding treatment Blatt & Zuroff (2005) differences TDCRP

Strength of evidence coding: RCTs

11-10 Predictors of therapist Fouad et al. (2009) effects; Competencies Lichtenberg et al. (2007) CP Core Competencies document

11-17 Culturally adapted Benish et al. (2011) treatments; multicultural Huey et al. (2014) competencies Imel et al. (2011) NBL Ch. 8

11-24 Predictors of therapist Tracey et al. (2014) effects: Therapist Faust & Ziskin (1988) experience and training Dawes et al. (1989)

[In class: finalize coding system for strength of evidence paper.]

12-1 Measurement of outcomes Goldfried (1997) chapter in treatment evaluation Lambert (2011) research Minami et al. (2007)

Assessment of outcomes in practice

12-8 Implications GPD Ch. 9 APA OEMA Special Report 12-15 Strength of evidence presentations

Final paper due 5 pm (LearnUW)

Evaluation Course grades are out of 100 points, with 94+ = A; 89-93 = AB; 84-88 = B; 79-83 = BC; 72-78 = C; 66-71 = D; 65- = F. Weekly reaction to readings 10 points Threat to validity presentation 10 points Strength of evidence coding system 20 points Strength of evidence paper 35 points Strength of evidence presentation 15 points Class participation 10 points TOTAL 100 points

Assignments

1. Weekly reactions to readings (10 points). Each student will upload to LearnUW a 250-word (minimum: about ½ page single spaced) reaction to the week’s readings during Weeks 2-13 of the course. Examples of constructive reactions include (but are not limited to): (a) exploration of apparent contradictions among the readings; (b) critique of one (or more) claims made in the readings, based on logical links to evidence provided; (c) consideration of theoretical or practical implications of claims made in the readings; (d) questions about links between evidence and conclusions in one (or more) of the readings.

Reactions should be cut and pasted into Dropbox on LearnUW (folder for the appropriate week) by 7 pm each Monday (starting Week 2). Grading will be based on timely submission and appropriate length. Students will lose 0.5 points for short or late submissions and 1 point for missing submissions after the first. (No penalty for first late or missing reaction.) If your reaction includes formatting that will not be rendered neatly in text, it is acceptable to upload a Word document to Dropbox.

2. Threats to validity (10 points). Each student will be assigned one threat to validity, as described (e.g.) by Shadish, Cook, & Campbell (2002). Your assignment is to (a) carefully read the description of this threat (in SCC or HKW, e.g.), (b) locate a published journal article that illustrates this threat to validity and (c) prepare a 15-minute in-class presentation in which you describe the threat, describe the methods and conclusions of the study in sufficient detail to illustrate how the threat makes interpretation problematic, and discuss alternatives to the authors’ proffered conclusions.

Upload your illustrative article to the Discussion Topic for your threat on LearnUW by Sunday 11:59 pm before the Tuesday on which you are scheduled to present. You can do this by navigating to the Topic, clicking on “Start a Thread,” then using “Add File” to upload a PDF of your article. If desired, you can also post a brief comment to call readers’ attention to salient features of the illustrative study, or to post discussion question(s) for readers to consider.

3. Quantifying strength of evidence (20 points). In this exercise, students will work in small groups to develop part of a coding system that can be used to index the strength of scientific support for claims about what makes psychotherapy work. The report of the Task Force on Evidence-Based Practice (2006, p. 274) considers 9 forms of evidence relevant to claims about practice:

1. Clinical observation 2. Qualitative research 3. Single-case studies 4. Single-case experiments 5. Public health and ethnographic research 6. **Process-outcome studies

7. **Intervention studies in naturalistic settings 8. **RCTs 9. Meta-analysis

For this assignment, we will focus on #6-8, as these are (a) relatively rigorous forms of evidence and (b) commonly available and relevant to the types of claims we study in this class. Each group will work on one of these evidence types, and should proceed as follows:

 Collect 5-10 exemplar studies. These should all include evidence of the designated type, but they do not all need to focus on the same research question. Read these carefully to identify common and unique features, focusing on similarities and differences that are relevant to threats to validity, for the claims each study seeks to investigate.  Based on your review of the studies, and on your knowledge of research design and scientific inference (consult HKW; SCC), make a list of features that are most important to assessing the strength of the research design for this type of study, for supporting the desired inferences.  Prioritize these features, making judgments about which are central and which are peripheral, from the perspective of providing strong support for logical inferences. Consider the various types of validity enumerated by SCC (i.e., statistical conclusion validity; internal validity; construct validity; external validity). Recognize that some of these features are likely to be negatively correlated (e.g., there is usually a trade-off between internal and external validity; it is difficult to design a study that gets high marks on both).  Based on this list, design a scoring system to allow raters to quantify methodological quality for studies of this type. (The goal is for the system to be usable to code the quality of a single study, although ultimately we want to use it to quantify quality in a set of studies cited in support of a particular claim.) For scaling purposes, design the scoring system to have a minimum score of 0 and a maximum score of 100.  Create a coding sheet briefly describing (a) each of your scoring categories or dimensions, (b) levels of each category, and (c) the score assigned to each.  Upload the scoring sheet, and three example studies, to the Strength of Evidence forum (Discussions on LearnUW), by Sunday 11:59 pm before the Tuesday on which you will present.  In class, bring copies of your scoring sheet and explain the scoring system to classmates, ideally using example studies different from those you uploaded to LearnUW. If revisions suggest themselves based on the class discussion, make these soon after class and upload the revised version of the scoring sheet to the Discussion forum. As an exercise in interrater reliability, all class participants will score each of the three example studies before the next class meeting, to determine how reliably we are able to apply the system.

NOTE. Do not assume that every threat to validity discussed by SCC is relevant (and certainly not equally relevant) to your specific research method. The list of threats in

SCC is designed to be heuristic, to alert you to some inferential challenges you may want to think about. Your job in this project is to choose those that are likely to be most important strengths or limitations of this type of study, from the perspective of providing evidence for claims about what works in psychotherapy and counseling.

4. Strength of evidence for a specific claim--paper (35 points). Pick one claim relevant to the question what makes counseling/psychotherapy effective. (See list of possible factors, next page.) Starting with assigned readings and expanding by examining their reference lists, evaluate the empirical basis for the claim that this factor contributes to therapeutic outcomes.  What is the historical basis for the claim? Who first advanced this possible explanation for effectiveness? Are there current trends (e.g., economic, political, scientific) that tend to make this claim attractive?  What evidence to proponents adduce in support of this claim? Give an overview of the forms of evidence (including the types listed by the Task Force for EBP, 2006), and a preliminary judgment of its persuasiveness.  Provide a detailed analysis of the 2-3 most critical bodies of evidence. (A “body of evidence” could be a meta-analysis of many related studies, a single seminal study that established the claim, or a group of observations (perhaps scientific, perhaps clinical) that are seen as supportive.)  Discuss the main conflicting findings. These are findings that tend to refute the claim (or are advanced by antagonists as evidence against the claim). How persuasive are these counterarguments?  Apply the “strength of evidence” coding system we developed as a group to the cumulative body of evidence (both supportive and contradictory). Overall, how do you judge the strength of the evidentiary basis for this claim? How confident can we be that the identified factor contributes to therapeutic efficacy (and how strong is its influence)?  What additional evidence would be helpful to confirm (or refute) this claim?  What changes to the “strength of evidence” coding system do you suggest to make it more applicable to the forms of evidence you reviewed in this paper?

Your paper should be 10-15 pages, including references, and should conform to APA style (APA Publication Manual, 6th ed.). You should submit your paper electronically via LearnUW Dropbox, by 5 pm the day of our final class meeting.

5. Strength of evidence for a specific claim—presentation (15 points). Use a powerpoint presentation (or other visual aids) to present the findings from your “strength of evidence” investigation to the class during one of the final two class meetings. Your presentation should run 15-20 minutes, with 10 minutes for questions and discussion.

6. Class participation (10 points). This is a doctoral seminar. I expect that you will have read material for the week carefully prior to class and come prepared to discuss your questions and reactions. Lateness will result in a 0.5 point deduction from your participation grade, and missing a class will result in a 1 point deduction after the first late arrival or absence. (No penalty for the first late arrival or excused absence.)

CP 951—Possible Paper Topics: Factors Related to Therapeutic Effectiveness

Client Characteristics  Expectations for therapy o Outcome expectations o Process expectations o Role expectations  Cultural and individual differences  Client extratherapeutic engagement

Therapist Characteristics  Evidence for therapist effects  Predictors of therapist effects o Credibility o Facilitative conditions . Empathy . Positive regard . Genuineness o Therapist competencies o Multicultural competencies

Therapeutic Relationship  Working Alliance o Bond o Tasks o Goals  Changes in working alliance over time  Therapist-client matching  Therapist flexibility to adapt intervention to client values and worldview  Considerations for establishing relationship with specific groups

Placebo Effects  Evidence for placebo effects (vs. no-treatment)  Predictors of placebo effects

Specific Effects  Evidence for specificity  Claims of disadvantages of encouraging/mandating specific treatments o Manualization versus flexibility o Evidence of value of treatment adaptation for particular groups o Manual-determined time limits versus therapist-determined time limits

General Issues in Studying Effectiveness  Generalizability of research findings to “clinically representative” conditions  Outcome measurement in psychotherapy research  Evidence for harmful effects of intervention

CP 951 Readings F15 APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. The American Psychologist, 61(4), 271–85. http://doi.org/10.1037/0003-066X.61.4.271 ASPE. (2014). Strategies for measuring the quality of psychotherapy: A white paper to inform measure development and implementation. Retrieved December 1, 2016, from https://aspe.hhs.gov/report/strategies-measuring-quality-psychotherapy-white-paper-inform- measure-development-and-implementation Baldwin, S. a, Wampold, B. E., & Imel, Z. E. (2007). Untangling the alliance-outcome correlation: exploring the relative importance of therapist and patient variability in the alliance. Journal of Consulting and Clinical Psychology, 75(6), 842–852. http://doi.org/10.1037/0022-006X.75.6.842 Baldwin, S., & Imel, Z. E. (2013). Therapist effects: Findings and methods. In M. J. Lambert (Ed.), Handbook of psychotherapy and behavior change (6th ed., pp. 258–297). New York: Wiley. Benish, S. G., Quintana, S., & Wampold, B. E. (2011). Culturally adapted psychotherapy and the legitimacy of myth: A direct-comparison meta-analysis. Journal of Counseling Psychology, 58(3), 279–289. http://doi.org/10.1037/a0023626 Beutler, L. E., Forrester, B., Holt, H., & Stein, M. (2013). Common, specific, and cross-cutting psychotherapy interventions. Psychotherapy, 50(3), 298–301. http://doi.org/10.1037/a0032405 Blatt, S. J., & Zuroff, D. C. (2005). Empirical evaluation of the assumptions in identifying evidence based treatments in mental health. Clinical Psychology Review, 25(4), 459–486. http://doi.org/10.1016/j.cpr.2005.03.001 Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66(1), 7–18. http://doi.org/10.1037/0022-006X.66.1.7 Counseling Psychology Core Competencies. (n.d.). Retrieved December 1, 2016, from http://www.ccptp.org/assets/docs/copsy competencies final2.pdf Crits-Christoph, P. (1997). Limitations of the dodo bird verdict and the role of clinical trials in psychotherapy research: Comment on Wampold et al. (1997). Psychological Bulletin, 122(3), 216– 220. http://doi.org/10.1037/0033-2909.122.3.216 Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science (New York, N.Y.), 243(4899), 1668–1674. http://doi.org/10.1126/science.2648573 Epstein, S. (1997). This I have learned from over 40 years of personality research. Journal of Personality, 65(1), 3–32. http://doi.org/10.1111/j.1467-6494.1997.tb00527.x Faust, D., & Ziskin, J. A. Y. (2008). The expert witness in psychology and psychiatry. Science, 241(4861), 31–35. Fouad, N. A., Grus, C. L., Hatcher, R. L., Kaslow, N. J., Hutchings, P. S., Madson, M. B., … Crossman, R. E. (2009). Competency benchmarks: A model for understanding and measuring competence in professional psychology across training levels. Training and Education in Professional Psychology, 3(4, Suppl), S5–S26. http://doi.org/10.1037/a0015832 Goldfried, M. R. (1997). Considerations in developing a core assessment battery. In H. H. Strupp, L. M. Horowitz, & M. J. Lambert (Eds.), Measuring patient change in mood, anxiety, and personality disorders: Toward a core battery (pp. 99–114). Washington, DC: American Psychological Association. Howard, K. I., Krause, M. S., Saunders, S. M., & Kopta, S. M. (1997). Trials and tribulations in the meta- analysis of treatment differences: Comment on Wampold et al. (1997). Psychological Bulletin, 122(3), 221–225. http://doi.org/10.1037/0033-2909.122.3.221 Huey, S. J., Tilley, J. L., Jones, E. O., & Smith, C. A. (2014). The Contribution of Cultural Competence to Evidence-Based Care for Ethnically Diverse Populations. Annu. Rev. Clin. Psychol, 10, 305–38. http://doi.org/10.1146/annurev-clinpsy-032813-153729

Hunsberger, P. H. (2007). Reestablishing clinical psychology’s subjective core. The American Psychologist, 62(6), 614–615. http://doi.org/10.1037/0003-066X62.6.614 Imel, Z. E., Baldwin, S., Atkins, D. C., Owen, J., Baardseth, T., & Wampold, B. E. (2011). Racial/ethnic disparities in therapist effectiveness: A conceptualization and initial study of cultural competence. Journal of Counseling Psychology, 58(3), 290–298. http://doi.org/10.1037/a0023284 JW, S., RJ, B., PS, A., Swanson, J. W., Sciences, B., & Bonnie, R. J. (2015). Getting serious about reducing suicide: More “how” and less “why.” Jama, 27514, 1–2. http://doi.org/10.1001/jama.2015.15566.Conflict Laska, K. M., Gurman, A. S., & Wampold, B. E. (2014). Expanding the lens of evidence-based practice in psychotherapy: A common factors perspective. Psychotherapy, 51(4), 467–481. http://doi.org/10.1037/a0034332 Lichtenberg, J. W., Portnoy, S. M., Bebeau, M. J., Leigh, I. W., Nelson, P. D., Rubin, N. J., … Kaslow, N. J. (2007). Challenges to the assessment of competence and competencies. Professional Psychology: Research and Practice, 38(5), 474–478. http://doi.org/10.1037/0735-7028.38.5.474 McKay, K. M., Imel, Z. E., & Wampold, B. E. (2006). Psychiatrist effects in the psychopharmacological treatment of depression. Journal of Affective Disorders, 92(2-3), 287–290. http://doi.org/10.1016/j.jad.2006.01.020 Minami, T., Wampold, B. E., Serlin, R. C., Kircher, J. C., & Brown, G. S. J. (2007). Benchmarks for psychotherapy efficacy in adult major depression. Journal of Consulting and Clinical Psychology, 75(2), 232–243. http://doi.org/10.1037/0022-006X.75.2.232 Norcross, J. C., Beutler, L. E., & Levant, R. F. (Eds.). (2006). Evidence-based practices in mental health: Debate and dialogue on the fundamental questions. Washington, DC: American Psychological Association. OEMA special section: Indigenous peoples: Promoting psychological healing and well-being. (n.d.). http://doi.org/10.1017/CBO9781107415324.004 Stuart, R. B., & Lilienfeld, S. O. (2007). The evidence missing from evidence-based practice. The American Psychologist, 62(6), 615–616. http://doi.org/10.1037/0003-066X62.6.615 Tracey, T. J. G., Wampold, B. E., Lichtenberg, J. W., & Goodyear, R. K. (2014). Expertise in psychotherapy: an elusive goal? The American Psychologist, 69(3), 218–29. http://doi.org/10.1037/a0035099 Wampold, B. E., & Bhati, K. S. (2004). Attending to the Omissions: A Historical Examination of Evidence- Based Practice Movements. Professional Psychology: Research and Practice, 35(6), 563–570. http://doi.org/10.1037/0735-7028.35.6.563 Wampold, B. E., Goodheart, C. D., & Levant, R. F. (2007). Clarification and elaboration on evidence- based practice in psychology. The American Psychologist, 62(6), 616–618. http://doi.org/10.1037/0003-066X62.6.616 Wampold, B. E., & Imel, Z. E. (2015). The great psychotherapy debate: The evidence for what makes psychotherapy work (2nd ed.). New York: Routledge. Wampold, B. E., Minami, T., Tierney, S. C., Baskin, T. W., & Bhati, K. S. (2005). The placebo is powerful: Estimating placebo effects in medicine and psychotherapy from randomized clinical trials. Journal of Clinical Psychology, 61(7), 835–854. http://doi.org/10.1002/jclp.20129 Wampold, B. E., Mondin, G. W., Moody, M., & Ahn, H. (1997). The flat earth as a metaphor for the evidence for uniform efficacy of bona fide psychotherapies: Reply to Crits-Christoph (1997) and Howard et al. (1997). Psychological Bulletin, 122(3), 226–230. http://doi.org/10.1037/0033- 2909.122.3.226 Wampold, B. E., Mondin, G. W., Moody, M., Stich, F., Benson, K., & Ahn, H. (1997). A meta-analysis of outcome studies comparing bona fide psychotherapies : Empirically , “All must have prizes .” Psychological Bulletin, 122(3), 203–215.

Wendt, D. C., & Slife, B. D. (2007). Is evidence-based practice diverse enough? Philosophy of science considerations. The American Psychologist, 62(6), 613–614. http://doi.org/10.1037/0003- 066X62.6.613 Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2004a). The empirical status of empirically supported psychotherapies: assumptions, findings, and reporting in controlled clinical trials. Psychological Bulletin, 130(4), 631–63. http://doi.org/10.1037/0033-2909.130.4.631 Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2004b). The Empirical Status of Empirically Supported Psychotherapies: Assumptions, Findings, and Reporting in Controlled Clinical Trials. Psychological Bulletin, 130(4), 631–663. http://doi.org/10.1037/0033-2909.130.4.631 Whipple, J. L., & Lambert, M. J. (2011). Outcome measures for practice. Annual Review of Clinical Psychology, 7, 87–111. http://doi.org/10.1146/annurev-clinpsy-040510-143938 Worthington, E. L., Wade, N. G., & Hoyt, W. T. (2016). Evidence-based practice in psychology: What psychotherapists can learn from research on treatment and psychotherapist responsiveness. In H. E. A. Tinsley, S. H. Lease, & N. S. Giffin Wiersma (Eds.), Contemporary theory and practice in counseling and psychotherapy (p. 511). Los Angeles: Sage.