Screening and Data Extraction Forms For

Screening and Data Extraction Forms for Systematic Review of Effects of Multisystemic Therapy (MST) Julia H. Littell, Margo Campbell, Stacy Green, Jessica Schaffner Wilen, & Barb Toews Bryn Mawr College

Steps in the screening and data extraction process:

1) Independent screening of documents from titles and abstracts (Level 1). If not excluded at Level 1, retrieve full text. Link reports to studies, assign study ID and report IDs (in excel spreadsheets).

2) Independent coding of study eligibility criteria (Level 2). Compare results, calculate kappas, resolve differences, and determine eligibility.

3) If eligible, independent coding of study Levels 3 and 5 on paper. Compare results, calculate kappas, and resolve differences.

4) Independent electronic entry of outcome measures (Level 4, 1st spreadsheet). Compare results, resolve differences, and clean data.

5) Independent electronic entry of outcome data (Level 4, 2nd spreadsheet). Compare results, resolve differences, and clean data.

6) Enter relevant data into RevMan.

1 Littell et al., MST review, data extraction forms, 19 February 2013 Level 1: Initial Screening (document level; from titles and abstracts) 1.1. Is this document about MST (perhaps in addition to other topics)? o Yes o No [STOP, code as unrelated] o Can’t tell [RETRIEVE FULL TEXT] 1.2. What is this? o Study of effects of MST for social, emotional or behavioral problems [GO TO LEVEL 2] o Study of effects of MST for medical condition(s) [STOP] o Review of MST outcome studies [HARVEST REFERENCES] o Descriptive, correlational, single-group, or case study [STOP] o Theoretical or position paper, editorial, or book review [STOP] o Practice guidelines or treatment manual [STOP] o Can’t tell [RETRIEVE FULL TEXT]

2 Level 2: Eligibility Decision (study level; based on full text, multiple reports if available)

Study ID _____ Coder’s initials ______Date ______

Reports associated with this study: Report ID First 3 Authors Date 1 2 3 [ATTACH INFORMATION ON ADDITIONAL REPORTS AS NEEDED]

2.1. Does this study include two or more parallel cohorts (groups that received different treatments and were assessed at the same points in time)? o Yes o No [STOP, excluded] o Can’t tell 2.2. Is it a randomized experiment? o Yes o No o Can’t tell 2.3. Does this study include a licensed MST program? o Yes o No [STOP, excluded] o Can’t tell 2.4. Does it include youth (ages 10-17) with social, emotional, or behavioral problems? o Yes o No [STOP, excluded] o Can’t tell 2.5. Is the primary presenting problem a medical condition (diabetes or HIV-positive status)? o Yes [STOP, excluded] o No o Can’t tell

3 Littell et al., MST review, data extraction forms, 19 February 2013 Level 3: Data Extraction (study level; based on full text, multiple reports if available)

PLEASE IDENTIFY REPORT DATE AND PAGE NUMBERS WHERE INFORMATION WAS FOUND (IN RIGHT MARGIN)

Research methods

3.1. How were comparison/control groups formed? o Random assignment o Other (SKIP TO QUESTION 3.NR) 3.2. If random assignment, specify design o Simple/systematic (individuals/families) o Stratified/blocked (IDENTIFY STRATIFYING/BLOCKING VARIABLES) o Yoked pairs (created by timing of enrollment into the study) o Matched pairs (IDENTIFY MATCHING VARIABLES) o Cluster (group) randomized [CHECK UNIT OF ANALYSIS] o Other (SPECIFY) o Unclear 3.3. Who performed group assignment? o Research staff o Program staff o Other (SPECIFY) o Unclear 3.4. How was random assignment performed? o Computer generated o Random numbers table o Coins or dice o Other (DESCRIBE) o Unclear

4 3.NR. FOR NONRANDOMIZED STUDIES ONLY: What method(s) were used to construct the comparison group(s)? o Convenience sample of individuals/families in the same geographic area o Comparison of different geographic areas or agency/school clusters [STOP] o Matched individual/family pairs on background characteristics (describe variables and methods used) o Matched groups on background or other characteristics (describe variables and methods used) o Propensity score matching o Other (SPECIFY)

5 Littell et al., MST review, data extraction forms, 19 February 2013 3.5 Start and end dates of enrollment in the study

3.6 Funding sources (list all) and contract/grant/project numbers if available

6 Settings 3.6. How many separate sites were included in the study? ______3.7. Location of interventions (check all that apply) o Urban o Suburban o Rural o Unclear 3.8. Location details (city, state, country) 3.9. Primary service sector o Juvenile justice o Mental Health o Child Welfare o Other (specify) o Unclear 3.10. Describe sample inclusion criteria

Sample Characteristics

3.11. Sample size

Number of cases MST Control Total Pg# & Notes Referred to study Consented Randomly assigned Completed treatment

3.12. Are outcome data available for program completers only?

o Yes o No o Unclear

7 Littell et al., MST review, data extraction forms, 19 February 2013 3.13. Demographic characteristics

Total sample Pg# & Notes Gender % Male Youth ages Mean (sd) Min and Max Race/ethnicity % Caucasian % Afr/Amer % Hisp/Latino % Asian/Pacific % Other Socioeconomic status % college educ % unemployed % receive public aid Family composition % single parent household Primary caregiver relation to youth Mean number of children Other information Other sample characteristics

3.14. Were there any significant differences between program and control groups at baseline? o Yes (describe differences) o No (how do we know?) o Unclear

8 3.15. Were there any significant differences between MST program completers and drop-outs? o Yes (describe differences) o No (how do we know?) o Unclear 3.16. Were there any significant differences between completers and drop-outs in the control group(s)? o Yes (describe differences) o No (how do we know?) o Unclear

3.17. MST Service Characteristics

Min Max Mean SD Pg# & Notes Duration in of services (in days) Total hours of contact

3.18. Other characteristics of MST services

3.19. Characteristics of MST staff (education, demographics, etc.)

3.20. Describe methods used to insure quality of MST services (supervision, training, consultation)

3.21. Is there any information on program adherence (fidelity) to MST? o Yes (describe) o No o Unclear 3.22. Were TAM scores collected or reported? o Yes (collected and reported) o Collected, not reported

9 Littell et al., MST review, data extraction forms, 19 February 2013 o No (not collected) o Unclear 3.33.Were there any implementation differences between sites? (TAM scores OR any qualitative/quantitative differences) o Yes (describe differences) o No (how do we know?) o Unclear 3.34. MST program costs o Cost per case o Total cost o No info

Services provided to control cases 3.35. Type of control group o Usual services (treatment as usual) o Alternative service (DESCRIBE) o No service 3.36 Describe services provided to control group o Duration of services: o Total hours of contact 3.37. Characteristics of staff who provided services to control cases (education, demographics, etc.)

10 Outcome measures (study level) 3.38. Were interviews conducted by research staff? o Yes o No, but interviews were conducted by program staff o Unclear who conducted interviews o No interviews were conducted 3.39. Who participated in interviews (check all that apply)? o Mothers o Fathers o Youth o Teachers o Clinicians o Other (SPECIFY) 3.40. When were data collected? (check all that apply) o Baseline o Immediate post-treatment o Follow-ups (when?) o Other 3.41. Were data collected in the same manner for MST and control groups? o Yes o No (what were the differences?) Can’t tell

11 Littell et al., MST review, data extraction forms, 19 February 2013 Level 4: Outcome measures (outcome level) Instructions: Enter outcomes measures in Excel in the order in which they are described in the text. Enter each conceptually-distinct outcome and instrument, regardless of whether data were collected (at the time of the report) or reported. Note that a single outcome measure can be completed by multiple sources and at multiple points in time (data from specific sources and time-points will be entered later).

Timing of Reliablity & Format Direction Sources Mode of Admin Blind? Pg# & data collection Validity (identify all) notes (all planned) Conceptual o Baseline Info from: o Dichoto High score or o Youth o Self (paper) o Yes domain code: o Post-tx o Other samples my event is o Mother o Interview o No o Follow-ups o This sample (e.g., o Father o Other o Unclear (identify o Unclear event) o Positive o Parents all) o Continu o Negative o Teacher(s) Description: Info provided: ous o Unclear o Clinician (e.g., o Admin data scale) o Other o Unclear Instrument or Multiple Definition: sources reported? o Separate o Averaged (which?) o Selected (which?)

Conceptual domains codes: 1= Placement (jail/hospital/foster) 2= Arrest/conviction 3= Substance use 4= Delinquency 5= Peer relations (include social skills) 6= Youth behavior and symptoms 7= Parent behavior and symptoms 8= Family functioning (include parental supervision 12 Littell et al., MST review, data extraction forms, 19 February 2013 9= School (grades, attendance)

Note: Repeat as often as necessary to code all outcome measures.

13 Littell et al., MST review, data extraction forms, 19 February 2013 Level 4: Outcome data

Please enter outcome data in the Excel sheet. Enter dichotomous outcomes first, then continuous outcomes. Outcome # refers to the measures described above.

Dichotomous outcome data Enter data only if it is provided (do not perform calculations). OR = odds ratio. Enter exact p-value if available. If covariates (control variables) are used in the analysis, please identify these variables under Statistics (cov). Outc # Timing Source Valid Ns n w/ event % w event Statistics Pg# & notes o Base o Youth MST MST MST OR line o Mother 95%CI (LB UB) o Post o Father Chi2 tx o Teacher Control Control Control Df o Foll o Clinician p-val Other ow- o Admin data Cov up o Other (whe n?) o Othe r

Subsa mple? (descri be) *Repeated as often as needed

Continuous outcome data If change/gain scores are provided, enter under "other data." If covariates (control variables) are used in the analysis, please identify these variables under Statistics (cov). Outc # Timing Source Valid Ns Means SDs Statistics Pg# & notes o Baseline o youth MST MST MST p o Post tx o parent t o 1st f-u o teacher F o 2nd f-u o clinician Control Control Control df o 3rd f-u o admin data ES 14 Littell et al., MST review, data extraction forms, 19 February 2013 o 4th f-u o other Other o 5th f-u Cov o Other *Repeated as often as needed

15 Littell et al., MST review, data extraction forms, 19 February 2013 Level 5: Risk of bias (study level; based on full text, multiple reports if available)

PLEASE IDENTIFY REPORT DATE AND PAGE NUMBERS WHERE INFORMATION WAS FOUND (IN RIGHT MARGIN)

5.1. Random generation of allocation (assignment) to groups (use of computer-generated random numbers, table of random numbers, drawing lots or envelopes, coin tossing, shuffling cards, or throwing dice) o Yes o Unclear (e.g., random assignment is mentioned, but not in detail) o No (allocation by some method other than random assignment) 5.2. Allocation concealment (participants and investigators cannot foresee assignment; e.g., central randomization performed at site remote from trial location or monitored use of sequentially numbered, sealed, opaque envelopes) o Yes o Unclear o No 5.3. Avoidance of performance bias (no treatment differences between groups other than the main intervention contrasts) o Yes o Unclear o No 5.4. Avoidance of attrition bias (losses to follow up less than or equal to 20% and equally distributed between comparison groups) o Yes for all outcomes o Yes for some outcomes o Unclear o No

16 Littell et al., MST review, data extraction forms, 19 February 2013 5.5. Avoidance of detection bias (assessor unaware of the assigned treatment when collecting outcome measures) o Yes for all outcomes o Yes for some outcomes o Unclear o No 5.6. Intention-to-treat (data analyzed according to assigned group whether or not assigned services were received/completed) o Yes for all outcomes o Yes for some outcomes o Unclear o No 5.7. Standardized observation periods (follow-up data were collected from each case at a fixed point in time after random assignment) o Yes for all outcomes o Yes for some outcomes o Unclear o No 5.8. Validated outcome measures (use of instruments with demonstrated reliability and validity in this sample or similar samples OR use of public agency administrative data, behavioral, or biologic measures) o Yes for all outcomes o Yes for some outcomes o Unclear o No 5.9. Free of selective reporting (any evidence of outcome measures collected but not reported? full reporting of outcomes regardless of direction and significance of results?) o Yes o Unclear o No

17 Littell et al., MST review, data extraction forms, 19 February 2013 5.10. Free of conflicts of interest (researchers or data collectors would not benefit if results favored MST OR the control group) o Yes o Unclear o No

18 Littell et al., MST review, data extraction forms, 19 February 2013