PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE FINAL RESEARCH REPORT Testing a New Software Program for Data Abstraction in Systematic Reviews Tianjing Li, MD, MHS, PhD1; Ian J. Saldanha, MBBS, MPH, PhD2; Jens Jap, BA2; Joseph Canner, MHS3; Christopher H. Schmid, PhD4; on behalf of the Data Abstractor Assistant investigators 1Center for Clinical Trials and Evidence Synthesis, Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland 2Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Brown University School of Public Health, Providence, Rhode Island 3Center for Outcomes Research, Department of Surgery, Johns Hopkins School of Medicine, Baltimore, Maryland 4Center for Evidence Synthesis in Health, Department of Biostatistics, Brown University School of Public Health, Providence, Rhode Island Institution Receiving the PCORI Award: Johns Hopkins University Original Project Title: Develop, Test, and Disseminate a New Technology to Modernize Data Abstraction in Systematic Reviews PCORI ID: ME-1310-07009 HSRProj ID: HSRP20152269 _______________________________ To cite this document, please use: Li T, Saldanha IJ, Jap J, Canner J, Schmid CH; Data Abstractor Assistant (DAA) Investigators. (2020). Testing a New Software Program for Data Abstraction in Systematic Reviews. Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/04.2020.ME.131007009 TABLE OF CONTENTS ABSTRACT ............................................................................................................................. 4 BACKGROUND ....................................................................................................................... 6 Figure 1. Steps in completing a systematic reviewa ........................................................... 6 Specific Aims .............................................................................................................................. 8 PARTICIPATION OF PATIENTS AND OTHER STAKEHOLDERS .................................................. 10 Impact of Stakeholder Engagement on Project ....................................................................... 10 METHODS ........................................................................................................................... 12 Aim 1: Developing DAA ............................................................................................................ 12 Aim 2: Conducting a Randomized Controlled Trial to Evaluate DAA ....................................... 14 Table 1. Assignment of 24 Pairs of Data Abstractors to 6 Sequences and 48 Articlesa ............................................................................................................................. 17 Figure 2. Screenshot from the Baseline Tab of a data abstraction form used during the DAA trial .......................................................................................................... 20 Aim 3: Disseminating the Study Findings ................................................................................. 24 RESULTS .............................................................................................................................. 25 Aim 1: Developing DAA ............................................................................................................ 25 Figure 3. Screenshot showing how DAA displays the source document in HTML format (right) adjacent to the data abstraction form in the data abstraction system (SRDR, left)a........................................................................................................... 26 Aim 2: Conducting a Randomized Controlled Trial to Evaluate DAA ....................................... 27 Figure 4. Participant flow during the DAA trial ................................................................. 27 Table 2. Baseline Characteristics of All 52 Participants in the DAA Trial .......................... 28 Table 3. Baseline Characteristics of All 52 Participants in the DAA Trial by Level of Experience With Data Abstraction .................................................................................... 30 Table 4. Proportion of Errors by Data Abstraction Approach, Type of Error, Type of Data Item, and Systematic Review Topic ..................................................................... 33 Table 4. Proportion of Errors by Data Abstraction Approach, Type of Error, Type of Data Item, and Systematic Review Topic (cont’d) ........................................................ 34 Table 5. Proportion of Errors Across All Approaches, by Type of Error, Type of Data Abstracted, and Systematic Review Topic ............................................................... 36 Table 6. Between-Approach Comparisons of Error Proportions by Type of Data Abstracteda ....................................................................................................................... 37 Table 7. Auto-recorded Time Spent (in minutes) by Data Abstraction Approach, Type of Data Item, and Systematic Review Topic ............................................................. 39 2 Table 8. Self-recorded Time (in minutes) Spent by Data Abstraction Approach, Step of Data Abstraction, and Systematic Review Topic .................................................. 41 Table 9. Self-recorded Time (in minutes) Spent Across All Approaches, by Step of Data Abstraction and Systematic Review Topic ............................................................... 42 Table 10. Between-Approach Comparisons of Auto-recorded Time by Type of Data Abstracteda ............................................................................................................... 44 Table 11. Between-Approach Comparisons of Self-recorded Time Across All Topicsa ............................................................................................................................... 44 Aim 3. Disseminating the Study Findings ................................................................................. 46 Table 12. Considerations When Selecting Data Abstraction Approaches During Systematic Reviews ........................................................................................................... 47 DISCUSSION ........................................................................................................................ 49 Error Proportions Observed and Context for Study Results .................................................... 49 Differences in Error Proportions and Time Among Data Abstraction Approaches ................. 50 Possible Reasons for Higher Error Proportions With DAA ....................................................... 51 Subpopulation Considerations ................................................................................................. 51 Value of Using DAA and Implications for Future Research ...................................................... 51 Challenges With Independent Dual Data Abstraction Plus Adjudication ................................ 52 Implications and Uptake of Study Results ............................................................................... 53 Study Limitations and Strengths .............................................................................................. 53 CONCLUSIONS ..................................................................................................................... 55 REFERENCES ........................................................................................................................ 56 RELATED PUBLICATIONS ...................................................................................................... 59 In Preparation .......................................................................................................................... 59 Published .................................................................................................................................. 59 ACKNOWLEDGMENTS .......................................................................................................... 60 APPENDICES ........................................................................................................................ 61 Appendix 1: Published paper describing the technical details of DAA .................................... 61 Appendix 2: Survey instrument ................................................................................................ 75 Appendix 3: Summary of survey responses, by level of experience with data abstraction ............................................................................................................................... 78 Appendix 4: Published paper describing the DAA trial protocol ............................................. 80 3 ABSTRACT Background: When performing systematic reviews, data abstraction, a predominantly manual process, is labor intensive and error prone. Current standards for abstraction rest on a weak evidence base. Objectives: Aim 1. Develop Data Abstraction Assistant (DAA), a software tool to identify and track the location of data in articles and to automatically enter data into the Systematic Review Data Repository. Aim 2. Conduct a randomized controlled trial to evaluate the comparative effectiveness of 3 approaches—(A) DAA-facilitated single data abstraction plus verification, (B) single data abstraction plus verification, and (C) independent dual data abstraction plus adjudication—on