Using Real-World Data and Artificial Intelligence to Advance Health Services Research Summary Background The field of health services research (HSR) can capitalize on In recent years, advances in computing power have enabled re- burgeoning sources of real-world data to parse new and perennial searchers to leverage complex, large, and novel data sources to reveal questions about health care costs, quality, and access, as well as new insights for decision-makers. While data science has spread into potentially increase the timeliness and relevance of research find- many sectors, HSR has been relatively slow to incorporate new data ings. But first, the field must prepare the infrastructure, including sources and analytics into the field’s methodological toolbox. academia, research funding, and peer-reviewed publications, to deliver on the promise and avoid the pitfalls of greater and more so- The AcademyHealth Paradigm Project, supported by the Robert phisticated use of real-world data. While researchers have long used Wood Johnson Foundation, is a concerted, collaborative effort structured real-world data like claims to answer questions in policy to increase the relevance, timeliness, quality, and impact of HSR and practice, myriad new unstructured data sources, including free through innovation.1 AcademyHealth, through the Paradigm text in electronic health records (EHRs) and images from X-rays Project, convened a February 2021 meeting to explore using real- and other technologies, are emerging for exploration. Similarly, world data—also sometimes referred to as “big data”—and related new methodologies, such as machine learning and natural language artificial intelligence methods, such as machine learning and natu- processing, are being applied to real-world data to gain deeper ral language processing, to enhance HSR capabilities to improve insights about which care is right for which patients. This brief health and health care. Over the course of two afternoons, a group summarizes key points from a February 2021 meeting convened by of health services researchers and data experts discussed how AcademyHealth to examine greater use of real-world data in HSR real-world data and evidence can complement traditional HSR ap- and related issues, including safeguarding against the introduction proaches to identify and answer questions relevant to health policy of racial and other biases; addressing privacy concerns; establish- and practice. Among the topics explored, participants discussed: ing data standards; developing data resources as public goods; and helping researchers gain needed skills to design and conduct studies • Using nontraditional data sources and artificial intelligence meth- and interpret and disseminate findings. ods alongside traditional HSR approaches to answer questions related to health care costs, quality, and access. Genesis of this Brief: This brief is based on a meeting of researchers and research users that took place virtually on February 24-25, 2021. AcademyHealth convened the meeting as part of its Paradigm Project, a concerted, collaborative effort to increase the relevance, timeliness, quality, and impact of health services research (HSR). Funded by the Robert Wood Johnson Foundation, the project is ideating and testing new ways to ensure HSR realizes its full potential to improve health and the delivery of health care. The Paradigm Project is designed to push HSR out of its comfort zone—to ask what works now, what doesn’t, and what might work in the future. Additional information may be found on the project’s website at https://academyhealth.org/ParadigmProject. Using Real-World Data and Artificial Intelligence to Advance Health Services Research • Tapping real-world data sources to untangle the causal inference prove access,” a participant said, adding that such analyses can help of policy and practice interventions on heterogeneous subgroups. counter “policymakers routinely saying to health services research- ers, we’re too slow.” • Preparing the HSR infrastructure, including academia, research funding, and peer-reviewed publications, to capitalize on the Volume. Velocity. Variety. Veracity. Value. promise and avoid the pitfalls of real-world data. The so-called five Vs offer a helpful context to grasp the concept of big data.4 As the volume, velocity, and variety of real-world data • Applying research findings to answer real-world policy and prac- increase—fed by the exponential growth of digital information tice questions related to improving health and health care. generated in the connected world we live in—ensuring the veracity This brief summarizes the February meeting discussion, including and unlocking the value of real-world health data falls squarely in using real-world data and analytics to move beyond average effects; the HSR wheelhouse. While the field has long used structured real- safeguarding against the introduction of racial, ethnic, and other world data like claims to answer research questions, myriad new biases in real-world data analysis; addressing privacy concerns; es- unstructured data sources, including free text in EHRs and images tablishing standards; developing data resources as public goods; and from X-rays and other technologies, are emerging for exploration. helping researchers gain needed skills to design and conduct studies Similarly, new methodologies, such as machine learning and natu- and interpret and disseminate findings. The brief also examines the ral language processing, are being applied to real-world data to gain implications of greater use of real-world data for research funders, deeper insights beyond traditional randomized controlled trials. academia, peer-reviewed journals, and other aspects of the HSR ecosystem, including AcademyHealth. Because the session was off A form of artificial intelligence, machine learning essentially the record, the brief conveys the general content of the meeting enables computers to learn and adapt by analyzing and drawing without attributing specific comments to particular participants. inferences from patterns in large datasets. Algorithms, or the step- The discussion was informed by existing research though neither by-step rules used in problem-solving calculations, are the fuel of the discussion nor this brief incorporates a systematic review of the machine learning. Similarly, natural language processing, or NLP, literature related to using real-world data in HSR. A bibliography of is another form of artificial intelligence that “helps computers un- some relevant, current literature is included at the end of the brief. derstand, interpret and manipulate human language.”5 An offspring of linguistics, NLP enables computer software not only to read text Real-World Data in Health Care and hear speech but interpret language, measure sentiment, and The U.S. Food and Drug Administration defines real-world data determine importance. as “data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.”2 Along Moving Beyond Average Effects to Precision HSR with more traditional retrospective observational and survey data, Increasingly, researchers are using real-world data and machine real-world data can include information from EHRs, administrative learning to move beyond average effects captured by random- and claims data, registries, patient-reported outcomes and wearable ized controlled trials. The goal is to pin down causal inference for sensors, measures of social determinants of health, environmental subgroups that may respond differently to new drugs and medical exposures, and even clicks on a webpage, tweets, and geolocation devices post market, or perhaps even more importantly, identify- data from smartphones and other mobile devices. Common users ing more generally across practice which tests and medical proce- of real-world data and related research include pharmaceutical dures work best for which patients, according to participants at the companies, payers and purchasers, providers, policymakers, and Paradigm Project gathering. Similarly, recent studies using machine patients.3 learning to analyze large complex datasets have pinpointed patient- level differences related to physicians ordering low-value care and In one example of how real-world data may increase the timeliness the impact of increased patient cost sharing for prescription drugs. of HSR findings for policymakers, researchers are using geoloca- tion data from 45 million smartphones to understand health care In the first study, researchers created algorithmic predictions of utilization and social mobility during the COVID-19 pandemic the results of testing patients in emergency departments for heart by examining people’s visits to hospitals, physician offices, dental attacks. Using traditional analytic approaches, the testing on average offices, and other health care sites. “This dataset has allowed us to appeared to be cost-effective. But the analysis powered by machine look at these utilization patterns and see how they’re changing in learning that accounted for more granular patient-level differences real time, such that policymakers may be able to make closer to found that almost half of the tests should never have been ordered, real-time adjustments, whether in payment policy or trying to im- and even more importantly, many patients who should have been 2 Using Real-World Data and Artificial Intelligence to Advance Health Services Research tested were not.6 In the second study, researchers