White paper in Value-Based Healthcare: Forecasting Risk, Utilization, and Outcomes Delivering effective value-based healthcare requires identifying Consider a model predicting the chance of a hospital and mitigating risk by anticipating and preventing adverse readmission. The input training dataset might contain events and outcomes. Predictive models have been a part of thousands of hospitalization events. Each hospital admission healthcare practice for decades. However, more advanced would include the information on a patient’s medical history, analytics have started to take shape to provide better visibility like chronic conditions and prior utilization. The dataset into characterizing a patient’s current state and future risk. would also contain the “answer” as to whether that particular With the use of , it is possible to build models around hospitalization resulted in a readmission within 30 days. predicting future events and outcomes, utilization, and overall Data scientists typically use specialized (or write risk. These predictive models can be: their own) to build models that “learn” from the results of a – incorporated into a clinical workflow to facilitate care readmission to improve predictive accuracy. The model is then management and identify individuals at risk evaluated using the testing dataset, where a similar set of data – used to perform risk adjustment on quality measures is provided without the “answer” and the model’s prediction to account for patient severity is compared to the correct result (whether the patient was – employed to understand the treatment pathway with truly readmitted or not). The accuracy of a model on a testing the greatest chance of success dataset is typically what is presented as the true performance Predictive models have many potential uses, but no of the model. single risk score serves as a magic bullet to answer all To indicate the accuracy of a given method, model questions regarding a patient or a population. It is important performance can be reported in a variety of ways, including: to understand what question a risk score is addressing to – a classification rate (the number of true predictions over determine the circumstances in which a predictive model all predictions) should be used. – sensitivity/specificity (rates of correct true positive/ negative predictions) Predictive model overview – R2 (the amount of output variability accounted for A traditional predictive model is a mathematical formula that by the model) uses historical information to predict the chance of an event – Mean Absolute Percent Error (MAPE) (the overall percent happening in the future. Most predictive models are built to difference of the predicted value from the true value) address a population. The models are built based on a training – Area Under the Curve (AUC) (a measurement of dataset (a set of patients where the input factors are defined performance accuracy allowing the user to see the and the resulting output is known). The model is then evaluated tradeoffs between true and false positive rates under to determine its performance using a test dataset (a separate different conditions) sample of inputs where the result is blinded from the system Predictive models can be designed to apply to individual and only used at the end of the process to determine the patients or to a population as a whole. However, the uncertainty prediction’s accuracy). of predicting events for individuals or small populations has a higher level of uncertainty or variability associated with it than when predictions are averaged across a larger population. The resulting accuracy of individual models is typically lower when compared to the prediction rate across a larger population.

Watson Health Predictive Analytics in Value-Based Healthcare: Page 2 Forecasting Risk, Utilization, and Outcomes Predictive models in healthcare practice Event prediction Predictive models have been used in healthcare decision- Several areas of predictive models are in use today. The making for decades. Perhaps the most well-known predictive first major use case is in predicting events and outcomes. model in healthcare comes from the Framingham Heart In healthcare, that typically relates to identifying the chance Study1. The study began in 1958 to track the progression of of unwanted events occurring such as avoidable healthcare cardiovascular disease in more than 5,000 subjects. The study utilization, specific clinical conditions and their progression, is still ongoing and resulted in several predictive models across and outcomes such as mortality. Examples of avoidable several diseases. The most commonly used model takes six utilizations include a hospital admission or emergency data points (age, diabetes, smoking status, systolic blood department (ED) visit within the next 30 days or hospital pressure, total cholesterol, and HDL cholesterol) and predicts readmissions to the hospital within 30 days of discharge. the probability that someone will develop cardiovascular Disease prediction can identify individuals with a higher than disease in the next 10 years. The Framingham Study is now average chance of developing a new chronic condition (e.g., in its third generation of participants and has led to a creation Framingham Risk Model to predict the onset of diabetes in the of family of risk models, including atrial fibrillation, congestive next 8 years). Disease progression is used to determine how heart failure, coronary heart disease, diabetes, hypertension, a condition might advance over time (e.g., the chance of that and stroke. diabetic progressing from chronic kidney disease to end stage Not all predictive models require a prospective study. renal disease requiring dialysis). Outcomes prediction can The modern age of big data analytics provides access to a assess the chance of any number of future events, from the wealth of deep clinical and administrative data to study large chance of complication after surgery to mortality within populations of individuals with significant longitudinal history. the next year. Paired with advanced data analysis methods, such as machine The goal of event prediction is to manage a population learning and cognitive analytics, it is possible to develop such that as many mitigatable, undesirable outcomes are new predictive models with a statistical power and scale that avoided as possible. For example, a care team might evaluate is unprecedented. Modern technologies allow for the initial the readmission risk for any patient currently admitted. That development of such models within hours. The ubiquitous use score could help inform the team to decide whether a patient is of electronic medical records and care management platforms kept in the hospital longer, allowed to go home with appropriate allow for the deployment of models, once validated, to be used follow-up care, or safe to be discharged without any extra in tracking and managing patient care. assistance. A risk score alone cannot determine the best course of action or treatment. However, a risk score can be a valuable component of the picture of a patient’s overall health to assist providers in making care decisions. Predicting these events and then avoiding them can have a profound impact on the overall value of healthcare.

Watson Health Predictive Analytics in Value-Based Healthcare: Page 3 Forecasting Risk, Utilization, and Outcomes Utilization and Risk Prediction An example of a more complex utilization prediction One of the most common forms of risk prediction involves is the IBM Explorys Risk Model4. Instead of relying identifying the relative financial risk of an individual within a on only demographics and diagnoses, the IBM Explorys population of patients in an actuarial, meaningful way. Typically, Risk Model also considers procedures, medications, and the healthcare spend of an individual is directly tied to his or prior healthcare costs to predict prospective utilization. The her condition, utilization, and overall risk. Utilization models are additional input factors increase the overall performance split into concurrent and prospective prediction. Concurrent compared to models using diagnoses and demographics models evaluate an individual’s risk and utilization for the alone. The IBM Explorys Risk Model is a good example of how current year. Prospective models evaluate an individual’s risk organizations can increase predictive performance of a model and utilization for the upcoming year. Naturally, the predictive by adding additional input features. power lessens the farther you attempt to predict in the future. Still, prospective models offer insight into how an organization Risk Adjustment should manage a population and where to place resources. One of the largest challenges in accurately reporting quality Several cost and utilization models exist in both the metrics is taking into account the severity of the population public domain and commercial vendor spaces. For example, being measured. Risk adjustment in healthcare is a means the Centers for Medicare & Medicaid Services (CMS) utilizes of using an individual’s severity, risk, or burden to normalize a model called the Hierarchical Condition Category (HCC) a reported outcome or quality metric. A simple example is a to provide prospective utilization calculations for individuals provider’s patient panel. The number of individuals being cared enrolled in a Medicare Advantage plan that are used in for indicates the general effort being spent by a provider on determining reimbursement2. The HCC score utilizes the delivering care. However, a straight panel count comparison patient’s demographics and diagnoses for the current year is not effective because not all patients require the same to predict the expected utilization for the following year3. HCC level of care. Instead, a utilization prediction score could be scores are normalized such that a score of 1.0 indicates that used to adjust the population count. Consider two providers an individual is expected to cost the average amount for that that each care for 1,000 patients. Provider A’s normalized population. A score of 1.5 indicates that an individual will cost population average utilization risk score is 1 while Provider B’s 50% more than average and a score of 3 indicates that an is 2. Provider B is expected to spend twice the resources of individual will cost three times the average. HCC is a good Provider A and is effectively treating 2,000 patients. example of why appropriate documentation and subsequent coding of conditions are important for both risk management and reimbursement. Failure to document a condition on a bill or claim could lead to underestimating the risk of an individual and result in lower reimbursement.

Watson Health Predictive Analytics in Value-Based Healthcare: Page 4 Forecasting Risk, Utilization, and Outcomes Risk adjustment is most commonly used to adjust Next Generation Analytics: utilization rates, including hospital admissions, ED visits, and Cognitive and readmissions. Naturally, the chance of a hospital readmission Predictive model development has seen a shift from relying increases greatly when an individual is sicker. Therefore, on traditional statistical techniques to utilizing new techniques adjusted counts provide a more apples-to-apples comparison and technologies in , deep learning, and when looking across different populations, providers, and cognitive computing. These newer technologies provide healthcare systems. For example, healthcare system A might improved predictive power by using significantly more data have a heart failure readmission rate of 30% while healthcare within a flexible modeling environment that can learn and adapt. system B has a rate of 15%. It might appear as if healthcare Where traditional regression models typically use 10-20 input system B is doing twice as well as system A. However, if variables across a fixed regression equation, machine learning healthcare system A is a tertiary care center, it may be models can be built using hundreds or thousands of inputs in dealing with individuals who are more likely to experience a dynamically generated model. Cognitive and other learning complications. Healthcare system B might be dealing with technologies expand the types of data available for modeling, more traditional treatment cases. After risk adjusting for patient including text from multiple sources, wearable devices, and severity, the risk adjusted readmission rate of both healthcare genetic data. Cognitive learning technologies extend predictive systems might be closer to 20% and their performance is power by introducing information context and reasoning to the effectively equivalent. information being assessed, to link knowledge and insights in Risk adjustment makes less sense when evaluating the process of model development. Cognitive models build and process metrics, such as weight screening and smoking improve upon a base set of knowledge by continually generating cessation counseling. The severity of a patient’s condition new insights and learnings in an ever-evolving system. should not have a direct effect on standard care processes. A downside of advanced predictive models is that their Additional factors can affect patient compliance, but they should complexity can present interpretation challenges to the end be accounted for in the measure definition, including patients at user. The advanced models can sometimes be viewed as a the end of life, in hospice, etc. More advanced risk adjustment black box and can lead to challenges in model adoption due and stratification can take into account patient features such as to a lack of understanding. One resolution is to simplify the socioeconomic status, disability status, accessibility of public model to the point that it can be clearly understood (often transportation, and adequate insurance coverage. However, at the sacrifice of model performance). Alternatively, well- most commonly used risk adjustment methodologies do not documented development and testing procedures can assist currently take these factors into consideration. others in understanding how a model was built and how it performs in real-world scenarios.

Watson Health Predictive Analytics in Value-Based Healthcare: Page 5 Forecasting Risk, Utilization, and Outcomes There are several major players in the next generation of Implementing Predictive Models predictive analytics. Many of these new solutions go beyond in the Clinical Workflow traditional predictions and provide solutions to solve real-world Predictive models can be used at different points in the care problems. Two of the most prominent contributors are management process to support benchmarking, drive quality and IBM. initiatives, and understand individual patient risk. Population- Google developed its deep learning technology, called based predictive models in benchmarking allow for a closer, DeepMind, which is a combination of machine learning and more accurate comparison between populations within and artificial intelligence5. Google successfully trained its AlphaGo across healthcare systems. For example, understanding the system to play the game Go and beat the European champion rate of readmissions considering patient risk and severity. Fan Hui in 20156. Google is expanding into the healthcare Similarly, population-based predictive models in quality space with DeepMind Health. Initial research projects involve management programs allow an organization to assess examining retinal scans to identify macular degeneration and a population as a whole to determine the overall risk and other eye-related conditions. expected utilization. Organizations can leverage individual- IBM has been involved in advanced analytics and artificial level predictive models to understand a single patient’s risk intelligence for decades. One of the first prominent contributions profile, including expected utilization and the probability of was IBM’s Deep Blue system for playing chess, which defeated adverse events occurring in the near future (e.g., readmission, world champion in 19977. In 2011, IBM unveiled hospitalization, complication). its Watson system of cognitive computing on the TV The greatest predictive models have value only if the Jeopardy!8. IBM’s cognitive computing is also an extension of information is reliable and can be acted upon. Actionable machine learning and where the system insights typically require timely data, analytics, and results that is able to comprehend, reason, and learn. IBM presents its are presented in a form that can be readily used. For example form of AI as “augmented intelligence,” as it is not expected to data or analyses that are a week or a month old often provide replace an individual’s decisions, only to inform them. Cognitive little value in establishing the risk of an individual currently technologies are at the center of IBM Watson Health to help admitted in the hospital. Similarly, a timely risk score on its inform decisions around multiple clinical areas, including own must have the proper context and workflow options to be oncology treatment, clinical trial matching, and drug discovery9. meaningful. The analytics need to be embedded in the care For example, Watson for Oncology examines an individual’s delivery and management workflow in order to be acted upon. DNA sequence from a healthy and a tumor cell to identify Changes in provider behavior and patient outcomes must be mutations. The cognitive system then examines the body of clearly measureable to track performance improvement and scientific literature to identify potential treatment pathways and the effectiveness of any analytics program. determine their likelihood of success. The results often include treatments normally administered for a different form of that may have not been traditionally considered10. For example, a patient with might be presented with a treatment option for stomach cancer because the individual happens to have a mutation found in several forms of stomach cancer. As the body of medical and scientific literature grows, so does the learning and reasoning power of Watson.

Watson Health Predictive Analytics in Value-Based Healthcare: Page 6 Forecasting Risk, Utilization, and Outcomes Conclusions Footnotes Healthcare is only beginning to see the potential of predictive 1. Framingham Heart Study. analytics. The power of predictive will increase https://www.framinghamheartstudy.org/ as more data and technology are available. Modern clinical datasets and analytic technologies present an enormous 2. Center for Medicare and Medicaid Services (CMS). Risk Adjustment. opportunity for new model development. The predictive power https://www.cms.gov/Medicare/Health- of a model developed on tens of millions of individuals across Plans/MedicareAdvtgSpecRateStats/ hundreds of input factors far exceeds traditional methods of Risk-Adjustors.html training a regression model of ten inputs across a sample of a 3. HCC University HCC Calculator. few thousand individuals. Big data technologies further empower http://www.hccuniversity.com/hcc- data scientists to quickly build and test models within a few university/risk-score-calculator/ hours instead of days or weeks with traditional techniques. 4. Rong Yi. “Evaluation of Explorys’ Risk Predictive algorithms are increasingly becoming a Adjustment and Predictive Models.” part of clinical healthcare practice. Clinical adoption will Milliman Client Report. 2014. grow as models gain in maturity, predictive power, and 5. Google DeepMind. usable applications. The results of predictive algorithms https://deepmind.com/ must be presented in a way that can be clearly understood. 6. Google DeepMind AlphaGo. The information presented must be actionable to drive an https://deepmind.com/alpha-go improvement in healthcare practice and made available to a practitioner in a timely manner. It will be necessary to provide 7. IBM Deep Blue. http://www-03.ibm. com/ibm/history/ibm100/us/en/icons/ true clinical value in order to gain differentiation and adoption. deepblue/ New systems will continue to be developed that help inform provider decisions and drive better outcomes. Predictive 8. IBM’s Watson computer takes the Jeopardy! challenge. https://www. models, whether traditional or cognitive, will be an expected ibm.com/midmarket/us/en/article_ component of care to provide the fullest, data-driven approach Smartercomm5_1209.html to a patient’s value-based health. 9. IBM Watson Health. http://www.ibm. com/watson/health/

10. IBM Watson Health Watson For Oncology. http://www.ibm.com/watson/ health/oncology/

Watson Health Predictive Analytics in Value-Based Healthcare: Page 7 Forecasting Risk, Utilization, and Outcomes © Copyright IBM Corporation 2017 IBM products are warranted according to the IBM Corporation terms and conditions of the agreements Route 100 under which they are provided. Somers, NY 10589 Statement of Good Security Practices: Produced in the IT system security involves protecting of America, January, 2017 systems and -mation through prevention, detection and response IBM, the IBM logo, ibm.com, and Watson to improper access from within and are trademarks of International Business outside your enterprise. Improper Machines Corp., registered in many access can result in information being jurisdictions worldwide. Other product altered, destroyed, misappropriated or and service names might be trademarks misused or can result in damage to or of IBM or other companies. A current list misuse of your systems, including for use of IBM trademarks is available on the Web in attacks on others. No IT system or at “Copyright and trademark information” product should be considered completely at www.ibm.com/legal/copytrade.shtml secure and no single product, service or security measure can be completely This document is current as of the initial effective in preventing improper use date of publication and may be changed or access. IBM systems, products by IBM at any time. Not all offerings are and services are designed to be part available in every country in which IBM of a lawful, comprehensive security operates. approach, which will necessarily involve additional operational procedures, and PROVIDED “AS IS” WITHOUT may require other systems, products or ANY WARRANTY, EXPRESS OR services to be most effective. IBM DOES IMPLIED, INCLUDING WITHOUT ANY NOT WARRANT THAT ANY SYSTEMS, WARRANTIES OF MERCHANTABILITY, PRODUCTS OR SERVICES ARE FITNESS FOR A PARTICULAR IMMUNE FROM, OR WILL MAKE YOUR PURPOSE AND ANY WARRANTY OR ENTERPRISE IMMUNE FROM, THE CONDITION OF NON-INFRINGEMENT. MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY. HLW03032-USEN-00