Considerations for Predictive Modeling in Insurance Applications May 2019 2 Considerations for Predictive Modeling in Insurance Applications AUTHORS Eileen Burns, FSA, MAAA SPONSORS Modeling Section Gene Dan, FCAS, MAAA, CSPA Predictive Analytics and Futurism Anders Larson, FSA, MAAA Section Bob Meyer, FCAS, MAAA Committee on Life Insurance Research Zohair Motiwalla, FSA, MAAA Product Development Section Guy Yollin Reinsurance Section Milliman Caveat and Disclaimer The opinions expressed and conclusions reached by the authors are their own and do not represent any official position or opinion of the Society of Actuaries or its members. The Society of Actuaries makes no representation or warranty to the accuracy of the information Copyright © 2019 by the Society of Actuaries. All rights reserved. Copyright © 2019 Society of Actuaries 3 CONTENTS Acknowledgments .................................................................................................................................................... 5 Section 1: Introduction ............................................................................................................................................. 6 Section 2: Literature Review ..................................................................................................................................... 8 2.1 BACKGROUND ...................................................................................................................................................... 8 2.2 PROJECT OBJECTIVE ............................................................................................................................................ 9 2.3 DATA ACQUISITION AND PREPARATION .......................................................................................................... 10 2.4 ALGORITHM SELECTION .................................................................................................................................... 12 2.5 FEATURE ENGINEERING AND SELECTION ........................................................................................................ 14 2.6 MODEL EVALUATION AND MEASURES OF SUCCESS ....................................................................................... 16 2.7 MODEL DEPLOYMENT ....................................................................................................................................... 18 2.8 MODEL GOVERNANCE ....................................................................................................................................... 19 2.9 SOFTWARE SELECTION ...................................................................................................................................... 21 Section 3: Predictive Analytics Considerations ........................................................................................................ 25 3.1 PROJECT OBJECTIVE .......................................................................................................................................... 25 3.2 DATA ACQUISITION AND PREPARATION .......................................................................................................... 26 3.3 ALGORITHM SELECTION .................................................................................................................................... 28 3.4 FEATURE ENGINEERING AND SELECTION ........................................................................................................ 28 3.5 MODEL EVALUATION AND MEASURES OF SUCCESS ....................................................................................... 30 3.6 MODEL DEPLOYMENT ....................................................................................................................................... 31 3.7 MODEL GOVERNANCE ....................................................................................................................................... 32 3.8 SOFTWARE SELECTION ...................................................................................................................................... 34 3.9 STAYING CURRENT ............................................................................................................................................ 35 Section 4: Case Study .............................................................................................................................................. 37 4.1 CASE STUDY BACKGROUND .............................................................................................................................. 37 4.2 PROJECT OBJECTIVE .......................................................................................................................................... 38 4.2.1 COMMENTARY ....................................................................................................................................... 39 4.3 DATA ACQUISITION AND PREPARATION .......................................................................................................... 39 4.3.1 DATA SOURCES ....................................................................................................................................... 39 4.3.2 DATA RECONCILIATION .......................................................................................................................... 40 4.3.3 COMMENTARY ....................................................................................................................................... 40 4.4 ALGORITHM SELECTION .................................................................................................................................... 40 4.4.1 CANDIDATE ALGORITHMS ..................................................................................................................... 40 4.4.2 CONCEPTUAL DESIGN AND ALGORITHM SELECTION ........................................................................... 41 4.4.3 COMMENTARY ....................................................................................................................................... 43 4.5 SOFTWARE SELECTION ...................................................................................................................................... 43 4.5.1 PROJECT STRUCTURE ............................................................................................................................. 44 4.5.2 COMMENTARY ....................................................................................................................................... 46 4.6 FEATURE ENGINEERING AND SELECTION ........................................................................................................ 46 4.6.1 COMMENTARY ....................................................................................................................................... 50 4.7 MODEL EVALUATION AND MEASURES OF SUCCESS ....................................................................................... 50 4.7.1 ACTUAL VERSUS EXPECTED ANALYSIS .................................................................................................. 50 4.7.2 PRECISON ................................................................................................................................................ 55 4.7.3 COMMENTARY ....................................................................................................................................... 59 4.8 MODEL DEPLOYMENT ....................................................................................................................................... 60 4.8.1 COMMENTARY ....................................................................................................................................... 61 4.9 MODEL GOVERNANCE ....................................................................................................................................... 61 4.9.1 ORGANIZATIONAL STRUCTURE ............................................................................................................. 61 Copyright © 2019 Society of Actuaries 4 4.9.2 MODEL INVENTORY ............................................................................................................................... 62 4.9.3 VERSION CONTROL PRACTICES ............................................................................................................. 62 4.9.4 INDEPENDENT REVIEW OF MODELS ..................................................................................................... 63 4.9.5 DOCUMENTATION PRACTICES .............................................................................................................. 63 4.9.6 COMMENTARY ....................................................................................................................................... 63 Appendix 1: Survey Results ..................................................................................................................................... 64 DEMOGRAPHICS........................................................................................................................................................... 64 BUSINESS PURPOSE ..................................................................................................................................................... 68 DATA ACQUISITION AND PREPARATION ..................................................................................................................... 71 ALGORITHM SELECTION