Bias-Variance Aware Integration of Judgmental Forecasts and Statistical Models
Total Page:16
File Type:pdf, Size:1020Kb
Bias–Variance Aware Integration of Judgmental Forecasts and Statistical Models Zur Erlangung des akademischen Grades eines Doktors der Ingenieurwissenschaften (Dr.-Ing.) von der Fakultät für Wirtschaftswissenschaften am Karlsruher Institut für Technologie (KIT) genehmigte DISSERTATION von M.Sc. Inform.-Wirt. Sebastian M. Blanc Tag der mündlichen Prüfung: 19.09.2016 Referent: Prof. Dr. Thomas Setzer Korreferent: Prof. Dr. Gerhard Satzger Karlsruhe, 2016 Acknowledgements I would like to express my special gratitude to my advisor Prof. Dr. Thomas Setzer for his great support and trust. He enabled me to pursue this research and motivated me to aim at a deeper understanding of the challenges. His enthusiasm has been an inspiration for my research as well as for my personal development. I would also like to acknowledge Prof. Dr. Gerhard Satzger, Prof. Dr. Christof Weinhardt, and Prof. Dr. Oliver Grothe, who served on my examination committee and who provided valuable comments on this thesis. My sincere thanks go to my colleagues in the research group Corporate Services and Systems at the Institute of Information Systems and Marketing and at the FZI Forschungszentrum Informatik. The coffee breaks, discussions, and all the other welcomed distractions greatly contributed to my motivation for writing this thesis. This work was accomplished in collaboration with the Bayer AG. I thank Bayer –especially Alexander Burck, Kati Schnürer, and the team of the corporate financial controlling– for the financial support and the fruitful cooperation as well as for providing their profound domain knowledge, expertise, and feedback on this research. Lastly, but most importantly, I thank Yasmin and my family. Yasmin continu- ously encouraged and supported me with loving care and patience. My family, especially my parents Peter and Karin, were always there for me and believed in me. Without them, this thesis would never have been written. Contents List of Figures vii List of Tables ix List of Abbreviations xi I Foundations1 1 Introduction and Motivation3 1.1 Motivation.................................4 1.2 Research Questions............................6 1.3 Structure of the Thesis.......................... 10 2 Integration of Judgmental Forecasts and Statistical Models 13 2.1 Judgmental Forecasting......................... 13 2.2 Integration Mechanisms......................... 16 2.2.1 Correction of Judgmental Forecasts.............. 17 2.2.2 Combination of Judgmental and Model-Based Forecasts.. 23 2.3 Statistical Learning Theory........................ 27 II Bias–Variance Aware Integration 31 3 Robust Forecast Correction 33 3.1 Model Estimation in Forecast Correction................ 34 3.2 Robustness in Forecast Correction................... 37 3.2.1 Training Sample Size....................... 38 3.2.2 Structural Change........................ 40 3.2.3 Non-Stationarity......................... 45 3.3 Illustration and Discussion........................ 46 3.3.1 Training Sample Size....................... 47 3.3.2 Structural Change........................ 48 3.3.3 Non-Stationarity......................... 52 3.4 Correction Considering Structural Change and Non-Stationarity.. 53 3.5 Conclusions and Limitations...................... 60 iii iv Contents 4 Advances in Forecast Combination 63 4.1 Theory and Issues of Forecast Combination.............. 64 4.2 Properties of Combination Methods.................. 68 4.2.1 Sampling Distribution of Weight Estimates.......... 69 4.2.2 Expected Combined Out-of-Sample Error Variance..... 72 4.3 Robustness in Forecast Combination.................. 74 4.3.1 Training Sample Size....................... 75 4.3.2 Error Covariance Change.................... 78 4.4 Combination Considering Bias–Variance and Structural Change.. 83 4.4.1 Optimal Bias–Variance Aware Shrinkage........... 84 4.4.2 Robust Shrinkage......................... 85 4.5 Illustration and Discussion........................ 86 4.5.1 Bias–Variance Trade-Off..................... 88 4.5.2 Training Sample Size....................... 90 4.5.3 Optimal Bias–Variance Aware Shrinkage........... 94 4.5.4 Error Covariance Change.................... 98 4.5.5 Robust Shrinkage......................... 105 4.6 Conclusions and Limitations...................... 112 III Application in Practice and Empirical Evaluation 115 5 Case Study: Corporate Cash Flow Forecasting 117 5.1 Sample Company............................. 118 5.2 Available Data and Preprocessing.................... 119 5.3 Data Characteristics and Classification................. 121 6 Empirical Evaluation 125 6.1 Research Design.............................. 125 6.1.1 Treatments Regarding Forecast Correction.......... 126 6.1.2 Calculation of Model-Based Forecasts............. 128 6.1.3 Treatments for Forecast Combination............. 130 6.2 Results on Forecast Correction...................... 131 6.2.1 Estimation Method........................ 132 6.2.2 Weighting of Past Observations................. 133 6.2.3 Approaches to Treating Structural Change.......... 136 6.2.4 Regression Analysis....................... 138 6.2.5 Discussion of Parameter Estimates............... 140 6.3 Results on Forecast Combination.................... 142 6.3.1 Two Model-Based Forecasts................... 143 6.3.2 One Judgmental and One Model-Based Forecast....... 145 6.3.3 One Judgmental and Two Model-Based Forecasts...... 147 6.3.4 Structural Change in Practice.................. 148 Contents v 6.4 Comparison of Correction and Combination............. 151 6.5 Conclusions and Limitations...................... 154 IV Finale 157 7 Conclusions and Outlook 159 7.1 Contribution................................ 160 7.2 Future Work................................ 164 V Appendix 169 A Proofs 171 References 193 List of Figures 1.1 Structure of the Thesis.......................... 12 2.1 Biases Considered in Theil’s Method.................. 19 2.2 Forecast-Actual Plots for Different Biases............... 21 2.3 In-Sample Combined Error Variance.................. 25 2.4 Visualization of the Bias–Variance Trade-Off............. 29 3.1 Minimal Training Sample Size...................... 48 3.2 Critical Change of Mean Error...................... 49 3.3 Critical Change of Correlation...................... 50 3.4 Critical Change of Forecast Variance.................. 51 3.5 Spurious Relationship in Forecast Correction............. 53 3.6 Weighting Schemes With a Structural Break.............. 54 4.1 Bias–Variance Trade-Off in Forecast Combination.......... 89 4.2 Minimal Training Sample Size in Forecast Combination....... 91 4.3 Minimal Training Sample Size in Forecast Combination with Two Forecasts.................................. 92 4.4 Results of Comparison to Schmittlein et al............... 94 4.5 Optimal Shrinkage Parameter...................... 95 4.6 Optimal Shrinkage for Two Forecasts.................. 96 4.7 Combined Error Variance for Optimal Shrinkage........... 97 4.8 Influence of Error Variance Change................... 99 4.9 Bivariate Error Correlation Change Thresholds............ 100 4.10 Bivariate Error Variance Change Thresholds.............. 101 4.11 Multi-Criteria Change Thresholds for Two Forecasts......... 102 4.12 Results of Comparison to Miller et al.................. 105 4.13 Distribution of Robust Shrinkage Parameter.............. 106 4.14 Distribution of Shrinked Weights with Robust Shrinkage...... 107 4.15 Combined Error Variance with Robust Shrinkage (Ordering by Ac- curacy)................................... 108 4.16 Combined Error Variance with Robust Shrinkage (Random Order- ing)..................................... 110 4.17 Error Correlation by Forecast...................... 111 5.1 Characteristics of the Available Time Series.............. 122 vii viii List of Figures 6.1 Correction Results by Estimation Method............... 133 6.2 Correction Results by Weighting Approach.............. 134 6.3 Correction Results with Fixed Weights................. 135 6.4 Correction Results with Breakpoint Treatment............ 137 6.5 Forecast Correction Model Parameter Estimates........... 140 6.6 Rolling Error Characteristic Estimates................. 149 6.7 Information Content of Rolling Estimates............... 150 List of Tables 4.1 Error Characteristics of the Forecasts of the M3 Competition.... 88 4.2 Treatments for the Comparison to Miller et al.............. 103 5.1 Delivery Structure of Forecasts..................... 119 5.2 Available Time Series in the Case Study................ 121 5.3 Results of Time Series Classification.................. 123 6.1 Evaluation Treatments Regarding Forecast Correction........ 127 6.2 Evaluation Treatments for Forecast Combination........... 131 6.3 Regression Analysis of Correction Results............... 139 6.4 Forecast Combination Results for DTES and ARIMA Forecasts... 144 6.5 Forecast Combination Results for Judgmental and ARIMA Forecasts146 6.6 Forecast Combination Results for Judgmental and DTES Forecasts. 146 6.7 Forecast Combination Results for Judgmental, DTES, and ARIMA Forecasts.................................. 147 6.8 Comparison of Forecast Correction and Combination Results.... 153 ix List of Abbreviations AIC Akaike Information Criterion AP Agricultural Products APE Absolute Percentage Error AR Autoregressive ARMA Autoregressive Moving Average ARIMA Autoregressive Integrated Moving Average BIC Bayesian Information Criterion CoV Coefficient of Variation DIV Diverse DJ Dow Jones DTES Damped Trend Exponential Smoothing HP Health and Pharmaceuticals IM Industrial Materials