medRxiv preprint doi: https://doi.org/10.1101/2020.11.11.20220962; this version posted December 4, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license . Short-term forecasts to inform the response to the Covid-19 epidemic in the UK 1* 1 2 3 4,5 6,7 6 1 8 Funk S ,​ Abbott S ,​ Atkins BD ,​ Baguelin M ,​ Baillie JK ,​ Birrell P ,​ Blake J ,​ Bosse NI ,​ Burton J ,​ ​ 7 ​ 1 ​ 6,7 ​ 2 ​ ​ 1 ​ 1 ​ 3 ​ Carruthers J ,​ Davies NG ,​ De Angelis D ,​ Dyson L ,​ Edmunds WJ ,​ Eggo RM ,​ Ferguson NM ,​ ​ 3 2​ ​ 2 ​ 1 ​ 2 ​ 2 9 ​ 10 Gaythorpe K ,​ Gorsich E ,​ Guyver-Fletcher G ,​ Hellewell J ,​ Hill EM ,​ Holmes A ,​ House TA ,​ Jewell C ,​ 1 ​ 1 ​ 11 2 ​ 12 ​ 3 ​ 1 ​ ​13 ​ Jit M ,​ Jombart T ,​ Joshi I ,​ Keeling MJ ,​ Kendall E ,​ Knock ES ,​ Kucharski AJ ,​ Lythgoe KA ,​ Meakin 1 ​ 1 ​ ​ 14 ​ 9 ​ 9 ​ 11 ​ ​ 3 9,15 SR ,​ Munday JD ,​ Openshaw PJM ,​ Overton CE ,​ Pagani F ,​ Pearson J ,​ Perez-Guzman PN ,​ Pellis L ​, ​ 16 ​ 17,18 ​ 1 ​ 12 ​ 2 ​ 7 ​ 3,7 ​ Scarabel F ,​ Semple MG ,​ Sherratt K ,​ Tang M ,​ Tildesley MJ ,​ Van Leeuwen E ,​ Whittles LK ,​ ​ ​ ​ ​ ​ ​ ​ CMMID COVID-19 Working Group, Imperial College COVID-19 Response Team, ISARIC4C Investigators 1 Centre​ for Mathematical Modelling of Infectious Diseases, London School of Hygiene & Tropical Medicine, Keppel St, London, WC1E 7HT, UK 2 The​ Zeeman Institute for Systems Biology & Infectious Disease Epidemiology Research, School of Life Sciences and Mathematics Institute, University of Warwick, Coventry, CV4 7AL, UK 3 MRC​ Centre for Global Infectious Disease Analysis, Jameel Institute for Disease and Emergency Analytics, Imperial College London, London, W2 1PG, UK 4 ​ Roslin Institute, University of Edinburgh, Edinburgh, EH25 9RG, UK 5 ​ Intensive Care Unit, Royal Infirmary Edinburgh, Edinburgh, UK 6 MRC​ Biostatistics Unit, University of Cambridge, Forvie Site, Robinson Way, Cambridge Biomedical Campus , Cambridge, CB2 0SR, UK 7 National​ Infection Service, Public Health England, 61 Colindale Avenue, London, NW9 5EQ, UK 8 Division​ of Informatics, Imaging and Data Sciences, Faculty of Biology Medicine and Health,The University of Manchester, Oxford Road, Manchester, M13 9PT, UK 9 Department​ of Mathematics, University of Manchester, Oxford Road, Manchester M13 9PL, UK 10 Centre​ for Health Informatics, Computing and Statistics, Lancaster Medical School, Lancaster University, Lancaster LA1 4YG, UK 11 NHSX,​ Skipton House, 80 London Road, London, SE1 6LH, UK 12 NHS​ England & NHSE Improvement, Quarry House, Quarry Hill, Leeds LS2 7UE, UK 13 Big​ Data Institute, University of Oxford, Old Road Campus, Oxford, OX3 7LF 14 ​ National Heart and Lung Institute, Imperial College London, London, SW3 6LY, UK 15 The​ Alan Turing Institute, British Library, 96 Euston Road, London, NW1 2DB 16 Laboratory​ for Industrial and Applied Mathematics, York University, Toronto, ON, M3J 1P3, Canada 17 ​ NIHR Health Protection Research Unit in Emerging and Zoonotic Infections an Institute of Infection, Veterinary and Ecological Sciences, Faculty of Health and Life Sciences, University of Liverpool, Liverpool, L69 7TX, UK 18 ​ Respiratory Medicine, Alder Hey Children’s Hospital, Liverpool L12 2AP, UK * [email protected] ​ NOTE: This preprint reports new research that has not been certified by peer review and should not be used to guide clinical practice. medRxiv preprint doi: https://doi.org/10.1101/2020.11.11.20220962; this version posted December 4, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license . Abstract Background: Short-term forecasts of infectious disease can aid situational awareness and planning for outbreak response. Here, we report on multi-model forecasts of Covid-19 in the UK that were generated at regular intervals starting at the end of March 2020, in order to monitor expected healthcare utilisation and population impacts in real time. Methods: We evaluated the performance of individual model forecasts generated between 24 March and 14 July 2020, using a variety of metrics including the weighted interval score as well as metrics that assess the calibration, sharpness, bias and absolute error of forecasts separately. We further combined the predictions from individual models into ensemble forecasts using a simple mean as well as a quantile regression average that aimed to maximise performance. We compared model performance to a null model of no change. Results: In most cases, individual models performed better than the null model, and ensembles models were well calibrated and performed comparatively to the best individual models. The quantile regression average did not noticeably outperform the mean ensemble. Conclusions: Ensembles of multi-model forecasts can inform the policy response to the Covid-19 pandemic by assessing future resource needs and expected population impact of morbidity and mortality. Introduction Since the first confirmation of a case on 31 January 2020, the Covid-19 epidemic in the UK has caused a large burden of morbidity and mortality. Following a rapid increase in cases throughout 1 February and March, triggered by repeated introduction and subsequent local transmission ,​ the UK ​ population was advised on 16 March to avoid non-essential travel and contact with others, and to work from home if possible. This advice became enforceable law a week later, which was followed by a decline in reported cases and deaths starting in the first half of April. During the same period, hospitals prepared for a rapid increase in seriously ill patients by maximising inpatient and critical 2 care capacity .​ ​ Short-term forecasts of infectious diseases are increasingly being used to inform public health policy 3 for a variety of diseases .​ Models for short-term forecasts can be statistical (investigating the ​ changing distribution of variables over time), mechanistic (explicitly incorporating plausible medRxiv preprint doi: https://doi.org/10.1101/2020.11.11.20220962; this version posted December 4, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license . biological and social mechanisms of transmission), or a hybrid of the two. While the best choice of models for short-term infectious disease forecasts is an ongoing topic of research, there is some evidence that introducing mechanistic assumptions does not necessarily improve short-term predictive performance compared to statistical models that have no specific assumptions related to 4 the disease transmission process .​ While short-term forecasts are most prominent for seasonal ​ influenza, more recently they have also been made for outbreaks such as Ebola, measles, Zika, and 5–9 diphtheria .​ Developing accurate and reliable short-term forecasts in real time for novel infectious ​ agents such as SARS-CoV-2 in early 2020 is particularly challenging because of uncertainty about 7,10 modes of transmission, severity profiles and other relevant parameters .​ ​ Here, we report on short-term forecasts of the Covid-19 epidemic produced by six groups in the UK, representing a mixture of academic and government institutions and using a variety of models and methods. A system for collating and aggregating these forecasts was set up through a short-term forecasting subgroup of the Scientific Pandemic Influenza Group on Modelling (SPI-M) on 23 March, 2020. The aim of these efforts was to predict the burden on the healthcare system, as well as key indicators of the current status of the epidemic faced by the UK at the time. Aggregate forecasts were made available to the Strategic Advisory Group of Experts (SAGE) and the UK government, marking the first time that a multitude of models for short-term forecasting models were explicitly combined to inform health policy in the UK. We review forecasts between the end of March and July and assess the quality of the predictions made at different times. Methods Targets and validation datasets Initially seven forecasting targets were set: 1) The number of intensive care unit (ICU) beds occupied by confirmed Covid-19 patients, 2) the total number of beds (including ICU) occupied by confirmed Covid-19 patients, 3) the total number of deaths by date of death, 4) the number of deaths in hospitals by date of report, 5) the number of new and newly admitted confirmed Covid-19 patients in hospital, 6) the number of new admissions to ICU and 7) the cumulative number of infections. Validation data sets for England and the seven English National Health Service (NHS) regions into which it is divided were derived from NHS England situational reports, and for the devolved nations in Northern Ireland, Scotland and Wales from their distinct reports and data sets. Because of differences in the structure of these reports and the data included in them, validation data sets started being used for the short-term forecasts at different times. Over time, this was reduced to four targets: the number of ICU beds and any beds occupied by confirmed Covid-19 patients, respectively; the number of new and newly admitted Covid-19 patients in hospital; and the number of deaths by date of death. Data sources and their interpretation changed at several points and definitions were updated as time progressed. Data sources that were not available at the time forecasts were made were excluded from the evaluation. medRxiv preprint doi: https://doi.org/10.1101/2020.11.11.20220962; this version posted December 4, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.