WK,(((,QWHUQDWLRQDO&RQIHUHQFHRQ&ROODERUDWLYH&RPSXWLQJ1HWZRUNLQJ$SSOLFDWLRQVDQG:RUNVKDULQJ &ROODERUDWH&RP Accurate Weather Forecasting Through Locality Based Collaborative Computing Bard˚ Fjukstad John Markus Bjørndalen Otto Anshus Department of Computer Science Department of Computer Science Department of Computer Science Faculty of Science and Technology Faculty of Science and Technology Faculty of Science and Technology University of Tromsø, Norway University of Tromsø, Norway University of Tromsø, Norway Email: [email protected] Email: [email protected] and the Norwegian Meteorological Institute Forecasting Division for Northern Norway Email: [email protected] Abstract—The Collaborative Symbiotic Weather Forecasting of collaboration is when users use forecasts from the national (CSWF) system lets a user compute a short time, high-resolution weather services to produce short-term, small area, and higher forecast for a small region around the user, in a few minutes, resolution forecasts. There is no feedback of these forecasts on-demand, on a PC. A collaborated forecast giving better uncertainty estimation is then created using forecasts from other from the users to the weather services. These forecasts take users in the same general region. A collaborated forecast can be minutes to compute on a single multi-core PC. The third is visualized on a range of devices and in a range of styles, typically a symbiotic collaboration where users share on-demand their as a composite of the individual forecasts. CSWF assumes locality locally produced forecasts with each other. between forecasts, regions, and PCs. Forecasts for a region are In a complex terrain like the fjords and mountains of Nor- computed by and stored on PCs located within the region. To locate forecasts, CSWF simply scans specific ports on public IP way, the topography have a significant impact on the weather addresses in the local area. Scanning is robust because it avoids on the very local scale. This represents a serious challenge maintaining state about others and fast because the number of for numerical models where the spatial resolution limits the computers is low and only a few forecasts are needed. ability to produce accurate forecasts. The weather services can Keywords: Weather Forecast, Distributed Computing, Col- increase the resolution to better reflect the topography. While laboration, Peer to Peer this is gradually happening, it is still primarily done for regions of special interest, like airports. This is because the compute I. INTRODUCTION resources applied are not sufficient to do a timely delivery Access to weather forecasts for practically any location of the forecasts for very large regions, let alone the whole on Earth is available free of charge over the Internet from of Earth. Many commercial weather services typically have meteorological services, like the Norwegian Meteorological the same lack of resolution because they repack the forecasts Institute Yr.no [1], the European Centre for Medium-Range from the national weather services. While they integrate this Weather Forecasts (ECMWF), and the US National Weather with other forms of weather related information few compute Service [2]. There are collaborations between the weather their own numerical atmospheric models, and if they do, this services in that lower resolution forecasts are used to compute is often for specialized purposes like wind mill farms for forecasts of higher resolution. For instance, the Norwegian premium customers, and not publicly available. Meteorological Institute uses the forecasts from ECMWF to Local weather forecasts have seen some industry attention. do higher resolution forecasts for Scandinavia. However, there IBM’s Deep Thunder project [3] has developed a system de- are no collaborations with users, and between users of weather livering targeted high-resolution weather forecasts for smaller forecasts. The resolution and accuracy of weather forecasts can areas. The intended use is limited in that the forecasting be increased if there is collaboration between national forecast typically is for a pre-determined fixed area and for a specific services and users, and between users. use like the 2014 World Cup in Rio. Three types of collaboration related to weather forecast- In summary, the accuracy of an actual forecast is a function ing can be identified. The first is the collaboration between of the characteristics of the models used to compute it, how national weather services. The national service can produce many models are computed, the resolution of the background forecasts that other services use as a starting point or as data, the size of the forecasted area, how far into the future boundary values in their own production of numerical fore- the forecast is for, and the time interval. The model area, casts. The forecasts at national level are medium to long-term, resolution and forecast interval must be determined in light of large area, and medium to high-resolution forecasts. They take the compute resources available, and the necessary hard wall several hours to compute on supercomputers. The second type clock deadlines before the numerical forecasts must be ready k,&67 '2,LFVWFROODERUDWHFRP for use. The national services do forecasts for larger regions II. COLLABORATIVE WEATHER FORECASTING and at lower resolutions. Commercial services do specific Modern weather forecasting is based on one of the most high-resolution forecasts for paying customers. successful international collaborative efforts [4]. Using ob- Even though forecasts are made available whenever a user servations and forecast products exchanged with a global 1 requires them, the organization around, as well as the ap- telecommunications networks, GTS , that predates the inter- proaches used to produce forecasts, lead to a situation where net, national weather services have access to all the back- they are pre-computed, instead of being done on-demand. ground information needed for both global, regional and local The weather services often strongly select the data they make weather forecasting. available, aiming at the most typical usage. Dedicated supercomputer clusters are used for running large numerical forecast models, and a very large storage This paper proposes a three-tier approach to producing fore- infrastructure is used for storing the forecasts and observa- casts. The first tier, the global forecasts, is the forecasts for tional background data. The size of this infrastructure can be large areas, low resolution, and long time periods produced by illustrated with the budgeted $ 23.7 million2 2013 update of the national weather services. These are typically computed as infrastructure and computing facilities to the National Weather parallel computations on a supercomputer or a compute cluster. Services, NWS, following the Sandy hurricane. The second tier, the local forecasts, is producing very high- It should be stressed that such very expensive systems are resolution forecasts for small areas and for short time periods indeed needed for providing forecasts for large areas. These using the tier 1 forecasts as a starting point. The computation systems provide the necessary background meteorological data is typically parallell and done on multi- and many-core modern for the personalized collaboration system described in this PCs. The PCs are located in private homes, and in private and paper. public offices. The geographical locations of these computers National weather services typically uses a client-server are typically in the area for which they produce a forecast. The model in the form of web-based systems for preparing and third tier, the collaborative symbiotic forecasts, is producing visualizing meteorological data, and for making available the amalgamated forecasts based on the local forecasts. This is raw datafiles. The user accesses the data through a regular done on and by the same computers used to produce the web browser or use local apps, which typically download local forecasts. The global and local forecasts are sufficient the data onto the device. One example of this model is the to produce very high-resolution forecasts that can be used NOAA Operational Model Archive and Distribution System as is. The symbiotic forecast is used to achieve better error (NOMADS) [5]. estimations of the forecasted weather, using multiple forecasts computed using a slightly different center position. III. COLLABORATIVE SYMBIOTIC WEATHER FORECASTING The usage scenario is comprised of users around the world The observation behind the Collaborative Symbiotic wanting to compute on-demand accurate and high-resolution Weather Forecasting (CSWF) model is that national and com- forecasts for a small area. They do so on their own PCs. Firstly, mercial weather forecasting services do not have the resources background data is pulled in, or is already pre-fetched, from a to offer high-resolution forecasts for arbitrary parts of the national service. Secondly, by restricting the forecast in space Earth selected on-demand by public users. Therefore, pre- and time, a high-resolution numerical forecast is produced in computed lower resolution forecasts are provided for large minutes using a professional model. To increase the accuracy areas, and higher resolution forecasts are created for pre- of the forecast, forecasts from other persons in and for the selected areas, like airports, and for paying customers. area are located and pulled in. Previous work [6] have shown that users can do their own This paper documents