Current State of Exploitation in AMS Community Eric B. Wendoloski, Timothy J. Hall, Kiley L. Yeakel, Peter J. Isaacson The Aerospace Corporation

Abstract Surge in Community AI Research/Interest Catalysts for Recent Surge Catalysts for Continuing Growth Continued… The environmental community has long produced a wealth of • Conference proceedings/recent publications suggest • Increased demand for hyperlocal forecasts • Scalable Cloud Computing; no need for local cluster mission specific observations, estimations, and simulations. Fusion rapid expansion in community AI capabilities • Renewable energy forecasting • Amazon Web Services (AWS) AMI of these sources traditionally occurs within numerical weather • 2015/16  2017: ~50% increase in conference material prediction frameworks through data-assimilation cycles that provide • Generic build of GPU-enabled deep-learning tools initial conditions to forecast models. Myriad environmental • 2015  2017: ~65% increase in published material. (TensorFlow, , Keras, , etc.) forecasting applications exist over all scales, but the environmental • Expansion accompanied by notable uptick in mature work Figure 5: Machine-learning-based • Best of both worlds programming languages community as whole has been slower to adopt the application of fusion of NWP and in-situ obs. better forecast solar irradiance • Julia –Python/R interactive style, speed of C++ artificial intelligence (AI) to these problem spaces than other during intermittent cloud cover • Python packages already ported (e.g., Scikit-Learn) industries (e.g., financial services, retail, etc.). However, a marked (Isaacson et al. 2016). increase in AI-based applications that leverage the wealth of data Physics is a Key Hurdle for Future Adoption available in the environmental sciences has been occurring over the last two years. This rapid increase in exploitation has been • Understanding the learned physics of AI-based models • Tree-based methods demonstrate variable importance manifesting itself as a jump in AI-related presentations and • Smart-phone age publications within the AMS community and increased utilization in • NN based applications tend to bury physics the operational meteorological domain. This presentation • Enhanced trust in machine-learning • Marzban and Viswanathan (2017) – intuitive example of • Higher data volume (e.g., GOES-16, JPSS-1) characterizes the increase in AI-based activity in AMS publications NN-learned logic gates. and identifies broad research areas that are reaching maturity • Ease of development using AI-based approaches. Additionally, this presentation Vectors Prime for Further AI Exploitation discusses the catalysts responsible for this increase in activity • Multi-scale automatic threat-area recognition along with research vectors that can benefit from AI-based data • Implications for forecaster-on-the-loop transition exploitation. (R-Project 2018) • Greater exploitation of lightning data including GOES GLM Catalysts for Continuing Growth • Nonlinear forecast calibration (enhanced MOS) Introduction • Enhanced Ease of Data Access • Environmental community provides huge volume of • Satellite Environmental Data Record (EDR) retrievals • Data assimilation with machine-learning basis environmental observations to the user community. Figure 1: Surge in peer-reviewed AMS articles containing AI-related • NOAA gathers > 20 TB of data per day NOAA OneStop Interface • Automated data source selection for problem domain content over last decade. Accompanying table lists most active NOAA • >>20 TB collected when next-gen satellite authors in AMS journals in this subject area. OneStop Amazon Web Services • Data extension to denied regions (i.e., marine environments) systems, international partner data, and private • Extreme heat waves/Long-term droughts & “flash” droughts Maturing Applications & Major Contributors industry sensor/IoT data considered Public Cloud Platform • Transportation forecasting with IoT backbone • Increased number of mature AI-based ideas Access • Water quality (harmful algal blooms, oil slick detection) • Fusing multi-source data to leverage the combined • Mature – products are/soon-to-be IBM NOAA • Seasonal land-falling Atlantic hurricanes “information” contained within is daunting operational Big Data Microsoft • Sea ice extent/navigability • Forecasters-in-the-loop lack time to deviate from Project trusted resources • Renewable Energy Figure 2: Decision-tree Structure Open Commons Consortium Useful Links • Fusion of measurements occurs in data assimilation • NCAR Boulder • AWS Deep Learning AMI: https://aws.amazon.com/amazon-ai/amis/ • Earth on AWS: https://aws.amazon.com/earth/ • Innumerable forecasting applications exist where • Often includes application of multi-source data fusion could yield valuable Key Datasets Hosted By Cloud Providers • Google Big Query: https://cloud.google.com/bigquery/public-data/ tree-based methods, • Google Earth Engine: https://earthengine.google.com/datasets/ Earth on AWS Google EarthEngine/BigQuery information but variety employed • IBM NOAA Earth Systems Data Portal (NESDP): https://noaa-crada.mybluemix.net/ Figure 3: Neural-Network Structure GOES NOAA GHCN • Julia Programming Language: https://julialang.org/ (e.g., Haupt et al. 2017) NEXRAD NOAA GSOD • NOAA OneStop: https://www.ncdc.noaa.gov/onestop/ ; Kearns, E., 2017: Facilitating Lag in AI Application to Weather Topics? new opportunities for data users via NOAA’s Big Data Project. Accessed 14 November UKMET MOGREPS NWP NOAA ICOADS – surface marine • Environmental community has lagged behind other 2017, http://www.nsc2017.org/wp- industries in using machine-learning/data-analytics- • Severe Weather NASA Earth Exchange DMSP content/uploads/presentations/NSC2017_Session_4.0_Ed_Kearns.pdf • University of Oklahoma/CIMMS Landsat, MODIS, Sentinel Landsat, MODIS, Sentinel • OCC Environmental Commons: http://edc.occ-data.org/ enabled AI to fully exploit observations. DigitalGlobe Events (pre/post) GFS, CFSv2, OpenAQ • Weather traditionally a problem solved with classical • Includes application of References tree-based or neural-network (NN) Agriculture Imagery Agriculture Imagery • Haupt, S., and coauthors, 2017: Building the Sun4Cast system: improvements in solar power physics Land Use (variety of sources) Land Use (variety of sources) forecasting. BAMS, Early Release, doi: 10.1175/BAMS-D-16-0221.1. based methods Figure 4: Convolutional Neural-Net Structure • Isaacson, P., G. Bloy, E. Wendoloski, and C. Mutchler, 2016: Short Term Forecasting of Solar Irradiance • Continuous innovation in physics modeling th (McGovern et al. 2017) IBM NESDP Microsoft Azure using Data Analytics and Probabilistic Cloud Cover Forecasts, 96 Annual American Meteorological • Difficult to break from physical constructs NOAA RAP NWP NASA EARTHDATA Society Conference, New Orleans, LA. • Marzban, C., and R. Viswanathan, 2017 : On the complexity of neural-network learned functions. 15th • Legacy approaches difficult to supplant NOAA NMFS Datasets WorldcLIM (gridded climate) Conf. on Artificial and Computational Intelligence and its Applications to Environmental • Aviation Sciences, Seattle, WA, Amer. Meteor. Soc., 2.1, • E.g., Legacy code persists older techniques OCC Environmental • MIT Lincoln Labs https://ams.confex.com/ams/97Annual/webprogram/Paper316115.html . • E.g., Forecaster-in-the-loop ingrained in operations GOES-16 • McGovern A., and coauthors, 2017: Using Artificial Intelligence to Improve Real-Time Decision Making • Includes mature for High-Impact Weather. BAMS, 98, 2073-2090, doi: 10.1175/BAMS-D-16-0123.1 • Past difficulties in implementing innovative approaches NEXRAD • R-Project, 2018: GNU general public license [Available online at https://www.r- • E.g., computational resources/software tools applications rooted in deep-learning project.org/Licenses/GPL-2] frameworks (Veillette et al. 2017) • Veillette, M., E.P. Hassey, C.J. Mattioli, H. Iskenderian, and P.M. Lamey: Using convolutional neural • Last two years demonstrate huge increase in networks to create radar-like precipitation analyses for aviation. 18th Conf. on Aviation, Range, and Aerospace Meteorology, Seattle, WA, Amer. Meteor. Soc., J2.2, community effort related to AI applications. © 2018 The Aerospace Corporation https://ams.confex.com/ams/97Annual/webprogram/Paper304433.html.