Volume 27, Issue 1 GEMSTONE The GEMstone Volume 27, Issue 1 May 2017 Focus Group Final Report Table of Contents Metrics and Validation Focus Group (2010-2015): Final Report 3 Masha Kuznetsova, Aaron Ridley, Tim Guild, Lutz Rastaetter, and Howard Singer Volume 27, Issue 1 GEMSTONE GEMstone Newsletters are available online at: http://aten.igpp.ucla.edu/gemwiki/index.php/Newsletters The GEMstone Newsletter is edited by Peter Chi ([email protected]) and Marjorie Sowmendran ([email protected]). The distribution of GEMstone is supported by the National Science Foundation under Grant AGS-1405565 . 2 May 2017 Metrics and Validation Focus Group 2010-2015 : Final Report GEMSTONE Masha Kuznetsova (NASA GSFC/CCMC), Aaron Ridley (2010-2012, University of Michigan), Tim Guild (Aerospace Corporation), Lutz Rastaetter (NASA GSFC/CCMC), and Howard Singer (2013-2015, NOAA/SWPC) The Metrics and Validation Focus Group NmF2, HmF2, drift velocities, and Total (M&V FG) was formed to address GEM needs Electron Content [Shim et al, 2012, 2014]. for systematic and quantitative evaluation of The partnership between the GEM M&V FG general geospace circulation models (GGCM) and the CEDAR community was further ex- with a goal of testing current space weather panded during the 2011 Joint GEM-CEDAR modeling capabilities and identifying areas in Summer Workshop, when a series of GEM- need of future scientific development. CEDAR Modeling Challenges were initiated. The GEM-CEDAR Challenges were focused The activities of the M&V FG in 2010-2012 on physical parameters, spatial domains and were focused on building upon successes of aspects of model validation of interest to both the series of GGCM Modeling Challenges ini- communities, such as the role of high altitude tiated by the M&V FG in 2008-2010. The drivers in storm-driven ionospheric and ther- goals of the Challenges were: mospheric disturbances, changes in regional To evaluate the current state of geo- TEC, and neutral densities, processes and space models, to demonstrate model boundaries in the auroral region. The GEM- capabilities, and to track model im- CEDAR Challenges also addressed issues re- provements over time by testing model lated to observational data quality and predictions against observations; availability, sensitivity of model outputs to To gather information on model valida- input parameters, boundary conditions, tion efforts and to define observables modeling assumptions, adjustable parameters. and numerical methods to compare models with measurements; In support of the Challenges the Community To facilitate interactions between re- Coordinated Modeling Center (CCMC, search communities and users of space http://ccmc.gsfc.nasa.gov) developed a suite weather products in developing metrics of interactive on-line metrics tools for for model evaluations tailored for spe- simulation results submissions and analysis. cific space weather applications; The CCMC continues maintaining the web To facilitate collaborations among site with interactive access to model output modelers and data providers, to address archive and observational data used for Chal- challenges of model-data comparisons; lenge projects. To facilitate further model improve- ment and to advance GEM science. The Modeling Challenge projects initiated by the Metrics and Validation Focus Group in In 2010 the GEM M&V FG supported the CE- 2008-2012 are listed in the Table 1. The table DAR community in setting up a CEDAR Elec- includes status, deliverables, GEM Focus trodynamics Thermosphere Ionosphere Group Co-Sponsors, relevant missions and (CEDAR ETI) Challenge based on the set of space weather applications. events selected by the GEM M&V FG and fo- cused on global ionosphere physical parame- The results of the Ground magnetic ters including neutral and electron densities, perturbation and Regional-K Challenges were 3 Volume 27, Issue 1 Table 1. The Modeling Challenge projects initiated by the Metrics and Validation Focus Group in 2008-2012 Phys. Parameter/ Status Deliverables (Publications, Reports) Relevant GEM research GEMSTONE Challenge Project (1st round) topics, Space weather ap- plications, Missions Magnetic field at completed Rastaetter et al., 2011; Pulkkinen et al., 2010 Inner Magnetosphere FG, geosyn. orbit. VA Probes Magnetopause loca- on-going Rastaetter et al, 2013, 2015; Dayside Research Area, tion Collado-Vega and Sibeck, 2015; Reconnection, MMS Analysis based on MMS data is expected in Dec 2016 Regional K* completed Glocer et al, 2016 Geomagnetically Induced A report (Phase II) to NOAA/SWPC on opera- Currents (GICs) tional geospace model selection Ground magnetic completed Pulkkinen et al., 2009, 2011, 2013 perturbations* Rastaetter et al., 2014 A report (Phase I) to NOAA/SWPC on opera- tional geospace model selection Dst Index completed Rastaetter et al., 2012 Inner Magnetosphere, Storm impact Heat flux into iono- completed Rastaetter et al, 2016 Dayside FACs and Energy sphere** Deposition FG, Satellite Drag Auroral boundaries *, on-going Zheng et al, 2015, Lane et al, 2015 Inner Magnetosphere, Sur- ** face charging CEDAR ETI (NmF2, completed Shim et al, 2012, 2014 Ionosphere Disturbances, hmF2, drift GPS, Communications, velocities, TEC) ** Satellite Drag Role of High-Latitude on-going M-I Coupling Patch-Panel Driver Swapping M-I Coupling Drivers on Iono- Tool, Library of drivers for selected events sphere/ Thermo- sphere ** (*) Metric studies of primary interest to operational community (NOAA/SWPC, AFWA). (**) Joint GEM-CEDAR Challenge utilized as a basis for the Operational Geospace Reconnection FG. Since 2013 the Heat Flux model selection by NOAA/SWPC and will be into Ionosphere, and the Role of Driver used as a benchmark for an upcoming 2nd round Challenges are carried on by the GEM-CEDAR of Challenges to trace model improvements. In Challenge Working Group hosted by CEDAR. addition, carrying out these challenges resulted The GEM-CEDAR Challenge discussion in model developers providing new and sessions are continuously organized at CEDAR improved versions of their models for use by Summer Workshops and at GEM Fall mini- the research community for runs-on-request. Workshops (GEM-CEDAR Challenge special Efforts to define and model auroral boundaries sessions). were started in 2011 as well as determining and comparing the magnetopause position and In 2013-2015 under new leadership the M&V standoff distance using geosynchronous FG continued to define and pursue modeling satellites. Both studies are still ongoing and are challenges encompassing different regions of being taken up by the GEM GSM Modeling the Earth magnetosphere-ionosphere coupled Methods and Validation Focus Group (MMV, system. The efforts were expanded into new 2016-2020). The MMS data opened new analysis methods (including climatological opportunities for these Challenges that are now techniques), model uncertainty and extreme also of interest to the GEM Magnetic event simulations such as Carrington-style events (that are beyond the range of events ob- 4 May 2017 served during the space age). In 2013-2014 Publications: presentations by various modelers addressed sensitivity of models to variations in input data Baker, D. N., X. Li, A. Pulkkinen, C. M. GEMSTONE and approaches to validate simulations of ex- Ngwira, M. L. Mays, A. B. Galvin, K. D. treme events by using different types of historic C. Simunac, A major solar eruptive event in GEMSTONE records related to storm impacts and by extrap- July 2012: Defining extreme space weather olation of well-observed storm events. To un- scenarios, Space Weather, 11 (10), 585- derstand significant differences in model out- 591, doi:10.1002/swe.20097 (2013). puts demonstrated by some metrics studies, a Claudepierre, S. G., F. R. Toffoletto, M. Wilt- series of baseline model comparisons were ini- berger, Global MHD modeling of resonant tiated. The researchers agreed to run different ULF waves: Simulations with and without a MHD codes for various solar wind and IMF plasmasphere, JGR Space Physics, 121 (1), conditions and compare the results. The sensi- 227-244, doi:10.1002/2015JA022048 tivity of models to boundary conditions and in- (2016). ternal (numerical) parameters were explored to Facsko, G., I. Honkonen, T. Zivkovic, L. uncover reasons for the differences among the Palin, E. Kallio, K. Agren, H. Opgenoorth, major global magnetosphere MHD models. E. I. Tanskanen, S. Milan, One year in the Earth's magnetosphere: A global MHD Consistent with the goal of being the glue that simulation and spacecraft measurements, binds GEM together, the M&V FG reached out Space Weather, 14 (5), 351-367, to other GEM FGs to facilitate relevant model- doi:10.1002/2015SW001355 (2016). ing challenges for the state of the art models in Glocer, A., L. Rastaetter, M. Kuznetsova, A. their groups. In 2013, modelers explored ways Pulkkinen, H. J. Singer, C. Balch, D. to validate and improve models through com- Weimer, D. Welling, M. Wiltberger, J. parisons with observational maps of electric Raeder and R. Weigel, J. McCollough, S. currents (from AMPERE). New sources of data Wing, Community-wide validation of were introduced (TIMED/GUVI) that can aid geospace model local K-index predictions model development in magnetosphere- to support model transition to operation, ionosphere coupling. The focus group also Space Weather, 14 (7), 469-480, doi: started a study to model ULF wave fields in 10.1002/2016SW001387, (2016). global magnetosphere MHD models for eventu- Gordeev E. V., V. Sergeev, I. Honkonen, M. al comparison with in-situ (Van Allen Probes) Kuznetsova, L. Rastaetter, M. Palmroth, P. and ground-based observations (PC-4 magnetic Janhunen, G. Toth, J. Lyon, M. Wiltberger, perturbations). In 2015, there was more discus- Assessing the performance of community- sion on the assessment of real-time model runs. available global {MHD} models using key After the completion of the Metrics and Valida- s ystem parameters and empirical tion Focus Group, the ULF modeling study is relationships, Space Weather, 13, 868-884, now being continued by the new UMEA and doi:10.1002/2015SW001307 (2015). MMV focus groups (2016-2020). Katus, R. M., M. Liemohn, E.L. Ionides, R.