The Earth System Grid: a Visualisation Solution
Total Page:16
File Type:pdf, Size:1020Kb
The Earth System Grid: A Visualisation Solution Gary Strand Introduction Acknowledgments •PI’s •ESG Development Team –Veronika Nefedova (ANL) –Ian Foster (ANL) –Ann Chervenak (ISI/USC) –Don Middleton (NCAR) –Carl Kesselman (ISI/USC) –Dean Williams (LLNL) –David Bernholdt (ORNL) –Kasidit Chanchio (ORNL) –Line Pouchard (ORNL) –Alex Sim (LBNL) –Arie Shoshani (LBNL) –Bob Drach (LLNL) –Dave Brown (NCAR) –Gary Strand (NCAR) –Jose Garcia (NCAR) –Luca Cinquini (NCAR) –Peter Fox (NCAR) Current Practices oScientist (or others) wants a visualisation oVisualisation person gets appropriate data after verifying with data manager as to the name, location, total size, etc. oData moved to “local” machine that has visualisation tools oVisualization created on “local” machine oHopefully, someone remembers to archive the visualisation Simple Vis Example Problems in the process oWhat if the data cannot be found (e.g. we have 1.2 million files, 73 TB of data), or the data manager is unavailable? oWhat if there isn’t enough disk space or sufficient other resources? oWhat if a better visualisation tool is located elsewhere? oWhat if the visualisation should be shared? oWhat if the visualisation is lost? o ESG is part of the answers to these questions What is ESG? ANL: Computational grids, LBNL: Climate storage & grid-based applications facility LLNL: Model diagnostics & inter-comparison USC/ISI: Computational grids, & grid-based applications NCAR: Climate change LANL: Next generation ORNL: Climate storage & predication and scenarios coupled models & computing computational resources ESG Architecture LBNL HPSS! High Performance! disk! Storage System! ANL CAS! Community Authorization Services! NCAR SRM! gridFTP! gridFTP! openDAPg! Storage Resource ! Striped! server! server! Management! server! MyProxy! Tomcat servlet engine ! server! MCS client! MyProxy client! disk! LLNL CAS client! RLS client! SRM! Storage Resource ! gridFTP! GRAM! Management! server! gatekeeper! gridFTP! server! gridFTP! ORNL SRM! ISI gridFTP! gridFTP! Storage Resource ! SRM! server! Management! SOAP" Storage Resource ! MCS! Management! Metadata Cataloguing Services! RLS! RMI" HPSS! MSS! disk! High Performance! Replica Location Services! disk! Mass Storage System! Storage System! Solutions oWhat! happens when data cannot be found, or the data manager is unavailable? •Metadata! catalogue service (MCS) •Replica! location service (RLS) MCS and RLS and Metadata Services ESG CLIENTS API ! PUBLISHING! ANALYSIS & VISUALIZATION! & USER INTERFACES! SEARCH & DISCOVERY! ADMINISTRATION! BROWSING & DISPLAY! HIGH LEVEL METADATA SERVICES! METADATA! METADATA! METADATA & DATA ! METADATA! METADATA! EXTRACTION! ANNOTATION! REGISTRATION! BROWSING! QUERY! METADATA! METADATA! METADATA! METADATA! AGGREGATION! VALIDATION! DISPLAY! DISCOVERY! CORE METADATA SERVICES! METADATA ACCESS! SERVICE TRANSLATION! (update, insert, delete, query)! LIBRARY! METADATA HOLDINGS! Data &! mirror! Dublin Core! Dublin Core! COARDS! COMMENTS! Metadata! Database! Database! XML Files! Catalog! XML Files! Solutions (contd.) oWhat! if there isn’t enough disk space or sufficient other resources? •Hierarchical! Resource Manager (HRM) HRM Solutions (contd.) oWhat! What if a better visualization tool is located elsewhere? •Distributed! visualization CDAT Example of an ESG Script Access ! The next-generation language, Python, is used to access the Earth System Grid (ESG) at LLNL Import cdms, vcs db = cdms.open(“ldap://localhost:389/database=demo,ou=PCMDI,o=LLNL,c=US”) f = db.open( “ncep_reanalysis_mo”) ds = f(‘ts’) x=vcs.init( ) x.plot(ds) CDAT: Example of an ESG GUI Client Access Solutions (contd.) oWhat! if the visualization should be shared? • ! Access Grid plus Visualisation Tool Collaborative Environments Science Portals + AccessGrid: University of Michigan (Knoop, Hardin) Vegetation & Ecosystem Mapping Program (VEMAP) NCAR/SCD VETS/KEG Argonne National Labs Conclusions " Visualisation! can require as many services and resources as the initial computation " Many! sites do not offer sufficient resources for the visualisations earth sciences require " ESG! provides, and will provide, the tools that enable visualisation on a grander scale Conclusions (contd.) " ESG! tools enable better data access, better data knowledge, and the processes of collaboration for the needs of investigating, visualising, and learning .