http://earthdata.nasa.gov

NASA Earthdata

Mark McInerney Deputy PM - Technical Data & Information System (ESDIS) November 2019

1 NASA’s Enterprise Approach to Cloud

Key Elements of the Enterprise Cloud Framework

. The NASA CIO is responsible for delivering enterprise-class commercial cloud computing services to the entire agency . Focus on smart consumption of commercial Cloud Services . Standardized Agency governance . Integrated hierarchical approach to cybersecurity . Standards and guidance for technical integration with Agency infrastructure, processes, services . Networking, security operations, authentication services . Common procurement vehicles with proper terms, conditions, best practices . All users obtain access to cloud computing directly through OCIO or All users leverage the CIO Cloud Framework to delegated authority minimize start-up time and costs . Payment system to facilitate “pay as you go” within Agency . Cuts months from cloud learning curve constraints . Significantly reduces “pioneering” costs . Integration with Agency IT service catalog and help desk . Reduces duplication of effort

NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 2 EOSDIS Comprises Data of the Whole Earth System Atmosphere Winds & Aerosols & Temperature & Humidity Solar radiation

Ocean Surface temperature Surface wind fields & Heat flux Surface topography Ocean color

Cryosphere Sea/Land Ice Snow Cover

Land Cover & Usage Soil Moisture Topography & elevation Temperature

The Earth Observing System Data and Information System (EOSDIS) Human Dimensions Population & Land Use Human & Environmental Health NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 3 Components of EOSDIS End-to-End

EOSDIS Research NASA’s EOSDIS provides end- to-end capabilities for distribute Applications managing NASA’s Earth science data from satellites, Education aircraft, field measurements, data subset and various other programs. downlink Users

EOSDIS is responsible for a archive data collection that is large in volume and projected to grow capture rapidly over the next several and process years. clean

NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 4 EOSDIS 12 Distributed Active Archive Center (DAACs) and EOSDIS Organization 13 Science Investigator-led Processing Systems (SIPS)

Socioeconomic Data and Applications Center Human Interactions, Suomi National Polar-orbiting Land Use, Environmental Partnership (Suomi-NPP) Sustainability, Atmosphere Geospatial Data Goddard Earth Sciences Data and Information Services Center Land Processes Global Precipitation, Solar Irradiance, DAAC Atmospheric Composition and Dynamics, Land Cover, Surface Reflectance, Global Modeling Radiance, Temperature, Topography, ◉ Vegetation Indices ◉ ◉ Crustal Dynamics Data Level 1 and Atmosphere Archive Information System Space Geodesy, Solid Earth Measurements and Distribution System (LAADS) MODIS Level-1 and Atmosphere Data Products ◉◉ of Pollution in the ◉◉◉◉ Physical Troposphere (MOPITT) ◉◉◉◉ Ocean Biology DAAC Oceanography DAAC Visible Infrared Imaging Ocean Biology, Sea Surface Temperature Gravity, Sea Surface Temperature, ◉◉ Radiometer Suite ◉ Microwave Ocean Winds, Topography, Circulation & Currents (VIIRS) Ocean Limb Oak Ridge National Sounder Laboratory DAAC (MLS) Biogeochemical Dynamics, ◉ Ozone Mapping Tropospheric Emission Ecological Data, Environmental Profiler Suite (OMPS) LaRC Atmospheric Spectrometer (TES) National Snow and Ice Processes Ozone ◉◉◉ Data Center DAAC Science Data Center • ◉ Frozen Ground, Glaciers, ◉◉ Radiation Budget, Clouds, Ozone Monitoring Ice Sheets, Sea Ice, Aerosols, Tropospheric Snow, Soil Moisture Chemistry Instrument (OMI) Suomi National Polar-orbiting • Partnership (Suomi-NPP) Global Hydrology Ozone Mapping (ATMS) and (CrIS) Sounder Resource Center DAAC Advanced Microwave Profiler Suite (OMPS) Hazardous Weather, Scanning Radiometer for Ozone , Tropical Cyclones and EOS 2 (AMSR-E/2) • Alaska Satellite Storm-induced Hazards Ocean Data Processing Facility DAAC System (OCDPS) SAR Products, Sea Ice, • Polar Processes, MODIS Adaptive ◉ Processing System (MODAPS) • f Visible Infrared Imaging Radiometer Suite (VIIRS) Land

NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 5 Motivation for Commercial Cloud Motivation for Cloud Projected Data Volumes Growth of Mission Data & Processing: Projected rapid archive growth and the need to effectively process significantly larger volumes of new mission data requires rethinking existing architectures.

Data Systems: More cost-effective, flexible, and scalable data system ingest, archive, and distribution solutions are needed to keep pace with new mission advancement.

Science Users: Significantly larger data volumes requires additional ways to access and utilize this data, with “Data Close to Compute” or Data Lake”. Bring Algorithms to the cloud

NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 6 ● “Managed” commercial cloud for EOSDIS on AWS

● Improves the efficiency of NASA’s data systems operations – maintaining free/open data policy

Earthdata Cloud (EDC) ● Designed for EOSDIS applications and mission data ingest, archive, distribution

● Increase opportunity for researchers and commercial users to access/process petabytes of data quickly without the need for data management. NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 7 Cloud Mission Data Data Services Organizational Budget Metrics Community Platform Management & analytics Governance Controls Management Development Infrastructure Model

01 02 03 04 05 06 07

Components of the Earthdata Cloud

NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 8 User Access / Networking Support Services

Security Cloud Organizational Budget Metrics Community Services Platform Governance Controls Management Development Infrastructure Model

01 04 05 06 07 Governance of Cloud Accounts

Certification Common and Services and Accreditation Controls Strategy

Project Level Components and Core Elements

NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 9 Cloud Platform Infrastructure

Common Services Common Services and Controls and Controls

1. NASA-Approved Amazon Services: vetted AWS and third- party SAAS services and process to add new. Focus is on using AWS cloud-native services

2. Code Deployment Services: DevOps CICD Pipeline to security scan code, build, and deploy into EDC

3. Use of Infrastructure as Code: including re- useable template to define a multi-account ecosystem

4. Single System Security Plan (SSP) and Authority to Operate (ATO)

5. Single Identity and Access Management Solution (CloudTamer.io): • Rotate AWS access keys • Apply session limits • Provide role-based access control • two-factor authentication

Components of the EOSDIS Earthdata Cloud (EDC) 10 Cloud Platform Infrastructure

Governance of Account Structure Cloud Accounts

• Single Payer Account using AWS Organization and Consolidated billing • Multi-Account structure divided into NASA / Mission-defined organizational units • Isolation based on organizational units • Isolation based on application development, test, and production accounts • Isolation of Management and Security accounts from end user environment • Track AWS expenses to NASA organizations and funding sources

Components of the EOSDIS Earthdata Cloud (EDC) 11 Cloud Platform Infrastructure Est 200 AWS Accounts AWS Account Current AWS Account Structure Metrics Fall 2020 EDC AWS Account Totals 100 89 AWS Regions by Accounts 90 83 82 80 73 70 63 us-west-2 60 55 55 (Oregon), 33, 37% 50 43 us-east-1 (N. 41 40 41 Virginia), 56, 36 37 37 63% 40 33 30 30 28 28 28 23 23 23 24 20 18 18 19 19 14 14 14 14 14 15 10 10 11 8 9 9 9 10 6 6 7 6 7 6 us-east-1 (N. Virginia) 4 3 3 4 4 0 0 1 1 0 us-west-2 (Oregon) 1/1/19 2/1/19 3/1/19 4/1/19 5/1/19 6/1/19 7/1/19 8/1/19 9/1/19 10/1/19 11/1/19

System / Management Sandbox Pre-Production (SIT/UAT) Production Total

155 Unique Users with 200 Roles Components of the EOSDIS Earthdata Cloud (EDC) 12 Cloud Mission Data Data Services Organizational Budget Metrics Community Platform Management & analytics Governance Controls Management Development Infrastructure Model

01 02 03 04 05 06 07

Components of the Earthdata Cloud

NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 13 Organizational Governance Development Project Management Board Framework

Platform Team

….. 01 IT Security Team

08 02

Additional Teams at Development Project DAAC Cloud discretion of ….. 07 03 PMB to meet Management Migration Team established Board (PMB) goals

06 04

05 … Services Team

Business Analysis Team

Components of the EOSDIS Earthdata Cloud (EDC) 14 Cloud Mission Data Data Services Organizational Budget Metrics Community Platform Management & analytics Governance Controls Management Development Infrastructure Model

01 02 03 04 05 06 07

Components of the Earthdata Cloud

NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 15 Budget Controls The Antideficiency Act (ADA) and Pay-as-You-Go

EOSDIS today has over 30 Petabytes The ADA act prohibits federal agencies from obligations of accessible Earth science data or expending federal funds in advance or in excess of an appropriation, and from accepting voluntary services. EOSDIS delivered over Federal employees who violate the Antideficiency Act 1.6 Billion data products are subject to two types of sanctions: administrative and to over 3.1 Million penal. Employees may be subject to appropriate science users from around the world administrative discipline including, when circumstances warrant, suspension from duty without pay or removal from office. In addition, employees may also be subject to fines, imprisonment, or both.

… with over 438 Million Science data files in the repository NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 16 Budget Controls

Cost Cloud Resource & Cost Model Accounting Cost accounting begins with first capturing individual mission/application level required AWS resources and cost for execution year and out to year 5.

Used to: 1. Project level cloud resource and cost capture 2. Feed NASA’s Planning, Programming, Budgeting and Execution (PPBE) 5 year budget request / cycle 3. Input into execution year “cost phasing plan” for each account to manage account level CloudTamer budget caps 4. Support performance and cost optimization processes, routine cloud account auditing 5. Initial capture of AWS service needs, vetting for availability and metrics for reserved instance discounts

Components of the EOSDIS Earthdata Cloud (EDC) 17 Budget Controls The cloudtamer.io account-level view

• Tool to push & fund individual AWS accounts under a single AWS payer account • Enforces individual AWS account-level budget through “budget caps” • Provides account alert spend monitoring and budget & egress control actions • Allows for flexible access levels: • Top-level view for management & business teams • Account view for local managers & Developers

NASA Earthdata Cloud (EDC) – http://earthdata.nasa.gov 18 http://earthdata.nasa.gov

NASA Earthdata Cloud

Mark McInerney Deputy PM - Technical Earth Science Data & Information System (ESDIS) November 2019

19