Perspectives on Risk‐Informed Decision‐Making Enabling a Compelling Tomorrow
Michael A. Johnson Chief Technologist, Engineering and Technology Directorate NASA Goddard Space Flight Center michael.a.johnson@nasa.gov
1 NASA/SP-2010-576 Version 1.0 April 2010
Questions considered‐ • What is NASA’s view on risk and risk‐informed decision making? • How does NASA incorporate risk insights into its day‐to‐day decision making? • What are challenges and successes with integrating risk‐informed decision making at NASA?
2 Background
National Aeronautics and Space Administration (NASA) capabilities are distributed across the Nation.
3 Background Goddard Space Flight Center
Goddard Space Flight Center Wallops Flight Facility (GSFC) is distributed across five locations. White Sands Complex
The main campus is in Greenbelt, MD.
Greenbelt
Goddard Institute for Independent Verification and Space Studies Validation Facility
4 Voyager QuikSCAT Background Wind ERBS EO‐1 RHESSI Stereo ACRIMSAT TRMM TOMS GSFC provides essential technical and Geotail SOHO THEMIS Aqua ICESat TIMED Landsat 7 TOPEX TRACE scientific support to address national and CALIPSO world societal needs through our work with ACE Terra GRACE SORCE Cluster POES and for other Federal agencies. SDO IMAGE AIM FAST Aura GOES Polar GPM MMS We derive and share information, solutions, Solar‐B Aquarius and technology for NASA, the Nation, and IBEX NPP TDRSS LDCM CloudSat RBSP Osiris‐Rex the world. (Sample WMAP TWINS Cassini Return) (Instrument) NuSTAR JWST HST We engage with other NASA centers, New WISE GALEX Messenger Horizons Spitzer SWAS universities, and industry in search of Astro‐H GEMS Galileo IUE RXTE opportunities to blend our competencies Juno MAVEN Integral with their specialized assets in order to Pioneer EUVE FUSE Swift obtain knowledge that can be applied to Mars Science COBE Laboratory Fermi Compton addressing some of the most profound and LADEE LRO GRO 5 compelling questions of our time. GSFC has flown over 300 Heliophysics, Earth Science, Astrophysics, and Planetary missions
5 Background 301 Successes Numerous Launch Vehicle (LV) failures GSFC has a history of 0 Spacecraft (SC) failures since 1975 mission success: * 1 Success Failure (due to LV) Failure (due to SC) No spacecraft failures * GSFC does not develop or deliver launch vehicles since 1975. 1 1 2 Success has been derived 3 13 13 12 in part, by enforcing a 11 1 2 mostly static set of 99 9 1 1 8 8 88 8 8 7 7 7 7 7 Mission Assurance 6 6 5 5 5 5 3 5 555 5 4 4 4 44 4 4 4 4 4 3 Requirements (MAR). 1 3 3 3 3 3 3 2 2 2 2 22 2 2 2 2 1 111 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 GSFC Mission Outcomes
6 NASA/SP-2010-576 Version 1.0 April 2010
• What is NASA’s GSFC’s view on risk and risk‐informed decision making? • How does NASA GSFC incorporate risk insights into its day‐to‐day decision making? • What are challenges and successes with integrating risk‐informed decision making at NASA GSFC?
7 What is GSFC’s view on risk and risk‐informed decision making?
It would be wrong to conclude that mission success derived from the MAR approach would Success Failure (due to LV)* Failure (due to SC) obviate a need for change. 1 * GSFC does not develop or deliver launch vehicles To effectively and efficiently meet today’s and 1 1 tomorrow’s space exploration challenges, GSFC 2 must not only make advancements in what we 3 13 13 do; we must also advance how we do it. 12 11 1 2 99 9 1 1 8 8 88 8 8 7 7 7 7 7 6 6 4 5 5 5 5 3 5 555 5 3 4 4 44 4 4 4 4 4 1 3 3 3 3 3 3 2 2 2 2 22 2 2 2 2 1 111 GSFC Mission Outcomes
8 What is GSFC’s view on risk and risk‐informed decision making? GSFC implements a Risk‐based Safety and Mission Assurance (SMA)‐ The process of applying limited resources to maximize the chance for safety and mission success by focusing on mitigating specific risks that are applicable to the project vs. simply enforcing a set of requirements because they have always worked.
Key Elements • Risk‐informed Framework • Risk‐informed Requirements Generation • Risk‐informed Decision Making • Risk‐informed Review and Audit
Risk‐based Safety and Mission Assurance (SMA) began formal GSFC implementation in 2016
9 • What is GSFC’s view on risk and risk‐informed decision making? • How does GSFC incorporate risk insights into its day‐to‐day decision making? • What are challenges and successes with integrating risk‐informed decision making at GSFC?
Voyager QuikSCAT Wind ERBS EO‐1 RHESSI Stereo ACRIMSAT TRMM TOMS Geotail SOHO THEMIS Aqua ICESat TIMED Landsat 7 TOPEX TRACE CALIPSO ACE Terra GRACE SORCE Cluster POES SDO IMAGE AIM FAST Aura GOES Polar GPM MMS Solar‐B Aquarius IBEX NPP TDRSS LDCM CloudSat RBSP Osiris‐Rex (Sample TWINS WMAP Cassini Return) (Instrument) NuSTAR JWST HST New WISE GALEX Messenger Horizons Spitzer SWAS Astro‐H GEMS Galileo IUE RXTE Juno MAVEN Integral
Pioneer EUVE FUSE Swift Mars COBE Science Fermi Compton LADEE Laboratory LRO GRO 10 Developing space flight systems is GSFC’s “day‐to‐day”
10 How does GSFC incorporate risk insights into its day‐to‐day decision making? Example: Printed Circuit Board Test Coupon Nonconformance Handling
Printed circuit boards are a foundational component of most spaceflight electrical, electronic, and electromechanical systems
How do we assess the quality of the bare board and its manufacturing process?
Ref: Jesse Leitner, Bhanu Sood, Eric Isaac, Jack Shue, Nancy Lindsey & Jeannett Plante (2018) Risk‐based safety and mission assurance: Approach and experiences in practice, Quality Engineering, 30:4, 648‐662, DOI: 10.1080/08982112.2018.1473584 Populated printed circuit board
11 How does GSFC incorporate risk insights into its day‐to‐day decision making? Example: Printed Circuit Board Test Coupon Nonconformance Handling
PCB PANEL
Printed Circuit Board (PCB) PCB COUPON COUPON
PCB PCB COUPON COUPON
One PCB panel typically yields multiple bare PCBs. Test “coupons” inform the board/ fabrication process quality assessment. Populated printed circuit board
12 How does GSFC incorporate risk insights into its day‐to‐day decision making? Copper wicking Example: Printed Circuit Board Test Coupon Nonconformance Handling
Traditional Approach: Evaluate test coupons to favored standard. Reject and refabricate all boards associated with nonconforming coupons.
Risk‐based Approach: Use requirements from one or more commonly‐ used standards to define minimum requirements for a range of risk levels. Assess risk when nonconformance is found and determine Capped vias whether risk is acceptable. Only refabricate if cause for nonconformance is understood and can be eliminated.
Risk‐based Results Summary: Out of 193 boards with nonconforming test coupons, over 85% had either no elevated risk or very low risk associated with the non‐conformance. The risk acceptance resulted in millions of dollars and hundreds of weeks of schedule saved relative to the Copper wicking and vias are two (of traditional approach. several) coupon elements evaluated during coupon analysis.
13 How does GSFC incorporate risk insights into its day‐to‐day decision making? Example: Printed Circuit Board Test Coupon Nonconformance Handling Copper wicking Traditional Approach: Evaluate test coupons to favored standard. Reject and refabricate all boards associated with nonconforming coupons.
Risk‐based Approach: Use requirements from one or more commonly‐ used standards to define minimum requirements for a range of risk Capped vias levels. Assess risk when nonconformance is found and determine whether risk is acceptable. Only refabricate if cause for nonconformance is understood and can be eliminated.
Risk‐based Results Summary: Out of 193 boards with nonconforming test coupons, over 85% had either no elevated risk or very low risk associated with the non‐conformance. The risk acceptance resulted in millions of Copper wicking and vias are two (of dollars and hundreds of weeks of schedule saved relative to the several) coupon elements evaluated during coupon analysis. traditional approach.
14 NASA/SP-2010-576 Version 1.0 April 2010
• What is GSFC’s view on risk and risk‐informed decision making? • How does GSFC incorporate risk insights into its day‐to‐day decision making? • What are challenges and successes with integrating risk‐informed decision making at GSFC?
15 What are challenges and successes with integrating risk‐informed decision making at NASA?
NASA is increasingly infusing small satellites (mass < 180 kg) into its missions and mission planning activities
• Increased cost and schedule efficiencies • Greater spatial, temporal, and spectral coverage • More frequent launch opportunities • Greater potential for innovative mission implementation approaches • Greater opportunities to leverage private sector innovation Dellingr CubeSat deployment from the International Space Station, December 2017 Development and use of SmallSats raises multiple challenges and successes, many relevant to RIDM
16 What are challenges and successes with integrating risk‐informed decision making at NASA? Example: The Dellingr CubeSat Mission
Leveraging private sector innovation The use of commercial off‐the‐shelf (COTS) components is increasing.
However, vendors may provide minimal mission assurance documentation, and components may have little flight heritage. Dellingr CubeSat. 10 x 20 x 30 cm, ~11 kg
This presents an SMA challenge.
Dellingr 6U CubeSat‐ internal view (during development)
17 What are challenges and successes with integrating risk‐informed decision making at NASA? Example: The Dellingr CubeSat Mission
Consistent with this SMA challenge, the Dellingr spacecraft exhibited numerous component anomalies and failures during development and on‐orbit.
Yet, the mission is successful because the “mission system” was resilient to component failure.
Ref: Cutting Edge‐ Goddard’s Emerging Technologies, Fall 2018
18 What are challenges and successes with integrating risk‐informed decision making at NASA? Example: The Dellingr CubeSat Mission Spacecraft Ground systems Success Lesson: RIDM should not be considered just a safety and mission assurance team activity.
The degree to which the approach is infused across the mission full‐lifecycle (formulation through operations) and throughout the mission team can provide mission resiliency and inform success. Science and engineering team Operations team
The Dellingr “system” mitigated numerous on‐orbit component failures/anomalies and infrastructure challenges.
19 What are challenges and successes with integrating risk‐informed decision making at NASA?
Other High‐level RIDM Integration Challenges
• The fear of failure: changing a “failure is not an option” dominant culture is not easy • Organizational compartmentalization is a barrier to “mission system‐level thinking” • Personal comfort and protection can be derived from strictly adhering to static SMA requirements • Organizational inertia can be hard to overcome • Organizations may need to re‐learn the “art of thinking”
An organization’s history of success can be its largest barrier to RIDM infusion
20 In Closing
• GSFC had a history of success prior to implementing risk‐based SMA • Yet, advancement essential and is not just informed by the “what”; it’s also the “how” • GSFC has migrated to risk‐based SMA, yielding millions of dollars and hundreds of weeks of schedule saved relative to a static requirement‐based approach • The degree to which a risk‐informed approach is infused across the mission full‐lifecycle (formulation through operations) and throughout the mission team can provide mission resiliency and inform success. • An organization’s history of success can be its most significant barrier to RIDM infusion
Re‐learning the “art of thinking” can lead to a compelling tomorrow.
21 For further information:
1 Dezfuli, Homayoon & Stamatelatos, Michael & Maggio, Gaspare & Everett, Chris. (2010). NASA/SP‐2010‐576, NASA Risk‐Informed Decision Making Handbook.
22 For further information:
The paper Risk‐based safety and mission assurance: Approach and experiences in practice includes additional PCB case study details and additional risk‐based SMA examples and results.
2Jesse Leitner, Bhanu Sood, Eric Isaac, Jack Shue, Nancy Lindsey & Jeannett Plante (2018) Risk‐based safety and mission assurance: Approach and experiences in practice, Quality Engineering, 30:4, 648‐662, DOI: 10.1080/08982112.2018.1473584
url https://doi.org/10.1080/08982112.2018.1473584
23 Thank You
24