Designing for System Value Sustainment using Interactive Epoch Era Analysis Space Tug Case Study Mike Curry and Adam Ross Conference on Systems Engineering Research Huntsville, AL March 24, 2016 Outline • Motivation • Background – EEA – Prior EEA Frameworks – Visual Analytics • Hypotheses • IEEA Framework • Space tug case study • Summary seari.mit.edu © 2016 Massachusetts Institute of Technology 2 Motivation and Challenges Motivation Challenges • Development of “resilient” • Need framework that explicitly systems identified by DoD as considers human interaction a strategic research priority during EEA to comprehend and make decisions • Desire sustained value delivery from systems in spite • Visual and analytic methods of perturbations in design, for making sense of high- context or needs over time dimensional multivariate data • Epoch-Era Analysis (EEA) • Methods and metrics for considers the time varying assessment of path- needs of stakeholders and dependent drivers of value in evolving contexts in which the dynamic environments that system operates, but allow the identification of long- challenges remain term strategies seari.mit.edu © 2016 Massachusetts Institute of Technology 3 Background (EEA) • Tradespace exploration tends to focus on analysis of system alternatives within a static context and needs [20] • EEA conceptualizes the effects of time and changing context on a system [4,5] – Epochs: periods of fixed context and needs (short run) – Eras: sequences of epochs simulating a potential future experienced by the system (long run) seari.mit.edu © 2016 Massachusetts Institute of Technology 4 Background (EEA Frameworks) • RSC is a method for applying EEA developed to study system value sustainment through changeability [6,7] • RMACS expanded RSC processes and explored the application of multi-attribute expense (MAE) to more effectively capture all resources expenditures required to realize a given system [8] • Key challenges: – Volume and complexity of data when conducting analysis across a potentially large number of epochs and eras – Sense-making when attempting to extract useful and actionable information from the analyses seari.mit.edu © 2016 Massachusetts Institute of Technology 5 Background (Visual Analytics) • Research in the field of visual analytics has demonstrated advantages/insights that can be gained by allowing humans to interact with and visualize data Visual analytics combines automated analysis with interactive visualizations for effective • Enhance analysts ability understanding, reasoning and decision making on the basis of a very large and complex dataset [Keim, to understand and 2010]. communicate data through interactive Data analysis mantra: “Overview first, zoom/filter, visualization and analysis details on demand” [Shneiderman, 1996] applications Visual analysis mantra: “Analyze first, show the important, zoom/filter, analyze further, details on demands” [Keim, 2006] seari.mit.edu © 2016 Massachusetts Institute of Technology 6 Hypotheses Incorporating research from the field of visual analytics to extend EEA can better address the challenges of ERS . A new design framework for EEA that Framework with incorporates research from the field of visual explicit consideration analytics should consider both algorithmic for human interaction and visual data analysis to enable effective decision making . Direct interaction through a visual interface will allow decision-makers to better analyze Interactive and explore EEA data Visualization . Interaction enables understanding, reasoning and decision making on the basis of a very large and complex EEA data seari.mit.edu © 2016 Massachusetts Institute of Technology 7 IEEA Framework 1 Problem Elicitation Definition • Inspired by visual 2 Design analytics process and Formulation extends processes of Generation/ RSC and RMACS 3 4 Epoch Era Sampling Characterization Construction Databases, Recommendation Engine, etc and Data Generationand Data Problem FormulationProblem • Prescriptive 5 Evaluation framework for Design-Epoch-Era Design of Experiments Evaluations (DOE), Adaptive DOE, applying EEA to gain Markov Chain Monte Carlo insights includes: 6 Single Epoch 8 Single Era Analyses – Data generation (1-5) Analysis Analysis Dijkstra's algorithm, based & - Path planning / search – Algorithmic and visual 7 Multi Epoch 9 Multi Era algorithms, Centrality Analysis Analysis Metrics, Minimax, A* data analysis (6-9) visual visual analysis Model Decisions 10 Visual and Evidence / Data / algorithmic data analysis, domain Insights knowledge Knowledge seari.mit.edu © 2016 Massachusetts Institute of Technology 8 Space tug case study Design Variable Levels • Scenario: A space tug Capability Low, Med, High, rental company needs to Extreme provide the services of Propulsion Type BiProp, Cryo, the system to customers Electric, Nuclear Fuel Mass 30, 100, 300, 600, with varying missions 1200, 3000, 10000, and preferences 30000, 50000 DFC Level 0, 1, 2 384 Design Alternatives • Goals: Provide value to customers, Performance Range regardless of changing mission Attribute requirements, for as long as possible Cost [96.9, 3952.8] Payload [300, 5000] Isp [300, 3000] Mass fraction [0.12, 0.30] Speed [0, 1] Decision-maker is attempting to satisfy sets of Delta V [24.4, 41604] preferences corresponding to different missions Base mass [0, 1000] Dry mass [604, 22830] Wet Mass [634, 69800] seari.mit.edu © 2016 Massachusetts Institute of Technology 9 Epoch-space Characterization • Each of the 8 missions has Missions different performance Baseline mission requirements for payload, Technology demonstration speed and DV GEO satellite rescue Satellite deployment assistance In-orbit refueling and maintenance • Single context variable to Garbage collection describe current technology level All-purpose military mission Satellite saboteur – Affects the transition costs of the change mechanisms – Affects fuel efficiencies and Epoch Variable Levels Mission 1 - 8 mass fractions of different Technology Level Present or Future propulsion systems 8 Preference Sets * 2 Contexts = 16 Epochs seari.mit.edu © 2016 Massachusetts Institute of Technology 10 Design Changeability • Desire systems that can No. Change Mechanism Effect DFC level maintain value delivery 1 Engine Swap Biprop/Cryo swap 0 across different missions 2 Fuel Tank Swap Change fuel mass 0 3 Engine Swap (reduced cost) Biprop/Cryo swap 1 or 2 or changing contexts 4 Fuel Tank Swap (reduced cost) Change fuel mass 1 or 2 – Passively robust 5 Change Capability Change Capability 1 or 2 Change fuel mass – Changeable (e.g. 6 Refuel in Orbit 2 adaptable, resilient) (no redesign) • Change options can allow sustained value at additional cost – Must be coupled with a Epoch Shift defined strategy or policy for how they will be used seari.mit.edu © 2016 Massachusetts Institute of Technology 11 Single Epoch Analysis • Large numbers of design candidates generated by enumerating combinations of design variables and evaluating using system performance models • Goal is equivalent to traditional tradespace exploration: identify Pareto efficient solutions seari.mit.edu © 2016 Massachusetts Institute of Technology 12 Multi Epoch Analysis • Goal of multi epoch analysis analogous to that of robust design – Identify designs that remain close to the Pareto front across many epochs through passive robustness or changeability • Internal mental representation of 3 Designs within 0% (FPN) of Pareto optimal in 87.5% of enumerated epochs all data not strictly necessary • Even with clever visual 3 Designs within 1% (FPN) of Pareto optimal in 100% of enumerated epochs encoding, visualizations that show all data could incur additional cognitive load for the user rather than reduce it 5 Designs within 1% (FPN) of Pareto optimal in 87.5% of enumerated epochs seari.mit.edu © 2016 Massachusetts Institute of Technology 13 Multi Epoch Analysis Bar Charts Row Charts • Click/Drag on charts to examine • Multi epoch analysis on only certain values of a data dimension • Grayed out values are filtered out of all a broader range of data other coordinated dimensions possible views Design Design Variables • Combination of online Cost analytical processing Change (OLAP) and binned Expense Trace aggregation techniques Pareto used for more degree - Filtered sophisticated Out interactive applications Designs Available seari.mit.edu © 2016 Massachusetts Institute of Technology 14 Value sustainment over time • Long-run value is a measure of performance for a design-strategy pair over a time period (era) • Long-term strategies – Maximize utility (U) – Minimize lifecycle cost (Acquisition, M&S) – Maximize efficiency (FPN) • Metrics of interest (MOI) may be defined on any criteria of interest • Era-level metrics defined to for the stakeholder capture the time-dependent nature – Performance attribute of value – Utility or value – Time-weighted average – Efficiency – Variability – Cost – Expedience – Greatest rise / fall – Range seari.mit.edu © 2016 Massachusetts Institute of Technology 15 Single Era Analysis • Goal is to evaluate value of Single-Era Interactive Filtering Era metrics: design-strategy pairs for a 1.Expedience single sequence of future 2.Variability epochs 3.Average 4.Greatest D 5. Range • Coordinated visualization for interactive filtering based on 5 Filtered subset of designs based era-level metrics that evaluate on era temporal aspects of value performance delivery Change Path Dependency Analysis • Interactive chord diagram for visualizing change option usage and transition behavior