Test & Evaluation of Autonomous Systems A
Total Page:16
File Type:pdf, Size:1020Kb
A “Get Well” Program for Test & Evaluation of Autonomous Systems by Jack Ring Fellow, INCOSE Member, Educe LLC for San Diego Chapter Mini-Conference 2010 Copyright © 2010 by Jack Ring. Published and used by INCOSE with permission. You don’t know Jack? • Worked 1957-1959. Played 1960–present. Built many systems. Some OK. • System T&E – Atlas Missile Radio Guidance System. Flight Test Conductor verifying operational site readiness. Project Golden Ram. • System Engineering – GE Range Safety and Information System, Sports car instrumentation, Dynamic Automated Retrieval Technique, Test Range Automated Planning and Scheduling Syste, Intelligence collection, analysis and production system, Computer-integrated manufacturing, Automotive performance systems, General Purpose Set Theoretic Processor, Software engineering, Systems engineering, and Intelligent enterprises. • Action research and mentoring – High Tech startups and turn-arounds – Object technology – Advancing systemics and systems praxis. • INCOSE (Fellow), ISSS, IEEE, ACM, etc. • UAST ‘subject matter expert’ 9/15/10 [email protected] Copies permitted with attribution 2 Agenda • Problematic Situation: Systems, Autonomous Systems – Suitability, Effectiveness, Safety, Survivability, SESS – New ways to excel --- or fail. – Lack of warfighter-trusted knowledge. • Reconceptualizing the solution: – T&E: What got you here won’t get you there. – Non-deterministic systems. – Knowledge: Acquire, Interpret, Produce, Convey, Utilize, KAIPCU • Get Well: – 1. Soldier systemists driving intelligent evolutionary acquisition. – 2. Prototypin. – 3. Refurbishing the knowledge workers. • Questions & Recommendations. What do we mean by System? e = entity e = relation S R = behavior = π e e e = system e S = Stimulus, R = Response e = system (explicit) e e e Entity can contain e e e = e a system = system (implicit) e e e e e e e e e e = system e e e e System of Systems (soft) 9/15/10 [email protected] Copies permitted with attribution 4 Three kinds of System of Systems Dedicated e e e S e S1 S2 e e S3 e Shared e e e e S e e S1’ S2’ e e Concurrent S3’ e e e e e S e e POSIWID: The purpose of a system is what it does, regardless of the intent of sponsor, designer, user or operator. 9/15/10 [email protected] Copies permitted with attribution 5 Systems Praxis Problem System; System Content, Process, 1. Behavior System Context Problem 2. Suppression System 3. Problem Suppression System: Content t2 Problem Suppression 4. System: Content, Adapted from t3 Science of Generic Design, t1 Process, Behavior John Warfield 9/15/10 [email protected] Copies permitted with attribution 6 Some Categories of Systems Situation Class Class Value Problem S π R Type Space Space Type f(t) f(t) Value Type π = f(k) = ballistic Class π = f(O) = governor Pr = Problem Space π = f(I) = anticipatory Val = Value Space π = f(Sit, O) = homeostatic S = Stimulus R = Response π = f(Val) = goal-seeking Sit = Situation π = f(Pr) = self-organizing π = Nth Order OODA Loop π = f(Pr, Val) = autopoietic π = f(all) = autocatalytic 9/15/10 [email protected] Copies permitted with attribution 7 Autonomous Systems • An autonomous system is characterized by its ability to: – Perceive the environment and itself – Reason based on perception, to achieve goals within constraints – Act to achieve goals within constraints ISS control room Stanford Robotics Lab Autonomous systems are created through the integration of various disciplines … Mission planning, guidance & control algorithms, human-system collaboration, communications, etc. …and can be anywhere along the mobility Orbital Express System spectrum Vehicle -> articulated robotic arm -> habitat management system MCM 8 PATFrame Mini-conference, MIT, August 24, 2010 Autonomous Vehicles [email protected] Copies permitted 9/15/10 9 with attribution The Warfighter’s Whole System Context mission profile Operator initialization informs Preparation System Anticipated Problematic Payload Actual Production Sys & Problematic Problem Vehicle Situation System Situation Operational Availability System Test System(s) 9/15/10 [email protected] Copies permitted with attribution 10 Systems Praxis Problem System; System Content, Process, 1. Behavior System Context Problem 2. Suppression System 3. Problem Suppression System: Content t2 Problem Suppression 4. System: Content, Adapted from t3 Science of Generic Design, t1 Process, Behavior John Warfield 9/15/10 [email protected] Copies permitted with attribution 11 Autonomic Actions Adjust: Gradients on relationships Arrange: Pattern of relationships Co-align: Content of system with context and resources. Within Dynamic and Integrity Limits X, d(X)/dt, d2(X)/dt2 •Thermodynamics: mass, momentum and energy •Informatics: data, information and knowledge •Teleonomics: skills, rate of learning, and rate of invention •Human social dynamics: trust, enthusiasm, co-evolution •Economic: Investment, ROI, Liquidity •Ecology: Waste, Fads, Unintended Consequences 9/15/10 [email protected] Copies permitted with attribution 12 Autonomous Engagement Categories Non-deterministic Probabilistic Deterministic Behavior is Locus of behaviors is Limits of integrity and predictable from predictable from knowing stability are estimatable knowing past content, relationships and from knowing whether trajectory of envelope of inputs. ruleset and resources can states and current always achieve requisite inputs. variety. 9/15/10 [email protected] Copies permitted with attribution 13 Anticipating SESS? • SESS may be treated as the outcome of a game between an angel and demons. Ideally our champion: the angel, can marshal, allocate and schedule resources to prevail. The demon may react to all the angelic moves attempting all possible tricks to defeat the angel. That means the angel must be able to counter continuously any new demonic action change her strategy and behavior in response to demonic actions 9/15/10 [email protected] Copies permitted with attribution 14 Angel Requisite Variety • The angel is supposed to be able to exhibit requisite variety for duration of the engagement. • If the environment is finite (the number of potential interactions with the angel is chosen from a finite set although they may occur erratically) it is possible to reach a Nash equilibrium wherein angel and demon use their respective best strategy thus no strategy change is needed. • If the environment is unbounded the game cannot reach a Nash equilibrium, therefore the angel may be defeated. • A System of systems contains a web of angels and a web of demons. • Our job is the discover the weak angel w/r warfighter situations. 9/15/10 [email protected] Copies permitted with attribution 15 Reconceptualizing The Solution Requisite Variety C>B>A Requisite Variety B>A Requisite Variety A [5a] [7] Adjust Descriptive Models Adapt UAS(1) Co-align designs [3] [2] [4] & Autonomous generates Autonomous UAS(2) T&E System T&E exercises Enterprise UAST(i) & (n.m) [6] produces observes T&E & conveys UAS(n ) Descriptive Assets Knowledge Models [1][1] [5b] 5000.02 Quick Reaction Missions Warfighters Oversight Programs Programs (1 … k) 9/15/10 [email protected] Copies permitted with attribution 16 Generic “T&E” Capabilities Environment Blue Force Red Force Knowledge Needs Observe Orient Decide Act Safety Effectiveness Agility Suitability Survivability Evaluation DOE & Plan Acquire and Interpret Produce and Convey Clarify Assess Constructive Virtual Live in situ Test Operations Test UAST Readiness Safety Protocol Sensor Data Quality Design Preparation Assessment Assurance Mgmt Mgmt Mgmt data, session & presentation management Staff interoperability infrastructure Associated Competencies communications & networking Authorities 9/15/10 [email protected] Copies permitted with attribution 17 It’s the LIMITS, Stupid Strategy The allocation and scheduling of resources sufficient to overcome impediments to objectives • Dynamic Limits – pursuit • Dynamic Limits – morph • Integrity Limits – recoverable • Integrity Limits -- unrecoverable The warfighter needs to know, and trust, the Suitability, Effectiveness, Safety, Survivability limits [email protected] Copies permitted 9/15/10 18 of each autonomouswith attribution system. Design for Prevention Precludes Unintended Consequences [email protected] Copies permitted 9/15/10 19 with attribution The “get well” paradigm shift 25 T&E Paradigm KAIPCU Paradigm 20 Shift Your 15 Paradigm! 10 1. Personnel Notional Levels Notional 2. Prototyping 5 3. Parsimony 0 9/15/10 [email protected] Copies permitted with attribution 20 Get Well 1: Personnel Objective: Bring a 12 person DOD cadre up to systemist level of competency for each kind of UAS (Space, Air, Ground, Marine, Undersea). Will lead prototyping. Will become exemplars for expanded cadre. Action: a) Interactive Management session to identify and stratify the UAS problematic situation, formulate a variety- sufficient response, and compose a scenario of systems architecting, engineering, and learning. b) Perspective session: SDOE 678 and 683 courses by Stevens Institute of Technology, School of Systems and Enterprises, or equivalent. c) Perspective session: Design for Prevention paradigm 9/15/10 [email protected] Copies permitted with attribution 21 Get Well 1: Personnel – MOE’s • Systemists acknowledge that the challenges are not about test and evaluation but about discovering the expected SESS of unmanned autonomous systems of systems in anticipated, albeit ambiguous situations. • Systemists