Spacecraft Architecture Based on Deterministic Ethernet

Ensuring Reliable Networks

SPACECRAFT ARCHITECTURES BASED ON DETERMINISTIC ETHERNET Pasadena, 17th Dec 2014

Mirko Jakovljevic, Christian Fidi [email protected] [email protected]

December 17th, 2014

Bombardier Boeing 787 CSeries

Distributed Embedded Platforms and NASA Orion

Deterministic Networks from TTTech Airbus A380 Embraer Legacy 450 / 500

Audi A8

ISO 26262 IEC 61508 EN 13849 DO 254/178 IEC 60601 Automotive Industrial Off-Highway Aerospace IEC 62304 Medical

Application- App App App App App Specific

Functions

SW SW

Abstraction

FT FT Services

Middleware Middleware Middleware Middleware Middleware Platform

Platform

Layers

/ /

RTOS RTOS RTOS RTOS (RT)OS /

Integration

Interfacing System

Ethernet Backbone

Requirements for deterministic networks • More bandwidth alone does not solve QoS challenges • Known (maximum or fixed) end-to-end latency • Bounded and small jitter • Proper peak-load handling • Proper handling of delays and faults in communication • Objective: Manageable design of integrated embedded systems and critical functions

www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 4 Interfacing in Advanced Integrated Systems: Different Traffic Classes / Different Functions on Shared Resources Ensuring Reliable Networks

Alarm/Protection Periodic RT Functions or Media Critical Event Streams Messages (Traffic Bursts)

Soft-Time Periodic RT Functions Control Loops Integrated AperiodicTraffic and Traffic Bursts hard RT & RT System * Typically not integrated with Platform critical functions (Deterministic Periodic Data Streams Ethernet / TTEthernet)

Computing in Integrated Systems Ensuring Reliable Networks • Resource sharing among different integrated functions shall be periodic and non-blocking – Failure of one function shall not influence other functions or generate resource starvation and deadlocks – Resource sharing shall be carefully planned at design time, but in „embedded clouds“ unknown „unknowns“ may exists IMA (Integrated Modular Avionics or Arch.): • closed system, safety-/time-critical, real-time, deterministic • Need to know about all functions and their resource requirements: critical and non-critical, to setup the system

Mixed Criticality Systems Distributed IMA / „Embedded Cloud“ Computing: Reconfigurable Open Generic Architecture • (ideally) open system, generic architecture Integrated Modular Architectures with Hard RT • time- and/or safety-critical, hard RT, deterministic for critical function Critical IoT Infrastructure / Advanced C4 Systems • Need to know only about critical functions in the system, to Scalable RT / HPC Computing setup the system

and Network Capabilities Ensuring Reliable Networks …Determine complexity!!! Network capabilities

Model Of Computation and Communication

Distributed SW Platform Architecture Design around Design the limitations of supported MoCC and network capability Application SW (Function) Design Methodology

System Lifecycle Costs System Lifecycle Costs

A family of frame-based standards for LAN/MAN networks by IEEE 802 Statistical • Standard physical medium Multiplexing (Asynchronous • Set of medium access Communication) control rules with fair arbitration • Variable size packets Priority-driven Best Effort in Ethernet format VLAN Traffic Traffic (802.1Q)

• IEEE 802.1 focuses on Layer 2 QoS enhancements (traffic classes)! 802.3 focuses on bandwidth growth! • Ethernet capabilities change over time!  Not a monolithic standard! • Ethernet device datasheets provide the list of supported functions and standards! www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 10 SAE & ARINC Standards for

Critical Ethernet Networking Ensuring Reliable Networks SAE International

(over 36000 standards and 138.000 members) Networking standards: ARINC429, ARINC629, ARINC659, ARINC664(AFDX), SAE AS15531 (MIL-1553), ARINC825 (CAN), SAE J1939 (CAN), SAE AS5643 (Firewire), SAE AS6003 (TTP), SAE AS6802 (TTEthernet), SAE AS 4075A (HSRB), SAE AS5659 (WDM LAN), SAE AS5653A (MIL-1760) …

Typically SAE provides original networking standards, or network services / profiling to 3rd party (e.g. IEEE, …) networking standards to enable their application in critical infrastructure and integrated system applications.

SAE ITC Aviation Industry Actvities SAE Standards (ARINC Standards) Focus Aerospace/Space/Defense/Automotive/ Commercial vehicle / Integrated Systems and Focus Commercial Aviation / Integrated Systems and Architectures Architectures / Datalinks Driven By: Aerospace / Automotive / Transportation Driven By: Airliners / Aerospace Industry Industry

(Released 2011)

(In work since 2012)

Applications & IEEE Ethernet Ensuring Reliable Networks

Time-triggered extensions for standard switched Gigabit-Ethernet • Startup

• Recovery • Robust fault-tolerant distributed clock • Foundation for design of scheduled /synchronous traffic class)

Makes Ethernet viable for safety-critical distributed applications!

System time available on switches and end stations • Scheduled traffic can have fixed latency and µs-jitter • Switch knows when the message is forwarded

By controlling jitter we also minimize latency for critical streams A large portion of latency in time- sensitive rate-constrained communication is the jitter!

Bandwidth Partitioning Ensuring Reliable Networks • ARINC664 and SAE AS6802 QoS Layer 2 rely on virtual links (VLs) with defined QoS and timing performance • VLs emulate point-to-point connections in integrated architectures • ARINC: max. latency per VL, SAE: fixed latency per VL

E/S E/S E/S E/S

VL1 VL22 VL21 V3 V3 Ethernet VL1 VL22 Network VL1 VL1 VL4 VL1 VL4

VL21 E/S E/S E/S E/S ...... E/S E/S E/S E/S Note: • Design of critical integrated systems not viable without VLs (VLANs cannot do the job!) • Synchonous VL (prereserved bandwidth not used if no message sent) • Asynchronous VL (require permanent reservation) www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 15 Distributed Fault-tolerant

Synchronization Ensuring Reliable Networks Robust algorithm based on exchange of asynchronous IEEE 802.3 messages

Synchronizes local clocks – system time (!) • no wall clock (external time source - e.g. GPS) required

Fail-operational: • tolerates multiple faults • tolerates byzantine synchronization faults • no search for best master (distributed clock!)

Synchronization Ensuring Reliable Networks

Protocol Control Frames called “Integration Frames” are used to perform all synchronization functions. They are transmitted accordingly: The Synchronization “Masters” Sync Comp send Integration Frames at the beginning of each Integration Sync Comp Cycle. The timing of these frames is used for the “voting” Sync Comp

The Compression “Masters” send Sync Comp Integration Frames to everybody, timing them in a special way so Sync Comp that everybody can correct their clocks. Sync Comp

Devices (Aerospace) Ensuring Reliable Networks

AFDX (ARINC664) and TTEthernet (SAE AS6802) Network Devices – Switches + Endsystems

IEEE IEEE ARINC664 SAE 802.1D 802.1Q Part 7 AS6802

ARINC664 Time- Triggered VLAN Aware Virtual Links Virtual Links Bridge (Asynchronous VL) (Synchronous VL) Layer 2 Switching Fault-Tolerant Packet Priority Policing Clock (QoS) Synchronization

Best Effort Asynchronous Synchronous

System Integration Capability Ensuring Reliable Networks

Distributed System with Sensors, Actuators (Effectors) and Hard Real-Time Control Loops

Partitions Partitions Module 1 Module N App1a App1b App2 AppN T T T T T T T T T T T T T T a a a a a a a a a a a a a a T T T T T T T T T T T T T T T T T T T T T T T T T T T T s s s s s s s ... s s s s s s s a a a a a a a a a a a a a a a a a a a a a a a a a a a a k k k k k k k k k k k k k k s s s s s s s s s s s s s s s s s s s s s s s s s s s s k k k k k k k k k k k k k k k k k k k k k k k k k k k k IPC & Platform Abstraction Middleware (A)

System-level IPC AND/OR Deterministic Network (B) Shared System Memory Distributed Embedded Computing Platform

• (A) Platform Abstraction Middleware and Services • separating application from architecture, simplifying distributed application design • (B) Deterministic Network • real-time communication guarantees, bandwidth partitioning and congestion management • defined interaction, interfacing and separation among different distributed functions • inter-partition communications (IPC) among different modules OR shared memory emulation www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 20 Key Design Ingredients: Virtualization in

Advanced Integrated Systems Ensuring Reliable Networks

MoCC: Model of Computation/ Communication (TTA and L-TTA)

Network: Robust Computing: Bandwidth Time/Space- Partitioning And Partitioning Virtual Links (VLs)

www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 21 MoCC (Models of Computation and Communication) support the integration of critical functions on shared embedded resources

Ensuring Reliable Networks

L-TTA (asynchronous model, but works on local time) • Cyclic/periodic processing of data based on local clock/timer • For RT control loops, defined max latency required (asynchronous VL!) • Enable deterministic control loop performance viable – limits on hard RT performance • Application Domains: Aerospace IMA/Railway Signalling/Nuclear • Note: GALS is L-TTA with several partitionins per LRM

TTA (synchronous model, works on system time) • Cyclic/periodic processing of data based on system time • For simple integration of hard RT control loops, fixed latency (synchronous VL!) • Full hard RT performance • Application Domains: Aerospace/Space/Automotive/Railway-Rolling Stock www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 22

Virtual links for Robust Bandwidth Partitioning:

Impact on System Architectures Ensuring Reliable Networks

Asynchronous Synchronous VL VL

Supports Supports TTA and L-TTA L-TTA Enables „Embedded Cloud“

System-Wide Hard RT Performance due to fixed latency Real-Time Performance (SW function separated from controlled object)

Closed Systems Open Systems (predefined critical and non-critical function (predefined critical function performance, performance) arbitrary non-critical performance)

TDMA Partitioning Ensuring Reliable Networks

Equivalent to physically separated Ethernet subnetworks Embedded system virtualization (time-critical/time-sensitive/soft-time) Allows "slicing" of shared computing/networking resources Design of safety-/time-critical functions in a distributed integrated systems

F4 F4 F4 F3

Partitioned OS F4 Partitioned OS Partitioned OS (e.g. VxWorks ARINC653) (e.g. VxWorks ARINC653) F3 Partitioned OS (e.g. VxWorks ARINC653) (e.g. VxWorks ARINC653)

Windows Partitioned OS PC (e.g. VxWorks ARINC653) Partitioned OS TTE (e.g. VxWorks ARINC653) Windows DISTRIBUTED PC DISTRIBUTED FUNCTION 4 FUNCTION 3 TTE TTE Linux Server F2 IEEE802.3 Ethernet network F2 (Office LAN) F1

F2 F2 Partitioned OS Partitioned OS (e.g. VxWorks ARINC653) F1 F1 Partitioned OS (e.g. VxWorks ARINC653) (e.g. VxWorks ARINC653)

Partitioned OS Partitioned OS (e.g. VxWorks ARINC653) (e.g. VxWorks ARINC653) Partitioned OS Partitioned OS (e.g. VxWorks ARINC653) (e.g. VxWorks ARINC653)

TTE DISTRIBUTED FUNCTION 1 DISTRIBUTED TTE FUNCTION 2 HINT: VLANs do not support the virtualization of time-critical functions! TDMA communication capability is required! www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 24 Industry Trend:

Time/Space Partitioning Ensuring Reliable Networks

• Multiple SW APPs are executed in • Time and Space Partitions on • A high performance, low cost HW (SoCs)

APP APP APP

OS1 Linux OS3

Hypervisor / TSP OS

SoC (CPU, FPGA, MEM, …)

System Level Partitioning Ensuring Reliable Networks

• Bandwith to Memory APP APP APP Partitioning mapping at Bandwith Partitioning at E/S based on VLs Switch Level

Win Linux OS • Redundancy management Hypervisor / ARINC 653

Mem Mem Mem

Strong Partitioning • Bandwidth partitioning at the network level • Bandwidth partitioning supported at the switch and E/S level • Memory partitioning at the E/S Level • Bandwidth to memory mapping at the E/S based on virtual links www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 26 Embedded Virtualization Cookbook for

Ethernet-based Integrated Systems Ensuring Reliable Networks

• Take care of resource use in critical functions Critical Functions (Time-Critical) Non-Critical/Soft-Time Functions • Use IMA methodology • L-TTA/GALS/TTA, Asynch. VL Embedded System Virtualization and/or Synch. VL, Partitioned OS • Ensure integration of hard RT, RT and MoCC (Model Of Computation And Communication) soft-time functions L-TTA / TTA Priority/Event-driven • Mixed L-TTA, TTA model Asynchronous Synchronous • Synchronous VL enables Asynchronous / RT / Hard RT • Full control of jitter and latency • Network devices know exactly Technology Baseline what is going to happen with hard RT and RT traffic Time Partitioning of Computing Priority-Driven VM and Task • Network devices know exactly Resources Execution when the resources are free for soft-time Network Bandwidh Partitioning Statistical Network Bandwidh • Non-critical/soft-time functions with defined QoS (Virtual Links) Multiplexing (VLAN, best effort) can take care of themselves • They use remaining resources Asynchronous Synchronous as background tasks Virtual Links (VL) Virtual Links (VL)

• Designed for N-redundant FT systems (typ. double or triple redundant networks) • Reduce the complexity of functional interactions • Built-in mechanisms for FT synchronization • Autonomous and scheduled operation • Critical function timing defined @ network layer • Influences design methodology and resource scheduling • Simplifies software design, layering and partitioning • Supports full separation and layering of temporal and functional behavior in the system • Enables design of „flat“ Distributed IMA Architectures (generic and scalable architectures) www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 28 Ensuring Reliable Networks

Supporting Slides

Synchronous vs. Asynchronous Ensuring Reliable Networks

Active standby avionics system model with three components… • Synchronous model: 185 reachable states (~2x102) • Asynchronous model & communication with no latency: >3x106 states • Asynchronous model with varying communication latency: The number of reachable states could not be calculated with 8Gb RAM…

8- 10 >10 10 ???

The number of system states in an integrated systems can be very high… And this is still a relatively simple system… https://www.ideals.illinois.edu/bitstream/handle/2142/17089/pals-formalization.pdf?sequence=2 www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 30 Synchronous Alignment: Resource Use & Complexity Reduction Ensuring Reliable Networks Maximize use of network bandwidth and computing resources for critical embedded functions • Ensure unambiguous design of key system interfaces • Reduce uncertainity, jitter and unintended system states (prevents system state explosion) Improve functional alignment (and separation!) • Simplified sensor fusion and distributed processing • Simplified redundancy management • Minimize software complexity / simplify functional alignment

Software Software Application Software Application Application

Software Software Software Middleware / Middleware / Middleware / Application Application Application Platform Platform Platform Abstraction Abstraction Abstraction Middleware / Middleware / Middleware / Platform Abstraction Platform Abstraction Platform Abstraction

Asynchronous Ethernet Synchronous/Asynchronous Communication Ethernet Communication www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 31 Clean Layered Model: Improved Control of

Latency and Jitter (TTA model) Ensuring Reliable Networks Interfaces and temporal behavior defined at network level • Middleware contains parameter-defined communication abstraction and redundancy management (voting) • Application can handle only functional aspects without temporal interdependencies (no busy waiting, watchdogs, semaphores, …) • All behavior related to progression of time, not dependant on HW or SW platform • Supports model-based application design (simple computation tasks!) • All sensors and actuator access synchronized to µs (using simple IO tasks) c

n Interface to physical SYNC LEVEL 3 y s (APP. LEVEL) systems synchronized (simpified sensor fusion)

SENDER RECEIVER

Redundancy Mgmt c n

SYNC LEVEL 2 y (Voting) (MIDDLEWARE) s Comm. Abstraction MIDDLEWARE MIDDLEWARE c n

SYNC LEVEL 1 y (NETWORK) s Network – Temporal behavior for all critical NETWORK NETWORK functions defined here!!! www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 32 Ensuring Reliable Networks

Use Cases

Launchers

Ensuring Reliable Networks

• Ariane 6 (replacement for Ariane 5) : Planned first flight 2020/2021 • Higher integration levels and SWaP reduction, lower physical complexity DASIA 2012

Ensuring Reliable Networks

 Single-fault-tolerance handled in the protocol (network level)  Robust and Highly Reliable Systems  One network configuration – different launcher configurations  Modular embedded platforms  Known latency and minimal jitter for critical communication  Fully deterministic, predictable WCET in complex integrated systems  Fault-tolerant synchronization  Lower software complexity, predictable operation  Ethernet physical layer 100Base-TX  Robust  Seamless integration since the sub-systems are tested with the flight configuration  „Composability“, design and verification in isolation does not create integration challenges  Make use of standard Ethernet for development, testing and operations  COTS based www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. Page 36 Launcher

Application Ensuring Reliable Networks

Ensuring Reliable Networks

 Up to dual-fault-tolerance handled in the protocol (network level)  Higly reliable  Full determinism (known latency and minimal jitter)  Highly deterministic  Full traffic partitioning (combine platform and payload)  Easy access to shared ressource e.g. TSP OS / Integrated Architecture  Fault-tolerant synchronization  Seamless integration since the sub-systems are tested with the flight configuration  Composeability

Purpose Crew Vehicle) Ensuring Reliable Networks

October 23rd, 2013 - 7th ADCSS Workshop, “NASA MPCV Use of Ethernet - Time Triggered Gigabit Ethernet on NASA’s Crewed Exploration Vehicle”, George Eger, LMCO

(5th December 2014) Ensuring Reliable Networks

 “Congratulation to TTTech! It was a fantastic mission and the TTEthernet Data Network worked perfectly! Thanks for all of the support over the years…it was great to see it come together and work so incredibly well.”

Advanced Integrated Systems Ensuring Reliable Networks

Ensuring Reliable Networks

 Globale synchronized time-base (important for platform and payload)  One trusted timebase  Synchronization to GPS (all data is timestamped with a precise absolutestamp)  One absolute timebase  Full determinism (known age of data important for platform an payload)  Allows distributed real-time computing  Full traffic partitioning (combine platform and payload)  Easy access to shared ressource e.g. TSP OS  Seamless integration since the sub-systems are tested with the flight configuration  Composeability  Real-time  Reduced memory needs (no large buffering SRAM necessary for TTEthernet switches and embedded systems)

Ensuring Reliable Networks

Space HW and IP

TTEthernet Common IP Core TTEthernet Space Pegasus IP Core (Automotive, Aerospace, Energy, ...) 2014

TTE-End System Rad tolerant/hard TTE-End System TTE-Switch IP Core Pluto FPGA IP Core Space IP Core Space Space >2016

Rad hard TTE-End System TTE-Switch ASIC Controller Space Controller Space ASIC ASIC

Development Hardware & Design Tools Support & Integration & Customization

Development Systems Support Customization

Integration Software Boards

Flight and Rugged Products Test Equipment & Verification Tools

Chip IP TTESMC TTEEnd System A664 Lab TTESwitch A664 A600 Pro TTESwitch 24 Ports Lab

w w w. tt t e ch .c o m

Asynchronous MoCC

• Resources are reserved for all functions in the system • It can be proven or assumed that no function will use more resources than planned (closed system!) • Strict resource use policing: • dropping data packets violating temporal boundaries • exiting non-compliant processes

Designed to avoid hand-shaking, deadlocks, CMFs • Periodic processing and sensor sampling • Defined maximum latency for all data communication • No temporal interdependancies or synchronism among computing modules and networking devices

Synchronous MoCC

www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved. TTA: Synchronous MoCC (I) Ensuring Reliable Networks Can be seen as a special L-TTA case, with network devices and computing modules in sync • Fixed latency for all data communication • Hard RT computing performance • Distributed (masterless) fault-tolerant system time

Deterministic Ethernet

Deterministic Unified Ethernet Ensuring Reliable Networks

What if synchronous links (VLs) are reserved, but the message is not sent? • … ECU/LRU is not installed • … Function is currently inactive

Dynamic Bandwidth Release: immediate availability for asynchronous traffic

w w w. tt t e ch .c o m