Deterministic Ethernet for Real-Time and Critical Applications Wilfried Steiner [email protected]
https://at.linkedin.com/in/wilfriedsteiner www.tttech.com
© NASA © Boeing 787 NASA Orion Vestas Wind Turbines
Reliable Networks and Safety Controls from TTTech
Thales Railway Signalling
Audi Piloted Driving Prinoth Leitwolf www.tttech.com
2 Virtualization of Control
Fully Integrated Safety Control Functions
Real-Time Machine-to-Machine Communication
Unconstrained Access www.tttech.com to Data From Machine to Fog / Cloud 3 • 3 Content • Deterministic Networking • Synchronized Communication • Non-Synchronized Communication • IT/OT Convergence • Recap of Audio/Video Bridging (AVB) • IEEE 802.1 Time-Sensitive Networking (TSN) • Complementing Deterministic Functions • Conclusions www.tttech.com 4 Deterministic Networking
5 Switched Ethernet • Today we are mostly concerned with switched Ethernet • Switches (correct technical term according to the IEEE 802.1 standards is “bridge”) • End Systems (or end points / end stations) • Sometimes Switch and End System are integrated
www.tttech.com
6 Ethernet Frame Format
7B 1B 6B 6B 4B 2B 42B – 1500B 4B 12B 802.1Q MAC MAC Ethertype/ Preamble “VLAN” Payload FCS IFG SOF Destination Source Length Tag
16 bits 3 bits 1 bit 12 bits Tag Protocol Priority Code Drop Eligible VLAN Identifier Point Indicator Identifier
www.tttech.com
7 Example Network 1 EN 3 4 5 EN EN EN
1 2 3 4 5 SW SW SW SW SW
2 EN 8 7 6 SW SW SW Physical Topology (Ethernet Links) Logical Topology Port 8 7 6 www.tttech.com EN EN EN
8 Switch overview 4 Traffic Policing Switching Traffic Shaping o Deterministic Fabric Queue of Ethernet Switches
Traffic Class 0 n o i t c e l e S
Queue of n
Traffic Class 1 o o Embedded i s s i
. Switch Chips m . s . n a r
Queue of T Traffic Class 7 o Deterministic Ethernet IP Solutions www.tttech.com
9 Deterministic Network Objectives • Known upper bounds on latency through the network • Can be achieved by • synchronized communication (TT) • constrained unsynchronized communication (e.g., RC)
www.tttech.com
10 Synchronized Communication (TT) • Time-Triggered Communication is build on two principles • Synchronized global notion of time • Communication schedule • Traditionally this is calculated and distributed offline as part of a device configuration
www.tttech.com
11
IC Synchronized Communication N
IC N
IC S N W IT S C W H I TC H IC N
IC N IC N
IC N IC NX
S Synchronous Communication W IT C IC H N
X
IC N Exactly one order of messages Mi IC N (in contrast to PERM(Mi) in async. comm) Synchronized Global Time • A protocol has to be used to synchronize the local clocks in the switches and end systems. • There are various protocols defined in the academic literature as well as in industrial standards. • Three important synchronization protocols are: • SAE AS6802 • IEEE 1588 • IEEE 802.1AS www.tttech.com
13 Synchronization Protocols (cont.) • Synchronization protocols synchronize the local clocks, which means formally, they ensure that: • at any point in real time, when the system is synchronized, then the difference of clock readings of any two non-faulty clocks in the system does not deviate more than a defined value (which we call the precision)
www.tttech.com Late Clock Perfect Clock Early Clock 14 Synchronization Protocols (cont.) • Synchronization protocols differentiate themselves with respect to how well they synchronize: • What is the precision (ms, us, ns, sub-ns)? • Is the precision probabilistic or deterministic? • What failures are tolerated (fail-silent, Byzantine)? • Are startup/recovery protocols defined? • How fast do they converge from an unsynchronized to a synchronized state? www.tttech.com 15
Communication Schedule • Message transmissions from different end systems and/or switches are separated in time to avoid congestions. • For examples, if two messages need to traverse the same switch-to-switch link (aka multi-hop link) then the senders of the two messages can agree to send their messages at different times.
www.tttech.com
16 Communication Schedule (cont.) • This scheduling problem becomes quite complex because of: • number of messages to be scheduled • size of the network • efficient network utilization • dependencies between messages (e.g., “m1 has to be sent x usec before m2”)
www.tttech.com
17 2 1
5 3 Dataflow Links are enumerated 4 6 on the x-axis
1 2 …
X
www.tttech.com
18 Communication Schedule (cont.) • Different strategies exist to solve the scheduling problem: • Genetic algorithms (like simulated annealing) • Usage of general purpose search tools: • Integer Linear Programming (e.g., CPLEX) • Constraint Programming • SMT Solving
www.tttech.com
19 Communication Schedule (cont.) • The problem of finding a communication schedule is not necessarily an optimization problem (i.e., search for the best solution out of a set of valid solutions) • Indeed it is in most use-cases a satisfiability problem (i.e., search for one existing solution) • E.g., upcoming TTTech tools will move from genetic algorithms towards SMT-based solutions. • Other tools are under research. www.tttech.com
20 Unsynchronized Communication IC N
IC N
IC S N W X IT S C W H I TC H IC X N
IC N IC N
IC N IC NX
S W IT C IC H N Asynchronous Communication . Transmission Points in Time are not predictable IC N Transmission Latency and Jitter accumulate IC N Number of Hops has a significant impact Deterministic Unsynch. Traffic Rate-Constrained Traffic (RC)
S w itc h/R er o iv ut e e ec r R
r de en S
min. duration min. duration min. duration Deterministic Unsynch. Traffic • Sophisticated tools are necessary (but available) to calculate the latency/jitter/buffer for unsynchronized traffic, e.g.: • Network Calculus • Trajectory Approach • Response-Time Analysis
www.tttech.com Information Technology (IT) Operations Technology (OT) Convergence
24 IT-OT Convergence • Information Technology: (office) Ethernet, SDN, Data Centers, Internet, high throughput, performance, etc. • Operations Technology: embedded systems, cyber- physical systems, real-time, fault-tolerance, robustness, etc. • Information Technology and Operations Technology converge • Motor that drives smart* developments www.tttech.com OT – Ethernet Variants • Commercial products: PROFINET, EtherCAT, Ethernet Powerlink, Ethernet IP, AFDX, etc.
• Academic results: Flexible Time-Triggered Ethernet, Time-Triggered Ethernet, switched Ethernet, ...
www.tttech.com
26 Two Main Drivers for Convergence • Industrial Automation: • Industrial Internet of Things, Industrie 4.0, etc. • Automotive Industry: • ADAS, etc.
Standard Ethernet becomes more and more ready for OT use. www.tttech.com
27 Organizations involved • IEEE 802.1: a working group within which the IEEE 802.1 TSN task group is operational – here switch functions are being defined • AVnu: industry consortium that defines inter- operability for AVB and TSN products; different profiles are defined depending on application area • Open Alliance: has distributed Broad-R-Reach and is now capturing Automotive switch requirements www.tttech.com
28 IEEE 802.1 AVB Summary • 802.1AS: clock synchronization protocol. • 802.1Qat: Stream Reservation Protocol (SRP). • 802.1Qav: Forwarding and Queuing Enhancements for Time-Sensitive Streams • 802.1BA: definition of profiles for AVB systems.
The AVB projects have been published in 2011 ! www.tttech.com
29 IEEE 802.1 TSN Introduction • IEEE 802.1 working group maintains several task groups • IEEE 802.1 Time-Sensitive Networking (TSN) is one of these task groups (others are, e.g., security) • For some projects IEEE 802.1 closely interoperates with IEEE 802.3 which maintains and extends the Ethernet PHY and MAC standards.
www.tttech.com
30 TSN projects (#10) overview
• .1AS-rev: synchronization improvements • .1Qbv: time-triggered queues • .1Qbu: frame preemption and resumption • .1CB: stream identification and redundancy management • .1Qca: redundant route configuration • .1Qcc: configuration and SRP improvements • .1Qch: cyclic queuing and forwarding • .1Qci: per-flow policing and filtering • .1Qcr: asynchronous traffic shaping • .1Qcs: improved reservation/registration protocol (a.k.a. MRP++) www.tttech.com
31 TSN projects status (Mar/2016)
www.tttech.com
32 .1AS-rev • Improvements to the .1AS synchronization standard. • Support for multiple grandmaster clocks • Support for multiple routes through the network • Merge of the time from the different grand masters is not standardized (for now) • One-step clock • Fine-Time Measurement for Wireless www.tttech.com
33 .1Qbv: time-aware shaping Time-Aware Shaping
Synchronized Clock Schedule T00: Oc...O T01: cO...O …
Queue of Gate
Traffic Class 0 n o i t c e l e S Switching Queue of Gate n
Traffic Class 1 o Fabric i s s i . m . s . n a r
Queue of T Gate Traffic Class 7 www.tttech.com
34 .1Qbu: frame preemption and resumption • Ongoing frame transmission can be interrupted. • Differentiation between preemptable and preemptive frames by traffic class. • Preemption generates framelets. • Minimum Ethernet frame size is respected, as a consequence a frame (or remaining frame) of 127 bytes cannot be preempted. • Standard PHYs can be used. www.tttech.com
35 .1CB: stream identification and redundancy management • Defines which bits to use for the identification of a stream (flow) • Defines a frame redundancy management algorithm similar to HSR/PRP – i.e., how to merge multiple redundant copies of the same frame into a single frame. • Redundancy information is carried in a Redundancy Tag. www.tttech.com 36 .1Qca: redundant route configuration
• Based in the Intermediate Station to Intermediate Station (IS-IS) protocol. • Allows to configure multiple routes through a network.
www.tttech.com
37 .1Qcc: configuration and SRP improvements
Centralized • Defines managed objects for User Configuration TSN (in YANG). UNI • Remote management (e.g., Centralized Network NETCONF) support Configuration • User/Network Interface (UNI)
• Also defines SRP Talker Listener improvements. Bridge A Bridge B Bridge C Remote Management Protocols www.tttech.com
38 .1Qci: per-flow policing and filtering
• defines both, unsynchronized and synchronized traffic policing schemes • frames that violate policing rules may be dropped or re-prioritized
www.tttech.com
39 .1Qch: cyclic queuing and forwarding • takes the frame arrival point in time (according to .1AS) into account when determining the output queue of the frame 4 Traffic Policing Switching Traffic Shaping Fabric Queue of
Traffic Class 0 m1 receivedn at time t1 o i t c e l e S
Queue of n
Traffic Class 1 o m1 receivedi at time t2 s s i . m . s . n a r
Queue of T Traffic Class 7 www.tttech.com
40 .1Qcr: asynchronous traffic shaping • better known as the “urgency-based scheduler” • driven by General Motors • targets at an improved asynchronous shaper with good real-time guarantees • uses elements from EDF scheduling for transmission selection
www.tttech.com
41 .1Qcs: improved reservation/registration protocol
• driven by key industrial players • improved SRP (MRP) protocol for decentralized TSN configuration
www.tttech.com
42 Complementing Deterministic Functions
43 Background: Time-Triggered Ethernet „This paper presents the rationale for and an outline of the design of a time-triggered (TT) Ethernet that unifies real- time and non-real-time traffic into a single coherent communication architecture.“
Kopetz, Hermann, Astrit Ademaj, Petr Grillinger, and Klaus Steinhammer. "The time-triggered ethernet (TTE) design." In Object-Oriented Real- Time Distributed Computing, 2005. ISORC 2005. Eighth IEEE International Symposium on, pp. 22- 33. IEEE, 2005. www.tttech.com
44 Background: TTEthernet
ARINC 664 + TT + Standard Ethernet www.tttech.com
45 Deterministic Ethernet Evolution Deterministic Ethernet TTEthernet TTP Products Prototypes Safe IT-OT Convergence Mixed-Criticality Paradigm Time-Triggered Paradigm Research www.tttech.com TTTech’s Deterministic Ethernet Implementation in Detail AVB (Audio Video Bridging) TSN (Time Sensitive Networking) TTTech’s Deterministic Ethernet Implementation
IEEE IEEE IEEE IEEE Fully 802.1D 802.1Q 802.1AS 802.1TSN Scheduled Networks Layer 2 VLAN Aware IEEE 802.1AS IEEE 802.1Qbv Switching Bridge Software Stack Time-Aware Shaper Packet Priority Hardware IEEE 802.1Qbu SAE AS6802 (QoS) Timestamping Preemption Fault-Tolerant Clock Sync. 802.1Qat Stream IEEE 802.1CB Reservation Redundancy
802.1Qav Credit IEEE 802.1Qci Time-Triggered -Based Shaper Policing Shaper
…
47 SAE AS6802 • Fault-tolerant synchronization protocol • End systems may fail arbitrarily • Switches may only fail inconsistent-omission faulty • Definition of the Commander/Monitor design • Targets at micro-second-level precision • Formally proven through model checking • Implemented fully in VHDL www.tttech.com
48 Fault-Tolerant Clock Synchronization
E Grand Master TT
E TT
E TT T TE IN 1 T TE
th E
E TT
TT TT E Grand Master E E TT
E TT E TT E TT Grand Master
15 Grand Master 88
88 Fault-tolerant synchronization services 15
th are needed for establishing a safe and E highly available synchronized time. Time-Triggered Shaper
www.tttech.com
50 Time-Triggered Shaper (cont.) • Messages are assigned to queues only at pre-defined points in time (according to a schedule) • Finer granularity of the timing of forwarding decisions of synchronized messages • E.g., synchronized messages can simply be re- ordered in the switch.
www.tttech.com
51 Integrated Dataflow Example TT BE TT BE TT BE t Dataflow – Integration 3ms cycle 3ms cycle 3ms cycle - Time-Triggered (TT)
r de - Rate-Constrained (RC) en S S 1 w itc h/R er o - Standardiv Ethernet (BE) ut e e ec r R
r de en S 2
TT TT RC BE TT TT BE BE TT RC TT TT BE t
TT BE BE TT RC TT BE 3ms cycle 3ms cycle 3ms cycle t 2ms cycle 2ms cycle 2ms cycle 2ms cycle 2ms cycle 2ms cycle 2ms cycle
6ms Cluster Cycle
TTEthernet Switches are non-preemptive store-and-forward switches using priorities Conclusions
57 Conclusions • Standard IT equipment will become more and more usable as operations technologies • For the Automotive and the Industrial markets, TSN- based products are becoming increasingly relevant • Existing Ethernet Variants will remain, especially in niche markets like Aerospace (e.g., TTEthernet will remain standard product for the Aerospace Market) www.tttech.com
58 Conclusions (cont.) • TSN and IT-based solutions are not a “silver bullet” for all industrial communications needs. • Significant challenges remain, e.g., combined wired/wireless, configuration flexibility, ... • New challenges arise, e.g., security, configuration of large-scaled synchronized networks, ... • Close cooperation of academic research and industrial practice has high relevance. www.tttech.com
59 Vienna, Austria (Headquarters) USA Japan China Phone +43 1 585 34 34-0 Phone +1 978 933 7979 Phone +81 52 485 5898 Phone +86 21 5015 2925-0 [email protected] [email protected] [email protected] [email protected] www.tttech.com Copyright © TTTech Computertechnik AG. All rights reserved.