SpaceWire Links Routers and Networks ECSS-E50-12A
Presented by Ph. Armbruster ESA/ESTEC/TOS-ED Data Systems Division [email protected]
Credit – University of Dundee - European Space Agency Contents Part 1 Contents Part 2
§ Introduction § SpaceWire Packets § Physical Level § Routing Switch Concepts § Signal Level – Worm-hole routing § Character Level – Header deletion – Path addressing § Exchange Level – Logical addressing § Packet Level – Priority – Arbitration – Group adaptive routing – Fault tolerance § Networks § Time Codes SpaceWire
§ SpaceWire is a standard for spacecraft onboard data-handling – ESA ECSS-E-50-12A § SpaceWire based on two commercial standards – IEEE-1355 – LVDS (Low Voltage Differential Signalling) § SpaceWire is the result of work by many people in ESA, NASA, the space industry and academia. SpaceWire Key Features
§ High data rate ( > 100 Mbps) § Distance of 10m + § Scalable § Low error rate § Good EMC performance § Low-power § Fault tolerance support § Radiation tolerant components Purpose
§ SpaceWire provides a unified high-speed data- handling infrastructure for connecting together – sensors, – processing elements, – mass-memory units, – downlink telemetry sub-systems, and – EGSE equipment. Purpose
§ The purpose of the SpaceWire standard is – to facilitate the construction of high-performance onboard data-handling systems – to help reduce system integration costs – to promote compatibility between data-handling equipment and sub-systems – to encourage re-use of data handling equipment across several different missions. Scope
§ SpaceWire links are – full-duplex, point-to-point, serial data communication links. § The scope of the SpaceWire standard is the – physical connectors and cables, – electrical properties, and – logical protocols. Scope
§ The SpaceWire standard covers the following normative protocol levels – Physical Level: Connectors, cables, cable assemblies and PCB tracks – Signal Level: Signal encoding, voltage levels, noise margins, and data rates. – Character Level: Data and control characters used to manage the flow of data across a link. – Exchange Level: Protocol for link initialisation, flow control, link error detection and link error recovery. – Packet Level: How data is split up into packets for transfer across a SpaceWire link. – Network Level: Structure of a SpaceWire network, transfer of packets across a network, handling of errors. Physical Level
§ Cables § Connectors § Cable Assemblies § PCB traces Cable Requirements
§ Allowing high data-rate over distances of at least 10m § Cable must have following characteristics – Characteristic impedance matched to line termination, 100 ohms – Low signal-signal skew – Low signal attenuation – Low cross-talk – Good EMC performance Cable Construction
Conductor 28 AWG (7 x 36 AWG) Insulating layer Filler
Twisted pair
Inner shield around twisted pair (40AWG) Jacket Filler Binder Outer shield (38AWG) Outer Jacket Connectors
§ 8 signal wires § Inner screen termination contact § 9-pin micro-miniature D-type
Din+ Sin+ Inner Sout- Dout- Shield
1 2 3 4 5
6 7 8 9
Din- Sin- Sout+ Dout+ Cable Assembly
Dout+ 1 Din+ 9 Dout- 6 Din- 5
Sout+ 2 Sin+ 8 Sout- 7 Sin- 4 GROUND 3 GROUND 3 Sin+ 8 Sout+ 2 Sin- 4 Sout - 7
Din+ 9 Dout+ 1 Din- 5 Dout- 6 Cable Assembly PCB Traces
§ SpaceWire can also be run over PCB traces § 100 ohm differential impedance must be maintained Signal Level
§ Low Voltage Differential Signalling (LVDS) § Signal Encoding - Data/Strobe Encoding LVDS Driver and Receiver
Vcc
~3.5mA
- +
100R Transmission Medium + DRIVER 100R - + - RECEIVER LVDS Features (1)
§ Near constant total drive current – 3.5mA for logic 1 and 3.5mA for logic 0 – decreases switching noise on power supplies. § Good immunity to ground potential difference – between driver and receiver – can tolerate at least ±1V ground difference. § High immunity to induced noise – because of differential signalling over twisted-pair cable. § Low EM emission – because small equal and opposite currents create small electromagnetic fields which tend to cancel one another out. § Not dependent upon particular device supply voltage(s). LVDS Features (2)
§ Simple 100 ohm termination at receiver. § Failsafe operation - – the receiver output goes to the high state (inactive) whenever § the receiver is powered and the driver is not powered, § the inputs short together, § input wires are disconnected. – the transmitter output is current limited § preventing damage when transmitter outputs shorted together § or shorted to ground. § Low power consumption – typically 50mW per driver/receiver pair for LVDS – 120mW for ECL/PECL LVDS Signalling Levels Driver Output Voltage 1 0 1 VB OFF ON OFF
+247 to 454 mV 1.2V typ. O V (diff.) -247 to 454 mV VA
Receiver Input Thresholds
+100 mV +Vi O V (diff.) Transition Region -100 mV -Vi Data Encoding § SpaceWire uses Data-Strobe (DS) encoding – Data values transmitted directly – Clock encoded with data to form strobe – XORing data and strobe recovers clock – provides improved jitter/skew tolerance compared to data/clock encoding § Used in IEEE-1355 and IEEE-1394 standards DATA-STROBE ENCODING Data 0 1 0 0 1 1 0 1 1 0 D S CLK § D and S signals transmitted differentially using LVDS Data Signalling Rate
§ Minimum data signalling rate 2 Mbps § Maximum data signalling rate Depends upon system skew and jitter >> 100 Mbps § Operational data signalling rate Any rate 2 Mbps to Maximum Rx and Tx can operate at different rates Tx rate may be altered at any time after link connection made § Initial data signalling rate 10 ± 1 Mbps Character Level
§ Character level defines: – data characters § used to hold data – control characters § used to manage flow of data across a link Data Characters
P 0 X X X X X X X X 0 1 2 3 4 5 6 7 LSB MSB Data-Control Flag Parity Bit Control Characters
P 1 0 0 FCT Flow Control Token
P 1 0 1 EOP Normal End of Packet
P 1 1 0 EEP Error End of Packet
P 1 1 1 ESC Escape Control Codes
ESC FCT P 1 1 1 0 1 0 0 NULL
P 1 1 1 1 0 T T T T T T T T Time-code ESC 0 1 2 3 4 5 6 7 LSB MSB Exchange Level
§ Exchange Level Protocol defines – Link Initialisation – Flow Control – Detection of Link Errors – Link Error Recovery § Concepts from IEEE-1355 with significant improvements SpaceWire Interface
SpaceWire SpaceWire Interface Interface HOST TRANSMITTER RECEIVER HOST SYSTEM SYSTEM END A END B RECEIVER TRANSMITTER N-Chars and L-Chars
§ L-Chars: Link characters (NULL, FCT) – used in exchange level – not passed on to higher level § N-Chars: Normal characters – Data characters – End of packet markers (EOP, EEP) – Passed up to higher level § Link interface interleaves L-Chars and N-Chars § N-Chars from one packet NOT interleaved with N- Chars from another packet § Received character must have parity checked before acted upon Disconnect
§ Disconnect detection is used for – monitoring physical connection – performing exchange of silence § Receive clock is D XOR S § Monitor time between one receive clock edge and next § If > 850nsec then disconnect timeout § Gives a lower limit to data signalling rate (2 Mbps) § Disconnect detection enabled after first bit received § i.e. after first receive clock edge Transmission Priorities
§ Character transmission is prioritised as follows:- – FCTs - highest priority – N-Chars – NULLs - lowest priority Basic State Machine Reset
ErrorReset Reset Tx [Link Disabled] Reset Rx After 6.4us
Run ErrorWait Send FCTs/NChars/Nulls Reset Tx Enable Rx Enable Rx After gotFCT 12.8us After After 12.8 us 12.8 us
Connecting Ready Send FCTs/Nulls Reset Tx Enable Rx Enable Rx
gotNull Started [Link Enabled] Send Nulls Enable Rx Link Initialisation (1) End A
ErrorReset Reset Tx, Reset Rx End B
After 6.4 us ErrorReset Reset Tx, Reset Rx ErrorWait Reset Tx, Enable Rx
After 12.8 us After 6.4 us
Ready Reset Tx, Enable Rx ErrorWait Reset Tx, Enable Rx Link Enabled After 12.8 us Started Send Nulls Send Nulls, Enable Rx Ready Reset Tx, Enable Rx Link Initialisation (2) End A End B
Started Ready Send Nulls, Enable Rx Reset Tx, Enable Rx
Link Enabled Started NULLs Send Nulls, Enable Rx gotNULL gotNULL
Connecting Connecting Send FCTs/Nulls, Enable Rx Send FCTs/Nulls, Enable Rx FCTs gotFCT gotFCT Run Send FCTs/NChars/Nulls Run Enable Rx Send FCTs/NChars/Nulls Enable Rx Flow Control
§ Flow control used to avoid overflow of host receive buffer § Use flow control tokens (FCTs) – Send FCT from end A to end B – when end A can accept 8 more N-Chars. – This enables end B to send up to 8 more N-Chars. – Multiple FCTs can be sent if there is more room in buffer. § FCT is exchanged for 8 N-Chars § TX must keep a count of credit. Normal Operation 1. Sends NULLs to maintain connection 2. Host System 4. FCT received indicates it has 3. Transmitter enabling 8 more room for 8 more sends FCT to chars to be sent chars request 8 more chars SpaceWire SpaceWire Interface Interface
HOST TRANSMITTER RECEIVER HOST SYSTEM SYSTEM END A END B RECEIVER TRANSMITTER
6. N-Chars received and 5. Transmitter passed on to host sends N-chars system from host AutoStart
§ Link interface can start – on command from host system – automatically on receipt of a NULL § [Link Enabled] = ( NOT [Link Disabled] ) AND ([LinkStart] OR ( [AutoStart] AND gotNull )) – LinkDisabled is the flag set by software or hardware to indicate that the link is disabled. – LinkStart is a flag set by software or hardware to start a link – AutoStart is a flag set by software or hardware to indicate that the link should start automatically on receipt of a NULL. – gotNull is the flag indicating that the link interface has received a NULL. § LinkStart and AutoStart are only acted upon when the link interface is not disabled i.e. [LinkDisabled] = False. Error Detection § Disconnect Error – No RX clock transition for more that 850 ns § Parity Error – Parity bit error § Escape Error – ESC character should only be used to form a NULL (ESC, FCT) or time-code (ESC, Data) – ESC followed by any character other than FCT or data character is an error § Credit Error – If there is no room in host RX buffer for data received then an error must have occurred which affected the FCTs § Empty Packet Error – EOP or EEP followed by another EOP or EEP – Represents an empty packet – Not permitted Exchange of Silence One End Other End of Link of Link Error Detected Reset Tx Disconnect Detected Reset Rx Reset Tx After 6.4 usec Reset Rx Reset Tx Exchange After 6.4 usec Enable Rx of Silence Reset Tx After 12.8 usec Enable Rx Send Nulls After 12.8 usec Enable Rx Send Nulls Enable Rx Null Received Send FCTs NULL/FCT Null Received Enable Rx Handshake Send FCTs FCT Received Enable Rx NORMAL FCT Received OPER. NORMAL OPER. Full State Machine Reset RxErr OR RxErr OR gotFCT OR CreditErr OR ErrorReset gotNChar EmptyPacketErr Reset Tx [Link Disabled] Reset Rx After 6.4us
Run ErrorWait Send FCTs/NChars/Nulls Reset Tx Enable Rx Enable Rx RxErr OR After gotFCT gotFCT OR 12.8us gotNChar RxErr OR RxErr OR gotNChar OR gotFCT OR after 12.8 us gotNChar OR Ready Connecting after 12.8 us Send FCTs/Nulls Reset Tx Enable Rx Enable Rx
gotNull Started [Link Enabled] Send Nulls Enable Rx Packet Level
§ Packet Level Protocol – defines packet structure – follows IEEE 1355 (with clarification) The SpaceWire Router
Credit – University of Dundee – Austrian Aerospace – EADS - ESA Contents Part 2
§ SpaceWire Packets § Routing Switch Concepts – Worm-hole routing – Header deletion – Path addressing – Logical addressing – Priority – Arbitration – Group adaptive routing – Fault tolerance § Networks SpaceWire Packets § Packet Format
§ Several link interfaces interconnected via a routing matrix. § Wormhole routing. § Uses leading data character of a packet as destination address / destination identifier. § Path and Logical addressing. § Output port arbitration. § Priority addressing. § Group adaptive routing. Wormhole Routing
ROUTER NODE NODE
Node sends out packet
Router receives header and checks requested output port Router connects input to output and packet flows through router Wormhole Routing
ROUTER NODE NODE
When EOP marker seen, router terminates connection and frees output port SpaceWire Routing Switches (1)
SpaceWire Routing Switch 1 3 Link Link Interface Interface ROUTING 2 MATRIX 4 Link Link Interface Interface
§ Packet arriving at any link input § Can be switched to any link output SpaceWire Routing Switches (2)
SpaceWire Routing Switch 1 3 Link Link Interface Interface ROUTING 2 MATRIX 4 Link Link Interface Interface Header Deletion (1)
NODE NODE
NODE NODE ROUTER ROUTER
Header selects output of router Header Deletion (2)
NODE NODE
NODE NODE ROUTER ROUTER
Header selects Header output of router deletion at each router Path Addressing (1) § destination is specified as router output port number § node 1 to node 3 <3>
4 Router R4 1 2 3
4 4 4 Router R1 Router R2 Router R3 1 2 3 1 2 3 1 2 3 3 N1 N2 N3 N4 N5 N6 N7 N8 N9 Path Addressing (2) destination is specified as router output port number node 1 to node 8 <4><3><2>
4 Router R4 1 2 3 3 2 4 4 4 Router R1 Router R2 Router R3 4 1 2 3 1 2 3 1 2 3 N1 N2 N3 N4 N5 N6 N7 N8 N9 Logical Addressing § each destination has a unique logical address § each router has a list of which port(s) to send data out for each possible destination § node 1 to node with logical address 43 <43>
43 1 4 163 3 Router R4 1 2 3 43 3 43 4 163 4 163 2 4 4 4 Router R1 Router R2 Router R3 1 2 3 1 2 3 1 2 3
N1 N2 N3 N4 N5 N6 N7 N8 N9 LA 41 LA 42 LA 43 LA 152 LA130 LA 131 LA 175 LA 163 LA 182 LA = Logical Address Routing Table
Address Port 0 Port 1 Port 2 Port 3 Port 4 Configuration 0 1 0 0 0 0 1 0 1 0 0 0 Path 2 0 0 1 0 0 Addressing … 32 0 1 0 0 0
Logical 33 0 0 1 0 0 Addressing 34 0 0 0 0 1 … Reserved 255 0 0 0 0 0 Priority
§ Arbitration in Router – Fair arbitration – Priority based § SpaceWire header contains address only § Assign priority to logical addresses Priority
Address Priority Port 0 Port 1 Port 2 Port 3 Port 4 Configuration 0 0 1 0 0 0 0 1 0 0 1 0 0 0 Path 2 0 0 0 1 0 0 Addressing … 32 0 0 1 0 0 0
Logical 33 1 0 1 0 0 0 Addressing 34 0 0 0 0 0 1 … Reserved 255 0 0 0 0 0 0 Arbitration LINK INTERFACE 1
INPUT OUTPUT PORT 1 PORT 1
INPUT OUTPUT PORT 2 PORT 2 INPUT OUTPUT PORT 3 PORT 3
INPUT Arbitration OUTPUT PORT 4 PORT 4 INPUT OUTPUT PORT 5 PORT 5 SWITCH INPUT MATRIX OUTPUT PORT 6 PORT 6
LINK INTERFACE 6 SpaceWire Network § Two or more SpaceWire nodes § Zero or more SpaceWire routing switches § Interconnected with SpaceWire links
ROUTER NODE
ROUTER NODE
NODE NODE NODE NODE NODE
§ Add bandwidth where required § Add redundant links where required SpaceWire Network
MEMORY 1
3 4 5 2 ROUTER 6 SENSOR 1 7 1 1 8 P 1 P 2
1 8 ROUTER 7 P 3 P 4 SENSOR 2 2 6 2 5 3 4 Processor Array MEMORY 2 SpaceWire Network
MEMORY 1
R R SENSOR 1 R P 1 P 2
SENSOR 2 R P 3 P 4 R R Processor Array MEMORY 2 SpaceWire Time-Codes
Credit – University of Dundee - European Space Agency Need for Time-Codes § Needed for control applications – Time accuracy better than 1msec (10 µs) across a fairly large network – Sub µs across links and small networks – Avoids having a separate network carrying time Time-Codes
§ Time-code comprises ESC, Data-Character – Data character holds an 8-bit time code § 6 bits of system time § 2 control flags – Time bits § used to distributed time § and to facilitate time-code broadcast – Control flags § General purpose § Broadcast to all nodes and routers Time-Codes
(P) P 1 1 1 1 0 T0 T1 T2 T3 T4 T5 T6 T7
ESC DATA CHARACTER Nodes and Routers
§ Each node or router has a 6-bit time counter § Link interfaces have TICK_IN and TICK_OUT signals § Asserting TICK_IN causes – time counter to be incremented – new time code to be transmitted – holding value of time counter § TICK_IN used by only one, time-master node § When valid time-code received – time counter is incremented – TICK_OUT is asserted Nodes and Routers
§ Valid time-codes have – Value of one more than the time-counter – At the receiver § If valid – Set time-counter to new time-code value – Propagate time-code § If NOT valid – Set time-counter to new time-code value – Do NOT propagate time-code Time Distribution
TICK_IN N1 R1 R2 N2 TICK_OUT 4041 4041 4041 4041 Time Distribution
R3 4041 TICK_IN N1 R1 R2 N2 TICK_OUT 4041 4041 4041 4041 Time Distribution
R3 4142 TICK_IN N1 R1 R2 N2 TICK_OUT 4142 4142 4142 4142 Time Distribution Error
TICK_IN N1 R1 R2 N2 TICK_OUT 4041424344 4041424344 40424344 404344 Time-Code Latency
§ Latency dependent upon – Number of links over which time-code travels – Operating rate of the links – Delay at time-code source – Delay at each router – Delay at receiving nodes Time-Code Latency § Assuming all links operating at same rate, A Mbits/s § Minimum delay at each router/node is – 14 bit periods (ESC + data character) § This gives rise to minimum time skew of – 14.S/A – where S is the number of routers traversed § Also get jitter at each router/node – Variation in time waiting for link to finish sending current character – Delay of 0 to 10-bits at each router/node § Total jitter is then – 10.S/A § Example A=100 Mbits/s, S = 10 – Tjitter = 1.0 µs – Tskew > 1.4 µs § Time-code distribution accuracy < 10 µs § Probably much better than software response time Time-Code Applications
§ Time distribution § Event signalling § Isochronous communications Conclusion
§ SpaceWire is an onboard communications standard designed specifically for spacecraft § SpaceWire ECSS-E50-12A standard has been issued § Many organisations and missions now using SpaceWire
§ SpaceWire is a platform for the implementation of many onboard communication architectures
§ Future work will focus on recommendations for – Application architectures – Transport layer protocols – etc