PHYSICAL REVIEW SPECIAL TOPICS - ACCELERATORS AND BEAMS 15, 082801 (2012) Open borders for system-on-a-chip buses: A wire format for connecting large physics controls M. Kreider,1,2 R. Ba¨r,1 D. Beck,1 W. Terpstra,1 J. Davies,2 V. Grout,2 J. Lewis,3 J. Serrano,3 and T. Wlostowski3 1GSI Helmholtz Centre for Heavy Ion Research, Darmstadt, Germany 2Glyndwˆ r University, Wrexham, United Kingdom 3CERN, Geneva, Switzerland (Received 27 January 2012; published 23 August 2012) System-on-a-chip (SoC) bus systems are typically confined on-chip and rely on higher level compo- nents to communicate with the outside world. The idea behind the EtherBone (EB) protocol is to extend the reach of the SoC bus to remote field-programmable gate arrays or processors. The EtherBone core implementation connects a Wishbone (WB) Ver. 4 Bus via a Gigabit Ethernet based network link to remote peripheral devices. EB acts as a transparent interconnect module towards attached WB Bus devices. EB was developed in the scope of the WhiteRabbit Timing Project at CERN and GSI/FAIR. WhiteRabbit will make use of EB as a means to issue commands to its timing nodes and control connected accelerator hardware. DOI: 10.1103/PhysRevSTAB.15.082801 PACS numbers: 84.40.Ua, 29.20.Àc, 07.05.Àt systems hosted inside a field-programmable gate array I. PURPOSE AND ENVIRONMENT (FPGA). EtherBone was named after these underlying This article builds on the paper by the title ‘‘EtherBone—A technologies, Ethernet and Wishbone. However, EB re- network Layer for the Wishbone SoC Bus’’ in the sides in the Open Systems Interconnection session layer ICALEPCS 2011 conference proceedings and aims to pro- (OSI layer 5) and does not depend on a specific choice of vide details on design choices and performance analysis for lower layer protocols in implementation. implementation. EtherBone (EB) functionality has been suc- cessfully demonstrated at the 2012 WhiteRabbit workshop. II. REQUIREMENTS EtherBone is a fast, low-level network protocol layer intended for either software to hardware or hardware to A. Timing challenge hardware communication. It connects a client to a distant Control of the accelerator’s machines is time critical, Wishbone (WB) system-on-a-Chip (SoC) bus and is capable varying from several n sec, over few sec , up to thousands of direct memory access to attached devices. This article of milliseconds. When compared with a delivery time of builds on the paper by the same title in the ICALEPCS 2011 approximately 100 secs for an EB packet, this leads to conference proceedings. EB will be used in the timing nodes several conclusions: (i) all actions need to be known in at the GSI/FAIR and CERN accelerator facilities. advance; (ii) all actions must be precisely timed; (iii) there At this point, a description of the intended environments is no time for acknowledgment or retransmission; (iv) a for the deployment of EB is in order. In synchrotron deterministic, low-latency command and parameter distri- machines, the particle beam can only be kept on its desired bution system is necessary. path by a tight interplay between a vast number of smaller For the efficient running of the accelerator it is clearly parts. There are beam guide components, like magnets necessary that the timing system must produce timing and radio frequency units, diagnostic equipment, and large signals that are deterministic, i.e., the time at which an sensor arrays at the targets. All of these machines need action is carried out must be precisely known and repro- accurately timed commands in the form of synchronous ducible. To facilitate this, it is necessary that there is triggers, timestamps, and control signals in order to minimum delay in the path from the production of the play their part at exactly the right time. When GSI is timing signal to its use. expanded with the planned Facility for Antiproton and Ion Research (FAIR), this means the coordination of B. Deterministic command distribution more than 2000 timing end points. WhiteRabbit [1] (WR), the system chosen for time synchronization, is based EB was designed to have very low latency and high on Ethernet technology and locally employs WB based determinism, giving secondary consideration to through- put. Because of this specialization, EB application focuses on commands rather than the transport of raw data. Separation between the data and the commands working Published by the American Physical Society under the terms of the Creative Commons Attribution 3.0 License. Further distri- with this data is possible in most cases. This means ma- bution of this work must maintain attribution to the author(s) and chines are fed with data via a standard network infrastruc- the published article’s title, journal citation, and DOI. ture and also receive EB packets over the timing network, 1098-4402=12=15(8)=082801(10) 082801-1 Published by the American Physical Society M. KREIDER et al. Phys. Rev. ST Accel. Beams 15, 082801 (2012) containing a command to carry out a preset action and a throughput, while short message latency is a secondary time of execution. EB is also able to address all specialized factor. PCI Express falls into the same category and will control blocks inside the WB hardware description lan- be addressed later in more detail. guage (HDL) designs directly, a fact that is very useful for There are also high level protocols available like this implementation. For example, consider remotely pro- CORBA [6] and SOAP [7], which aim for abstract software gramming a function generator with a parameter set for to software communication in heterogeneous environ- output level, gate length, and a sequence of trigger pulses ments. While being very versatile, due to their higher plus a time stamp to execute all this. The function genera- logistics overhead and generic nature, they are not well tor would be a WB memory mapped device, so EB will suited for fast communication between software to hard- provide direct and easy access to all its registers and ware or hardware to hardware. All of the above have in functions. common that they are not tied to a specific underlying bus protocol at their end points. While they keep data content, C. Compatibility and expandability they will not preserve syntax during transport. A comparison to widely used field buses, like USB, PCI, Particle accelerator facilities like GSI and CERN host and PCI Express (PCIe) showed PCIe as most fitting for large heterogeneous pools of equipment, which have this scenario but it still does not quite achieve the require- evolved for more than 40, or in CERN’s case almost 60, ments, mainly because of difficulties associated with rout- years. Most equipment at the Large Hadron Collider ing over WANs. It has nevertheless many features that are (LHC) and on the FAIR project is quite new, but there are still many legacy systems to cover. FPGA based tech- desirable for EB [8]. nology enables all sorts of adapters and converter logic Like the GbE Interface, EB uses a 125 MHz clock rate. with little or no extra hardware effort, while Ethernet based While the network end point uses an 8 Bit interface, the infrastructure makes the distances in growing facilities Wishbone interface connected to EB is 32 Bit wide, giving easily covered. This shall also be reflected by the next it 4 times the bandwidth. Regardless of delays for process- generation control systems. ing the packet structure, the difference in bandwidth en- sures EtherBone to be fully streaming capable. EB has several design traits in common with PCI Express. Both III. FURTHER APPLICATIONS are serial field bus protocols, they feature error detection in There are many places on site where well-known hard- the form of a cyclic redundancy check (CRC) to ensure ware tools like debug modules, in-system programming packet integrity, carry routing information, and provide adapters, logic analyzers, and similar are needed to deploy, quality of service (QoS). Also, both protocols go all the maintain, and upgrade hardware. Because of the distances way down to the physical layer. PCIe features autodiscov- between nodes, which are about 2 km at FAIR and 10 km at ery of bus devices, which is also present in WB and (and CERN, and the quantity of nodes involved, routing was therefore EB) since March 2012 under the name self- another requirement of EtherBone. It will not only reduce describing wishbone [9]. the time it takes engineers to travel on site, but also makes However, there are also differences. While PCIe is automated testing easier. Personnel can also collaborate packet-based from the bottom up, EB’s underlying WB more easily, since access to the hardware tools can be bus is cycle based. PCIe was also designed for higher shared over the network. Last but not least, it is possible throughput than WB, by bundling several lanes into one that the electronics are not accessible during beam time connection, and is meant to run at higher frequencies. This and for some time afterwards due to radiation concerns. All said, there are also differences to EB on the upper layer. in all, this feature will reduce time requirements for main- The first lies in the routing capabilities. While PCIe can be tenance and deployment. switched much like the MAC layer of Ethernet, it cannot do complex routing and most importantly is not native to Wide Area Networks (WANs). This makes long distance IV. RELATED WORK connections over common network architectures impos- There are many different examples of available proto- sible. PCIe could of course be encapsulated in IP packets to cols for direct data exchange. Among the most commonly do just that, but given the similarity between PCIe and used low-level were Myrinet in the supercomputing sector Ethernet/IP packet headers, this would be almost com- (almost completely replaced now by Ethernet based equip- pletely redundant and therefore double the overhead.