Compactpci Serial Specification for Modular Embedded Computer Systems

COMPACTPCI

CompactPCI Serial specification for modular embedded computer systems

By Alexander Buravlev, Fastwel

This article reviews the new basic specification of CompactPCI Serial, and explains why it was necessary to develop this specification when two basic specifications already existed in the PICMG consortium (ATCA and MicroTCA) and one in the VITA consortium (VPX/OpenVPX). It highlights the advantages of the differ- Figure 1. Development of ent system architectures for interconnects of intermodular specific tasks. data exchange of the PICMG CompactPCI standards family n Embedded computer systems fall into two creation of integral systems. The question of ity about 1 Gbit/s. This bus fulfilled the function classes according to their internal architecture: the right selection of standards is difficult for of universal interconnect, providing both data monolithic (for example, the automobile com- systems developers mainly due to two reasons. exchange for collaborative computing and data puter or the electronics modules of printers The first is that the standards themselves evolve exchange with peripheral and storage modules. or scanners), and modular, consisting of dif- over time in the technical aspect. This can be In the following 10 years two specifications ferent units or boards (for example, radar data rather difficult to follow: changes must be were issued, each of which added serial inter- processing). The first class are generally made monitored and upgraded versions acquired. connects to the parallel PCI bus: 2 Ethernet by big companies in millions of copies, allowing The second reason is that the standard itself channels were added in PICMG 2.16 and 4x1 to allocate huge engineering resources for their can be very interesting from the technical point PCI-E channels and 2 more Ethernet channels development, and quickly recovering the de- of view but not popular in the market. Thus, it were added in PICMG 2.30. And at last, in velopment cost. With the second class of sys- will be difficult to find the necessary modules 2011 the CompactPCI Serial specification was tems it is far more complicated, since volumes both at the system development stage and, adopted, substantially increasing throughput range from a few units to a maximum of what is worse, at the mass production stage. capacities in all types of interconnects: for data some thousands, and the intended tasks are exchange - PCI-E and Ethernet; for storage sys- often impossible for just one controller or The CompactPCI specifications were supple- tems – SAS/SATA; and for input/output systems processor, while the consolidated resources mented in 2011 by the CompactPCI Serial for general profile peripherals - USB 2.0/3.0. consume tens, if not hundreds of watts. For specification. It was developed to combining electronic modules, it is necessary renew the data exchange intercon- to use the one or the other technology. nects between modules in Com- pctPCI systems, and thereby to en- Most of the world’s modular embedded com- sure the solution of the task of cre- puter systems are developed on the basis of ating modular systems over the open standards, describing mechanical con- next 15 to 20 years. struction, heat sink, electrical connections and very often even logical protocols of communi- The CompactPCI history began in cation between the system modules. Such stan- 1999 (figure 1) when the first basic dards simplify the development of industrial specification was published, com- electronics products, using modules from dif- bining the Euromechanics standard ferent vendors without losing the novelty and (IEC 60297) with the PCI bus. At maintainability of the solution itself. The tech- that time the data exchange between Figure 2. External view of the FCI CompactPCI Serial nology of standards lies in ensuring compati- modules was provided by the 32- AirMax VS connectors installed on motherboards (on the left) and blade modules (on the right) bility of modules from different vendors for bit PCI bus with throughput capac-

25 September 2012 COMPACTPCI

CompactPCI Serial at first looks to be a logical system controller (Syst) as a host, while Ether- continuation of the tendency of step-by-step net has a full mesh topology or, in other words, transition from parallel buses to serial inter- “each-to-each” topology (figure 3). It is impor- connects of point-to-point type, but if you tant to note that the CompactPCI Serial stan- look at it more deeply and familiarize yourself dard does not impose restrictions on the con- with details, you will see that the CompactPCI figuration of motherboards, leaving this deci- Serial specification represents the new basic sion up to the customer and motherboard specification of the standard family, and this manufacturer. Respectively, some systems can is very important for the industry and devel- be designed for connection of only one appli- opers of embedded systems. cation or peripheral (Per.) module, the others - for connection of up to 24 application mod- CompactPCI Serial has five key innovations: ules. CompactPCI Serial for 6U modules de- new connectors, high speed interconnects, new scribes an additional connector on the system layout of connectors on 6U boards, a new controller module, providing two additional power supply scheme and conductive cooling Ethernet channels which can be used for en- technology. Connectors: CompactPCI Serial suring compatibility with PICMG 2.16, and uses new high-density connectors for data an additional power supply. transfer both on the side of blade modules and on the side of motherboards (figure 2). The mechanical structure underwent few Testing of connectors, for example, FCI AirMax changes but resulted in significant consequences connectors, showed high quality of signals for the 6U systems. Thus, all connectors for transmission up to 12.5 GHz. connection to the motherboard are now located in the upper half of the 6U plate, while the Connectors structure allows installation from lower half remains free and can be used for in- different sides of the board, allowing thereby stallation of connectors according to customer to apply the mezzanine concept of blade mod- specifications and/or for direct connection to ules creation with connection of mezzanines a rear I/O module (figure 4). If the organization directly to motherboards. Interconnects: Com- of the 6U system does not require support of pactPCI Serial outlines expressly the four types PICMG 2.16 and additional power supply, 3U of interconnects pins (pinout) designation motherboards can be used in such systems. and one I2C check bus on the system controllers and peripheral connectors: 8 PCI-E Figure 5 illustrates different options for the channels, two of which are 8 and the remaining implementation of CompactPCI Serial systems six are 4; eight Ethernet Base-T channels; (a, b, c) and hybrid systems (d) with monolithic eight SAS/SATA channels; eight USB 2.0 or (a, c) and composite (b, d) motherboards. Ad- USB 3.0 channels; I2C bus for control and vantages consist in the reduction of prices for monitoring of system service parameters. At motherboards - they now became smaller and the physical level each channel consists of two simpler to manufacture, as well as the possibil- differential pairs, providing data transfer from ity of supporting inherited (i.e. inherited from source to receiver and reverse. As all Compact- previous specifications) interconnects (Ethernet PCI Serial interconnects are serial, the specifi- and PCI), and the creation of hybrid systems. cation describes their topology, namely: PCI- In addition to the improvements already men- E, SATA and USB have star topology with the tioned, there is one special mechanical element

Figure 3. Intermodular interconnects topologies in the CompactPCI Serial specification: a) PCI-E, SATA and USB; b) Ethernet

September 2012 26 COMPACTPCI

Figure 4. Examples of connection of front and rear I/O cards to CompactPCI Serial motherboards for the 6U (a) and 3U (b) systems in 3U modules and two mechanical elements vision of power supply to modules with con- in 6U modules, directing and centering the ductive or liquid cooling. CompactPCI Serial boards when connecting to the motherboard provides the system controller with the possi- to simplify the processes of systems assembling bility to control the power supply, for example, and modules replacement for repair. upon occurrence of events (wake on LAN or wake on modem). Therefore, the functions im- New connectors also allowed improving the plemented in all modern processors and chipsets power supply system. Specifically, CompactPCI can be used for creating an embedded system. Serial requires only one supply voltage (+12V) The CompactPCI Serial specification offers with maximum input power of 79.8W for each developers a simple option for implementation 3U module and 171W for each 6U module, of systems with conductive cooling. This option needed for provision of power supply to 8HP assumes packing of the standard board in a or 12HP high-productive modules, or for pro- metal holder which is inserted into the system with the 5HP step between modules. Very often the success of new technical solutions is based on the support of legacy technologies. A good example is the success of the x86 processor architecture, supporting operation of the previously written program code. In the family of CompactPCI specifications, the new CompactPCI Serial specification provides systems developers with very wide possibilities of integration of inherited PICMG 2.0, 2.30 and 2.16 modules within one system. The CompactPCI 2.30 (PICMG 2.30) peripheral and application modules use identical connectors and are completely compatible to Com- pactPCI Serial.

The system controllers and peripheral 3U modules can be used in 6U systems. More Infor- mation on the compatibility of standard modules of the CompactPCI specifications family is provided in table 1. The CompactPCI Serial specification allows to implement hybrid systems in which the peripherals are based both on serial interconnects and on the inherited 32 or 64-bit PCI buses. In the 3U format the hybrid system can be implemented, for exam- Figure 5. Options of possible motherboard ple, by means of the PCI-E/PCI bridge module creation: a) 3U CompactPCI Serial, 8 periph- connected by a cable to the system controller, eral slots; b) 6U CompactPCI Serial, 8 and two motherboards, locating, for example, peripheral slots, composite motherboard; c) 6U CompactPCI Serial, 8 peripheral slots, the PCI segment on the left, and the Compact- monolithic motherboard; d) 6U hybrid com- PCI Serial segment on the right. Such an posite motherboard with support of 2 option is implemented by the MEN company. CompactPCI 2.0 peripheral blade modules In the option offered by Fastwel, the bridge is and 6 CompactPCI Serial peripheral slots. implemented on the mezzanine on the left for

27 September 2012 COMPACTPCI

the CPC510 system controller, which allows consortium (ATCA and MicroTCA) and one while the CPCI/VME modules in 6U format installing it in the PICMG 2.30 system con- in the VITA consortium (VPX/OpenVPX). show substantial advantages (plus 60 %) over troller slot and supporting both of the om- This is a good question, and the answer is very MicroTCA. pactPCI and PICMG 2.30 peripherals. The important from the point of view of under- implementation of compatibility with inherited standing the advantages of the different system It is necessary to note that the CPCI/VME modules on the basis of PCI bus in the 6U for- architectures for specific tasks. Let’s look at manufacturers widely use the possibility of in- mat is already described. Compatibility with this in more detail. creasing the board width up to 8HP, and even PICMG 2.16 is supported in the 6U format by up to 12HP, by installing additional boards an additional connector with two Ethernet First of all, let’s compare such basic parameters and locating the functions required thereon, channels. as board areas, the possibilities of routing var- while the AMC functional modules manufac- ious interfaces for data input-output, and in- turers cannot go over the maximum size of The increase in input power up to 79.8 W (for stalling different components on board, as 6HP, as this size is defined by the restriction 3U) and 171 W (for 6U) for the CompactPCI well as heat budgets. The overall dimensions on mezzanine thickness in (AMC is a mez- Serial system controllers allows installing prac- and heat budgets of CompactPCI (CPCI) and zanine for ATCA). The conclusion is thus: the tically any processors of PowerPC or x86 ar- VME modules are identical, therefore they CPCI/ VME specifications allow installing a chitecture. This way, using the advantages of can be reviewed jointly. The AMC blade mod- bigger number of components, and compo- mechanical structure, on the double width ules used in the MicroTCA standard may have nents of a bigger size, on the blade modules, modules one can install not only soldered 3HP, 4HP or 6HP width and two board sizes – as well as tolerating heavier heat loads with air processors from the CPU suppliers mobile single or double. If we compare the areas of cooling, than the AMC modules in the Mi- segment, but also server socket processors and the CPCI/VME boards with the size of the croTCA systems. If we look at the areas of the vertically installed DIMM or SO-DIMM. We AMC and ATCA boards, we arrive at two obvi- front panels which determine the possibilities recall that the CompactPCI specification de- ous conclusions as regards locating electronic for installing connectors and the input/output fined a power input of only 50W for both 3U components, radiators and air heat sinks. 1) possibilities for the different interfaces, the ad- and 6U modules, requiring application of pro- The ATCA modules are non-comparable and vantage of the CPCI/VME modules becomes prietary methods for installation of server therefore do not compete in any way with obvious (figure 6b). processors. The informed reader may ask why CPCI/VME and MicroTCA (figure 6a). 2) The it was necessary to develop a new specification CPCI/VME modules in 3U format have com- To be fair, it is necessary to note that the new based on serial interconnects, if two basic spec- parable characteristics with the single-size MicroTCA.4 standard adopted in 2011 sup- ifications have already existed in the PICMG AMC blade modules for the MicroTCA systems, ports the possibility of directly connecting COMPACTPCI

Figure 6. Comparison of CPCI/VME, MicroTCA and ATCA technologies: a) according to useful areas on boards for location of electronic components and heat sink and air cooling capabilities; b) according to useful areas of panels for location of I/O connectors. front modules to rear modules of the same (table 2). Besides, the list of standardized in- size, thereby doubling the area for components terconnects is wider than in CompactPCI installation, and increasing the output capacity Serial, and the topologies of their connections by roughly 50%. Thus, in regard to board area include a double star, a ring, a mesh and extension and components installation thereon, others. In fact, OpenVPX appears as the refer- it can be said that the CompactPCI and VME ence manual (and the OpenVPX developers specifications benefit compared with MicroT- do not hide it even in the standard name: CA, while in regard to the input-output capa- ANSI/VITA 65.0-2010 OpenVPX Architectural bilities, they have substantial advantage over Framework for VPX), describing a great diver- MicroTCA. sity of intermodular exchange technology implementations, while CompactPCI Serial de- The VPX/OpenVPX family standards were describes specific implementations of technology. veloped by a group of contracting companies The consequences are simple and complex at for the United States Department of Defense, the same time: intermodular compatibility is with the target task of updating the VME possible only within compatible profiles, which specification and inventing a new technology in practice means – from one manufacturer. for creating modular embedded systems for By contrast, CompactPCI Serial allows the as- their customer. Originally the VPX standard sembly of systems with modules produced by (ANSI/VITA 46.0-2007 VPX: Base specifica- different companies, since the intermodular tion) was based on the use of new plated compatibility is guaranteed by the specification. MultiGig RT2 connectors from Tyco Electronics And logically we see that CompactPCI Serial with throughput capacity of 6.25 GHz; this al- performs the function of facilitating creation lowed using 64 differential pairs for ensuring of modular systems, so long as the modules intermodule communication in the system on are known to be compatible with each other. the basis of 3U boards and 192 pairs for 6U. Recently, a new connector with a throughput The VPX/OpenVPX specification allows very capacity of 10 GHz comparable to CompactPCI high values of input power for the module Serial has been standardized for the VPX/Open- which can be used for construction of systems VPX systems (ANSI/VITA 60.0-2012 VPX: Al- with liquid cooling, as well as a bigger number ternative connector for VPX). Accordingly, the of differential pairs in the 6U VPX/OpenVPX VPX modules and systems manufacturers have systems in comparison with ompactPCI Serial, no barrier to increasing the speed; only the enabling higher speeds of intermodular ex- timely and organized migration to use the change when using a number of high-produc- new connector is required as the VITA 46 tive profiles. modules and motherboards are incompatible with VITA 60. With reference to modular platforms, granularity is an important parameter for developers who The next and, probably, the most important plan to release a line of products based on one difference is the approach to description of of these technologies but with different charac- pinouts in the CompactPCI Serial and teristics, for example, for provision of services VPX/OpenVPX standards. Namely, the Com- to a different number of subscribers or with pactPCI Serial standard expressly defines at different computational power. The granularity which connector pin the signal should be pres- metrics can be different and be expressed in ent. The OpenVPX standard gives this deter- units of power, volume or cost, measured by a mination to the so-named profiles (pins desig- number of operations per second, etc. nation and interconnection topology). These are subdivided into the modules, slots and Systems built on the basis of the 3U Compact- motherboard profiles, with the number of PCI Serial standard have the lowest granularity, profiles of each type described in the OpenVPX namely about 30W and $1000, and the 6U sys- specification for 3U and 6U modules desig- tems – about 70W and $3000. Such values of nated in tens and sometimes in hundreds granularity are well-suited for creation of dif-

29 September 2012 COMPACTPCI

the public networks core is occupied by the 10 to 40 Gbps ATCA systems and the rack-mount server-based solutions. But how about Com- pactPCI Serial?

With updated high values of the internal interconnects throughput capacity, the Com- pactPCI Serial standard can be successfully used in the IP-based systems intended for creation of network infrastructure. But new possibilities for CompactPCI Serial now more likely exist not in the cores of public telecommunication networks, but in their peripheral and boundary sections. Regardless of the customer type, whether a digital house, digital transport, digital office or subscribers, the Table 1. Cross-compatibility of system controllers and peripherals modules for the CompactPCI equipment for their network access should be range standards able to recognize different wire and wireless data transfer protocols, to aggregate and convert these flows in TCP/IP and to carry out their preprocessing. The CompactPCI Serial capacity, competitive prices and low granularity values – this is a partial list of the advantages which play a key role here.

If we have a look at a private network, for example at the process equipment control network or at the vehicle control network, the CompactPCI Serial system switching capacities allow its use for resolution of tasks in the core of such a private network. Already there are some successful applications, for example, the Wi-Fi Internet system in European trains.

The possibilities of the 6U CompactPCI Serial standard are also interesting for creating the Table 2. Comparison of the CompactPCI Serial and VPX/OpenVPX technologies wire network infrastructure connected with the support of technology of the customer de- ferent control and supervisory complexes, and tem off (hot-swap). The full intrasystem du- vices power supply via the network cable measuring equipment. And if the 3U systems plication can be organized by synchronization (Power over Ethernet – PoE). Such devices in- are more suitable due to their capacity for exe- of operation of two CompactPCI Serial seg- clude IP surveillance cameras, panel computers, cution of customer tasks, the 6U systems have, ments through Ethernet channels (for the 3U controllers and other devices, consuming up mainly, server functionality and designation. and 6U systems), or using intrasystem switches to 40W. By means of the P0/J0 connector, the The VPX systems given the same size and (only for the 6U system). 6U CompactPCI Serial modules can be pro- computing parameters show at least twice the vided with an additional power supply of – cost. Accordingly, their granularity in relation It will be fair to note that complete intrasystem 48V via the motherboard, with maximum cur- to cost is higher, which may lead in some cases back-up is rarely used nowadays. More often rent of 1.9A (91.2W). to going over the limits of the permissible distributed back-up is applied, allowing to in- budget when creating a line of products. crease the systems operability with power sup- The CPCI/VME standards have taken a strong ply from different sources and use of distributed position in many niche markets, such as pro- The ATCA standard-based systems have the communication lines. The high level of inter- duction automation, complex process equip- biggest granularity – about 300W and the cost modular switching and support of a large ment and programmable-controlled machines of modules about $50 to $8000. Thus it makes number of standard interfaces, such as Ethernet, management, control and communications sense to use these systems for tasks with heavy is the key to success in creating systems with on transport, defense systems and many others. loads on the switching or computing subsys- distributed back-up on the basis of the Com- The new CompactPCI Serial standard allows tems, measured in teraflops and tens of Gbit/s. pactPCI Serial specification. simplifying the system creation process. The system developer has the possibility to choose The CompactPCI Serial standard supports dif- Originally, the first CompactPCI standard was between 4 types of interconnects, allowing to ferent back-up and hot-swap technologies. designed for use in the public telecommunica- reduce significantly the number of bridges for The systems developers have all possibilities tion networks core. But since the end of the communication of peripherals with the system open for back-up implementation. For example, 1990s telecommunication networks have controller. Direct connection of SATA drives, peripheral modules can be duplicated or trip- changed a lot, and the switching capacity pro- including the use of chipset-integrated RAID, licated, and the replacement of an inoperable vided by the CompactPCI systems (1 to 2 connection of USB cryptokeys or controllers module is possible without switching the sys- Gbps) became insufficient. Now their place in with USB ports, CompactFlash, SD and mi-

September 2012 30 COMPACTPCI croSD USB card readers – and this is a far coprocessors on the basis of 18 PCI-E. It is low latency (PCI-E is supported by the system from full list of the CompactPCI Serial capa- important to note that CompactPCI Serial controller, there are no additional switches bilities whose implementation does not require overall dimensions allow creating these appli- with their time delays) and high speeds of bridge microchips and their drivers. The pos- cation boards in the form of monolithic boards, the intermodular exchange, thus allowing the sibilities of releasing the PCI-E-based periph- or in the form of carriers of MiniPCI Express, creation of clusters combining the resources erals are extremely wide. Let’s begin the list of PCI-E, XMC, FMC cards and even of the of several modules. examples with the simplest ones – expansion PC/104 family. modules for wire or wireless data transfer, For example, the Fastwel CPC510 module has CAN controllers and COM ports on the basis The real-time system developers and the de- the semi-transparent PCI-E bridge, allowing to of 11 PCI-E. Then we continue with the lower velopers of the high-productive complexes insert it into the system controller slot and into level graphic coprocessors, controllers with intended, for example, for processing of Fouri- the PCI-E peripherals slot, using it as the 4-core „copper“ or optical interfaces – Gigabit or er conversions, should see the true value of computing coprocessor where each core can ex- 10Gigabit Ethernet. We close the list with CompactPCI Serial, as the specification pro- ecute up to 8 instructions of double accuracy graphic and network DSP or FPGA top level viding the data exchange interconnect with with floating decimal point in one cycle. n Product News