Making the Switch to Rapidio

QNX Software Systems Ltd. 175 Terence Matthews Crescent Ottawa, Ontario, Canada, K2M 1W8 Voice: +1 613 591-0931 1 800 676-0566 Fax: +1 613 591-3579 Email: [email protected] Web: www.qnx.com Making the Switch to RapidIO Using a Message-passing Microkernel OS to Realize the Full Potential of the RapidIO Interconnect Paul N. Leroux Technology Analyst QNX Software Systems Ltd. [email protected] Introduction Manufacturers of networking equipment have hit a bottleneck. On the one hand, they can now move traffic from one network element to another at phenomenal speeds, using line cards that transmit data at 10 Gigabits per second or higher. But, once inside the box, data moves between boards, processors, and peripherals at a much slower clip: typically a few hundred megabits per second. To break this bottleneck, equipment manufacturers are seeking a new, high-speed — and broadly supported — interconnect. In fact, many have already set their sights on RapidIO, an open-standard interconnect developed by the RapidIO Trade Association and designed for both chip-to-chip and board-to-board communications. Why RapidIO? Because it offers low latency and extremely high bandwidth, as well as a low pin count and a small silicon footprint — a RapidIO interface can easily fit Making the Switch to RapidIO into the corner of a processor, FPGA, or ASIC. Locked Out by Default: The RapidIO is also transparent to software, allowing Problem with Conventional any type of data protocol to run over the intercon- Software Architectures nect. And, last but not least, RapidIO addresses the demand for reliability by offering built-in While RapidIO provides a hardware bus that is error recovery mechanisms and a point-to-point both fast and reliable, system designers must find or architecture that helps eliminate single points develop software that can fully realize the benefits of failure. of its advanced features — especially its support for Of course, other potential interconnect standards distributed, multiprocessor systems. The problem isn’t exist, including HyperTransport, Infiniband, Fibre with RapidIO itself, which is generally software- Channel, Gigabit/10G Ethernet, and PCI-based fabric transparent, but with software that can’t easily migrate interconnects. But RapidIO doesn’t seek to supplant to high-speed distributed designs. Sometimes, the any of these. Rather, it is complementary, offering problem lies with application software that is written high performance — up to 64 Gbit/s per port — over to be hardware- or protocol-specific. All too often, the short distances needed to connect processors, however, the problem lies not with the application, memory, and local I/O within a single system. but with its underlying operating system (OS). For instance, consider what happens whenever an An Emerging Standard application needs to access a system service, such RapidIO is gaining momentum. Already, Motorola as a device driver or protocol stack. In most OSs, and IBM have begun to implement RapidIO inter- such services run in the kernel and, as a result, must faces on next-generation processors. Meanwhile, be accessed by OS kernel calls. Since kernel calls Altera has released a RapidIO solution for program- don’t cross processor boundaries, these services can mable systems-on-a-chip, and Tundra Semiconductor be accessed only by applications on the local node. has announced both switch chips and host bridge It thus becomes difficult, if not impossible, to distri- chips, including PCI/PCI-X to RapidIO bridge chips bute an application and its related services across for migration from existing designs. RapidIO has multiple nodes — they’re effectively locked together also gained acceptance from leading vendors such on the same processor. as Alcatel, Cisco, Ericsson, Lucent, Nokia, and Nortel — in fact, most of these companies are active members of the RapidIO trade association. Distributed by Design: The Advantage of Microkernel RapidIO itself continues to mature. Just recently, the RapidIO Trade Association released a serial Architecture specification that, combined with the existing To make distributed systems much simpler to RapidIO parallel specification, allows systems implement, the QNX® Neutrino® RTOS uses a designers to standardize on a single interconnect message-passing microkernel architecture. In a technology. For example, semiconductor and ASIC microkernel OS, only the most fundamental OS developers bringing serial chips to market can now primitives (e.g. threads, mutexes, timers) run in the reuse much of their original RapidIO parallel kernel itself. All other services, including drivers, design. file systems, protocols, and user applications, run outside of the kernel as separate, memory-protected processes. Page 2 Making the Switch to RapidIO Making the Switch to RapidIO In this architecture, process are inherently distributed. send messages over — it can be an Ethernet LAN That’s because they can use just one mechanism, today or a RapidIO backplane tomorrow. synchronous message passing, to communicate with As indicated, software developers can implement any other application or service, either local or distributed message passing using industry-standard remote. For instance, let’s say an application process POSIX calls such as open(), read(), and write(). But needs to send a message to a device driver. To do this, note that developers can also choose to access the it simply issues a POSIX open() call on a symbolic messaging framework directly, using three simple name identifying that driver. The underlying C library calls: MsgSend(), MsgReceive(), and MsgReply(). will then convert the call into a message, and the OS will route the message to its destination (the driver). If the driver is local, the OS microkernel will route Achieving Predictable Response the message directly; if the driver is on another node, in a Distributed System an OS service called the QNX micronetwork will When implementing remote calls in a distributed transparently forward the message to that node. system, the software developer typically has to deal In effect, it doesn’t matter whether the driver is local with complex synchronization issues and ensure that or remote — the application uses the exact same the calls return in a timely fashion. Remote calls can code either way. Consequently, any process can, be especially problematic for a realtime application, given appropriate permissions, transparently access virtually any resource on any other node, as if that MASTER BOARD resource were local. The QNX Neutrino RTOS thus Qnet Flash TCP/IP eliminates much of the complexity in building a Manager Fsys SNMP Manager distributed system, whether that system is based on RapidIO or any other form of interconnect. POSIX TCP/IP HA Ethernet Moreover, existing software can migrate easily to App Manager Manager RapidIO. Because of the clean division of labor provided by microkernel architecture, applications LINE CARD Qnet don’t effectively “care” what protocol or media they Manager VoIP TCP/IP Microkernel HA Flash Ethernet POSIX ATM Ethernet Manager Fsys Driver App Message Passing LINE CARD Microkernel Qnet Manager SS7 ATM POSIX TCP/IP VoIP App Manager Microkernel POSIX POSIX HDLC Figure 1 — In the QNX Neutrino RTOS, the kernel App App contains only a small set of core primitives. All other system services — drivers, file systems, protocol Figure 2 — QNX message passing integrates a network stacks — are provided by separate, memory-protected of individual nodes into a single logical machine. As a processes that can be stopped and started dynamically. result, an application can access resources on any other To achieve this modularity, QNX Neutrino uses message node transparently, without having to invoke remote passing as the fundamental means of IPC for the entire procedure calls. system. Making the Switch to RapidIO Page 3 Making the Switch to RapidIO since they typically provide no guarantee that requests inheritance applies whether the message originates going to and from other nodes will be processed in from a process on the local node or on another node. priority order. Predictable response times become This feature is key to ensuring that a system will difficult, if not impossible, to achieve. respond predictably even when its applications and services are distributed across a number of CPUs. The synchronous message passing provided by QNX Neutrino helps eliminate these issues. First, it In fact, QNX synchronous message passing generally automatically coordinates the execution of cooper- makes designs simpler to implement and easier to ating threads and processes. For instance, when maintain: it eliminates the need to keep track of process A sends a message to process B, process A complicated queuing behavior; it makes it easy to will immediately stop running, or become blocked, map a logical design to an actual implementation; until it has received a reply from B. This model and it fosters better encapsulation and information- ensures that the processing performed by B for A is hiding by restricting interactions to well-defined, complete before A can resume executing. It also message-based interfaces between processes. 1 eliminates the need to hand-code and debug synchronization services in either process. Redundant Links for Greater QNX message passing also ensures predictable Throughput and Fault Tolerance response times in a multi-node system by providing To enable fast, predictable response times, the distributed priority inheritance. For instance, if a low- RapidIO architecture offers both determinism priority process sends a message to a driver, asking it (achieved through multiple message-priority levels) to perform some work, the receiving thread in the and low latency. This low latency can be maintained driver will inherit that process’s low priority. This under even heavy network loads, thanks to several ensures that the work done for the low-priority features. For instance, if a link is busy, a packet process doesn’t execute at a higher priority than it doesn’t have to return to the original source device; should, thereby preventing a higher-priority process rather, it simply waits for the link to become from accessing the CPU (a condition known as available.

Load more