Tenasys Real-Time Hypervisor

BACKGROUNDER

Host Real-time and General-purpose Operating Systems on a Single Hardware Platform with Intel® Virtualization Technology

August, 2006

TenAsys Corporation 1400 NW Compton Drive, #301 Beaverton, OR 97006 USA +1 503 748-4720 fax +1 503 748-4730 [email protected] www.tenasys.com

Copyright © 2006, TenAsys Corporation. TENASYS, INTIME, and IRMX are registered trademarks of TenAsys Corporation. 060803 Other trademarks and brand names are the property of their respective owners. Virtualization Background

Virtualization of computer hardware has been used for many decades. The most widely noted early examples are those implemented by IBM on their mainframe hardware giving their customers a means to easily upgrade from old “iron” to new “iron.” In this case a primary goal of virtualization was to allow legacy applications to run on newer machines, alongside applications designed for the new OS and hardware. [1]

Software that manages computer hardware virtualization is referred to as a Virtual Machine Manager (VMM) or hypervisor. [2] A VMM is akin to a machine emulator. A machine emulator typically simulates the CPU instruction set, some key hardware elements, and the operating system calls from a real system; usually for the purpose of running applications developed for obsolete hardware on newer hardware. For example, emulators have been written to run the code copied from ROM cartridges of old gaming systems on a desktop computer. Like an emulator, a VMM creates the illusion of a hardware platform, for the purpose of hosting an entire operating system (the guest OS) and its applications. A key difference between an emulator and a virtual machine is that a virtual machine can be built from virtual hardware; that is, the I/O devices inside a virtual machine need not emulate real hardware.

Most modern VMMs operate without the need for CPU instruction set emulation, since the instruction set of the virtual machine is identical to, or is a subset of, the underlying hardware. The guest OS and applications running inside each virtual machine are native to the underlying processor instruction set. Also, instead of perfectly emulating a specific hardware platform, the typical VMM presents a virtual computing system that can be replicated multiple times on a single hardware platform and contains sufficient virtual I/O to support common client and server applications.

Emulation on the IA platform VM86 mode, part of the Intel® architecture since the Intel® architecture (IA) processors, 386 processor, provides a means by which a VMM ubiquitous on desktops and widely used can efficiently host a 16-bit guest operating system for many embedded applications, supports (e.g., DOS and Windows versions thru ME). The four distinct privilege levels of application VM86 hardware traps accesses to key processor execution named rings 0 thru 3. Ring 0 resources so the VMM can maintain control of the executes at the highest privilege level and machine hardware. This x86 virtualization feature ring 3 at the lowest. In typical practice only was exploited over fifteen years ago by TenAsys two levels are ever used: rings 0 and 3, engineers, in DOS RMX and the original iRMX® for referred to, respectively, as “supervisor- Windows. Instances of these virtual real-time mode” and “user-mode.” Operating embedded applications are still in use today, systems that utilize these execution modes running the iRMX® RTOS and a general-purpose OS are referred to as “protected-mode” side-by-side, on a single hardware platform. operating systems; the OS executes its instructions in supervisor-mode and its applications execute in user-mode. Drivers might execute either in supervisor or user-mode, depending on the architecture of the OS and the nature of the driver.

On an IA processor, the typical emulator simulates an application’s operating system environment by intercepting legacy OS calls made by the emulated application and

page 2 of 11 Copyright © 2006, TenAsys Corporation • www.tenasys.com • August, 2006 translating them into equivalent calls to the hosting OS. The emulated application runs in user-mode, making it very easy to trap and substitute system-level calls using the x86 ring hardware. A VMM, on the other hand, hosts a complete protected-mode OS (the guest OS) plus its applications. A protected-mode guest OS expects to run in supervisor-mode with full access to the underlying CPU registers and data structures. One of the most difficult jobs a VMM must do is intercept guest OS instructions that modify supervisor-mode registers and data structures and then simulate the impact of those instructions inside the virtual machine on which the guest OS is running.

Static virtualization and real-time

In 1997 TenAsys® Corporation introduced INtime®, an RTOS that runs deterministically alongside 32-bit and 64-bit versions of Microsoft® Windows® on a single IA hardware platform. A unique form of virtualization makes this feat possible, allowing Windows to run unmodified as the lowest priority task (i.e., the idle task) in the INtime real-time task list. This “dual-OS, single-platform” arrangement gives developers a means to build deterministic embedded Windows systems that can reliably control critical machine functions and simultaneously present high-level interfaces for system monitoring, enterprise connectivity, and complex user interaction.

INtime and Windows share a single hardware platform. Windows runs as the lowest priority task (the idle task) in the INtime real-time task list.

The INtime RTOS is a 32-bit protected-mode operating system, as is Windows XP, the desktop OS which “shares” a single hardware platform with INtime. Running two specific protected-mode operating systems on a single platform is sometimes referred to as “static virtualization.” [3] In this case Windows boots first and then INtime inserts itself between the system hardware and Windows. Memory and I/O hardware designated for use by the INtime RTOS are “hidden” from Windows and dedicated for use by RTOS processes. All standard user interface I/O, such as the video, keyboard, and mouse, remain under the ownership of Windows. Windows is unaware that the INtime RTOS and its processes exist, even as it is relegated to the lowest-priority task in the INtime task list.

Copyright © 2006, TenAsys Corporation • www.tenasys.com • August, 2006 page 3 of 11 Static virtualization of an RTOS with Windows has many applications: • Phoenix Contact of Ann Arbor, Michigan implements their Steeplechase soft PLC engine on INtime and a PLC-to-enterprise network interface, called Transaction Express, on Windows, resulting in one of the fastest and most flexible Windows-based PLC engines for factory floor automation. • AVL in Graz, Austria, drives their PUMA Open test bed software with INtime. PUMA Open is a real-time Windows test automation system for measuring and processing high-speed data from engines, transmissions, and power trains. AVL products are used by major car and truck manufacturers worldwide. • CNC maker ANCA from Melbourne, Australia puts an INtime plus Windows powered PC at the heart of their CNC machines, giving their customers fast and accurate machining along with the ability to model and preview grinding operations in 3D, directly on the CNC machine. The combination reduces their customer’s cost of operation by simplifying equipment requirements and avoiding expensive tool crashes during trial runs.

Conventional IT virtual machine shortcomings

The static virtualization method used by INtime succeeds where the conventional approach to virtualization used by traditional VMMs simply will not work; that is, for real-time applications. Like a general-purpose OS, such as Windows or Linux, a conventional VMM must be “fair” in its approach to scheduling CPU time for virtual machines and allocating I/O resources among the virtual machines. The conventional VMM is targeted at solving problems for a corporate IT network: maximizing the use of server resources and simplifying the deployment and maintenance of client desktops. The “IT virtual machine” presented by a conventional VMM is not capable of supporting the deterministic scheduling and dedicated I/O required of real-time applications and their operating systems.

The I/O presented by a conventional IT virtual machine usually consists of a CPU, RAM, disk, video, keyboard, mouse, and a network interface. A few other generic I/O devices may be available to “install” inside the IT virtual machine, but due to the difficulty associated with multiplexing a wide range of I/O between multiple virtual machines, most virtual machine I/O is limited to these basic devices. Despite these limitations, this I/O complement satisfies a large percentage of IT server and desktop applications, making this form of virtualization popular with corporate IT groups as a means to quickly buildup and teardown server applications and client desktops.

Intel® Virtualization Technology

Until the introduction of Intel® Virtualization Technology (Intel® VT) as part of the Intel® Core™ microarchitecture, a software-only VMM designed to support multiple protected- mode operating systems on an x86 virtual machine encountered significant challenges. Both the VMM and the protected-mode guest OS expect to maintain supervisor-level control over the hardware platform; however, absent some form of cooperation between the VMM and the guest operating systems (sometimes referred to as “paravirtualization”) the VMM must resort to trickery. Supervisor-level control can be reliably maintained by only one software system, resulting in a conflict between the VMM and the guest OS. The tricks a VMM must

page 4 of 11 Copyright © 2006, TenAsys Corporation • www.tenasys.com • August, 2006 use, without Intel VT, involve modifying the guest OS binary code and running the guest operating systems at ring levels for which they were not written.

The downside to such VMM trickery is a decrease in performance and limited guest OS compatibility. For example, binary files of a guest OS may be modified to trap supervisor- level CPU instructions, requiring the VMM to emulate these instructions. Instruction emulation slows down the execution speed of the guest OS, and the need to “fix up” binary files limits your guest OS options to those that have been certified for use with the VMM. Likewise, problems exist in the area of address translation and interrupt masking where, again, the software solutions result in performance overhead that cannot be tolerated by real-time applications.

Intel Virtualization Technology is designed to overcome the problems outlined above. In those processors that include Intel VT an overarching operating-mode has been added, called VMX root, where a hypervisor executes with final control of the CPU hardware. A hypervisor that uses Intel VT can intercept key supervisor-mode operations executed by any software operating outside of VMX root without requiring a priori knowledge of the guest OS binaries or internals. Using this Intel VT hardware assist for virtualization, one can build a hypervisor VMM that hosts protected-mode operating systems executing in ring 0 without giving up control of key CPU resources. Also, Intel VT provides a way for the VMM to implement virtual interrupts, an essential feature for hosting a real-time guest OS. [4]

Dynamic real-time virtualization

Low interrupt latency, direct access to specialized I/O, and the assurance that a VMM won’t “time slice away” the determinism and priority of real-time tasks are all key requirements of a real-time virtual machine. The combination of multi-core CPUs and Intel VT are an ideal platform on which to move beyond multi-OS real-time systems that utilize static virtualization and build a real-time hypervisor based on dynamic virtualization.

A real-time hypervisor is a VMM that uses hardware virtualization technology to isolate and simultaneously host general-purpose operating systems and real-time operating systems. Unlike static virtualization (described above), the dynamic virtualization implemented by a real-time hypervisor uses an “early start” technique, taking over complete control of the hardware platform. This means that guest operating systems are allowed to “boot” only after the real-time hypervisor has constructed a virtual machine for them.

There are two significant advantages to building a multi-OS real-time system by using dynamic virtualization rather than static virtualization:

A wide range of operating systems, both general-purpose and real-time, can be supported.

The boot sequence for each guest OS is under the control of the hypervisor.

This second advantage means it is possible to restart one guest OS while other guest operating systems continues to run without interruption.

TenAsys Real-time Hypervisor

Intel Virtualization Technology has enabled TenAsys to develop a hypervisor capable of supporting the demands of an RTOS while simultaneously hosting a general-purpose operating system (GPOS), like Windows or Linux. Further, by leveraging Intel VT, the

Copyright © 2006, TenAsys Corporation • www.tenasys.com • August, 2006 page 5 of 11 TenAsys real-time hypervisor enhances real-time application responsiveness and reliability in a “dual-OS, single-platform” environment, with even more precise control over interrupt latency and finer partitioning of I/O resources between multiple guest operating systems than previous solutions.

The TenAsys real-time hypervisor leverages Intel Virtual Technology to partition processor resources between multiple guest operating systems.

Using a real-time hypervisor to simultaneously support execution of a real- time OS and a general-purpose OS on a single hardware platform is quite different from installing a real-time thread scheduler, in the form of a device driver or subsystem, inside a general-purpose OS kernel driver and subsystem solutions lack the context to apply Intel VT and insure isolation, protection, and independence of operation between the RTOS and the GPOS.

A key difference between the TenAsys real-time hypervisor and conventional VMM solutions is how physical resources are allocated to each virtual machine. Resources, such as CPU cycles, RAM, I/O, and interrupts, must be allocated by any VMM. In the simplest case, a conventional VMM evenly multiplexes these resources between the virtual machines. A conventional VMM may allow priorities to be assigned to specific virtual machines, but these “tuning parameters” only slightly modify the bias of a system that, by design, attempts to fairly distribute physical resources among all the virtual machines on a given hardware platform.

When determinism and performance are more important than equal access, the processor virtualization features can be used to isolate resources for use by a specific virtual machine and its guest OS rather than to create virtual I/O for shared access between multiple virtual machines.

To maintain the determinism of a guest RTOS in a virtual machine environment, the TenAsys real-time hypervisor distinguishes between resources that can be multiplexed by the VMM and those that are exclusive to a virtual machine. For example, user interface I/O is usually not associated with time-critical events, so devices like the keyboard, mouse, console, disk, and an enterprise Ethernet interface can be multiplexed and shared between all virtual machines. However, hardware that is specific to a real-time control application, such as a video capture card, fieldbus interface, or an Ethernet NIC designated for communication with real-time I/O devices, should not be multiplexed between virtual machines. Specialized real-time I/O needs to be dedicated to its real-time virtual machine, so the RTOS and application using that I/O can maintain real-time determinism and control.

This notion of applying I/O exclusively to a specific virtual machine is essential to guarantee real-time responsiveness, because it allows a real-time virtual machine to have direct physical access to its dedicated I/O. Without exclusive physical assignment of pertinent I/O you run the risk of waiting indeterminately for access to key devices. If another virtual machine has access to an I/O device, because the I/O is multiplexed, the wait can be significant. Even if only one virtual machine ever accesses a specific I/O device, when a request is made to access that I/O a VMM that virtualizes I/O must translate the request by the virtual machine into real I/O accesses to the physical hardware, an unnecessary and time consuming process.

Similar arguments can be made for access to RAM. In a conventional VMM some or all of the memory in each virtual machine could be swapped to disk, in order to more efficiently allocate limited physical RAM among multiple virtual machines. An RTOS, and all of its real-time applications, must always be resident in physical RAM. The TenAsys real-time hypervisor guarantees that each real-time virtual machine is locked into physical RAM, and is never swapped to disk. This is the only way to insure that every real-time event is serviced consistently, with deterministic timing.

Exclusivity for performance

Exclusivity of I/O doesn’t apply only to a real-time virtual machine. Graphics intensive applications need access to real hardware for maximum performance. For example, a virtual frame buffer may be too slow and inadequate in features for an application that renders 3D moving images. In that case, the virtual machine containing the GPOS needs direct access to the physical frame buffer and its control I/O.

Dedicating the physical video frame buffer to a single virtual machine guarantees optimum video features and performance for the GPOS and applications running inside that virtual machine, at the expense of leaving all other virtual machines with no output console. A serial console or a virtual frame buffer can be presented to the remaining virtual machines, depending on their video output needs, with the output of the virtual video devices displayed in a window on the user interface of the virtual machine containing the dedicated video buffer. This approach requires a video helper function residing on the virtual machine that contains the physical frame buffer.

For those systems where virtual frame buffers have sufficient performance and features for all virtual machines hosted by the hypervisor a simple keyboard switch (e.g., Ctrl-Alt-Fn) can swap between virtual displays, showing the output of any one virtual buffer at a time on

Copyright © 2006, TenAsys Corporation • www.tenasys.com • August, 2006 page 7 of 11 the real monitor. The advantage, of course, to this fully virtualized video approach is that no video helper function is required.

Legacy RTOS support

Many systems exist today that depend on real-time code written many years ago. These legacy real-time applications continue to be used because they work; they are proven, they are reliable, and they may even be certified for an application (e.g., medical equipment, defense, and aerospace). Unfortunately, these legacy applications may also be running on obsolete hardware or be in need of an update, such as a graphical user interface or access to an enterprise network. Rewriting a proven and/or certified real-time application is rarely desirable or economical.

The TenAsys real-time hypervisor can directly host legacy real-time applications or host a legacy RTOS and its application(s). With minimal modification to existing code, new features can be added to the legacy application by simultaneously hosting a GPOS, thereby enabling developers to maximize code preservation and maintain performance. Interprocess communication, facilitated by the real-time hypervisor, is used to augment the legacy application by adding new features via processes running in parallel virtual machines.

Running legacy RTOS code in a virtual real-time machine is also how legacy real-time code can be migrated from obsolete hardware to modern embedded platforms. Because I/O can be virtualized, it is possible to simulate old hardware devices, minimizing rewrite of proven legacy code. For example, a VMEbus system could be converted to a less expensive single- board computer system by intercepting I/O requests to VMEbus I/O and redirecting it to equivalent on-board I/O devices.

Interrupt latency

Control over hardware interrupts is substantially improved by Intel Virtualization Technology. The Intel VT hardware in an Intel Core microarchitecture CPU provides the TenAsys real-time hypervisor with the ability to “see” a hardware interrupt even if it has been masked by a guest OS. This is an important feature, it means the real-time hypervisor can field hardware interrupts and insure that interrupts for a real-time virtual machine will always be serviced without delay.

The ability to monitor interrupts at all times, regardless of the state of their mask bits inside a virtual machine, insures that real-time interrupt latencies are minimized and that virtual machines cannot interfere with each other, even if they share hardware devices on a single interrupt line.

Multiple RTOS Support

The use of the TenAsys real-time hypervisor is not limited to just “dual-OS, single platform” applications. Three or more virtual machines can be hosted by the real-time hypervisor. For example, a single GPOS in one virtual machine and two real-time virtual machines, each containing a dedicated RTOS.

Take, for example, a conventional system consisting of a general-purpose OS board serving the user-interface and enterprise nexus function, an RTOS board implementing machine control, and a DSP daughterboard dedicated to high-performance numeric algorithms, such as image processing. Using real-time virtual machine technology, three hardware elements

page 8 of 11 Copyright © 2006, TenAsys Corporation • www.tenasys.com • August, 2006 can be condensed into a single platform solution, all hosted on the TenAsys real-time hypervisor; hosting three virtual machines, one each for what was previously three separate (and expensive) pieces of computational hardware.

Multi-Core CPUs

Multi-core processors are composed of multiple CPU cores in a single-socket package. Intel Core microarchitecture multi-core processors access their external bus via a shared level- two cache, called the Intel® Smart Cache. Multi-core technology runs at lower clock speeds than single-core processors, reducing overall power consumption—an important consideration for embedded applications. [5]

Intel multi-core processors combine the benefits of multiple execution cores on a single die of silicon. The multi-core architecture can deliver significantly greater performance at far lower heat dissipation than equivalent performance single-core processors. These gains are of particular benefit to applications requiring high performance in a small embedded form factor. Intel Core microarchitecture processors include multiple CPU cores on a single silicon die. (Image courtesy of Intel)

From a software application perspective, multi-core systems look just like a multi-socket, symmetric multiprocessor platform (SMP); a server hardware platform that has been available for many years. The key differences between multi-socket SMP platforms and single-socket multi-core platforms are cost, size, power consumption, and availability to the embedded market. Most general-purpose operating systems, such as Windows XP and Linux, are designed to operate on and take advantage of SMP machines.

Real-time virtual machine solutions derive maximum benefit from multi-core processors. For example, on a dual-core processor the TenAsys real-time hypervisor dedicates one CPU core to the RTOS and the other to the GPOS; providing real-time applications with 100% of the CPU instruction cycles of that core dedicated to the RTOS. The CPU cycles of the remaining core are used exclusively by the GPOS virtual machine. Contention for key CPU resources, such as pipelines and the FPU, are avoided, maximizing performance and responsiveness of both operating systems. Coordination between the RTOS virtual machine and the GPOS virtual machine is managed by the real-time hypervisor which uses built-in interprocessor interrupt mechanisms, eliminating inter-OS context switch times.

Removing contention for resources in a multi-OS platform has a dramatic impact on real- time performance metrics, such as interrupt latency. TenAsys has measured a 10 to 1 improvement for interrupt latency on dual-core platforms compared to equivalent clock speed single-core platforms, latencies measured in the 10-30 microseconds range on single- core systems are reduced by an order of magnitude to 1-3 microseconds on dual-core systems. With such low latencies real-time application control loops can execute in the 50- 200 microsecond range with very high precision. Multi-core processors mean faster scan times can be implemented and higher quality controllers can be deployed.

Unlike the application of a conventional VMM to an IT server platform, the TenAsys real- time hypervisor is best suited for control and data acquisition applications that require determinism, performance, and security and would also benefit from the use of a GPOS, such as Windows or Linux.

Industrial Control

Take, for example, a Windows-based industrial control application where the real-time tasks run on a PLC engine that must operate continuously, and with very precise timing. Windows is used as a man-machine interface to set important parameters, monitor system status, provide disk logging, and act as an interface to the factory enterprise network. Normally, two hardware platforms would be employed because the Windows OS can guarantee neither continuous operation nor precision timing to satisfy the needs of the PLC engine.

Combining the two functions on a dual-core hardware platform saves time, money, and space and simplifies development and communication between the two parts of the system: the Windows enterprise interface and the PLC control engine. The TenAsys real-time hypervisor hosts each part in a separate virtual machine, where each virtual machine has a dedicated CPU core to insure timeliness of response and ample performance.

Because Windows and the PLC engine reside in separate virtual machines, the PLC engine is assured of its need for continuous operation, regardless of the state of Windows. This means that Windows can be rebooted, for whatever reason, without impeding operation of the PLC engine. By dedicating a CPU core to each virtual machine, the real-time integrity and performance of the PLC engine is also maintained.

Legacy Real-time

Legacy real-time systems are a prime application for the TenAsys real-time hypervisor. Proven real-time applications can be hosted in a guest real-time virtual machine with little or no modification—enabling developers to maximize code preservation and maintain performance while using the real-time hypervisor to add new features via a parallel general purpose operating system. Helper functions can be added to the real-time virtual machine to communicate with the GPOS virtual machine for coordinating between the existing legacy code and new feature code. Shared memory or a virtual Ethernet channel can be used to communicate between the two environments. If necessary, obsolete hardware can be simulated because the VT-x technology allows the hypervisor to isolate I/O accesses.

DSP Replacement

Modern processors contain instructions designed to efficiently perform DSP arithmetic operations. As a group these are known as SIMD instructions (Single Instruction, Multiple Data). On Intel architecture processors the SIMD instructions are referred to as the MMX and SSE instructions. MMX instructions were introduced first and are limited to integer operations. SSE instructions were introduced later and can efficiently handle floating point DSP operations and vector arithmetic. These SIMD instructions are ideal for implementing digital filters, the basis of complex digital control algorithms, pattern recognition systems, and video streaming and mixing applications.

page 10 of 11 Copyright © 2006, TenAsys Corporation • www.tenasys.com • August, 2006 Rather than dedicate a costly DSP board along with an RTOS and/or a GPOS system to implement a complex control system it makes economic sense to combine these functions onto a single hardware platform. Where previously a GPOS system was used for human interface and enterprise network access, an RTOS box for primary machine control, and a DSP board for high-performance data acquisition and filtering, it is now possible to combine them on a single multi-core processor system. Using the TenAsys real-time hypervisor, one could simultaneously operate three virtual machines, dedicating a CPU core to each function, without the cost and complicated logistics associated with multiple separate pieces of hardware.

Conclusion

The net gains from the application of TenAsys real-time virtual machine technology on Intel multi-core processor platforms are elimination of redundant computer and communication hardware, faster communication and coordination between RTOS and GPOS subsystems, improved reliability and robustness, re-use of proven legacy applications, and simplified development and debugging. The ability to reboot one OS while the other runs without interruption promotes significant cost savings by condensing systems comprised of separate GPOS, DSP, and real-time hardware subsystems onto a single, multi-core, hardware platform.

Multi-core processors easily support multiple operating systems and high-performance, low- latency, real-time applications by dedicating a CPU core to the RTOS. The CPU instruction cycles of the RTOS core are available 100% of the time for its real-time applications. Contention for key resources, such as CPU cycles, pipelines, and the FPU, are avoided. Intel Virtualization Technology is used to remove I/O contentions by isolating and dedicating those devices that belong to the RTOS for exclusive use by real-time applications. I/O that can be shared, like keyboard, mouse, and enterprise network interfaces, are presented as virtual devices for use by all virtual machines.

Coordination between multiple CPU cores is accomplished via built-in inter-processor communication mechanisms, eliminating context switches between multiple operating systems. Real-time interrupt latencies are reduced by an order of magnitude, from 10-30 microseconds down to 1-3 microseconds. Complex real-time application loop cycle times in the 50-200 microsecond range are supported with very high precision and accuracy. The result is an order of magnitude improvement in the quality and bandwidth of real-time control algorithms that can be deployed on a real-time Windows or Linux platform.

[1] Singh, Amit, An Introduction to Virtualization, ca. 2004, http://www.kernelthread.com/publications/virtualization [2] Answers.com Encyclopedia, Virtual Machine, http://www.answers.com/topic/virtual-machine [3] Neumann, Dean, et.al., Intel Virtualization Technology in Embedded and Communications Infrastructure Applications, Intel Technology Journal, August 2006, http://www.intel.com/technology/itj/ [4] Uhlig, Rich, et.al., Intel Virtualization Technology, Computer Magazine, IEEE Computer Society, vol. 38, issue 5, May 2005, pp. 48–56, http://doi.ieeecomputersociety.org/10.1109/MC.2005.163 [5] Intel Corporation, Intel® Advanced Platform Technologies, January 2006, pp. 2-3, http://www.intel.com/technology/advanced_comm/311321.pdf