To the light of extinguished STAR VAX MP A multiprocessor VAX simulator Technical Overview Sergey Oboguev [email protected] Initial edition: November 21, 2012 Last update: December 31, 2012 (rev. 3) Content and intended audience: This document describes internal design of VAX MP multiprocessor simulator. It is intended for VAX MP developers or those interested in VAX MP internal architecture, not for end users. End users should refer to “VAX MP OpenVMS User Manual”. This document discusses VAX MP design principles and decisions, key problems met during VAX MP design and implementation, their possible solutions and rationales for chosen and rejected solutions. The document does not attempt to describe VAX MP internal interfaces, data structures or other internal details, since that would have required a much longer document. Only a brief description is made of some basic interfaces introduced in the portability layer and key interfaces changed compared to the original uniprocessor SIMH, as well as interfaces relevant for modifying SIMH virtual device handlers for multiprocessing environment. 2 Introduction VAX MP is a derivative variant of popular SIMH simulator1 that extends original SIMH to simulate multiprocessor VAX system. The intent of VAX MP is to run OpenVMS and perhaps in the future also BSD Unix in SMP (symmetric shared-memory multiprocessor) mode. In technical parlance this is also known as SMP virtualization. VAX MP targets primarily hobbyist use of OpenVMS and possibly BSD Unix for the purposes of retro- computing, i.e. preservation of computer history and the past of computer and software technology. While the intent of VAX MP is to provide reasonably stable SMP VAX system simulation environment for hobbyist purposes, multiprocessing poses unique challenges to quality assurance, as explained below – challenges that go well beyond the resources available within a single-developer free-time non- commercial project. Therefore VAX MP is not recommended for production use. If you are looking for a production solution, please examine commercial options, such as VAX simulator from Migration Specialties or Charon-VAX/nuVAX product ranges by Stromasys.2 * * * There are two conceivable high-level approaches to simulating a multiprocessor. One straightforward approach is for the simulator to use a single thread of execution that, during each single cycle of execution, loops across all active virtual processors executing one instruction at a time from every virtual processor’s instruction stream (skipping VCPUs marked idle) and thus automatically keeps VCPUs in a lockstep. This approach may have advantages for the purposes of system testing during its development3, but is less attractive if the primary interest is the use of legacy software, such as the exactly case with retro-computing. Single-threaded approach avoids many problems of multi- threaded approach described below, but at the cost of mostly defeating the purpose of multiprocessing. 1 This document does not make semantic distinction between terms simulator and emulator and uses them interchangeably, as is the common practice in the recent years, save for terminological purists. 2 In addition to increased stability, these commercial products offer improved performance, since unlike SIMH which is an instruction-level interpreter, they perform JIT dynamic binary translation. As ballpark figures, whereas single-CPU version of VAX MP executes on Intel i7 3.2 GHz based PC at about 30-40 VUPS, commercial version of Charon-VAX simulating single-processor MicroVAX according to the manufacturer executes at 125 VUPS, and Charon-VAX/66x0 with six processors delivers up to 700 VUPS. Some of the mentioned commercial products also offer increased memory size beyond virtual MicroVAX 3900 limit of 512 MB (for example Charon-VAX 66x0 simulator offers up to 3 GB), making them altogether more suitable for production use. 3 Or rather, instruction-level simulator is just one of possible testing simulators. It may simulate real-world computer only to an extent: for example, clock drifts may be simulated but only somewhat, whereas inter-cache interaction would be much more difficult to simulate and would probably require register-transfer-level or logical- block-level or even gate-level simulator. 3 Another approach, one that VAX MP takes, is to simulate virtual multiprocessor on top of a real hardware multiprocessor. The advent of on-chip multiprocessors readily available in consumer PCs makes it an attractive solution for hobbyist use. The targets of VAX MP project are: • Primary and initial targeted guest operating system is OpenVMS, not only because it is the most venerable system associated with VAX, but also the only currently available operating system that supports VAX multiprocessors. (Save Ultrix, which we do not target as having much lower historical interest and uniqueness.) • It might also be possible to add in the future support for OpenBSD and/or NetBSD Unix. These operating systems do support VAX and they do support some multiprocessor platforms, but they do not support multiprocessing on VAX platform, hence they would have to be updated to add multiprocessing support for VAX to them, which obviously would be Das Glasperlenspiel of the second order, hence the possibility of adding support for these systems is kept in mind but they are not targeted by the initial release of VAX MP which keeps focused solely on VMS. Thus the discussion in this document will be somewhat VMS-centric, however much of VMS-related notes do also apply to OpenBSD and/or NetBSD as well should they ever be updated to support multiprocessing on VAX. • No support for Ultrix SMP is planned for VAX MP at this point, owing to the difficulties obtaining Ultrix source code and listings and also very low expected level of interest for running Ultrix in SMP mode. However, just like all other VAX operating systems, it is possible to boot and run Ultrix on VAX MP in uniprocessor mode. • Host systems targeted by general design of VAX MP are, theoretically, any cache-coherent SMP systems4 with no or low NUMA factors. VAX/VMS and VAX version of OpenVMS assume flat uniform symmetric memory access for all the processors and do not accommodate NUMA factors. Any significant NUMA factors would cause inefficient process-to-resource allocation within the guest operating system and, most importantly, jeopardize guest operating system stability; for this reason host systems with significant NUMA factors are not supported. However it might be possible to run VAX MP on high-NUMA system as long as VAX MP is allocated to a partition with low intra-region NUMA factor. It might still very well be that virtualized OpenVMS will run fine (perhaps somewhat sub- efficiently, but stably) on a multiprocessing host system with moderate NUMA factors, however this aspect had not been thought through for the initial VAX MP release and the author does not have NUMA system in his possession available for testing. Should the need arise, the issue of 4 Host systems with no cache coherency are not targeted, as they are unable to virtualize existing guest operating systems. 4 adaptability of VAX MP to NUMA configurations may be revisited later. • On a related matter, 2-way hyper-threaded processors are supported for the host system. The effect of 2-way hyper-threading is that virtual processor can execute quite a bit slower when the physical core it is mapped to is shared by two threads, compared to when the virtual CPU’s thread has the core exclusively to itself.5 The accommodation of temporal variance in VCPU speed induced by hyper-threading is made possible by two factors: o If hyper-threaded processors are detected on the host system, VAX MP paravirtualization layer adjusts operating system’s busy-wait loop calibration counters increasing them approximately two-fold (more precisely, by 1.8).6 This effectively increases sanity intervals for spinlock acquisition, for inter-processor request-response time and (less important) for virtual device interaction delays in the kernel to offset any possible CPU speed variability.7 o Default values of VAX/VMS (OpenVMS) SMP sanity timers have large safety reserve margin to them, about 10x even on a processor with 1 VUPS performance, and effectively even larger on more powerful processors, as would be for today’s hosting environments.8 This creates ample safety reserve against false premature expiration of SMP sanity timers. 5 Let us say, perhaps about 1.7 times slower as a ballpark figure, albeit the actual number can vary widely, depending on the behavior of the second thread executed by the hyper-threaded processor. The slowdown is caused by the threads competing for execution units, cache space and memory access bandwidth of the CPU core. 6 For VMS these are kernel variables EXE$GL_TENUSEC, EXE$GL_UBDELAY and their shadow copies CPU$L_TENUSEC and CPU$L_UBDELAY in the per-CPU database. 7 If host system is itself virtualized, such as Windows or Linux ran inside VMWare or other virtual machine monitor that conceals underlying physical topology from virtualized host machine, this VM monitor may fail to properly expose physical processor topology and packaging to guest OS, and hence to VAX MP running on top of that virtual OS instance, and falsely represent all of its virtual processors as independent cores, whereas in fact they are implemented as threads executing on hyper-threaded cores. Such a failure to convey information about processor topology might negatively affect VAX MP stability and reduce guest OS safety margin. When running such a configuration, it is therefore important to verify that SMT slow-down factor printed by VAX MP at startup matches actual host system processor topology. For 2-way SMT, SMT slow down factor should be about 1.8. If printed slow-down factor mismatches actual host topology, it should be adjusted manually via VAX MP SIMH console command CPU SMT <slow-down-factor> 8 Single-CPU VAX performance of VAX MP running on the machine with Intel i7 3.2 GHz CPU is about 30-40 VUPS, whereas multiprocessor performance with cores hyperthreaded would be on the order of 15 - 25 VUPS per VCPU.