Compute Node OS for the XT3 Jim Harrell Cray Inc
ABSTRACT: Cray is working on different kernels for the compute node operating system. This talk will describe the rationale, requirements, and progress.
KEYWORDS: Light Weight Kernel, Linux, XT
a review of options for a CNOS in the 1. Introduction near future.
The talk upon which this paper is based A review of the current CNOS, was designed to discuss the purpose and Catamount, provides the basis for issues around a small project working judging alternatives. Catamount is well on common Compute Node Operating understood and has set the standard on Systems, CNOS, for Several Cray XT for what a CNOS can be expected to products. The goal is to apprise the do. audience of the status of the work and next steps, but also to ensure that The current work on an alternative CNOS there is an understanding of the has focused on a Linux kernel. This has complexities that this project faces. caused a lot of discussion both For the purpose of this discussion we positive and negative. The use of a will use the CNOS term for the kernel Linux kernel for this project is not to that runs on a compute node. There are be construed as an intention to run a variety of other terms like Light full Linux on compute nodes. The Linux Weight Kernel, LWK, that would be kernel is modified to only support equally useful – but rather than use application processes. This differs multiple terms in a single discussion sharply from what a Linux user would we will use just the CNOS term to avoid expect on a standard server. In confusion. This talk was broken into a general, talking about CNOS and Linux number of sections. We will provide an appears to be less productive than overview of each section. talking about a different CNOS without explaining its derivation. We may The basis for a discussion about actually pursue this approach in the Compute Node Operating Systems, CNOS, future as it helps focus on the purpose is built around the software of the CNOS. Architecture of MPP style systems. The first segment of this paper will review There are some other alternative CNOSs the Architecture to re-establish the that could be used. Several are either importance of the CNOS in the System under development or currently more and help to define the functionality prototypes than a kernel Cray could use that is required of the CNOS. in a production environment. However, there are good reasons to track the The next section will describe a research and development in this area. rationale and set of requirements for a Promising developments might be useful CNOS. This project was started because in the future and should not be there is a set of new hardware ignored. We will review a few of these requirements in the future that will be CNOS possibilities. difficult to meet with the current CNOS. There is also a set of software Finally, we will discuss the current features that are requested to provide status of the project and the next a broader scope of applications for XT. steps that are being pursued. The These software features are considered project does have some commitments and difficult to add to the current CNOS. these will be ddiscussed. Together these new requirements started
CUG 2006 Proceedings 1 of 8
2. A Review of XT System Architecture •Provides limited set of system call functions – most system calls forwarded The basic organization of XT is very to Service nodes familiar to anyone who has worked on an MPP in the past decades. An MPP is a •Small memory footprint and CPU form of distributed system where the utilization (when not servicing operating system services are provided requests) from a set of nodes that have complete systems installed including the servers Overall, a Compute Node Operating for all the distributed services. These System takes little of the processor nodes are called Service Nodes in the time available and performs the minimum XT Architecture. These nodes are set of functions required to support specialized to provide specific application process(es) assigned to the functions and to scale specific node. All of this ensure the forward functions, such as the number of login progress of a distributed application – nodes to scale the access for which is the purpose of the system. individual users or the number of I/O nodes to scale the access and quantity of disks for the system. The Compute 4. Rationale for a Different Compute Node Nodes, where all the user application processes are run, have a minimal Operating System kernel that provides a small set of There are three basic components of the functions and uses the Service Nodes rationale for a different CNOS – for all the application processes’ changes in processor/node technology, service requests – such as I/O. new features and functions required in the software, and the ability to 3. Compute Node Definition leverage a different CNOS between multiple Cray products. A definition of a Compute Node is often based on describing what the node does There are changes in Socket technology not do as opposed to what the node that are adding more and more cores per does. This is because a Compute Node is socket. The ability to take advantage designed to provide support only for an of these additional cores does factor application process and not for all the into a decision about a new CNOS. services that a normal operating system might provide. For instance, a Compute There are a number of applications that Node will support virtual memory would benefit from some additional CNOS because all processors today implement functionality. These include, memory support using Virtual memory networking access, support for new or techniques. However, Compute Node different file systems, support for Operating Systems will not support Global Arrays, and support for OpenMP. Demand Paging because the cost of the Support for OpenMP is obviously related I/O and the performance lost for a to the number of cores available in a distributed application because of the socket. OpenMP support would be limited time involved in managing the demand to within a node. paging. Cray has other products that use A Compute Node has the following Compute Node style Operating Systems. characteristics: It would be a benefit to Cray to be able to share the development of a CNOS •Provides support for the machine across several products. It would also dependent aspects of the node help move Cray software towards the longer-term goal of Adaptive •Supports the High Speed network (HSN) SuperComputing if the CNOS could be connection – HSN is “allocated” to operated and managed in the same way application usage across products.
•Provides support for application processes allocated to the node
CUG 2006 Proceedings 2 of 8
the application process. The following 5. Cray Requirements for a New CNOS diagram shows the structure of Catamount.
The requirements for a new CNOS are based on the current CNOS, Catamount. Catamount has been successful at setting the standard by which a new CNOS will be judged. The areas Catamount has most influenced are boot time, application start time, and I/O performance. Boot time is an area of influence because Catamount can boot on several thousand nodes in seconds. The second major area of influence is application start time, because the start up time of an application is also a matter of small numbers of seconds for thousands of nodes. The area of I/O performance is a bar set by Catamount that any new CNOS must clear. Catamount support for I/O is sufficient for most applications and scales well. Catamount has also had influence in the number of nodes a CNOS can be expected to support. By supporting more than 10 thousand nodes the expectation is that Catamount memory footprint is minimal. a compute node operating system should Much of the space not left for user be able to support 30 thousand nodes. process(es) is buffer space for Portals. The following graph shows a There are features that are required or 4GB node and the Catamount kernel with desired by customers for their all components broken down to show applications that add to the space used. Obviously the majority of capabilities now supplied by Catamount. the space is left to the user These are support for sockets or process(es). networking, support for SHMEM, UPC, CAF, and Global arrays, signal support, and as already mentioned support for OpenMP. Each of these additional Catamount Memory Usage features brings some issues as well. Support for networking means that each Compute Node will have to have an IP address if the implementation adds a network stack to each compute node. 1 page tables 1.3 QK text and data 1.7 alignment pool (page tables) 1 I/O mapping 10 Page tables (4GB node) 10 Network buffer 6. Catamount Review 64 Shared Memory and Event Queues 1 Alignment space 2 PCT code and data The Catamount kernel is currently used 2 Alignment space 2 PCT Stack on all XT3 systems. The basic structure 16 PCT Buffers 3 mimimum program and runtime librart of Catamount is similar to many 3885 User memory operating systems. There is a low level machine dependent kernel which supports the hardware and system call functionality. There is a set of libraries and support services. In the case of Catamount the support services 7. Catamount Positives are reduced to just the Process Control Thread (PCT) which provides support for Catamount obviously has positive attributes. Just to ensure they get
CUG 2006 Proceedings 3 of 8 mentioned together – they are – •Fully productizing Catamount is scaling, CPU usage, small memory foot substantial work. Logging and debugging print, support for MPI, and a “mature” are still difficult problems code base. There are several existing systems with 5 thousand and 10 thousand •Catamount design has some limitations nodes. This is a proof of existence of that will be difficult to overcome: scaling. The CPU utilization of • File System support Catamount has been documented to be • Application Network Support less than .0008%. This is the • Many system calls missing percentage of time used by the •User/system time operating system when the application • Debugging support (kernel) is making no system calls. This is • Signal support – for exit substantially less than many other • Multiple threads in the kernel systems that use a CNOS. The size of • Lack of support for multi-core or the memory footprint is about 115 SMP Megabytes. This includes all the • Robustness problems Portals buffers. The Portals buffers are over half the total size of the •Sandia had not been planning future Catamoount kernel. In the past versions of Catamount to support 4 microkernels had a suggested size of cores about 10% of memory. Catamount is at this size even for a single Gigabyte •The support for more than 1 or 2 memory size. Catamount does benefit processors/cores is required. from the growth in memory sizes. *Productizing dual-core Catamount has Microkernels had memory sizes of 128 proven problematic. The asymmetry of Megabytes, which made 10% a much the design results in unpredictable smaller number. Catamount supports MPI performance. applications that are the greater proportion of HPC applications currently. The Catamount code base is maturing rapidly as the systems provide 9. Catamount Extensions it with more run time. Any time there is a discussion of issues with software, such as with 8. Catamount Issues Catamount, it is necessary to acknowledge that some of the issues All software has issues. Catamount is could be mitigated by adding the no exception. The following list functionality. There have been some describes the current issues that are discussions about adding some features being pursued. to Catamount and this section describes a few of those features. •Catamount was developed to support a set of MPI applications. In order to First among the features to be added to broaden the application set a number of Catamount is support for multiple features need to be considered. cores, beyond just two cores. A project is underway at Sandia to use the same •Sandia and Cray are sole sources for Master/Slave implementation as exists Catamount development and support. The with Catamount Virtual Node (CVN) and design and development of any new extend it to more than two cores. There feature or fix is done by one or the are obviously positive and negative other organization. points that could be made about the approach, but the work is underway and •Support for new processors and devices will provide a benchmark for any other have to be added after the hardware implementation. becomes available – Support for other software is available at processor or It is also possible to implement other device release. features in Catamount such as OpenMP. This method of implementing OpenMP might provide a lighter weight implementation than a full operating
CUG 2006 Proceedings 4 of 8 system support for OpenMP. Similarly, it may be possible to support Global Arrays by adding the feature to Catamount.
There is a reason for concern in adding many features to Catamount. First, new code takes time to mature and the lightweight nature of Catamount could The current version of compute node easily be lost if too many features are Linux has only the processes needed to added. It is also disheartening how run application processes, a few test often new features take longer to services to help with debugging, a few complete than expected, create logging services, and Lustre. Lustre is unexpected problems, and don’t perform a special problem, as it adds more as well as an existing implementation. services and demons than would be expected. However, to date there have not been substantial problems reported 10. Linux Review as a result of Lustre services.
The current plans for a Linux based compute node operating system is not
Linux as is available for use on a syslogd server. The Linux kernel for a compute node is configured and modified to support just the application processes sent to run on the node. This fits with the description of a compute node
Linux
nscd cron klogd
Portmap mingetty(s
qmgr … This is not proof that Linux can be cupsd used for a compute node operating sshd master system, but it is proof of Linux’s configurability.
powersaved pickup init slpd 11. Linux Positives
resmgrd There are reasons why Linux would be kdm ndbd attractive as a compute node operating system. A number of the reasons are listed here.
Linux CUG 2006 Proceedings 5 of 8
•Linux is an accepted standard for x86 [Portals is a hold over from Catmount – architectures (for Unix) it seems that this imposed requirement]
• Almost all of HPC is using Linux or a •Root file system is required for Linux Unix variant – but •Use of a RAM file system takes •Development is “Shared” among the memory Linux community •RAM file system would have to be •HPC does have a limited following created and maintained and limited influence •RAM file system would limit shared library usage •Linux supports a broad variety of •Shared root – like NFS or Lustre may applications and application models simply not work at 30K nodes •New hardware and new features are available in Linux before other systems •Work in Programming Environment – MPI, libraries, and tools •Linux is still a lightweight •Although much of this work would architecture make us more standard
•Networking adds complexity 12. Linux Issues •Each node needs an IP address •ARP and other noise needs to be There are a number of concerns and eliminated from the HSN network issues with using Linux as a compute •NAT, RSIP, or alternative for node operating system. None of the routing outside the machine problems appear insurmountable, but will require some work and changes to •Memory footprint is larger. How to make the Linux kernel a suitable constrain it? alternative. The following list of concerns are currently being pursued •User credentials – Need to support and work is going on to understand what some mechanisms changes may be required. •File Systems – Need to aid Lustre •Linux has no concrete examples of its client scaling to 30K nodes use on an HPC system at 5K or 10K nodes
•Linux as a compute node will not 13. Linux – Decisions and Investigations provide all the services that a standard Linux system would provide Using Linux as a compute node operating •Most services and demons would be system does not change the fundamental removed to ensure proper performance architecture of the system. The use of service sand compute nodes remains the •Changes to Linux to provide a better same. A goal of the current project is compute node LWK will have to be to be able to use the CRMS supported outside of the distributions infrastructure without changes. This has been successful so far and enables •Linux memory management does not easy testing of Compute Node Linux, handle alternate page sizes as well as CNL. The initial test vehicle has more would be desired functionality than Catamount, but •Some “patches” from other vendors substantially less than a full Linux may help, but the patches would then be distribution. a support burden The ALPS scheduling package is being •Linux scheduler and clock tick used instead of CPA/YOD/PCT. ALPS is features will likely need to be the placement scheduler for Linux modified to optimize the performance compute node operating system support on Black Widow. Job scheduling is still •Portals performance issues in Linux supported using PBS. LSF and MOAB are need to be investigated and fixed planned.
CUG 2006 Proceedings 6 of 8
Lustre is the file system supported in •Job start the prototype. The Lustre organization is similar to the standard Linux •I/O performance cluster configuration, except that the compute nodes only support clients. •Network performance There is work to scale and tune Lustre. If Lustre clients prove untenable it •OS jitter/noise on nodes would be possible to use libLustre as an alternative. •Boot times
There is a standard IP based network •Console/CRMS load set up on the High Speed Network, HSN.
The hosts and IP addresses are fixed and generated automatically. This •Memory usage/availability simplifies configuration and administration. The routes are static •Syslog loading and there is no use of ARP. This ensures than there will not be •Lustre client cpu and memory unnecessary traffic on the HSN. utilization. Currently we are looking at RSIP for a way of providing gateway support to As these measurements are completed the external networks. next set of project requirements will be put into place. At this point the The kernel boots on a compute node in initial numbers look promising, but are 15 seconds. This needs some attention. not complete. We will be looking at a number of changes including reducing the flow of 14. Other Alternatives console messages at boot.
The prototype uses a RAM based root There are other alternatives to file system. The contents are currently Catamount and Linux. There is a growing limited to init, apinit (for ALPS), the community of researchers and developers RCA kernel module, sshd, and a few who are looking at compute node debugging tools. There has been some operating systems specifically for use experimenting with shared libraries, with large scalable systems. This is a but currently the plan is to force change from the past and is quite applications to load all the libraries welcome. A few examples of the work during the application build. going on are virtualisation layers such as Xen. This is quite popular in a User credentials, user ids etc. are number of research communities, but may currently managed by ALPS. This limits take some time to mature. DOE supports the use of some library requests to a group called FastOS. The FastOS group change users and groups, but allows the is looking at several ideas including prototype to avoid the use of LDAP and library based runtimes and some NIS. versions of Linux.
There is work to do to improve the One alternative to a single Linux performance of Portals specifically compute node operating environment with the copying of data and page would be to have several Linux management. There is some potential to environments with different use GART and MRT to help with the capabilities. “Fatter and thinner” memory access. Linux compute node environments could offer a variety of services at Scaling studies have just begun. There has been different scales for different application requirements. a little testing at about two thousand nodes. A number of measurements need to be taken One potential in the future is to have including: a variety of compute node operating systems running on the same distributed
CUG 2006 Proceedings 7 of 8 system. This would require careful support for configuration and scheduling but would enable alternatives to be used as needed or required by different applications. In the long run the purpose of a compute node operating system is the support of application processes – and perhaps no one operating environment is sufficient.
About the Author Jim Harrell is Director of the Software Architecture Group. He can be reached at [email protected].
CUG 2006 Proceedings 8 of 8