Error Sensitivity of the Linux Kernel Executing on PowerPC G4 and Pentium 4 Processors Weining Gu, Zbigniew Kalbarczyk, Ravishankar K. Iyer Center for Reliable and High-Performance Computing Coordinated Science Laboratory University of Illinois at Urbana-Champaign 1308 West Main Street, Urbana, IL 61801 {wngu, kalbar, iyer}@crhc.uiuc.edu Abstract: The goals of this study are: (i) to compare Linux • While the activation1 of errors is generally similar for kernel (2.4.22) behavior under a broad range of errors on both processors, the manifestation percentages for the two target processors—the Intel Pentium 4 (P4) running Pentium 4 are about twice as high. RedHat Linux 9.0 and the Motorola PowerPC (G4) running For stack errors, there is a significant difference between YellowDog Linux 3.0—and (ii) to understand how architec- • the two processors in manifestation percentages (56% for tural characteristics of the target processors impact the error P4 versus 21% for G4). A similar trend is observed in sensitivity of the operating system. Extensive error injection kernel data errors, where 66% (for P4) and 21% (for G4) experiments involving over 115,000 faults/errors are con- of injected errors manifest as crashes. ducted targeting the kernel code, data, stack, and CPU sys- tem registers. Analysis of the obtained data indicates signifi- • The variable-length instruction format of the P4 makes it cant differences between the two platforms in how errors possible for a bit error to alter a single instruction into a manifest and how they are detected in the hardware and the sequence of multiple valid (but incorrect from the appli- operating system. In addition to quantifying the observed cation semantic point of view) instructions. On one hand, differences and similarities, the paper provides several ex- this leads to poorer diagnosability: executing an incorrect amples to support the insights gained from this research. instruction sequence may crash the system and generate 1 Introduction an exception, which does not isolate the actual cause of The dependability of a computing system (and hence of the the problem and obstructs diagnosis. On the other hand, services it provides to the end user) depends to large extent the same feature has the potential to reduce crash la- on the failure-resilience of the underlying hardware (proces- tency: executing an incorrect instruction sequence is sor) and operating system. Understanding a system’s sensitiv- likely to make the system fail fast. ity to errors and identifying its error propagation patterns and • Less compact fixed 32-bit data and stack access makes single points of failure are thus of primary importance in se- the G4 platform less sensitive to errors. The sparseness lecting a computing platform and in assessing tradeoffs in- of the data can mask errors, e.g., a larger presence of un- volving cost, reliability, and performance. This study investi- used bits in data items means that altering any unused bit gates two related aspects of this issue: (i) how the Linux ker- is inconsequential, even if the corrupted data is used. nel responds to errors that impact kernel code, kernel data, The more optimized access patterns on the P4 increase kernel stack, and processor system registers, and (ii) how the chances that accessing a corrupted memory location processor hardware architecture (instruction set architecture will lead to problems. and register set) impacts kernel behavior in the presence of errors. The goal is to compare Linux behavior under a broad 2 Related Work range of errors on two different platforms and to understand Failure behavior/characterization of operating systems has how architectural characteristics of the target processors im- been the focus of several studies. This section briefly reviews pact the error sensitivity of the operating system. Two target representative examples. Linux-2.4.22 systems are used: the Intel Pentium 4 (P4) run- User-level testing by executing API/system calls with errone- ning RedHat Linux 9.0 and the Motorola PowerPC (G4) run- ous arguments. Ballista [14] provides a comprehensive as- ning YellowDog Linux 3.0. sessment of 15 POSIX-compliant operating systems and li- Fault/error injection is used to stress the Linux system while braries as well as the Microsoft Win32 API. Fuzz [17] tests running workload programs. The UnixBench [23] benchmark system calls for responses to randomized input streams; the suite is used to profile kernel behavior and to identify the study addresses the reliability of a large collection of UNIX most frequently used functions representing at least 95% of utility programs and X-Window applications, servers, and kernel usage. Over 115,000 faults/errors are injected into the network services. The Crashme benchmark [6] uses random kernel code, data, stack, and CPU registers. The response of input response analysis to test the robustness of an operating the kernel (e.g., crash, hang, fail silence violation, not- environment in terms of exceptional conditions under fail- manifested) on each target system is automatically monitored ures. and logged. While our previous study focused on errors in the code segment of the Linux kernel [9], the current work inves- tigates the effect of the architecture on kernel sensitivity to a 1 Error activation cannot be determined while injecting into system registers, broad range of errors. Major findings are: as we do not have the ability to monitor kernel access to these registers. Error injection into both kernel and user space. Several stud- The error injection environment, shown in Figure 1, consists ies have directly injected faults into the kernel space and of (i) kernel-embedded components (injectors, crash handlers, monitored and quantified the responses. FIAT [2], an early and data deposit module) for P4 and G4 architectures, (ii) a fault injection and monitoring environment, experiments on user-level NFTAPE control host, which prepares the target SunOS 4.1.2 to study fault/error propagation in the UNIX addresses/registers (to be injected), starts the workload using kernel. FINE [12] injects hardware-induced software errors benchmark (UnixBench [23]), and logs injection data for and software faults into UNIX and traces the execution flow analysis, (iii) a hardware monitor (watchdog card) to detect and key variables of the kernel. system hangs/crashes in order to provide auto reboot if needed, and (iv) a remote crash data collector that resides on Xception [5] uses the advanced debugging and performance monitoring features existing in most modern processors to the control host computer to receive crash data by UDP con- inject faults and monitor their activation and impact on target nections. The latter capability is an important extension to the system behavior. Xception targets PowerPC and Pentium existing NFTAPE framework and allows reliably collecting processors and operating systems ranging from Windows NT the crash data even if the underlying file system (on the target node) is not accessible. The instrumentation to collect data on to proprietary, real-time kernels (e.g., SMX) and parallel op- erating systems (e.g., Parix). error latency (cycles-to-crash) is also added. MAFALDA [1] analyzes the behavior of Chorus and LynxOS … microkernels in the presence of faults. In [4], User Mode PPC Pentium Linux (equivalent to a virtual machine, representing a kernel) NFTAPE Target System executing on top of the real Linux kernel is used to perform Control Data Linux kernel fault injection via the ptrace interface. Host (Linux Kernel 2.4.22) Analysis •Injector In [19], SWIFI is employed to guide the design and Workload • Crash Handler • Data Deposit implementation of the Rio File Cache system on top of Remote FreeBSD operating system. Edwards and Matassa in [8] use Workload Crash Data ReColmolteector fault injection for hardening kernel device drivers. A state Crash Data Hardware Monitor machine is constructed to track the state of hardware and to Collector support the injection of faults at a specific run-time state. Figure 1: Error Injection Environment Other methods to evaluate the operating system. Operating 3.2 Approach systems have been evaluated by studying the source code, An automated error injection process, illustrated in Figure 2, collecting memory dumps, and inspecting error logs. For ex- includes the following three major steps: ample, Chou et al. [7] present a study of Linux and OpenBSD STEP 1: Generate injection targets. The target ad- kernel errors found by automatic, static, compiler analysis at dress/register generator provides error injection targets for the source code level. Lee et al. [15] use a collection of mem- the following: (i) code injection—an instruction breakpoint ory dump analyses of field software failures in the Tandem location based on selected kernel functions, kernel subsys- GUARDIAN90 operating system to identify the effects of tems, and kernel code address ranges; (ii) stack injection— software faults. Xu et al. [24] examine Windows NT cluster the bit patterns to inject at randomly chosen (at runtime) ker- reboot logs to measure dependability. Sullivant et al. [22] nel process; (iii) system register injection—system registers study MVS operating system failures using 250 randomly to inject; and (iv) data injection—random locations in the sampled reports. kernel data (both initialized and uninitialized) section. The 3 Methodology error injection targets are generated and stored before an error Software-implemented error injection is employed to experi- injection campaign is initiated. As a result, the activation rate mentally assess the error sensitivity of the Linux kernel exe- may not be 100%, as some of the pre-generated errors are cuting on PowerPC G4 and Pentium 4 processors. Single-bit never injected because a corresponding breakpoint is never errors are injected into kernel stacks, kernel code sections, reached. kernel data structures, and CPU system registers while run- STEP 2: Inject errors. The kernel injector obtains pre- ning benchmark programs.