Robustness Testing of the Microsoft Win32 API Charles P. Shelton Philip Koopman Kobey DeVale ECE Department & ICES ECE Department ECE Department Carnegie Mellon University Carnegie Mellon University Carnegie Mellon University Pittsburgh, PA, USA Pittsburgh, PA, USA Pittsburgh, PA, USA
[email protected] [email protected] [email protected] Abstract Unfortunately, Windows operating systems have ac- quired a general reputation of being less dependable than Although Microsoft Windows is being deployed in Unix-based operating systems. In particular, the infamous mission-critical applications, little quantitative data has "Blue Screen Of Death" that is displayed as a result of Win- been published about its robustness. We present the results dows system crashes is perceived by many as being far of executing over two million Ballista-generated exception handling tests across 237 functions and system calls more prevalent than the equivalent kernel panics of Unix involving six Windows variants, as well as similar tests operating systems. Additionally, it is a common (although conducted on the Linux operating system. Windows 95, meagerly documented) experience that Windows systems Windows 98, and Windows CE were found to be vulnerable need to be rebooted more often than Unix systems. How- to complete system crashes caused by very simple C ever, there is little if any quantitative data published on the programs for several different functions. No system dependability of Windows, and no objective way to predict crashes were observed on Windows NT, Windows 2000, whether the impending move to Windows 2000 will actu- and Linux. Linux was significantly more graceful at ally improve dependability over either Windows 98 or Win- handling exceptions from system calls in a dows NT.