Comprehensive and Efficient Runtime Checking in System Software through Watchdogs Chang Lou Peng Huang Scott Smith Johns Hopkins University Johns Hopkins University Johns Hopkins University ACM Reference Format: check whether it is functioning properly at runtime and au- Chang Lou, Peng Huang, and Scott Smith. 2019. Comprehensive and tonomously take corrective action when serious problems are Efficient Runtime Checking in System Software through Watchdogs. detected, and this requires effective failure detectors. In Workshop on Hot Topics in Operating Systems (HotOS ’19), May Despite their importance, failure detectors have tradition- 13–15, 2019, Bertinoro, Italy. ACM, New York, NY, USA,7 pages. ally been implemented as a thin module with simple logic https://doi.org/10.1145/3317550.3321440 independent of the underlying code: a monitored process is as- sumed to be working as long as it does something periodically Abstract based on the contract with the external detector, e.g., replies to pings, sends heartbeat messages, or maintains sessions. Systems software today is composed of numerous modules This works fine for fail-stop failures, but it cannot detect com- and exhibits complex failure modes. Existing failure detec- plex gray failures [23] including partial disk failures [31], tors focus on catching simple, complete failures and treat limplock [17], fail-slow hardware [11, 20], and state corrup- programs uniformly at the process level. In this paper, we ar- tion [16], which are common in large cloud infrastructure. gue that modern software needs intrinsic failure detectors that These failure detectors are overly generic, treating all mon- are tailored to individual systems and can detect anomalies itored software as coarse-grained “nodes”. The resulting fail- within a process at finer granularity. We particularly advo- ure modes are inherently too simple to represent the com- cate a notion of intrinsic software watchdogs and propose plex behavior of modern software. The recently proposed an abstraction for it. Among the different styles of watch- Panorama [22] attempts to address this limitation by convert- dogs, we believe watchdogs that imitate the main program ing any requester of a monitored process into a logical ob- can provide the best combination of completeness, accuracy server and captures error evidence in the request paths. While and localization for detecting gray failures. But, manually this approach can enhance failure detection, the observers constructing such mimic-type watchdogs is challenging and cannot identify why the failure occurs or isolate which part of time-consuming. To close this gap, we present an early explo- the failing process is problematic, making subsequent failure ration for automatically generating mimic-type watchdogs. diagnosis painful and time-consuming [4]. Hierarchical spies as proposed in Falcon [27] has similar limitations. 1 Introduction In this position paper, we argue that system software needs Software inevitably fails. In programs designed for interac- intrinsic failure detectors that detect anomalies within a pro- tive usage, a user can observe anomalies in the software and cess at finer granularity. An intrinsic detector monitors inter- react. For example, the user of a text editor may find that nally for subtle issues specific to a process, e.g., checking the backspace key can no long delete characters so she may if a Cassandra background task of SSTable compaction is restart the process and restore the last saved file. But a large stuck. Importantly, an intrinsic detector could pinpoint the fraction of software today is a component of some online problematic code region along with the payload for diagnos- service that demands high availability with minimal man- ing and reproducing production failures. This precise fault ual intervention [13, 29]. Such software needs to proactively information could further help expedite recovery. In particular, we advocate a promising yet under-explored watchdogs Permission to make digital or hard copies of all or part of this work for type of intrinsic failure detector – intrinsic – for personal or classroom use is granted without fee provided that copies are not system software. Watchdogs are widely used in embedded made or distributed for profit or commercial advantage and that copies bear devices such as deep space probes whose operation environ- this notice and the full citation on the first page. Copyrights for components ment can be remote and hostile due to extreme temperature, of this work owned by others than the author(s) must be honored. Abstracting high-intensity radiation, etc. They are hardware circuits con- with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request trolled by embedded software to automatically detect anom- permissions from [email protected]. alies within the system, and reset the processor or microcon- HotOS ’19, May 13–15, 2019, Bertinoro, Italy troller in response to failure. In the broader software domain, © 2019 Copyright held by the owner/author(s). Publication rights licensed to however, only a few mature systems have adopted the watch- ACM. dog concept [1]. Even for these systems, the watchdog mod- ACM ISBN 978-1-4503-6727-1/19/05. $15.00 ules are designed in a shallow way (e.g., to only periodically https://doi.org/10.1145/3317550.3321440 HotOS ’19, May 13–15, 2019, Bertinoro, Italy Chang Lou, Peng Huang, and Scott Smith Table 1. Comparison of three abstractions: crash failure detector, error handler and watchdog. Abstraction Scope Execution Goal Checks Target Analogy Crash FD Extrinsic Concurrent Inform another process about a process’s status to benefit a group Liveness Protocol failure Heartbeat Error handler Intrinsic In-place Mitigate a known error in a specific program point to prevent failure Safety Low-level error Immune sys. Watchdog Intrinsic Concurrent Monitor overall software health and assess if software is functioning Safety+Live. Partial failure Blood test check some status code), which are in fact equivalent to ex- use that information to adjust its behavior for e.g., consensus, trinsic detectors. It is unclear what a truly intrinsic software broadcasting, load balancing, or deciding a replica set. In watchdog should be like and how to systematically write it. comparison, the primary goal for a watchdog is to benefit To shed some light on this under-explored type of intrinsic the individual process it monitors, and a fault that does not failure detector, we propose a unique software watchdog ab- immediately concern the process group protocol may still be straction (§3.1), elucidate characteristics of a good watchdog of interest to a watchdog. Watchdogs and traditional failure (§3.2), and describe design principles for coding watchdogs detectors can and should co-exist. Watchdogs have better ac- (§3.3). Based on these design principles we believe that a cess to internal system states and thus will be particularly particularly effective type of watchdog is one that mimics the suitable for dealing with partial failures. Crash failure detec- main program. But manually writing such a watchdog is time- tors have stronger isolation. So if severe thrashing causes an consuming and challenging to get right as it needs to simulate entire process to become unresponsive and jeopardizes the the key interactions of the main program without negatively watchdogs as well, a crash detector can still catch the failure. impacting execution. To address this gap we present an early In contrast with the popularity of watchdogs in embedded exploration of how customized, mimic-type watchdogs can systems, mainstream software engineering does not put much be generated for a given piece of software (§4), and conclude emphasis on designing intrinsic watchdogs despite a similar with a discussion of challenges and future directions (§5). need for robustness. The primary way for system software to catch subtle failures today is to rely on error handlers or 2 Background and Motivation in-place assertions. Although error handlers do a good job of detecting some program-specific faults, they are designed as A Watchdog Timer (WDT) [12] is an essential component part of the main business logic, with the goal of mitigating found in embedded systems to detect and handle hardware some specific known error condition associated with a spe- faults or software errors. WDT’s use internal counters that cific program point for the program to continue execution. So start from an initial value and count down to zero. When the for example catching an EINTR error during a write system counter reaches zero, the watchdog resets the processor. In a call is the target of an error handler, and it may retry suc- multi-stage watchdog, it will initiate a series of actions upon cessfully. Watchdogs on the other hand are tasked to monitor timeout, such as generating an interrupt, activating fail-safe overall software health and give a definitive assessment as to states, logging debug information and resetting the processor. whether the software is still functioning properly. In addition, To prevent a reset, the software must keep “kicking” the complex failures such as metadata inconsistency and data in- watchdog, typically by writing to some control port. tegrity require complex checks beyond a single operation and To effectively use a watchdog, the program must do more are thus not suitable for error handlers. Watchdogs can run than just kick the watchdog at regular intervals – the software complex fsck-like checks in parallel to the normal execution should perform sanity checks [19], on e.g. the stack depth, to catch such failures. Liveness-related failures often also do program counter, resource usage, and status of the mechanical not have explicit error signals that can trigger a handler: there components before kicking the watchdog. It is also good is no signal for e.g., write being blocked indefinitely or some practice to insert a flag at each important point of the main thread deadlocking or infinitely looping.