DOCSLIB.ORG

Fault Tolerance

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Fault Tolerance Fault Tolerance 1/65 Fault Tolerance I Fault tolerance is the ability of a distributed system to provide its services even in the presence of faults. I A distributed system should be able to recover automatically from partial failures without seriously affecting availability and performance. 2/65 Fault Tolerance: Basic Concepts (1) I Being fault tolerant is strongly related to what are called dependable systems. Dependability implies the following: I Availability: The percentage of the time the system is operational. I Reliability: The time interval for which the system is operational. I Safety I Maintainability I In-class Exercise: How is is availability different from reliability? How about a system that goes down for 1 millisecond every hour? How about a system that never goes down but has to be shut down two weeks every year? 3/65 Fault Tolerance: Basic Concepts (2) I A system is said to fail when it cannot meet its promises. For example, if it cannot provide a service (or only provide it partially). I An error is a part of the system’s state that may lead to failure. The cause of an error is called a fault. I Types of faults: I Transient: Occurs once and then disappears. If a operation is repeated, the fault goes away. I Intermittent: Occurs, then vanishes of its own accord, then reappears and so on. I Permanent: Continues to exist until the faulty component is replaced. I In-class Exercise: Give examples of each type of fault for your project! 4/65 Failure Models (1) 5/65 Failure Models (2) I Fail-stop: failures that can be reliably detected I Fail-noisy: failures that are eventually detected I Fail-silent: we cannot detect between a crash failure or omission failure I Fail-safe: failures that do no harm I Fail-arbitrary or Byzantine Failures: I Arbitrary failures where a server is producing output that it should never have produced, but which cannot be detected as being incorrect. I A faulty server may even be working with other servers to produce intentionally wrong answers! 6/65 Failure Masking by Redundancy I Information Redundancy. For example, adding extra bits (like in Hamming Codes, see the book Coding and Information Theory) to allow recovery from garbled bits. I Time Redundancy. Repeat actions if need be. I Physical Redundancy. Extra equipment or processes are added to make the system tolerate loss of some components. 7/65 Process Resilience Achieved by replicating processes into groups. I How to design fault-tolerant groups? I How to reach an agreement within a group when some members cannot be trusted to give correct answers? 8/65 Flat Groups Versus Hierarchical Groups 9/65 Failure Masking Via Replication I Primary-backup protocol. A primary coordinates all write operations. If it fails, then the others hold an election to replace the primary. I Replicated-write protocols. Active replication as well as quorum based protocols. Corresponds to a flat group. I A system is said to be k fault tolerant if it can survive faults in k components and still meet its specifications. I For fail-silent components, k + 1 are enough to be k fault tolerant. I For Byzantine failures, at least 2k + 1 extra components are needed to achieve k fault tolerance. I Requires atomic multicasting: all requests arrive at all servers in same order. This can be relaxed to just be for write operations. 10/65 Consensus in Faulty Systems The model: a very large collection of clients send commands to a group of processes that jointly behave as a single, highly robust process. To make this work, we need to make an important assumption: In a fault-tolerant process group, each nonfaulty process executes the same commands, in the same order, as every other nonfaulty process. I Flooding Consensus (for fail-stop failures) I Paxos Consensus (for fail-noisy failures) I Byzantine Agreement (for arbitrary failures) 11/65 Flooding Consensus I A client contacts a group member (P1;:::;Pn) requesting it to execute a command. Every group member maintains a list of commands: some from clients and some from other group members. I In each round, a process Pi sends its list of proposed commands it has seen to every other process. At the end of a round, each process merges all received proposed commands into a new list, from which it deterministically selects a new command to execute (if possible) I A process will move on to the next round without having made a decision, only when it detects that another process has failed. P1 P2 decide P3 decide P4 decide 12/65 Consensus in Systems with Arbitrary Faults I Two Army Problem I Non-faulty generals with unreliable communication. 13/65 Byzantine Agreement Problem (1) Problem: I Red army in the valley, n blue generals each with their own army surrounding the red army. I Communication is pairwise, instantaneous and perfect. I However m of the blue generals are traitors (faulty processes) and are actively trying to prevent the loyal generals from reaching agreement. The generals know the value m. Goal: The generals need to exchange their troop strengths. At the end of the algorithm, each general has a vector of length n. If ith general is loyal, then the ith element has their correct troop strength otherwise it is undefined. Conditions for a solution: I All loyal generals decide upon the same plan of action I A small number of traitors cannot cause the loyal generals to adopt a bad plan 14/65 Byzantine Agreement Problem (2) Setup: I Assume that we have n processes, where each process i will provide a value vi to other processes. The goal is to let each process construct a vector of length n such that if process i is non-faulty, V [i] = vi . Otherwise, V [i] is undefined. I We assume that there are at most m faulty processes. Algorithm: Step 1 Every non-faulty process i sends vi to every other process using reliable unicasting. Faulty processes may send anything. I Moreover, since we are using unicasting, the faulty processes may send different values to different processes. Step 2 The results of the announcements of Step 1 are collected together in the form of a vector of length n. Step 3 Every process passes its vector from Step 2 to every other process. Step 4 Each process examines the ith element of each of the newly received vectors. If any value has a majority, that value is put into the result vector. If no value has a majority, the corresponding element of the result vector is set to UNKNOWN. 15/65 Byzantine Agreement Example (1) I We have 2 loyal generals and one traitor. I Note that process 1 sees a value of 2 from process 2 (process 2 vector) but sees a value of b for process 2 (process 3 vector). But process 1 has no way of knowing whether process 2 or process 3 is a traitor. So it cannot decide the right value for V [2]. I Similarly, process 2 cannot ascertain the right value for V [1]. Hence, they cannot reach agreement. I For m faulty processes, we need 2m + 1 non-faulty processes to overcome the traitors. In total, we need a total of 3m + 1 processes to reach agreement. 16/65 Byzantine Agreement Example (2) The Byzantine generals problem for 3 loyal generals and 1 traitor: (a) The generals announce their troop strengths (let’s say, in units of 1 kilo soldiers). (b) The vectors that each general assembles based on previous step. (c) The vectors that each general receives. (d) If a value has a majority, then we know it correctly, else it is unknown. In this case, we can reach agreement amongst the non-faulty processes. 17/65 Limitations on Reaching Consensus (1) The general goal of distributed consensus algorithms is to have all the non-faulty processes reach consensus on some issue, and to establish that consensus within a finite number of steps. Possible cases: I Synchronous versus asynchronous systems I Communication delay is bounded or not I Message delivery from the same sender is ordered or not I Message transmission is done through unicasting or multicasting 18/65 Limitations on Reaching Consensus (2) Message ordering Unordered Ordered Communication delay XX X X Bounded Synchronous X X Unbounded X Bounded Asynchronous X Unbounded Process behavior UnicastMulticast Unicast Multicast Message transmission I Most distributed systems in practice assume that processes behave asynchronously, message transmission is unicast, and communication delays are unbounded. 19/65 The CAP Theorem I CAP Theorem (aka Brewer’s Theorem): states that it is impossible for a distributed computer system to simultaneously provide all three of the following guarantees: I Consistency (all nodes see the same data at the same time) I Availability (a guarantee that every request receives a response about whether it was successful or failed) I Partition tolerance (the system continues to operate despite arbitrary message loss or failure of part of the system) This is a chord, this is another, this is a third. Now form a band! We know three chords but you can only pick two Brewer’s CAP Theorem (julianbrowne.com) 20/65 Dealing with CAP I Drop Partition Tolerance I Drop Availability I Drop Consistency I The BASE Jump: The notion of accepting eventual consistency is supported via an architectural approach known as BASE (Basically Available, Soft-state, Eventually consistent). BASE, as its name indicates, is the logical opposite of ACID. I Design around it! 21/65 Reliable Client-Server Communication RMI semantics in the presence of failures.
Load more

Recommended publications
  • The Google File System (GFS)
    The Google File System (GFS), as described by Ghemwat, Gobioff, and Leung in 2003, provided the architecture for scalable, fault-tolerant data management within the context of Google. These architectural choices, however, resulted in sub-optimal performance in regard to data trustworthiness (security) and simplicity. Additionally, the application- specific nature of GFS limits the general scalability of the system outside of the specific design considerations within the context of Google. SYSTEM DESIGN: The authors enumerate their specific considerations as: (1) commodity components with high expectation of failure, (2) a system optimized to handle relatively large files, particularly multi-GB files, (3) most writes to the system are concurrent append operations, rather than internally modifying the extant files, and (4) a high rate of sustained bandwidth (Section 1, 2.1). Utilizing these considerations, this paper analyzes the success of GFS’s design in achieving a fault-tolerant, scalable system while also considering the faults of the system with regards to data-trustworthiness and simplicity. Data-trustworthiness: In designing the GFS system, the authors made the conscious decision not prioritize data trustworthiness by allowing applications to access ‘stale’, or not up-to-date, data. In particular, although the system does inform applications of the chunk-server version number, the designers of GFS encouraged user applications to cache chunkserver information, allowing stale data accesses (Section 2.7.2, 4.5). Although possibly troubling
    [Show full text]
  • Introspection-Based Verification and Validation
    Introspection-Based Verification and Validation Hans P. Zima Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA 91109 E-mail: [email protected] Abstract This paper describes an introspection-based approach to fault tolerance that provides support for run- time monitoring and analysis of program execution. Whereas introspection is mainly motivated by the need to deal with transient faults in embedded systems operating in hazardous environments, we show in this paper that it can also be seen as an extension of traditional V&V methods for dealing with program design faults. Introspection—a technology supporting runtime monitoring and analysis—is motivated primarily by dealing with faults caused by hardware malfunction or environmental influences such as radiation and thermal effects [4]. This paper argues that introspection can also be used to handle certain classes of faults caused by program design errors, complementing the classical approaches for dealing with design errors summarized under the term Verification and Validation (V&V). Consider a program, P , over a given input domain, and assume that the intended behavior of P is defined by a formal specification. Verification of P implies a static proof—performed before execution of the program—that for all legal inputs, the result of applying P to a legal input conforms to the specification. Thus, verification is a methodology that seeks to avoid faults. Model checking [5, 3] refers to a special verification technology that uses exploration of the full state space based on a simplified program model. Verification techniques have been highly successful when judiciously applied under the right conditions in well-defined, limited contexts.
    [Show full text]
  • Fault-Tolerant Components on AWS
    Fault-Tolerant Components on AWS November 2019 This paper has been archived For the latest technical information, see the AWS Whitepapers & Guides page: Archivedhttps://aws.amazon.com/whitepapers Notices Customers are responsible for making their own independent assessment of the information in this document. This document: (a) is for informational purposes only, (b) represents current AWS product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers. © 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved. Archived Contents Introduction .......................................................................................................................... 1 Failures Shouldn’t Be THAT Interesting ............................................................................. 1 Amazon Elastic Compute Cloud ...................................................................................... 1 Elastic Block Store ........................................................................................................... 3 Auto Scaling ....................................................................................................................
    [Show full text]
  • Network Reliability and Fault Tolerance
    Network Reliability and Fault Tolerance Muriel Medard´ [email protected] Laboratory for Information and Decision Systems Room 35-212 Massachusetts Institute of Technology 77 Massachusetts Avenue, Cambridge, MA 02139 Steven S. Lumetta [email protected] Coordinated Science Laboratory University of Illinois Urbana-Champaign 1308 W. Main Street, Urbana, IL 61801 1 Introduction The majority of communications applications, from cellular telephone conversations to credit card transactions, assume the availability of a reliable network. At this level, data are expected to tra- verse the network and to arrive intact at their destination. The physical systems that compose a network, on the other hand, are subjected to a wide range of problems, ranging from signal distor- tion to component failures. Similarly, the software that supports the high-level semantic interface 1 often contains unknown bugs and other latent reliability problems. Redundancy underlies all ap- proaches to fault tolerance. Definitive definitions for all concepts and terms related to reliability, and, more broadly, dependability, can be found in [AAC+92]. Designing any system to tolerate faults first requires the selection of a fault model, a set of possible failure scenarios along with an understanding of the frequency, duration, and impact of each scenario. A simple fault model merely lists the set of faults to be considered; inclusion in the set is decided based on a combination of expected frequency, impact on the system, and feasibility or cost of providing protection. Most reliable network designs address the failure of any single component, and some designs tolerate multiple failures. In contrast, few attempt to handle the adversarial conditions that might occur in a terrorist attack, and cataclysmic events are almost never addressed at any scale larger than a city.
    [Show full text]
  • Fault Tolerance
    Distributed Systems Fault Tolerance Paul Krzyzanowski Except as otherwise noted, the content of this presentation is licensed under the Creative Commons Attribution 2.5 License. Page Faults • Deviation from expected behavior • Due to a variety of factors: – Hardware failure – Software bugs – Operator errors – Network errors/outages Page A fault in a system is some deviation from the expected behavior of the system -- a malfunction. Faults may be due to a variety of factors, including hardware, software, operator (user), and network errors. Faults • Three categories – transient faults – intermittent faults – permanent faults • Any fault may be – fail-silent (fail-stop) – Byzantine • synchronous system vs. asynchronous system – E.g., IP packet versus serial port transmission Page Faults can be classified into one of three categories: transient faults: these occur once and then disappear. For example, a network message transmission times out but works fine when attempted a second time. Intermittent faults: these are the most annoying of component faults. This fault is characterized by a fault occurring, then vanishing again, then occurring, … An example of this kind of fault is a loose connection. Permanent faults: this fault is persistent: it continues to exist until the faulty component is repaired or replaced. Examples of this fault are disk head crashes, software bugs, and burnt-out hardware. Any of these faults may be either a fail-silent failure (also known as fail-stop) or a Byzantine failure. A fail-silent fault is one where the faulty unit stops functioning and produces no ill output (it produces no output or produces output to indicate failure). A Byzantine fault is one where the faulty unit continues to run but produces incorrect results.
    [Show full text]
  • Detecting and Tolerating Byzantine Faults in Database Systems Benjamin Mead Vandiver
    Computer Science and Artificial Intelligence Laboratory Technical Report MIT-CSAIL-TR-2008-040 June 30, 2008 Detecting and Tolerating Byzantine Faults in Database Systems Benjamin Mead Vandiver massachusetts institute of technology, cambridge, ma 02139 usa — www.csail.mit.edu Detecting and Tolerating Byzantine Faults in Database Systems by Benjamin Mead Vandiver Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY June 2008 c Massachusetts Institute of Technology 2008. All rights reserved. Author.............................................................. Department of Electrical Engineering and Computer Science May 23, 2008 Certified by.......................................................... Barbara Liskov Ford Professor of Engineering Thesis Supervisor Accepted by......................................................... Terry P. Orlando Chairman, Department Committee on Graduate Students 2 Detecting and Tolerating Byzantine Faults in Database Systems by Benjamin Mead Vandiver Submitted to the Department of Electrical Engineering and Computer Science on May 23, 2008, in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science Abstract This thesis describes the design, implementation, and evaluation of a replication scheme to handle Byzantine faults in transaction processing database systems. The scheme compares answers from queries and updates on multiple replicas which are off-the-shelf database systems, to provide a single database that is Byzantine fault tolerant. The scheme works when the replicas are homogeneous, but it also allows heterogeneous replication in which replicas come from different vendors. Heteroge- neous replicas reduce the impact of bugs and security compromises because they are implemented independently and are thus less likely to suffer correlated failures.
    [Show full text]
  • Fault Tolerance Techniques for Scalable Computing∗
    Fault Tolerance Techniques for Scalable Computing∗ Pavan Balaji, Darius Buntinas, and Dries Kimpe Mathematics and Computer Science Division Argonne National Laboratory fbalaji, buntinas, [email protected] Abstract The largest systems in the world today already scale to hundreds of thousands of cores. With plans under way for exascale systems to emerge within the next decade, we will soon have systems comprising more than a million processing elements. As researchers work toward architecting these enormous systems, it is becoming increas- ingly clear that, at such scales, resilience to hardware faults is going to be a prominent issue that needs to be addressed. This chapter discusses techniques being used for fault tolerance on such systems, including checkpoint-restart techniques (system-level and application-level; complete, partial, and hybrid checkpoints), application-based fault- tolerance techniques, and hardware features for resilience. 1 Introduction and Trends in Large-Scale Computing Sys- tems The largest systems in the world already use close to a million cores. With upcoming systems expected to use tens to hundreds of millions of cores, and exascale systems going up to a billion cores, the number of hardware components these systems would comprise would be staggering. Unfortunately, the reliability of each hardware component is not improving at ∗This work was supported by the Office of Advanced Scientific Computing Research, Office of Science, U.S. Department of Energy, under Contract DE-AC02-06CH11357. 1 the same rate as the number of components in the system is growing. Consequently, faults are increasingly becoming common. For the largest supercomputers that will be available over the next decade, faults will become a norm rather than an exception.
    [Show full text]
  • Fault-Tolerant Operating System for Many-Core Processors
    Fault-Tolerant Operating System for Many-core Processors Amit Fuchs Technion - Computer Science Department - M.Sc. Thesis MSC-2018-28 - 2018 Technion - Computer Science Department - M.Sc. Thesis MSC-2018-28 - 2018 Fault-Tolerant Operating System for Many-core Processors Research Thesis In partial fulfillment of the requirements for the degree of Master of Science in Computer Science Amit Fuchs Submitted to the Senate of the Technion — Israel Institute of Technology Tevet 5778 Haifa December 2017 Technion - Computer Science Department - M.Sc. Thesis MSC-2018-28 - 2018 Technion - Computer Science Department - M.Sc. Thesis MSC-2018-28 - 2018 The research thesis was carried out under the supervision of prof. Avi Mendelson, in the Faculty of Computer Science. Technion - Computer Science Department - M.Sc. Thesis MSC-2018-28 - 2018 Technion - Computer Science Department - M.Sc. Thesis MSC-2018-28 - 2018 Contents List of Figures xi List of Listings xiii Abstract1 Glossary3 Abreviations5 1 Introduction7 1.1 Achievements.......................... 10 1.2 Related Work.......................... 12 1.2.1 Distributed Operating Systems ............ 12 1.2.2 Parallelization Techniques............... 13 1.2.2.1 Hardware Parallelism ............ 13 1.2.2.2 Native Multi-threading Libraries . 14 1.2.2.3 Message Passing............... 14 1.2.2.4 Transactional Memory............ 14 2 Architecture Model 17 2.1 Clustered Architecture..................... 17 2.2 Distributed Memory ...................... 19 2.2.1 Local Memories..................... 20 2.2.2 Globally Shared Memory................ 20 2.2.2.1 Scalable Consistency Model......... 21 2.2.2.2 No Hardware-based Coherency . 21 2.2.2.3 Distance Dependent Latency (NUMA) . 22 2.2.2.4 Address Translation ............
    [Show full text]
  • Fault Tolerance in Tandem Computer Systems
    1'TANDEM Fault Tolerance in Tandem Computer Systems Joel Bartlett * Wendy Bartlett Richard Carr Dave Garcia Jim Gray Robert Horst Robert Jardine Dan Lenoski DixMcGuire • Preselll address: Digital Equipmelll CorporQlioll Western Regional Laboralory. Palo Alto. California Technical Report 90.5 May 1990 Part Number: 40666 ~ TANDEM COMPUTERS Fault Tolerance in Tandem Computer Systems Joel Bartlett* Wendy Bartlett Richard Carr Dave Garcia Jim Gray Robert Horst Robert Jardine Dan Lenoski Dix McGuire * Present address: Digital Equipment Corporation Western Regional Laboratory, Palo Alto, California Technical Report 90.5 May 1990 Part Nurnber: 40666 Fault Tolerance in Tandem Computer Systems! Wendy Bartlett, Richard Carr, Dave Garcia, Jim Gray, Robert Horst, Robert Jardine, Dan Lenoski, Dix McGuire Tandem Computers Incorporated Cupertino, California Joel Bartlett Digital Equipment Corporation, Western Regional Laboratory Palo Alto, California Tandem Technical Report 90.5, Tandem Part Number 40666 March 1990 ABSTRACT Tandem produces high-availability, general-purpose computers that provide fault tolerance through fail­ fast hardware modules and fault-tolerant software2. This chapter presents a historical perspective of the Tandem systems' evolution and provides a synopsis of the company's current approach to implementing these systems. The article does not cover products announced since January 1990. At the hardware level, a Tandem system is a loosely-coupled multiprocessor with fail-fast modules connected with dual paths. A system can include a range of processors, interconnected through a hierarchical fault-tolerant local network. A system can also include a variety of peripherals, attached with dual-ported controllers. A novel disk subsystem allows a choice between low cost-per-byte and low cost-per-access.
    [Show full text]
  • Reliability and Fault-Tolerance by Choreographic Design∗
    Reliability and Fault-Tolerance by Choreographic Design∗ Ian Cassary Adrian Francalanza Claudio Antares Mezzina Reykjavik University University of Malta IMT School for Advanced Studies Lucca, Italy [email protected] [email protected] [email protected] Emilio Tuosto University of Leicester, UK [email protected] Distributed programs are hard to get right because they are required to be open, scalable, long-running, and tolerant to faults. In particular, the recent approaches to distributed software based on (micro- )services where different services are developed independently by disparate teams exacerbate the problem. In fact, services are meant to be composed together and run in open context where unpre- dictable behaviours can emerge. This makes it necessary to adopt suitable strategies for monitoring the execution and incorporate recovery and adaptation mechanisms so to make distributed programs more flexible and robust. The typical approach that is currently adopted is to embed such mechanisms in the program logic, which makes it hard to extract, compare and debug. We propose an approach that employs formal abstractions for specifying failure recovery and adaptation strategies. Although implementation agnostic, these abstractions would be amenable to algorithmic synthesis of code, monitoring and tests. We consider message-passing programs (a la Erlang, Go, or MPI) that are gaining momentum both in academia and industry. Our research agenda consists of (1) the definition of formal behavioural models encompassing failures, (2) the specification of the relevant properties of adaptation and recovery strategy, (3) the automatic generation of monitoring, recovery, and adaptation logic in target languages of interest. 1 Introduction Distributed applications are notoriously complex and guaranteeing their correctness, robustness, and resilience is particularly challenging.
    [Show full text]
  • Milesight-Troubleshooting RAID
    Milesight-Troubleshooting RAID 1 NVR Version XX.9.0.2 Update 2018.12.5 1.What is RAID? RAID (Redundant Array of Independent Disks) is a storage technology that combines multiple disk drive components into a logical unit. A RAID setup stores data over multiple hard disk drives to provide enough redundancy so that data can be recovered if one disk fails. 2.Different kinds of RAID RAID 0 It's also known as "disk striping." With RAID 0, data is written across multiple disks. Advantage: The work that the NVR is doing is handled by multiple disks rather than just one. It increases the performance because multiple drives read and write data at the same time, and improves disk I/O. Disadvantage: There is no fault tolerance. It will affect the entire array and increase the possibility of data loss or corruption if one disk is failed. The minimum quantity of disks required is two. RAID 1 A fault-tolerance configuration known as "disk mirroring." With RAID 1, data is copied 2 seamlessly and simultaneously from one disk to another, creating a replica or mirror. The other disks would keep working if one disk is dead. Advantage: It's the simplest way to implement fault tolerance and it's relatively low cost. Disadvantage: The downside is that RAID 1 causes a slight drag on performance. Also, it cuts half of the total disk capacity: For example, the total storage capacity of a server would be 1 TB but not 2TB if it features two 1TB drives and is configured with RAID 1.
    [Show full text]
  • Scalable Design of Fault-Tolerance for Wireless Sensor Networks
    SCALABLE DESIGN OF FAULT-TOLERANCE FOR WIRELESS SENSOR NETWORKS DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By Murat Demirbas, M.S., B.S. * * * * * The Ohio State University 2004 Dissertation Committee: Approved by Anish Arora, Adviser Neelam Soundarajan Adviser Paolo Sivilotti Computer & Information Science c Copyright by Murat Demirbas 2004 ABSTRACT Since wireless sensor networks are inherently fault-prone and since their on-site maintenance is infeasible, scalable self-healing is crucial for enabling the deployment of large-scale sensor network applications. To achieve scalability of self-healing, in this dissertation we focus on addressing (1) the scalability of the cost-overhead of self-healing with respect to the size of the network, and (2) the scalability of the design effort for self-healing with respect to the size of the application software. Our research on fault-containment addresses the first problem: By confining the contamination of faults within a small area, this approach achieves healing within work and time proportional to the size of the perturbation, independent of the size of the network. Our research on specification-based design of self-healing addresses the second problem: Since specifications are more succinct than implementations, this approach yields efficient design of self-healing even for large implementations. These two research directions are complementary, and together enable a scalable design of local self-healing for large-scale sensor network applications. ii To my family. iii ACKNOWLEDGMENTS I am indebted to several people for their help and support throughout my Ph.D.
    [Show full text]