A Single System Image Java Operating System for Sensor Networks

A SINGLE SYSTEM IMAGE JAVA OPERATING SYSTEM FOR SENSOR NETWORKS

Emin Gun Sirer Rimon Barr John C. Bicket Daniel S. Dantas Computer Science Department Cornell University Ithaca, NY 14853 {egs, barr, bicket, ddantas}@cs.cornell.edu

Abstract

In this paper we describe the design and implementation of a distributed operating system for sensor networks. The goal of our system is to extend total system lifetime through power-aware adaptation for sensor networking applications. Our system achieves this goal by providing a single system image of a unified Java virtual machine to applications over an ad hoc collection of heterogeneous sensors. It automatically and transparently partitions applications into components and dynamically finds a placement of these components on nodes within the sensor network to reduce energy consumption and increase system longevity. This paper describes the design and implementation of our system and examines the question of where and when to migrate components in a sensor network. We evaluate two practical, power-aware, general-purpose algorithms for object placement, as well as an adaptive scheme for deciding the time granularity of object migration. We demonstrate that our algorithms can increase sensor network longevity by a factor of four to five by effectively distributing energy consumption and avoiding hotspots. 1. Introduction able to components at each node, in particular the available power and bandwidth may change over Sensor networks simultaneously promise a radi- time and necessitate the relocation of application cally new class of applications and pose signifi- components. Further, event sources that are being cant challenges for application development. Re- sensed in the external environment, such as cent advances in low-power, high-performance tracked objects or chemical concentrations, may processors and medium to high-speed wireless move rapidly, thereby shifting network loads and networking have enabled large scale deployment requiring applications to adapt by migrating com- of sensor nodes, capable of local processing and ponents. Finally, an application's behavior might multi-hop communication. Many interesting sens- change, as in the transition from defensive to of- ing applications, however, entail collaboration fensive mode in a battlefront sensing application, between components distributed throughout an modifying its communication pattern and necessi- ad hoc network. For example, sensor networks are tating a reorganization of its deployed components often composed of three types of components: within the network. sensors acting as data sources, information con- sumers operating as data sinks, and numerous in- Currently, sensor networking applications either termediate filters for performing application- rely on a static assignment of components to specific data processing. While the data sources nodes or use ad hoc, manual policies and mecha- and sinks may be coupled tightly to the nodes to nisms for migrating code and data in response to which they are attached, there is often a high de- change. A static assignment of functionality to gree of freedom in the placement of the data proc- nodes simplifies application design by obviating essing components. This freedom, coupled with code migration and reduces meta-traffic in the the dynamic environment posed by sensor net- network by eliminating component mobility, but it works, makes it difficult to find the optimal distri- also leads to non-adaptive, fragile and energy- bution of application components among nodes. inefficient systems. For example, many current- sensor networks perform all data aggregation and Adapting to dynamically changing conditions by processing at a central node. Such a network will changing the distribution of components across a stall as soon as the critical nodes on the dataflow network is critical for many distributed network- path run out of power or move out of transmission ing applications. For example, the resources avail-

range. In contrast, manual approaches to code and data mobility suffer from being platform- dependent, error-prone and hard to develop. Each application using a manual approach needs to re- implement the same migration, monitoring and Static partitioning communication mechanisms, correctly, on every Monolithic application platform. Further, locally optimal policies pursued Distributed application by applications that share a common network may Figure 1: A static partitioning service converts mono- lead to globally unstable and energy-inefficient lithic Java applications into distributed applications that behavior when they conflict. For instance, a high- can run on an ad hoc network and transparently com- priority submarine tracking application running on municate via RPC. an acoustic sensor grid may be forced to behave • sub-optimally by a lower-priority whale tracking Adaptive: The system should respond auto- application, if the applications use ad hoc mecha- matically to significant changes in network to- nisms to adapt to the current power levels in the pology, resource availability, and the communi- network. In essence, application-level adaptation cation pattern of the applications. suffers from building on an abstraction level that • General purpose: The system should support a is too low. An operating system that provides the wide range of applications and porting an exist- requisite mechanisms and policies for code mobil- ing centralized sensing application to execute ity would not only simplify application develop- efficiently on a sensor network should require ment, but also ensure the integrity of system-wide little effort. Applications should be able to di- goals in the face of multiple applications compet- rect the adaptation using application-specific in- ing for resources. formation. The system should provide effective Unlike distributed programming on the Internet, default adaptation policies for applications that where energy is not a constraint, delay is low, and are not power-aware. bandwidth is plentiful, physical limitations of sen- • Platform independent: Applications should be sor networks lead to some unique requirements. able to execute on ad hoc networks of heteroge- Technology trends indicate that the primary limi- neous nodes. tation of sensor networks is energy consumption, and communication is the primary energy con- Our operating system meets these goals by provid- sumer. Measurements from first-generation sensor ing the abstraction of a single unified Java virtual nodes [Pottie & Kaiser 00] show that sending one machine over an ad hoc network of heterogeneous, bit may consume as much energy as executing physically separate, potentially mobile sensor 3000 instructions. This motivates a system that nodes. MagnetOS consists of a static application performs more computation to reduce communica- partitioning service that resides on border hosts tion. capable of injecting new code into the network, and a runtime on each node that performs dynamic In this paper, we outline the design of MagnetOS, monitoring and component migration. The static a single system image (SSI) operating system for partitioning service takes regular Java applications sensor networks. Based on our target domain of and converts them into distributed components energy-constrained sensor networks, we identify that communicate via RPC [Birrell & Nelson 84] the following set of design goals for an operating by rewriting them at the bytecode level (Figure system. 1). The code injector then finds a suitable initial • Efficient: The system should execute distrib- layout of these components and starts the execu- uted sensor network applications in a manner tion of the application. The runtime monitors the that conserves power and extends system life- performance of the application and migrates ap- time. Policies and mechanisms used for adapta- plication components when doing so would bene- tion in the systems layer should not require ex- fit the system. cessive communication or power consumption.

The algorithms used to decide where and when to ponents are partitioned among the nodes auto- move application components form the core of our matically and migrated transparently at runtime. system. While MagnetOS is designed such that Second, we propose practical, adaptive, online these algorithms can be transparently replaced to algorithms for deciding where and when to move optimize for differing goals, such as minimizing application components. Finally, we demonstrate application latency, response time, or bandwidth that these algorithms achieve high-energy utiliza- consumption, in this paper we tackle what we be- tion, extract low overhead, and improve system lieve to be the most important goal in energy- longevity. constrained ad hoc networks of mobile hosts: we In the next section, we describe related work on examine how to maximize total application and operating system support for ad hoc sensor net- system lifetime by utilizing power more effi- works and their applications. Section 3 outlines ciently. We present two practical, online algo- our system implementation, including the code rithms, named NetPull and NetCenter, for finding partitioning and distribution technique. Section 4 a distribution of application components on nodes presents our network and application model, de- in an ad hoc network that increases total system scribes our simulation framework and evaluates lifetime by increasing the effective energy utiliza- NetPull and NetCenter within this environment. tion (Figure 2). We evaluate these algorithms in We summarize our contributions and results in the context of a generic sensing application and Section 5. examine their impact on system longevity, which we define as the length of time that a sensing ap- 2. Related Work plication can maintain sensor coverage above a There has been much research on distributed op- given threshold area. Both algorithms operate by erating systems, ad hoc sensor networks, and dividing time into epochs, monitoring the commu- power management, though few systems have ex- nication pattern of the application components amined all three. within each epoch, and migrating components at the end of the epoch when doing so would result 2.1. Distributed Systems in more efficient power utilization. We also pre- Single system image (SSI) operating systems have sent an online algorithm for selecting an epoch been examined extensively in the context of wired length that matches application behavior through networks of workstations. Early landmark sys- online, adaptive statistical testing. We show that tems, such as V [Cheriton 88], Sprite [Ousterhout the MagnetOS system can achieve a factor of four et al. 88], Ameoba [Tanenbaum et al. 90, Steketee to five improvement in system longevity over na- et al. 95], Accent [Rashid & Robertson 81], and ive or static partitioning techniques. LOCUS [Popek & Walker 85], implemented na- This paper makes three contributions. It outlines tive operating system facilities for migrating proc- the design and implementation of a single system esses between nodes on a tightly coupled cluster. image operating system for sensor networks, More recently, the cJVM [Aridor et al. 99] and where the entire network appears to be a single JESSICA [Ma et al. 99] projects have examined Java virtual machine to applications, whose com- how to extend a Java virtual machine-across a high-performance cluster. Others, including Con- dor [Litzkow et al. 97], libckpt [Plank et al. 95] and CoCheck [Stellner 96], provide user-level mechanisms for checkpointing and process migration without operating system support. These projects target high-performance, well-connected clusters. Their main goals are to balance load and achieve high performance in a local area network for interactive desktop programs or CPU-intensive batch jobs. In contrast, MagnetOS targets wireless node application filter multi-hop networks, where utilizing power effec- data source network packet tively and maximizing system longevity is more

Figure 2: Migrating components closer to their data important than traditional application perform- sources in a sensor network increases system longevity ance. and decreases power consumption by reducing total network communication cost.

Distributed object systems have examined how to geted specifically for ad hoc networks and provide support distributed computations in the wide area. an adaptive-QoS programming interface. XMID- Emerald [Jul et al. 88] provides transparent code DLE [Mascolo 01] assists with data management migration for programs written in the Emerald and synchronization. MagnetOS takes a systems language, where the migration is directed by approach instead of providing a programmer source-level programmer annotations. Thor driven toolkit and automatically manages the [Liskov et al. 93] provides persistent objects in a shared network and energy resources among language-independent framework. It enables cach- ad hoc sensor applications. This approach unifies ing, replication and migration of objects stored at the system layer and ensures that disparate appli- object repositories. More recently, grid computing cations, regardless of which toolkits they use, be- systems such as Legion [Lewis & Grimshaw 95] have in a cooperative manner. and Globus [Foster & Kesselman 97] have fo- 2.2. Ad hoc Routing Protocols cused on the construction of scalable, geographi- cally distributed computing systems. These sys- There has been much prior research on ad hoc tems differ fundamentally from MagnetOS in that routing algorithms. Proactive (e.g. DSDV [Perkins they require explicit programmer control to trigger & Bhagwat 94], WRP [Murthy & Garcia- migration, do not support an ad hoc network Luna_Aceves 96]), reactive (e.g. DSR [Broch et model and target traditional applications. al. 98], AODV [Perkins 97], TORA [Park & Corson 98]) and hybrid (e.g. ZRP [Haas & Some recent systems have focused on how to par- MagnetOSman 98], HARP [Nikaein et al. 01]) tition applications within a conventional wired routing protocols seek to pick efficient routes by network. The Coign system [Hunt & Scott 99] has proactively disseminating or reactively discover- examined how to partition COM applications being route information, or both. While some proto- tween two tightly interconnected hosts within a cols, such as PARO [Gomez et al. 01] and MBLR local-area network. Coign performs static spatial [Toh 01], have examined how to make power- partitioning of desktop applications via a two-way aware routing decisions, all of these routing minimum cut based on summary application pro- algorithms assume that the communication files collected on previous runs. Extending this endpoints are fixed. Directed diffusion work, the ABACUS system [Amiri et al. 00] has [Heidemann et al. 01] provides a data-centric examined how to migrate functionality in a stor- programming model for sensor networks by age cluster. MagnetOS shares the same insight as labeling sensor data using attribute-value pairs and Coign, in that it also focuses on the automatic re- routing based on a gradient. MagnetOS com- location of application components, but differs in plements the routing layer to move application that it dynamically moves application components code around the network, changing the location of in response to changes in the network, instead of the communication endpoints and radically computing a static partitioning from a profile. altering the communication pattern of the overall [Kremer et al. 00] proposes using static analysis to application. It provides increased system and select tasks that can be executed remotely to save application longevity by bringing application com- energy. J-Orchestra [Tilevich & Smaragdakis 02] ponents closer to the data sources, which performs application partitioning via rewriting, complements the route selection performed by the leaving dynamic migration decisions under appli- ad hoc routing protocol. cation control. Spectra [Flinn et al. 01] monitors 2.3. Power Management resource consumption, collects resource usage Prior work has also examined how to minimize histories and uses quality of service (fidelity) in- power consumption within an independent host formation supplied by the application to make re- through various mechanisms [Pillai & Shin 01, source allocation decisions. Spectra is invoked Grunwald et al. 00, Weiser et al. 94, Douglis et al. prior to operation startup, and statically deter- 95, Stemm & Katz 96], including low-power proc- mines a location at which to execute the operation. essor modes, disk spin-down policies, adapting wireless transmission strength and selectively Middleware projects have looked at constructing turning off unused devices. Our system is com- toolkits to support mobile applications. The Rover plementary to this work and opens up further op- toolkit [Joseph et al. 95] provides relocation and portunities for minimizing power consumption by messaging services to facilitate the construction of shipping computation out of hosts limited in mobile applications. The Mobiware [Campbell 98] power to less critical nodes. and DOMT [Kunz and Omar 00] toolkits are tar-

3. System Implementation and Distribu- face (MagnetInterface) and a class object (Mag- tion Model netStatic). MagnetOS implements a single system image op- A Magnet is a modified implementation of the erating system for sensor networks in two steps. original class that stores the instance variables of First, a monolithic application is partitioned, dis- the object. Each Magnet is an object instance and tributing its functionality across the ad hoc net- is free to move across nodes in the network. Mag- work. The MagnetOS runtime then coordinates the netPoles, on the other hand, are remote references communication and migration of these application to the corresponding Magnet instance. That is, segments across the nodes in the sensor network MagnetPoles are used to invoke procedure calls in order for the newly distributed application to on remote Magnets residing on other nodes. Calls appear as if running on a single Java virtual ma- to the MagnetPole are intercepted by the Magne- chine. We will now discuss the implementation tOS runtime and converted into RPCs. This level details of the two components, the partitioning of indirection enables code migration. As a Mag- mechanism and the MagnetOS runtime. net moves, the method calls to the corresponding 3.1. Application Partitioning MagnetPoles are tracked by the MagnetOS runtime and directed to the new Magnet location. The partitioning mechanism of MagnetOS con- MagnetInterfaces capture the interface that the verts Java applications written and compiled for a original class exposes to the rest of the applica- single virtual machine into remote objects that can tion. Magnet and MagnetPole instances implement be dispersed and executed across an ad hoc net- the MagnetInterface of the original class. work of many virtual machines. The transformed application code, though modified to interact with Several modifications to the application binaries the MagnetOS runtime, retains its original are required for this remote object mechanism to application semantics. work seamlessly. First, object creations (new instructions and matching constructor invocations) MagnetOS partitions applications at class granu- are replaced by calls to the local MagnetOS run- larity; consequently, the unit of mobility in Mag- time. The runtime selects an appropriate node and netOS is an object instance. This transformation at constructs a new Magnet instance at that location. class boundaries preserves existing object inter- This operation returns a corresponding, properly faces. The entire transformation is performed at initialized MagnetPole, which is then used in sub- the byte-code level via binary rewriting, without sequent method invocations. In addition, Magne- requiring source-code access. tOS adds accessor methods of the appropriate type Our approach to partitioning applications stati- for each field, converting field accesses into cally is patterned after distributed virtual ma- method calls to the MagnetPole reference, which chines [Sirer et al. 99]. Static partitioning confers subsequently remotely call the accessor on the several advantages. First, the complex partitioning corresponding Magnet instance. Finally, services need only be supported at code-injection typechecking and synchronizing instructions points, and can be performed offline. Second, (checkcast, instanceof, monitorenter since the run-time operation of the system and its and monitorexit instructions, and synchro- integrity do not depend on the partitioning tech- nized methods) are rewritten to trap into the nique, users can partition their applications into MagnetOS runtime. The runtime reconstructs the arbitrary components if they so choose. Further, appropriate type check on top of the modified type since applications are verified prior to injection hierarchy, retaining the original application's be- into the network, individual MagnetOS nodes havior. Similarly, it converts lock acquisitions and need not re-run a costly verifier on application releases into centralized operations at the Magnet. components. Finally, binary rewriting provides a Unlike regular JVM monitor operations, each convenient, default mechanism for transitioning thread in MagnetOS is identified via a globally legacy, monolithic applications to execute over unique identifier, consisting of a ad hoc networks. tuple. This enables MagnetOS to identify cases of recursively locked mutexes The static partitioning takes original application and support the lock semantics required by the classes, and for each class, creates an instance Java virtual machine. (Magnet), a remote stub (MagnetPole), an inter-

The final component created for a class is a Mag- proach for its simplicity, and rely on our dynamic netStatic object. It contains static field members, object migration algorithms to find the optimal also known as class instances. In Java, static fields placement of objects over time. Furthermore, are shared across all instances of an object. The short-lived, tightly scoped objects do not travel partitioning service coalesces the static fields of across the network unnecessarily. The application the original class into a single object, to which all binaries, containing all of the object constructors, the Magnet instances of that class retain a refer- are distributed to all nodes at the time that the ap- ence. Static field and method instructions are replication is introduced into the network. Once cre- placed with calls to the MagnetOS runtime, which ated, the (remote) runtime simply initializes the tracks the location of the MagnetStatic object and object by calling its constructor and returns a forwards the static operations to the appropriate MagnetPole instance referring to the (remote) ob- node in the ad hoc network. ject. While MagnetOS attempts to provide reasonable The runtime transparently handles invocations defaults for converting regular JVM applications among the application components distributed to work in an ad hoc network, it explicitly does across the network. Each runtime keeps a list of not try to make the process completely transparent the live, local Magnets, which can optionally be and support network-oblivious applications. Ap- named. MagnetPoles maintain the current location plications running on networks have diverse fail- of their corresponding Magnet, and make runtime ure modes that cannot be masked, and MagnetOS calls on behalf of application invocations to mar- reflects such failures to the programmer by map- shal arguments to, and results from, the appropri- ping them to implicit runtime exceptions, similar ate node and object. to the way other exceptional events, such as run- The MagnetOS runtime implements a lease-based ning out of memory, are handled. This mapping garbage collector for remote objects, with leases conforms to the JVM specification and provides a automatically renewed by MagnetPoles. As in way for the application programmer to react to RMI and Network Objects [Birrell et al. 94], we runtime events. We have found that most non- do not collect cycles in the object reference graph. trivial, stateful applications will require some Local objects are handled by the standard Java amount of failure recovery in all but the most garbage collector. densely connected and static networks. 3.2. Runtime Mechanisms for Object Mi- MagnetOS migrates application components at gration runtime by serializing Magnet state and moving it to a new node. MagnetPoles are informed of the The MagnetOS runtime provides the dynamic ser- relocation lazily, the next time they invoke a Mag- vices that facilitate the distributed execution of net method or renew their object lease, via a componentized applications across an ad hoc net- forwarding reference left behind when an object work. Its services include component creation, migrates. Long chains of forwarding pointers, if inter-component communication, object migra- allowed to persist for a long time, would pose a tion, garbage collection, naming, and object dis- vulnerability – as nodes die, out-of-date Magnet- covery. These runtime services are invoked in Poles may not be able to trace a path to the current three ways: through background processes that location of the object to which they are connected. automatically manage the running system, indi- MagnetOS collapses these paths whenever they rectly via MagnetPoles, and directly via explicit are traversed. Periodic lease updates in lease- API calls from the application. based garbage collection requires periodic com- In order to create a new instance of an object, an munication between MagnetPoles and Magnets, application will contact the local runtime and pass which provides an upper-bound on the amount of the requisite type descriptor and parameters for time such linear chains are permitted to form in object creation. The runtime then has the option of the network. placing the newly created object at a suitable loca- The MagnetOS runtime provides an explicit inter- tion with little cost. It may choose to locate the face by which application writers can manually object on the local node, at a well-known node or direct component placement. This interface allows at its best guess of an optimal location within the programmers to establish affinities between com- network. In our current implementation, all new ponents and ad-hoc nodes. We provide two levels objects are created locally. We chose this ap- of affinity. Specifying a “strong” affinity between

a component and a node effectively anchors the code to that node. This is intended for attaching 1 - NetPull components like device drivers to the nodes with 2 - NetCenter the installed device in them. Specifying a “weak” affinity immediately migrates the component to node the named node, and allows the automated code placement techniques described in the next section data source 2 1 to adapt to the application’s communication pattern from the new starting point. Note that today’s application component manually constructed applications correspond to network packet the use of strong affinity in our system – unless explicitly moved, components are bound to nodes.

The result of overusing strong affinity is a fragile Figure 3: NetPull moves one hop towards the source of system, where unforeseen communication and data whereas NetCenter moves directly to the source of mobility patterns can leave an application most packets. stranded. While we provide these primitives, we do not advocate their use and believe that auto- munication between MagnetPoles and correspond- mated techniques can outperform manual efforts ing Magnets. to place components. Finally, the higher-level policies in MagnetOS 3.3. Runtime Support for Ad hoc Networks require information on component behavior to The sensor networking domain places additional make intelligent migration decisions. The runtime constraints on the runtime implementation. assists in this task by collecting, for each component, information on the amount of data it ex- First, multi-hop ad hoc networks require an ad hoc changes with other components. The runtime in- routing protocol to connect non-neighboring tercepts all RPCs and records, for all incoming nodes. MagnetOS relies on a standard ad hoc rout- and outgoing invocations per component, the ing protocol below the runtime to provide message source and destination. It keeps a cumulative sum routing. Currently, our system runs on any plat- per component per epoch, and periodically in- form that supports Java JDK1.4. On Linux, we use forms the migration policy in the system of the an efficient in-kernel AODV implementation we current tally. While this approach has worst case developed. On other platforms, we use a user-level space requirement that is O(N2), where N is the version of AODV written in Java to provide uni- number of components in the network, most com- cast routing. The choice of a routing algorithm is ponents communicate with few others, and the independent from the rest of the runtime, as the space requirements are typically small. For in- runtime makes no assumptions of the routing layer stance, in the sensor benchmark examined in Sec- besides unicast routing. tion 4, the storage requirements are linear. The In addition, standard communication packages next section describes how MagnetOS uses these such as Sun’s RMI are designed for infrastructure statistics to automatically migrate components. networks, and are inadequate when operating on 3.4. Finding an Energy-Minimizing Object multi-hop ad hoc networks. Frequent changes in Placement network topology and variance in available band- In this section, we describe two algorithms, Net- width require MagnetOS to migrate objects. How- Pull and NetCenter, which use the information ever, standard RMI does not provide an easy-to- gathered by the runtime to migrate components in use mechanism by which the endpoints of an ac- a manner that increases system longevity. tive connection can be modified. Consequently, we have had to develop our own RPC package. Both NetPull and NetCenter share the same basic Similar to Sun’s RMI, it supports a synchronous insight. They shorten the mean path length of data interface and uses a reliable datagram protocol packets by automatically moving communicating resembling RDP [Hinden & Partridge 90] instead objects closer together. They perform this by pro- of TCP. The MagnetOS RPC package allows us to filing the communication pattern of each applica- easily modify the communication endpoints when tion in discrete time units, called epochs. In each components move and is responsible for all com- epoch, every runtime keeps track of the number of incoming and outgoing packets for every object.

At the end of each epoch, the migration algorithm tions and normal system perturbation. Adapting decides whether to move that object, based on its too slowly can also result in wasted energy when recent pattern of behavior. Under both algorithms, significant changes in application behavior or the the decision is made locally, based on information environment are not addressed. collected during recent epochs at that node. Net- The insight guiding our approach to epoch selec- Pull and NetCenter differ in the type of information is that a well-placed component should get tion they collect and how they pick the destination equal numbers of packets from all directions host. Depending on the environment, one may be within an epoch. Consequently, a distinct direc- easier to implement. tion that dominates the communication pattern NetPull collects information about the communi- marks the boundary of an epoch. We implement cation pattern of the application at the physical an adaptive epoch selection algorithm based on a link level, and migrates components over physical statistical test that determines whether the data links one hop at a time. This requires very little collected thus far is sufficient to make a migration support from the network; namely, the runtime decision with high confidence. We accumulate the needs to be able to examine the link level packet number of packets sent to each component by its headers to determine the last or next hop for in- communicating peers, and then compute the like- coming and outgoing packets, respectively. For lihood of this event occurring from a multinomial every object, we keep a count of the messages sent distribution with equally weighted priors as fol- to and from each neighboring node. At the end of lows: an epoch, the runtime examines all of these links ()xi and the object is moved one hop along the link ∏ pi P()x ,..., x = ()x !⋅ with greatest communication. 1 n ∑ i () ∏ xi ! NetCenter operates at the network level, and mi- where: n – number of peers grates components multiple hops at a time. In each xi – number of packets from node i epoch, NetCenter examines the network source pi – 1/n. addresses of all incoming messages, and the destination addresses of outgoing messages for each If the probability calculated above is below a object. This information is part of the transmitted threshold value, we continue to accumulate data, packet, and requires no additional burden on the and defer the migration decision. We migrate network. At the end of an epoch, NetCenter finds components only when we can determine with the host with which a given object communicates high probability that one direction in the network the most and migrates the object directly to that dominates the communication pattern. host. 4. Evaluation Both of these algorithms improve system longev- In this section, we evaluate the performance of ity by using the available power within the net- MagnetOS. We first evaluate the core object mi- work more effectively. By migrating communicat- gration algorithms, NetPull and NetCenter, and ing components closer to each other, they reduce show that they achieve good energy utilization, the total distance packets travel, and thereby re- improve system longevity, and are thus suitable duce the overall power consumption. Further, for use in a general-purpose, automatic object mi- moving application components from node to gration system. We then discuss the benefits of node helps avoid hot spots and balance out the epoch length adaptation. Finally, we report results communication load in the network. As a result, from some microbenchmarks to show that auto- both algorithms can significantly improve the total matically partitioning applications does not extract system longevity for an energy-constrained ad hoc a large performance cost, and that the memory network. costs of a specially-tuned Java virtual machine is 3.5. Determining Adaptation Granularity within the resource-budget of next generation sensors. Both NetPull and NetCenter are epoch-based algorithms, and are therefore sensitive to the time 4.1. Simulation Framework granularity at which they adapt. Adapting too Our system targets general-purpose sensor net- quickly can result in wasted energy due to poor works. We developed a fast, packet-level, stati- migration decisions in response to transient condi- cally parameterizable ad hoc networking simulator

in order to simulate large networks. The simulator • NetPull moves components one hop along the accounts for all communication costs, including most active adjacent communication link at each AODV routing and route repair overhead. It mod- epoch to the most active neighbor. els the movement of every unicast and multicast • NetCenter moves components directly to the packet and incurs the cost of moving a condenser node with greatest activity in the previous ep- and notifying all its sensors of the new location. och. We initialize the simulator with a uniform distri- 4.4. Simulation Parameters bution of nodes on a plane, and vary parameters such as noise levels, field size, density, battery A simulation of a complex system such as this power, and communication and sensing radii. requires many parameters. In the following ex- Sensing events are generated at random locations periments, we model a terrestrial sensor network on the field, and with random durations and veloc- consisting of nodes with seismic sensors for object ity vectors. Sensors that are in range detect these tracking. We examine large sensor networks, con- signals and generate application events. They can sisting of 3600 nodes. The field size is 300 by 300 distance units, which, if scaled to a meter, corre- also, with small probability, generate fictitious 2 events due to sensor noise. spond to a density of 0.04 sensors/m , a practical density for distribution from the air. Node sensing 4.2. Network Model and Benchmark Ap- radius is 20 units; communication radius is 10. plication Sensors generate spurious readings, or noise, due In our simulations, all nodes have the same com- to local vibrations and sensor noise; we assume munication radius and they are connected to the that sensors are fairly accurate, and that only 1% fixed networking infrastructure via a single, cen- of the messages are attributable to noise. We have trally placed node. Each node initially stores a examined, but do not present results from, simula- fixed, finite amount of energy. Sending a packet tions with higher sensor density (0.02 through between any neighboring nodes exacts a constant 0.06) and noise level (1% through 10%). The communication cost, and the cost of local compu- choice of a particular density, or noise level does tation on a host is negligible in comparison. not qualitatively impact our results; they shift the curves without affecting any of the trends. The We examine a generic, reconfigurable sensing choice of epoch duration is initially fixed – we benchmark we developed named SenseNet. This address the epoch length selection question sepa- application consists of sensors, condensers and rately in Section 4.6. Each epoch contains at least displays. Sensors are fixed at particular ad hoc one event in our simulations, where an event cor- nodes, where they monitor events within their responds to an object being tracked, and moves sensing radius and send a packet to a condenser in through the sensor field at a randomly chosen ve- response to an event. Condensers can reside on locity. An event may span up to 10 epochs and any node, where they process and aggregate sen- move through the field at a velocity between 0.0 sor events and filter noise. The display runs on the and 2.0 distance units per epoch. We assume that central node, extracts high-level data out of the nodes are stationary after the initial deployment, sensor network and sends it to the wired network. though none of the nodes make any assumptions 4.3. Algorithms about geographical location of other nodes. Every We compare four different algorithms for auto- data point represents an average of five runs. matic object migration: • Static corresponds to a static, fixed assignment of objects to nodes within the network. Our components remain at the home node for the entire duration of the simulation. • Random selects a random destination for each component at each epoch. It corresponds to a simple load-balancing algorithm, designed to avoid network hotspots.

1800 6 100 1600 5 90 1400 80

1200 4 70

1000 60 3

epochs 800 speedup 50 static 600 2 40 random netpull 400 30 netcenter

1 percent of field area sensed 200 20

0 0 10 static random netpull netcenter 0 0 200 400 600 800 1000 1200 1400 1600 1800 epoch Figure 4: System longevity improvement. Figure 5: Sensor coverage degradation over time.

Figure 5 shows how quickly the field coverage degrades when components are assigned to nodes

in a manner that is oblivious to the underlying ap- 4.5. Results and Discussion plication communication pattern. In contrast, Net- Figure 4 illustrates the impact of our algorithms Pull and NetCenter migrate components close to on system longevity. In this simulation, we define the source of events, thus preserving energy and system failure as the point when half of the field is shedding load from the critical nodes in the net- no longer being sensed, that is, only half of the work. field area is within the sensing radius of at least The slopes of the curves in Figure 6 demonstrate one live sensor which can communicate along that actively migrating components in the network some functioning route with the home node. Static drains fewer nodes than a static placement. This is corresponds to current, naïve implementations of because active migration algorithms avoid creat- sensing applications, where all data is pooled off ing hotspots around critical nodes. In addition, of the sensor network for processing on a central both Static and Random reach system failure with node. The network becomes unoperational as soon far fewer drained nodes, indicating that these al- as the gateway nodes around the central node run gorithms distribute load unevenly and lead to hot- out of power. Random performs better by 50% spots. because it distributes the load more evenly and avoids hot spots. NetPull and NetCenter lengthen Figure 7 shows that NetPull and NetCenter lie the operational lifetime of the system by a factor between Static and Random in terms of energy of four to five by performing in-network process- consumption per epoch. The low slope for Static ing at suitably-selected locations. shows that this algorithm uses less energy per epoch because it does not expend any power on ac-

600 100 static random 90 500 netpull netcenter 80

70 400 60

300 50 static 40 random 200 netpull 30 netcenter number of sensors drained percent of field area sensed 20 100 10

0 0 0 200 400 600 800 1000 1200 1400 1600 1800 0 200 400 600 800 1000 1200 1400 1600 1800 epoch epoch Figure 6: Sensor drainage over time. Figure 7: Field energy used over time.

3000 80 static random 70 netpull 2500 netcenter 60

2000 50

1500 40

30 1000

number of sensors disconnected static 500 random 10

netpull percent of total sensor power used at failure netcenter 0 0 0 200 400 600 800 1000 1200 1400 1600 1800 50 100 150 200 250 epoch field size Figure 8: Sensors disconnected over time. Figure 9: Energy used at breakdown vs. field size. tive object migration. Most of this power usage is Finally, Figure 9 shows that all algorithms, except concentrated in a ring around critical nodes, how- for the static placement of components, are unaf- ever, and the application terminates early, leaving fected by variations of field size, and will scale to more than 95% of the total energy on the field un- large networks. Static does not scale, because the utilized. NetPull and NetCenter consume more number of critical nodes is a constant function of energy than Static, because they need to move the application data flow graph, and is not propor- components, but consume less energy than Ran- tional to the size of the field. dom, because they take application behavior into Overall, NetPull and NetCenter achieve good en- account. Overall, they outlast both random and ergy utilization and improved system longevity. static despite lying between them. Their simplicity makes them strong candidates for The graph of disconnected nodes over time, use in automated object migration systems. shown in Figure 8, indicates that the number of 4.6. Epoch length selection disconnected nodes increases more gradually for NetPull and NetCenter because they distribute In this section, we evaluate epoch length selection load more evenly across the network. In the case algorithm described in section 3.5. To test the sen- of Static and Random, even though only a small sitivity of object migration to epoch length, we set number of nodes are drained, they are all located up an experiment where the field contains four around the home node and thus quickly disconnect event sources located at the midpoint of each the entire field from the wired network. edge. Events are generated for half a 10 second period in a pair-wise alternating manner. Figure 10 evaluates the performance of automatic object

System longevity versus epoch length migration algorithms at different, statically fixed 1400 static epoch lengths. It demonstrates the dangers of an random netpull epoch length that is not matched to the temporal 1200 netcenter communication patterns in the network. A highly adaptive algorithm like NetCenter performs best 1000 when its epoch length matches that of the underlying event source. At shorter epoch lengths it 800 wastes energy by performing excessive migra-

600 tions. At harmonics of the beaconing frequency, epochs to failure some of the migration algorithms perform slightly

400 better, though at long epoch lengths all the migration algorithms fail to adapt to network changes. 200 In contrast with statically selected epoch lengths, yhe adaptive epoch length selection algorithm de- 0 0 5 10 15 20 25 30 35 40 45 50 epoch length scribed in section 3.5 achieves an average system Figure 10: Sensitivity of migration algorithms to epoch longevity of 2370 when combined with NetCenter. length.

It outperforms any static epoch length selection However, through simple space optimizations, shown in Figure 10 by 50%. including lazy loading, discarding basic block in- 4.7. MagnetOS RPC performance formation, stack reduction, and eliminating the space allocated for reserved but unused fields, we An automatic approach to application partitioning have reduced these resource requirements 20-fold and transparent object migration would be unten- without compromising any VM functionality. able if the performance of automatically partitioned applications suffered significantly. In this Even further space savings can be achieved by section, we show that the performance of auto- trading off functionality for space. For instance, matically partitioned and rewritten applications is Sun’s J2ME KVM and Java Card VM have been competitive with manually partitioned applica- specifically targeted for embedded platforms. tions. In the micro-benchmark below, we compare They consume less memory by supporting fewer the overhead of our RPC implementation to that of Java libraries, not providing features like code remote invocations performed via Java RMI, on a verification and reflection, and modifying the pro- 1.7 GHz P4 with 256 MB of RAM JDK 1.4 on gramming model to avoid rich data types. For in- Linux 2.4.17 with AODV. On all micro- stance, the Java Card VM can execute Java appli- benchmarks, automatically decomposed applica- cations using only 512 bytes. Overall, we consider tions are competitive with manually coded, the problem of running a Java interpreter, or equivalent RMI implementations. equivalent functionality, on sensors to be solvable. Our VM is capable of executing the MagnetOS Remote call Java RMI MagnetOS runtime and Java applications with full access to Null 430 ± 16 172 ± 6 the complete Java class libraries with 2 MB of Int 446 ± 9 180 ± 8 RAM, which we expect to be available on sensor Obj. w/ 32ints 991 ± 35 174 ± 4 nodes of the near future. Obj. w/ 4int, 2obj 844 ± 21 177 ± 7 5. Conclusion all times in µs, average of 1000 calls. In this paper, we present the design and imple- Table 1: Remote method invocation comparison. mentation of a single system image operating system for sensor networks. Our system implements 4.8. Space-optimized Java Virtual Machine the Java Virtual Machine interface on top of a col- The applicability of a Java-based SSI OS is lim- lection of sensor nodes. An application partition- ited by the ability of sensor network nodes to sup- ing tool takes monolithic Java applications and port the requisite services of a Java VM. Java vir- converts them into distributed, componentized tual machines on the desktop can indeed have ex- applications. A small runtime on each node is re- cessive resource requirements; the Java virtual sponsible for object creation, invocation and mi- machine we started out with, the Kimera VM, also gration. We rely on a transparent RPC for node- had significant memory requirements. Table 2 independent communication between components. shows the size of different Java virtual machines Overall, this distributed system provides a well- at application startup. understood programming model for sensor network applications, while simultaneously providing Virtual Machines Peak Memory the system with sufficient freedom to transpar- Consumption ently move components in order to extend achieve Sun Java JDK 1.4 9000 KB power savings and extend sensor network lifetime. Kimera Unmodified 22822 KB We propose algorithms for automatically deter- Kimera Optimized 1172 KB mining where to locate application components in Sun J2ME KVM 160 KB the network to minimize energy consumption and Sun Java Card VM 512B + 16 KB ROM to determine when to migrate application compo- Table 2: Space consumption of Java virtual machines. nents by adaptively selecting an epoch duration. Combined, these algorithms enable MagnetOS to Traditional virtual machines, such as Sun Java find an assignment of components to nodes that JDK 1.4 and the original Kimera VM support yields good utilization of available energy in the many features and are not optimized for space. network. These algorithms are practical, entail low Consequently, they have high memory require- overhead and are easy to implement because they ments and are not suitable for sensor networks. rely only on local information that is readily avail-

able. We have demonstrated that they can con- [Gomez et al. 01] J. Gomez, A. T. Campbell, M. serve power and achieve a factor of four to five Naghshineh and C. Bisdikian. PARO: Conserving improvement in system longevity. Transmission Power in Wireless Ad hoc Networks. In Proceedings of the 9th International Conference Ad hoc sensor networking is a rapidly emerging on Network Protocols, Riverside, California, No- area with few established mechanisms, policies vember 2001. and benchmarks. We hope that high-level abstrac- [Grunwald et al. 00] Dirk Grunwald, Philip Levis, Keith tions, such as single system image operating sys- I. Farkas, Charles B. Morrey III and Michael Neu- tems combined with automatic object migration feld. Policies for Dynamic Clock Scheduling. In algorithms, will create a familiar and power- Proceedings of the Fourth OSDI, San Diego, Cali- efficient programming environment, thereby ena- fornia, October 2000. bling rapid development of platform-independent, [Haas & MagnetOSman 98] Z. J. Haas and M. R. Mag- power-adaptive applications for sensor networks. netOSman, The Zone Routing Protocol (ZRP) for References Ad hoc networks (Internet-Draft). Mobile Ad hoc Network (MANET) Working Group, IETF, Aug. [Amiri et al. 00] Khalil Amiri, David Petrou, Greg 1998. Ganager and Garth Gibson. Dynamic Function [Harold 00] Harold, E. R. Java Network Programming. Placement in Active Storage Clusters. USENIX O'Reilly & Associates, Aug 2000. Annual Technical Conference, San Diego, CA, [Hinden & Partridge 90] B. Hinden and C. Partridge, June 2000. "Version 2 of the reliable data protocol (RDP)," [Aridor et al. 99] Yariv Aridor, Michael Factor and Avi RFC 1151, IETF, Apr 1990. Teperman. cJVM: a Single System Image of a JVM [Hunt & Scott 99] Galen C. Hunt and Michael L. Scott. on a Cluster. IEEE International Conference on The Coign Automatic Distributed Partitioning Sys- Parallel Processing, September 1999. tem. In Proceedings of the Third Symposium on [Birrell & Nelson 84] A. D. Birrell and B. J. Nelson. Operating System Design and Implementation, pp. Implementing remote procedure calls. ACM Trans- 187-200. New Orleans, Louisiana, February 1999. actions on Computer Systems, 2(1):39--59, Febru- [Joseph et al. 95] Anthony D. Joseph, Alan F. De ary 1984. Lespinasse, Joshua A. Tauber, David K. Gifford, [Birrell et al. 94] Andrew Birrell, Greg Nelson, Susan and M. Frans Kaashoek., Rover: A Toolkit for Mo- Owicki, and Edward Wobber. Network Objects. bile Information Access. In Proceedings of the Fif- SRC Tech Report 115, Feb 1994. teenth SOSP, Dec 1995. [Broch et al. 98] J. Broch, D. B. Johnson, and D. A. [Jul et al. 88] Eric Jul, Henry Levy, Norman Hut- Maltz, The Dynamic Source Routing Protocol for chinson, Andrew Black. Fine-Grained Mobility in Mobile Ad hoc Networks. Internet-Draft, draft-ietf- the Emerald System. ACM TOCS, 6(1), Feb. 1988, manet-dsr-01.txt, Dec. 1998. pp. 109-133. [Cheriton 88] David Cheriton. The V Distributed [Kremer et al. 00] U. Kremer, J. Hicks, and J. Rehg. System. Communications of the ACM, 31(3), Compiler-directed remote task execution for power March 1988, pp.314-333. management: A case study. Workshop on Compil- [Douglis et al. 95] Fred Douglis, P. Krishnan and ers and Operating Systems for Low Power, PA, Oc- Brian Bershad. Adaptive Disk Spin-down Policies tober 2000. for Mobile Computers. In 2nd USENIX Sympo- [Kunz & Omar 00] T. Kunz and S. Omar. A Mobile sium on Mobile and Location-Independent Com- Code Toolkit for Adaptive Mobile Applications. puting, April 1995. IEEE Workshop on Mobile Comp. Syst. and Apps, [Flinn 01] Jason Flinn, Dushyanth, Narayanan, and M. Monterey, CA Dec 2000. Satyanarayanan. Self-Tuned Remote Execution for [Lewis & Grimshaw 95] Mike Lewis and Andrew Pervasive Computing. In Proceedings of the 8th Grimshaw. The Core Legion Object Model. Pro- Workshop on Hot Topics in Operating Systems ceedings of the Fifth IEEE International Sympo- (HotOS-VIII), Schloss Elmau, Germany, May sium on High Performance Distributed Computing, 2001. August 1995. [Foster & Kesselman 97] I. Foster, C. Kesselman. [Liskov et al. 92] Barbara Liskov and Mark Day and Globus: A Metacomputing Infrastructure Toolkit. Liuba Shrira. Distributed Object Management in Intl J. Supercomputer Applications, 11(2):115-128, Thor. In Proc. of the International Workshop on 1997. Distributed Object Management, 1992, pp. 79-91.

[Litzkow et al. 97] Michael Litzkow, Todd [Popek & Walker 85] G. Popek and B. Walker, eds. Tannenbaum, Jim Basney, and Miron Livny. The LOCUS Distributed System Architecture. MIT Checkpoint and migration of UNIX processes in Press, Cambridge, MA 1985. the Condor distributed processing system. Techni- [Pottie & Kaiser 00] G.J. Pottie and W.J. Kaiser. Wire- cal Report #1346, University of Wisconsin- less integrated network sensors. Communications Madison, April 1997. of the ACM, 43(5):51--58, May 2000. [Lorch & Smith 98] Jacob R. Lorch and Alan Jay [Rashid & Robertson 81] Rashid, R.F., Robertson, G.G. Smith. Software Strategies for Portable Computer Accent: A Communication Oriented Network Op- Energy Management. IEEE Personal Communica- erating System Kernel. 8th ACM SOSP. Pacific tions Magazine, 5(3), June 1998. Grove, California, 1981. [Ma et al. 99] Matchy J. M. Ma, Cho-Li Wang, Francis [Sirer et al. 99] Emin Gün Sirer, Robert Grimm, Ar- C. M. Lau and Zhiwei Xu. JESSICA: Java-Enabled thur J. Gregory and Brian N. Bershad. Design and Single System Image Computing Architecture. The Implementation of a Distributed Virtual Machine International Conference on Parallel and Distrib- for Networked Computers. 17th SOSP, South Caro- uted Processing Techniques and Applications, June lina, December 1999. 1999. [Steketee et al. 95] Chris Steketee, Piotr Socko, [Mascolo 01] Cecilia Mascolo, Licia Capra and Wolf- Bartosz Kiepuszewski. Experiences with the Im- gang Emmerich. XMIDDLE - A Middleware of plementation of a Process Migration Mechanism Ad hoc Networks. UCL-CS Research Note 00/54, for Amoeba. In Proceedings of the 19th Austral- 2001. asian Computer Science Conference, January 1995, [Murthy & Garcia-Luna-Aceves 96] S. Murthy and J.J. pp. 213-224. Garcia-Luna-Aceves. An Efficient Routing Proto- [Stellner 96] Georg Stellner. CoCheck: Checkpointing col for Wireless Networks. ACM Mobile Networks and Process Migration for MPI. International Paral- and App. J., Special Issue on Routing in Mobile lel Processing Symposium, pp. 526--531, Hono- Communication Networks, Oct. 1996, pages 183- lulu, HI, April 1996. 97. [Stemm & Katz 96] Mark Stemm and Randy Katz. [Nikaein et al. 01] Navid Nikaein, Christian Bonnet and Measuring and Reducing energy consumption of Neda Nikaein. HARP - Hybrid Ad hoc Routing network interfaces in hand-held devices. 3rd Inter- Protocol. In Proc. of the International Symposium national Workshop on Mobile Multimedia Com- on Telecommunications, 2001. munications, Sept. 1996. [Ousterhout et al. 88] J. Ousterhout, A. Cherenson, [Tanenbaum et al. 90] Tanenbaum, A.S., Renesse, R. F. Douglis, M. Nelson, and B. Welch. The Sprite van, Staveren, H. van., Sharp, G.J., Mullender, S.J., network operating system. IEEE Computer, Jansen, A.J., and Rossum, G. van: Experiences with 21(2):23--36, February 1988. the Amoeba Distributed Operating System, Com- [Park & Corson 98] Vincent D. Park and M. mun. ACM, vol. 33, pp. 46-63, Dec. 1990. Scott Corson. Temporally-Ordered Routing Algo- [Tennenhouse & Wetherall 96] D. L. Tennenhouse and rithm (TORA) version 1: Functional Specification. D. Wetherall. Towards an Active Network Archi- Internet-Draft, draft-ietf-manet-tora-spec01. txt, tecture. In Multimedia Computing and Networking, August 1998. San Jose, California, January 1996. [Perkins 97] Perkins, C.E. Ad hoc On-Demand [Tilevich & Smaragdakis 02] E. Tilevich and Y. Distance Vector (AODV) Routing. IETF MANET, Smaragdakis. J-Orchestra: Automatic Java Applica- Internet Draft, Dec.1997. tion Partitioning. European Conference on Object- [Perkins & Bhagwat 94] C. Perkins and P. Bhagwat. Oriented Programming, 2002. Highly Dynamic Destination-Sequenced Distance- [Toh 01] C.K. Toh. Maximum Battery Life Routing to Vector Routing (DSDV) for Mobile Computers. Support Ubiquitous Mobile Computing in Wireless ACM SIGCOMM, October 1994. Ad hoc Networks. IEEE Communications, June [Pillai & Shin 01] Padmanabhan Pillai and Kang G. 2001. Shin. Real-Time Dynamic Voltage Scaling for [Weiser et al. 94] Mark Weiser, Brent Welch, Alan Low-Power Embedded Operating Systems. SOSP Demers, and Scott Shenker. Scheduling for reduced 2001, pp. 89-102. CPU energy. In Proc. of the First OSDI, Monterey, [Plank et al. 95] James S. Plank, Micah Beck, Gerry California, November 1994. Kingsley and Kai Li. Libckpt: Transparent Check- pointing under Unix. Usenix Winter 1995 Techni- cal Conference, New Orleans, LA, January 1995.