A Portable Interface for Runtime Energy Monitoring: Extended Analysis

Connor Imes ∗ Lars Bergstrom Henry Hoffmann University of Chicago Mozilla Research University of Chicago [email protected] [email protected] [email protected]

ABSTRACT ers may even have to implement different protocols and data for- As energy consumption becomes a first class concern for comput- mats [32, 42]. Furthermore, even a single system equipped with ff ing systems, there is an increasing need for application-level ac- power or energy monitoring can o er multiple approaches to ob- cess to runtime power/energy measurements. To support this need, tain it [10, 21, 35]. Correctly and accurately collecting power and a growing number of power and energy monitors are being de- energy data requires understanding and properly managing sensor veloped, each with their own interfaces. In fact, the approaches properties and behavior. ffi are extremely diverse, and porting energy-aware code to new plat- Despite these di culties, software increasingly relies on power forms with new hardware can involve significant rewriting effort. and energy metrics. In general, software systems that use this data To reduce this effort and support portable, application-level energy fall into two categories: profilers that characterize an application’s monitoring, a common interface is needed. In this paper, we pro- power and energy usage, and self-adaptive applications that ac- pose EnergyMon, a portable application interface that is indepen- tively modify their behavior to manage power or energy consump- / dent of underlying power/energy data sources. We demonstrate En- tion. Power energy profiling is important for applications rang- / ergyMon’s flexibility with multiple case studies, including energy- ing from mobile embedded platforms [2, 13, 29, 33, 36, 46] to aware profiling and self-adaptive systems, which require monitor- more traditional end user systems and High Performance Com- ing energy across a range of hardware from different manufactur- puting environments [9, 25, 43]. Other approaches create self- ers. We release the EnergyMon interface, implementations, utili- adaptive applications [6, 24, 37] that change their runtime behavior ffi / ties, and Java and Rust bindings and abstractions as open source. to achieve better energy e ciency or respect runtime power energy constraints [1, 12, 14, 17, 20, 40]. However, all this prior work uses ad-hoc power/energy monitoring approaches. None have proposed 1. INTRODUCTION a common, portable approach for exposing the required energy con- Power and energy consumption are becoming increasingly im- sumption data to software. portant measures of software quality, especially as battery-driven A common application interface for portable and practical access mobile platforms become ubiquitous. Even small improvements in to power and energy monitoring hardware is clearly needed. To power/energy management can save significant amounts of battery this end, we propose EnergyMon, the first application-level inter- and extend a system’s usable runtime by hours. As a result, there face designed specifically for exposing energy metrics from diverse is a growing need for software to access power/energy data in-situ. sources in a portable manner. However, accessing that data requires writing system-specific code This paper makes the following contributions: which is challenging for engineers and results in applications that • Describes the challenges and importance of capturing run- are not portable across platforms. time energy metrics in software, motivating the need for a The diversity in sensor properties and current ad-hoc approaches common interface. for using them presents a significant challenge in accessing runtime • Proposes EnergyMon, a portable API for capturing energy power/energy measurements. The problem is more complex than data from diverse sources. simply locating and reading the correct files, hardware registers, • Demonstrates EnergyMon’s versatility using performance/power or memory addresses. Other factors must be considered. Some application and system characterization, power/energy-aware sensors report energy while others report power. Energy counters profiling, and self-adaptive applications as case studies, exe- eventually overflow. Some sensors, particularly external devices, cuted on various platforms with different power/energy data require exclusive access. Each may use different units of measure- sources and distinct performance/energy behavior. ment with various levels of precision. All sensors have refresh in- • Open-source release of the EnergyMon API, current imple- tervals. Sensors which only report power data must be polled at mentations, bindings and abstractions in Java and Rust, and 1 their refresh interval so as not to lose data after the sensor refreshes. additional utilities. Power/energy readings may need to be extracted from other data Furthermore, EnergyMon is publicly available in Servo, Mozilla’s or transformed before they are meaningful [32, 35, 42]. Develop- parallel web browsing engine [3] (see Section 5).

∗This work was partially conducted while Connor Imes was at Mozilla Research. 2. BACKGROUND AND MOTIVATION The University of Chicago’s effort on this project is funded by This section motivates the need for a portable, application-level the U.S. Government under the DARPA BRASS program, by the energy monitoring interface. We first discuss the challenges pre- Dept. of Energy under DOE DE-AC02-06CH11357, by the NSF under CCF 1439156, and by a DOE Early Career Award. 1Available at https://github.com/energymon Table 1: Diverse properties of power/energy data sources. Nickname Data Source Data Type Access Units (Dec. Precision) Overflow Refresh Interval Interface msr Intel MSR via msr kernel module [35] Energy Shared platform-specific 32 bits 1 ms dev file(s) rapl Intel MSR via intel_rapl kernel module [10] Energy Shared microjoules platform-specific 1 ms sysfs files odroid TI INA-231 sensors [21] Power Shared watts (6) - 264 ms sysfs files odroid-ioctl TI INA-231 sensors [21] Power Shared microwatts - 264 ms dev files osp ODROID Smart Power via hidapi [15, 32] Energy Exclusive watt-hours (3) 1000 Wh 200 ms USB osp-polling ODROID Smart Power via hidapi [15, 32] Power Exclusive watts (3) - 200 ms USB wattsup WattsUp? meter [42] Power Exclusive deciwatts - 1 sec USB shmem Shared memory provider Energy Shared microjoules provider-managed provider-dependent memory dummy None - Shared - - - memory sented by the array of different power and energy sensors avail- Algorithm 1 Computing Energy from Power Sensors. able across modern hardware platforms. We then demonstrate the Require: PowerS ensor ⊲ The power sensor importance of collecting runtime energy data by discussing prior Require: Re f reshInterval ⊲ The sensor’s refresh interval work in building software systems for both energy profiling and Energy ← 0 ⊲ Energy counter, initialized to 0 self-adaptive power/energy management. 2.1 Challenges function PollPowerSensor ⊲ Local function loop As power and energy consumption become ever more crucial Wait(Re f reshInterval) measures of software quality, hardware power and energy sensors Power ← ReadPower(PowerS ensor) are becoming more prevalent in computing systems. The diversity Energy ← Energy + Power × Re f reshInterval of available sensors complicates the process of collecting and in- end loop terpreting the data they produce. Any portable interface for energy end function monitoring must account for the following differences: • Data Type: Whether sensors expose power or energy data. function ReadEnergy ⊲ Externally visible function • Access: Access privileges/constraints, e.g., exclusiveness. return Energy • Units of measurement: The power or energy units that sen- end function sors expose measurements in, and their precision. • Overflow: Maximum values for sensor counters. relies on power or energy data must contain all of this platform- • Refresh Interval: How frequently sensors update. specific code to account for the diversity in data sources. • Interface: System abstractions, protocols, and data formats. Table 1 demonstrates these differences for the data sources used 2.2 Power and Energy-Aware Software in this work. The first column are the nicknames we use to refer Software requires runtime access to energy data for various pur- to the sensor access implementations, the second describes the un- poses including profiling and self-adaptation. derlying data source, and the rest present the details for properties Power and Energy-Aware Profiling: With the proliferation of listed above. The fact that refresh intervals span three orders of energy-constrained systems like smartphones, tablets, and now the magnitude, from the MSR’s 1 ms to the WattsUp?’s 1 second, ex- Internet of Things, power and energy-aware profiling has become emplifies how diverse sensors can be. In some cases there are even increasingly important for developers who previously have not been multiple implementations for the same data source because they concerned with this aspect of software behavior. PowerScope is one expose multiple data types or interfaces. of the early works that recognized the importance of profiling en- One property that necessitates further discussion is the data type ergy usage for mobile applications [13]. More recently, Banerjee et – whether sensors expose power or energy data. Assume a sensor al. used Android device power profiles to detect energy hotspots exposes cumulative energy data. Software that requires energy data and bugs in applications [2]. Other research has proposed new simply takes the difference in readings at the beginning and end of power modeling techniques, investigated the behavior of SoCs or the event being evaluated. Software interested in power does the components like GPS and cameras, and explored power behav- same, but divides the difference in energy by the elapsed time to ior that is unique to smartphones like tail power states and wake- get an average power. Power sensors expose a value that is the av- locks [30, 33, 34, 36, 46]. Power and energy-aware profiling is not erage power over an interval (e.g., one second), and are more com- limited to mobile applications and devices. Cui et al. instrument plicated to use. To record energy, software must read the power in PC hardware to capture power with fine-grained time granularity each interval, multiply that value by the refresh interval to compute as low as 20 microseconds [9]. PAPI 5 provides access to hardware energy, and keep a running total. Algorithm 1 demonstrates this performance counters including those that measure power/energy approach. If the power sensor refreshes before its value is read, the data exposed by common hardware in HPC environments [43]. energy consumption over that interval is lost since the next power Self-Adapting Applications: Applications that monitor and ad- value may be different. Software interested in power must also poll just their behavior at runtime to obey performance, power, and en- at the refresh interval and keep a running average of the readings ergy constraints are becoming more common [37]. Software can during the desired time interval. even exhibit contrasting power/energy behavior on different hard- Software that reads from a power or energy data source must un- ware [18]. To address these problems, some applications access derstand its behavior in order to use it properly. Such a task may in- power/energy data in deployment to automatically adapt their be- volve managing sensor polling threads, tracking elapsed time, lim- havior. The Green language creates adaptive applications that min- iting the frequency of sensor accesses, performing I/O, parsing and imize energy given an acceptable accuracy [1], while the Eon lan- transforming data readings to consistent units, and sharing the data guage and runtime adapt embedded applications to the availability with other software components. For example, our energy monitors of harvested energy [40]. The Odyssey [14], GRACE [41, 44], for the data sources listed in Table 1 require roughly 250-300 lines CoAdapt [16], JouleGuard[17] and xTune [22] projects automati- of code on average. Without a portable interface, software that cally coordinate applications and systems to meet real-time goals with reduced energy in mobile and embedded systems, while Mo- data length to avoid overflow. Applications therefore do not need to hapatra et al. provide a similar framework for distributed systems track various unit types and do conversions. A deceptively simple [31]. The POET library enables applications to tune their own re- but important insight is not our choice of units or data length, but source usage to meet soft real-time constraints with minimal en- rather that counter overflow is one of the diverse sensor properties ergy [20]. Similar projects automatically tune server applications that applications should not have to worry about. We choose micro- to meet power goals with maximum performance [7, 19, 45]. Klein joules ( µJ) as 64-bit unsigned integers2, which are precise enough et al. recently proposed a technique to alter webpage content based for the use cases we anticipate and avoid the need for floating point on available energy and power resources [23]. data types. In our experience, modern sensors are too imprecise and models too inaccurate to justify smaller units. 2.3 The Need for a Common Interface The energymon C interface defines a struct that contains seven Prior works have used an assortment of sensors, but none have function pointers and a state variable: proposed a common interface specifically for making energy data • finit: Initialize the energy monitor. available to software in a portable manner. Tools like PAPI are use- • fread: Get the cumulative energy in microjoules. ful in some cases, but do not present data in a consistent format and • ffinish: Destroy the energy monitor. instead just provide direct access to the hardware sensors. Thus, • fsource: Get the name for the energy monitoring source. PAPI and similar low-level tools require making application code • finterval: Get the refresh interval in microseconds. changes to support new power/energy data sources [43]. This re- • fprecision: Get the best possible precision in microjoules. quirement might be reasonable for the HPC domain it is designed • fexclusive: Get if exclusive access is required. for, but fast becoming impractical for software like mobile appli- • state: Pointer to a struct that the implementation uses to cations that are deployed on a wide variety of continually evolving maintain internal state, e.g., file descriptors or thread data. systems. In POET, the authors created a primitive interface that This interface exposes some of the diverse properties described pre- was internal to their tool in an attempt to access energy data in a viously, like refresh interval and sensor precision, while hiding more portable manner than had been done previously [20]. POET other properties that are not important to most software, like the is representative of many self-adaptive systems in that it is not de- underlying interfaces, device protocols, and data formats. signed to report energy usage, but simply use that data internally. A user typically calls energymon_get_default provided by Such systems do not expose implementation properties like refresh the energymon-default interface to populate an energymon struct. interval, precision, or exclusiveness, nor do they enable sensible If the implementation is known in advance, its getter function can bindings to other programming languages. Additionally, their im- be called directly. The default interface enables maximum portabil- plementations do not manage practical concerns like counter over- ity since application code does not require any modification when flows or preservation of error codes. changing data sources. Using the function pointers in the energy- As software matures, its shelf life increases. That software will mon struct provides portability for all other operations. The deci- continue to be deployed on new hardware for years to come, with sion to use function pointers instead of explicitly declaring com- few upfront guarantees about the hardware’s power/energy proper- mon header functions was made for practical reasons – in C, only ties or how it will expose such data. Additionally, it is infeasible one implementation of an interface can be implemented in a run- for most software to manage the challenges in correctly using pow- time. Reducing the number of common header functions min- er/energy sensors for more than a handful of data sources. Much imizes the need for conditionally including extra wrapper func- of the information needed to effectively use power/energy sensors tions at compile time (energymon_get_default is the only one). like the access restrictions and operating system abstractions are Therefore, implementations conditionally include the default get- not important to the software that needs the data – they are periph- ter function, allowing multiple EnergyMon implementations to be eral concerns. An interface can hide these details while exposing used simultaneously when there are multiple power/energy data important properties in a common manner, like the actual energy sources known in advance. values, the refresh interval, and the sensor’s precision, while requir- EnergyMon implementations are initialized by calling the finit ing only a few lines of platform-independent code. We address the function pointer. If the implementation reads from a power sensor, challenges described in Section 2.1 and aforementioned problems finit starts a separate thread to poll the sensor at its refresh interval in prior work with EnergyMon, a portable interface for accessing and update the cumulative energy value after each sensor reading. energy data in-situ that is independent of the data source. It can eas- The application reads from the energy monitor by invoking fread. ily be used by current and future power and energy-aware software For any time τ, where τ = 0 is the energy monitor’s initialization, without the need to change code for different deployment scenarios the following invariants hold for energy readings E(τ): or even know the available sensors in advance. ∀τ ≥ 0 τi ≤ τ j =⇒ 0 ≤ E(τi) ≤ E(τ j) (1) 3. DESIGN AND IMPLEMENTATIONS τ = 0 6 =⇒ E(τ) = 0 (2) Eqn. 1 states that for any time at or after initialization, energy is 3.1 Interface Design always non-negative and monotonically increasing. The only ex- The first important design decision is which data type to expose ception should be in case of a sensor read failure – in C, a 0 value in a common interface – we choose energy. Unlike power, energy is is returned and errno set appropriately; abstractions in other lan- not explicitly a function of time and can be recorded cumulatively, guages may use their own error handling mechanisms, like excep- simplifying the interface. With an energy provider that records total tions. Eqn. 2 declares that the energy counter does not have to begin energy consumed since some time τ = 0, a power-aware applica- at 0 µJ. This flexibility facilitates sharing energy data with multi- tion can simply record energy E at the beginning and end of a time ple applications and avoids the need to reset hardware counters, interval and compute the average power as ∆E/∆τ. In this way, the which is often a privileged operation. When the energy monitor interface easily supports applications interested in energy or power. is no longer needed, ffinish is called to cleanup resources like files Next, a common interface can expose energy metrics in a stan- dard and sufficiently precise unit of measurement at a large enough 2At 100 W of power, it would take about 5,850 years to overflow. or threads. The following example of the EnergyMon C interface 3.3 Bindings and Abstractions demonstrates computing the average power of a worker function: Current EnergyMon implementations are written in C, but we

1 energymon em; also provide interfaces in Java and Rust. Additionally, we make 2 uint64_t start_uj, end_uj, start_usec, end_usec; bindings and abstractions over the energymon-default C interface 3 available for those languages. Bindings provide as close to direct 4 // get the energymon instance and initialize 5 energymon_get_default(&em); access to the underlying interfaces and implementations as possi- 6 em.finit(&em); ble. Abstractions provide the safety guarantees these languages 7 demand and the RAII3 idioms they commonly use. 8 // get start time/energy 9 start_usec = gettime_us(); // e.g. wrap clock_gettime In Java, we provide the EnergyMon interface, which implemen- 10 start_uj = em.fread(&em); tations written in pure Java may implement. Its methods throw ex- 11 ceptions in error cases. The DefaultEnergyMon abstraction man- 12 // application-specific processing 13 do_work(); ages a JNI boundary, encapsulates pointers to heap allocations, and 14 cleans up automatically. Users may also safely destroy an instance 15 // get end time/energy themselves to immediately clean up resources, which might include 16 end_usec = gettime_us(); background threads, if they do not wish to wait for garbage collec- 17 end_uj = em.fread(&em); 18 tion. For example, to use the default Java abstraction: 19 printf("Average power of do_work() in Watts: %f\n", 20 (end_uj-start_uj) / (double) (end_usec-start_usec)); 1 // get the EnergyMon instance and initialize 21 2 EnergyMon em = DefaultEnergyMon.create(); 22 // destroy the instance 3 23 em.ffinish(&em); 4 // get start time/energy 5 long startMs = System.currentTimeMillis(); Listing 1: EnergyMon C example. 6 long startUj = em.readTotal(); 7 Other functions provide additional information about the imple- 8 // application-specific processing 9 doWork(); mentation that software can use to make runtime decisions. For 10 example, applications can use the name provided by fsource for 11 // get end time/energy debugging or logging. finterval can be used to determine a lower 12 long endMs = System.currentTimeMillis(); 13 long endUj = em.readTotal(); bound on sleep time for polling the energy monitor at regular in- 14 tervals. fprecision might inform a self-adaptive system how much 15 System.out.println("Average power of doWork() in Watts: " noise to expect in power/energy data. fexclusive could be used in 16 + (endUj - startUj) / (1000.0 * (endMs - startMs))); 17 deciding to share energy data with other software components. 18 // destroy the instance (optional, but recommended) 19 em.finish(); 3.2 Implementations Listing 2: EnergyMon Java interface and abstraction. We provide EnergyMon implementations for all the data sources detailed in Table 1, which access various power and energy sensors, In Rust, we provide the EnergyMonitor trait where functions ff both embedded and external. The table describes their di erences that are allowed to fail return a Result. EnergyMon implementa- in more detail. While not strictly a requirement, the interface is best tions written in Rust may implement this trait. The default Ener- leveraged when the underlying energy source makes data available gyMon abstraction encapsulates the struct bindings and prevents to userspace. This allows sandboxed or other unprivileged applica- accidental or malicious destruction of an instance. For example, to tions to measure their own behavior, or even to adjust their behavior use the default Rust abstraction: at runtime, e.g., to meet power cap requirements. Some systems expose power or energy data for multiple sources. 1 // get the EnergyMon instance and initialize For example, the Intel MSR provides energy data for core and un- 2 // EnergyMon implements the EnergyMonitor trait 3 let em: EnergyMon = EnergyMon::new().unwrap(); core components, as well as DRAM. Both the msr and rapl Energy- 4 Mon implementations use the MSR as their data source. Similarly, 5 // get start time/energy the ARM big.LITTLE system used in this paper exposes power 6 let start_time: SystemTime = SystemTime::now(); 7 let start_uj: u64 = em.read_uj().unwrap(); data for the SoC’s big cluster, LITTLE cluster, GPU, and DRAM. 8 The osp, osp-polling, and wattsup implementations read from 9 // application-specific processing external sensors. The first two read from an ODROID Smart Power 10 do_work(); 11 meter [32] – osp reads energy data in Watt-hours, accurate to three 12 // get end time/energy decimal places; osp-polling polls the sensor’s power readings in 13 let end_time: SystemTime = SystemTime::now(); Watts, also accurate to three decimal places. The polling imple- 14 let end_uj: u64 = em.read_uj().unwrap(); mentation provides more accurate energy data given that Watt-hours 15 let duration: Duration = end_time.duration_since( 16 start_time).unwrap(); are so imprecise (1 Wh = 3600 J). The third implementation reads 17 from a WattsUp? power meter, which provides power in deciwatts 18 println!("Average power of do_work() in Watts: {}", at coarse-grained one second intervals [42]. 19 (end_uj - start_uj) as f64 / 20 (duration.as_secs() * 1000000 + The shmem implementation reads from a shared memory loca- 21 duration.subsec_nanos() as u64 / 1000) as f64); tion. Sensors that require exclusive access and/or elevated privi- 22 leges are ideal candidates to run in separate processes and expose 23 // instance dropped automatically 24 // (when it falls out of scope or is consumed) their data to multiple unprivileged applications over shared mem- ory. This is especially true for external meters, so we include shared Listing 3: EnergyMon Rust trait and abstraction. memory providers for the osp-polling and wattsup implementa- tions. The dummy implementation is used when no other sources are available. 3Resource Acquisition is Initialization 4. EXPERIMENTAL DESIGN Idle All Idle big LITTLE 4.1 Evaluation Platforms 1.0 1.0 We use three very different hardware systems for our analysis. The first is a Lenovo Thinkpad X1 Carbon laptop (3rd gen) with 0.75 0.75 an Intel i7-5600U dual-core processor with Hyperthreading [11], 0.5 0.5 running Ubuntu 15.10 with Linux kernel 4.2.0. The second is a Power Power 0.25 0.25 Hardkernel ODROID-XU3 mobile development platform with an 0 0 Exynos5 Octa 5422 SoC with quad core ARM Cortex-A15 “big” 0 0.25 0.5 0.75 1.0 0 0.25 0.5 0.75 1.0 and Cortex-A7 “LITTLE” clusters [38]. The ODROID runs ei- Per f ormance Per f ormance ther Ubuntu 14.04 LTS with Hardkernel’s modified Linux kernel 3.10.58+ or Android 4.4 “KitKat”, depending on the case study. (a) Intel Haswell (wattsup) (b) ARM big.LITTLE (osp- The third system is a Sony VAIO SVT11226CXB tablet with a polling) dual-core Intel Haswell mobile processor with Hyperthreading [35]. Figure 1: Performance/power tradeoffs using DVFS and core allo- It runs Ubuntu 14.04 LTS with Linux kernel 3.13.0. Prior work has cation for a parallel video encoding application. shown that the Haswell processor and the Exynos big.LITTLE SoC tion runtime. This experiment not only demonstrates EnergyMon’s / exhibit contrasting performance energy behavior [18]. potential for self-adaptive applications, but demonstrates its flexi- 4.2 Case Studies bility and portability by using sensors the self-adapting controller was not originally designed for. Evaluating and exploiting performance and power tradeoffs are In summary, our case studies use Linux and Android and run on common research topics, particularly in the embedded and mo- three different hardware platforms – a tablet with an Intel Haswell bile systems domain. Our first case study demonstrates diversity processor, an ARM big.LITTLE development platform, and a lap- in both application and system performance/power tradeoffs, moti- top with an Intel i7 processor. We use four distinct power/energy vating the need for energy awareness in applications. We capture data sources – the Intel MSR, embedded TI INA-231 power sen- coarse-grained timing and energy behavior for two distinct, unmod- sors, an external WattsUp? PRO meter, and an external ODROID ified applications using the Vaio and the ODROID, both running Smart Power meter. Linux. It is common for engineers to use external instrumentation when evaluating software behavior on different systems, so we use a WattsUp? PRO meter with the wattsup EnergyMon implemen- 5. EXPERIMENTAL EVALUATION tation on the Vaio, and an ODROID Smart Power meter with the This section presents our analysis of EnergyMon with case stud- osp-polling implementation on the ODROID. Since video encod- ies that motivate the importance of energy awareness in software ing is a common task for embedded systems like mobile phones and and demonstrate EnergyMon’s utility. We also analyze Energy- tablets that have on-board cameras, we use a compute-bound paral- Mon’s runtime overhead, then discuss limitations and future work. lel video encoder for the first application [4]. We use the STREAM benchmark as the second example to represent memory-intensive 5.1 Performance/Power Tradeoffs applications [28]. Our first case study demonstrates diversity in performance/power The second case study profiles Servo, Mozilla’s parallel web tradeoffs for systems and applications, motivating the importance browsing engine written in Rust [3, 27], at a fine-grained level. This of energy awareness in software. To properly manage system re- experiment uses the Thinkpad and the ODROID. The Thinkpad source allocation, engineers must understand their application’s be- runs Linux and uses the msr EnergyMon implementation to read havior, even if only at a coarse-grained level. This is especially from the Intel MSR [35]. The ODROID now runs Android and true for applications that run on energy-constrained platforms like uses the odroid-ioctl implementation to poll embedded TI INA-231 embedded and mobile systems. Software exhibits different perfor- power sensors [21]. We use a real, industrial-strength application to mance/power tradeoffs depending on the system it is executed on, demonstrate that EnergyMon provides in-situ runtime energy data and behavior can even vary dramatically for different applications on different hardware and software platforms suitable for profiling running on the same system. Performance/power tradeoffs are a in a practical setting. function of both the application and the system. We first demon- Our third case study returns to the parallel video encoding appli- strate tradeoff diversity between systems with a parallel video en- cation used in the first experiment [4], but now uses a version that coding application [4]. We then show diversity across applications was previously modified to configure resource allocation at runtime on the same system by comparing these results with those produced to obey soft power constraints. Again we use the Vaio tablet and the by the STREAM benchmark, which represents memory-intensive ODROID, both running Linux. The modified application’s feed- applications [28]. back controller was originally designed for the systems’ embedded We connect the tablet with the Intel Haswell mobile processor sensors, but we substitute the energymon-default interface and use and the ARM big.LITTLE development system to external WattsUp? external power meters. This provides an additional challenge and PRO and ODROID Smart Power meters, respectively. We allocate further demonstrates EnergyMon’s utility and portability. We once unique combinations of DVFS frequency and core assignment – again connect the Vaio to an external WattsUp? PRO power me- 44 configurations for the Intel Haswell and 128 configurations for ter [42] and the ODROID to an external ODROID Smart Power the ARM big.LITTLE. We record the total application runtime and meter [32]. Because these meters require elevated privileges and system energy consumption for each configuration, which we use exclusive access, we now launch the wattsup and osp-polling En- to compute normalized average performance and power values. ergyMon implementations in a separate process and expose energy Figure 1 is the resulting tradeoff space for the video encoder, nor- data to the application using shared memory. This makes the en- malized to the maximum performance and power achieved on each ergy data available to any application that wants it, which we ac- system. System idle power is included for reference. The Intel cess with the shmem EnergyMon implementation during applica- Haswell has a mostly linear tradeoff space, with maximum perfor- 7000 Idle All Idle big LITTLE Time 80 6000 Energy 1.0 1.0 5000 60 0.75 0.75 4000 0.5 0.5 3000 40 Power Power Time (ms) 2000 0.25 0.25 20 Energy (Joules) 1000 0 0 0 0.25 0.5 0.75 1.0 0 0.25 0.5 0.75 1.0 0 0 Per f ormance Per f ormance

(a) Intel Haswell (wattsup) (b) ARM big.LITTLE (osp- Painting

LayoutMain ScriptEvent polling Compositing ScriptConste ScriptResizeScriptSetVie ) ApplicationH LayoutParallLayoutPerforLayoutRestylLayoutStyleRPaintingPerT ScriptNetwor ScriptUpdate LayoutDispLiLayoutGenera ScriptDocumeScriptImageC Figure 2: Performance/power tradeoffs using DVFS and core allo- cation for a parallel memory-intensive application. (a) Intel i7 on Linux (msr) mance about 8× the minimum, and maximum power slightly less than 2× the minimum. The ARM big.LITTLE’s tradeoff space is 140 Time much richer and noticeably more convex, with a performance and 30000 120 Energy power range much greater than the Intel Haswell. Its maximum per- 25000 formance is nearly 70× greater than the minimum, and maximum 100 power is more than 5× the minimum. The shapes of the tradeoff 20000 80 spaces for the video encoding application are similar to those of 15000 60

other parallel compute-bound applications we evaluated on these Time (ms) 10000 40

systems. Energy (Joules) 5000 20 Figure 2 shows how different the tradeoffs are for memory-intensive applications compared to the video encoder’s in Figure 1. The 0 0 Intel Haswell no longer has any configurations between the per- formance/power extremes. Without this knowledge, a developer Painting

LayoutMain ScriptEvent ScriptResize or user might naively assume that mid-range resource allocations Compositing LayoutParall ScriptConste ScriptSetVie ApplicationHLayoutDispLi LayoutPerforLayoutRestylLayoutStyleRPaintingPerT ScriptNetwor ScriptUpdate would provide mid-range performance and power consumption. The LayoutGenera ScriptDocumeScriptImageC ff ARM big.LITTLE still o ers configurations that cover the range of (b) ARM big.LITTLE on Android (odroid-ioctl) performance/power values, but a performance wall is eventually Figure 3: Cumulative time and energy consumption in Servo. reached as more resources are allocated. In fact, many configura- tions that were high-performance/high-power for the video encoder to save space, and unused categories are not shown. Engineers use are still high-power but actually have lower performance than some timing and energy data to decide where to focus their optimization configurations that require fewer resources and less power. Failure efforts. Developers will recognize that tasks running in parallel, to recognize these differences may result in inefficient or even in- like PaintingPerTile, record their own elapsed time which results in adequate system resource allocations for applications. total time values that exceed actual runtime. Energy is measured The specific tradeoffs for the systems and applications we ex- for shared system components, so some energy consumption is as- amine are not the important conclusions in this case study. What signed to multiple profiler categories and is therefore counted more is important is that power and energy-aware applications must rec- than once (the sum of the energy values exceeds the actual energy ognize and manage these behavioral characteristics to make intel- consumed). More advanced profiling and analysis techniques be- ligent runtime decisions. This diversity in application and system yond the scope of this paper are needed to better assign time and behavior motivates the need for energy awareness in software – a energy consumption to parallel tasks. ApplicationHeartbeat pro- capability that EnergyMon facilitates. vides the true wall clock time and total energy consumption for the execution. Layout operations require a significant portion of the 5.2 Servo total time and energy on the Intel i7 in Linux, but painting con- sumes a larger percentage of the runtime on the ARM big.LITTLE This case study uses Servo, Mozilla’s parallel web browsing en- in Android. This means developers need to concentrate their ef- gine. We add energy monitoring to Servo’s existing fine-grained forts in different places depending on the system they are trying to time profiling interface and add a background profiler category optimize for. called ApplicationHeartbeat to record runtime power behavior. We Figures 4a and 4b present the time series for a single execution, profile Servo on the Thinkpad with an Intel i7 processor running overlayed with system power derived from ApplicationHeartbeat’s Linux, and the ODROID with an ARM big.LITTLE SoC running energy readings. The ApplicationHeartbeat polling interval is lim- Android. We capture energy readings for each platform’s proces- ited by the refresh rate of the sensors underlying the EnergyMon sors rather than using external equipment. The experiment loads implementation, as exposed by the finterval function, hence the Servo, fetches the page mozilla.org from the remote server, pro- more coarse-grained power data on the ARM big.LITTLE system cesses the page, and displays it. We instruct Servo to use four lay- (its power sensors refresh about 4× per second). The Intel i7’s MSR out threads on each evaluation system (the Intel i7 has four virtual refreshes every 1 ms, but we limit the profiler’s polling interval to cores, and each of the ARM big.LITTLE clusters has four cores). 50 ms to avoid unnecessary overhead. Figures 3a and 3b break down by profiler category where time The LayoutPerform profiler category is an umbrella event for all and energy are spent within Servo while loading on the two plat- layout operations. LayoutStyleRecalc determines which CSS styles forms, averaged over four trials. Profiler category names are trimmed ScriptUpdateRepl 20 ScriptSetViewpor ScriptResize ScriptNetworkEve ScriptImageCache ScriptEvent 15 ScriptDocumentEv ScriptConstellat PaintingPerTile Painting 10 LayoutStyleRecal LayoutRestyleDam

LayoutPerform Power (Watts) LayoutParallelWa LayoutMain 5 LayoutGeneratedC LayoutDispListBu Compositing ApplicationHeart 0 0 1000 2000 3000 4000 5000 Time (ms)

(a) Intel i7 on Linux (msr)

ScriptUpdateRepl ScriptSetViewpor ScriptResize 8 ScriptNetworkEve ScriptImageCache ScriptEvent ScriptDocumentEv 6 ScriptConstellat PaintingPerTile Painting LayoutStyleRecal 4 LayoutRestyleDam

LayoutPerform Power (Watts) LayoutParallelWa LayoutMain LayoutGeneratedC 2 LayoutDispListBu Compositing ApplicationHeart 0 0 2000 4000 6000 8000 10000 12000 Time (ms)

(b) ARM big.LITTLE on Android (odroid-ioctl) Figure 4: Timing and power behavior while loading mozilla.org in Servo. apply to each of the nodes in the Document Object Model (DOM) Despite the differences in the two platforms, we are able to cap- tree and creates a structure that layout will use to determine element ture useful profiling energy data from both using the common En- positioning. This is a time-consuming operation given that it must ergyMon interface. This energy profiling capability is now publicly traverse the DOM tree. LayoutMain then determines where to posi- available as part of Servo via the energy-profiling feature. tion elements when rendering on the screen. The time series makes it clear that layout tasks, especially LayoutStyleRecalc, consume more power than average on the Intel i7. The power data captured 5.3 Self-Adaptive Application on the ARM big.LITTLE in Android is more coarse-grained, but This experiment uses an energy-aware parallel video encoding accounting for the delays also indicates the same trend. application that adjusts its resource usage at runtime to meet soft On ARM, there is a large power spike about halfway through the power constraints [19]. Given a power target and set of config- execution – something we saw in all of our trials. The current pro- urable system resources, it adapts to meet the power target while filing does not point to an obvious reason for this, indicating that maximizing performance. We substitute EnergyMon for the con- the application developers should investigate further. More impor- troller’s custom energy monitoring code. To further challenge En- tantly, this behavior would not be detected by time profiling alone, ergyMon and demonstrate its versatility, we again use external in- yet is critical on power and energy-constrained systems. struments instead of the platforms’ embedded sensors that the feed- back controller originally intended to use. The experiment is par- Intel Haswell (wattsup) ARM big.LITTLE (osp-polling) f init f read f f inish 1.0 1 sec 0.75 1 ms 0.50 1 µs 0.25 Performance (Normalized)

Function Latency 1 ns 1.50 msr osp rapl dummy odroid shmemwattsup 1.25 odroid-ioctl osp-polling 1.0 Figure 6: Average function overhead (log scale).

Power 0.75 have not attempted to optimize our implementations’ performance (Normalized) 0.50 beyond the obvious design decisions required to achieve accurate 0 500 1000 1500 results, like caching file descriptors as part of the state variable (see time [frame] Section 3.1). Figure 5: Meeting soft power constraints with an energy-aware Figure 6 shows the average latency for these three functions – video encoder using EnergyMon and external power meters. note the log scale. The odroid, odroid-ioctl, osp, and osp-polling ticularly challenging because of the relatively slow refresh rate, implementations were tested on the ODROID, the rest on the Thinkpad. coarse-grained accuracy, and additional I/O overhead of commu- Initialization and destruction of energy monitors are the most time- nicating with the devices over USB. We select a window period of consuming tasks (finit and ffinish), but only need to be done once. / 50 frames – the work interval at which the application adjusts its Implementations that read from external devices must perform I O resource usage. For each system, we set the power target to be the communications with the devices and may need to wait while they mean of the minimum and maximum processing power. We then reset internally. For example, the wattsup implementation must launch the application with an input that is the combination of three sleep for a whole second after clearing the device memory before videos with distinct levels of encoding difficulty. it starts capturing, so it took an average of about 1.5 seconds to Figure 5 is a time series of the resulting behavior. The top por- completely initialize. On the other hand, the msr and rapl im- tion of the figure is the window performance (frames encoded per plementations initialized in about 50 µs. The odroid implementa- second), normalized to highest for each system; the bottom portion tion took 1.6 ms since it scans the sysfs filesystem for the sensors; shows the window power, normalized to the target. The vertical odroid-ioctl took 384 µs to open the four device files and enable the lines at frames 500 and 1000 indicate the boundary between the in- sensors. If the overhead of finit and ffinish truly matter to an ap- put video’s phases. The system power hovers around the target as plication, the energy monitor can be run in a background process the application adjusts resource usage while the performance fluc- and expose data using shared memory. Applications can then read tuates depending on the level of encoding difficulty for each frame. energy data using the shmem implementation, like in Section 5.3, Maintaining the desired power on the Intel Haswell is more dif- which initialized and finished in 9.0 µs and 2.3 µs, respectively. ficult than on the ARM big.LITTLE. The slower refresh rate and The fread function provides the core functionality and is natu- lower precision of the WattsUp? power meter, compounded by rally executed the most. The msr implementation took less than the Haswell’s faster processing speed and the variable difficulty of 1 µs on average and rapl took 15 µs. The osp implementation reads encoding the input video (even within a single phase) make the from an external ODROID Smart Power meter over USB, which task especially challenging. During the first window period (50 caused relatively high latency – about 12 ms. However, the refresh frames), the application observes system behavior, then makes its interval of the device is 200 ms, so the implementation is not very first resource allocation adjustment. For frames after the first period useful for applications that need to poll more frequently. Imple- of resource adjustment (frame 100), the mean window power per mentations that read from power sensors have background threads / frame is 0.14% above the target on the Intel Haswell and 0.027% to poll the sensors, and account for the I O and runtime latency below the target on the ARM big.LITTLE, with 4.4% and 2.0% when computing energy from power. In these cases, reading the coefficient of variation, respectively. Of these frames, 44.6% on energy data from an application only involves accessing 8 bytes of the Haswell and 43.4% on the big.LITTLE had an average win- memory. Their response times were typically around 1 µs, at which dow power above the target. This is expected when meeting soft point the precision, refresh rate, and overhead of reading the system ff power targets for an application that exhibits as much variability clock a ects the accuracy of the measurement. as a video encoder [20]. The application could be tuned further to keep more frames below the power target, but that is beyond the 5.5 Limitations and Future Work scope of this work. What is important for this case study is that As with any power/energy monitoring, whether performed in- EnergyMon successfully provided the necessary energy data to the situ or externally, results are limited by the precision, accuracy, and application’s feedback controller. Despite the challenges, Energy- refresh rate of the sensors. A sensor that only updates once per sec- Mon’s portable interface allows the application to use data from ond may not be as useful as one that updates more frequently. In external power meters it was not originally designed for to meet its cases of relatively slow refresh intervals, tasks sometimes complete power constraints. too quickly to capture an accurate change in energy values. Most of the time no change in energy will be recorded, but occasionally 5.4 Overhead Analysis a task execution will overlap with a sensor refresh and record an Runtime overhead is an important consideration when collecting inaccurately large change in energy for a short-lived task. With a power/energy data, whether it be for profiling, a feedback control statistically large enough sampling, the total energy recorded for system, or something else. We took simple in-situ timing measure- relatively short-lived tasks will average to reasonably accurate val- ments for the finit, fread, and ffinish functions on current Energy- ues. Servo’s PaintingPerTile profiler category in Figures 3 and 4 Mon implementations and averaged the results of 100 trials. We is a good example of this behavior. The most informative results we observed were captured using the Intel MSR, which refreshes [5] S. Browne, J. Dongarra, N. Garner, G. Ho, and P. Mucci. at millisecond intervals. “A Portable Programming Interface for Performance Eval- Some power/energy sensors require exclusive access, particu- uation on Modern Processors”. In: Int. J. High Perform. larly external meters. If multiple parties are interested in using the Comput. Appl. 14.3 (Aug. 2000), pp. 189–204. sensor simultaneously, a reader can run in a separate process and [6] B. Cheng, R. de Lemos, H. Giese, P. Inverardi, J. Magee, expose the data over shared memory or other IPC mechanisms. For J. Andersson, B. Becker, N. Bencomo, Y. Brun, B. Cu- example, we wrote shared memory providers for the osp-polling kic, G. Di Marzo Serugendo, S. Dustdar, A. Finkelstein, and wattsup EnergyMon implementations. Applications then just C. Gacek, K. Geihs, V. Grassi, G. Karsai, H. Kienle, J. link with the shmem implementation. Kramer, M. Litoiu, S. Malek, R. Mirandola, H. Müller, Automatic discovery of power/energy sensors is an important S. Park, M. Shaw, M. Tichy, M. Tivoli, D. Weyns, and J. next step for hardware and operating systems designers to address. Whittle. “A Research Roadmap”. In: Software Engineer- A standard operating system interface would offer greater flexibil- ing for Self-Adaptive Systems. 2009. ity to power and energy-aware applications. Advances in operating [7] R. Cochran, C. Hankendi, A. K. Coskun, and S. Reda. system support would not eliminate the need for a portable appli- “Pack & Cap: adaptive DVFS and thread packing under cation interface like EnergyMon, but rather expand its utility. power caps”. In: MICRO. 2011. Unfortunately, not all systems have power/energy sensors or are [8] G. Contreras and M. Martonosi. “Power Prediction for connected to external meters. EnergyMon implementations can in- Intel XScale R Processors Using Performance Monitoring stead use models based on other hardware counters or sensors to Unit Events”. In: ISLPED. 2005. provide energy estimates. There is prior work on building pow- [9] Z. Cui, Y. Zhu, Y. Bao, and M. Chen. “A fine-grained er/energy models based on other hardware counters, but is beyond component-level power measurement method”. In: IGCC. the scope of this work [8, 26, 34, 39, 46]. However, future work 2011. includes building implementations that can interpret models in a [10] H. David, E. Gorbatov, U. R. Hanebutte, R. Khanna, and common format, perhaps using tools like PAPI [5] to read perfor- C. Le. “RAPL: Memory Power Estimation and Capping”. mance counters as needed to drive the model. In: ISLPED. 2010. [11] A. Deval, A. Ananthakrishnan, and C. Forbell. “Power 6. CONCLUSION management on 14 nm Intel Core-M processor”. In: Low- Power and High-Speed Chips (COOL CHIPS XVIII), 2015 This paper describes the challenges arising from the diversity in IEEE Symposium in. 2015, pp. 1–3. power and energy data sources, motivating the need for a common [12] J. Durango, M. Dellkrantz, M. Maggio, C. Klein, A. V. Pa- interface to expose energy data to software. In response, we pro- padopoulos, F. Hernández-Rodriguez, E. Elmroth, and K.- pose EnergyMon, a portable interface that is independent of under- E. Arzén. “Control-theoretical load-balancing for cloud lying power/energy sensors. We demonstrate the interface’s flexi- applications with brownout”. In: CDC. 2014. bility and practicality with three use cases. We first show diversity [13] J. Flinn and M. Satyanarayanan. “PowerScope: a tool for in performance/power tradeoffs across platforms, and for applica- profiling the energy usage of mobile applications”. In: WM- tions running on the same platform, motivating the need for en- CSA. 1999. ergy awareness in software. Next, we instrument Servo, Mozilla’s [14] J. Flinn and M. Satyanarayanan. “Energy-aware adapta- parallel web browsing engine, with energy profiling and describe tion for mobile applications”. In: SOSP. 1999. the insight it provides. This feature is now publicly available in [15] HID API for Linux, Mac OS X, and Windows. http:// Servo. We then substitute EnergyMon for an energy-aware video www.signal11.us/oss/hidapi/. encoder’s custom energy monitoring code and show that it contin- [16] H. Hoffmann. “CoAdapt: Predictable Behavior for Accuracy- ues to meet imposed power constraints, even using different sensors aware Applications Running on Power-aware Systems”. than it was originally designed for. Our experiments use both in- In: ECRTS. 2014. ternal and external power and energy sensors with widely varying [17] H. Hoffmann. “JouleGuard: energy guarantees for approx- properties on platforms with different, sometimes contrasting, per- imate applications”. In: SOSP. 2015. formance and energy characteristics. We release the EnergyMon [18] C. Imes and H. Hoffmann. “Minimizing Energy Under interface, implementations, Java and Rust bindings, and other util- Performance Constraints on Embedded Platforms: Resource ities as open source. Allocation Heuristics for Homogeneous and Single-ISA Heterogeneous Multi-Cores”. In: EWiLi. 2014. References [19] C. Imes and H. Hoffmann. “Bard: A Unified Framework [1] W. Baek and T. Chilimbi. “Green: A Framework for Sup- for Managing Soft Timing and Power Constraints”. In: porting Energy-Conscious Programming using Controlled SAMOS. 2016. Approximation”. In: PLDI. 2010. [20] C. Imes, D. H. K. Kim, M. Maggio, and H. Hoffmann. [2] A. Banerjee, L. K. Chong, S. Chattopadhyay, and A. Roy- “POET: A Portable Approach to Minimizing Energy Un- choudhury. “Detecting Energy Bugs and Hotspots in Mo- der Soft Real-time Constraints”. In: RTAS. 2015. bile Apps”. In: FSE. 2014. [21] T. Instruments. http://www.ti.com/product/ina231. [3] L. Bergstrom, B. Anderson, M. Goregaokar, J. Matthews, [22] M. Kim, M.-O. Stehr, C. Talcott, N. Dutt, and N. Venkata- K. McAllister, J. Moffitt, and S. Sapin. “Engineering the subramanian. “xTune: A Formal Methodology for Cross- Servo Web Browser Engine using Rust”. In: ICSE. 2016. layer Tuning of Mobile Embedded Systems”. In: ACM [4] C. Bienia, S. Kumar, J. P. Singh, and K. Li. “The PARSEC Trans. Embed. Comput. Syst. 11.4 (Jan. 2013). Benchmark Suite: Characterization and Architectural Im- [23] C. Klein, M. Maggio, K.-E. Årzén, and F. Hernández- plications”. In: PACT. 2008. Rodriguez. “Brownout: Building more robust cloud ap- plications”. In: ICSE. 2014. [24] J. Kramer and J. Magee. “Self-Managed Systems: an Ar- [43] V. Weaver, M. Johnson, K. Kasichayanula, J. Ralph, P. chitectural Challenge”. In: FOSE. 2007. Luszczek, D. Terpstra, and S. Moore. “Measuring Energy [25] J. Laros, P. Pokorny, and D. DeBonis. “PowerInsight - and Power with PAPI”. In: ICPPW. 2012. A commodity power measurement capability”. In: IGCC. [44] W. Yuan and K. Nahrstedt. “Energy-efficient soft real-time 2013. CPU scheduling for mobile multimedia systems”. In: SOSP. [26] B. C. Lee and D. M. Brooks. “Accurate and Efficient Re- 2003. gression Modeling for Microarchitectural Performance and [45] H. Zhang and H. Hoffmann. “Maximizing Performance Power Prediction”. In: ASPLOS. 2006. Under a Power Cap: A Comparison of Hardware, Soft- [27] N. D. Matsakis and F. S. Klock II. “The Rust Language”. ware, and Hybrid Techniques”. In: ASPLOS. 2016. In: HILT. 2014. [46] L. Zhang, B. Tiwana, Z. Qian, Z. Wang, R. P. Dick, Z. [28] J. D. McCalpin. “Memory Bandwidth and Machine Bal- M. Mao, and L. Yang. “Accurate Online Power Estima- ance in Current High Performance Computers”. In: IEEE tion and Automatic Battery Behavior Based Power Model TCCA Newsletter (1995). Generation for Smartphones”. In: CODES/ISSS. 2010. [29] D. McIntire, T. Stathopoulos, and W. Kaiser. “Etop: Sen- sor Network Application Energy Profiling on the LEAP2 Platform”. In: IPSN. 2007. [30] G. Metri, A. Agrawal, R. Peri, M. Brockmeyer, and W. Shi. “A simplistic way for power profiling of mobile de- vices”. In: ICEAC. 2012. [31] S. Mohapatra, R. Cornea, H. Oh, K. Lee, M. Kim, N. Dutt, R. Gupta, A. Nicolau, S. Shukla, and N. Venkatasubra- manian. “A cross-layer approach for power-performance optimization in distributed mobile systems”. In: IPDPS. 2005. [32] ODROID Smart Power. http : / / ameridroid . com / products/smart-power. [33] A. Pathak, Y. C. Hu, and M. Zhang. “Where is the Energy Spent Inside My App?: Fine Grained Energy Accounting on Smartphones with Eprof”. In: EuroSys. 2012. [34] A. Pathak, Y. C. Hu, M. Zhang, P. Bahl, and Y.-M. Wang. “Fine-grained Power Modeling for Smartphones Using Sys- tem Call Tracing”. In: EuroSys. 2011. [35] E. Rotem, A. Naveh, D. R. amd Avinash Ananthakrish- nan, and E. Weissmann. “Power management architecture of the 2nd generation Intel Core microarchitecture, for- merly codenamed Sandy Bridge”. In: Hot Chips. 2011. [36] M. Sabharwal, G. Metri, C. Huang, and A. Agrawal. “To- wards fine grain power profiling tools for SoC based mo- bile devices”. In: ICEAC. 2013. [37] M. Salehie and L. Tahvildari. “Self-adaptive software: Land- scape and research challenges”. In: ACM Trans. Auton. Adapt. Syst. 4.2 (2009). [38] Y. Shin, K. Shin, P. Kenkare, R. Kashyap, H.-J. Lee, D. Seo, B. Millar, Y.Kwon, R. Iyengar, M.-S. Kim, A. Chowd- hury, S.-I. Bae, I. Hon, W. Jeong, A. Lindner, U. Cho, K. Hawkins, J. Son, and S. Hwang. “28nm High-K Metal- Gate Heterogeneous Quad-Core CPUs for High-Performance and Energy-Efficient Mobile Application Processor”. In: ISSCC. 2013. [39] K. Singh, M. Bhadauria, and S. A. McKee. “Real Time Power Estimation and Thread Scheduling via Performance Counters”. In: SIGARCH Comput. Archit. News 37.2 (July 2009), pp. 46–55. [40] J. Sorber, A. Kostadinov, M. Garber, M. Brennan, M. D. Corner, and E. D. Berger. “Eon: a language and runtime system for perpetual systems”. In: SenSys. 2007. [41] V. Vardhan, W. Yuan, A. F. H. III, S. V. Adve, R. Kravets, K. Nahrstedt, D. G. Sachs, and D. L. Jones. “GRACE-2: integrating fine-grained application adaptation with global adaptation for saving energy”. In: IJES 4.2 (2009). [42] WattsUp .Net Meter. http : / / www . wattsupmeters . com/.