Atomicity and Visibility in Tiny Embedded Systems John Regehr Nathan Cooprider David Gay University of Utah, School of Computing Intel Research Berkeley {regehr, coop}@cs.utah.edu [email protected] Abstract bool flag = false; // interrupt handler Visibility is a property of a programming language’s void __vector_5 (void) memory model that determines when values stored by { one concurrent computation become visible to other atomic flag = true; computations. Our work exploits the insight that in } nesC, a C-like language with explicit atomicity, the traditional way of ensuring timely visibility—volatile void wait_for_interrupt(void) variables—can be entirely avoided. This is advan- { tageous because the volatile qualifier is a notorious bool done = false; do { source of programming errors and misunderstandings. atomic if (!flag) done = true; Furthermore, the volatile qualifier hurts performance } while (!done); by inhibiting many more optimizations than are neces- } sary to ensure visibility. In this paper we extend the semantics of nesC’s atomic statements to include a visi- bility guarantee, we show two ways that these semantics Figure 1. nesC code containing a visibility er- can be implemented, and we also show that our better ror implementation has no drawbacks in terms of resource usage. Until now, nesC has left it to programmers to en- 1. Introduction sure visibility by judicious use of C’s volatile qual- ifier. In this paper we investigate the idea of strength- ening nesC’s memory model such that the semantics of The nesC [5] language is a C dialect designed for pro- atomic are augmented with a visibility guarantee. This gramming embedded wireless sensor network nodes. It change results in a novel design point for lightweight supports safe concurrent programming using an atomic atomicity in a C-like language that reduces program- construct. This construct guarantees that a statement “is mers’ effort and opportunities to make subtle errors. We executed ‘as-if’ no other computation occurred simul- present, and evaluate the performance and code size im- taneously.” These semantics do not address visibility, pact of, two implementations of the new semantics. which Alexandrescu et al. [1] define as addressing the question: “Under what conditions will the effects of a write action by one thread be seen by a read from an- 2. Problems with volatile other thread?” In situations where a computation spins waiting for the result of another computation, visibility In C and nesC, a volatile variable is one where every errors lead to deadlocks. In other situations data corrup- source-level read or write is guaranteed to correspond tion is possible. to a load from or store to a physical memory location. Atomicity on uniprocessor embedded systems is Furthermore, the compiler must not reorder these loads easily implemented by disabling interrupts. This im- and stores. In effect, volatile variables are exempt from plementation is extremely lightweight compared to soft- most compiler optimizations. Volatiles are used to im- ware transactional memory, but it lacks a visibility guar- plement communication between concurrent flows and antee. Since small uniprocessor embedded systems do also to ensure that hardware registers are accessed prop- not have caches, compiler optimizations are the only erly. issue that must be considered when making visibility Figure 1 shows nesC code that is unlikely to guarantees. work. When this code is compiled for two pop- ular sensor network platforms, Mica2 and TelosB, current code. Asynchronous variables may be modi- wait for interrupt spins on a register instead of on fied by concurrent code, but only inside atomic sections. the actual flag variable. The problem would not have Racing variables are those that may be concurrently occurred if the developer had declared flag as volatile. modified from outside of an atomic section. Of these Our experience is that developers do not do a good categories, we are interested only in asynchronous vari- job of declaring variables as volatile. This problem is ables. Synchronous variables require no visibility guar- common enough to appear at the top of the “frequently antee, and racing variables have platform-dependent be- asked question” list for the AVR port of the GNU C havior. Library [2]. Similarly, a simple inspection of the code We propose updating the semantics of nesC’s base of TinyOS 1.x [6], an operating system for wire- atomic construct, so it guarantees that a statement less sensor network nodes written in nesC, shows that volatile is used to declare 18 variables in 157k lines of ...is executed ‘as-if’ no other computation oc- code. Clearly, most shared data is not declared volatile. curred simultaneously, and furthermore any The data in Table 1, described in more detail in Sec- values stored to asynchronous variables from tion 5, support this claim. Part of the problem is that inside an atomic statement are visible inside there are a variety of situations in which code may work all subsequent atomic statements. No guaran- even when a shared variable is not declared volatile. For tee is made about asynchronous variables ac- example, a register may not be available in which to cessed outside of atomic sections (racing vari- cache a shared variable. Similarly, the scope in which a ables). shared variable is allocated to a register may end in time for a value to be pushed to RAM before it is needed. 4. Implementing Visibility Our sense is that the latter case is most common in TinyOS applications, where an important synchroniza- The nesC compiler produces a single C file as output, tion idiom is for an interrupt handler to post a task that which is then passed to a regular C compiler. Atomic runs at a later time. The end of the interrupt handler statements are compiled by calling target-specific in- serves as an implicit memory barrier, forcing the results trinsic functions at the start and end of each atomic sec- of its computations to be flushed to RAM. However, it tion. We extended this compiler with two implementa- is clear that in many cases, implicit visibility is fragile, tions for our visibility semantics; both are straightfor- and code making improper use of volatile can be bro- ward. ken by a change in compiler version, compiler flags, or The first is to mark all asynchronous variables as phase of the moon. volatile in the generated C code, and any pointers used Another problem with volatile variables is that to access them as pointer-to-volatile. This is overkill they are overkill: they affect all accesses to a vari- but it is clearly correct: all loads and stores to these able, whereas visibility only requires that the final ac- variables are forced to go to RAM. cess in an atomic section be forced to RAM. Since The second implementation is to strengthen the they are such a blunt tool for inhibiting compiler opti- nesC intrinsic functions that start and end atomic sec- mizations, declaring too many variables as volatile hurts tions (which until now simply disabled and enabled in- performance. Developers of resource constrained sys- terrupts) with compiler-level memory barriers. These tems tend to use a trial-and-error approach to declaring barriers stop the compiler from retaining copies of vari- volatiles: initially few variables are volatile, and then ables in registers across a barrier: values are flushed to more volatile qualifiers are added in order to fix bugs RAM before the barrier and reloaded afterwards. Again that are observed during testing. we believe this implementation to be correct, even though the analogous use of memory barriers around 3. Visibility Semantics for nesC mutex lock and unlock operations is not sufficient, as pointed out by Boehm [3]. We discuss this issue further nesC encourages the use of a static programming model under related work. Our second transformation enables with no dynamic memory allocation. Thus, in the rest a minor performance optimization where the volatile of this paper we will discuss concurrency, atomicity and qualifier can be removed from asynchronous variables visibility in terms of variables rather than more general that have already been declared as volatile. memory objects. While memory barriers cannot be portably speci- fied,1 they are supported by many C compilers. For in- nesC’s concurrency model divides global variables into three categories: synchronous, asynchronous, and 1The common idiom of calling an external function would not racing. Synchronous variables are not modified by con- seem sufficient to prevent optimization of static global variables. 2 application (loc) sync async race vol memory barriers strictly reduce the number of optimiza- blinktask (1871) 18 4 0 3 tions that the compiler can apply, and gcc is invoked osc (5587) 34 25 7 3 with its optimize-for-size flag. genericbase (6941) 51 57 1 4 Figure 3 shows the impact of our transformations rfmtoleds (7401) 57 46 7 4 on application duty cycles. Again the baseline is the cnttoledsandrfm (7810) 68 50 8 4 duty cycle of the executable generated by the unmodi- micahwverify (8131) 68 50 8 4 fied nesC toolchain. Adding the visibility guarantee has sensetorfm (8259) 67 54 8 4 no clear effect in either direction on application perfor- testtimestamping (8308) 60 57 22 5 mance, and the differences are at most a few percent. surge (10657) 117 53 10 4 ident (11146) 101 50 8 4 6. Related Work hfs (12284) 118 51 17 4 Boehm [3] discusses why C cannot reliably be used for threaded (concurrent) programming without compiler Table 1. Kinds of variables in the TinyOS ap- knowledge of threads. In particular, he shows that the plications that we use to evaluate our work. common approach of ensuring that mutex lock and un- The volatility of a variable is independent of lock operations contain compiler-level memory barri- whether it is synchronous, asynchronous, or ers is not sufficient to ensure correctness.