A Tutorial on Performance Evaluation and Validation Methodology for Low-Power and Lossy Networks

A Tutorial on Performance Evaluation and Validation Methodology for Low-Power and Lossy Networks Kosmas Kritsis, Georgios Papadopoulos, Antoine Gallais, Periklis Chatzimisios, Fabrice Theoleyre

To cite this version:

Kosmas Kritsis, Georgios Papadopoulos, Antoine Gallais, Periklis Chatzimisios, Fabrice Theoleyre. A Tutorial on Performance Evaluation and Validation Methodology for Low-Power and Lossy Networks. Communications Surveys and Tutorials, IEEE Communications Society, Institute of Electrical and Electronics Engineers, 2018, 20 (3), pp.1799 - 1825. 10.1109/COMST.2018.2820810. hal-01886690

HAL Id: hal-01886690 https://hal.archives-ouvertes.fr/hal-01886690 Submitted on 23 Apr 2020

HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. 1

A Tutorial on Performance Evaluation and Validation Methodology for Low-Power and Lossy Networks Kosmas Kritsis, Georgios Z. Papadopoulos, Member, IEEE, Antoine Gallais, Periklis Chatzimisios, Senior Member, IEEE, and Fabrice Theoleyre,´ Senior Member, IEEE,

Abstract—Envisioned communication densities in Internet of may be used for counting the number of vehicles, such to Things (IoT) applications are increasing continuously. Because control optimally the street traffic lights and to reduce the these wireless devices are often battery powered, we need waiting time [3]. specific energy efficient (low-power) solutions. Moreover, these smart objects use low-cost hardware with possibly weak links, This type of sensor networks, due to their embedded nature, leading to a lossy network. Once deployed, these Low-power often requires to operate with limited power, constrained mem- Lossy Networks (LLNs) are intended to collect the expected ory and processing resources. Therefore, they require specific measurements, handle transient faults and topology changes, approaches to make the wireless network energy efficient. To etc. Consequently, validation and verification during the protocol do so, the wireless devices have to implement protocols to development are a matter of prime importance. A large range of theoretical or practical tools are available for performance reduce their energy consumption, e.g., by turning off their evaluation. A theoretical analysis may demonstrate that the radio interface for most of the time. performance guarantees are respected, while simulations or Furthermore, multiple environmental factors interfere with experiments aim on estimating the behaviour of a set of protocols the system, while adding extra noise and promoting high within real-world scenarios. In this article, we review the various variances in the communication links. Therefore, the Packet parameters that should be taken into account during such a performance evaluation. Our primary purpose is to provide Error Rate (PER) may be high for some of the radio links, a tutorial that specifies guidelines for conducting performance and requires to design robust algorithms and protocols. In evaluation campaigns of network protocols in LLNs. We detail the particular, we have to make the system reliable even if it general approach adopted in order to evaluate the performance depends on unreliable radio links. The wireless infrastructure of layer 2 and 3 protocols in LLNs. Furthermore, we also specify has to deliver most of the packets to their destinations, the methodology that should be adopted during the performance evaluation, while reviewing the numerous models and tools that without duplicating them (energy efficiency), and by handling are available to the research community. transparently the packets losses at the link layer. Actuators may also be part of the wireless infrastructure Index Terms—Low-power Lossy Networks; Internet of Things; Protocols; Algorithms; Performance Evaluation; Validation; Ex- (e.g., the heating system in smart buildings [4]). To avoid periments; Simulation; Testbeds; Models any ambiguity, we adopt here the Internet Engineering Task Force (IETF) terminology, which designates a Wireless Sensor Network as a Low-power Lossy Network (LLN). A LLN I.INTRODUCTION comprises routers, sensors and actuators, which use wireless After many decades of research, wireless networks have transmissions to exchange packets, possibly via multiple hops. evolved from Ad Hoc Wi-Fi technology to low-power and The great importance of LLNs becomes clear, considering large-scale Wireless Sensor Networks (WSNs). This miniatur- how they have affected the emergence of modern IoT applica- ization extended de facto their usage, and enabled the design tions, where information must be shared intact across different of radically new applications that follow the modern concept platforms, while enhancing the data from a distributed network of Internet of Things (IoT). For instance, Smart Cities are of sensors and actuators. expected to rely heavily on a myriad of devices, able to Globally, the IoT paradigm encompasses a large variety of measure their surrounding environment and take decisions in devices connected to the Internet, including amongst others: (i) order to manage the city efficiently [1], [2]. Moreover, sensors multimedia objects (e.g., a video camera) that require a high throughput and, thus, rendering them unable to save energy, (ii) K.Kritsis is with the Research and Innovation Center Athen (ATHENA Radio-Frequency Identification System (RFID) and tags that RC), Institute for Language and Speech Processing (ILSP), 15125 Marousi, Athens, Greece (e-mail: [email protected]). enable the tracking of any object, typically within a supply G. Z. Papadopoulos is with the IRISA, IMT Atlantique, 35510, Cesson- chain context, (iii) small sensors and actuators, which are Sevign´ e,´ France (e-mail: [email protected]). mostly battery-powered wireless hardware platforms. A. Gallais and F. Theoleyre´ are with the ICube Laboratory, CNRS / University of Strasbourg, 67412 Illkirch, France, (e-mail: The number of embedded devices involved in a typical {gallais,theoleyre}@unistra.fr). LLN scenario varies from tens to thousands of nodes, which P. Chatzimisios is with the CSSN Research Lab, Department of Informatics, introduces further density as well as scalability issues. Alexander TEI of Thessaloniki (ATEITHE), 57400 Thessaloniki, Greece and with the Department of Computing & Informatics, Bournemouth University, Validation and verification during the protocol develop- BH12 5BB, United Kingdom (e-mail: [email protected]). ment has become a matter of prime importance. Envisioned 2

solutions must be intensively tested before being deployed Application home intrusion HVAC in a real-world environment, by employing either simula- (e.g. periodic Traffic, Event-based) automation detection tors, emulators, or even conducting experimentations over real physical testbeds [5]. More specifically, simulators have TRANSPORT CoAP allowed users to exploit available models (e.g., link quality, (end-to-end, security) radio propagation, medium interference, topologies) in order to anticipate the behaviour of their proposals under real-life RPL conditions. Some open testbeds have also emerged, providing access to pre-deployed low-power devices and, thus, a certain NETWORK IPv6 (e.g. routing, IP address, self- level of repeatable experimental setups [6]. However, a great configuration) 6LoWPAN number of issues, such as weather conditions, interference from other wireless technologies or even obstacles, may have MAC IEEE 802.15.4-2015 an impact on the radio links among the sensor nodes. Thus, (e.g. medium access, PHY addr) (e.g. TSCH, DSME, LLDN) before proceeding to the experimental evaluation of a network Networking protocol or an algorithm, apprehending the wireless links and PHY the overall topology is an essential step [7]. It is also manda- IEEE 802.15.4 PHY (e.g. modulation, error code) tory to consider the correct energy models since preliminary results must help end users to get a flavour of the ensued Fig. 1: Typical stack of protocols in the Internet of very energy consumption. We here detail numerous tools that shall low-power Things. ease such investigations. In order to initially validate their concepts or models, researchers must be advised of the numerous constraints that the setup (e.g., reproducibility). We then describe the various arise in order to define the scientific methodology that will lead characteristics that may be present in various radio topologies to pertinent experimental results. Theoretical and numerical (SectionIV), as well as we detail the traffic characterization analysis, along with packet level simulation and emulation (SectionV) and the different energy models (SectionVI). We appear as essential pieces of the long-term validation of a then present most of the existing tools that may be used during whole solution. a performance evaluation campaign (Section VII). Finally, In this tutorial, we review in depth various parameters Section VIII discusses the current related challenges, before that should be taken into account during such performance giving some concluding remarks in SectionIX. evaluation. We especially focus on the main networking layers of the communication stack (e.g., Medium Access Control (MAC), routing). Techniques related to the physical layer are II.PERFORMANCE EVALUATION LIFE-CIRCLE out of the scope of this document, since the performance A LLN is a combination of software, network and embedded evaluation often uses specific hardware for the experiments, or engineering. These fields are well defined and therefore, devel- it is conducted based on specific models. The link and network opers should be aware of the currently employed technologies layers are rather based on protocols and algorithms, and form and methods in the previously mentioned domains in order to a specific, distinct research subject, with different tools. efficiently design new solutions. Our primary purpose is to provide a tutorial both with basic However, some of these practices should be modified in and enhanced guidelines for evaluating the performance of order to fit the specifications of wireless LLNs. There are network protocols in LLNs. various methodologies that can be adapted, concerning the The contributions of this paper are fourfold: development life-cycle of applications that incorporate LLN 1) We detail the general approach for evaluating the per- technologies. formance of networking protocols for LLNs; Most LLN solutions are built upon a modular stack of 2) We detail the methodology to be adopted, what should protocols (Figure1): the most accurate protocols / standards be measured (i.e., metrics) and what should be taken may be chosen before the deployment, based on the require- into account when evaluating the performance; ments of the targeted specific application. To reduce the cost 3) We provide a comprehensive view of the different of deployment, several applications (e.g., intrusion detection, models (radio communication, interference, traffic) and heating, ventilating and air-conditioning), may operate on their typical characteristics and limitations. In particular, top of the same wireless infrastructure [8]. Transport layer we provide key use cases in which the protocols should solutions dedicated to LLNs include CoAP and MQTT-SN, be evaluated; which both rely on publish/subscribe mechanisms, while using 4) Finally, we detail the existing tools, from theory to UDP as the underlying protocol [9], [10]. experimentations, and their typical utilization. Agile methodologies are particularly appropriate for devel- The paper is organized as follows. In SectionII, we expose oping IoT research applications. Short and iterative develop- the suggested methodology that allows a complete and relevant ment cycles allow researchers to quickly correct an error in performance evaluation. Section III details the main require- the conception or in their assumptions [11], [12]. ments of a consistent performance evaluation process, ranging Our approach considers that research in LLNs follows the the monitored parameters that are related to the accuracy of life-cycle as illustrated in Figure2. A certain number of steps 3

models (PHY layer): interference, packet losses, isotropic environment, etc.

Inputs Inputs Inputs models Theoretical Analysis Numerical Analsis Network Simulation adaptation idea and/or Emulation Proof of convergence, typical input values, distributions, Bounds, Correctness monte-carlo simulations Packet level simulations

iterations

Deployments Experimental Validation validation Real Life Applications Research testbeds, prototyping

Fig. 2: Typical long-term Validation of a whole solution. are required to evaluate the performance of a protocol, an analysis uses different values as input to study practically what algorithm or a whole networking stack. For instance, switching the obtained results would be. too quickly from the idea to protocol experiments may be With Monte-Carlo simulations, we generate a set of different prejudicial [105], [14]. input values, using a given distribution [17]. For instance, a network should behave similarly, with a different random A. Theoretical Analysis location of each source node. Monte-Carlo simulations may help to estimate the distribution of the different values (e.g., We often bootstrap a performance evaluation by analysing delay, packet losses) when the models are too complicated to theoretically the proposed algorithm. While it is considered to obtain the result analytically. Basically, the characteristics of be a preliminary step, it proves that the algorithm convergence the sampled random values are close to the real distribution was correctly designed. Typically, we may prove the following main properties: of the values when the number of samples is very large (law of large numbers). • Convergence: An algorithm is designed to get various Monte-Carlo simulations may use any mathematical tool to input and produce specific output. The convergence may pick random samples. Soua et al. use [18] GNU Octave to not be trivial in some cases, and a formal proof is evaluate the robustness of a scheduling algorithm for a LLN sometimes required; scenario, while Palattella et al. [19] use Python scripts and • Self-stabilization: Whatever the initial state is, the algo- Keshavarzian et al. use Matlab [20] for the same purpose. rithm must converge to a legal state in a finite number of steps [15]. This self-stabilization property is vital for self- C. Simulation and/or Emulation healing, while the network is able to correct the transient In this subsection, we first describe the tools that simulate a errors autonomously; whole network (e.g., energy consumption, radio propagation, • Complexity: The algorithmic complexity is of prime protocols). We then will detail the approach to emulate a whole importance for wireless LLNs, since they are composed device, while simulating the radio environment only. of embedded, low-cost nodes. The complexity is tightly 1) Network Simulation: Simulation has been widely used coupled with the resource allocation (e.g., memory, num- for a few decades to validate the behaviour of a protocol. ber of control messages) and hence with the energy Discrete event simulation is the most popular tool, where a consumption; collection of events (packet reception, time-out) are handled • Approximation: Most of the interesting problems are chronologically by the simulator. Typically, a pseudorandom NP-hard in LLNs, thus requiring to propose heuristic number generator is used to mimic uncertainty when prob- approaches, so that the problem is still tractable by small abilities have to be respected (e.g., the Packet Error Rate devices with limited capabilities. In particular, we may for a transmission). Thus, simulations help to improve the prove that an algorithm is an α-approximation, i.e., the reproducibility: providing the seed, the simulator version, and algorithm leads to a solution which differs by α from the the code is sufficient to re-obtain the same results. optimal one [16]; For LLNs, the simulations have to integrate PHY models, • Lower and upper bounds: When proposing an heuristic including the radio propagation. Unfortunately, these radio solution, it is always relevant to provide upper and lower characteristics are very complicated to capture [21], since bounds. While the average case might be of interest, they may exhibit very different behaviour depending on many focusing on the worst case helps to answer which are properties (e.g., indoor vs outdoor, ISM band, modulation, the guarantees that could be provided by the heuristic. urban vs. rural). Similarly, most of the simulators are not hardware specific, and do not model the fine-grained behaviour B. Numerical Analysis of each device (i.e., instruction level). For instance, most Obtaining mathematical closed form expressions is prac- simulators do not model clock drifts while it may impact very tically complicated for most of the problems. A numerical significantly the performance of a network [22]. 4

A large variety of network simulators exists in the literature, (cf. TableI). Hereafter, we list some common criteria for each either as open source (ns-2 [23], ns-3 [24], OMNeT++ [25]) network layer as follows. or proprietary (GloMoSim [26], Riverbed Modeler [27], Qual- 1) Common metrics: Some metrics are not relative to a par- net [28]). Their respective interests and limits are detailed in ticular communication layer and should always be monitored Section VII-B. in wireless LLN scenarios. Unfortunately, no common API or framework exist for the a) Energy consumption: First of all, a LLN aims at different simulators. This means that a network stack evaluated integrating low-power nodes using batteries. Thus, the radio on a given simulator cannot be transferred directly to another chipset has to be turned off to save energy [34]. one. All the protocols have to be rewritten, jeopardizing the We may measure the average duty cycle ratio that represents reproducibility. a first approximation of the network energy consumption. Because reproducibility is now a major concern in the More precisely: networking community [29], [6], emulation appears as a very T DCR = active (1) relevant solution for a performance evaluation. T 2) Emulation: Emulation uses a very fine-grained model of with T being the considered time-interval and Tactive the time the device to consider, and is consequently hardware specific. duration which the radio chipset of the node is set to active. Actually, most of the emulators for wireless networks comprise The popular CC2420 chipset typically consumes 18.8mA in two parts: RX mode, between 8.5 and 17.4mA in TX mode, 0.4mA 1) the node: the emulator mimics the exact instruction- in idle mode, and 0.02µA when sleeping. If we neglect the set processing. This way, we can also consider memory difference between the idle, TX and RX modes, the duty cycle and CPU constraints. Energy consumption tends to be ratio can be a good representation of the energy consumption more and more accurately modelled if some criteria are of a network node. respected [30]. b) Network Lifetime: We may measure the network life- 2) the PHY layer: most of the emulators simulate the radio time, which could be defined as [35]: propagation. We can use here the same PHY models as • First death: When the first node runs out of energy; for simulators. • Last death: When the last node dies [36]. However, such Thus, the same implementation can be used to execute the definition assumes implicitly a single hop topology, with code on both the real and the emulated hardware. It greatly all the sensors being equivalent (i.e., with no heterogene- reduces the development costs. ity); The code is often specific to the hardware, at least for the • Connectivity: The size of the largest connected compo- lower layers of the communication stack. Fortunately, there are nent exceeds a threshold value. However, this metric is some very popular platforms which are frequently employed in extremely dependent on the application, since it assumes LLN scenarios (such as the TelosB motes [31]), thus creating that the sensors are redundant, which may not hold in a de facto standard for experimenting a set of protocols. practice; Emulation tools such as Cooja [32] and OpenWSN [33] are • Coverage and Connectivity: A collection of regions now very popular and concentrate most of the developments. have to be monitored and, thus, a sensor is able to By using a limited set of emulators, the research community measure a given phenomenon in a given geographic area, also encourages code re-use, and reappropriation. i.e., it actually covers this area. Each area must be covered III.METHODOLOGY by at least k sensors and these sensors are able to send In this Section, we propose a methodology to evaluate their measurements to the central sink (i.e., connectivity). the protocol performance. While this could be considered The energy consumption and the network lifetime are as a straightforward problem, many research papers still do closely related metrics. However, most of the researchers focus not present their simulation or experimental results in an on the primer because it is more convenient to measure, accurate and detailed fashion. In this article, we focus on while on the other hand, measuring accurately its lifetime, protocols of layers 2 and 3, considering that physical layer the network has to operate and get monitored for very long is a given parameter of simulations and experiments while durations. upper layers strongly depend on the target application. In our c) Overhead: A chatty protocol needs to exchange many humble opinion, MAC and routing protocols thus, constitute packets and consequently it consumes both bandwidth and the backbone of the communication stack of LLN devices. energy. Therefore, it is important to measure the network over- Consequently, hereafter, we present the different metrics and head, expressed as the quantity of control packets generated guidelines that appear specific to the thorough study of these by all the nodes. two aforementioned layers. We also highlight the importance This overhead may be measured in: of conducting reproducible, robust and consistent performance • packets per second, when the cost to access the medium is evaluations. high (e.g., long preamble sampling, the data packet length being small compared with the sum of the preamble and A. What should we measure? the data frame); The first thing a researcher should wonder when evaluating a • bits per second, when the transmitter and the receivers protocol / algorithm concerns the target metrics to be measured have to stay awake only during the packet’s transmission; 5

Layer Metric Section Deﬁnition Energy Consumption III-A1a Quantity of energy consumed by all the nodes Transversal Network Lifetime III-A1b Time before the network stops doing its job Overhead III-A1c Amount of control packets (bits or packets per second) Delay III-A2a Medium access delay, comprising also the retransmissions MAC Packet Delivery Ratio III-A2b Ratio of packets correctly received by the receivers Fairness III-A2c Jain Index Duplicates III-A2d Number of packets duplicated (i.e. forwarded in multiple exemplars) by the MAC layer Delay III-A3a End-to-end delay, considering both medium access and buffer- Routing ing delays Packet Delivery Ratio III-A3b End-to-end packet delivery ratio from the source to the ﬁnal destination

TABLE I: Deﬁnition of the different metrics.

Some researchers have proposed to divide the number of buffer and its reception by the next hop. However, we need control packets by the number of packets received by the to clarify that we do not refer to the buffering delay, since it sink in order to measure network efﬁciency [37]. We consider corresponds rather to a problem concerning higher layers. that this corresponds to two different criteria that should be We may consider the transmission successful as soon as measured separately. the data frame is received by the next hop. Unfortunately, the 2) Medium Access Control (MAC): Medium access repre- transmitter is not certain if its transmission was successful, be- sents a key challenge in wireless LLNs. MAC layer is in charge cause the communication channel is half-duplex and the radio of determining when nodes should transmit, receive, listen link may be unreliable. Thus, the latency is often approximated or simply remain idle. It thus handles all operations related by the time difference between the data generation and the to the main source of energy consumption, which is packet reception of the corresponding received acknowledgement. transmission and reception [38]. In particular, a node must If we consider broadcast packets, the problem is more turn its radio off most of the time because it is the only way complicated since we have to consider the following aspects: to save a signiﬁcant amount of energy in radio networks [34]. • Time distribution: A broadcast packet may not be The lowest this duty cycle ratio is, the larger the network received at the same time since some MAC layers im- lifetime will eventually be. plement a duty cycle approach and, thus, a copy of the Similarly, the MAC protocol must solve the contention same packet may be received by neighbours according to amongst different interfering transmitters. This approach often their sampling frequency; provides a trade-off between reducing the medium access de- • Reliability: Only some neighbours may have received lay, while increasing the collision probability, and conversely. the broadcast packet. In this case, for instance, we could In the literature, several MAC strategies have been proposed compute the time at which 75% of the neighbours have for LLNs [39] (Figure3): correctly received the broadcast packet. • Preamble sampling: The transmitter sends a long pream- b) Reliability: Packets may be lost because of collisions ble, and the receiver has to turn its radio on periodically or link unreliability. Thus, it is mandatory to measure the to detect this preamble. Then, the receiver has to stay Packet Delivery Ratio (PDR) as: awake in order to receive the data packet, which follows N the preamble [40]. PDR = rx (2) Contiki-MAC [41] represents a widely used preamble Ntx

sampling protocol, which exploits several optimizations with Nrx the number of received packets, and Ntx the number for reducing the energy consumption. of transmitted packets. This approach is considered as a • Synchronous protocols: The receiver and the transmitter layer 2 metric; if a packet is received after two consecutive must agree on a common schedule to wake-up syn- 1 retransmissions, the PDR is considered equal to 3 . Since these chronously so as to be able to exchange frames. The retransmissions are hidden to upper layers, measuring the PDR transmitter maintains a guard-time before its transmis- at layer 2 is of primary importance. More retransmissions also sions for compensating possible clock errors [42], [43]. In mean a higher energy and bandwidth consumption. its Time-Synchronized Channel Hopping (TSCH) mode, The throughput consists in the PDR of a given flow multi- IEEE 802.15.4 [44] uses a Time Division Multiple Access plied by the rate of the corresponding flow. Since it does not (TDMA) matrix paired with slow channel hopping to reflect the actual pressure on the network (e.g., some packets combat external interference; may be dropped at intermediary nodes), this metric is less • Wake-up radio: Each node must have two interfaces. relevant for LLNs. The (always on) wake-up radio is in charge of waking- It is also important to distinguish the packet losses between up the receiver [45]. The second radio provides higher those that are caused by collisions, and those that emerged bit rates and exchanges data packets. from low quality radio links. In simulations, we can measure a) Medium Access Delay: Medium access delay corre- accurately the number of packets that overlap at some receivers sponds to the time separating the frame dequeued from the (i.e., the collisions). In particular, the centralized scheduler 6

data preamble data T T

ack ack R (TDMA) R sampling preamble synchronous

data NIC 1 rx mode sleeping mode T NIC 2 tx mode listenning wake-up signal ack NIC 1 R

Wake-Up Radio NIC 2 Fig. 3: MAC approaches for low-power and lossy networks.

R data receive the acknowledgement, it schedules a retransmission, leading to a duplicated packet. Mechanisms such as sequence ack R1 numbers help to limit the impact of such problems [51]. lost Furthermore, opportunistic forwarding in the MAC layer ack R2 exploits the broadcast nature of the radio transmissions in order to alleviate unreliability problems [52]. A single packet is transmitted using the anycast mode to a list of (ordered) T rx mode sleeping mode next hops (see Figure4). R1 correctly receives the packet and acknowledges it. The ack is received by T but not by tx mode listenning R1 R2 R2: the second receiver will also consequently send an ack and will enqueue the packet. Since in this case duplicates are Fig. 4: General of duplicates with an opportunistic MAC. generated in different nodes (R1 and R2), their detection is more challenging to be achieved. computes the signal strength of all frames in every receiver. We usually measure the duplicate ratio (ratio of the number Unfortunately, isolating collisions in experiments is a much of packets received by the destination and the number of more complicated task. Giustiniano et al. [46] proposed to packets really generated by the source). We consider the classify the source of packet loss for IEEE 802.11 networks by number of end-to-end duplicates because it captures both the measuring the loss. However, this technique has to be adapted negative (higher overhead) and the positive impacts (higher to each considered MAC layer. reliability). Finally, we may also isolate the source of external in- 3) Routing: Routing aims at deciding which node should terference that results in packet loss. Techniques such as be used to relay the packets toward a given destination [53]. SONIC [47] classify the source of interference (e.g., from The main following approaches exist in the literature: WiFi or Bluetooth networks) based on the variation of the • Proactive: Routes are constructed a priori toward the signal strength and payload corruption. destination, thus, all nodes in the network are aware of c) Fairness: Especially in wireless LLNs, we have to the routes to any destination at any time. Therefore, since guarantee a fair behaviour, which means that each transmitter all routes are stored in the routing tables, a node may should receive the same amount of radio resources. We often transmit its data packets to the destination at very low utilize the Jain Index [48] in order to measure fairness as delay. To maintain the routing tables constantly updated, follows: periodic routing-related control packets are required to be Pn 2 transmitted. ( i=1 xi) JainIndex(x1, .., xn) = (3) The Routing Protocol for LLNs (RPL) represents the n ∗ Pn x2 i=1 i standardized proactive routing protocol for LLNs, speci- where xi is the considered performance metric (i.e. PDR, delay fied by the IETF ROLL WG [54]. It is a distance vector or throughput) achieved by the flow / source i. routing protocol and it is defined as link-layer agnostic. A Jain Index equal to 1 means a perfectly fair solution. The Thus, RPL may operate over wireless or PLC networks. lower is the Jain Index, the more unfair the solution behaves. RPL builds a Destination Oriented Directed Acyclic d) Duplicates: Since transmissions are unreliable, data Graph (DODAG) whereby each node chooses one or packets may be correctly received and the corresponding ac- more parent(s). Each node compute its rank representing knowledgement may be lost (due to an unreliable link) or even its virtual distance from the root. More precisely, the node may collide (due to the hidden terminal problem [66]). Asym- uses an Objective Function (OF) to derive its rank from metrical links [50] tend to exacerbate the loss probability of the rank and the link quality of its best neighbour (its this acknowledgement packet. When the transmitter does not parent); 7

+ + • Reactive: Routes are built on-demand, when the source + + + + + has packets to transmit, which means that it is not + + + + + + + necessary to maintain a route if there is no traffic. + + + + + + ++ + + ++ + + ++ + + + ++ ++ + + + ++++++ + +++ + + +++ + + + +++ ++ + + + + +++ +++++ + + +++ +++++ + + + + + +++++ + + + + ++ +++++ ++ + ++ + ++ +++ + ++ + + + + + + + + + + + + + ++ + +++++++ + + ++ +++ + + + + + +++ + Thus, reactive protocols do not require routing-related ++ + +++ ++ +++ + ++ + ++ + +++ ++ ++++ + ++ + ++ + ++++++ + + + ++++ + + ++ + + ++++ ++ + ++ + ++++ + + + + + + + +++ + ++ + + + ++ + ++ ++++ + ++ + + + ++++++ ++ ++++ + ++ + + +++++++ ++ +++ +++ ++++ + ++++++ + +++ + + + + + + + + + + + + + + + + ++ + + ++ + + ++++ + + +++ +++ + ++++++++ + ++++ +++ + ++ + +++++++ + + + ++ + +++++++ ++ + + + + +++++ + ++ + + information to be transmitted periodically. It relies on + + + +++ + + + ++++ + ++ + + +++++ + y ++ ++ + ++ + + + +++ + ++ + + + +++ + + + + + + ++++ ++ + + ++ + ++++ ++ + + ++ ++ + + + ++ + + + + ++ + + ++ + + + +++ + + ++ ++ + +++ + ++ + + ++ + + +++ + ++ + + ++ + + + ++ +++ +++ + + + + + ++ + + + + + + ++ + + +++ + + + + +++ +++ + +++ +++ + + +++ + ++++ + a flooding mechanism to reach the destination, and to + + ++ ++++ + + + ++ + + + ++ ++++ + + + ++ + + + + + + + ++ + + ++ + + +++ + + ++ + +++ + + + ++++++ + +++ + + + ++ + + ++ + + + + + + + + ++ + + + + + + + + + +++ + + ++ + + + +++ ++ + + + +++ ++ + + + + + ++ ++++ + + ++ +++ + + ++ + ++ ++ + + + + + + + + +++ + + ++ + + + +++ + + ++ + + + +++ construct the routes. ++ + ++ + + + ++ + ++ + + + + ++ + + ++ + + ++ + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + ++ + ++ + Lightweight On-demand Ad Hoc Distance-vector Routing + + + + + + + + ++ + + + + + + + + + ++ + + + + + + + + + + ++ + Protocol Next Generation (LOADng) [55] is reactive + + + + + + routing protocol based on Ad Hoc On-Demande Distance + + + average values Vector Protocol (AODV). At its core, it floods the network scatter plot box plots with a route request, and the destination unicasts a route with conf. intervals reply after having inserted the route in its routing table; Fig. 5: Representation of experimental data: average values, • Geographic: If all the nodes are aware of their geo- scatter plot, boxplots. graphic location, they may select greedily the next hop as the neighbour closest to the destination. However, such local rule leads sometimes to a cul-de-sac, not easy to where NP os is the number of times that the system detected tackle [56]. a phenomenon (e.g. an intrusion, a location with a pollution A low-power lossy network may comprise multiple sinks, higher than a threshold, etc.), NfalseP os is the number of connected to the rest of the Internet [57]. These different sinks false positive detections (detected erroneously as true), and may be used for instance to balance the load, or to increase NfalseNe is the number of false negative detections (not the available bandwidth for each individual device. The sinks detected at all). may also be used to improve the robustness, each device We often target to achieve UnderDetectRatio ≈ 0 because having several independent routes to the Internet (through each we cannot accept to miss detecting an event, while minimiz- different sink). ing OverDetectRatio since it triggers useless actions (e.g. When considering a multihop network, we aim at focusing network reconfiguration or human intervention). on the end-to-end performance. To simplify this interpretation, we often only consider the performance in the multihop B. How should we measure? wireless part. a) L3 End-To-End Delay: The layer 3 (routing) delay We here provide some general guidelines about measuring corresponds to the multihop delay taking into account the the aforementioned metrics. following: 1) Average value: In order to obtain robust results, we • Medium access: The MAC delay (contention, possible have to verify that they are representative. Thus, we often retransmissions, and reception of the acknowledgement); run a simulation / experiment several times by utilizing the • Queueing: A packet is received by a node and stored in same input. We then present the average value x¯ for all a temporary queue before being forwarded. the experiments / simulations with the same input values. In particular, a routing protocol, which efficiently balances Alternatively, the median value is less sensitive to outliers and the load may help to reduce queuing delay. In the same provides an accurate manner to synthesize a set of different way, different queuing strategies (e.g. FIFO, LIFO, Priority experiments. Queuing) may impact the distribution of the end-to-end delay. However, we have also to appreciate the distribution of the b) End-to-End reliability: The end-to-end PDR counts different values, particularly when we compare different solu- the ratio of the number of packets received by the destinations tions. The confidence interval helps to quantify the variability and transmitted by the sources. We often neglect the incurred and to verify that a difference is significant. Let us assume loss outside the wireless part. that the values follow a normal distribution. The typical 95% High reliability is a requirement for many industrial LLN confidence interval is defined as: " # applications [58]. Thus, achieving an almost perfect PDR σ(X) σ(X) x¯ − 1.96 , x¯ + 1.96 (6) (close to 99.99%) may constitute an important requirement p|E| p|E| that needs to be satisfied. Since there is no need to receive all the measures when they where x¯ represents the average value, σ(X) is the standard are redundant, reliability may also be application specific. For deviation of the random variable and |E| is the number of instance, we consider that an intrusion detection system fulfills experiments carried out. A 95% confidence interval means that its objective if the event was detected by the system, whatever if we pick randomly an experiment, it has 95% probability to PDR is obtained. We then propose to measure both the Over perform within the computed interval (see Figure5). and Under Detection Ratios as follows: 2) Distribution and worst case: While the average value NfalseP os provides to us an overview of the average case, we should OverDetectRatio = (4) also consider the best and worst cases. NP os Boxplots are a convenient way to plot the distribution of NfalseNeg UnderDetectRatio = (5) the values as well as the minimum and maximum values. We NP os 8

lost received 35 35 lost received 30 30

25 25

20 20

15 15 Number of packets regular grid mixed: grid and random 10 10

5 Cumulative number of packets 5 device

0 0 1000200030004000500060007000800090001000011000120001300014000150001600017000180001900020000210002200023000240002500026000270002800029000300003100032000330003400030000350003600037000380003900040000410004200043000440004500046000470004800049000500005100052000530005400055000560005700058000590006000061000620006300064000650006600067000600006800069000700007100072000730007400075000760007700078000790008000081000820008300084000850008600087000880008900090000910009200093000940009500096000970009000098000 0 30,000 60,000 90,000 TimeTime (in(in ASN) ASN) relay node Fig. 6: Convergence of a distributed scheduling algorithm in a 6TiSCH stack [60].

random deployment may also use a beanplot [59] to compare two protocols, two Fig. 7: Common topologies for performance evaluation. by two. Thus, we are able to quantify visually the difference in the distribution. This visual representation is complementary to the Jain Index presented in Section III-A2c. For instance, the simulation model is assumed to remain Let us now consider Figure5 in which one boxplot repre- unchanged for the whole simulation period. Unfortunately, x sents the measurements for a given coordinate interval. Each node failure [61] or link quality changes [62] are practically box represents the median value, the first and third quartiles very common. for all these measurements. The whiskers here represent the A routing protocol may even not converge under stable minimum and maximum values. We are able to detect tenden- topologies when stochastic metrics are used [63]. Some sta- cies and to estimate more accurately the scattering. tistical estimators such as a Kalman Filter have often to be Particular attention has to be given to the outliers, i.e. used [64]. measures distant from other observations. They may mean: Thus, these time variations have to be carefully studied • Measurement error: The metric was erroneously mea- during a long time period. sured (e.g. due to inaccuracy in the sensor device); • Bug: The protocol was erroneously implemented; IV. RADIOTOPOLOGY • Algorithm: The algorithm was badly designed and it Many papers evaluate their proposition in a random topol- does not converge under certain situations; ogy, using a synthetic model to mimic the radio characteristics. • Heavy-tailed behaviour: The values do not follow a We propose here a comprehensive view of the possible key normal distribution and certain situations lead to a very scenarios and their respective limits. We make a distinction bad case. between: Depending on the situation, researchers will have to identify • Network topology: It provides a high-level view of the the exact cause(s) and if possible to patch the solution. set of motes which can directly communicate with each 3) Initialization vs. steady-state: We make a distinction other. Too many papers focus uniquely on a random between the following two steps: topology, which complicates the interpretation and which • Initialization: The node just booted and has not joined may hide the pathological situations; the network since it has to acquire an address (e.g. local, • Communication model: It describes formally the rules global), the cryptographic keys (security), a route (routing which enable a pair of nodes to communicate with each protocol) or a parent (MAC and topology control). other. We often associate the Bit Error Rate (BER) metric • Steady-state: The node has finished its configuration and to each radio link; it denotes the probability that a given can now send and receive packets. bit in a frame is erroneously received; Let us consider Figure6 that illustrates the PDR of a • Interference model: If two transmissions start simultane- 6TiSCH stack [60], with a distributed scheduling approach, ously and the receivers are close from both transmitters, during 30 minutes. The bars represent the volume of packets the signals will probably collide, wasting the radio band- generated during a time window. The violet (bottom) part width. The interference model defines formally in which depicts the packets that are never received by the sink. We case a transmission undergoes a collision. can identify the initialization period (before the slot 50, 000) and the steady-state period (i.e. the algorithm has converged). A. Network Topologies 4) Time Dependency and Stability: Surprisingly, most pro- The topology strongly impacts the performance of the tocols and algorithms are evaluated under stable conditions. network. We make a distinction between the following devices: 9

A sensor measures a physical phenomenon (e.g. temperature, to its destination. Unfortunately, this simple topology has been wind) and sends these measurements to a processing proved to perform poorly with random access [67]. entity; For medium access, we should evaluate the behavior of the An actuator has a physical action on the environment (e.g. MAC protocol in the following situations: open a door/window, control of HVAC); • Unidirectional: One of the extremities send frames to the A relay node just forwards the packet via a low-rate radio other extremities. This models well a sensor that sends interface; its measurements in the cloud via the sink; A sink (or border router, or gateway) connects the low power • Bidirectional: Both extremities exchange frames with wireless network to the Internet and is often considered each other. Indeed, communication with actuators or plugged in. mote’s reconfiguration require a bidirectional exchange All these devices are not involved similarly in the traffic to make reliable the transaction. process (see sectionV). For instance, an actuator consumes d) Heterogeneous radio link qualities: The evaluation most of the time a command while a sensor generates a should also consider heterogeneous radio link quality. In measure. Figure9, we consider a star where two radio links present Some scenarios have been identified in the literature to a large Packet Error Rate (PER); a frame has a non negligible create some specific problems. Thus, they should be studied probability to not be received correctly. in isolation, to verify that the protocols and algorithms work The MAC protocol may adopt one of the following ap- accurately in these situations. We distinguish the scenarios proaches: common to any protocol for LLNs, and those specific to a • Transmission opportunity fairness: Each transmitter has particular layer, where a particular problem may arise. the same probability to access the medium. The receiver 1) Common metrics: Some scenarios are layer-agnostic, with the poorest radio link quality will have the lowest and should be studied regardless of the protocol. bit rate; a) Regular grid: The easiest large scale topology to set- • Bandwidth fairness: Each receiver has the same goodput up consists of a regular 2D or 3D grid (Figure7). A node is (bit rate of frames correctly received). a radio neighbour with the closest nodes in the grid (with an 3) Routing protocol: At the routing layer, we should con- almost perfect link quality). This ensures that only good links sider the fairness among different flows, and its ability to are used to forward the packets. support load-balancing. b) Mixed Grid: Alternatively, the regular grid would a) Ladder (path quality): The simplest scenario consists consist of the relay nodes (i.e. they don’t generate traffic but of a ladder in which the source sends packets to the sink and just forward the packets of others) and the devices (sensors several redundant paths exist. All the radio links experience and actuators) are placed randomly in the area. In this way, we different quality (i.e. PER). Thus, the different paths provide guarantee a network-wide connectivity, while also considering different end-to-end reliability and latency. a heterogeneous deployment of the clients. We can consequently evaluate the ability of the routing c) Random: The random topologies are the most com- protocol to exploit the most efficient route. mon scenario; a fixed number of devices is randomly placed b) Ladder bi-source (load-balancing): An efficient rout- in the considered area. If we distribute a sufficient number of ing protocol should be able to balance the network load nodes, the topology is connected with a high probability [65]. for congestion avoidance. If the objective corresponds to 2) MAC layer: maximizing the network’s lifetime (see Section III-A1b), we a) Star topology: The simplest topology is the star must balance the energy consumption. More precisely, all the topology in which all the devices send their packets to a central nodes should consume an equivalent quantity of energy, while node (Figure9). We are able to evaluate the performance of minimizing the maximum energy consumption [68]. the MAC layer under the most basic scenarios, where all the With a ladder of two sources, we can validate the ability devices hear each other and the radio links are perfect (due of a routing protocol to optimize the load/energy-balancing to the fact that all devices are sufficiently close to the central property under the simplest scenario. node). c) Synchronous vs. Gradual Bootstrapping: In a syn- Throughput and fairness performance should be optimal in chronous scenario, all the nodes boot simultaneously and start this case. to execute the algorithm / protocol. While most of the perfor- b) Hidden terminal: The hidden terminal problem is mance evaluation focus on this particular scenario, its practical known for a long time in radio networks [66]. When two interest is limited. Indeed, it assumes that all the nodes are transmitters do not hear each other and send their packets first deployed and then the operator commands remotely their to the same receiver, collisions are very frequent. A modern start-up. The trigger of this start-up is never described, while it MAC layer should address this problem. actually represents a key challenge. Indeed, even the flooding Isolating this particular situation helps to focus on the of a startup control packet may be insufficient since a blind hidden terminal problem, proving that the protocol deals flooding is expensive and unreliable [69]. efficiently with this unwanted situation. It seems that the gradual bootstrapping is far more realistic. c) Line: A line represents the atomic pattern of a multi- The sinks are first deployed and then new devices are gradually hop network; packets are forwarded along a path from a source inserted in the network. This scenario models a network where 10

radio link (good) radio link (weak) …

device

ﬂow (traﬃc) star hidden terminal (bi)-line heterogeneous radio link qualities Fig. 8: Benchmarking topologies for the MAC layer.

D D radio link quality is assumed to be perfect for any radio link with a length inferior or equal to 1 unit. Let G(V,E) be a graph G with the set of vertices V and edges E. A UDG is defined by: ∀(u, v) ∈ E, d(u, v) ≤ 1 (7) ∀(u, v) ∈/ E, d(u, v) > 1 (8) where d(u, v) denotes the euclidean (geographical) distance from u to v. S S S’ A variant of the UDG consists of using a fixed radio range (6= 1). However, it just consists of rescaling the geographical ladder ladder single source bi-source (load-balancing) distance. Comments: the UDG model is very simple and is still used device data flow radio link for this purpose. However, it has been proved to largely over- estimate the performance achievable in realistic conditions. Fig. 9: Benchmarking topologies for the routing layer. Since they do not consider at all the unreliability problem (radio link quality), we consider that the UDG model is inappropriate, and should not be used for a performance evaluation in multihop wireless networks. new applications / nodes / services are iteratively deployed. 2) Free space: The Free Space Path Loss (FSPL) model For instance, smart meters may be progressively deployed estimates the attenuation of the signal when no obstacle is in a given district [70]. Consequently, they start to generate present between the receiver and the transmitter (line-of-sight periodical measures as soon as they are installed (or replaced) scenario). Let Pt (resp. Pr) be the transmission power (resp by the subscriber. the reception power). We have: To avoid wasting the energy resource of the nodes, the new devices should be able to join the network as soon as they start. λ 2 P = G (u)G (v)P ∗ (9) Thus, we may assume a gradual deployment with connectivity r t r t 4πd(u, v) preservation; a neighbour with a sufficient link quality already f exists in the vicinity of the new node. with λ denotes the wavelength ( c ), Gt(u) and Gr(v) respec- Surprisingly, this gradual bootstrapping scenario is very tively the transmission and reception gain of u and v. seldom studied. Most of the experiments focus rather on an We can compute with this path loss the strength of the ideal case, where all the nodes wake-up and have to join the received signals at the receivers and we can deduce the Signal network simultaneously. to Interference plus Noise Ratio (SINR). The SINR and the modulation then define the BER which corresponds to the probability that a bit is received in error. B. Communication model Comments: This free space model considers an ideal sce- Simulation and theoretical analysis rely on a PHY layer nario, without obstacle. In conclusion, this model should not model and they try to mimic radio propagation. Such a be used for a performance evaluation since more sophisti- model often associates with each pair of nodes a success cated models provide a better realism without decreasing the probability (i.e. a packet would be received correctly or not). tractability. In a asymmetrical link, this probability is different for the two 3) Two ray ground: The two ray ground model was widely directions; popularized by the NS2 simulator 1 and it considers both the 1) Unit Disk Graph (UDG) communication model: The direct and ground reflection paths. When the transmitter and simplest model consists of a geometric communication model receivers are close to each other, the path loss is defined by in which two nodes can exchange packets only if they are located at most 1 unit distance far apart. Furthermore, the 1http://www.isi.edu/nsnam/ns 11

Equation9 and the impact of the reflected wave is considered 1) Fixed range: Similarly to the UDG, see Section IV-B1, negligible. In all cases, the received signal strength is denoted we assume a fixed interference range. Any node that is located by: in the interference area is not allowed to transmit or receive any packet. This area is modeled by a disk, centered in h(u)h(v) the transmitter, with a radius equal to the interference range Pr = Gt(u)Gr(v)Pt ∗ 4 (10) d(u, v) (typically twice the radio range). where h(u) is the height of the antenna of node u. Comments: This fixed range cannot model the capture Comments: This model reflects more accurately the gray effect, where a signal can be decoded correctly only if the zone in which the radio link is neither short nor long and transmitter is very close to the receiver. Furthermore, it also the radio link quality significantly varies even for a small assumes a circular interfering region, which leads to an homo- change of distance. However, it keeps on considering only geneous network topology, and tends to simplify the task of ideal environments, e.g. omnidirectional antennas, isotropic the network algorithms. This model should not be considered and homogeneous environments. In particular, this model does for a realistic performance evaluation. not capture the time variability which makes the radio link so 2) k-hops interference model: This graph based model difficult to exploit [71]. expresses the interference rule as a hop distance condition. 4) Log-normal path loss model: The Log-normal path Basically, a given transmission fails if another transmitter, lo- loss model tries to empirically estimate signal strength. It cated less than k hops away from the transmitter or the receiver accurately estimates the path loss inside a building or in a is active at the same instant. Such model assumes implicitly high-dense populated area (e.g. a smart city). The path loss in that the transmission (and interference) is bidirectional (the Decibels is defined by: data and acknowledgement frames). d(u, v) Let consider the topology illustrated in Figure 10-b (1-hop PL + 10γlog + X (11) 0 10 d g interference model). All the nodes neighbours of S and D 0 cannot neither transmit nor receive any packet. where PL0 is the path loss measured at a distance of d0, and Comments: This interference model is often used in the Xg is a normal gaussian variable with zero mean modeling algorithms executed to e.g. allocate resource. However, the the flat fading. model does not capture the radio effect, nor heterogeneous This constitutes, to our mind the most accurate model for situations. Another more realistic interference model should Wireless Sensor Networks. It is sufficiently generic to mimic a be preferred. common situation, while still considering heterogeneous and unreliable radio links. More specific models, tailored for a 3) SINR: Since radio signals are additive, two pairs may given scenario (forest, indoor, etc.) can rather be addressed not collide but inserting a third one may create collisions. using a genuine dataset (see section IV-E). The SINR estimates the radio signal strength at the receiver. Depending on the modulation scheme, this level is then 5) Experimental measures to calibrate the models: The translated into a BER. parameters values strongly impact the characteristics of the radio topology. Testbeds may have this purpose: measures Any of the previous radio propagation models may be used in realistic conditions may help to find the most accurate to add up the signals received from the interfering nodes parameters values. and the ones from the transmitter: we have to compute the For instance, Chen et al. proved that a Log-normal path ratio of the signal strength of the transmitter and those of loss model is realistic to mimic indoor situations [72]. They the interfering nodes at the receiver. Then, the radio chipset identified several scenarios for which they calibrated the specifications (with the modulation, the bitrate, etc.) provide models appropriately. the corresponding Bit Error Rate (BER). Vazquez et al. [73] considered rather a smart city scenario, Comments: The SINR model represents the most generic using the 868MHz band, and estimated the path loss in model, able to capture realistic situations when it is coupled different situations, depending on the location of the antenna with an accurate radio propagation model. This model is the (on top of a building vs. in the street). most realistic one for simulations or a theoretical analysis. 4) Graph of conflict: The conflict graph is a very convenient tool to identify colliding situations. Each radio link C. Interference Model consists of a vertex in the conflict graph and an edge exists If two transmitters simultaneously send a frame, both radio between the two vertices if the pair of radio links mutually signals may interfere at one of the receivers. This collision interferes. With a k-hop interference model, it corresponds to consumes energy and bandwidth and, thus, should be avoided. the transitive closure of the line graph of the communication The interference models aim to reference the list of mutually graph. The radio links that are authorized to simultaneously interfering radio links. transmit without colliding form a Maximum Independent Set We have to carefully select the right interference model, in the conflict graph (i.e. they are not neighbours of the since it strongly impacts performance, whatever the stack of conflict graph), (e.g. the radio links SD and BU1 in Figure 10). protocols is [74]. We detail the most common models in However, this model is still unable to take into account the Figure 10. capture effect 12

U1 U unblocked node U U2 U B blocked node 1 U B1 B U1 interf. 2 B1 S D pair in transmission range conﬂict graph U2 B1 S radio link B2 SB SB S 1 2 D interfering radio links SD U1 U2 unblocked radio link D

B1 blocked radio link U2 a) ﬁxed range b) 1 hop interference model Fig. 10: Different interference models.

D. Key realistic characteristics Moreover, these probes have to be achieved continuously The characteristics of multihop wireless networks have been for a very long time (i.e. days or weeks). The simulator will extensively studied in the last years. The different experimental then pick-up the experimental result which is the closest from studies have highlighted the existence of some key character- the simulation time. istics which are often not captured by the different existing Comments: this method is very powerful to mimic real- synthetical models. istic conditions, while still providing reproducibility. It can 1) Time Variability: A multihop wireless network exhibits accurately model the capture effect, the time variability, etc. in most cases a very large time variability. Srinivasan et al. However, a dataset if specific to a given scenario, and its demonstrated the existence of a very short term radio link con- exhaustivity is very complicated to guarantee. Public datasets 2 3 4 sistency [75]. Basically, some links exhibit a strong burstiness: such as soda or tutornet (buildings), hydrobionets (industrial 5 packet losses are not independent, and the burstiness metric plant), or intel (lab) may be used. denotes this conditional probability of correct reception. V. TRAFFIC CHARACTERIZATION Cerpa et al. studied long and short term temporal aspects [76]. They advocate the obligation for protocols to As previously presented, LLNs support a large variety of consider the variations, and to not consider uniquely the real-world use-cases, ranging from industrial to monitoring average case. Because a radio link may drop all the packets applications. In this Section, we detail all possible traffic during a given duration, a huge number of retransmissions patterns and profiles that LLN deployment may support. may be required during these bad periods. Hereafter, we make a distinction between the following To the best of our knowledge, no model succeeds to capture different types of traffic: such variability. • Flooding: A packet is generated by a node or the sink 2) Asymmetry: Radio link asymmetry is frequent in and has to be delivered to all the nodes in the wireless LLNs [50]; the Packet Reception Rate is different in both network. Blind flooding has been proved to perform directions. This asymmetry may come from different antennas, poorly in wireless multihop networks, leading to the transmission power, or the presence of external interference. so-called broadcast storm problem [79]. Protocols such Radio irregularity impacts MAC and routing layers, but also as Gossiping [80] have been proposed to improve the the accuracy for localization, or the coverage [77]. reliability while reducing the incurred overhead. • Multicast: A packet has to be delivered to a group of motes. Directed Diffusion [81] was one of the first E. Datasets protocols to implement natively multicast: a node floods Instead of defining synthetic models, probably unable to its interest (a kind of multicast address). When a packet is capture all this complexity, we can also directly inject mea- generated, it is forwarded along the gradient of interests sures in the simulation / emulation. Typically, a set of packets and, thus, delivered natively to all the interested nodes. is generated to test all the radio links, to record their quality, • Unicast: The most common scenario consists of generat- and the occurence of collisions. ing packets with a specific unicast address. Alternatively, Concretely, we have to test all the radio links. Let n denote a mote may specify an anycast address, e.g., any sink the number of nodes. Since signals are additive, we have connected to Internet. theoretically to test all the possible subsets of radio links to During the previous decade, in most of the deployments, estimate the amount of collisions: the employed protocol addresses only one of the previously

n ∗ (n − 1) ∗ (n − 2)! (12) 2http://wsn.eecs.berkeley.edu/connectivity/ 3http://anrg.usc.edu/www/tutornet/ which becomes quickly intractable. Padhye et al. proposed 4https://github.com/apanouso/wsn-indfeat-dataset some heuristics to reduce the number of probes [78]. 5http://db.csail.mit.edu/labdata/labdata.html 13

a n a n n n n n n n n S n n S n n n a n s sink n n n n node n n (sensor or actuator)

n n n a actuator n n a (a) convergecast (all to a (c) localized (most of traffic is exchanged in a sink, often connected to the (b) any2Any (any device geographical area between the sensors and data flow rest of the Internet). with any other device). actuators)). Fig. 11: Model of traffic in low-power lossy networks, i.e. type of communications privileged in the topology.

mentioned scenarios. However, since the requirements for each 3) Localized: M2M traffic tends to be present in many of them are often too heterogeneous to be fulfilled by a single of the current deployments such as in smart homes [88]. In solution, the current standardization bodies and research lead this case, the considered topology does not rely on boarder- to design and developing standards that may provide solutions routers to collect the measurements and push them in a for multiple applications over single deployments [82]. For cloud. In fact, when data is not transmitted to a centralized instance, having different traffic flows (i.e., tracks) for different entity, the response time and the energy consumption may be applications [60]. significantly reduced [89]. In Figure 11c, each actuator directly collects the measurements of a group of sensors and take autonomously a decision. A. Traffic pattern On the contrary, in Figure 11b, the data flows do not traverse 1) Convergecast: Most common use-cases comprise a low- into the whole network. Such application is more scalable, it power devices connected to a cloud infrastructure. The sensor may multiplex more efficiently transmissions, without creating nodes transmit their data packets to the cloud through the bottleneck nodes. sink. The term convergecast is used, since all multihop traffic Unfortunately, the traffic models are very specific to each traverse through the border-router (Figure 11a). considered application. For instance, while a home automation Under convergecast traffic pattern, the heavily loaded area switch may command a neighbouring light, a boiler may is located around the boarder-router, forming the so-called use the temperature measurements coming from the whole funnelling effect [83]. Thus, all traffic is forwarded by the building. 1-hop away from the boarder-router nodes, leading to the formation of a congested zone [84]. Therefore, optimizing the traffic pattern in the interfering zone is essential to improve B. Traffic Profile the network capacity [85]. Finally, in this Section, we also define the traffic profile for The research community has been working on MAC and each traffic flow. routing protocols to particularly focus on addressing this 1) Constant Bit Rate (CBR): CBR flows represent the most specific challenge. Indeed, the standardization community is common traffic model in which a data packet is generated progressing to design an efficient time-slotted and frequency- periodically, possibly fragmented into several frames. This hopping MAC standard [44]. models well the smart metering scenario, where some meters 2) Up and Download: The download direction is often measure a physical phenomenon and report regularly their not considered in the academic community although it is of measurements. A CBR flow is defined by its period, i.e., time primary importance in most deployments. In particular, the interval between two packets and we often assume fixed length sink has to be able to send packets to the nodes in the following messages. scenarios: In Figure 12, we consider five nodes generating CBR flows. • Actuator: Some motes may be able to have a physical ac- The inter-packet time may be fixed independently for each tion on the environment. They must consequently receive node. In our scenario, S1 and S3 have the same CBR period some commands, either from the sink or from another and generate their packets simultaneously. Thus, they will con- node [86]; tend and possibly collide for each one of their transmissions. • Over-the-air reprogramming: Since the wireless in- On the contrary, node S2 has the same period but the packet dustrial network has to support multiple applications or generation is asynchronous. Similarly, node S5 has a different to react after an event detection, some nodes may have CBR period and contends only for the first transmission. to be reconfigured [87]. These commands are transmitted Consequently, we have to carefully consider fairness (see from the sink to the motes typically in multicast. Section III-A2c) among the different flows. 14

2s 2s of the participating nodes run on batteries. We already pre- S1 sented in Section III-A1a the different metrics that help to 2s S2 estimate energy consumption and network lifetime in general. 2s A. Residual Energy Estimation S3 Measuring accurately the residual energy is practically a S4 7.2s complicated task [92]. Most techniques rely on measuring S5 5.3s the battery voltage, (non linearly) correlated with the residual battery level. However, this correlation function depends on the Constant Bit Rate (CBR) event 1 battery model, the temperature, etc. [93] making an inaccurate estimation. S1 To estimate network lifetime would consequently require to S2 execute the application until e.g. the first node depletion. For S3 lifetime of months or even years, this approach is unrealistic. Furthermore, many algorithms would operate more efficiently S4 Event based if they are aware of the residual energy of each node, to S5 e.g. change routing decisions for a better load balancing. Thus, most of the solutions rather try to estimate energy packet event 5.3s inter packet consumption. generation occurence time Fig. 12: Traffic profiles: Constant Bit Rate vs. Event based B. Packet Based Estimation packet generation. Heinzelman et al. [94] propose to focus uniquely on the communication subsystem, because it represents the main source of energy consumption. The energy drained by a 2) Event detection: the LLN infrastructure may also be transmission takes into account: employed for event detection [90]. Instead of executing com- • The joule per bit energy consumption; putations in the cloud, low-power devices may implement • Fixed energy drained for any transmission, whatever the more sophisticated techniques to process data. For instance, packet length is; a node may transmit a measurement only if the temperature • Distance: The euclidean distance between the transmitter exceeds a specified threshold. By pushing the computations and the receiver. The authors consequently assume an close to the producers, less transmissions are required and, ideal radio propagation, without noise and interference, thus, network lifetime is optimized. with ideal radio chipsets. The event-detection property has a direct impact on the Inversely, the energy to receive a packet is linear with the traffic profile since packets are not uniformly generated. In packet size. The model has to be parameterized according to Figure 12, the first event is detected by nodes S1, S2, S3 and the radio chipset. S5 that all generate a burst of transmissions. We can remark Polastre et al. [95] also consider the energy for listening, that even if the global volume is similar to the CBR scenario, sampling and receiving. The radio state may be represented collisions may be more frequent since all nodes will intend to by a state machine (sleep, TX, RX, etc.), and the transition send their packets simultaneously. duration has also to be considered. For instance, almost 2ms Hereafter, we propose some potential approaches that can are required to initialize the radio before a reception when a be adopted: CC1000 chipset was in sleeping mode. • Real measurements may be collected in a real-life Wang et al. [96] use such a model to derive the whole deployment. These measurements can then be re-injected network lifetime, taking into account interference, multihop in simulations, emulating a real sensor or a real phe- routes, channel acquisition, etc. However, such analytical nomenon. models tend to under-estimate the impact of several environ- • Models to mimic the physical environment are used in mental parameters (such as temperature, radio link quality, simulations. The simulator computes the list of simulated time variability, etc.) measurements acquired by each sensor. These measure- Muller et al. [97] developed a specific circuit to measure the ments may trigger a packet generation. However, the battery voltage, from a home-made mote. The authors combine models are very application specific, and require a very this circuit with a packet-level method in order to estimate precise knowledge of the physical phenomenon, compli- more accurately the energy consumption. cated to obtain in complex environments. Participatory sensing represents a promising way to collect data and to construct accurate models [91]. C. Transmission power adaptation In topology control, a node selects the list of neighbours VI.ENERGY CONSUMPTION MODELSFOR WIRELESS with which it will communicate. To reduce the level of DEVICES interference and its energy consumption, it may adjust its The network lifetime is a major requirement because most transmission power so that the received signal strength is just 15

prior to real deployment. Such deployments require to learn about available operating systems and supported platforms. refill radio solar chipset This section thus provides an overview of existing emulators, consumed accumulator wind (e.g. Li-ion) which help to provide more accurate results by hosting specific lightweight OSs that run on top of specific target hardware. !controler kinetic We discuss to what extent emulators may become a mandatory modules battery device step for thorough evaluation of networking protocols.

Fig. 13: Energy harvesting components. A. Numerical Analysis The simplest way to study the behaviour of an algorithm consists of replacing its different inputs by realistic values. above the threshold required to decode it correctly. Aziz et For instance, a centralized algorithm can use a graph based al. [98] present a comprehensive view of the different topology model, to define which nodes (aka vertices) have a common control techniques to prolong network lifetime. radio (edge). To evaluate the dependency with the input parameters, we often use Monte Carlo simulations; different D. Energy harvesting values are used and should lead to a similar behaviour. As an Energy harvesting consists of incorporating a module into example, 20 source nodes should generate approximately as a device that collects energy from the environment [99]. A many collisions, wherever they are located. solar panel, a piezoelectric device or an anemometer provide Those numerical computations can be addressed with var- energy to the ious programming languages, such as MATLAB, Python and device, which stores often this energy in its battery (Fig- GNU Octave, where each one of them have their pros and ure 13). Energy harvesting requires to redesign the stack of cons. The first difference someone can notice is that MATLAB protocols. Indeed, the routing decision should depend on the requires to purchase a licence in order to utilize its core level of battery of the relay nodes [100]. functions, while both Python and Octave are open and free pro- A super capacitor helps to temporarily store the harvested gramming languages. Furthermore, additional packages and energy [101]. Since it does not need a dedicated charging libraries for MATLAB, such as SIMULINK, add extra charges circuit, it presents a better conversion ratio. However, sub- in the overall cost. stantial leakage power losses need to be considered since they Regarding their syntax, MATLAB and Octave are high- significantly impact efficiency. level languages. Also, Octave provides a MATLAB-like en- The harvested energy depends on the source; solar energy vironment that facilitates the transition between the two lan- obviously is ineffective during the night. The level of charge guages. On the other hand, Python aims to be used as a full can be modelled by employing a discrete Markov Chain [102]. scale programming language which can support OS specific A transmission or reception consumes energy, while the har- modules like process scheduling and multi-threading, as well vesting source will fill the battery. Because the harvested as networking and databases. Consequently, programming in power is often much smaller than the transmission/reception Python requires a longer learning curve to get familiar with power, the communication is considered instantaneous. its core libraries and programming flow, but it is easy to learn even from novice programmers. Additionally, a great advantage of Python is that it can support fast prototyping VII.AVAILABLE TOOLSFOR PERFORMANCE EVALUATION for different functionalities in LLN applications, considering In this Section, we detail a variety of tools and methods some recent efforts from the community to develop smarter that can be utilized in order to evaluate the performance of implementations of Python, like PyPy6, as well as tiny micro- different LLN applications, algorithms or protocols. Usually controllers that runs in Python7. A brief description of each the researchers start with the numerical analysis, where they language follows up. test their ideas in a more theoretical approach, based on 1) MATLAB: MATLAB (MATrix LABoratory) is a soft- mathematical formulas with realistic values. Next, the network ware package and a high-level scripting language that enables simulators can be used so as to evaluate those theoretical as- high performance numerical computation and visualizations of sumptions, but with an essence of abstraction. However, there new ideas. It represents one of the most popular software pack- are some advanced simulators known as emulators, which are ages for scientific research. The mathematical framework can able to communicate with real network devices such to provide provide solutions on a broad field of mathematical problems more realistic simulations of hardware aspects (e.g., processor such as matrix algebra, complex arithmetic, linear and non- performance during duty cycles, energy consumption). Fur- linear systems, differential equations, signal processing etc. thermore, due to the current low cost of acquiring simple 2) GNU Octave: GNU Octave8 is an interesting tool, actuators, sensors, microprocessors and other modern IoT initially published in 1993 and is compatible with Matlab hardware, the research community has developed numerous open physical testbeds that allow for network experimenta- 6http://pypy.org/ tions. These testbeds can provide the ability to the scientists to 7https://micropython.org/ expose their solutions to the most realistic conditions possible, 8https://www.gnu.org/software/octave/ 16 scripts, . In particular, it is used to validate algorithms using directly capture the different packets transmitted/received the queueing theory and Markov chains9. by the nodes. By analyzing this packet capture, we 3) Python: This language is now widely used for scientific can compute e.g. the Packet Delivery Ratio, the end-to- computing, in particular with the SciPy library10. In order to end delay (see section III-A) using the same scripts as evaluate network algorithms, the library NetworkX11 provides for experiments on real testbeds. Such method reduces routines for basic graph computation, simplifying the scien- the development costs when moving from simulation to tific development. For instance, scheduling algorithms (e.g. experiments. TASA [19]) often use Python to validate performance in an • Visualization tools are not required stricto sensu to mea- ideal graph. sure the performance, but simplify greatly the debugging 4) Limitations: Numerical analysis represents the first step tasks. For instance, ns3 relies on PyViz [112] to debug a to validate an algorithm. However, results tightly depend on simulation, e.g. where a packet is dropped, the movement the accuracy of the models. Unfortunately, the most realistic of a specific device. To our mind, offline visualization models (interference, radio link quality) make the numerical tools, which may exploit traces, should be preferred. They analysis quickly intractable. Thus, another tool to validate allow to replay the simulation, and to detect and localize performance should be employed in most of the scenarios. a fault when an inconsistent result is obtained; • Clock drifts: to the best of our knowledge, the simulators B. Network Simulation do not mimic clock drifts while they may have a significant impact on the performance [113]. Some additional Large-scale testbeds are expensive to be developed and modules or the engine have to be developed to quantify need great effort to be managed [103]. Therefore, network the impact of clock drifts. simulators have been designed with an aim to provide a software platform which can address the key aspects of the 2) Solutions: We detail here the most popular simulation overall performance of different types of networks. However, tools used by the LLN community. TableII summarizes their it is almost impossible to duplicate the exact same conditions main characteristics. of real deployments. Simulations have been proved to not • ns2 [23](“Network Simulator”) is a discrete-event sim- estimate correctly the impact of physical phenomenon [104]. ulator mostly used for a research or educational purpose. On the other hand, simulators can be considered essential Since it was primarily designed to simulate the Internet, for exploring LLN applications, acting as a common ground it provides the full IP stack (TCP, IPv4, IPv6, etc.). Its for the scientists to test their ideas [105]. In order to effectively models (protocols) are written in C, while Tcl/Tk scripts evaluate a study by employing simulation campaigns, it is control the simulation (e.g. when a TCP flow starts). important to have a good knowledge of the existing simulators However, ns2 requires a long learning curve and advanced and their capabilities. skills in order to conduct valuable and repeatable simu- Some examples of widely used simulators from the research lations. Besides, ns2 does not target specifically LLNs, community, include the ns2[23] ns3[24], OMNeT++[25], and requires huge development effort to have a full LLN Riverbed Modeler[27], Qualnet[28] and WSNet[106]. How- compliant stack. ever their individual capabilities and approaches on simulating • ns3 [24] is the latest version of the Network Simulator. the different details and types of networks, are quite distinct. Instead of an evolution, the team has re-wrote from 1) Proposed services:: the event-discrete simulators may scratch the simulator, making it incompatible with its propose some additional services to simplify the development. previous version. The ns-3 scenarios and models are • Topology generation: to mimic the different scenar- implemented entirely in C++ with optional use of Python ios, we have to simulate different topologies (see sec- and PERL bindings. tion IV-A). The topology may be defined as a scenario ns-3 provides an interesting hybrid mode, mixing em- with a GUI (e.g. riverbed), or via a configuration file (e.g. ulation and simulation to focus on a specific physical WSNet). To replay the same setup, flat configuration files phenomenon. Besides, it provides tools to measure the should be preferred since they simplify the distribution, performance of the network by analyzing the packets and can be easily converted into other formats; captures by each device. This genericity is a strong • Mobility models: most simulators provide a way to asset: the same tools can be used for simulations and control the trajectories for some of the nodes. Because experiments, reducing the development costs, and making of the discrete-event engine, the location of a device is the analysis more reproducible. updated periodically, simulating a movement. • OMNeT++ [25], is a component-based and discrete-event • Statistics collection: to measure the performance, the framework, which was primarily developed for building devices have to be instrumented. Some of the simula- network simulators for wired and wireless networks. tors provide proprietary API to collect statistics (e.g. Support for specific domain networks, such as sensor Riverbed, ns3). ns3 provides also a generic way to networks has been provided through separate packages such as Castalia [114] WSN simulator. Furthermore, its 9 http://www.moreno.marzolla.name/software/queueing/ core is totally implemented in C++, with tools for traffic 10Scientific Computing Tools for Python, http://www.scipy.org 11High-productivity software for complex networks, https://networkx. visualisations, packet tracing and debugging. github.io/ Most state of the art standards for LLNs are not na- 17

Simulator Type12 Language Models13 Speciﬁc features Mobility IEEE802.15.4-2006 Bluetooth DSME TSCH RPL CoAP ns2 O C & Tcl/Tk y y y n n n n parallel processing for scalable simulation ns3 O C y y n n n y [107] packet capture (libpcap) for statistics omnet++ O C & XML y y n n n [108] n riverbed C C y y y n n n n GUI for a smoother learning curve modeler Qualnet C y y n n n [109] n Takes beneﬁt from parallel architectures WSNet O C & XML y [110] n n n [111] n dedicated for low-power nodes TABLE II: Summarized characteristics of popular network simulators.

tively supported. For instance, an RPL implementation maintain a reasonable computation time, we have consequently is provided in [108], but some of the features such to consider only simple radio propagation models or a small as the Destination Advertisement Object (DAO) are not number of nodes. Besides, Colesanti et al. [116] have high- supported. lighted that simulation results may differ significantly from • Riverbed Modeler is a commercial, object-oriented, real experimental results. discrete-event network simulation software that is capa- Most of the simulators are not specialized and target the ble of operating in packet-level [27]. Riverbed Modeler large scale networks in general. Some of they provide a provides a powerful GUI for the designing of simulation wireless library but with a very limited (and not up-to-date) scenarios, which tends to reduce the learning curve for set of protocols. Re-implementing the different standards and beginners. Recent versions also provide a large set of protocols increase the development costs, and the number propagation models, with an implementation of the IEEE of possible bugs. The reproducibility and the comparison of 802.15.4 and ZigBee standards. different implementations are also problematic. However, Riverbed Modeler does not target specifically The emulation/experimentation community (e.g., the low-power devices. Thus, most of the protocols and COOJA/Contiki, OpenSim/OpenWSN) is much more active. standards for LLNs are not supported natively. Most of the latest version of the IETF drafts and the latest • Qualnet is a proprietary simulator, supporting parallel standards are often available. Thus, we recommend now processing to make the simulation scalable [28]. It mimics to skip this simulation step to go directly to emulation, real communications networks, enabling to model the simulating only the PHY layer. Possibly, simulation may whole propagation environment in a variety of complex still be used to study in depth a very specific phenomenon, situations (e.g. a collection of buildings for smart cities). and not the whole behavior of a full stack. It provides a library comprising a set of protocols (mainly Zigbee). C. Testbeds As a descendant of GloMoSim[26], Qualnet simulator was rather designed for ad-hoc networks and its usage As it was previously exposed, testing and verifying new in LLN scenarios is not well recognized by the research protocols and applications only over simulations may imply community. simplified assumptions, given the absence of accurate energy • WSNet and radio simulation models, as well as considering the great WSNet [106] is an event-driven simulator dedicated to complexity of real deployments. WSNs, which has been extensively evaluated [115]. It Therefore, performing experiments with real hardware al- implements a large collection of radio propagation mod- lows scientists to analyse performance of their applications els, and the PHY layer can be modified easily (MIMO, under realistic environments. Indeed, due to the recent tech- modulation, multiple radio interfaces). A node may be nological advances, the cost of such hardware has been re- mobile, and a physical environment may be simulated duced, thus, allowing the deployment of various large-scale (e.g. fire detection). controllable and manageable experimental testbeds. The simulator and its protocols are fully implemented Many testbeds are open to the research community for in C and XML. Unfortunately, each protocol requires conducting their experiments with different characteristics, as a specific implementation, and the current library and they are summarized in Table III. packages are quite limited. In order to ease the choice of the testbed and facilitate the experiments, we propose some guidelines for users to use 3) Limits: The limits of simulations correspond to the repeatable setups and obtain reproducible results. limits of the simulation models. For instance, the most so- 1) Characteristics: Existing testbeds present their own phisticated radio propagation models should be preferred, but characteristics that should be carefully studied before opting their complexity has an impact on the computation time. To for a given facility. 18

• Hardware requirements: They play a critical role for with robots over reliable links (e.g., WiFi). conducting realistic performance evaluations. The testbed 2) Proposed services: In addition to these characteristics, of choice should correspond to the appropriate hardware open testbeds may propose some services that would ease requirements of the tested application in order to enable the setup of experiments while guaranteeing uninterrupted the researchers to investigate in depth its functionality operations. prior to real deployment. The hardware parameters of a physical LLN platform include the network heterogeneity • Maintenance and configuration: Regular maintenance and scalability, where the underlying devices play dif- is appropriate to verify that the hardware and the software ferent roles and reserve various resources. According to architecture of the testbed are still operational so as to Yarvis et al. [117], heterogeneity can be distinguished manage effectively the experimental queue. Additionally, into three types. The first type is the computational scheduled maintenance must take place in order to update heterogeneity where some of the nodes have increased the provided services and hardware components like computational abilities, such as the sink nodes. Another the motes’ batteries, such to optimize the functionality type that could be considered is the link heterogeneity and extend the lifetime of the facility. End users would where some of the nodes may have wired interfaces in expect such information to be available in order to select order to establish reliable communication links. The last the most appropriate facility. Furthermore, regarding the type is the energy heterogeneity where the nodes may configuration of experiments, existing platforms present utilize different energy resources. various tools that allow to design, conduct and analyse • Radio environment and physical topology: Radio prop- the experiments in a convenient and reliable way. The agation necessarily depends on the environment of a prime objective for end users is to finely configure their testbed (e.g., materials, thickness of ceiling and walls, experimental setup. This includes the communication density of plants in case of outdoor platform). Users interfaces and protocols between the user and the underly- may select a testbed whose deployment area presents ing hardware in the different network layers, the scenario the more similarities with the one of the target applica- repeatability and possible simulation interoperability. tion. In addition, some facilities either offer only indoor • Radio link characterization: In addition to the physical nodes (e.g., Indriya, FIT IoT-lab) while others include topology (i.e., location of nodes), testbeds may expose the outdoor devices (e.g., ORBIT, FLOCKLAB). Testbeds radio topologies that can be tested. For instance, most of also greatly differ in terms of number of devices that routing and MAC protocols require bidirectional links so are made available. As FLOCKLAB presents 40 sensors, that two nodes can exchange information (e.g., data and TWIST or Indriya involve more than 100 motes, while acknowledgement frames). Due to the realistic environ- FIT IoT-lab announces more than 2,500 nodes that can be ments in which testbeds are deployed, such assumption booked for experiments. Once again, those characteristics may not hold. Thus, end users should be able to identify should be studied carefully before choosing a platform. the different radio links and their associated properties Note that some testbeds may allow end users to book only (e.g., average bit error rate, distribution of radio link a subset of the available devices. Consequently, on some quality over time) before running their solution. Some facilities, various experiments may co-exist and exper- dedicated tools would assist them to characterize the imented solutions would then face noisy environments. used topologies and their associated trust level. The latter This may be considered as a perfect environment for would allow users to select the most appropriate nodes users aiming to test their solutions under unanticipated and links for testing their own solutions and to compare interferences. the obtained results against the existing ones. • Mobile nodes: Many LLN applications involve mobile • Topology control: Once the radio topology and link nodes that require special communicating schemes in or- characterization are known to the users, the topology may der to interchange sensory data and information. Testing be controlled. Some facilities would allow for topology and executing mobile scenarios during an experimenta- control with either basic selection of used nodes or tion procedure requires to involve and combine advanced enhanced transmission power adaptations. Users may also and intelligent technologies such as robots. Consequently, be able to impact wireless transmission by inducing few of the widely popular open platforms do support radio noise and interferences within the network. Some mobility [118], even though more and more modern testbeds have incorporated such features (e.g., FIT IoT- experimental facilities employ robotic and automation lab). Even though a fine tuning of the jamming devices is systems (e.g., Emulab, FIT IoT-LAB, KANSEI). Numer- required to reproduce observed realistic radio conditions, ous challenges need to be addressed when having mobile such a service could counterbalance the too idealistic robots in a testbed, namely, charging, remote administra- environments in which some testbeds would be deployed. tion and maintenance of the robots. Indeed, robots must Additionally, the majority of LLN testbeds are equipped be able to reach their docking stations automatically. They with commercially successful motes that provide different must also be able to follow the assigned trajectories. out-of-the-box functionalities such as to enable users to Thus, accurate positioning and path planning mechanisms conveniently test various types of applications. Another with obstacle avoidance should be supported by such option for addressing scalability and heterogeneity issues, testbeds. Conversely, remote users must be able to interact is the federated model, which enables local experimental 19

Testbed Nodes Description Emulab 580 PC nodes with USRP A combination of hardware and software tools. There are many instances of the Emulab [119] 6 MICA2 robots framework deployed in more than two dozens sites around the world. 30 stationary MICA2 FIT IoT- 1144 MSP430-based motes Over 2700 indoor wireless sensor nodes spread across six different sites in France. Fixed LAB [?] 956 ARM Cortex M3 nodes and mobile nodes, embedding a variety of wireless sensors are available, with different 561 ARM Cortex-A8 nodes processor architectures (MSP430, STM32 and Cortex-A8), and different wireless chips 108 open host nodes (802.15.4 PHY @ 800 MHz or 2.4 GHz). In addition, open nodes can receive custom 50 wireless mesh routers wireless sensors for inclusion in IoT-LAB testbed. six robots (127 planned) FLOCKLAB 30 Observers equipped with any four of A mixed indoor/outdoor topology, able to support different services such as measuring [121] Tmote Sky, OpenMote, MSP430-CCRF, power consumption and time accurate tracing and actuation. TinyNode, Opal, and Iris motes Indriya 139 TelosB Based on MoteLab. The nodes are powered over the USB backchannel and equipped [122] with light, temperature, acoustic, magnetometer, 2- axis accelerometer and infra-red sensors. Intel Mirage 97 MICA2 Based on a resource allocation system where the testbed resources are allocated, [123] 51 MICA2DOT motes according to a repeated combinatorial auction. The motes are equipped with pressure, temperature, light and humidity sensors and powered over Ethernet. KANSEI 210 stationary nodes with Stargates, Heterogeneous, hybrid experimental LLN laboratory that combines hardware motes, [124] TmoteSky and Extreme Scale Motes simulation engines and data generation devices 50 portable Trio motes five robots MAP [125] 32 static mesh routers An experimental WMN laboratory. The testbed does not provide power consumption 5 laptops and 16 PDAs awareness. MoteLab Fixed array of 30 MICAz One of the first open LLN testbeds, MySQL back-end server, a PHP web server, Java- [126] 190 TelosB based data logger and a Job Daemon for assigning tasks to the motes, Wall-powered with in-situ power measurement device in addition to temperature, humidity and light sensors. NetEye 130 TelosB motes An open LLN experimental testbed equipped with light sensors and a mixed USB and [127] 15 laptops Ethernet backchannel. ORBIT 400 nodes with more than 1,500 radio de- A radio grid network testbed that consist of a remotely accessible indoor testbed, in [128] vices addition to an outdoor trial network with mobile nodes. Tutornet 13 Stargates A simple three-tiered, clustered LLN testbed [129] 91 TmoteSky 13 MICAz motes TWIST 102 TmoteSky A central PostgreSQL server which is hierarchically organized in three layers, the [130] 102 eyesIFX servers, the super nodes and the sensor nodes. USB powered with light and temperature sensors. The super nodes are Network Link Storage Units. UMass 40 buses with GPS devices A vehicular DTN of 40 public transport busses and various throwboxes that work as DieselNet HaCom Open Brick computer with 3 radios relays that promote the messages to the central repository. [131] WARPLab Up to16 WARP nodes An experimental framework for experimentation of physical layer protocols controlled [132] by a single PC by interfacing WARP nodes directly with MATLAB. TABLE III: Summarized characteristics of popular LLN testbeds. platforms to interconnect under a common framework in operating systems. Hahm et al. [135] provide a very detailed order to share their resources and provide more powerful and comprehensive survey about LLN operating systems. evaluations, similar to WISEBED [133] and FIT-IoT [?] 1) TinyOS: TinyOS, based on event driven design, is so testbeds. far one of the most used open source OSs for LLN scenarios [136]. It is written in the nesC programming language, a dialect of the C language, as a set of cooperating tasks D. Which OS to choose? and processes. TinyOS is designed for very low-power and Billions of sensor nodes, actuators smart meters and home constrained devices, i.e., 8 bit and 16 bit platforms, and is appliances will be interconnected by wireless networks or popular for its sophisticated design. Finally, it comes with Power Line Communication (PLC) [134], thus, giving birth BLIP network stack, an implementation of the 6LoWPAN to the IoT. Therefore, it is essential to have a lightweight stack. operating systems that fulfills the requirements of such ultra 2) Contiki OS: Contiki OS is one of the leading oper- low-power and extremely constrained, in terms of computing ating systems for embedded systems both in industry and power and memory, devices. in academia. Contiki is developed in C language, while In this subsection, we briefly introduce the operating sys- some parts make use of macro-based abstractions (e.g., Pro- tems for embedded and constrained devices that represent the tothreads [137]), and has been ported to a number of mi- most promising approaches towards realizing industrial (and crocontroller architectures, including the Texas Instruments not only) Low Power Lossy Networks. We focus on smart MSP430 and the Atmel AVR [142]. Contiki is running on the things with sufficient autonomous capabilities (i.e. not RFID ESB platform [139] that uses MSP430 microcontroller with 2 tags) but which must also be energy aware (i.e. save energy). kilobytes of RAM and 60 kilobytes of ROM running at 1 MHz. Note that most OSs (if not all) are written in the C program- Furthermore, Contiki comes with a number of network stacks ming language. Hereafter, we will detail the most popular such as the uIP stack (with support for IEEE802.15.4-TSCH, 20

Emulator Language Models14 Speciﬁc features Mobility IEEE802.15.4-2006 Bluetooth DSME TSCH preamble sampling RPL CoAP Cooja C y y n n y y y y GUI interface TOSSIM C & nesc n y n n n y y y complex learning curve for nesc MSPSim C n n n n n n n n independent from the OS, reused by many platform and OS speciﬁc emulators WSim/WSNetC y y n n n n y n independent from the OS, to be plugged with WSNet [106]. OpenWSN/ C& n n n n y n y y GUI (web-based) interface OpenSim Python TABLE IV: Summarized characteristics of popular network emulators (all of them are opensource).

IPv6, 6LoWPAN, RPL, and CoAP) and the Rime stack, which OS and protocol stack to use. Only WSim and MSPsim are provides a set of distributed programming abstractions. independent from the OS, but some key features (such as phys- 3) FreeRTOS: FreeRTOS is developed since 2002 and is so ical communications, mobility support, high-level description, far one of the most used open-source and Real-Time Operating etc.) are not emulated. Systems (RTOSs) for embedded and constrained devices. It However, emulation is a very convenient tool for perfor- has been ported to a large number of micro-controllers, and is mance evaluation. The whole hardware platform is emulated, the de-facto standard solution for micro-controllers and small so that the same implementation may be used in testbeds or micro-processors. FreeRTOS is written mostly in the C lan- even in real-life. By debugging the emulated mote, we can guage, but there are a few assembly functions (i.e., mostly in also identify more efficiently the potential bugs. architecture-specific scheduler routines). Even though, it does TableIV illustrates a summary of the different characteris- not come with its own full network stack, however, third-party tics and protocols supported by the major emulators for LLNs. network stacks can be employed for Internet connectivity. 1) COOJA: is an open source and flexible simulator for 4) RIOT: RIOT is designed and developed since 2012, by the Contiki OS specialized for sensor nodes [142]. COOJA a growing, world-wide open-source community. RIOT is a supports a large variety of hardware, including the most microkernel-based RTOS with multi-threading support, while popular platforms (TelosB, Zolertia Z1, etc.) This emulator employing an architecture inherited from FireKernel [140]. is written in Java and is supported consequently on mist of The OS is written in C, while applications and libraries can the OS. also be implemented in C++. Furthermore, RIOT comes with 2) MSPSim: MSPSim is an open source instruction set several network stacks, such as the implementation of the emulator able to simulate complete motes such as Tmote Sky, full 6LoWPAN stack (i.e., the gnrc stack), a port of the as well as custom motes based on Texas Instruments MSP430 6TiSCH stack OpenWSN, and a port of the information centric microcontroller [143]. Extendibility is supported through a networking stack CCN-lite15. variety of available build-in implementations of different pe- 5) OpenWSN: The OpenWSN project16 was founded in ripheral devices as components, which are further simulated 2010 (from the Berkeley Sensor & Actuator Center at UC based on a discrete-event approach. Berkeley) and is developed since then by a growing open- 3) TOSSIM: is a discrete event simulator and part of source community that contributes to the Industrial Internet the TinyOS project, which is an embedded operating OS of Things (IIoT). OpenWSN comes with an implementa- specialized for sensor networks [144]. Thus, TinyOS application of a fully standards-based protocol stack (i.e., 6TiSCH tions can be compiled directly into the TOSSIM framework. network [141]) designed for capillary networks, rooted in Furthermore, TOSSIM replaces the low-level components of the new IEEE802.15.4-2015 TSCH standard, and providing a TinyOS system, such as the Analog-to-Digital Converter IPv6 connectivity to ultra-reliable and low-power industrial (ADC), the Master Clock, the EEPROM and several of the networks. Moreover, OpenWSN comprises a Board Support components in the radio stack in order to emulate their Package (BSP) i.e., a simple hardware abstraction. real behavior. It also provides a GUI that enables handy In TableV, We here focus on most commonly available visualization, designing and debugging of running simulation platforms, as summarized in Table III. scenarios in a controlled and repeatable environment. 4) WSim: is a platform simulator that was developed as part E. Emulation of the Worldsense framework [145]. It relies on cycle-based simulations using microprocessor instruction-driven timings. To complete our presentation of the available tools with The simulator is able to perform a full simulation of hardware emulators since many solutions are tightly related with the events that occur in the platform and to give back to the 15http://ccn-lite.net developer a precise timing analysis of the simulated software. 16http://openwsn.org/ FreeRTOS, Contiki and TinyOS operating systems have been 21

Operating Latest release Supported Stacks RAM / ROM Real-Time Emulator system architectures (min. required) support Atmel AVR PIC32 TI MSP430 STM32 Standard TCP/IP 6LoWPAN / RPL / CoAP 6TiSCH Contiki-NG 4.0 (Nov. 6th, 2017) x x x x x x 2kB / 30kB partial COOJA FreeRTOS 10.0.0 (Nov. 28th, 2017) x x x x x x 1kB / 5kB full Windows FreeRTOS port OpenWSN 1.8.0 (Oct. 5th, 2014) x x x x x 3,7kB / 31kB partial OpenSim RIOT 2017.10 (Oct. 27th, 2017) x x x x x 1,5kB / 5kB full None official, even though RIOT OS executables can be simulated with Cooja. TinyOS 2.1.2 (Aug. 20th, 2012) x x x 1kB / 4kB none TOSSIM TABLE V: Summarized characteristics of popular and open-source operating systems. successfully tested on Worldsense. WSim can be used in For scenarios considering the mobility, the testbeds standalone mode for debugging purpose, or interfaced with should provide a way to replay the same trajectory, the WSNet simulator to perform the simulation of a complete with the same signal quality. This represents currently sensor network. a key challenge since GPS is inaccurate for the indoor 5) OpenSim: is an open source emulator and simulator environments. Currently, even with perfect localization which was tailored to work with OpenWSN [33]. It is written of all robots, trajectories are very difficult to replay, in Python, and can be handled via a web-based interface to especially due to the odometer drifts. control the devices, push some commands, change the link • A raw dataset should be available in order to provide quality, etc. Moreover, it can even mix emulated devices with the whole set of results (before filtering). These measures real devices. It supports currently a very large variety of have to be stored durably, to guarantee a free access in hardware platforms. the next decades. This dataset can be easily re-interpreted 6) Conclusions: Emulating a device is CPU intensive, later, using the latest, up-to-date scientific knowledge, like while emulating a large-scale network requires a powerful Raman et al. did with the MIT roofnet dataset [147]. machine. Thus, emulation/simulation is limited to small scale • A public repository should collect all the traces, the topologies for prototyping to focus on a given phenomenon implementation and the scripts. Crawdad [148] aims to while large-scale evaluations may use a public testbed. collect a list of traces for different wireless networks Emulators are often tied with a particular OS. Thus, the (mainly mobility traces). This good practice would sim- communication stack will probably guide the choice for plify the quantitative comparison against related work and the performance evaluation. For instance, COOJA seems the would also allow other researchers to analyse the results most relevant platform for preamble sampling protocols while in order to identify new phenomena/problems. OpenWSN is strongly recommended for a 6TiSCH stack since Regarding experimentations, we shall be able to replay the it integrates the last updates from the IETF Working Group. same experimental setup over stable and finely controlled hardware components. The real-world environment should VIII.GUIDELINESAND DIRECTIONS be controllable to provide exactly the same conditions. This imposes to guarantee stability of hardware and environment A. Reproducibility components over time. Similarly to Guix-HPC17, an effort to An experiment has to be reproducible, leading to the same optimize GNU Guix for reproducible scientific workflows in results, when facing to the exactly same conditions [146]. With High-Performance Computing (HPC), a future direction would simulation, it should be sufficient to provide a detailed descrip- consist in designing building blocks that could be configured tion of the simulation setup and the implementation of the and used within IoT testbeds (e.g., sets of topologies, mobility algorithm/protocol. Unfortunately, such detailed description is and traffic patterns, interference maps) in order to conduct very infrequent in the literature. repeatable setups and publish reproducible results. According to our perspective, performance evaluation should describe the following points: B. Diversity and robustness • Exhaustive description of the setup: the authors have to describe without any ambiguity their performance Performance evaluation may be very dependent on the evaluation setup (traffic, topology, models). The source conditions. The testbed or the simulation models have to be the code should be ideally distributed, or at least the binary most realistic as possible to predict the actual performance. An (firmware) of the protocol’s stack used for the perfor- experimental validation is even not the panacea since different mance evaluation; testbeds do not provide the same environmental conditions • Scenario Replay: a set of scripts that permit to replay all (e.g. indoor vs outdoor, etc). the experiments should be provided, so that anyone can reply the same scenario on the same or different testbeds. 17https://guix-hpc.bordeaux.inria.fr/ 22

In order to make the performance evaluation robust, we layer. Unfortunately, the interference model deeply impacts the have consequently to evaluate an algorithm that in the most performance [74]. Even worse, the behavior of the network has diverse situations it will have to take into account the following been proved to deeply depend on the simulator for wireless aspects: networks [153]. On the contrary, experiments are complicated • Hardware: If a proposition is very closely related to the to reproduce and interpret, because may external factors (e.g., hardware used, we should proceed to the performance external interference) impact the performance of the network. evaluation on top of different hardware (e.g. different Emulation seems a promising way to simplify the performance radio chipsets to verify the presence of asymmetrical evaluation, but it relies also on a synthetic PHY model to links); mimic the transmissions. • Topology: To verify that a solution performs well in any A promising approach seems to combine experiments situation, we should study its behaviour in various con- through a testbed with simulation or emulation: the experi- ditions and to measure the impact of different parameters ments provide the PHY measures concerning the success of (e.g. density, scalability, traffic intensity); each transmission, information re-injected in simulation or • External Interference: To make the experiments re- emulation to mimic the behavior of a real radio medium. producible, some nodes (with the same or different The same measures may be re-injected to make the evaluation technologies) may be used to generate an interfering reproducible. signal. Such experimental setup would maximize the re- A few initiatives already proposed a free access to such producibility, while estimating the robustness to external datasets: interference [7]. • the packet delivery ratio for 55 devices, over 16 frequen- Federated testbeds should make such comparison easier, cies at USC18 [154]. letting researchers to reserve several different testbeds [149]. • aggregated statistics (e.g., Packet Delivery Ratio, mean LQI) for a testbed with 10 nodes executing RPL during 19 C. Long term evaluation one day . • packet delivery statistics (i.e. packet received or not) While low power lossy networks expect to be deployed for a for 267 radio links, over the 16 IEEE802.15.4 channels, long-term use, most of the performance evaluations focus sur- during 1.5 hour20. prisingly on experiments which last for a few minutes. Many papers even do not specify the duration of their experiments / Having the raw (non aggregated) results should be preferred simulations [12]. to be the most generic as possible. To demonstrate the relevance of LLNs, solutions have Unfortunately, probing each radio link individually to esti- been deployed and monitored. Zebranet [150] and Sen- mate the level of interference, and the probability of collisions sorscope [151] provide results over one month. However, they is very expensive. Indeed, we have to test each subset of radio often consist in real deployments, where the application is the links iteratively, with a train of probe packets [7]. Such an main concern. exhaustive method is quickly inapplicable, and some heuristics To more accurately capture the actual performance of a low have to be proposed to mimic actually a real deployment. power lossy network, we need to study more in depth the following points: E. Definition of Scenarii and Benchmarking • Convergence: Many papers propose to discard the first For low power and lossy networks, we have a profusion of minutes of the experiments / simulations and focus on solutions which are evaluated in very different conditions [12]. the steady state only. However, the bootstrapping phase Unfortunately, multiplying the conditions complicates signifi- plays a major role in the convergence time and thus cantly the interpretation and the comparison. The IETF ROLL appears of primary importance as well. For instance, with WG has defined in the past the requirements for smart build- RPL it is common that many routes may change initially, ing [4], smart homes [155] and industrial [156] applications. consuming a large overhead [152]. Thus, we have to Similarly, the ETSI has defined the traffic profile for many study more carefully this phase in order to accelerate the applications in smart cities, detailing the requirements in delay, convergence of the solutions: waiting for a few hours may bandwidth, etc. [157]. be unacceptable for many deployments. We need to define clearly some scenarii: • Long-term evaluations: we need to start experiments • we need to define scenarii that each researcher should use which should last at least for few days. Rare phenomena for a collection of different applications. The scenario may have significant impact to the performance of the should be entirely defined (topology, link quality, vari- network, but can only be captured if the experiment lasts ability, etc.) so that the different results are reproducible long enough to capture all the situations. Furthermore, the and so that we can compare easily different algorithms / characteristics of the network may even be time-variant protocols; (see Section IV-A3c). • we need to collect real data on large-scale testbeds corresponding to these scenarii. These measures can then D. Re-injection of real measures 18http://anrg.usc.edu/www/tutornet/ Simulations are often far from the reality, since they are 19https://github.com/apanouso/wsn-indfeat-dataset based on simple models, concerning for instance the PHY 20https://github.com/ftheoleyre/fitiotlab-multichannel-dataset 23

be re-injected to replay the same experiment, so that [7] V. Kotsiou, G. Z. Papadopoulos, P. Chatzimisios, and F. Theoleyre, emulation can then be used to automatize the comparisons “Is Local Blacklisting Relevant in Slow Channel Hopping Low-Power Wireless Networks?,” in Proceedings of the IEEE International Con- (cf section VIII-D). ference on Communications (ICC), 2017. [8] G. Gaillard, D. Barthel, F. Theoleyre, and F. Valois, “Service level IX.CONCLUSIONS agreements for wireless sensor networks: A wsn operator’s point of view,” in 2014 IEEE Network Operations and Management Symposium Validation and verification during the development of pro- (NOMS), pp. 1–8, May 2014. tocols are a matter of prime importance. A large variety of [9] M. R. Palattella, N. Accettura, X. Vilajosana, T. Watteyne, L. A. Grieco, G. Boggia, and M. Dohler, “Standardized protocol stack for practical and theoretical tools has been developed to assist the internet of (important) things,” IEEE Communications Surveys researchers to assess the performance of their solutions. This Tutorials, vol. 15, pp. 1389–1406, Third 2013. tutorial aims at providing guidelines and a list of good prac- [10] V. Karagiannis, P. Chatzimisios, F. Vazquez-Gallego,´ and J. Alonso- Zarate, “A survey on application layer protocols for the internet of tices for researchers involved in evaluating the performance of things,” Transaction on IoT and Cloud Computing,, vol. 1, January protocols for low-power and lossy networks. Several papers 2015. already provide a comprehensive view of existing testbeds [11] R. C. Martin, Agile Software Development: Principles, Patterns, and Practices. Upper Saddle River, NJ, USA: Prentice Hall PTR, 2003. and network simulators, as well as describing some of their [12] G. Z. Papadopoulos, K. Kritsis, A. Gallais, P. Chatzimisios, and individual pitfalls. T. Noel, “Performance evaluation methods in ad hoc and wireless Thus, we here adopted an approach focused rather on sensor networks: A literature study,” IEEE Communications Magazine, vol. 54, no. 1, pp. 122–128, 2016. the methodology and in particular what should a researcher [13] I. Stojmenovic, “Simulations in Wireless Sensor and Ad Hoc Networks: take into consideration when evaluating performance of an Matching and Advancing Models, Metrics, and Solutions,” IEEE Com- algorithm. Therefore, we reviewed various parameters that munications Magazine, vol. 46, no. 12, pp. 102–107, 2008. [14] K. Langendoen, A. Baggio, and O. Visser, “Murphy loves potatoes: should be considered during such performance evaluation. experiences from a pilot sensor network deployment in precision Furthermore, we detailed the general approach adopted to agriculture.,” IPDPS 2006, p. 8 pp., 2006. evaluate the performance of networking protocols (i.e., layers [15] S. Lohs, J. Nolte, G. Siegemund, and V. Turau, “Self-stabilization - a mechanism to make networked embedded systems more reliable?,” 2 and 3) for low-power and lossy networks. In particular, in Symposium on Reliable Distributed Systems (SRDS), pp. 317–326, we have tried to provide detailed information that could IEEE, Sept 2016. help answering questions like “How should a performance [16] V. Vazirani, Approximation Algorithms. Springer, 2002. [17] P. Bremaud, Markov Chains: Gibbs Fields, Monte Carlo Simulation, evaluation be started?” or “What are the different choices and Queues. Springer, 2008. provided to evaluate performance and the different steps to [18] R. Soua, P. Minet, and E. Livolant, “Modesa: An optimized multichan- be followed in order to definitely validate a contribution?”. nel slot assignment for raw data convergecast in wireless sensor networks,” in Performance Computing and Communications Conference Therefore, we especially surveyed numerous models and tools (IPCCC), 2012 IEEE 31st International, pp. 91–100, Dec 2012. that are available to the research community. [19] M. R. Palattella, N. Accettura, M. Dohler, L. A. Grieco, and G. Boggia, Typically, such a tutorial article makes sense for any re- “Traffic aware scheduling algorithm for reliable low-power multihop ieee 802.15.4e networks,” in Personal Indoor and Mobile Radio searcher (i.e., MSc or PhD student, engineer) starting in this Communications (PIMRC), 2012 IEEE 23rd International Symposium research area. By reading this paper, one could avoid the on, pp. 327–332, Sept 2012. common pitfalls that most of the researchers face when they [20] A. Keshavarzian, H. Lee, and L. Venkatraman, “Wakeup scheduling in attempt to carry out a performance evaluation. The recent wireless sensor networks,” in Proceedings of the 7th ACM International Symposium on Mobile Ad Hoc Networking and Computing, MobiHoc huge attention drawn on the reproducibility of the related ’06, (New York, NY, USA), pp. 322–333, ACM, 2006. research domain is nothing else but the willingness to adopt [21] X. Yin and X. Cheng, Propagation Channel Characterization, Param- a methodical approach by considering all the different key eter Estimation, and Modeling for Wireless Communications. Wiley & Sons, 2016. aspects of LLNs. [22] D. Djenouri and M. Bagaa, “Synchronization protocols and implementation issues in wireless sensor networks: A review,” IEEE Systems REFERENCES Journal, vol. 10, pp. 617–627, June 2016. [23] “Network simulator 2 (ns-2).” https://www.isi.edu/nsnam/ns/. [1] J. M. Schleicher, M. Vogler,¨ S. Dustdar, and C. Inzinger, “Application [24] “Network simulator 3 (ns-3).” http://www.isi.edu/nsnam/ns/, 2011. architecture for the internet of cities: Blueprints for future smart city [25] “Omnet++.” https://omnetpp.org/. applications,” IEEE Internet Computing, vol. 20, pp. 68–75, Nov 2016. [26] X. Zeng, R. Bagrodia, and M. Gerla, “Glomosim: A library for parallel [2] B. Ahlgren, M. Hidell, and E. C. H. Ngai, “Internet of things for simulation of large-scale wireless networks,” SIGSIM Simul. Dig., smart cities: Interoperability and open data,” IEEE Internet Computing, vol. 28, pp. 154–161, July 1998. vol. 20, pp. 52–56, Nov 2016. [27] “Riverbed modeler.” https://www.riverbed.com. [3] B. Zhou, J. Cao, X. Zeng, and H. Wu, “Adaptive traffic light control [28] “The fastest, most scalable network modeling platform.” http://web. in wireless sensor network-based intelligent transportation system,” in scalable-networks.com/qualnet-network-simulator. 2010 IEEE 72nd Vehicular Technology Conference - Fall, pp. 1–5, Sept [29] L. Yan and N. McKeown, “Learning networking by reproducing 2010. research results,” SIGCOMM Comput. Commun. Rev., vol. 47, pp. 19– [4] J. Martocci, P. D. Mil, N. Riou, and W. Vermeylen, “Building au- 26, May 2017. tomation routing requirements in low-power and lossy networks,” RFC [30] C. Haas, J. Wilke, and V. Stohr,¨ “Realistic simulation of energy 5687, IETF, 2010. consumption in wireless sensor networks,” in European Conference [5] G. Z. Papadopoulos, J. Beaudaux, A. Gallais, T. Noel, and G. Schreiner, Wireless Sensor Networks (EWSN), (Trento, Italy), pp. 82–97, 2012. “Adding value to WSN simulation using the IoT-LAB experimental [31] J. Polastre, R. Szewczyk, and D. Culler, “Telos: enabling ultra-low platform,” in International Conference on Wireless and Mobile Com- power wireless research,” in Proceedings of the 4th ACM/IEEE Inter- puting, Networking and Communications (WiMob), IEEE, 2013. national Conference on Information Processing in Sensor Networks [6] G. Z. Papadopoulos, A. Gallais, G. Schreiner, and T. Noel,¨ “Importance (IPSN), pp. 364–369, 2005. of Repeatable Setups for Reproducible Experimental Results in IoT,” in [32] F. Osterlind, A. Dunkels, J. Eriksson, N. Finne, and T. Voigt, “Cross- Proceedings of the 13th ACM Symposium on Performance Evaluation Level Sensor Network Simulation with COOJA,” in Proceedings of of Wireless Ad Hoc, Sensor, and Ubiquitous Networks (PE-WASUN the 31st Annual IEEE International Conference on Local Computer ’16), pp. 51–59, 2016. Networks (LCN), pp. 641–648, 2006. 24

[33] T. Watteyne, X. Vilajosana, B. Kerkez, F. Chraim, K. Weekly, Q. Wang, [55] T. Clausen, et al., “The lightweight on-demand ad hoc distance-vector S. Glaser, and K. Pister, “Openwsn: a standards-based low-power wire- routing protocol - next generation (loadng),” draft 15, IETF, 2016. less development environment,” Transactions on Emerging Telecommu- [56] H. Huang, H. Yin, Y. Luo, X. Zhang, G. Min, and Q. Fan, “Three- nications Technologies, vol. 23, no. 5, pp. 480–493, 2012. dimensional geographic routing in wireless mobile ad hoc and sensor [34] L. M. Feeney and M. Nilsson, “Investigating the energy consumption networks,” IEEE Network, vol. 30, pp. 82–90, March 2016. of a wireless network interface in an ad hoc networking environment,” [57] M. O. Farooq, C. J. Sreenan, K. N. Brown, and T. Kunz, “Design and in INFOCOM 2001. Twentieth Annual Joint Conference of the IEEE analysis of rpl objective functions for multi-gateway ad-hoc low-power Computer and Communications Societies. Proceedings. IEEE, vol. 3, and lossy networks,” Ad Hoc Networks, vol. 65, pp. 78 – 90, 2017. pp. 1548–1557 vol.3, 2001. [58] A. A. K. Somappa, K. Øvsthus, and L. M. Kristensen, “An industrial [35] I. Dietrich and F. Dressler, “On the lifetime of wireless sensor net- perspective on wireless sensor networks: A survey of requirements, works,” ACM Transactions on Sensor Networks (TOSN), vol. 5, pp. 1– protocols, and challenges,” IEEE Communications Surveys Tutorials, 39, Feb. 2009. vol. 16, pp. 1391–1412, Third 2014. [36] D. Tian and N. D. Georganas, “A coverage-preserving node scheduling [59] P. Kampstra, “Beanplot: A boxplot alternative for visual comparison scheme for large wireless sensor networks,” in Proceedings of the of distributions,” Journal of Statistical Software, 2008. 1st ACM International Workshop on Wireless Sensor Networks and [60] F. Theoleyre and G. Z. Papadopoulos, “Experimental Validation of Applications, WSNA ’02, (New York, NY, USA), pp. 32–41, ACM, a Distributed Self-Configured 6TiSCH with Traffic Isolation in Low 2002. Power Lossy Networks ,” in Proceedings of the 19th ACM International [37] M. Saleem, I. Ullah, and M. Farooq, “Beesensor: An energy-efficient Conference on Modeling, Analysis and Simulation of Wireless and and scalable routing protocol for wireless sensor networks,” Informa- Mobile Systems (MSWiM), 2016. tion Sciences, vol. 200, pp. 38 – 56, 2012. [61] S. Chessa and P. Santi, “Crash faults identification in wireless sensor [38] G. Z. Papadopoulos, V. Kotsiou, A. Gallais, P. Chatzimisios, and networks,” Computer Communications, vol. 25, no. 14, pp. 1273 – T. Noel, “Low-Power Neighbor Discovery for Mobility-Aware Wireless 1282, 2002. Sensor Networks,” Elsevier Ad Hoc Networks, vol. 48, pp. 66–79, 2016. [62] M. Eskola and T. Heikkila,¨ “Classification of radio channel distur- [39] P. Huang, L. Xiao, S. Soltani, M. W. Mutka, and N. Xi, “The bances for industrial wireless sensor networks,” Ad Hoc Networks, evolution of mac protocols in wireless sensor networks: A survey,” vol. 42, pp. 19 – 33, 2016. IEEE Communications Surveys Tutorials, vol. 15, pp. 101–120, First [63] O. Iova, F. Theoleyre, and T. Noel, “Stability and efficiency of rpl 2013. under realistic conditions in wireless sensor networks,” in International [40] G. Z. Papadopoulos, J. Beaudaux, A. Gallais, and T. Noel, “T-AAD: Symposium on Personal, Indoor, and Mobile Radio Communications Lightweight Traffic Auto-ADaptations for Low-power MAC Proto- (PIMRC), pp. 2098–2102, IEEE, Sept 2013. cols,” in Proceedings of the 13th IEEE IFIP Annual Mediterranean [64] M. Senel, K. Chintalapudi, D. Lal, A. Keshavarzian, and E. J. Coyle, “A Ad Hoc Networking Workshop (MED-HOC-NET), pp. 79–86, 2014. kalman filter based link quality estimation scheme for wireless sensor [41] A. Dunkels, “The contikimac radio duty cycling protocol,” Tech. Rep. networks,” in Global Telecommunications Conference (GLOBECOM), T2011:13, SICS, 2011. pp. 875–880, IEEE, Nov 2007. [42] A. Mavromatis, G. Z. Papadopoulos, X. Fafoutis, A. Elsts, [65] E. Schiller, P. Starzetz, F. Theoleyre, and A. Duda, “Properties of G. Oikonomou, and T. Tryfonas, “Impact of Guard Time Length on greedy geographical routing in spontaneous wireless mesh networks,” IEEE 802.15.4e TSCH Energy Consumption,” in Proceedings of the in Global Telecommunications Conference, 2007. GLOBECOM ’07. 13th Annual IEEE International Conference on Sensing, Communica- IEEE, pp. 4941–4945, Nov 2007. tion, and Networking (SECON), pp. 1–3, 2016. [66] F. Tobagi and L. Kleinrock, “Packet switching in radio channels: Part [43] G. Z. Papadopoulos, A. Mavromatis, X. Fafoutis, N. Montavont, ii–the hidden terminal problem in carrier sense multiple-access and the R. Piechocki, T. Tryfonas, and G. Oikonomou, “Guard Time Optimisa- busy-tone solution,” IEEE Transactions on Communications, vol. 23, tion and Adaptation for Energy Efficient Multi-hop TSCH Networks,” pp. 1417–1433, Dec 1975. in Proceedings of the IEEE 3rd World Forum on Internet of Things [67] C. Chaudet, D. Dhoutaut, and I. G. Lassous, “Performance issues with (WF-IoT), pp. 301–306, 2016. ieee 802.11 in ad hoc networking,” IEEE Communications Magazine, [44] “IEEE Standard for Low-Rate Wireless Personal Area Networks (LR- vol. 43, pp. 110–116, July 2005. WPANs).” IEEE Std 802.15.4-2015 (Revision of IEEE Std 802.15.4- [68] O. Iova, F. Theoleyre, and T. Noel, “Using multiparent routing in 2011), April 2016. {RPL} to increase the stability and the lifetime of the network,” Ad [45] L. Gu and J. A. Stankovic, “Radio-triggered wake-up for wireless Hoc Networks, vol. 29, pp. 45 – 62, 2015. sensor networks,” Real-Time Systems, vol. 29, no. 2, pp. 157–182, 2005. [69] F. Stann, J. Heidemann, R. Shroff, and M. Z. Murtaza, “Rbp: Robust [46] D. Giustiniano, D. Malone, D. J. Leith, and K. Papagiannaki, “Mea- broadcast propagation in wireless networks,” in International Confer- suring transmission opportunities in 802.11 links,” IEEE/ACM Trans- ence on Embedded Networked Sensor Systems (SenSys), pp. 85–98, actions on Networking, vol. 18, pp. 1516–1529, Oct 2010. ACM, 2006. [47] F. Hermans, O. Rensfelt, T. Voigt, E. Ngai, L. A. Norden, and [70] C. H. Lo and N. Ansari, “The progressive smart grid system from both P. Gunningberg, “Sonic: Classifying interference in 802.15.4 sensor power and communications aspects,” IEEE Communications Surveys networks,” in Information Processing in Sensor Networks (IPSN), 2013 Tutorials, vol. 14, pp. 799–821, Third 2012. ACM/IEEE International Conference on, pp. 55–66, April 2013. [71] R. D. Gomes, D. V. Queiroz, A. C. L. Filho, I. E. Fonseca, and [48] R. Jain, D. Chiu, and W. Hawe, “A quantitative measure of fairness M. S. Alencar, “Real-time link quality estimation for industrial wireless and discrimination for resource allocation in shared computer systems,” sensor networks using dedicated nodes,” Ad Hoc Networks, vol. 59, Technical Report DEC-TR-301, Digital Equipment Corporation, 1984. pp. 116 – 133, 2017. [49] F. Tobagi and L. Kleinrock, “Packet switching in radio channels: Part ii [72] Y. Chen and A. Terzis, “On the implications of the log-normal path - the hidden terminal problem in carrier sense multiple-access and the loss model: An efficient method to deploy and move sensor motes,” in busy-tone solution,” IEEE Transactions on Communications, vol. 23, Conference on Embedded Networked Sensor Systems (SenSys), pp. 26– pp. 1417–1433, December 1975. 39, ACM, 2011. [50] R. P. Liu, Z. Rosberg, I. B. Collings, C. Wilson, A. Y. Dong, and S. Jha, [73] A. Angles-V` azquez,´ X. Vilajosana-Guillen,` J. Lopez-Vicario,´ “Overcoming radio link asymmetry in wireless sensor networks,” in A. Morell-Perez,´ P. Tuset-Peiro,´ and I. Vilajosana-Guillen,` “Generic Personal, Indoor and Mobile Radio Communications, 2008. PIMRC empiric propagation model for low power wireless networks operating 2008. IEEE 19th International Symposium on, pp. 1–5, Sept 2008. at the 868 mhz band in smart cities,” IET Microwaves, Antennas [51] G. Z. Papadopoulos, J. Beaudaux, A. Gallais, P. Chatzimisios, and Propagation, vol. 8, no. 14, pp. 1143–1153, 2014. T. Noel, “Toward a Packet Duplication Control for Opportunistic Rout- [74] A. Iyer, C. Rosenberg, and A. Karnik, “What is the right model ing in WSNs,” in Proceedings of the IEEE Global Communications for wireless channel interference?,” IEEE Transactions on Wireless Conference (GLOBECOM), pp. 94–99, 2014. Communications, vol. 8, pp. 2662–2671, May 2009. [52] S. Biswas and R. Morris, “Exor: Opportunistic multi-hop routing for [75] K. Srinivasan, M. A. Kazandjieva, S. Agarwal, and P. Levis, “The β- wireless networks,” in SIGCOMM, pp. 133–144, 2005. factor: Measuring wireless link burstiness,” in Proceedings of the 6th [53] N. A. Pantazis, S. A. Nikolidakis, and D. D. Vergados, “Energy- ACM Conference on Embedded Network Sensor Systems, SenSys ’08, efficient routing protocols in wireless sensor networks: A survey,” IEEE (New York, NY, USA), pp. 29–42, ACM, 2008. Communications Surveys Tutorials, vol. 15, pp. 551–591, Second 2013. [76] A. Cerpa, J. L. Wong, M. Potkonjak, and D. Estrin, “Temporal [54] T. Winter, et al., “RPL: IPv6 Routing Protocol for Low-Power and Properties of Low Power Wireless Links: Modeling and Implications Lossy Networks,” rfc 6550, IETF, 2012. on Multi-Hop Routing,” in International Symposium on Mobile Ad 25

Hoc Networking and Computing (MOBIHOC), (New York, New York, sensor networks,” in International Conference on Industrial Informatics USA), pp. 414–425, ACM Press, 2005. (INDIN), pp. 625–630, IEEE, July 2016. [77] G. Zhou, T. He, S. Krishnamurthy, and J. A. Stankovic, “Models [98] A. A. Aziz, Y. A. Sekercioglu, P. Fitzpatrick, and M. Ivanovich, “A and solutions for radio irregularity in wireless sensor networks,” ACM survey on distributed topology control techniques for extending the Trans. Sen. Netw., vol. 2, pp. 221–262, May 2006. lifetime of battery powered wireless sensor networks,” IEEE Commu- [78] J. Padhye, S. Agarwal, V. N. Padmanabhan, L. Qiu, A. Rao, and B. Zill, nications Surveys Tutorials, vol. 15, pp. 121–144, First 2013. “Estimation of link interference in static multi-hop wireless networks,” [99] W. Ejaz, M. Naeem, A. Shahid, A. Anpalagan, and M. Jo, “Efficient in SIGCOMM Conference on Internet Measurement (IMC), pp. 28–28, energy management for the internet of things in smart cities,” IEEE ACM, 2005. Communications Magazine, vol. 55, pp. 84–91, January 2017. [79] Y.-C. Tseng, S.-Y. Ni, Y.-S. Chen, and J.-P. Sheu, “The broadcast [100] Y.-H. Chen, B. Ng, W. K. Seah, and A.-C. Pang, “Modeling and storm problem in a mobile ad hoc network,” Wireless Networks, vol. 8, analysis: Energy harvesting in the internet of things,” in International pp. 153–167, March 2002. Conference on Modeling, Analysis and Simulation of Wireless and [80] J. N. Al-Karaki and A. E. Kamal, “Routing techniques in wireless Mobile Systems (MSWiM), pp. 156–165, ACM, 2016. sensor networks: a survey,” IEEE Wireless Communications, vol. 11, [101] A. S. Weddell, G. V. Merrett, T. J. Kazmierski, and B. M. Al- pp. 6–28, Dec 2004. Hashimi, “Accurate supercapacitor modeling for energy harvesting [81] C. Intanagonwiwat, R. Govindan, D. Estrin, J. Heidemann, and F. Silva, wireless sensor nodes,” IEEE Transactions on Circuits and Systems “Directed diffusion for wireless sensor networking,” IEEE/ACM Trans. II: Express Briefs, vol. 58, pp. 911–915, Dec 2011. Netw., vol. 11, pp. 2–16, Feb. 2003. [102] A. Biason and M. Zorzi, “On the effects of battery imperfections in an [82] P. Thubert, “An Architecture for IPv6 over the TSCH mode of IEEE energy harvesting device,” in International Conference on Computing, 802.15.4.” draft-ietf-6tisch-architecture-11, July 2017. Networking and Communications (ICNC), pp. 1–7, IEEE, Feb 2016. [83] C.-Y. Wan, S. B. Eisenman, A. T. Campbell, and J. Crowcroft, “Siphon: [103] A. Stajkic, M. D. Abrignani, C. Buratti, A. Bettinelli, D. Vigo, and Overload traffic management using multi-radio virtual sinks in sensor R. Verdone, “From a real deployment to a downscaled testbed: A networks,” in Proceedings of the 3rd International Conference on methodological approach,” IEEE Internet of Things Journal, vol. 3, Embedded Networked Sensor Systems, SenSys ’05, (New York, NY, pp. 647–657, Oct 2016. USA), pp. 116–129, ACM, 2005. [104] D. Kotz, C. Newport, R. S. Gray, J. Liu, Y. Yuan, and C. Elliott, [84] G. Z. Papadopoulos, N. Pappas, A. Gallais, T. Noel, and V. Angelakis, “Experimental evaluation of wireless simulation assumptions,” in Inter- “Distributed Adaptive Scheme for Reliable Data Collection in Fault national Symposium on Modeling, Analysis and Simulation of Wireless Tolerant WSNs,” in Proceedings of the 2nd IEEE World Forum on and Mobile Systems (MSWiM), pp. 78–82, ACM, 2004. Internet of Things (WF-IoT), pp. 116–121, 2015. [105] I. Stojmenovic, “Simulations in Wireless Sensor and Ad Hoc Networks: [85] A. Karnik, A. Iyer, and C. Rosenberg, “Throughput-optimal configu- Matching and Advancing Models, Metrics, and Solutions,” IEEE Com- ration of fixed wireless networks,” IEEE/ACM Trans. Netw., vol. 16, munications Magazine, vol. 46, no. 12, pp. 102–107, 2008. pp. 1161–1174, Oct. 2008. [106] “Wsnet.” http://wsnet.gforge.inria.fr/. [86] J. Chen, X. Cao, P. Cheng, Y. Xiao, and Y. Sun, “Distributed [107] “ns3-coap.” https://github.com/maesoser/ns3-coap. collaborative control for industrial automation with wireless sensor [108] H. Kermajani and C. Gomez, “On the network convergence process in and actuator networks,” IEEE Transactions on Industrial Electronics, rpl over ieee 802.15.4 multihop networks: Improvement and trade-offs,” vol. 57, pp. 4219–4230, Dec 2010. Sensors, vol. 14, no. 7, pp. 11993–12022, 2014. [87] M. Szczodrak, O. Gnawali, and L. P. Carloni, “Dynamic reconfiguration [109] V. Kathuria, G. Mohanasundaram, and S. R. Das, “A simulation study of wireless sensor networks to support heterogeneous applications,” of routing protocols for smart meter networks,” in International Confer- in 2013 IEEE International Conference on Distributed Computing in ence on Smart Grid Communications (SmartGridComm), pp. 384–389, Sensor Systems, pp. 52–61, May 2013. IEEE, Oct 2013. [88] M. Chen, J. Wan, S. Gonzalez, X. Liao, and V. C. M. Leung, ¨ “A survey of recent developments in home m2m networks,” IEEE [110] N. Abdeddaım and F. Theoleyre, “Implementation of a WSNet Module Communications Surveys Tutorials, vol. 16, pp. 98–114, First 2014. to Simulate the IEEE 802.15.4 Beacon-Enabled Mode in Multihop Topologies,” tech. rep., HAL, 2011. https://hal.archives-ouvertes.fr/ [89] C. M. de Farias, L. Pirmez, F. C. Delicato, P. F. Pires, W. Li, A. Y. hal-00590853. Zomaya, E. N. de L. F. Jorge, and R. Juarez-Ramirez, “Grown: A control and decision system for smart greenhouses using wireless sensor [111] L. B. Saad, C. Chauvenet, and B. Tourancheau, “Simulation of the networks,” in Australasian Computer Science Week Multiconference rpl routing protocol for ipv6 sensor networks: two cases studies,” in (ACSW), pp. 48:1–48:8, ACM, 2017. SENSORCOMM, 2011. https://hal.inria.fr/hal-00647869. [90] D. C. Harrison, W. K. G. Seah, and R. Rayudu, “Rare event detection [112] “Pyviz - ns3.” https://www.nsnam.org/wiki/PyViz. and propagation in wireless sensor networks,” ACM Comput. Surv., [113] M. P. Uwase, M. Bezunartea, J. Tiberghien, J. M. Dricot, and K. Steen- vol. 48, pp. 58:1–58:22, Mar. 2016. haut, “Experimental comparison of radio duty cycling protocols for [91] L. Wang, D. Zhang, Y. Wang, C. Chen, X. Han, and A. M’hamed, wireless sensor networks,” IEEE Sensors Journal, vol. 17, pp. 6474– “Sparse mobile crowdsensing: challenges and opportunities,” IEEE 6482, Oct 2017. Communications Magazine, vol. 54, pp. 161–167, July 2016. [114] Castalia, “Castalia - Wireless Sensor Network Simulator.” https:// [92] V. Shnayder, M. Hempstead, B.-r. Chen, G. W. Allen, and M. Welsh, castalia.forge.nicta.com.au/index.php/en/index.html, 2013. “Simulating the power consumption of large-scale sensor network [115] E. Ben Hamida et al., “On the Complexity of an Accurate and Precise applications,” in Proceedings of the 2Nd International Conference on Performance Evaluation of Wireless Networks using Simulations,” in Embedded Networked Sensor Systems, SenSys ’04, (New York, NY, MSWiM, ACM, 2008. USA), pp. 188–200, ACM, 2004. [116] U. M. Colesanti, C. Crociani, and A. Vitaletti, “On the accuracy [93] J. Kim and D. K. Noh, “Voltage-based estimation of residual battery of omnet++ in the wireless sensor networks domain: simulation vs. energy in wireless sensor systems,” in SENSORS, pp. 1–4, IEEE, Nov testbed,” in Proceedings of the 4th ACM workshop on performance 2013. evaluation of wireless ad hoc, sensor, and ubiquitous networks, pp. 25– [94] W. R. Heinzelman, A. Chandrakasan, and H. Balakrishnan, “Energy- 31, ACM, 2007. efficient communication protocol for wireless microsensor networks,” [117] M. Yarvis, N. Kushalnagar, H. Singh, A. Rangarajan, Y. Liu, and in Proceedings of the 33rd Annual Hawaii International Conference S. Singh, “Exploiting heterogeneity in sensor networks,” in Proceedings on System Sciences, pp. 10 pp. vol.2–, Jan 2000. IEEE 24th Annual Joint Conference of the IEEE Computer and [95] J. Polastre, J. Hill, and D. Culler, “Versatile low power media access Communications Societies., vol. 2, pp. 878–890 vol. 2, March 2005. for wireless sensor networks,” in Proceedings of the 2Nd International [118] A.-S. Tonneau, N. Mitton, and J. Vandaele, “How to choose an Conference on Embedded Networked Sensor Systems, SenSys ’04, experimentation platform for wireless sensor networks? a survey on (New York, NY, USA), pp. 95–107, ACM, 2004. static and mobile wireless sensor network experimentation facilities,” [96] Q. Wang, M. Hempstead, and W. Yang, “A realistic power consumption Ad Hoc Netw., vol. 30, pp. 115–127, July 2015. model for wireless sensor network devices,” in Communications Society [119] University of Utah, “Emulab - Network Emulation Testbed.” http:// on Sensor and Ad Hoc Communications and Networks (SECON), vol. 1, www.emulab.net/index.php3, 2017. pp. 286–295, IEEE, Sept 2006. [120] O. Fambon, E. Fleury, G. Harter, R. Pissard-Gibollet, and F. Saint- [97]I.M uller,¨ J. Winter, C. Pereira, V. Brusamarello, and J. C. Netto, Marcel, “Fit iot-lab tutorial: hands-on practice with a very large scale “Energy consumption estimation for tdma-based industrial wireless testbed tool for the internet of things,” in UbiMob, 2014. 26

[121] R. Lim, F. Ferrari, M. Zimmerling, C. Walser, P. Sommer, and Conference on Local Computer Networks (LCN), pp. 455–462, IEEE, J. Beutel, “Flocklab: A testbed for distributed, synchronized tracing Nov 2004. and profiling of wireless embedded systems,” in Proceedings of the [143] J. Eriksson, F. Osterlind,¨ N. Finne, N. Tsiftes, A. Dunkels, T. Voigt, 12th International Conference on Information Processing in Sensor R. Sauter, and P. J. Marron,´ “Cooja/mspsim: Interoperability testing for Networks, IPSN ’13, (New York, NY, USA), pp. 153–166, ACM, 2013. wireless sensor networks,” in International Conference on Simulation [122] M. Doddavenkatappa, M. C. Chan, and A. L. Ananda, “Indriya: A low- Tools and Techniques (Simutools), pp. 27:1–27:7, 2009. cost, 3d wireless sensor network testbed,” in International Conference [144] P. Levis, N. Lee, M. Welsh, and D. Culler, “Tossim: Accurate and on Testbeds and Research Infrastructures for the Development of Net- scalable simulation of entire tinyos applications,” in International Con- works & Communities (TRIDENTCOM), (Shanghai, China), pp. 302– ference on Embedded Networked Sensor Systems (SenSys), pp. 126– 316, EAI, 2012. 137, ACM, 2003. [123] B. N. Chun, P. Buonadonna, A. AuYoung, C. Ng, D. C. Parkes, [145] A. Fraboulet, G. Chelius, and E. Fleury, “Worldsens: Development and J. Shneidman, A. C. Snoeren, and A. Vahdat, “Mirage: A microeco- prototyping tools for application specific wireless sensors networks,” nomic resource allocation system for sensornet testbeds,” in The Second in 2007 6th International Symposium on Information Processing in IEEE Workshop on Embedded Networked Sensors, 2005. EmNetS-II., Sensor Networks, pp. 176–185, April 2007. pp. 19–28, May 2005. [146] G. Z. Papadopoulos, A. Gallais, G. Schreiner, E. Jou, and T. Noel, [124] A. Arora, E. Ertin, R. Ramnath, M. Nesterenko, and W. Leal, “Kansei: “Thorough IoT testbed Characterization: from Proof-of-concept to a high-fidelity sensing testbed,” IEEE Internet Computing, vol. 10, Repeatable Experimentations,” Elsevier Computer Networks, vol. 119, pp. 35–47, March 2006. pp. 86–101, 2017. [125] Purdue University, “Purdue University Wireless Mesh Network [147] B. Raman, K. Chebrolu, D. Gokhale, and S. Sen, “On the feasibility Testbed.” https://engineering.purdue.edu/MESH, 2017. of the link abstraction in wireless mesh networks,” IEEE/ACM Trans. [126] G. Werner-Allen, P. Swieskowski, and M. Welsh, “Motelab: a wireless Netw., vol. 17, pp. 528–541, Apr. 2009. sensor network testbed,” in IPSN 2005. Fourth International Sympo- [148] “A community resource for archiving wireless data at dartmouth.” http: sium on Information Processing in Sensor Networks, 2005., pp. 483– //crawdad.org. 488, April 2005. [149] G. Coulson, B. Porter, I. Chatzigiannakis, C. Koninis, S. Fischer, [127] X. Ju, H. Zhang, and D. Sakamuri, “Neteye: A user-centered wireless D. Pfisterer, D. Bimschas, T. Braun, P. Hurni, M. Anwander, G. Wa- sensor network testbed for high-fidelity, robust experimentation,” Int. genknecht, S. P. Fekete, A. Kroller,¨ and T. Baumgartner, “Flexible J. Commun. Syst., vol. 25, pp. 1213–1229, Sept. 2012. experimentation in wireless sensor networks,” Commun. ACM, vol. 55, [128] Rutgers University, “ORBIT – Open-Access Research Testbed for pp. 82–90, Jan. 2012. Next-Generation Wireless Networks.” http://www.orbit-lab.org, 2017. [150] P. Juang, H. Oki, Y. Wang, M. Martonosi, L. S. Peh, and D. Rubenstein, [129] University of South California, “Tutornet: A Low Power Wireless IoT “Energy-efficient computing for wildlife tracking: Design tradeoffs and Testbed.” http://anrg.usc.edu/www/tutornet/, 2017. early experiences with zebranet,” SIGARCH Comput. Archit. News, [130] V. Handziski, A. Kopke,¨ A. Willig, and A. Wolisz, “Twist: A scalable vol. 30, pp. 96–107, Oct. 2002. and reconfigurable testbed for wireless indoor experiments with sensor [151] F. Ingelrest, G. Barrenetxea, G. Schaefer, M. Vetterli, O. Couach, and networks,” in Proceedings of the 2Nd International Workshop on Multi- M. Parlange, “Sensorscope: Application-specific sensor network for ACM Trans. Sen. Netw. hop Ad Hoc Networks: From Theory to Reality, REALMAN ’06, (New environmental monitoring,” , vol. 6, pp. 17:1– York, NY, USA), pp. 63–70, ACM, 2006. 17:32, Mar. 2010. [152] R. T. Hermeto, A. Gallais, K. Van Laerhoven, and F. Theoleyre, “Pas- [131] H. Soroush, N. Banerjee, M. Corner, B. Levine, and B. Lynn, “A sive and Stable Initial Preferred Parent Selection through Neighbors retrospective look at the umass dome mobile testbed,” SIGMOBILE Ranking for a Fast Convergence in 6TiSCH Networks,” in European Mob. Comput. Commun. Rev., vol. 15, pp. 2–15, Mar. 2012. Wireless Sensor Networks (EWSN), ACM, 2018. [132] N. Anand, E. Aryafar, and E. W. Knightly, “Warplab: A flexible [153] D. Cavin, Y. Sasson, and A. Schiper, “On the accuracy of manet framework for rapid physical layer design,” in Proceedings of the 2010 simulators,” in Proceedings of the Second ACM International Workshop ACM Workshop on Wireless of the Students, by the Students, for the on Principles of Mobile Computing, POMC ’02, (New York, NY, USA), Students, S3 ’10, (New York, NY, USA), pp. 53–56, ACM, 2010. pp. 38–43, ACM, 2002. [133] I. Chatzigiannakis, S. Fischer, C. Koninis, G. Mylonas, and D. Pfisterer, [154] P. H. Gomes, Y. Chen, T. Watteyne, and B. Krishnamachari, “Insights Sensor Applications, Experimentation, and Logistics: First Interna- into frequency diversity from measurements on an indoor low power tional Conference, SENSAPPEAL 2009, Athens, Greece, September wireless network testbed,” in 2016 IEEE Globecom Workshops (GC 25, 2009, Revised Selected Papers , ch. WISEBED: An Open Large- Wkshps), pp. 1–6, Dec 2016. Scale Wireless Sensor Network Testbed, pp. 68–87. Berlin, Heidelberg: [155] A. Brandt, J. Buron, and G. Porcu, “Home automation routing require- Springer Berlin Heidelberg, 2010. ments in low-power and lossy networks,” RFC 5826, IETF, 2010. [134] C. Cano, A. Pittolo, D. Malone, L. Lampe, A. M. Tonello, and [156] K. Pister, P. Thubert, S. Dwars, and T. Phinney, “Industrial routing A. G. Dabak, “State of the art in power line communications: From requirements in low-power and lossy networks,” RFC 5673, IETF, the applications to the medium,” IEEE Journal on Selected Areas in 2009. Communications, vol. 34, pp. 1935–1952, July 2016. [157] “Spectrum Requirements for Short Range Device, Metropolitan Mesh [135] O. Hahm, E. Baccelli, H. Petersen, and N. Tsiftes, “Operating systems Machine Networks (M3N) and Smart Metering (SM) applications,” for low-end devices in the internet of things: A survey,” IEEE Internet ETSI TC ERM, TR 103 055, v1.1.1, Sept. 2011. of Things Journal, vol. 3, pp. 720–734, Oct 2016. [136] P. Levis, S. Madden, J. Polastre, R. Szewczyk, A. Woo, D. Gay, J. Hill, M. Welsh, E. Brewer, and D. Culler, “Tinyos: An operating system for sensor networks,” in in Ambient Intelligence, Springer Verlag, 2004. [137] A. Dunkels, O. Schmidt, T. Voigt, and M. Ali, “Protothreads: Simpli- fying Event-driven Programming of Memory-constrained Embedded Systems,” in Proceedings of the 4th International Conference on Embedded Networked Sensor Systems (SenSys), pp. 29–42, 2006. [138] A. Dunkels, B. Gronvall, and T. Voigt, “Contiki - a Lightweight and Flexible Operating System for Tiny Networked Sensors,” in Proceedings of the 29th Annual IEEE International Conference on Local Computer Networks (LCN), pp. 455–462, 2004. [139] “CST Group at FU Berlin. Scatter web Embedded Sensor Board.” http: //www.csc.kth.se/∼ronniej/project/Scatterweb/ESB.html. [140] H. Will, K. Schleiser, and J. Schiller, “A real-time kernel for wireless sensor networks employed in rescue scenarios,” in 2009 IEEE 34th Conference on Local Computer Networks, pp. 834–841, 2009. [141] “IPv6 over the TSCH mode of IEEE 802.15.4e.” https://datatracker. ietf.org/wg/6tisch. [142] A. Dunkels, B. Gronvall, and T. Voigt, “Contiki - a lightweight and flexible operating system for tiny networked sensors,” in International 27

Kosmas Kritsis is a Research Associate at the In- stitute for Language and Speech Processing, Athena Periklis Chatzimisios (S02-M05-SM12) received Research and Innovation Center (ILSP/ATHENA the B.Sc. degree in informatics from Alexander RC). He is also a Ph.D. Candidate at the Dept. of Technological Educational Institute of Thessaloniki Informatics, University of Piraeus (UniPi). He holds (ATEITHE), Thessaloniki, Greece, in 2000, and a M.Sc. in Sound & Music Computing from Pompeu the Ph.D. degree in wireless communications from Fabra University of Barcelona (UPF) and a B.Sc. Bournemouth University, Poole, U.K., in 2005. He in Informatics from the Alexander Technological serves as an Associate Professor and the Director Educational Institute of Thessaloniki (ATEITHE). of the Computing Systems, Security and Networks During his studies he was admitted twice the Eras- (CSSN) Research Laboratory in the Department of mus EU scholarship for studying at the Computer Informatics, ATEITHE. He has edited/authored eight Science Dept. of Carlos III University of Madrid (UC3M). Additionally, he books and more than 130 peer-reviewed papers and received the Erasmus Placement EU scholarship as a student researcher at the book chapters. His published research work has received more than 2800 ICube lab, University of Strasbourg (UNISTRA). His academic experience and citations by other researchers. His research interests include performance research interest includes diverse scientiﬁc areas, such as computer networks, evaluation and standardization activities of mobile/wireless communications, signal processing, computer vision, Human-Computer Interaction and machine Internet of Things, and big data. Dr. Chatzimisios is involved in several learning. standardization and IEEE activities serving as a member of the Standards Development Board for the IEEE Communication Society (ComSoc), the IEEE ComSoc Standards Program Development Board, and the IEEE ComSoc Education and Training Board, as well as the Vice Chair of the IEEE ComSoc Technical Committees on Big Data (TCBD) and Information Infrastructure Georgios Z. Papadopoulos (S10-M16) serves as and Networking (TCIIN). an Associate Professor at the IMT Atlantique in Rennes, France. Previously, he was a Postdoctoral Researcher at the University of Bristol. He received his Ph.D. from University of Strasbourg, in 2015 with honors, his M.Sc. in Telematics Engineering from University Carlos III of Madrid in 2012 and his B.Sc. in Informatics from Alexander T.E.I. of Thessaloniki in 2011. Dr. Papadopoulos has par- ticipated in various international and national (FP7 RERUM, FIT Equipex) research projects. Moreover, he has received the prestigious French national ANR JCJC grant for young researchers. He has been involved in the organization of many international events (AdHoc-Now18, IEEE CSCN18, IEEE ISCC17). His research interests include Industrial IoT, 6TiSCH, LPWAN, Battery Management System and Smart Grid. Dr. Papadopoulos has received the Best Ph.D. Thesis Award granted by the University of Strasbourg and he was a recipient of two Best Paper Awards (IFIP Med-Hoc-Net14 and IEEE SENSORS14).

Antoine Gallais is an Associate Professor at the University of Strasbourg since 2008 (ICube laboratory). He holds a master (2004), a PhD degree (2007) Fabrice Tholeyre (S05-M09-SM16) is a researcher in computer science from the University of Lille, at the CNRS. After having spent 2 years in the and an HDR thesis (2017) from Univ. Strasbourg. Grenoble Informatics Laboratory (France), he is part Since sept. 2017, he is a visiting researcher at Inria of the ICube lab (Strasbourg, France) since 2009. He Lille - Nord Europe. His research topics include received his PhD in computer science from INSA, wireless sensor and mobile ad hoc networking, mo- Lyon (France) in 2006. He was a visiting scholar bility management and performance evaluation. He at the University of Waterloo (Canada) in 2006, is serving as TPC member for several events (e.g., and a visiting researcher at INRIA Sophia Antipolis IEEE COMNETSAT, IEEE ICNC, IEEE Globe- (France) in 2005. He serves in about ten TPC com), was program co-chair of ICST Adhocnets’14-15 and local co-chair per year, and is area editor for Ad Hoc Networks of IEEE Wimob’13. He regularly serves as an external reviewer for several since 2018. He has been associate editor for IEEE international journals an conferences and was an active member of several Communications Letters and guest editor for Computer Communications and national and international research projects (ANR TLCOM SensLAB, ANR Eurasip JWCN. His research interests mainly concern distributed algorithms FIT Equipex, ANR INFRA IRIS, PHC EXPRESS). and experimental design for the Internet of Things.