A Tutorial on Performance Evaluation and Validation Methodology for Low-Power and Lossy Networks Kosmas Kritsis, Georgios Papadopoulos, Antoine Gallais, Periklis Chatzimisios, Fabrice Theoleyre To cite this version: Kosmas Kritsis, Georgios Papadopoulos, Antoine Gallais, Periklis Chatzimisios, Fabrice Theoleyre. A Tutorial on Performance Evaluation and Validation Methodology for Low-Power and Lossy Networks. Communications Surveys and Tutorials, IEEE Communications Society, Institute of Electrical and Electronics Engineers, 2018, 20 (3), pp.1799 - 1825. 10.1109/COMST.2018.2820810. hal-01886690 HAL Id: hal-01886690 https://hal.archives-ouvertes.fr/hal-01886690 Submitted on 23 Apr 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. 1 A Tutorial on Performance Evaluation and Validation Methodology for Low-Power and Lossy Networks Kosmas Kritsis, Georgios Z. Papadopoulos, Member, IEEE, Antoine Gallais, Periklis Chatzimisios, Senior Member, IEEE, and Fabrice Theoleyre,´ Senior Member, IEEE, Abstract—Envisioned communication densities in Internet of may be used for counting the number of vehicles, such to Things (IoT) applications are increasing continuously. Because control optimally the street traffic lights and to reduce the these wireless devices are often battery powered, we need waiting time [3]. specific energy efficient (low-power) solutions. Moreover, these smart objects use low-cost hardware with possibly weak links, This type of sensor networks, due to their embedded nature, leading to a lossy network. Once deployed, these Low-power often requires to operate with limited power, constrained mem- Lossy Networks (LLNs) are intended to collect the expected ory and processing resources. Therefore, they require specific measurements, handle transient faults and topology changes, approaches to make the wireless network energy efficient. To etc. Consequently, validation and verification during the protocol do so, the wireless devices have to implement protocols to development are a matter of prime importance. A large range of theoretical or practical tools are available for performance reduce their energy consumption, e.g., by turning off their evaluation. A theoretical analysis may demonstrate that the radio interface for most of the time. performance guarantees are respected, while simulations or Furthermore, multiple environmental factors interfere with experiments aim on estimating the behaviour of a set of protocols the system, while adding extra noise and promoting high within real-world scenarios. In this article, we review the various variances in the communication links. Therefore, the Packet parameters that should be taken into account during such a performance evaluation. Our primary purpose is to provide Error Rate (PER) may be high for some of the radio links, a tutorial that specifies guidelines for conducting performance and requires to design robust algorithms and protocols. In evaluation campaigns of network protocols in LLNs. We detail the particular, we have to make the system reliable even if it general approach adopted in order to evaluate the performance depends on unreliable radio links. The wireless infrastructure of layer 2 and 3 protocols in LLNs. Furthermore, we also specify has to deliver most of the packets to their destinations, the methodology that should be adopted during the performance evaluation, while reviewing the numerous models and tools that without duplicating them (energy efficiency), and by handling are available to the research community. transparently the packets losses at the link layer. Actuators may also be part of the wireless infrastructure Index Terms—Low-power Lossy Networks; Internet of Things; Protocols; Algorithms; Performance Evaluation; Validation; Ex- (e.g., the heating system in smart buildings [4]). To avoid periments; Simulation; Testbeds; Models any ambiguity, we adopt here the Internet Engineering Task Force (IETF) terminology, which designates a Wireless Sensor Network as a Low-power Lossy Network (LLN). A LLN I. INTRODUCTION comprises routers, sensors and actuators, which use wireless After many decades of research, wireless networks have transmissions to exchange packets, possibly via multiple hops. evolved from Ad Hoc Wi-Fi technology to low-power and The great importance of LLNs becomes clear, considering large-scale Wireless Sensor Networks (WSNs). This miniatur- how they have affected the emergence of modern IoT applica- ization extended de facto their usage, and enabled the design tions, where information must be shared intact across different of radically new applications that follow the modern concept platforms, while enhancing the data from a distributed network of Internet of Things (IoT). For instance, Smart Cities are of sensors and actuators. expected to rely heavily on a myriad of devices, able to Globally, the IoT paradigm encompasses a large variety of measure their surrounding environment and take decisions in devices connected to the Internet, including amongst others: (i) order to manage the city efficiently [1], [2]. Moreover, sensors multimedia objects (e.g., a video camera) that require a high throughput and, thus, rendering them unable to save energy, (ii) K.Kritsis is with the Research and Innovation Center Athen (ATHENA Radio-Frequency Identification System (RFID) and tags that RC), Institute for Language and Speech Processing (ILSP), 15125 Marousi, Athens, Greece (e-mail: [email protected]). enable the tracking of any object, typically within a supply G. Z. Papadopoulos is with the IRISA, IMT Atlantique, 35510, Cesson- chain context, (iii) small sensors and actuators, which are Sevign´ e,´ France (e-mail: [email protected]). mostly battery-powered wireless hardware platforms. A. Gallais and F. Theoleyre´ are with the ICube Laboratory, CNRS / University of Strasbourg, 67412 Illkirch, France, (e-mail: The number of embedded devices involved in a typical fgallais,[email protected]). LLN scenario varies from tens to thousands of nodes, which P. Chatzimisios is with the CSSN Research Lab, Department of Informatics, introduces further density as well as scalability issues. Alexander TEI of Thessaloniki (ATEITHE), 57400 Thessaloniki, Greece and with the Department of Computing & Informatics, Bournemouth University, Validation and verification during the protocol develop- BH12 5BB, United Kingdom (e-mail: [email protected]). ment has become a matter of prime importance. Envisioned 2 solutions must be intensively tested before being deployed Application home intrusion HVAC in a real-world environment, by employing either simula- (e.g. periodic Traffic, Event-based) automation detection tors, emulators, or even conducting experimentations over real physical testbeds [5]. More specifically, simulators have TRANSPORT CoAP allowed users to exploit available models (e.g., link quality, (end-to-end, security) radio propagation, medium interference, topologies) in order to anticipate the behaviour of their proposals under real-life RPL conditions. Some open testbeds have also emerged, providing access to pre-deployed low-power devices and, thus, a certain NETWORK IPv6 (e.g. routing, IP address, self- level of repeatable experimental setups [6]. However, a great configuration) 6LoWPAN number of issues, such as weather conditions, interference from other wireless technologies or even obstacles, may have MAC IEEE 802.15.4-2015 an impact on the radio links among the sensor nodes. Thus, (e.g. medium access, PHY addr) (e.g. TSCH, DSME, LLDN) before proceeding to the experimental evaluation of a network Networking protocol or an algorithm, apprehending the wireless links and PHY the overall topology is an essential step [7]. It is also manda- IEEE 802.15.4 PHY (e.g. modulation, error code) tory to consider the correct energy models since preliminary results must help end users to get a flavour of the ensued Fig. 1: Typical stack of protocols in the Internet of very energy consumption. We here detail numerous tools that shall low-power Things. ease such investigations. In order to initially validate their concepts or models, researchers must be advised of the numerous constraints that the setup (e.g., reproducibility). We then describe the various arise in order to define the scientific methodology that will lead characteristics that may be present in various radio topologies to pertinent experimental results. Theoretical and numerical (SectionIV), as well as we detail the traffic characterization analysis, along with packet level simulation and emulation (SectionV) and the different energy models (SectionVI). We appear as essential pieces of the long-term validation of a then present most of the existing tools that may be used during whole solution. a performance evaluation campaign (Section VII). Finally, In this tutorial, we review in depth various parameters Section VIII discusses the current related challenges, before that should be taken into account during such performance giving some concluding remarks in SectionIX. evaluation. We especially focus