All One Needs to Know About Fog Computing and Related Edge Computing ☆,☆☆ Paradigms: a Complete Survey

Journal of Systems Architecture 98 (2019) 289–330

Contents lists available at ScienceDirect

Journal of Systems Architecture

journal homepage: www.elsevier.com/locate/sysarc

All one needs to know about fog computing and related edge computing ☆,☆☆ paradigms: A complete survey

Ashkan Yousefpour a,∗, Caleb Fung b, Tam Nguyen c, Krishna Kadiyala b, Fatemeh Jalali d,

Amirreza Niakanlahiji e, Jian Kong b, Jason P. Jue b a University of California Berkeley, Berkeley, USA b University of Texas at Dallas, Richardson, USA c Georgia Institute of Technology, Atlanta, USA d IBM Research, Melbourne, Australia e University of North Carolina at Charlotte, Charlotte, USA

a r t i c l e i n f o a b s t r a c t

Keywords: With the Internet of Things (IoT) becoming part of our daily life and our environment, we expect rapid growth in Fog computing the number of connected devices. IoT is expected to connect billions of devices and humans to bring promising

Edge computing advantages for us. With this growth, fog computing, along with its related edge computing paradigms, such as

Cloud computing multi-access edge computing (MEC) and cloudlet, are seen as promising solutions for handling the large volume Internet of things (IoT) of security-critical and time-sensitive data that is being produced by the IoT. In this paper, we ﬁrst provide a Cloudlet

Mobile edge computing tutorial on fog computing and its related computing paradigms, including their similarities and diﬀerences. Next,

Multi-access edge computing we provide a taxonomy of research topics in fog computing, and through a comprehensive survey, we summarize Mist computing and categorize the eﬀorts on fog computing and its related computing paradigms. Finally, we provide challenges and future directions for research in fog computing.

1. Introduction as time-sensitive and location-aware applications emerge (such as patient monitoring, real-time manufacturing, self-driving cars, ﬂocks of In today’s information technology age, data is the main commodity, drones, or cognitive assistance), the distant cloud will not be able to and possessing more data typically generates more value in data-driven satisfy the ultra-low latency requirements of these applications, pro- businesses. According to the International Data Corporation (IDC), the vide location-aware services, or scale to the magnitude of the data that amount of digital data generated surpassed 1 zettabyte in 2010 [1] . Fur- these applications produce [5] . Moreover, in some applications, send- thermore, 2.5 exabytes of new data is generated each day since 2012 [2] . ing the data to the cloud may not be a feasible solution due to privacy Cisco estimates that there will be around 50 billion connected devices concerns. by 2020 [3] . These connected devices constitute the Internet of Things In order to address the issues of high-bandwidth, geographically- (IoT) and possibly generate a massive amount of data. With this astro- dispersed, ultra-low latency, and privacy-sensitive applications, there is nomical amount of data, the current mobile network architectures will a quintessential need for a computing paradigm that takes place closer to have trouble managing the momentum and magnitude of data. In cur- connected devices. Fog computing has been proposed by both industry rent implementations of cloud-based applications, most data that needs and academia [6,7] to address the above issues and to quench the need storage, analysis, and decision making is sent to the data centers in the for computing paradigm closer to connected devices. Fog computing cloud [4] . bridges the gap between the cloud and IoT devices by enabling comput- As the data velocity and volume increases, moving the big data from ing, storage, networking, and data management on the network nodes IoT devices to the cloud might not be eﬃcient, or might be even infea- within the close vicinity of IoT devices. Therefore, computation, stor- sible in some cases due to bandwidth constraints. On the other hand, age, networking, decision making, and data management occur along

☆ A complete list of conferences, journals, and magazines that publish state-of-the-art research papers on fog computing and its related edge computing paradigms is available at https://anrlutdallas.github.io/resource/projects/fog-computing-conferences.html . We have included papers from the above list in this survey. ☆☆ The data (categories and features/objectives of the papers) of this survey are available at https://github.com/ashkan-software/fog-survey-data .

∗ Corresponding author. E-mail address: [email protected] (A. Yousefpour). https://doi.org/10.1016/j.sysarc.2019.02.009 Received 6 December 2018; Accepted 7 February 2019 Available online 12 February 2019 1383-7621/© 2019 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license. ( http://creativecommons.org/licenses/by/4.0/ ) A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

computing can benefit from Software Defined Networking (SDN). They note the technical challenges in edge computing and propose using SDN as a solution for implementing edge computing infrastructure. The authors survey publications primarily on edge computing and SDN to support their argument and give future directions for SDN developments. The authors in [10] compile a comprehensive survey of recent efforts in fog-enabled network architectures, and provide various network applications of fog computing. The recent survey in [11] focuses on connectivity and device configuration aspects of the fog computing and identifies major features that fog computing platforms need to build infrastructure for smart city applications. They further review existing approaches that have been proposed to tackle the challenges in fog computing for building such smart city infrastructure. Comparably, the authors of [12] focus on architecture design and system management of edge computing to provide a detailed and focused survey in the edge computing field. They also characterize fog and edge computing by comparing a list of related computing concepts, including peer-to-peer computing, mobile grid computing, and mobile crowd computing. The authors in [13] take a closer look at fog- assisted IoT applications, discuss security and privacy challenges in fog computing, and review and analyze promising techniques to resolve security and privacy issues in fog-assisted IoT applications. There are a number of surveys in the area of MEC that also discuss similar concepts to fog computing and summarize papers applicable to fog computing research. The survey in [14] introduces a survey on MEC and focuses on the fundamental key enabling technologies in MEC. The paper also analyzes the MEC reference architecture, overviews the current standardization activities, and introduces main deployment scenarios. Similarly, the survey in [15] provides a survey of the recent state of MEC research with a focus on joint radio and computation resource Fig. 1. The structure of the survey. management. the path between IoT devices and the cloud, as data moves to the cloud 2.1. Contribution from the IoT devices. Other similar computing paradigms to fog computing such as edge computing, mist computing, cloud of things, and Different from the mentioned surveys, the contribution of this paper cloudlets, have been proposed by the research community to address is three-fold: (1) We provide a detailed tutorial on fog computing, how the mentioned issues. In this survey, we compare fog computing with it is defined, and how it is related to or different from other similar com- other related computing paradigms, and argue that fog computing is a puting paradigms, such as cloud computing, cloudlets, edge computing, more general form of computing, mainly due to its comprehensive def- and MEC (2) We propose an exhaustive taxonomy of research topics inition scope and flexibility. in the area of fog computing, and present a comprehensive survey on In this article, we present a comprehensive survey on fog comput- fog computing. 1 (3) We have compiled a list of challenges and future ing, and discuss how fog computing can meet the growing demand of directions for research in fog computing. applications with strict latency, privacy, and bandwidth requirements.

A comparison of the related survey papers in the area of fog computing 3. A comparison of fog computing and related computing is included in Section 2. To gain a thorough understanding of fog com- paradigms puting, in Section 3 , we will first look at cloud computing, then discuss how fog computing extends cloud computing to address the above issues This section focuses on the comparison of fog computing and re- of cloud, and finally, compare fog computing to other similar computing lated computing paradigms to demonstrate the value of fog computing paradigms. Next, in Section 4, we describe our taxonomy of research top- in a variety of use cases. Moreover, this section provides a better un- ics in fog computing. Later, in a comprehensive survey, we summarize derstanding of how these computing paradigms can benefit the current and categorize the efforts on fog computing and its related computing and future landscape of connected devices. We compare fog computing paradigms. In Section 5, we present the challenges and limitations in the with cloud computing as well as other related computing paradigms and fog computing area and provide future directions and potential starting summarize this information in Tables 2 and 3 . points for those challenges. Finally, Section 6 concludes the paper. Fig. 1 shows the structure of the survey and a reading map for the reader. 3.1. Cloud computing 2. Related surveys Cloud computing has been instrumental in expanding the reach and

There are are some existing related studies in the area of fog comput- capabilities of computing, storage, and networking infrastructure to ing that have attempted to provide a survey of the papers in the ﬁeld of the applications. The National Institute of Standards and Technology fog computing, edge computing, or MEC. The authors in [8] present a comprehensive review of current literature in fog computing with 1 We compiled a comprehensive list of conferences, journals, and a focus on architectures and algorithms in fog systems. They further magazines that publish state-of-the-art research papers on fog comput- sketch the prospects of fog computing in terms of emerging technologies ing and its related edge computing paradigms The list is available at with a focus on Tactile Internet. Cihat et al. [9] state the importance of https://anrlutdallas.github.io/resource/projects/fog-computing-conferences. cooperation between edge and cloud computing, and motivate how edge html .

290 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

(NIST) defines cloud computing as a model that promotes ubiquitous, lationship among IaaS, PaaS, and SaaS with the underlying cloud infras- on-demand network access to shared computing resources [16] . Cloud tructure, and illustrates what portion of the application stack is managed data centers are large pools of highly accessible virtualized resources by cloud providers. that can be dynamically reconfigured for a scalable workload; this re- configurability is beneficial for clouds services that are offered with a 3.1.2. Cloud resource provisioning pay-as-you-go cost model [17] . The pay-as-you-go cost model allows Since the demand for cloud resources is not fixed and can change users to conveniently access remote computing resources and data man- over time, setting a fixed amount of resources results in either over- agement services, while only being charged for the amount of resources provisioning or under-provisioning, as depicted in Fig. 3 . A foundation they use. Cloud providers, such as Google, IBM, Microsoft, and Ama- of cloud computing is based on provisioning only the required resources zon provide and provision large data centers to host these cloud-based for the demand. This includes the use of virtualization for on-demand resources. application deployment, and the use of resource provisioning to manage hardware and software in cloud data centers. Provisioning resources is 3.1.1. Cloud services an important topic in cloud computing that is widely explored. Since it Cloud offers infrastructure, platform, and software as services (IaaS, is difficult to estimate service usage from tenants, most cloud providers PaaS, SaaS). Application developers can use a variety of these services have a pay-as-you-go payment scheme. As a result, providers can be depending on the needs of the applications they develop. Infrastructure more flexible on how to provision resources, and clients only pay for as a service (IaaS) allows cloud consumers to directly access IT infras- the amount of resources they actually use. tructures for processing, storage, and networking resources [18] . Sup- pose Sam wants to set up a high-tech agricultural system that utilizes IoT 3.1.3. Types of cloud devices to monitor the condition of crops. Sam contacts a cloud provider There are four types of cloud deployments: private cloud, community and acquires an IaaS for development of his system. Sam now can config- cloud, public cloud, and hybrid cloud [16] . Private clouds are designed ure the IaaS (often offered as a standalone VM) in terms of hardware and for use by a singular entity and ensure high privacy and configurabil- corresponding software for his need. Control over infrastructure (IaaS) ity. Private clouds are a good choice for organizations that require an allows Sam to customize hardware configuration, such as the number infrastructure for their applications. This type of deployment is simi- of CPU cores and RAM capacity, in addition to systems-level software. lar to traditional company-owned server farms and often do not bene- Sam can obtain an IaaS from Amazon Web Services (AWS), Microsoft fit from a pay-as-you-go cost model. Community clouds are used by a Azure, or Google Compute Engine (GCE). community of users, and the infrastructure is shared between several On the other hand, platform as a service (PaaS) allows cloud con- organizations. A community cloud results in decentralized ownership sumers to develop software and fully supports software lifecycle –often of the cloud by multiple organizations within the community without with the help of a middleware –for software management and configu- relying on a large cloud vendor for the IT infrastructure. Public clouds ration. If Sam does not need to configure the infrastructure of the cloud, are the typical model of cloud computing, where the cloud services are managing and configuration of hardware and software may detract from offered by cloud service providers, such as Amazon, IBM, Google, Mi- the productivity of Sam’s business. Now, Sam could consider using PaaS crosoft, etc. Public clouds are generally more popular, easy-to-maintain, offered by Apache Stratos, Azure App Services, or Google App Engine and cost-effective compared to private clouds. In contrast to private for his business. PaaS manages the underlying low-level processes and clouds, public clouds may benefit from the pay-as-you-go pricing model. allows Sam to focus on managing software for his IoT-specific interac- However, public clouds do not always offer users complete customiza- tions. Moreover, PaaS providers often include tools for convenient man- tion of hardware, middleware, network, and security settings. Hybrid agement of databases and scaling applications. clouds are simply a combination of the cloud types mentioned above. Now suppose Sam is willing to spend more money and likes to get Hybrid clouds allow users to have finer control over virtualized infras- full software packages, and he does not want to take care of software tructure, and combining the capabilities from different types of cloud de- issues, such as database scalability, socket management, etc. Software ployments is accomplished through standardized or proprietary techno- as a service (SaaS) provides Sam an environment to centrally host his logy [19] . applications and removes the need for him to install software manually. The cloud computing paradigm was initially established to allow Sam’s client software now can be hosted on Google Apps or as a Web users to access a pool of computing resources for ubiquitous computing. application. Even though cloud computing has helped bring forth accessible com- As demonstrated by these examples, cloud services can be utilized puting, the time required to access cloud-based applications may be too for distinct use cases for a variety of end users. Fig. 2 illustrates the re- high and may not be practical for some mission-critical applications, or applications with ultra-low latency requirements. Also, the rapid growth in the amount of data generated at the network edge by an increasing number of connected devices requires cloud resources to be closer to where the data is generated. Greater demand for high-bandwidth, geographically-dispersed, low-latency, and privacy-sensitive data processing has emerged –a quintessential need for computing paradigms that take place closer to connected devices and that support low-latency, high-bandwidth, decentralized applications. To address these needs, fog computing has been proposed by both industry and academia [6,7] . In order to provide a detailed comparison among fog computing related paradigms, we introduce various computing paradigms, starting with fog computing.

3.2. Fog computing

Fog computing bridges the gap between the cloud and end devices (e.g., IoT nodes) by enabling computing, storage, networking, and data Fig. 2. Common cloud service models and their classiﬁcations relative what management on network nodes within the close vicinity of IoT devices. portion of the application stack is managed by cloud providers. Consequentially, computation, storage, networking, decision making,

291 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Fig. 3. Cloud provisioning is done based on the application demand [22] . and data management not only occur in the cloud, but also occur along other hand, requires devices to be connected when the cloud service is the IoT-to-Cloud path as data traverses to the cloud (preferably close to in progress. the IoT devices). For instance, compressing the GPS data can happen at Fog helps devices measure, monitor, process, analyze, and react, and the edge before transmission to the cloud in Intelligent Transportation distributes computation, communication, storage, control, and decision Systems (ITS) [20] . Fog computing is defined by the OpenFog Consor- making closer to IoT devices [6] (refer to Fig. 5 ). Many industries could tium [6] as “a horizontal system-level architecture that distributes com- use fog to their benefit: energy, manufacturing, transportation, health- puting, storage, control and networking functions closer to the users care, smart cities, to mention a few. along a cloud-to-thing continuum. ”The “horizontal ” platform in fog computing allows computing functions to be distributed between dif- 3.2.2. Fog–cloud federation ferent platforms and industries, whereas a vertical platform promotes There are clear differences and trade-offs between cloud and fog siloed applications [21] . A vertical platform may provide strong sup- computing, and one might ask which one to choose. However, fog and port for a single type of application (silo), but it does not account for cloud complement each other; one cannot replace the need of the other. platform-to-platform interaction in other vertically focused platforms. By coupling cloud and fog computing, the services that connected de- In addition to facilitating a horizontal architecture, fog computing pro- vices use can be optimized even further. Federation between fog and vides a flexible platform to meet the data-driven needs of operators and cloud allows enhanced capabilities for data aggregation, processing, and users. Fog computing is intended to provide strong support for the In- storage. For instance, in a stream processing application, the fog could ternet of Things. filter, preprocess, and aggregate traffic streams from source devices, while queries with heavy analytical processing, or archival results could 3.2.1. Fog vs. cloud be sent to the cloud. An orchestrator could handle the cooperation be- A common example that is often used to distinguished fog and cloud tween cloud and fog. Specifically, a fog orchestrator could provide an in- computing is whether latency-sensitive applications can be supported teroperable resource pool, deploy and schedule resources to application while maintaining satisfactory quality of service (QoS). Fog nodes can workflows, and control QoS [24] . Through the use of SDN, fog service be placed close to IoT source nodes, allowing latency to be noticeably providers will have greater control over how the network is configured reduced compared to traditional cloud computing. While this example with a large number of fog nodes that transfer data between the cloud gives an intuitive motivation for fog, latency-sensitive applications are and IoT devices. only one of the many applications that warrant the need for fog computing. Nodes in fog computing are generally deployed in less central- 3.2.3. Fog RAN ized locations compared to centralized cloud data centers. Fog nodes Fog computing can be integrated into mobile technologies in the are wide-spread and geographically available in large numbers. In fog form of radio access networks (RAN), to form what is referred to as fog computing, security must be provided at the edge or in the dedicated RAN (F-RAN). Computing resources on F-RANs may be used for caching locations of fog nodes, as opposed to the centrally-developed security at the edge of the network, which enables faster retrieval of content and mechanisms in dedicated buildings for cloud data centers. The decen- a lower burden on the front-haul. F-RAN can be implemented through tralized nature of fog computing allows devices to either serve as fog 5G related mobile technologies [25] . On the other hand, cloud RAN computing nodes themselves (e.g. a car acts as a fog node for onboard (C-RAN) provides centralized control over F-RAN nodes. C-RAN takes sensors) or use fog resources as the clients of the fog. advantage of virtualization, and decouples the base stations within a The majority of differences between cloud and fog computing are cell of the mobile network from its baseband functions by virtualizing attributed to the scale of hardware components associated with these those functions [26] . In C-RAN a large number of low-cost Remote Ra- computing paradigms. Cloud computing provides high availability of dio Heads (RRHs) are randomly deployed and connected to the Base computing resources at relatively high power consumption, whereas Band Unit (BBU) pool through the front-haul links. Both F-RAN and fog computing provides moderate availability of computing resources C-RAN are suited for mobile networks with base stations and are can- at lower power consumption [23] . Cloud computing typically utilizes didates for 5G deployments. Also, the use of F-RAN and C-RAN brings large data centers, whereas fog computing utilizes small servers, routers, a more energy efficient form of network operation. We encourage the switches, gateways, set-top boxes, or access points. Since hardware for motivated reader to refer to reference [27] for more information about fog computing occupies much less space than that of cloud computing, F-RAN. hardware can be located closer to users. Fog computing can be accessed Fig. 4 shows a classification of computing paradigms related to fog through connected devices from the edge of the network to the network computing and their overlap in terms of their scope. The figure il- core, whereas cloud computing must be accessed through the network lustrates our comparison of fog computing and its related computing core. Moreover, continuous Internet connectivity is not essential for the paradigms. Table 1 lists the acronyms used for this figure and in the fog-based services to work. That is, the services can work independently paper. We discuss the related computing paradigms in the order of with low or no Internet connectivity and send necessary updates to the their trend and show how some paradigms resulted in the emergence of cloud whenever the connection is available. Cloud computing, on the others.

292 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

transmission hardware and protocols, and compression algorithms [28] . Due to the evolving requirements of connected consumer devices, mobile computing alone is not suitable for many recent computing challenges. With fog and cloud computing, computation is no longer tied to a local network; fog and cloud computing expand the scale and scope of mobile computing. The only type of hardware that mobile computing requires are mobile devices, which can be connected through Bluetooth, WiFi, ZigBee, and other cellular protocols. In contrast, fog and cloud computing require more resource-rich hardware with virtualization capabilities. Security in mobile computing must be provided on the mobile device itself. Compared to fog and cloud computing, mobile computing is more resource-constrained, but in recent years, advancements in mobile hardware and wireless protocols have signiﬁcantly reduced this gap. The power of mobile computing is from its distributed computing

Fig. 4. A classiﬁcation of scope of fog computing and its related computing architecture. Distributed applications beneﬁt from this architecture be- paradigms. (Intersection of cloud computing and mist computing is Cloud of cause mobile machines do not need a centralized location to oper- Things, and intersection of mobile computing and cloud computing is mobile ate. Mobile computing, however, comes with many drawbacks such as cloud computing.). poor-resource constraints, the balance between autonomy and interde- pendence (prevalent in all distributed architectures), communication Table 1

A list of computing paradigm acronyms used in this section. latency, and the need for mobile clients to eﬃciently adapt to changing environments [29] . These drawbacks often make mobile computing IoT Internet of Things CC Cloud Computing unsuitable for current applications that require low-latency or robust-

MC Mobile Computing FC Fog Computing ness, or that need large amounts of data to be generated, processed, and EC Edge Computing MEC Multi-access Edge Computing

MCC Mobile Cloud Computing MACC Mobile ad hoc Cloud Computing stored on devices. CoT Cloud of Things mist Mist Computing

3.4. Mobile cloud computing 3.3. Mobile computing As cloud computing matured, it became a valuable complement to The advancement in fog and cloud computing is influenced by the mobile computing. This combination resulted in mobile cloud comput- groundwork set forth by the development of mobile computing. Mobile ing (MCC), which is defined as an infrastructure where both the data computing, or nomadic computing, is when computing is performed via storage and data processing occur outside of the mobile device, bring- mobile, portable devices, such as laptops, tablets, or mobile phones. Mo- ing mobile computing applications to not just smartphone users but a bile computing can be utilized to create pervasive context-aware appli- much broader range of mobile subscribers [30] . NIST extends this defini- cations, such as location-based reminders. tion to include mobile devices: cloud computing is the synergy between At the heart of mobile computing is the vision for adaptation in IoT devices, mobile devices, and cloud computing that enables data- an environment of low processing power and intermittent, sparse net- intensive and CPU-intensive applications for IoT environments [31] . work connectivity. The peak of mobile computing technologies precedes Some of these applications in MCC include crowdsourcing, healthcare, cloud computing. A large number of fundamental challenges (such as sensor data processing (such as optical character recognition and im- user mobility, network heterogeneity, and low bandwidth) in mobile age processing), and task offloading [32,33] . Mobile applications can be computing have been addressed in the literature before 2000. These partitioned at runtime so that computationally intensive components of issues have been addressed by advancements such as robust caching, the application can be handled through adaptive offloading [34] .

Fig. 5. Fog brings several beneﬁts for the application developers, applications, and diﬀerent industries by distributing the core functions.

293 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

In mobile cloud computing, resource contained mobile devices can 3.5.2. MACC vs. MCC leverage resource-rich cloud services. MCC shifts the majority of compu- MACC is also different from MCC in the hardware, service access tation from mobile devices to the cloud. MCC helps to run computation- method, and the distance from users, since computation is done on mo- intensive applications and to increase the battery life of mobile devices. bile devices in MACC, whereas it is far from mobile devices in MCC. MCC shares a blend of capabilities and characteristics in mobile com- MACC only requires mobile devices to operate, whereas MCC requires puting and cloud computing. By adopting a combination of mobile com- large-scale data centers used for cloud computing in addition to mobile puting and cloud computing objectives, high availability of computing devices. This results in high computation power, but also higher latency resources is present in MCC as opposed to resource-constrained mo- in MCC. Security in MACC must be provided only in mobile devices, but bile computing. This allows for the emergence of high computation ap- ensuring trust may be challenging in MACC without a secure collabora- plications, such as mobile augmented reality. Also, the availability of tion framework. Finally, in MACC, services are only accessed through cloud-based services in MCC is considerably higher than that of mobile mobile devices that are connected via Bluetooth, WiFi, and other cellu- computing. Similar to cloud computing and fog computing, MCC relies lar protocols. on cloud services for operating high-computation services. Computation in MCC can also be operated by mobile devices. Similar to cloud com- 3.5.3. MACC vs. fog puting, security in MCC must be provisioned in both mobile devices and Although fog computing can be performed across a variety of in the cloud. The authors in [35] design and implement an Android app resource-rich and resource deficient devices, mobile ad hoc cloud com- that helps drivers find parking space availability using MCC. puting is better suited for highly decentralized, dynamic network topolo-

MCC also suffers from the same limitations of mobile computing and gies in which Internet connection is not guaranteed. Connected devices cloud computing. First, while a centralized architecture in MCC is great in MACC are more decentralized compared to fog computing, and this for sharing a pool of computation resources, this may not be well suited allows the devices to form a more dynamic network in places of sparely for applications where pervasiveness of devices is desired. Second, since connected devices or a constantly changing network. An example of this both cloud computing and MCC require cloud-based services, and as ac- is an ad hoc network for peer-to-peer file sharing [39] . cess to those services is through the network core by WAN connection, applications running on these platforms require connection to the Inter- net all the time. MCC shifts the majority of computation from mobile de- 3.5.4. MACC vs. MANET vices to the cloud, and this introduces connectivity challenges that were A similar concept to MACC is a mobile ad hoc network (MANET). not present in mobile computing. Finally, offloading computation to the MANETs consist of mobile host devices that are connected to each cloud causes the latency to be relatively high for delay-sensitive appli- other with single hop without base stations [40]. MANET devices cations. The authors in [36] design a food recognition system based on form dynamic networks but do not necessarily form a cloud. In other

MCC that distributes the data analytics between the mobile devices and words, the computing or storage resource pools are not necessary for the servers in the cloud. Mobile phones can perform light-weight com- MANETs. However, many solutions to MANETS, such as redundancy putation on food images for food recognition, which allows the system and broadcasting, can be applied to MACC. In a resource-constrained to overcome some inherent limitations of traditional MCC paradigm, environment, peers may want to pool resources together to achieve a such as high latency and low battery life of mobile devices. computationally demanding task that may not be feasible on a single mobile device. A use case for this is an unmanned vehicular system that

3.5. Mobile ad hoc cloud computing consists of multiple unmanned vehicles and traﬃc devices.

Despite the pervasive nature of MCC, this computing paradigm is 3.6. Edge computing not always suitable for scenarios in which there is a lack of infrastructure or a centralized cloud. An ad hoc mobile network consists of nodes Similar to how MCC extends the capabilities of mobile devices, that form a temporary, dynamic network through routing and transport edge computing also enhances the management, storage, and process- protocols; it is the most decentralized form of a network [37] . Mobile ing power of data generated by connected devices. Unlike MCC, edge devices in an ad hoc mobile network form a highly dynamic network computing is located at the edge of the network close to IoT devices; topology; the network formed by the mobile devices is highly dynamic note that the edge is not located on the IoT devices, but as close as one and must accommodate for devices that continuously join or leave the hop to them. It is worth noting that the edge can be more than one hop network. Ad hoc mobile devices can form clouds that can be used for away from IoT devices in the local IoT network. OpenEdge Computing networking, storage, and computing. MACC could include use cases such defines edge computing as computation done at the edge of the network as disaster relief, group live video streaming, and unmanned vehicular through small data centers that are close to users [41] . The original vi- systems. sion for edge computing is to provide compute and storage resources close to the user in open standards and ubiquitous manner [41] . Edge 3.5.1. MACC vs. cloud computing computing is a crucial computing paradigm in the current landscape Mobile ad hoc cloud computing (MACC) is fundamentally different of IoT devices; it integrates the IoT devices with the cloud by filtering, from cloud computing, mainly due to the ad hoc nature of the resources. preprocessing, and aggregating IoT data intelligently via cloud services MACC involves mobile devices that function as data providers, storage, deployed close to IoT devices [42] . and processing devices. Mobile devices in a mobile ad hoc cloud net- Some issues that edge computing is well equipped to handle are pri- work are also responsible for routing traffic among themselves, because vacy, latency, and connectivity. Due to its proximity to the users, latency of the lack of network infrastructure. By pooling local mobile resources in edge computing is typically lower than in MCC and cloud computing, to form an ad hoc cloud, MACC offers reasonably high computation. if enough local computation power is provided; latency in edge comput- These attributes differ from the target users, architecture, and connec- ing can be slower than cloud or MCC if the local computation unit is not tivity in cloud computing. In a study done by researchers of Carnegie powerful enough. Service availability is also higher in edge computing Mellon University [38] , there is a tradeoff between offloading computa- because connected devices do not have to wait for a highly centralized tion to distant clouds (labeled as “infrastructure cloud ”) versus running platform to provide a service, nor are connected devices limited by the them on nearby mobile devices (labeled as “mobile edge-clouds, ”but limited resources in traditional mobile computing. Compared to MACC, in this paper we call them “mobile ad hoc clouds ”). The authors com- edge computing has small data centers, whereas MACC fundamentally pare the performance of executing some applications on a traditional does not need data centers. As a result, edge computing has higher infrastructure cloud versus running them in mobile ad hoc clouds. service availability. Edge computing also can expand with broader

294 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Fig. 6. Comparison of fog computing and its related computing paradigms in terms of their location and distance from the core clouds.

computing capabilities than MACC by forming hybrid architectures with Provider) access/edge networks. Yet, the term edge that is recently peer-to-peer and cloud computing models [43] . used in the IoT landscape [42,45] refers to the local network where sensors and IoT devices are located. In other words, the edge is the 3.6.1. Edge computing vs. fog computing immediate first hop from the IoT devices (not the IoT nodes them- Although fog computing and edge computing both move the compu- selves), such as the WiFi access points or gateways. If the computation tation and storage to the edge of the network and closer to end-nodes, is done on IoT devices themselves, this computing paradigm is referred these paradigms are not identical. In fact, the OpenFog Consortium to as mist computing (see Section 3.9 ). General Electric notes that fog states that edge computing is often erroneously called fog computing; computing focuses on interactions between edge devices (e.g., RANs, OpenFog Consortium makes the distinction that fog computing is hierar- base stations, or edge routers), whereas edge computing focuses on the chical and it provides computing, networking, storage, control, and ac- technology attached to the connected things (e.g., WiFi access points) celeration anywhere from cloud to things; while, edge computing tends [45] . to be limited to computing at the edge [6] . (Refer to Fig. 6 .) Moreover, in a tutorial article [44] about fog and edge, the authors explain that 3.7. Multi-access edge computing “fog is inclusive of cloud, core, metro, edge, clients, and things, ”and “fog seeks to realize a seamless continuum of computing services from Mobile cloud computing is an extension of mobile computing the cloud to the things rather than treating the network edges as isolated through cloud computing. Analogously, multi-access edge (MEC) com- computing platforms, ”and “fog envisions a horizontal platform that will puting is an extension of mobile computing through edge computing. support the common fog computing functions for multiple industries ETSI defines MEC as a platform that provides IT and cloud-computing and application domains, including but not limited to traditional telco capabilities within the Radio Access Network (RAN) in 4G and 5G, services. ” [44] in close proximity to mobile subscribers [46] . Multi-access edge computing was previously referred to as “mobile edge computing, ”but 3.6.2. Where is edge? the paradigm has been expanded to include a broader range of appli- It is worth mentioning that edge computing, cloudlets, fog comput- cations beyond mobile device-specific tasks. Examples of multi-access ing, and mist computing (to be discussed in Section 3.9 ) are used in- edge computing applications include video analytics, connected vehi- terchangeably in some papers, as they all have “edge ”as a common cles, health monitoring, and augmented reality. term. The term edge used by the telecommunications industry usu- MEC extends edge computing by providing compute and storage really refers to 4G/5G base stations, RANs, and ISP (Internet Service sources near low energy, low resource mobile devices. MEC allows RAN

295 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 operators to add edge computing functionality to existing base stations. proximity of mobile devices [52] . Although cloudlet computing fits well Similar to edge computing, small-scale data centers with virtualization with the mobile-cloudlet-cloud framework [53] , fog computing offers a capacity can also be used in MEC. Due to underlying hardware used in more generic alternative that natively supports large amounts of traffic, MEC and edge computing, available computing resources is moderate, in and allows resources to be anywhere along the thing-to-cloud contin- comparison to cloud computing. Furthermore, low-latency applications uum. The concept of mobile cloudlets is similar to cloudlets, in which the can be supported in MEC. MEC applications can benefit from real-time cloudlets are a group of nearby mobile devices that are connected wire- radio and network information hence can offer a personalized and con- lessly, e.g., using WiFi or Bluetooth [54] . In mobile cloudlets, mobile textualized experience to the mobile subscriber. devices can be providers as well as clients of computing service. Both edge computing and MEC computing services operate from the edge of the Internet and can function with little to no Internet connectiv- 3.9. Mist computing ity. MEC, however, establishes connectivity through a WAN, WiFi, and cellular connections, whereas edge computing generally can establish Recently, mist computing has been introduced to capture a more ex- any form of connectivity (e.g., LAN, WiFi, cellular). MEC also primar- treme edge –the endpoints –of connected devices [55] . This computing ily differs from MCC in its operations: Research in MCC focuses on the paradigm describes dispersed computing at the extreme edge (the IoT relationship between cloud service users (on mobile devices) and cloud devices themselves) and has been proposed with future self-aware and service providers, whereas research in MEC focuses on (RAN-based) net- autonomic systems in mind [56] . Mist computing could be seen as the work infrastructure providers. MEC is expected to benefit significantly first computing location in the IoT-fog-cloud continuum; it can be infor- from the up-and-coming 5G platform [47]. Likewise, 5G is seen as an mally labeled as “IoT computing ”or “things computing. ”An IoT device enabler of MEC as it allows for lower latency and higher bandwidth may be wearable, a mobile device, a smart watch, or a smart fridge. among mobile devices, and it supports a wide range of mobile devices Mist computing extends compute, storage, and networking across the with finer granularity. fog through the things. In a sense, mist computing is a superset of MACC;

MEC allows edge computing to be accessible to a wide range of since in mist, the networking may not be necessarily ad hoc, and the de- mobile devices with reduced latency and more eﬃcient mobile core vices may not be mobile devices (refer to Fig. 4 ). networks [14]. MEC also allows for mission-critical delay-sensitive ap- The authors in [57] introduce the idea of using mobile devices in plications over the mobile network [47]. It has also incorporated the the vicinity as a cloud computing environment for storage, caching, and use of SDN and network function virtualization (NFV) capabilities, in computing purposes. They study the use of mist computing to reduce the addition to 5G technologies. SDN allows for virtual networking devices load in traditional WiFi infrastructures for video dissemination applica- to be easily managed through software APIs [48], and NFV allows for tions. In this study, the spectators of a sport event organize themselves reduced deployment times for networking services through virtualized into WiFi-Direct groups and exchange video replays whenever possible, infrastructure. Moreover, through SDN and NFV, network engineers and bypassing the central server and access points. This study is also another possibly enterprise application developers can develop their own orches- example of mist computing, in which IoT devices act not only as “thin trator, whose goal is to coordinate the resource provisioning across mul- clients, ”but also as “thin servers. ”Some other uses of mist comput- tiple layers [49]. ing are to preserve the privacy of users’ data via local processing [58] , and to eﬃciently deploy virtualized instances on single-board com-

3.8. Cloudlet computing puters [59] .

Proposed by Carnegie Mellon University, cloudlet computing is another direction in mobile computing that shares many traits with MCC 3.10. Other similar computing paradigms and MEC. In fact, it addresses some of the disadvantages of MCC. A cloudlet is a trusted resource-rich computer or a cluster of computers 3.10.1. Micro data center with strong connection to the Internet that is utilized by nearby mobile Cloudlet is sometimes referred to as micro data center (MDC) in some devices [50] . Cloudlets are small data centers (miniature clouds, as the studies [60]. The term micro data center (MDC) was proposed by Mi- name suggests) that are typically one hop away from mobile devices. crosoft Research in 2015 [60] and is deﬁned as “an extension of tradi-

The idea is to offload computation from mobile devices to VM-based tional data centers used in cloud computing.”AnMDC can be an edge cloudlets located on the network edge [51] . Although academia mostly node or a cloudlet that is deployed between IoT devices and the cloud. drives current studies in cloudlet computing, it has high potential in domains such as wearable cognitive assistance and web applications 3.10.2. Cloud of things companies. Another similar concept to mist computing is the Cloud of Cloudlet is the middle tier of a 3-tier continuum: mobile device- Things (CoT) [61] , where IoT devices form a virtualized cloud cloudlet-cloud. Given the nature of cloudlets as a small cloud close to infrastructure. In mist computing computation is done on IoT mobile devices, operators for cloudlet computing could be cloud ser- devices, possibly via message exchange, and not necessarily in a vice providers who want their services to be accessible closer to mobile cloud of pooled resources. However, in Cloud of Things, computa- devices. Network infrastructure owners (e.g., AT&T, Nokia, etc.) can en- tion is done over the cloud that is formed by pooled resources of able cloudlets with virtualization capacity to be situated closer to mobile IoT devices. Abdelwahab et al. [61] introduce the notion of Cloud devices, in smaller hardware footprints compared to the massive data of Things for sensing-as a service, which uses edge nodes as cloud centers used in cloud computing. The small footprint of cloudlets result agents sitting close to IoT nodes. The authors propose the idea of dy- in more moderate computing resources, but lower latency and energy namically scaling up existing cloud resources (compute, storage, and consumption compared to cloud computing. Cloudlet computing is in- network) by using the sensing capability of IoT devices. Edge nodes are tended to serve devices in the local area. used as cloud agents near the edge to discover, virtualize, and form a Just as MACC greatly differs from cloud computing, it also highly dif- cloud network of IoT devices (CoT). This network is a geographically fers from cloudlet computing. Cloudlet needs infrastructure with virtu- distributed infrastructure, in which cloud agents constantly discover realization in the form of virtual machine (VM) capability, whereas MACC sources of IoT devices and pool them as cloud resources. CoT enables does not require such infrastructure. MACC and cloudlet computing both remote sensing and in-network distributed processing of data. For in- support mobility, but MACC is resource constrained and lacks virtual- stance, a cloud user can view pollution levels in cities from real-time ization support for real-time IoT applications. Cloudlets support local temperature and CO2 concentration sensors in vehicles with defined ac- services for mobile clients by dividing tasks among cloudlet nodes in the curacy. The CoT framework is scalable to IoT networks, supports het-

296 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

on low local

WiFi,

devices vehicles, with phones,

(e.g. processing

distributed

IoT ZigBee provided connectivity

cell computation

devices, devices

appliance Bluetooth, be

processing close

operate

devices devices

IoT IoT intermittent

Parallel on autonomous privacy-preserving local Self-organized businesses IoT sensors, home devices) Distributed on Must IoT Can or Internet LAN, cellular, Through Localized/ — Low — mist Low General Limited Local Moderate Very Academia

the of

nano

WiFi,

Glass)

with

no and

edge

providers the often

virtualization

infrastructure

devices

with Internet

WLAN, (OCR),

nodes

(Google (micro

cognitive

resource-rich

connection provided

with

character

LAN, service latency

be computation mobile

centers)

operate network

Internet

Optical recognition wearable assistance Cloud and providers Devices capability data High lower Localized Must participant Near Can intermittent connectivity; requires Internet cellular Through computers the Moderate WAN, OpenEdge cC High Mobile Moderate Local Low Close Academia

the

AP)

RAN

in or the

devices

ITU-T

data reality

(RAN,

the

vehicles,

RAN

provided through

Delivery, to

with installed

monitoring,

edge cellular analytics,

3GPP,

be network

virtualization operate

latency

global

the

Content Video connected health augmented infrastructure providers (RAN-based) centers with capacity, and Low computation hierarchical Must edge equipment edge May autonomously connect Internet WAN, At Internet High Localized/ Network Small-scale MEC Average Mobile Less Moderate Low Close Academia/industry

video

the limited

disaster

with

live and

latency for

unmanned WiFi,

less

system

mobile ZigBee provided

devices devices

group

be computation close

lower

operate

intermittent

Networking computing relief, streaming, vehicular Self-organized Mobile with Distributed Must mobile —Can Low Can autonomously or Internet cellular, devices —ETSI, Bluetooth, Through High MACC Relatively Low Mobile Local Moderate Very Academia

with

results devices

capability

large

health for mobile and

data

tasks

high

cloud buildings

Internet cloud in core

provided

mobile

cloud with providers devices devices

data

networking,

cloud-to-things and

be computation

the

virtual

on (tele-monitoring

tele-surgery)

Social sensor processing, care and Users service Mobile large-scale centers with High distributed devices Must along continuum mobile dedicated devices Requires connection oﬄoading obtaining computation from WAN Through Central Installed MCC High Mobile High Relatively Local Low Far Academia

with ZigBee the

with video

local

WLAN,

capability distributed devices

traﬃc provided connectivity

edge LAN,

cellular, video

be devices edge devices

latency

operate

the intermittent

Local surveillance, caching, control Network infrastructure providers businesses Edge computing —NIST Low computation Localized/ Must edge Near Can autonomously or Internet WAN, WiFi, At Internet Moderate EC General Average Low Local Low Academia/industry Close

the

with

city,

edge

at service points)

vehicles,

Internet delivery),

with

capacity

the

smart

WLAN,

nodes

dedicated close cloud

access

subsurface Consortium,

video connected routers,

(high-scale drone

provided

with in

from

installed

care,

LAN, latency grid/smart

and cellular

be computation

global

Connected operate be

core

intermittent

IoT, smart health delivery package real-time imaging, surveillance providers virtualization (servers, switches, IEEE High lower hierarchical Must participant edge locations Can autonomously or connectivity WAN, WiFi, Through devices the OpenFog Decentralized/ Users Moderate Devices FC General High Less Low Academia/industry Relatively

low

location on with

and

queries WiFi,

mobile ZigBee provided connectivity

sales devices devices processing devices

be close

operate

intermittent

Mobile transactions, dependent (travel recommendations), multimedia applications mobile Self-organized Mobile MobileInfo mobile Distributed Must mobile —Can Can or Internet cellular, devices Limited Bluetooth, MC Mobile Low Local Moderate Very Academia Through Distributed —Low

paradigms.

the

sets storage,

devices NIST,

for

large services data

computing apps,

high high buildings

in of core data data connected provided

MapReduce)

with computation

related service

cloud-to-things

GICTF be be

DMFT,

virtualization

Internet large

virtualized distributed for (Google providers centers with capacity High CSA, OCC, hierarchical along continuum dedicated Must the duration WAN General CC Relatively Must Cloud Large-scale Centralized/ High Global Academia/industry Ample Installed Through Relatively Scalable Far

Users

fog-computing

Cases

Computing

from

Use

Type Access

Location Consumption

Driver 2

Attribute Users General Operators Service Hardware Available Resources Main Distance Main Standardization Entity ApplicationType Architecture Availability Latency Security Server Power Internet Connectivity Hardware Connectivity Service Table Attributes

297 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Table 3 Features of fog-computing related paradigms.

Feature CC MC FC EC MCC MACC MEC cC mist √ √√√ √ Heterogeneity support ✗ ✗ ✗ ✗ √ √√√ √√√ Infrastructure need ✗ ✗ √√ √√√ Geographically distributed ✗ ✗ ✗ ✗ √√√ √√√√ Location awareness ✗ ✗ √√ √√√ Ultra-low latency ✗ ✗ ✗ ✗ √√√√ √ √ √√ Mobility support ✗ √√ √√√ Real-time application support ✗ ✗ ✗ ✗ √ √√ √ √ Large-scale application support ✗ ✗ ✗ ✗ √√ √√ √ Standardized ✗ ✗ ✗ ✗ √ √ √√ Multiple IoT Applications ✗ ✗ ✗ ✗ ✗ √ √ √√ Virtualization support ✗ ✗ ✗ ✗ ✗

erogeneity of IoT devices and edge computing nodes, and provides a balanced between edge and cloud is a significant advantage of osmotic foundation of sensing-as a service using fog computing. computing. Similar to CoT, the authors in [62] propose the concept of PClouds (personal clouds), which are distributed networked resources that are 3.11. Concluding remarks from both local/personal and remote/public devices and machines. PCloud can service end users even when remote cloud resources are The previous discussion about fog computing and related paradigms not present or difficult to access due to insufficient network connec- demonstrate the importance of understanding the characteristics of tivity. Another novel idea similar to Cloud of Things and MACC is the these platforms in the changing IT landscape. As demonstrated by the work of the authors in [63] , where they propose Cloudrone, an idea of strength and weaknesses attributed to these computing paradigms, some deploying ad hoc micro cloud infrastructures in the sky using low-cost paradigms may be better suited for a particular use case than others. drones, single-board computers, and lightweight OS virtualization tech- Even so, fog computing is suited for a large number of use cases in the nologies. The drones in this scheme form a cloud computing cluster in current landscape of IoT and connected devices. The versatility of fog the sky, which provisions the cloud services nearer to the user, even in computing makes it suitable for many cases of data-driven computing the absence of a terrestrial infrastructure to access the remote cloud. and low-latency applications, even though it may not be suitable for Similar to the concept of Cloud of Things, Femtoclouds have been a few extreme applications, such as disaster zones or sparse network proposed to tap into the computational capability and pervasiveness of topologies where ad hoc computing (e.g., MACC) or extreme edge clouds underutilized mobile devices. Femtoclouds take advantage of clusters (e.g., mist, CoT) may be a better fit. Nonetheless, fog computing is con- of devices that tend to be co-located in places such as schools, public sidered a more general form of computing when compared to other sim- transit, or malls. A hybrid edge-cloud workload management scheme is ilar paradigms (e.g., EC, MEC, cloudlet), because of its comprehensive proposed in [64] for management of resources and tasks in femtoclouds, definition scope, generality, and extensive presence along the thing-to- to provide low latency. cloud continuum. Tables 2 and 3 summarize these characteristics. Fog computing offers a bright future for an open-standards environment of connected devices, as it is evident by IEEE Standard’s adoption of Open- Fog Reference Architecture [69] . 3.10.3. Edge cloud There does not yet exist a globally unanimous distinction between When we talk about cloud computing, we mainly talk about “core ” fog computing and related computing paradigms, such as edge comput- or “distant ” clouds, which are far from the user or devices. Core clouds ing, mist computing, and cloudlets across researchers and industries, as are further from connected things and are responsible for heavy com- shown in the previous sections of this paper. We attempt in this survey putation. In contrast, “edge ”clouds are smaller scale compared to core paper to clarify the distinctions between fog computing and the related clouds and are closer to the devices. The concept of edge cloud [65] is computing paradigms. A comparison of the underlying infrastructure of similar to edge computing. The edge cloud extends cloud capabilities at fog computing and its related computing paradigms from the network- the edge by leveraging user or operator-contributed compute nodes at ing perspective is shown in Fig. 7 . In the rest of this paper, we will the edge of the network. Similar to fog, in edge clouds the ability to run mainly survey and discuss the recent literature on fog computing, but an application in a coordinated manner in both edge and the distant mention the studies on other related computing paradigms that could cloud is envisaged. Edge clouds are nodes at the edge, such as micro be easily extended or directly applied in fog. data centers, cloudlets, and MEC [66] . Researchers have begun studying federation of both edge clouds and 4. Taxonomy of fog computing through a complete survey core clouds, and proposed the “osmotic computing ” paradigm [67,68] . Osmotic computing implies “the dynamic management of services and In this section, we will introduce a taxonomy of the research in fog micro-services across cloud and edge data centers, addressing issues computing that is the basis of this survey 2 . This taxonomy categorizes related to deployment, networking, and security ” [67] . Osmotic computing utilizes both edge and cloud resources, each contained in two separate layers. Application delivery follows an osmotic behavior where 2 The data (categories and features/objectives of the papers) of this survey virtualized micro-services are deployed opportunistically either in the are available in the form of several datasets at https://github.com/ashkan- cloud or edge layers. The ability to control how micro-services can be software/fog-survey-data .

298 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Fig. 7. Comparison of the infrastructure of fog computing and its related computing paradigms from the networking perspective.

Table 4 Overall categories of the papers cited in this survey.

Category Subcategory Papers

Foundations Surveys [8,9,13,43,67,71,78–86,97–105] Deﬁnitions & Standards [6,66,68,70–77,85,87,106–123]

Frameworks and Programming Architectures and Frameworks for Fog [6,27,70–72,75,77,83,87–96,106,109,110,112,115,124–170] Models Concepts and Frameworks using Fog [61,64,118,123,171–193] Programming Models and Data Modeling [98,107,132,134,135,143,184,194–209]

Design and Planning Infrastructure Design [27,121,128–130,134,210–220] Resource Analysis and Estimation [23,76,113,117,119,121,211,216,221–230]

Resource Management and Service Provisioning (Orchestration & [24,126,132,135,138,139,145,156,162,169,179,197,217,231–254] Provisioning Migration) Placement (VM/Service) [92,110,149,155,157,166–168,174,184,217,255–278] Control and Monitoring [9,54,106,127,129,130,134,135,137,151,160,238,239,245,259,279–287]

Operation Scheduling, Oﬄoading, and Load [54,64,88,90,111,124,131,140,145,160,163,164,186,210,212,218,219,230,235,255,262,286,288– Balancing 321] Resource Discovery [61,116,123,138,143,196,197,204,221,239,290,310,322–329] Applications [20,51,61,100,154,159,174,175,190,191,247,263,271,276,287,306,320,321,323,330–379]

Software & Tools [66,150,195,207,240,322,344,355,380–396]

Testbeds & Experiments [123,139,142,169,173,185,193,348,376,386,394,397–418]

Security & Privacy [66,82,99,102,104,105,108,114,163,165,178,282,283,304,313,326,337,339,341,349,364,368,371, 373,389,419–433]

Hardware & Protocol Stack [27,75,116,122,128–131,146,152,158,164,170,171,176,179,200,212,214,219,266,322,384,408, 409,434,435]

the research articles that focus primarily on fog computing from the the features they provide (e.g., scalability), and we summarize them in networking perspective. We have also included research articles from Appendix, Table 7 . The explanation of the features and objectives along other similar computing paradigms, such as edge computing, if the ar- with several examples for each objective/feature are included in Table 5 . ticle is relevant and general enough that it could be easily extended to We extract these objectives/features such that they are comprehensive fog computing. The taxonomy is shown in Fig. 8 , and the papers on fog and useful for a fog system design, and are also closely in compliance computing, edge computing, cloudlet, etc. that are referenced in this with the pillars of OpenFog architecture [6] . survey are categorized into different categories in Table 4 . We will dis- The “Foundation ”category consists of the research papers that either cuss the literature on fog computing in this section, and we categorize survey the fog computing area, or try to define and standardize the field the research papers according to this taxonomy. Moreover, we have rig- of fog computing. The “Frameworks and Programming Model ”category orously checked the objectives of papers (e.g., QoS improvement) and is where the reader can find research articles that introduce frameworks,

299 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Fig. 8. Taxonomy of fog computing that is used in this survey.

architectures, and programming models for fog, or that use fog comput- the infrastructure and topology of the network, and determine the hard- ing to introduce a new concept (such a vehicular fog computing). The ware and resources that must be placed in a particular design. Next, in next category is “Design and Planning, ”which includes the papers that the Resource Management and Provisioning step, network operators try discuss the design and planning of the network and computing infras- to manage and provision the resources for better utility and efficiency. tructure. The “Resource Management and Provisioning ”category con- For instance, service orchestration and migration methods are used to in- sists of the research papers that study the management and provisioning telligently allocate and provision the available resources of the network of the resources (e.g., service provisioning, VM placement, control and nodes; monitoring techniques are used to monitor the resource usage of monitoring). The category “Operation ” includes the papers that discuss the nodes, for instance for placement decisions (e.g., VM or container operational aspects of fog computing systems (e.g., task scheduling, load placement). Finally, in the Operation step, the final improvements for balancing, and resource discovery). Each of the mentioned categories resource usage and efficiency are performed, such as task offloading and has subcategories, and we will discuss each subcategory in the follow- scheduling, load balancing, and efficient resource discovery. ing subsections. The category “Software and Tools ”will list papers that focus on software, simulators, and tools for fog computing. Likewise, the 4.1. Foundations: definition and standards “Hardware and Protocol Stack ”category showcases articles that propose a protocol stack or introduce particular hardware for fog computing. Pa- In this subsection, we survey the articles that are concerned with pers that focus on developing testbeds or doing extensive experiments defining and standardizing fog computing and concepts related to fog for fog are summarized under the “Testbeds and Experiments ”category. computing. The very definition of fog computing and fog nodes is a topic Finally, papers discussing security and privacy aspects of fog computing of ongoing discussion, and there is no common consensus on what a fog are included under the “Security and Privacy ” category. node is [70] . There are some early efforts to define fog and fog nodes Note that the three categories Design and Planning, Resource Man- [6,70] . OpenFog Consortium is one of the pioneers in standardizing and agement and Provisioning, and Operation reflect conventional steps in defining fog computing. The OpenFog architecture is established to pro- design and operation of distributed computing systems and computer vide a nonproprietary fog architecture and standard to support current networks. First, in the Design and Planning step, network designers es- cloud computing in addition to diverse IoT and edge-oriented ecosys- timate and analyze the required resources for a given network, design tems. The white paper introduces security, scalability, openness, agility,

300 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Table 5 Objectives of the papers and their corresponding explanation.

Objective Explanation Example

QoS The proposed scheme deals with improving the Quality of Service or quality Algorithms for enabling real-time applications, migration engines, task of experience (e.g., by minimizing or controlling latency or success rate) in oﬄoading, dynamic fog service provisioning. the fog.

Cost The work considers cost parameters, such as operational cost or capital cost, Cost-aware replica placement, cost estimation or capacity planning for in the proposed scheme. designing a fog network.

Energy The authors analyze the energy consumption or power in the paper. Energy-aware computation oﬄoading, energy-aware mobility management, federation of constrained devices.

Bandwidth (eﬃciency) The paper discusses and proposes algorithms that aﬀect Fog resource sharing, in-network processing, edge analytics, capacity bandwidth and throughput using fog computing. planning for designing a fog network.

Security Security and privacy aspects of fog computing are considered in these Anomaly detection using fog, location privacy, authentication schemes for studies. These include vulnerabilities, security mechanisms, privacy issues, fog nodes. and security protocols (e.g.authentication).

Foundation The fundamental and foundations of fog computing are discussed in these Surveys, standards, reference architectures, reference frameworks, new papers. concepts based on fog.

RAS Reliability, Availability, Survivability. The proposed scheme improves the Survivable replica placement, availability analysis of fog services, reliability, availability or survivability of the fog, or uses fog to availability-aware VM placement. provide/improve the reliability, availability or survivability, in the event of a network/node failure.

Mobility The paper considers the mobility of IoT devices or fog nodes. Mobility-aware fog node placement, service migration based on mobility. mobility-aware service placement.

Scalability The proposed scheme can eﬃciently scale to the large magnitude of IoT Edge analytics, scalable IoT node management, computation oﬄoading and networks. task assignment (not per task).

Heterogeneity The paper discusses heterogeneity or proposes frameworks that handle Cloud of things, computation oﬄoading among fog nodes, federation of fog heterogeneity of devices. The algorithms and frameworks in the paper do nodes or IoT nodes. not assume any particular type of node or network.

Management The paper proposes management, monitoring, or federation schemes, where SDN-enabled control of fog nodes, fog operating system, orchestration of fog nodes, (or IoT nodes) are managed, monitored or federated using some IoT services on fog nodes, orchestration of fog nodes. method of management (e.g., SDN).

Programmability The proposed framework is a programming language, programming Fog YANG models, distributed data ﬂow for fog, data modeling and labeling. framework or data modeling for fog computing.

among other “pillars ”of an open fog architecture [6] . Later, IEEE Stan- and processing among themselves and with other content islands. The dards Association adopted OpenFog Consortium’s reference architecture islands are based on the integration of a publish/subscribe system with as a standard for fog computing through IEEE 1934 [69] . disruption-tolerant network (DTN) techniques to provide higher flexi- Vaquero et al. [71] take into account mobile device ubiquity, net- bility with respect to data and computation sharing. Another definition work management, fog network connectivity, and privacy to propose in fog computing area is the “reliability factor ”of a node, which de- their definition for fog: a large amount of heterogeneous, ubiquitous, and fines the probability of a node being online or available, and is defined decentralized devices that can cooperate to form a network for storage and in [76] . In [77] ten economic aspects of fog computing (referred to as processing without third-party intervention . The authors in [72] focused fogonomics) are introduced. on the theoretical modeling and performance metrics of the fog computing architecture. They propose a mathematical formulation for fog 4.2. Foundations: surveys computing by defining its components for a generic fog architecture. Current communication technologies and standards that could be In this section, we discuss the previous work in the fog computing used in fog networks are presented in [73] . In the paper, a classifica- domain that are of survey or tutorial nature. Several comprehensive sur- tion of layers and technology settings related to IoT and fog comput- vey papers are already discussed in Section 2 ; in this subsection, we will ing is described. On the other hand, the authors in [74] define “class discuss some other survey papers that are not as comprehensive as the of service ”for fog applications, a classification of fog services accord- ones discussed in Section 2 . Klas from the Vodafone Group presents a ing to their QoS requirements. They also introduce a mapping between clear picture of edge computing and its benefits [78] . This paper high- certain classes of services and the corresponding processing layers of the lights relevant edge computing applications and similar research areas fog computing reference architecture. (e.g., fog and cloudlet), surveys the latest industrial efforts and available With emerging availability of IoT devices and their large volume of edge computing technologies, and discusses edge computing’s potential data that they produce, timely and reliable transfer of large data streams to improve the telecommunications industry. to a centralized location is a requirement of deep learning models. The Historically, the field of computing has seen cycles between de- authors in [68] introduce “deep osmosis ”and analyze the research chal- centralization and centralization [43] . The authors of [43] , in a sur- lenges involved with developing edge-cloud-based holistic distributed vey study, advocate for edge-centric computing, a more decentralized deep learning algorithms and their corresponding resource models and paradigm that utilizes peer to peer (P2P) networking at the edge of the architecture for cloud and fog computing. network while maintaining access to the cloud. The study in [75] defines the concept of “content island ”for fog The authors in [79] present a comprehensive study of edge comput- computing, which interconnects groups of devices to interchange data ing, starting with the factors leading up to its development, the advan-

301 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Fig. 9. Our three-layer architecture for IoT-fog-cloud ecosystem. tages of edge computing, requirements for successful implementation, computing and MEC separately have weaknesses and incompleteness; application use-cases, and challenges. Several use cases, such as, gam- they further claim the need for convergence of the two computing ing, real-time image processing, smart grid, smart transportation, are paradigms for overcoming such limitations. A three-layer general logi- used to emphasize the range of problems that edge computing can help cal architecture for fog computing is introduced in [88] and [89] . The alleviate. Weisong et al. define edge computing and survey its use cases, layers are IoT, fog, and cloud, where each layer is partitioned into do- research issues, and future research directions in [80] . A recent survey mains. Similarly, a three-layer architecture including the cloud, MEC, on MEC can be found in [81] . and IoT is proposed in [90] . In the three-layer architecture proposed, Stojmenovic and Wen in [82] highlight privacy and security concerns the user plane consists of mobile users and IoT devices, the edge com- of fog computing gateways as major issues. These issues include man- puting plane consists of edge servers in close proximity to the users, in-the-middle attacks and lack of encryption in gateways that serve as and the cloud computing plane is the core of the network and contains fog nodes. The study in [83] surveys the main features of fog computing, multiple cloud servers and data centers. Comparably, we propose our describes its architecture and design goals, and discusses some potential three-layer architecture for fog computing in Fig. 9 . issues of fog computing in 5G. One of the early surveys in fog architec- The authors in [91] propose a fog-to-cloud architecture, consisting ture and taxonomy is the work of Zhang and Chiang [84] . The authors of a layered management structure that can bring together different het- further describe the IoT challenges for which fog computing can provide erogeneous cloud/fog layers into a hierarchical architecture. The paper solutions. The survey in [85] distinguishes and explains edge computing, [92] designs a hierarchical edge cloud architecture, to efficiently uti- fog computing, and cloud computing. The study reviews various system lize the cloud resources for serving the peak loads from mobile users. architectures, application characteristics, and platform abstractions of The proposed architecture consists of servers at the edge, which directly fog, edge, and cloud. receive workloads from mobile devices via wireless links. These edge In the paper [86] , the authors overview fog computing model servers are connected to higher tiers of edge cloud servers and remote architecture, key technologies, and applications. They present the data centers through the Internet backbone. Different from directly us- hierarchical architecture of fog computing and its characteristics and ing a flat collection of edge cloud servers, the proposed architecture compare it with cloud computing and edge computing. Then, the key aggregates the peak loads that exceed the capacities of lower tiers of technologies for fog are introduced to see how they support fog com- edge cloud servers to other servers at higher tiers in the edge cloud puting deployments. hierarchy.

4.3. Frameworks and programming models: architectures and frameworks 4.3.2. Fog computing resource model for fog One challenge in fog computing is defining who the fog resource providers are. Is it the case that fog service providers must provide fog Many researchers have independently proposed various architec- resources? Can end users can bring their devices and share their re- tures and frameworks for fog computing. In this section, we summarize sources? Do network providers offer their edge resources for renting? the previous work that have proposed general architectures or frame- The articles [93] and [94] present a unified computing, caching, and works for fog computing. communication (3C) solution for 5G that allow service, content, and function providers to deploy their services/content/functions near the 4.3.1. General architecture for fog computing end users. The solution also allows for the exploitation of extreme edge A recent study suggests an architectural model for combining MEC resources by enabling their owners to form virtual fogs (vFogs) cooper- and fog computing for 5G networks [87] . The authors claim that fog atively; that is, end users will have the ability to become 3C resource

302 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 providers to the 5G ecosystem. The authors also propose their architec- user privacy [106] . Sun and Nirwan use OpenFlow SDN switches to pro- ture for fog computing, which consists of vFogs, hyper fogs (constella- vide network management for aggregated data from IoT devices. The tions of vFog networks to facilitate processing and data exchange that authors of [110] propose a hierarchical MEC architecture for resource requires resources from more than one vFog), super extreme edge node, allocation in MEC. The architecture introduces the notion of field, shal- regular extreme edge node, and orchestrator. low, and deep cloudlets, where the field cloudlets are collocated with One of the many efforts to design a reference framework and in- the base stations, the shallow cloudlets are at aggregation points, and frastructure for the fog-based IoT that considers resource sharing of the deep cloudlet is at the mobile backhaul. The architecture can handle consumers is Indie Fog [95] . The Indie Fog infrastructure utilizes con- peak loads efficiently by utilizing the shallow and deep computing fa- sumers’ equipment (e.g., WiFi access points) to provide fog services for cilities at higher levels when the computing capacity of a field cloudlet IoT devices. The authors suggest that network infrastructure providers is not enough to handle the loads from its corresponding mobile users. or cloud service providers can make use of the consumer premises Lingen et al. [134] focus on a unified approach for computing in fog equipment to provide their fog-based services. Under this model, they and cloud computing. They argue that fog computing and cloud com- claim that consumers will be willing to share their equipment with the puting should not be complementary paradigms, but instead should be providers for offering their services. Indie Fog uses the general fog ar- fused together. As a result, through the authors’ architecture, compute chitecture proposed by the OpenFog Consortium [6] and adds the Indie nodes in the fog and cloud have the same architecture, and resources Fog services to it, which are interconnected via virtual connections to are managed in a unified way. The architecture is extended from the private and public fog networks. European Telecommunications Standards Institute (ETSI)’s standardized reference architecture for NVF management and orchestration (MANO). 4.3.3. Fog architecture design decisions In an article from Cisco [96] , the author proposes a general high- 4.4. Frameworks and programming models: concepts and frameworks level architecture for fog networks, fog software, and fog nodes. This using fog paper is an early attempt to review and characterize the design/decision parameters of fog networks. The author names these decision criteria Several studies utilized the concept of fog computing to propose new “fog architectural imperatives, ”and discusses them in detail. The fog concepts, ideas, and frameworks based on fog computing. architectural imperatives are decisions related to design requirements that are difficult to implement on networks with sole reliance on cloud or IoT devices, and that can only be satisfied by using fog resources [96] . 4.4.1. Vehicular fog computing The authors in [172,176] proposed the idea of vehicular fog comput- 4.3.4. Fog architectures for 5G and IoV ing (VFC) by utilizing vehicles as the infrastructures for communication Fog computing is seen as a promising enabler for some of the emerg- and computation. VFC takes advantage of a dynamic group of vehicles ing paradigms, such as 5G, autonomous cars, and Internet of Vehicles to help increase computational power and decrease latency issues. Dif- (IoV). In their article [158] , the authors propose an SDN-based frame- ferent from the vehicular cloud computing, the proposed VFC supports work for cloud-fog interoperation in 5G wireless networks. Vilalta et al. geo-distribution, local decision making, and real-time load-balancing. propose TelcoFog - a fog computing architecture that is deployed at the Moreover, VFC depends on the collaboration of near-located vehicles, network edge for telecom operators to provide cost-effective 5G services instead of relying on the remote servers, which reduces the deployment for low latency and scalability [135] . TelcoFog consists of three main costs and delay. types of components: scalable TelcoFog nodes, TelcoFog controller, and An architecture for VFC is presented in [176] and is comprised of TelcoFog services. The paper [145] introduces the challenges of han- three layers: the application and services layer, the policy manage- dling big data in the IoV environments. The authors emphasize on the ment layer, and the abstraction layer. The application and services layer role of fog servers and describe a regional cooperative fog computing offers a variety of real-time applications as well as new services to (CFC) architecture to support IoV applications. The proposed CFC-IoV users, whereas the policy management layer is responsible for allocat- architecture consists of two layers - the fog layer and edge layer. The fog ing resources to the tasks and handling basic issues such as monitoring layer is a federation of geographically distributed fog servers, a coordi- system state dynamically. The abstraction layer is responsible for man- nator server, and the cloud servers, whereas the edge layer includes the aging, provisioning, and interfacing with the physical resources and for vehicular ad hoc network (VANET), IoT applications, and mobile cellu- the security and privacy of the VFC architecture. The benefits, architec- lar networks. Other effort suggesting fog architectures for 5G or IoV are ture, use cases, and potential issues of VFC are presented in [178] . The [83,87,93,128,134] . authors proposed a high-level architecture of vehicular fog computing, which comprises of three types of entities, namely smart vehicles as the 4.3.5. ICN-based fog architecture data generation layer, roadside units/fog nodes as the fog layer, and The study in [143] brings together fog computing and information- cloud servers as the cloud layer. centric networking (ICN), which enables flexible and efficient data dis- Similar to VFC, unmanned aerial vehicles (UAVs) have been consid- tribution at the network layer. In the introduced ICN-Fog architecture, ered as means to provide computing capabilities [186] . In this model, at the lowest layer are heterogeneous end devices that connect to fog UAVs act as fog nodes and provide computing capabilities with en- nodes, which run ICN-specific protocols to communicate with other fog hanced coverage for IoT nodes. Similarly, the concept of vehicular micro nodes. Apart from connecting to other fog nodes, each fog node is also clouds based on map information is introduced, and, by a simulation connected to the cloud. The authors note that ICN-Fog relies on the prin- study, investigated in [183] . Vehicular micro clouds are virtual edge ciples of ICN for building smart, horizontal fog-to-fog data communica- servers and are essentially clusters of cars that help to aggregate and tion that leads to reducing the application’s dependency on the cloud preprocess data that is transferred to the cloud. and distributed processing in the fog. Similarly, the authors of [100] ex- The study in [177] argues that cloud and fog computing using the plore the idea of combining Information-Centric Networking(ICN) with current mobile networks may not be ideally suited to provide the desired MEC to address mobility related issues in the MEC approach that relies levels of QoS for moving electric vehicles in vehicle-to-grid (V2G) ser- heavily on the underlying host-centric networking model. vices. They propose a hybrid computing model called “Foud, ”in which the cloud and fog come together and are made available to the V2G sys- 4.3.6. Resource allocation frameworks tems. In the proposed model, the cloud allows virtualized computing, Sun and Ansari introduced EdgeIoT, a hierarchical architecture that storage, and network resources to be available to the V2G system enti- aims to allocate resources through the use of VMs while maintaining ties, whereas the fog temporarily integrates the stationary and mobile

303 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 computing resources located at the edge of V2G networks to expand the CloudPath (a platform for path computing). CloudPath enables dynamic computing capacity. installation of light-weight stateless functions, and a distributed even- tual consistent storage system. CloudPath also automatically migrates application data across data centers to minimize latency and bandwidth 4.4.2. Beyond conventional fog nodes usage. Prazeres et al. [173] proposed a new paradigm called fog of things

(FoT) which uses fog computing platforms for the IoT. The authors note 4.4.6. Fog nodes as IoT hubs that, in the proposed FoT, IoT services are defined at the edge of the A proposed application of fog nodes to allow for the interoperability network and are distributed through message and service-oriented mid- of heterogeneous IoT devices is presented in [171] . These specialized fog dleware. Additionally, fog of things is self-organized, consists of FoT nodes are referred to as “IoT Hubs. ”IoT devices have different computa- devices, FoT gateways, and FoT servers, and can deliver IoT services tional power and different energy requirements; thus they are restricted in a distributed manner. With the described FoT paradigm, the authors to different communication protocols. The IoT Hub serves as a bridge further propose a platform for the actual implementation of the FoT between all the different physical networks and then merges them all paradigm. The authors in [185] propose human-driven edge computing using an all-IP network. (HEC) as a new model to ease the provisioning and to extend the coverage of traditional fixed MEC solutions by utilizing devices that humans carry. 4.5. Frameworks and programming models: programming models and data

The study in [175] looks at the latency issues that may be ex- modeling perienced by delay-sensitive IoT applications due to the geographical distances between the physical IoT devices and the data centers. In this subsection, we survey the studies that introduce and propose

The authors consider the mobile IoT federation as a service (MIFaaS) programming models or data modeling tools for fog computing. paradigm that leverages the pool of devices managed by individual cloud providers as a whole in order to help support delay-sensitive ap- 4.5.1. Distributed data modeling and frameworks plications. The network model considered is a cellular IoT environment With the rise of IoT, the demand for distributed big data analytics with multiple LTE femtocells as the network edge nodes that supports follows; however, data is rarely shared between stakeholders due to re- the MIFaaS paradigm. source, security, and privacy concerns. Zhang et al. [194] proposed a new programming model, called Firework, that takes advantage of fog computing to enable safe and reliable distributed data sharing between 4.4.3. Fog for transparent computing stakeholders. Firework merges geographically distributed data through The paper [179] examines the question of how to leverage trans- the creation of virtually shared data views that end users can access via parent computing to build scalable IoT platforms and proposes a interfaces defined by the data owners. An example use case of Firework tailored, transparent computing architecture for IoT applications. Trans- proposed by Zhang et al. is the shared data from security cameras. Secu- parent computing eliminates the dependency of hardware and software rity cameras from different sources in a common geographical location and allows the provisioning of cross-platform and on-demand services can be a part of a single Firework instance. Police departments can then on lightweight IoT devices. The proposed architecture consists of sev- request access to a specific footage at a particular time from the datasets eral layers - end user layer, edge server layer, core network layer, cloud in the Firework instance to track a person or object of interest. layer, and the management and interface layer. The end user layer con- One fundamental question in fog computing is the distribution of sists of a variety of IoT devices, and the edge server layer is responsible tasks between fog nodes and the cloud. How should the workload be for distributing computing, control, and storage functions to end user balanced? Should the user be able to choose the balance? If yes, how so? devices at the edge of the network. The core network layer is the core The authors in [98] propose the design of WM-FOG, a flexible software of the Internet, and the cloud layer is composed of a cluster of servers architecture that allows developers to customize policies regarding the with massive computing and storage resources. workflow. WM-FOG allows the developer to define the synchronization policy, to choose how much data is sent to the back-end cloud and how much data is sent to the fog node. 4.4.4. Volunteer edge computing The study in [198] focuses on the distributed deployment of process- Researchers from the University of Minnesota studied the possibil- aware IoT applications. The authors propose a mechanism to design ity of using volunteer resources near the edge for both computing and a distributed IoT application in one place and annotate the different storage, and proposed Nebula. Some existing systems that exploit volun- components of the application which are supposed to be deployed at teer edge computing and data sharing are Grid and peer-to-peer systems different edge nodes with their location information. After defining de- such as BitTorrent, BOINC [436] , and SETI@home [437] . While these ployment locations, the model identifies application fragments which volunteer platforms either are for compute-intensive applications (such need to be deployed in different locations, and annotates each fragment as BOINC and SETI), or file-sharing systems (e.g., BitTorrent), Nebula with the deployment location. It then decomposes the process-aware IoT supports distributed data-intensive applications through a close interac- application into a set of fragments based on the annotated locations. tion between compute and storage resources [182] . Nebula utilizes edge Crystal [205] provides an abstraction for fault-tolerant and distributed resources for in-situ data-intensive computing, through location-aware fog application development. A fog application using Crystal can take data and computation placement, replication, and recovery. full advantage of location transparency, self-healing, auto-scaling, and mobility support.

4.4.5. Path computing 4.5.2. Fog programming models Path computing is paradigm based on the fog computing paradigm, The paper [107] studied QoS-aware application deployment in fog where a multi-tier cloud architecture is deployed over the geographic infrastructure from the programming perspective. They present a proto- span of the network. Path computing provides storage and compute re- type to support application deployment in the IoT-fog-cloud scenario 3 . sources along a succession of data centers of increasing size, located As a future direction, one could consider other QoS attributes and in- between the IoT devices and the cloud data centers, and enables the deployment of a multi-level hierarchy of data centers along the path that traﬃc follows [184] . The authors in [184] propose path computing and 3 Available at https://github.com/di-unipi-socc/FogTorch .

304 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 clude cost information to get a richer classification of eligible fog de- is due to lack of standardized protocols and definitions for fog in gen- ployments. Moreover, one could account for multiple and multi-tenant eral. Nevertheless, there are some early efforts, and we discuss them in deployments on the same fog infrastructure. Renart et al. proposed this subsection. The authors of [214] propose a framework for cloudlet- an edge-based programming framework that enables users to define based network design and planning. The goal of the study is to design data-driven reactive behaviors based on the content and source of data a network based on time division multiplexed passive optical networks streams [202] . Users define how data stream must be processed on the (TDM-PON) to optimize the network infrastructure cost while satisfy- edge based on their content and the location of the data source. ing latency constraints. Fan et al. address the problem of placing the Saurez et al. propose Foglets, a programming model that fa- cloudlets to minimize the deployment cost of cloudlet providers and cilitates distributed programming across fog nodes [197] . Foglets to minimize the end-to-end delay of user requests [210] . Since the to- provides APIs for spatio-temporal data abstraction for storing and re- tal deployment cost of a cloudlet provider depends on the location of trieving application-generated data on the local nodes. Through the cloudlets and the amount of resources for cloudlets, cloudlet providers Foglets API, Foglets processes are set for a certain geospatial region and must take into account both the end-to-end delay of user requests and Foglets manages the application components on the Fog nodes. Foglets the deployment cost. is implemented through container-based visualization. The Foglets API takes into account QoS and load balancing when migrating persistent 4.6.1. Virtualization-based infrastructure (stateful) data between fog nodes. The paper [213] addresses the tradeoff between computing and communication and propose an architecture based on F-RAN. The F-RAN- 4.5.3. Fog software stack for android based architecture consists of radio access equipment, F-RAN nodes that The authors in [195] have developed a software stack for Android to provide the computing resources, end devices, and an F-RAN controller implement fog computing by using smartphones as fog nodes. To do this, that is in charge of receiving service requests and distributing tasks to several challenges need to be considered. For example, mobile operat- fog nodes. The authors observe that this architecture can meet the de- ing systems typically do not have priorities when dealing with memory mands of ultra-low latency applications by relying on front-haul wireless allocation, it is hard to track the resources used by each app, and it communications and distributing computation tasks to multiple F-RAN is difficult to ensure adaptability with current software. The proposed nodes near the end users. software stack is split into four components, each handling the above Fog networks can be implemented by SDN and virtualization to re- challenges. duce the management costs, and to improve the scalability and resource 4.5.4. Paas for fog utilization. The authors in [130] propose an integrated network archi- Current platform as a service (PaaS) models are intended for tradi- tecture for software-defined and virtualized radio access networks with tional cloud applications that are not latency-sensitive nor large-scale. fog computing. The proposed architecture is hierarchical and has two However, using fog computing can be difficult due to the orchestra- control levels: the higher level that is the SDN controller and the lower tion of highly dynamic and heterogeneous fog resources. Hong et al. layer that is the local controller, which could be collocated with fog [199] propose a PaaS model called Mobile Fog that provides a pro- nodes. The SDN controller instructs the local controllers how to process gramming abstraction and allows applications to easily use fog resources specific applications or requests. The authors further introduce a SaaS while supporting dynamic scaling at runtime. The Mobile Fog communi- called OpenPine that enables virtualization at the network level and user cation API is composed of event handlers that must be implemented and control of network operation. standard functions that can be called by the application. The same code can be run on a different device like smartphones, vehicles, or cameras; 4.6.2. Capacity planning the developer only needs to write the code once. The authors of [216] address the questions of where the edge data Similarly, the authors in [132] have proposed a PaaS architecture for centers must be located and how much compute capacity needs to be automating application provisioning in hybrid fog/cloud environment. allocated to each DC for cost-effectiveness while also satisfying the To extend existing PaaS, the authors used Cloud Foundry droplets (an bandwidth and performance requirements of applications. They con- open-source PaaS service), Docker containers, and REST to provide include that adding edge layer data centers results in high-cost savings for teractions between PaaS and the fog. This enabled their architecture to network-intensive applications while adding an intermediate data cen- provide development, deployment, and management phases of hybrid ter close to the root data center is beneficial for medium to low demand cloud/fog applications. In the proposed architecture, the controller ex- compute-intensive applications. Noreikis et al. improve edge resource poses an API for PaaS users to allocate resources for their applications utilization by taking advantage of network resources such as GPUs in in the deployment phase. The Cloud Foundry droplets are responsible addition to CPUs [211] . Their study includes initial capacity planning for running the applications in the management phase. in edge nodes to meet QoS requirements.

4.5.5. Service modeling 4.7. Design and planning: resource analysis and estimation Lingen et al. [134] used the YANG modeling language to model fog services and devices, which allows for better resource orchestration and As many publications on fog computing highlight the importance of system design. In addition, modeling services and devices using YANG this technology for ubiquitous access, service provisioning, and service allows the authors to transform service intention into service instantia- discovery, several studies also focus on the resource pricing, estimation, tion in their architecture. The authors then break down the complexity and analysis of fog computing. In this subsection, we describe these stud- of service modeling for reuse and building higher-level services for bet- ies. ter overall service management. Through the YANG modeling language, the authors in [135] are able to eﬀectively model IoT services descrip- tion for their proposed controller to handle a large volume of data. By 4.7.1. Fog resource pricing using the YANG modeling language, the controller can optimally allo- In fog computing, fog service providers aim to oﬀer their services cate resources across multiple networks. to their customers as close as possible to the customers’ locations. On the other hand, infrastructure providers try to maximize their infras- 4.6. Design and planning: infrastructure design tructure utility by renting their edge resources to the cloud/fog service providers. An attempt to model the edge resource pricing and auction Network planning and design of fog networks is an important re- between service and content providers and edge infrastructure providers search topic, and yet not many studies are performed in this area. This is described in [113] . Aazam and Huh in [221] formulated an estimation

305 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 for pricing of fog services based on CPU, memory, storage, and band- 4.8.1. Service provisioning width. Their pricing model also includes incentivized pricing for active The study in [248] focuses on dynamic service provisioning in edge customers and takes into account fog device mobility and customer his- clouds from a theoretical perspective. The model in the study captures tory to determine a more fair price. the limited capacity of fog nodes, the unknown arrival process of requests, the cost of forwarding requests to the remote cloud, and the cost 4.7.2. Fog energy estimation of onloading a new service on a fog node. In a simulation study, the Centralized data centers consume a significant amount of energy authors of [126] suggest a conceptual framework for fog resource pro- compared to small distributed servers, or nano data centers. Jalali et al. visioning. They introduce the concept of “fog cell, ”which is a software identify applications that are more energy-efficient when implemented component running on fog devices that controls and monitors a partic- on nano data centers than centralized data centers [23] . In their models, ular group of IoT devices. Using this and other related concepts, they Jalali et al. use a “flow-based ” energy consumption model for equipment model orchestration of IoT devices using a hierarchical cloud/fog re- shared by many users and services, while for network equipment close source control and provide a suitable resource provisioning solution for to end-users, the authors use a “time-based ” energy consumption model distributing tasks among them. Recently, researchers in [169] present based on the amount of time the equipment needs to provide access to the Red Wedding Problem , a real-world scenario motivating the need for services. The authors conclude the best energy savings with nano data stateful computations at the edge. They implement a prototype database centers is in applications that generate and distribute a large quantity for operation at the edge that addresses the issues presented in the Red of data near end-users that is not frequently accessed (e.g., local video Wedding Problem. surveillance in homes). Similar to the above study, the study in [222] focuses on the energy consumption of IoT applications using both fog and cloud. To increase 4.8.2. Service migration the power efficiency of IoT applications, Jalali et al. propose the use of The authors of [232] study the service migration problem in edge both fog and microgrids [222] . In their paper, the authors find that if a clouds, in response to user movement and network performance. The centralized grid provides power, fog computing is more energy efficient solution is based on based on Markov Decision Process (MDP) that con- if there is little computation to be done with the data. However, cloud siders network state and server response time in making migration deci- computing is more efficient if high data processing is required. sions. In [249] , it is suggested to use multi-path TCP for live migration of VMs across edge nodes to improve VM migration time and network 4.7.3. Fog resource estimation transparency of applications. The allocated resources to fog nodes must be elastic, scalable, and Farris et al. define two integer linear programming (ILP) opti- dynamic, because heterogeneous IoT nodes forming IoT networks are mization schemes to minimize QoE degradation and cost of replica highly dynamic, both spatially and temporally. To address this chal- deployment in service replication for MEC [234] . They distinguish lenge, the authors in [223] propose an analytical model that analyzes classic reactive service migration from proactive migration: reactive ser- the required fog resources for some offered IoT workload and scales vice migration is dependent on user movement and accommodates this fog nodes according to incoming IoT workloads. Dynamic scaling of fog movement by locating the most suitable target edge and then starting nodes is scaling up or down allocated resources according to the incom- the process for migration; however, proactive service migration deploys ing IoT workload. Ideally, when the workload is high, more fog nodes multiple replicas of the user service to neighboring nodes. As a future (or more resources on the fog nodes) are provisioned, whereas when the direction, one could study path-oriented proactive migration and trade- workload is low, it may be possible to free up some fog nodes or release offs between the probability of reactive migration and the cost of service the allocated fog resources. replications. Similarly, the authors in [224] take a close look at server utilization. Specifically, the paper studies the relationship between the extent to which MEC is deployed and the resulting average server utilization and 4.8.3. Orchestration frameworks latency. The metric for latency is defined as the distance between the To align deployed applications in distributed systems, Wen et al. un- base stations and the servers processing the traffic, and the efficiency derscore the importance of fog orchestration [24] . The authors develop metric for server deployment and utilization is defined as the ratio be- methodologies for studying fog orchestration systems with a focus on tween the average and peak traffic processed by the servers. challenges in reliability, scalability, and security for fog orchestration. Another effort for service orchestration is the work of the authors in 4.7.4. Estimating load and response time [233] called “Foggy. ” Foggy is a framework for continuous and au- Fog computing can reduce the web response times of modern web- tomated application deployment in the fog. It facilitates dynamic re- sites by enabling the generation of dynamic web contents at edge nodes source provisioning and automated IoT application deployment in fog close to end users. In [225] , the authors proposed a simple formula to architectures. Foggy assumes that IoT nodes can host Docker contain- estimate the lower bound of this reduction of response time. Moreover, ers. Developers push their containerized application package and their they measured the response times of about 1000 popular web pages from requirement specification to an orchestration center, which is a central 12 locations across the world to evaluate whether edge computing is authority that is in charge of monitoring each IoT node’s resources and suitable for offering dynamic web content. Based on their experiments, deploying services on IoT nodes. edge computing can reduce web responses across the world except in The authors in [239] propose a service-oriented middleware that North America and Europe where the round-trip times are considerably aims to distribute services over fog nodes for scalability, and with the short. In [226] , a location-aware load prediction for edge data centers help of SDN, performs QoS-aware orchestration by scheduling flows be- is proposed. For each edge data center, the load is predicted using its tween services. The architecture proposed mainly consists of two com- historical load time series and those of its neighboring data centers. ponents - the service-oriented middleware and the distributed service orchestration engine. The service-oriented middleware abstracts device 4.8. Resource management and provisioning: service provisioning functionalities, allowing all the nodes to act as service hosting platforms, (orchestration and migration) whereas the distributed service orchestration performs orchestration. Similarly, the authors in [132] proposed a Platform as a service (PaaS) Due to the limited storage capacity of fog nodes, proper resource architecture for automating application provisioning and orchestration utilization of fog nodes has a significant impact on their performance. in hybrid fog/cloud environments.

306 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

4.8.4. Virtualization technologies for fog computing VM placement, and QoS constraints. Souza et al. also formulate service The study in [236] explores container-oriented operational frame- allocation in a combined fog-cloud ecosystem in [256] . The authors in works for IoT from the perspective of how lightweight virtualization can [274] address the problem of multi-component application placement help exploit the resources offered by IoT devices. The authors propose to on fog nodes. Each application could be modeled as a graph, where use Docker-based service provisioning in wireless resource-constrained each node is a component of the application, and the edges indicate the IoT environments. To analyze and identify the impact of Docker man- communication between them. agement, the authors consider two use cases, based on which they The researchers in [258] investigate the problem of optimal resource propose two distinct approaches to container-based IoT service provi- provisioning and fog service placement while taking into account their sioning. QoS requirements. The authors define the fog service placement prob- In their proposed framework called FADES, researchers from TUM lem and formulate it as an integer linear programming problem with advertise the use of unikernels (single-purpose standalone kernels) to the objective of maximizing the utilization of the fog landscape. As a isolate and embed application logic into bootable images [245] . FADES research direction, we motivate the reader to consider other constraints is a modular orchestration architecture for handling heavy computa- such as the availability of resources, the reliability of services, and the tion from IoT devices to edge nodes. Each FADES unit supervises a cost of resources. Similarly, an uncoordinated strategy for service place- subset of IoT devices. When needed, FADES pulls the necessary applica- ment in edge-clouds is studied in [273] . tion/service from the cloud to handle the requests of IoT devices. Similar Virtual network functions (VNFs) are network services that provide to MCC, IoT devices resort to computational resources of edge nodes to some network functionality and provide flexible ways to deploy network handle heavy computations. In this sense, FADES is an orchestration services [438] . The study in [257] focuses on the QoS-aware VNF place- framework for pulling required services for IoT devices from the cloud. ment and provisioning problem over an edge-cloud infrastructure. They propose a strategy to determine the required resources and placement of 4.8.5. Provisioning of resource-limited IoT devices VNFs in the two-tier carrier cloud infrastructure while considering SLA To satisfy QoS requirements in an edge cloud built by single-board requirements. They formulate the VNF placement problem as Mixed In- devices (e.g., Raspberry Pi), the University of Cambridge researchers teger Linear Program (MILP). in [240] present a platform, PiCasso 4 , for service deployment based on The paper [260] addresses the problem of allocating computing re- the service specifications and the status of the resources at the host- sources (specifically, containers) in edge networks. The authors intro- ing devices. The architecture of PiCasso consists of edge nodes and a duce a contract model between cloud service providers and edge infras- service orchestrator. Similar to PiCasso, an edge node resource manage- tructure providers for resource sharing. Based on the contract, service ment framework with provisioning and deployment capability is intro- providers can provision their services on the micro data centers owned duced in [242] . The framework integrates a fog node with a cloud server by edge infrastructure provides. This essentially decouples the manage- and supports auto-scaling to manage edge resources dynamically. The ment of the edge infrastructures with that of the service placement per- framework is developed for resource-limited environments and hence formed by service providers. is shown to be simple. The authors then validate the proposed framework on a location-aware and latency-sensitive online game use case 4.9.2. VM placement (PokeMon Go). The paper [261] analyzes the VM placement problem in fog radio access networks (F-RANs) with the objective to minimize the overall 4.8.6. Handover back-haul traffic. The back-haul traffic is incurred in two ways: the VM Handover issues of mobile IoT devices between access points should replication and data transmission to the cloud. When a user connects be considered in designing orchestration or migration frameworks. In a to a fog node and requests an application service, there is no back-haul recent study, it is shown that VM handoff in the edge across cloudlets bandwidth consumption if the fog node has the application VM. Oth- is more than an order of magnitude faster than live migration meth- erwise, the VM has to be replicated on the fog node, or the request is ods currently used in data centers, for typical WAN bandwidths [250] . forwarded to the cloud. They formulate the replica-based VM placement The authors in [243] observe that traditional mobile network handover problem by considering the computing and storage of fog nodes, the user mechanisms cannot handle the demands of fog computation resources service constraint, as well as the edge bandwidth constraint. Similarly, and the low-latency requirements of mobile IoT applications. The au- the study [265] addresses the issues of launching VM replicas and mi- thors propose Follow Me Fog framework to guarantee service continuity grating them for latency-sensitive, computation and memory intensive and reduce latency during handovers. The key idea proposed is to con- applications in a MEC environment. tinuously monitor the received signal strength of the fog nodes at the Guaranteeing availability in a fog network needs a careful design, mobile IoT device, and to trigger pre-migration of computation jobs be- as fog nodes are presumed to be less reliable than always-provisioned fore disconnecting the IoT device from the existing fog node. Compara- cloud data centers. The authors in [268] study VM placement in MEC bly, the authors in [244] use Docker container migration between edge with respect to availability. The goal is to find placement strategies for nodes for service handoff. different types of multi-access edge applications with low cost while satisfying the availability requirements of an application. Comparably,

4.9. Resource management and provisioning: placement (VM and service) the traﬃc-aware VM placement problem in the cloudlet mesh with the objective of minimizing the total inter-cloudlet communication traﬃc

In this subsection, we survey research articles that address placement is studied in [272]. The study in [264] proposes a messaging method problems (e.g., service placement, VM placement, content placement, with low overhead that notiﬁes fog nodes about nearby replica nodes and caching) in fog networks. so that the replica nodes can be used for handling requests instead of depending on cloud storage. 4.9.1. Application (service) placement 4.9.3. Caching Gu et al. developed a linear programming-based two-phase heuristic Analogous to service placement, caching data on fog nodes can also algorithm to compute an optimal solution for service placement in fog considerably reduces the data retrieval delay compared to solely relying computing medical cyber-physical systems [255] . The authors formulate on a central data store. In [267] , the authors propose a cache grouping service placement problem based on user association, task distribution, mechanism for managing content in edge clouds. They also present a cache coherency mechanism to ensure data consistency within a cache

4 Available at https://github.com/AdL1398/PiCasso . group. Another study on cache placement is discussed in [149] , where

307 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 cache placement is done via the MEC paradigm in wireless networks to logic based on fog node status provided by the Resource monitoring satisfy the demands of automated driving services. The service requests module. submitted by autonomous vehicles are first processed by the edge server to determine if they can be processed locally or need to be handed to 4.10.3. Control and monitoring other networks via fog the remote cloud. Vehicular ad hoc networks (VANETs) face many issues such as unre- In their recent study, the authors of [269] study the cache place- liable connectivity, delay constraints, and poor scalability. The authors ment problem in F-RANs and consider flexible physical-layer transmis- of [279] suggest that using principles of fog computing along with SDN sion schemes. They develop centralized and distributed cache placement could solve many of the current problems with VANETs. In their archi- strategies to minimize users’ average download delay while meeting tecture, the vehicles, which act as end-users, are SDN wireless nodes. the fog storage capacity constraints. Similar to cache placement, the These wireless nodes send their data to Road Side Units (RSUs) which cloudlet placement problem is of prominent importance. Xiang et al. are installed alongside road systems. Once the data is sent to an RSU, [270] introduces the concept of movable cloudlets and explores the it is then sent to an RSU controller (RSUC) which is a cluster of RSUs problem of how to cost-effectively deploy these movable cloudlets to connected by broadband. The RSUC is capable of data storage and pro- enhance cloud services for dynamic context-aware mobile applications. cessing. Finally, the RSUCs communicate with the SDN controller, which To this end, the authors propose an adaptive cloudlet placement method has global knowledge of the VANET system. for mobile applications. Another use of edge caching may be for 360° vedeo streaming. Streaming high-quality 360° video is challenging due 4.11. Operation: scheduling, offloading, and load balancing to high bandwidth and low latency requirements [439] . In [440] a prob- abilistic model of common field of view (FoV) for 360° video based on In this subsection, we survey the studies that address job scheduling, previous users’ viewing histories to improve caching performance at the job dispatching, task offloading, and load balancing in fog networks. edge is introduced. 4.11.1. Offline vs. online It is worth mentioning that we discuss several candidate papers 4.10. Resource management and provisioning: control and monitoring among all of the papers in this area since there are a large number of papers in this area and most of them study “centralized ” (referred to 4.10.1. Control and monitoring of fog-based networks as offline ) scheduling, dispatching, offloading, and load balancing. In The traditional network architecture was not designed with high lev- centralized settings, either full information about the tasks, network, or els of scalability of IoT devices in mind. Tomovic et al. [127] and Xu nodes is known; or a centralized entity decides where each task could et al. [281] propose a control and monitoring framework for the fog- be sent. Even though there is a large body of research in this area, a based IoT network that utilizes SDN. Using SDN allows a system to know more challenging setting is the “distributed ” version of the problem (re- the requirements of the network and all of the resources it has, giving it ferred to as online ) where there is no central authority knowledgeable of the ability to handle large waves of data [441] . In the proposed model, the tasks, nor is the full information about the tasks, network, or nodes a global view of the network allows for easy fog orchestration. The SDN known. Note that, in the online scenario, information on each job is controller controls fog orchestration as it can see all the available fog unknown before its release. nodes and their resources such as RAM, storage, and software applications. The authors of [281] use the message queuing telemetry transport 4.11.2. Cooperative offloading and load sharing (MQTT) protocol in addition to SDN to enable effective and reliable de- The authors in [88] propose a delay-minimizing collaboration and livery of IoT data. The SDN controller is a fog node that is the broker offloading policy for fog-capable devices that aims to reduce the service for MQTT clients. delay for IoT applications. They also develop an analytical model to The study [137] explores the idea of implementing a wide area SDN evaluate the policy and show how the proposed framework helps to re- controller for fog and cloud while also taking into consideration the duce IoT service delay. In [310] , the authors proposed a locality-aware problem of minimizing the overall carbon footprint of data centers. The workload sharing scheme for mobile edge computing environments. In paper [280] proposed an SDN-enabled wireless fog architecture by com- the proposed scheme, each node is aware of its neighboring nodes and bining Openflow and distributed wireless protocols. The architecture de- their current workloads and utilizes such information for workload shar- ploys a hybrid SDN control plane to address the limitations of both the ing in case of high workload. Another distributed latency-aware task centralized and distributed SDN controllers. In case of a controller fail- processing and offloading model is proposed in the study [307] . In this ure or overhead, additional controllers are added at runtime as required model, gateways, which act as fog nodes, dynamically exchange their to balance the network performance. Likewise, the study [9] motivates processing and storage capability information. Based on this informa- how edge computing can benefit from SDN, proposes a collaboration tion, fog nodes probabilistically forward their task to their neighbor- model between SDN and edge computing through practical architec- ing fog nodes or the cloud, when there is a limit in local processing or tures, and shows that SDN can feasibly operate within the edge comput- storage. ing infrastructures. The authors in [358] argue that offloading computation-intensive tasks, such as those of image recognition, to fog nodes is not always a 4.10.2. Virtualization for control and monitoring right decision. The last-mile network latency due to the wireless commu- Some studies suggest bringing virtualization of network services to nication may not be tolerable for some applications; also, the fog nodes the edge devices. The authors in [238] argue that the heavy footprint of themselves may become the bottleneck of processing delay, if many today’s virtualization platforms is responsible for preventing them from tasks are offloaded to them. Hence, they propose to use the available being utilized at the network edge. They present Glasgow Network Func- resources on edge devices and propose Precog, a collaborative scheme tions (GNF), a container-based NFV platform that runs and orchestrates between edge devices and fog nodes to prefetch parts of the trained lightweight VNFs. They show that the presented framework has low VNF models of image recognition onto the device. Comparably, the authors instantiation time and memory requirements as compared with other ex- in [313] introduce the notion of edge computing coalition, which is a isting virtualization technologies, making it suited to run on a variety of collaborative edge-based resource pool of small cell base stations with edge devices. On the other hand, the TelcoFog controller [135] extends cloud computing capabilities to serve computation requests. The col- the functionalities of an NFV orchestrator for dynamic deployment of laborative edge computing scheme accommodates more computation virtualized functions. The main component in the TelcoFog controller workloads by offloading workload from overloaded nodes to lightly- is the Resource Orchestration, which defines and enforces orchestration loaded nodes. The coalition framework is based on coalition game

308 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 theory, which follows a payment-based incentive mechanism to form propose a distributed optimization algorithm that solves the problem of stable groups, and which builds a social trust network for managing optimal workload allocation to maximize QoS in terms of response time. risks among edge nodes. The authors in [309] propose and formulate a cooperative offloading 4.11.7. Quality of results in offloading policy between two edge data centers for load balancing. The model is The authors of [111] propose a systematic optimization framework based on a simple rule: if a service request arrives at one data center with the key idea that relaxing Quality of Results (QoR) in applications when its buffer is full, the request is offloaded to the other cooperating where a perfect result is not always necessary. Relaxing QoR alleviates data center and served by that data center. On the other hand, the study the required computation workload and enables a significant reduction in [312] analyzes an offloading policy between multiple fog data centers of response time and energy consumption. For the proposed framework, installed at the edge of the network in a ring topology. The study also the authors consider a mobile edge environment where the computing quantitatively models and estimates the gain achieved via cooperation tasks can be divided, offloaded, and processed in parallel by distributed between neighboring fog data centers in a ring topology. edge nodes. Thus, the goal of the framework is to minimize both response time and energy consumption by jointly optimizing the selection 4.11.3. Offloading in dynamic/uncertain fog networks of edge nodes’ QoR levels and task assignments to all edges. Lee et al. study the problem of fog network formation and task distribution in a hybrid cloud-fog architecture [296] . Their framework differs 4.11.8. Scalability issues in task scheduling and dispatching from other studies on task allocation for fog nodes by accounting for the The problem of task scheduling and dispatching in edge clouds dynamic formation of a fog network. Since the locations fog node neigh- has been studied extensively by several researchers in the area bors is an uncertainty, the authors use an online approach for quickly [231,262,288,303,311] . One example is the work of Tan et al. [303] , obtaining information of the fog network and minimizing computational where they formulate the problem of online job scheduling and dispatch- latency accordingly. Their online k-secretary algorithm allows a given ing in edge clouds. According to this study, the job scheduling problem fog node to observe its unknown environment and determine how to is to determine which task should be served first, and the job dispatch- offload computational tasks. A recent study that investigates the coming problem is to determine where to send the job, based on latency, putation offloading in an uncertain wireless environment is the work of required resources, etc. As a future direction for task scheduling and authors in [308] . The authors study the computation offloading when dispatching, it is suggested to investigate the scalability of the proposed the mobile users’ behavior is subjective and may deviate from rational- frameworks. Often, the framework fails to scale to the large magnitude ity. In this framework, which is modeled as a non-cooperative game, of edge cloud networks, since, for example, when an IoT device gener- users compete for limited communication resources. ates a job, it has to be sent to all edge cloud nodes, or all edge cloud nodes need to calculate some function and inform the IoT device about 4.11.4. Fog offloading for robotics the suitable edge cloud. Often, this has to be done for every job of every The study [292] proposes a scheme to estimate the processing time IoT device; hence the scalability and communication overhead are the of robotic tasks in fog networks. They use OS profiling tools to roughly two major issues associated with the proposed algorithms. measure the execution times of runtime task on heterogeneous hardware. Based on the estimated computation time, the robot can decide 4.12. Operation: resource discovery (using a basic offloading condition) whether to do the computation task locally or offload to a nearby fog node. The authors also profile con- In this subsection, we summarize the studies that have investigated ventional robotic runtime algorithms to estimate computational times the problem of resource discovery or selection in fog computing. By more accurately. To do so, they estimate processing time of multiple im- “resources, ”we mean resources in the fog networks such as IoT nodes, age, video, and map processing algorithms using OpenCV. In their other fog nodes, nearby devices, fog services, etc. study, the authors investigate the problem of computation offloading in fog computing [293] . An optimization formulation is presented con- 4.12.1. IoT resource discovery and selection serving the computation, energy, and communication overheads when The goal in IoT resource discovery and selection is to provide appli- making an offloading decision. Networked robotics is proposed as the cations with global discovery and access of IoT resources irrespective of motivating example, where the objective is to perform a computation- their location [327] . In order to preserve the distributed nature of the ally intensive task over the collected data by robots, to actuate in a short federation of IoT devices, in [327] the service is realized by IoT gateways time. (fog nodes) through a P2P overlay. The service is implemented using a distributed hash table (DHT), where information about all available IoT 4.11.5. Privacy-aware offloading and scheduling resources is stored for global lookup store. Privacy protection is also a challenge when offloading tasks in fog ACACIA [322] uses context awareness and employs LTE-direct for networks. The authors in [304] consider a system in which the mobile service discovery, which is a proximity service discovery technique us- device generates tasks and the objective of the mobile device is to find a ing D2D communication in IoT. In [61] , edge computing nodes are used scheduling policy that minimizes expected long-term cost. The authors as cloud agents near the edge to discover, virtualize, and form a cloud observe that in their formulated problem, the privacy issues of location network of IoT devices, named Cloud of Things. This network is a ge- and usage pattern are ignored. To this end, they propose a heuristic ographically distributed infrastructure, in which cloud agents contin- privacy metric that jointly quantifies location and usage pattern. uously discover resources of IoT devices and pool them as cloud resources. Similarly, in FocusStack [138] , IoT devices can be selected 4.11.6. Energy-aware offloading and scheduling and orchestrated using their geolocation information. Comparably, in Deng et al. focused on investigating power consumption and network [442] a semantic-based and space-efficient routing protocol for IoT ser- delay tradeoff in cloud-fog services by developing a computation-based vice discovery is proposed. framework and workload scheduling [124] . Similarly, Xiao et al. study the workload offloading problem in fog networks [295] . In their scheme, 4.12.2. Fog resource discovery and selection fog nodes in their scheme can process or offload to other fog nodes part The paper [325] examines the problem of discovering surrogates, of the workload that was initially sent to the cloud. Fog nodes decide to which are micro-clouds, fog nodes, or cloudlets, used by client devices either offload the workload to neighbors or locally process it, under a to offload computation tasks in a fog computing environment. In order given power constraint. The authors study the tradeoff between quality to enable the discovery and selection of available surrogates, the authors of service and power efficiency when considering fog offloading. They propose a brokering mechanism in which available surrogates advertise

309 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 themselves to the broker. The broker receives client requests and con- both within and outside the vehicle. The system can further use the siders a number of attributes such as network information, hardware data that is collected by the vehicle-mounted wireless device to track capabilities, and distance to find the best available surrogate for the passenger movements, detect pedestrian flows, and evaluate how exter- client. The proposed mechanism is implemented on off-the-shelf home nal factors impact human mobility, which provides useful analytics to routers. The authors in [196] discuss a comprehensive architecture for transit operators. Another use of resources near the edge and edge com- resource (container) selection in fog nano data centers. They introduced puting is in vehicular applications [356] , shown by the researchers of a 5-layered framework for task scheduling over containers, which selects the University of Michigan. Edge computing is used to do some anal- a container based on energy-efficiency to meet the users’ SLA require- ysis on the user’s interactions with the car’s application, to determine ments. Container selection and task scheduling occur through a coopera- what priority the current interaction should have and how much of the tive game between special middle entities called brokers. For container driver’s attention should be demanded. scheduling and migration, Docker is used, which also helps schedule tasks over containers. 4.13.4. Healthcare Several studies have considered the use of fog in healthcare 4.13. Operation: applications [340,359] . In [359] the authors introduce processing ECG features using fog nodes, which results in low-bandwidth and low-latency data process- In this subsection, we survey the papers related to fog computing that ing. In [340] , the authors present a hierarchical fog-assisted computing have used the concept of fog to develop and propose new applications architecture for remote IoT-based patient monitoring systems. The hi- in other domains. erarchical computing scheme enables partitioning of analytics and decision making between the fog and the cloud and deploys. The idea 4.13.1. Data stream processing is based on mapping the heavy training procedures in the cloud while Cloud-based data stream processing applications are not able to keep outsourcing the trained hypothesis to the fog nodes periodically, and up with geo-distributed IoT systems. Cheng et al. designed GeeLytics, an exploiting the knowledge at the edge. edge analytics platform, to process real-time data streams from network Through the utilization of the MQTT publish/subscribe protocol and edges and in the cloud [336] . To process IoT data streams, the authors fog computing paradigm, Zao et al. implemented a pervasive neuroimag- design their platform to account for unstructured stream data that is ing system to demonstrate the benefits of fog computing [334] . They constantly generated, mobility and colocation of sensors, low latency, used mobile devices to act as an interface, fog servers for brain state clas- heterogeneity, and ubiquity. Jayaraman et al. propose a context-aware sification, and cloud servers for further brain state analysis and archiv- real-time data analytics platform, CARDAP, to enable energy-efficient ing. Thus, the authors were able to take advantage of fog computing to data delivery strategies in mobile crowdsourcing applications [333] . classify brain states in real-time. Other researchers introduced the use of Microsoft Research in their paper [335] has developed a real-time fog computing in measuring ultraviolet (UV) radiation via smartphones video analytics system which relies on edge computing. The proposed [357] . Due to the sensitivity of CMOS sensors in mobile phone cameras system processes live camera feeds from the traffic intersections in the to UV, the researchers have found that mobile phones have potential to city of Bellevue, Washington, and raises alerts on anomalous traffic pat- measure UV radiation. Through fog servers, UV measurement results can terns. The system could be expanded to operate in other cities and can be gathered and improved to provide more accurate UV measurements. identify dangerous traffic patterns to reduce traffic casualties eventually. Similarly, Yang in the article [332] describes general model and archi- 4.13.5. Video and game analytics tecture of fog data stream processing and analytics. The framework in Video analytics are either computation-intensive or bandwidth hun- [331] enables fog nodes to support query evaluations, specifically with gry. Even with mobile cloud computing, there are still issues of un- weather data, in a federated environment in which high data volume is predictable latency, unexpected service outage, and limited bandwidth. expected. The authors in their study [361] propose the use of vehicles as To address this, the authors in [306] present an edge computing plat- fog nodes, to process live video streams from the in-vehicle dashboard- form called Latency-Aware Video Edge Analytics (LAVEA) to provide mounted cameras; unlike smartphones that are constrained in compute low-latency video analytics at places closer to the users. Cloud gaming resources, vehicles can support efficient computing platforms. comes with disadvantages such as long response latency, user cover- 4.13.2. Bandwidth savings age, QoS, and bandwidth cost. The authors in [323] explore approaches The authors of [349] propose a fog-based crowdsourcing framework to deal with the challenges of thin-client massively multiplayer online for precise task allocation and secure data deduplication. Fog nodes in gaming and propose a lightweight fog-based system called CloudFog. their scheme can detect and erase the repeated data in crowdsourced Fog is formed by idle machines that are close to the end-users and con- reports without deducing any information. Fog helps in data size re- nect to the cloud. In CloudFog, the intensive computation of the new duction by erasing duplicate data. The researchers from Georgia Insti- game state of the virtual world is conducted in the cloud which then tute of Technology built AppFlux, a novel mobile app streaming system sends update messages to fog nodes. The fog nodes update the virtual based on edge-clouds, for fast and efficient delivery of mobile apps and world, render game videos, and stream videos to the players. their updates [352] . This approach also relieves users from having to deal with app updates and could potentially save bytes on their mobile 4.13.6. Image and face recognition plans. EdgeCourier proposed in [51] uses edge computing to address The authors of [271] focus on image recognition based mobile appli- the bandwidth issues caused by cloud-based office applications. In a cations that are latency sensitive and are soft real-time in nature. They measurement study, they show that contemporary cloud storage office present the idea of using an edge server as a cache with computing re- services (e.g., word editor, or spreadsheet) consume unnecessary band- sources. The authors show that using an edge server as a typical web width since they transmit the whole file when an update happens. They cache does not reduce latencies much, and therefore propose Cachier. design an effective office-document-aware incremental sync approach, Cachier is an image recognition cache that leverages the spatiotemporal and EdgeCourier, that uses edge-hosted unikernels for low-bandwidth locality of requests by storing appropriate requests locally and mini- mobile document synchronization in cloud storage services. mizes expected latency by dynamically adjusting its cache size. Traditionally, face recognition includes face identification and reso- 4.13.3. Data analytics lution, and requires performing computation-intensive tasks in the cloud Another study to make use of edge computing is the work in and transmitting raw facial images to the cloud, which is bandwidth in- [354] that introduces a Wi-Fi-based in-bus monitoring and tracking sys- tensive. The authors in [342] observe that migrating part of the reso- tem that observes mobile devices and provides analytics about people lution tasks to fog nodes and transmitting only the feature value to the

310 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 cloud can significantly reduce network traffic. To this end, they propose ParaDrop is available as an open source project 11 , along with the doc- fog-based face identification and face resolution frameworks. uments and tutorials. ParaDrop has three components: ParaDrop access points, the ParaDrop controller, and the ParaDrop API. Using ParaDrop 4.13.7. Artificial intelligence and machine learning API, cloud services could be dynamically deployed on access points us- [345] presents edge stochastic gradient descent (EdgeSGD), a decen- ing the ParaDrop controller, which is a back-end controller that devel- tralized SGD algorithm for solving linear regression problem with the opers interact with to develop their desired services. objective of estimating the feature vector on the edge node. EdgeSGD is Analogous to offloading computation and task to either cloud or fog, used to predict subsurface seismic anomaly via real-time imaging. The a group of researchers from Georgia Institute of Technology argue the edge nodes form a mesh network, and the algorithm obtains the im- utility of “onloading ”cloud services to the edge of the network to ad- age by collaboratively optimizing the objective function over the edge dress the bandwidth and latency challenges of IoT networks [66] . They network. The proposed algorithm is entirely decentralized and does not view the cyber-foraging (e.g., code/task offloading) research domain as require synchronization. Comparably, an edge-assisted adaptive deep client-based methodology, while their proposed approach is “backend- learning framework for mobile object recognition is introduced in [350] . based ” cyber foraging or onloading. They define onloading specific services (e.g., caching, or aggregating traffic) near end-users, with a goal of minimizing user-perceived latency. They propose AirBox, a software 4.14. Software and tools platform based on containers for fast, scalable and secure onloading of edge services. In this subsection, we describe the software and tools that are devel- Bruneo et al. designed Stack4Things 12 , a framework based on Open- oped for fog computing. Stack IaaS middleware that adopts a cloud-oriented model for IoT resource provisioning [381] . Their framework allows injected code at 4.14.1. Simulation and emulation runtime through the cloud, which they define as “contextualization. ” Gupta et al. focused on implementing resource management tech- Similarly, the authors in [150] revise OpenStack Nova service for com- niques in fog computing to measure latency and throughput, and patibility with fog/edge systems, by leveraging a distributed key/value implemented iFogSim 5, a Java-based tool for simulation of fog networks store instead of the centralized SQL backend 13. [380] . iFogSim relies on CloudSim, a cloud simulation tool that enables One commercial edge computing platform for IoT gateways is modeling and simulation of cloud systems and application provisioning the Everyware Software Framework recently developed by Eurotech 14. environments. iFogSim supports cloud-only placement and edge-ward Similarly, EdgeX Foundry 15 is a vendor-neutral open source project placement to demonstrate the scalability of fog-based applications. An building a common open framework for edge computing. UC Berke- extension 6 of iFogSim to support mobility through migration of VMs be- ley researchers also implemented a generic and platform-agnostic open tween cloudlets is implemented in [385] . Another extension is proposed source edge computing framework 16 called Open Carrier Interface (OCI) in [390] to simulate scenarios with strategies aiming to optimize data [393] . placement 7 . Similarly, the authors in [388] propose another edge computing sim- 4.14.3. Edge-based data analytics ulation environment, EdgeCloudSim 8 , that considers both network and Xu et al. proposed edge analytics as a service (EAaaS) to promote computational resources and covers all aspects of edge computing simu- a lightweight, scalable, and low-latency service model for IoT devices lation modeling, including network and computational modeling. Simi- [382] . The primary motivations for EAaaS are the lack of desired real- lar to iFogSim, EdgeCloudSim relies on CloudSim as well. Additionally, time responsiveness, a pricy pay-as-you-go model, and data privacy EdgeCloudSim provides a modular architecture to provide support for concerns associated with cloud-centered IoT analytic services. EAaaS a variety of critical functionality and supports simulating multi-tier sce- provides RESTful interfaces for outside applications, an edge analytics narios where multiple edge servers are running in coordination with agent on the gateway side, and an edge analytics SDK to allow users to upper layer cloud solutions. develop methods for integrating with devices. EAaaS is provided as a Nevertheless, compared to simulation, emulation supports both re- part of IBM Watson IoT platform on IBM Bluemix Cloud. In their work, peatable and controllable experiments with real applications. The au- the authors plan to provide software upgrade capabilities as part of the thors in [392] propose their software implementation of EmuFog 9 , an existing RESTful services and utilize machine learning for existing an- extensible emulation framework for fog computing. EmuFog enables the alytic models. Comparably, SpanEdge 17 provides a programming envi- user to design network topologies, embed fog nodes in the topology, and ronment that allows programmers to specify parts of their applications run Docker-based applications on those nodes connected by an emulated that need to be close to the data source, without knowledge of the num- network. The modules of EmuFog are easily extensible, although Emu- ber of data sources and their geographical distributions [355] . Fog provides a default implementation for them [392] . Another emulation environment for fog computing is FogBed 10 that is based on Mininet 4.14.4. Distributed deep learning and Docker [395] . Harvard University researchers recently proposed a framework and its software implementation 18 for distributed deep neural networks

4.14.2. Edge computing middleware (DDNN), that can span over cloud, fog, and end devices [344] . The

One of the software implementations for fog is developed by the framework map sections of a deep neural network (DNN) onto a dis- researchers of the University of Wisconsin, named ParaDrop [383]. tributed computing graph. While the resulting DDNN allows for deep

ParaDrop is an edge computing platform that runs on WiFi access inferences in the cloud, it accommodates fast and localized inference points to enable edge computing at the extreme edge of the network. via some shallow layers of the DNN at the edge and end devices. DDNN Developers can use this edge computing platform to deploy services, which should be based on Docker containers, on the WiFi access points.

11 Available at https://paradrop.org .

12 Available at http://stack4things.unime.it .

5 Available at https://github.com/cloudslab/ifogsim . 13 More information and code available at http://beyondtheclouds.github.io .

6 Available at http://www.lrc.ic.unicamp.br/fogcomputing/ . 14 Available at https://esf.eurotech.com .

7 Available at https://github.com/medislam/iFogSimWithDataPlacement . 15 Available at https://www.edgexfoundry.org .

8 Available at https://github.com/CagataySonmez/EdgeCloudSim . 16 Available at https://github.com/marckoerner/oci .

9 Available at https://github.com/emufog/emufog . 17 Available at https://github.com/Telolets/StormOnEdge .

10 Available at https://github.com/fogbed/fogbed . 18 Available at https://github.com/kunglab/ddnn .

311 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 inherently enhances sensor fusion, data privacy, and fault tolerance for study their applicability for IoT networks. To do so, they defined three DNN applications. According to this study, DDNN can reduce the com- requirements that an effective container orchestration solution for such munication overhead by a factor of 20 × , compared with the traditional an environment must meet. First, adding/removing a new fog node to a method of sending raw sensor data to the cloud. cluster must be seamless, requiring a minimal software package installation on that node. Second, scheduling an application to a specific fog 4.14.5. Trust establishment node must be possible. Third, all available hardware resources on a fog A traditional solution for establishing trust between two entities is node must be accessible by the container on top of it. Therefore, they to create and share credentials in advance, and then to use a third-party proposed a new container orchestration framework based on Docker to validate the credentials of the nodes. Nevertheless, the characteris- Swarm for fog computing environment that can meet all of the require- tics of some environments (e.g., environments with intermittent or no ments above. Internet connectivity), do not consistently provide access to third-party Similarly, The paper [404] presents an evaluation of Docker as a con- authority. A team of researchers from CMU designed and implemented tainer to host applications at the edge for enabling edge computing. The a system for establishing a trusted identity solution based on cloudlets evaluation is based on four fundamental requirements for edge comput-

[389] . The authors discuss a software implementation 19 of an in-the- ing, namely, deployment and termination, resource and service manage- field solution for establishing trusted identities in disconnected edge ment, fault tolerance, and caching. To evaluate and examine Docker as environments. a candidate for edge computing, a testbed with a data center and three edge sites is simulated, and the four requirements of edge are evaluated 4.15. Testbeds and experiments in this testbed. Authors in [443] study how lightweight virtualization technologies, such as containers and unikernels, can be integrated with Several studies in the fog computing research area implement edge architectures and be suitable for IoT pervasive environments. They testbeds or conduct experiments to verify the concepts and ideas ex- present three IoT use cases, in which lightweight virtualization solutions perimentally. In this subsection, we look at such studies. can bring benefits and desirable design flexibility.

4.15.1. Where is the edge? 4.15.4. Experiments with high computation applications One might ask “Where is the edge of the network? ”Edge comput- Interactive wearable cognitive assistance application is often consid- ing or MEC as general concepts do not restrict the specific location of ered a killer app for edge computing and cloudlets [410] . In an empirical the edge compute and storage nodes, and most researchers assume they study on latency [410] , the performance of several such applications is are “at the edge of the network ”or “in the users’ proximity. ”A recent evaluated. The authors in [406] proposed a new serverless architecture measurement study by Bell Labs researchers [409] shed light on actual for MEC environments, in which mobile app computation is offloaded values of delay and provided a realistic picture of LTE deployments for among edge nodes to reach high throughput while keeping the latency edge computing. They found that the first hop (UE to base station) im- low. To evaluate their architecture, they studied a mobile augmented poses latency of 10–12 ms and adds a sawtooth pattern with an ampli- reality application, in which captured frames from camera must be an- tude of about 40 ms. They also found that in some cases the latency of alyzed to detect the point of interest of objects. The authors in their the first aggregation stage dominates the end-to-end latency. recent study [413] investigate the benefits game developers may obtain by exploring emerging edge cloud technology, specifically use of 4.15.2. Experiments with single-board computers as enablers of fog Google’s Edge Network. They demonstrate how massively multiplayer computing online games benefit from this new system through simulations. Another In [401] , the authors proposed a container-based architecture for recent experimental study is the work in [416] where authors compare edge-based PaaS, in which applications are orchestrated between nodes the performance of machine learning packages on the edges, including at the edge. To show that the proposed architecture can meet the re- TensorFlow, Caffe2, MXNet, PyTorch, and TensorFlow Lite. quirements of edge computing, such as cost-efficiency, and low power consumption, the researchers implemented their solution on a cluster 4.16. Hardware and protocol stack of Raspberry Pi (RPi) devices, which are resource-limited devices. Re- searchers in [408] have looked at the problem of bringing cloud services In this subsection, we summarize the literature that introduced spe- to rural and remote areas in developing countries, where building large cific hardware for implementation of fog computing, cloudlet, F-RAN, and expensive data centers may not be feasible. They propose a hard- etc., or that proposed a protocol stack for fog. ware platform based on a cluster of single-board computer (e.g., RPi) for making cloudlet nodes in rural and remote areas that offer cloud 4.16.1. Hardware services. Intel recently has released a documentation for Fog Reference Unit,

Elkhatib et al. [398] propose the use of micro-clouds, which are small a reference design in a self-contained enclosed chassis for testing and deployable computational infrastructures, to deploy fog networks. A de- demonstration of fog use cases [444] . Intel’s Fog Reference Unit can be vice as small as an RPi can be used to create a micro-cloud. The authors seen as a generic fog node. The authors of [212] propose to use optical note that, unlike mini data centers, micro-clouds are portable, easy to fiber networks with MEC and present a generic fiber-wireless (FiWi) set up, low cost, and can be deployed in rougher environments. They run architecture for IoT. The network architecture consists of two parts: the tests to compare the traditional cloud architecture to the micro-clouds backbone network for connecting to centralized cloud servers, and the and concluded that micro-clouds are more suitable for scenarios where FiWi network for providing MEC services for IoT devices. The authors the cloud server is far away, suffering from high latency. Besides, the in [214] propose the use of time division multiplexed passive optical boot time of micro-clouds is significantly faster than cloud. New RPi networks (TDM-PON) and optical network units (ONUs) for designing a models are capable of booting up 40% faster than an Amazon EC2. cloudlet-based network. The architecture for fog computing in [128] is based on the general-purpose processor (GPP) platform, which allows 4.15.3. Experiments with lightweight virtualization technologies as enablers the architecture for processing general and shared resources. of fog computing In [434] , the authors focus on the requirements of an industrial IoT In [403] , the authors evaluated three container orchestration tools, (IIoT) gateway with respect to heterogeneous network communication, namely Google Kubernetes, Docker Swarm and Apache Marathon, to management, big data, and other services. They describe the advantages of using a multi-Microcontroller architecture. Their architecture incor-

19 Available at https://github.com/SEI-TTG/pycloud/wiki . porates a high-speed parallel bridge controller using a reconﬁgurable

312 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

field programmable gate array (FPGA) to overcome the serial commu- the entropy of the user movement between edge clouds is sufficiently nication bottlenecks in a traditional MCU architecture. The proposed high. architecture of the IIoT multi-Microcontroller gateway consists of three To protect the privacy of users using applications with location-based major modules: a master controller for the cloud, a high-speed bridge services, several private proximity detection algorithms exist. However, controller for data and instruction exchange, and slave controllers for these algorithms fall short in the vicinity range and come with high com- IoT management and database operations. The high-speed bridge con- munication and computation costs. Huo et al. [339] presents a secure troller module is the core of the IIoT gateway and is responsible for homomorphic protocol for private proximity detection in applications packaging tasks and controlling communication with the other modules. of location-based services that addresses the drawbacks of existing ap- The paper [266] proposed a Cloud-Fog Radio Access Network proaches. To achieve privacy in the data transmission process, the au- (CF-RAN) by jointly considering the fog computing paradigm over a thors propose to use a homomorphic encryption scheme that provides Time and Wavelength-Division Multiplexing Passive Optical Networks fast encryption and decryption of data. (TWDM-PON). The proposed CF-RAN can place services provided by the cloud onto the fog nodes by adopting NFV to process the baseband 4.17.2. Data privacy signals sent from the RRHs. Similarly, [146] focuses on interference mit- The authors in [421] propose a privacy-preserving protocol for ve- igation, resource optimization, and mobility management in F-RAN. The hicular road surface condition monitoring. They propose a certificate- authors first present the system architecture that illustrates how the var- less signcryption scheme and a data transmission protocol for road ious components in F-RAN, such as macro RRHs (MRRHs), small RRHs surface condition monitoring that provides confidentiality, mutual au-

(SRRHs), fog computing access points (F-APs), and smart user equip- thenticity, integrity, privacy, and anonymity. In [425] , the authors pro- ment (F-UE), work together for the successful implementation of F-RAN. posed a new privacy-preserving scheme based on diﬀerential privacy

The MRRHs, SRRHs and the F-APs connect to the BBU pool which sup- technology for fog computing. The primary goal in this scheme is to ports resource optimization and provides centralized storage and com- prevent colluded nodes in a fog network from learning the data shared munications in F-RAN. by the IoT devices, thus preserving their privacy. The underlying idea is to introduce artiﬁcial noise to the data before outsourcing, such that

4.16.2. Protocol stack the colluded node cannot infer the original data by conducting statistical

The IoT hub proposed in [171] introduces a protocol stack for an analysis.

IoT gateway (e.g., fog node) that interacts with the smart object devices A lightweight privacy-preserving data aggregation scheme for IoT using a variety of network protocols such as IEEE 802.15.4, IEEE 802.11, networks is proposed in [424] . The scheme enables a control center to or Bluetooth Low-Energy. The job of the application layer includes ser- compute the average and variance of data collected by various types of vice and resource discovery, maintaining a resource directory, acting as IoT sensors while preserving their privacy; fog nodes and control cen- an origin server, and providing a CoAP-to-CoAP/HTTP-to-CoAP Proxy ters cannot learn the original data collected by devices. In this scheme, and cache. The network and physical/link layer both work together to IoT devices are authenticated by the fog nodes. Also, the scheme is fault provide border router functionality, allowing the IoT Hub to act as a tolerant in the sense that if some IoT devices malfunction, the control gateway/bridge between multiple constrained networks. center still can calculate the correct average and variance of the remain-

Another protocol stack for fog is introduced as FogOS [200]. The ing devices without breaching their privacy. Similarly, for information authors view the entire IoT ecosystem as a computer and utilize operat- sharing, a lightweight privacy-preserving fog-assisted information sharing system concepts to create FogOS to manage this abstract computer. ing scheme for health data collected by medical IoT devices is proposed

FogOS is composed of four main layers: service abstraction layer, appli- in [341] . cation manager layer, resource manager layer, and device abstraction layer. The role of the service and device abstraction component is to provide a service APIs and device data model, respectively. Resource 4.17.3. Intrusion detection management then pools or slices resources of the fog and devices as Although utilizing SDN switches as fog nodes seems promising (as needed. The application manager manages IoT applications, ﬁnds the discussed in Section 4.10), it can increase the security risks in SDN net- proper edge resource for a service request, and resources of currently works. The attacker who compromises a fog node can also attempt to running applications. take advantage of the SDN switch to control the network. In [282], the authors described a man-in-the-middle attack on SDN networks relying

4.17. Security and privacy only on the TLS protocol to secure their control channels between SDN switches and controllers. They also proposed a lightweight countermea-

In this subsection, we discuss the studies that consider and address sure utilizing Bloom Filters to detect such an attempt. Fog can be used security and privacy issues in fog computing or utilize fog computing to improve the current security system. In [426], the authors propose a to improve current security systems and protocols. In [99] , the authors new fog-based intrusion system. discussed various privacy and security challenges in fog computing and surveyed the existing literature addressing these problems. Alrawis et al. 4.17.4. Secure protocols and secure data transfer [422] explain security concerns with current IoT environments. They A recent study in [427] investigates secure data transmission be- proposed a scheme that utilizes fog computing to address the security tween IoT nodes and fog nodes. The study analytically models a threat issues faced in IoT environments, specifically the distribution of certifi- model and some possible attacks; it then provides the necessary proofs cate revocation information. Security and privacy issues of vehicular to show that the framework is secure against the discussed attacks. The crowdsensing (VFCS) are discussed in [102] . authors of [420] propose constructing leakage-resilient functional encryption schemes for the fog that provide privacy, fine-grained access 4.17.1. Location privacy control, and security against side channel attacks. Ting et al. [428] proposed several strategies that can be utilized to Identification and resolution of human subjects are crucial in cyber- prevent cyber eavesdroppers from tracking a user’s location by observ- physical systems where humans are involved. Hu et al. [337] proposed ing the user’s service migrations across edge clouds. The underlying idea a new face identification and resolution framework that utilizes fog is to use chaff services, or fake services, in addition to the real services computing to address bandwidth issues that arise in conventional cloud- and move them between the edge clouds to confuse eavesdroppers. The based schemes. In [337] , they provided the details of the security proto- authors showed that by carefully moving the chaff services in conjunc- cols that they devised to address the security and privacy concerns that tion with the real ones, tracking accuracy could be close to zero when arise in their system, such as confidentiality and integrity.

313 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Fig. 10. A bar chart showing the number of research articles in this survey paper under each category of our proposed taxonomy.

5.0.1. Fog system SLA Service level agreements (SLAs) are not currently defined for fog systems. Current SLAs that are used for fog systems are defined for cloud services (e.g., 99.99% availability guarantee for cloud services) or network infrastructure. Moreover, a fog system may have multiple providers/operators and span across multiple operating domains. A potential research direction is defining new and compatible SLA for fog systems (e.g., guaranteeing latency and bandwidth). Additionally, designing SLA management techniques and framework for fog computing that supports for multi-vendor or provider is another potential direction.

5.0.2. Multi-objective fog system design Most of the existing schemes that are proposed for fog systems, such as oﬄoading, load balancing, or service provisioning, only consider few objectives (e.g., QoS, cost) and assume other objectives do not aﬀect the problem (e.g., [107,239,257,295,305] ). A new research direction will be

Fig. 11. A pie chart showing number of research articles in this survey paper to design schemes that consider many objectives (e.g., QoS, bandwidth, that address/support a particular objective/feature. energy, cost) simultaneously. For instance, developing an eﬃcient task oﬄoading scheme that considers bandwidth, waiting time, availability, security, and energy at the same time is a promising direction.

5. Challenges and future research directions 5.0.3. Bandwidth-aware fog system design Few studies consider bandwidth savings through the use of fog com- In this section, we discuss the current challenges and limitations of puting (e.g., [51] ), even though one of the promising features of fog the research in the fog computing area, and we provide future direc- computing is to reduce bandwidth usage in the core of the Internet. tions and potential starting points for those challenges. The summary of There is a need for more research on bandwidth savings through the the challenges and future research directions are included in Table 6 . use of fog computing. These studies could be measurement studies that Fig. 10 shows the number of research articles in this survey that fall capture the actual bandwidth usage in the presence of fog computing. under each category of our taxonomy. Fig. 11 illustrates the number of research articles in this survey paper that address/support a particular 5.0.4. Scalable design of fog schemes objective/feature. The number of research articles may be an indicator Many of the existing schemes and algorithms for fog do not scale of potential future research directions. to the magnitude of IoT networks, since the authors neglect scalability

314 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Table 6 Summary of challenges and future research directions.

Related Features or Challenge Current Limitation Research direction or potential solution Objectives Related Categories

Fog system SLA SLAs are not defined for fog systems. • Define new and compatible SLA for fog QoS, Cost Architectures and Frameworks for Current SLAs are defined for cloud systems. Fog; Control and Monitoring services or network infrastructure. • Design SLA management techniques and framework for fog computing. • Support for multi-vendor or provider SLA for fog systems

Multi-objective fog Many schemes (e.g., offloading, load • Propose schemes that consider multiple QoS, Cost, Energy, Resource Analysis and Estimation; system design balancing) consider few objectives and objectives (e.g., latency, bandwidth, energy) Bandwidth Scheduling, offloading, and Load ignore other objectives. simultaneously (e.g., an efficient task Balancing; Testbeds and Experiments offloading scheme that considers bandwidth, waiting time, availability, security, and energy).

Bandwidth-aware Few works consider bandwidth savings • Need more studies on bandwidth savings Bandwidth Testbeds and Experiments; fog system design through the use of fog computing, even through the use of fog computing. Scheduling, oﬄoading, and Load though one of the promising features of • Perform measurement studies to capture the Balancing; Control and Monitoring; fog computing is to reduce bandwidth actual bandwidth usage in the presence of fog. Resource Analysis and Estimation; usage of the core. Infrastructure Design

Scalable design of Many of the existing schemes and • Design scalable algorithms and schemes for Scalability Service Provisioning; Placement; fog schemes algorithms for fog do not scale to the fog systems, e.g., online task oﬄoading Scheduling, oﬄoading, and Load magnitude of IoT networks. scheme that does not consider individual IoT Balancing; Applications nodes for decision making • Verify scalability of the algorithm and schemes by actual implementation.

Mobile fog Most of the existing literature assumes fog • Propose mobile fog computing, where fog Mobility, Resource Discovery; Concepts and computing nodes are fixed, or focus on the mobility nodes can move. Management Frameworks using Fog; Programming of IoT devices. If fog nodes are mobile, • Scheme for management or federation of Models and Data Modeling; Service resource availability, offloading, and mobile fog nodes. Provisioning; Security and Privacy; resources provisioning will be more • Provisioning method for mobile fog services Scheduling, offloading, and Load challenging. to keep the service always-available for IoT Balancing nodes. • Design of mobility-aware task offloading and scheduling schemes when fog nodes are mobile.

Fog resource Few studies address monitoring of fog • Multi-operator fog resource monitoring Management, Control and Monitoring; Software and monitoring resources. Monitoring is more challenging techniques. programmability Tools if multiple operators use a fog node. • SDN-based monitoring software for resource monitoring and resource advertisement.

Green fog computing Improving the overall energy • Use of energy harvesters and battery Energy Infrastructure Design; Resource consumption of fog has not been well storages for IoT devices and sensors. Analysis and Estimation studied (literature considered • Energy-aware fog node placement, e.g., close energy-aware computation oﬄoading, to renewable energy resources (solar, wind, or energy-aware mobility management, and vibration) federation of IoT devices to improve energy consumption).

Support of Current communication protocols do not • Develop fast or stateless handshake protocols Mobility, RAS Architectures and Frameworks for high-speed users support high-speed users. for high-speed users, e.g., users in vehicles or Fog; Service Provisioning; Resource for automotive communication. Discovery • Develop machine-learning-based mobility prediction algorithms.

Fog node security Fog nodes normally are located close to • Design of physically secure fog nodes against Security, Security and Privacy; Infrastructure users, e.g., at the base stations, routers, or site attacks Heterogeneity Design; Hardware and Protocol Stack even at extreme network edge such as • Design secure hardware, safe against WiFi access points. This makes their physical damage, jamming, etc. security challenging. • Design strong access-control policies for fog nodes.

SDN support for fog SDN does not have native support for fog • Enhancing and standardizing SDN (e.g., Foundation, Deﬁnition and Standards; Software computing. OpenFlow northbound, southbound, east-west programmability and Tools bound interface) for fog use cases.

Fog node site Few studies address the fog node site • Developing fog node site selection strategies Cost, RAS, QoS, Infrastructure Design; Resource selection selection problem, which is a design that considers communication, storage, and Energy Analysis and Estimation problem for ﬁnding appropriate locations computing (a communication hotspot may not for deploying for nodes. be a storage or computing hotspot). • Considering cost in fog node site selection strategies (e.g., deploying fog nodes in Manhattan may be a good decision concerning latency and bandwidth, but may not be a good decision with respect to rental costs). ( continued on next page )

315 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Table 6 ( continued )

Related Features or Challenge Current Limitation Research direction or potential solution Objectives Related Categories

Resilient fog system Current fog networks do not consider • Fault detection, fault prevention, and fault RAS, Security Control and Monitoring; design failure or fault in the network. Also, recovery in fog-based networks Infrastructure Design; Service denial of service (DoS) attacks are more • DoS-resilient fog system design Provisioning; Security and Privacy possible on fog nodes, since they are more • Design a coordinated protection mechanism resource-constrained than cloud servers. that considers fog and cloud to guarantee availability.

Fog federation There is no fog federation framework or • Design new schemes for the federation of fog Management, Placement; Software and Tools; software similar to that of hybrid cloud nodes, across different operating domains. programmability Resource Discovery. Service federation schemes. • Design resource sharing models for fog Provisioning nodes from different vendors/operators • Define new pricing models for federated fog resource sharing schemes

P2P fog computing Current fog computing resource models • Design a complete P2P resource framework Foundation, Concepts and Frameworks using Fog; assume simple client-server model. for fog computing that handles heterogeneity Mobility, Resource Discovery; Service and mobility of fog nodes. Heterogeneity Provisioning; Security and Privacy • P2P fog computing is beneﬁcial when there is no connectivity to the cloud (e.g. in disasters such as ﬂooding, earthquake, etc.) • P2P fog resource pricing, where blockchain could be used for recording fog resource transactions.

Trust and Heterogeneity of fog nodes and IoT nodes • Design new authentication and trust Heterogeneity, Deﬁnition and Standards; Security authentication in makes the conventional trust and mechanisms that could cope with Security and Privacy; Hardware and Protocol heterogeneous fog authentication protocols unsuitable for heterogeneity of fog nodes and IoT nodes. Stack systems fog systems. • Design authentication protocols for fog nodes of diﬀerent vendors/operators.

Secure fog offloading Offloading tasks to fog nodes might incur • Design secure and private offloading and Security, QoS Scheduling, offloading, and Load some security and privacy risks. load balancing schemes. Balancing; Security and Privacy • A mechanism for receivers to verify the correctness and integrity of the offloaded task.

PaaS for fog Lack of a PaaS for fog systems, where • Developing a PaaS for fog computing, which programmability, Software and Tools; Service computing developers can easily develop software is transparent to users and supports diﬀerent Management Provisioning; Programming Models across fog, IoT, and cloud. communication- and application-level and Data Modeling protocols and APIs. • Developing plugins for PaaS for diﬀerent fog computing applications

Standardizing fog Many independent definitions for fog (and • Unanimous and universally-agreed on the Foundation Definition and Standards computing fog-related computing paradigms) are definition of fog computing. being proposed.

Hardware Most studies do not use new available • Use of new hardware and communication Scalability Hardware and Protocol Stack technologies for fog hardware or communication technologies. technologies, such as non-volatile storage technologies, optical networks, and FPGAs.

in their fundamental design. We believe scalability is critical in design- exist new provisioning methods for mobile fog services, such that fog ing fog systems; fog systems should be scalable so that they could be services are available for IoT nodes and users. Similarly, one could de- implemented in IoT networks. For instance, a scalable algorithm for fog sign a task offloading and scheduling scheme when fog nodes are mobile. offloading is an online offloading scheme that does not need information An early effort for mobile fog nodes can be found in [160] . of individual IoT nodes for decision making (e.g., [88,312,313] ). We encourage researchers in the fog computing area to verify the scalability 5.0.6. Fog resource monitoring of their proposed algorithms and schemes (e.g., by an actual implemen- Few studies in the literature propose monitoring schemes for fog re- tation). sources [151] . Monitoring is useful when multiple operators use a fog node, or when a fog node is located in a location where many users

5.0.5. Mobile fog computing use the fog node. A possible direction is developing fog resource moni-

Most of the existing literature assumes that the fog nodes are fixed, toring techniques that support multi-operator access. Use of SDN-based or only considers the mobility of IoT devices (e.g., [146,291] ). Less at- monitoring software for fog resource monitoring and fog resource ad- tention has been paid to mobile fog computing and how the mobile fog vertisement is also a promising approach. nodes can improve the QoS, cost, and energy consumption. When fog nodes are mobile, fog resource availability, resource discovery, task of- 5.0.7. Green fog computing floading, and resources provisioning will be more challenging. Mobile Few studies in the reviewed literature have addressed the energy cri- fog computing, where fog nodes can move and form new networks, is terion in their system design (e.g. [23,196,294,295,333,384] ). Most of an interesting and challenging research direction. Moreover, designing the studies on energy are about energy-aware computation offloading, a scheme for management or federation of mobile fog nodes is another energy-aware mobility management, and federation of IoT devices to possible direction. Along with mobile fog computing, there needs to improve the energy consumption of fog systems. However, improving

316 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 the overall energy consumption of fog has not been well studied. En- tial starting point for access control in fog computing can be found in ergy consumption of a fog network includes three major portions: (1) [104,420] . energy consumption of IoT devices sending data to the fog, (2) energy consumption of the network interconnecting IoT devices and the fog 5.0.11. Fog node site selection nodes, and (3) energy consumption of the fog nodes. Few studies address the fog node site selection problem, which is a To reduce the energy consumption of IoT devices, use of energy har- design problem for finding appropriate locations for deploying nodes. vesters and battery storage for IoT devices and sensors are potential Fog node site selection strategies should consider communication, stor- research directions. Energy harvesters can improve energy consump- age, and computing at the same time for finding an appropriate lo- tions while bringing new challenges to the system, such as uncertainty cation (a communication hotspot may not necessarily be a storage or and unpredictability. In order to reduce the energy consumption of the computing hotspot). Moreover, cost should also be a deciding factor network interconnecting IoT devices and the fog nodes, one of the po- in fog node site selection strategies; deploying fog nodes in Manhattan tential research directions is to identify where to put fog nodes and may be a good decision concerning reduction in latency and bandwidth, how close they should be to the end users. Mobile fog nodes is also but may not be a good decision with respect to rental costs. Further- a compelling use case for energy consumption. To reduce the energy more, fog node security considerations that are discussed in “Fog Node consumption of fog nodes, one potential research direction is to reduce Security ” previously could also affect the fog node site selection de- the distance between fog servers and local renewable energy sources cision. We refer the interested reader to the recent article [445] that (such as solar, wind, or vibration). This problem can be addressed in describes the security requirements and approaches of an open fog different ways: the traffic from IoT devices can be rerouted to the near- architecture. est fog node that is powered by renewable energy. The other way is for telecommunication companies to identify the location of fog nodes that 5.0.12. Resilient fog system design need the high amount of power to serve the traffic, and to encourage From the reliability and availability perspective, fog services and fog people to use their local renewable energy for their local micro-grid to networks bring new challenges to the current network and service provi- power up their local fog nodes. sioning methods. To guarantee the availability and reliability of the fog services, a coordinated service provisioning mechanism that considers

5.0.8. SDN support for fog both fog and cloud computing is needed. For example, if a fog service

SDN software does not have native support for fog computing. needs some functions to process a stream of data, providing extra repli-

SDN is mostly commercially viable inside large data centers or campus cas of those functions can improve the availability of the service. On networks [9] . Enhancing and standardizing SDN software (e.g., Open- the other hand, due to the limited computing resources of the fog nodes

Flow northbound, southbound, east-west bound interface) for fog use compared to the cloud data centers, allocation of the function replicas cases is a direction we believe will ease the development of fog comput- to provide availability and reliability is not a straightforward decision. ing software. Moreover, with multiple vendors/operators in fog systems, As a future direction, availability may be considered in addition to con- there will be a need for new SDN architectures with multiple domains straints, such as latency, throughput, and security when designing pro- and hierarchies of SDN controllers. visioning methods for fog services. Most of the articles in the literature about fog computing do not consider failure or fault in the fog network. Another research direction is

5.0.9. Support of high-speed users to provide diﬀerent protection and restoration mechanism across dif-

Current communication protocols that are proposed for fog com- ferent layers. In addition, failure detection, prevention, and recovery puting environments do not support high-speed users, such as users in are efficient ways to improve the availability of the fog services. Addi- cars, users on trains, and vehicular computing. A research direction is tionally, fog nodes are more prone to denial of service (DoS) attacks, to develop quick or stateless handshake and authentication protocols since they are more resource-constrained than adequately-secure cloud for high-speed users and automotive communication. Note that there data centers; also, recently compromising IoT nodes and embedded sys- have been already some early efforts in this field, such as the articles tems are becoming new sources of distributed DoS attacks [446] . Novel

[100,145,172]; nevertheless, we are still far from having a working classes of proactive defense techniques based on moving target defense and resilient communication protocol for high-speed users and automo- paradigm (sometimes referred to as address mutation/randomization ) tive communication for fog computing. One can use machine-learning- could be used to thwart DoS attacks [447,448] . Researchers of UC Berke- based mobility prediction algorithms in the design of the handshake and ley recently have proposed a resilient edge computing framework [116] , authentication protocols, to predict the location of the high-speed users which is a good starting point in the direction of resilient fog system and analyze their mobility patterns for fog computing. design. In addition to handshake and authentication protocols for high-speed users, fog service provisioning for IoT applications is required to be dy- 5.0.13. Fog federation namic and proactive due to the rapid changes (such as connectivity, Currently, there is no fog federation framework or software (similar bandwidth fluctuations, or failure) in mobile and high-speed IoT envi- to that of hybrid cloud federation schemes), which controls and feder- ronments. To address dynamic and proactive fog service provisioning, ates fog resources across multiple operating domains. There is a need predicting the behavior and location of IoT devices and high-speed users for new schemes for the federation of fog nodes, especially when they based on historical data or machine learning methods is another poten- belong to different operating domains. The federation scheme should tial solution that requires further investigation. account for resource sharing models for fog nodes from different vendors/operators. Similarly, one can define new pricing models for feder- 5.0.10. Fog node security ated fog resources. Finally, one can propose policies for new fog resource Fog nodes are going to be placed near users, in locations such as on sharing schemes (e.g., P2P fog computing resource sharing model) un- the base stations, or routers, or even at the extreme network edge on der the federation framework. The recent article in [228] might be a WiFi access points. This makes it challenging to provide security for fog good start for potential research about fog federation. nodes. Site attacks are more possible on the fog nodes than cloud data centers. A research direction may be designing secure fog node sites, safe 5.0.14. P2P fog computing against physical damage, jamming, etc. Also, another direction may be Current fog computing resource models assume a simple client- to designing strong access-control policies for fog nodes, so that they server model, where IoT devices (clients) use cloud computing or fog are secure in the presence of malicious users in the vicinity. A poten- computing (server) to process their requests, while fog nodes may also

317 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330 offload computation among themselves or to the cloud. We argue that services proactively and automatically with minimal effort from devel- design of a complete peer-to-peer (P2P) resource framework for fog com- opers, and support different communication- and application-level pro- puting is a promising direction. Under the P2P fog computing model, tocols and APIs. Once this PaaS is available, its modular design could fog nodes share resources in a P2P manner, where individual fog nodes also be taken into account so that various plugins for different fog com- share their resources (e.g., compute or storage), without an intermedi- puting applications and services could be easily integrated with the ary third-party. The P2P fog computing resource model could further PaaS. handle heterogeneity and mobility of fog nodes (e.g., when a fog node leaves the P2P network). P2P fog computing is the most beneficial when 5.0.18. Standardizing fog computing there is no connectivity to the cloud, for instance when there are disas- Different research teams are proposing many independent defini- ters such as flooding or earthquake and the connection to the cloud re- tions of fog (and fog-related computing paradigms, such as edge com- sources is gone. P2P fog resource pricing is another research direction, puting). In Section 3 we saw that the definitions of fog computing and where different pricing and incentive methods could be proposed for its related computing paradigms are not completely standardized. We P2P resource sharing model. Blockchain technology could be a choice believe there is a research gap in the definitions and standards for fog for keeping track of fog resource transactions when designing a P2P fog computing and other fog-related computing paradigms that needs to be resource pricing model. filled by standards and universally-agreed definitions. Once the definitions are agreed upon, researchers become more clear when defining 5.0.15. Trust and authentication in heterogenous fog systems problems, and there will be more agreement among researchers and in- In addition to mobility that is discussed in “Support of High-speed dustry about these paradigms. Organizations such as OpenFog Consor- Users, ” heterogeneity of fog nodes and IoT devices makes the conven- tium and OpenEdge Computing are already developing standards and tional trust and authentication protocols unsuitable for fog systems. definitions for fog computing and edge computing. Also, fog nodes may belong to different vendors and operators. Hence, we need to design new authentication and trust mechanisms for new 5.0.19. Hardware technologies for fog fog systems that could cope with the heterogeneity of fog nodes and IoT Most studies in the area of fog computing or edge computing do not devices. Some researchers have started to address heterogeneity in their make use of new hardware or communication technologies, such as non- fog system design, such as the articles [171,324] . volatile storage technologies, optical networks, fiber-wireless (FiWi), or FPGAs. Use of new hardware and communication technologies for the 5.0.16. Secure fog offloading design of fog networks, (e.g., fog-to-cloud interconnection) is a direction Offloading tasks among fog nodes might incur some security worth exploring. and privacy risks. The risk is when tasks containing security- and privacy-critical information is offloaded. Also, a security risk might be 6. Conclusion when a fog node becomes over-loaded (e.g., by the requests sent by a malicious user) and starts offloading security- and privacy-critical in- The Internet of Things accelerates digital transformation and pro- formation to other fog nodes (which may be accessible to the mali- vides benefits to many industries, including manufacturing, energy, cious user). A research direction, hence, is to design and implement se- transportation, smart cities, education, retail, healthcare, and govern- cure offloading and load balancing schemes. An early effort for privacy- ment. Due to IoT’s fundamental benefits, the number of connected de- aware offloading in MEC is included in [304] . Furthermore, designing vices and the IoT networks is on the rise, as individuals and companies a lightweight and efficient mechanism for IoT receivers to verify the deploy more and more IoT devices. IoT is expected to connect billions correctness and integrity of the offloaded tasks. of devices and humans to bring promising advantages for us. Fog computing is one of the promising solutions for handling the big data that 5.0.17. Paas for fog computing is being produced by the IoT, which is often security-critical and time- There are not any solid implementations of a PaaS for fog systems, sensitive. In this paper, we provided a tutorial on what fog comput- where developers can easily develop software across fog, IoT, and cloud. ing is and how it relates to or differs from other computing paradigms, [132,401] are some of the very few efforts in this direction. Developing such as cloudlets, MEC, and edge computing. Next, we provided a tax- a PaaS for fog computing can ease the general development and ac- onomy of research topics in fog computing and summarized the rele- ceptability of fog computing. The future PaaS for fog computing should vant papers on fog computing and its related computing paradigms. Fi- hide the fog configuration specification (e.g., location of fog nodes, their nally, we provided challenges and future directions for research in fog interconnection, their capacity) for users, provision applications, and computing.

318 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Appendix

Table 7 (Appendix 1) Features and objectives of the cited papers in this survey.

QoS Cost Energy Bandwidth Security Foundation RAS Mobility Scalability Heterogeneity Management Programmability Paper QoS Cost Energy Bandwidth Security Foundation RAS Mobility Scalability Heterogeneity Management Programmability Paper √ √ √ √ √ √ Deng et al. [124] Wen et al. [24] √√ √√ √ √ Zhao et al. [288] Jayaraman et al. [333] √√ √ Vaquero and Rodero-Merino [71] Zao et al. [334] √ √ Dastjerdi and Buyya [449] Li et al. [133] √√ √ √√ √ Streiﬀer et al. [397] Hou et al. [172] √ √√ Garcia Lopez et al. [43] Jalali et al. [23] √√ √ Aazam and Huh [221] Jalali et al. [222] √√ √ √√ Gu et al. [255] Peng et al. [27] √√ √ √ Urgaonkar et al. [231] Bruneo et al. [381] √√ √√√ Guo et al. [289] Prazeres and Serrano [173] √√ √ Souza et al. [256] Jain and Tata [198] √√ √√ Stojmenovic and Wen [82] Xu et al. [382] √√√√ √√ Yannuzzi et al. [97] Yu et al. [261] √√ √√ Truong et al. [279] Lee et al. [296] √√ √√ Gao et al. [83] Chen and Zhang [297] √√ √ √√ Sarkar and Misra [72] Hakiri et al. [280] √√ √ √√ Li et al. [290] Baktir et al. [9] √√√√ √ √ Sarkar et al. [89] Xu et al. [281] √√ √ √√ Jemaa et al. [257] Chang et al. [95] √ √√ √√√ Alippi et al. [125] Liu et al. [383] √√ √ √√√ √√ √ Skarlat et al. [126] Yigitoglu et al. [233] √√√ √√√ Sun and Ansari [106] Hong et al. [199] √√√ √√ √√ Tomovic et al. [127] Yin et al. [298] √√ √√ √√ Skarlat et al. [258] Ananthanarayanan et al. [335] √ √√ Shi et al. [80] Ni et al. [13] √ √ Chiang and Zhang [84] Cheng et al. [336] √ √√ √ Villari et al. [67] Fan and Ansari [210] √√√√ Brogi and Forti [107] Wang and Zhou [299] √ √√ √ OpenFogConsortium [6] Sun et al. [300] √ √√√ Zhang et al. [194] Ti and Le [301] √ √ Li et al. [330] Farris et al. [234] √ √√ Dantu et al. [195] Zhang et al. [302] √√ √√ √ Ku et al. [128] Xiao et al. [211] √√ √√√ √√√ Gupta et al. [380] Abdelwahab et al. [61] √√ √√√ √√ Dsouza et al. [108] Abdelwahab and Hamdaoui [235] √ √ √√√ √ Yi et al. [109] van Lingen et al. [134] √√√ √√√√√ Hong et al. [259] Vilalta et al. [135] √√ √ √√√ Hao et al. [98] Choi et al. [200] √√√ √ Cirani et al. [171] Zhang et al. [136] √√ √√ √ Malensek et al. [331] Kaminski et al. [174] √√√√ √√ √√√ Moreno-Vozmediano et al. [129] Farris et al. [175] √ √√ √√ Sill [73] Borylo et al. [137] √√ √√ Bittencourt et al. [291] Morabito et al. [236] √√ √ √√√ Esposito et al. [419] Amento et al. [138] √√ √√√ √√ √ Liang et al. [130] Ottenwälder et al. [237] √ √√√ √ Kapsalis et al. [131] Giang et al. [201] √ √√√√ Yu et al. [420] Giang et al. [238] √√ √√ Kattepur et al. [292] Zeng et al. [262] √√ √ √ √√√ Kattepur et al. [293] Gupta et al. [239] √ √ √√√ Basudan et al. [421] Yannuzzi et al. [139] √√√ √√ Yangui et al. [132] Elkhatib et al. [398] √√ √ √ Xu et al. [260] Alrawais et al. [422] √√ √ √ Kaur et al. [196] Muñoz et al. [399] √√√√ √√√ Saurez et al. [197] Lertsinsrubtavee et al. [240] √√√ √√ √√ Nan et al. [294] Hong et al. [263] √√ √√ Xiao and Krunz [295] Rodrigues et al. [241] √ √√√√ Byers [96] Tong et al. [92] √√ √√ √ √√ Yang [332] Aral and Ovatman [264] √ √ √√√√ √√ Zhang et al. [232] Tan et al. [303] √ √√√ Klas [78] Sookhak et al. [176] √ √ √√√ Mouradian et al. [8] El Kafhali and Salah [223] ( continued on next page )

319 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Table 7 ( continued )

QoS Cost Energy Bandwidth Security Foundation RAS Mobility Scalability Heterogeneity Management Programmability Paper QoS Cost Energy Bandwidth Security Foundation RAS Mobility Scalability Heterogeneity Management Programmability Paper √ √ √ √ Hu et al. [337] Schäfer et al. [324] √√ √ √√ Lyu et al. [338] Teerapittayanon et al. [344] √√ √√ √ √ Huo et al. [339] Li et al. [111] √√√ √√ √ Al Faruque and Vatanparvar [384] Abdallah et al. [402] √√ √ √ Li et al. [282] Hoque et al. [403] √ √√√ √ Wang et al. [242] He et al. [428] √√ √√ √√ √√ Tao et al. [177] Drolia et al. [271] √√ √ √√√ Nan et al. [140] Gedeon et al. [325] √ √√√ √√√ Azimi et al. [340] Yousefpour et al. [88] √√√ √ √√ He et al. [304] Vallati et al. [215] √√ √√ √√√√ Zhao et al. [265] Farris et al. [112] √√ √√ Chang et al. [305] Kamath et al. [345] √√√ √ √√√√√ Tinini et al. [266] Lebre et al. [150] √√ √ √√ Shen et al. [341] Bellavista et al. [87] √√√ √√√√ Iotti et al. [400] Lopes et al. [385] √ √√ √ Yi et al. [99] Ismail et al. [404] √√√ √√√ Hu et al. [342] Confais et al. [386] √ √√√ Wang et al. [141] Hong et al. [346] √√√ √√ Bao et al. [243] Santoro et al. [405] √√ √ √√√ Yi et al. [306] Li and Nabrzyski [272] √ √ √√√√ √ Ma et al. [244] Khan and Freitag [387] √√√ √√√ Ficco et al. [142] Ascigil et al. [273] √√ √√√√ Huang et al. [178] Abderrahim et al. [151] √ √√ √√ Ahmed et al. [79] Mehta et al. [216] √ √ √√√ √√ √√√√ Markakis et al. [93] Garcia-Perez and Merino [152] √ √ √√√ √ Markakis et al. [94] Dhakal and Ramakrishnan [347] √ √√√ √√√ Li et al. [143] Le Tan et al. [226] √√ √√ √ Yu et al. [144] Sonmez et al. [388] √√ √ √√√√√ Basudan et al. [423] Alonso-Monsalve et al. [180] √√ √√ Nikoloudakis et al. [343] Beraldi et al. [309] √√√ √ √√ √ √ Lu et al. [424] Le et al. [153] √√ √√ Wang et al. [425] Baresi et al. [406] √√ √√√ Guo et al. [212] Lujic et al. [154] √ √ √√√ √√ √ Masip-Bruin et al. [91] Li et al. [181] √√ √√ √ An et al. [426] Mangiante et al. [348] √ √√ √√√ Guevara et al. [74] Tärneberg et al. [155] √√√ √√ √√ Desikan et al. [307] Shekhar et al. [156] √√√√√ √√√√ Shih et al. [213] Jonathan et al. [310] √√ √√√√ √√√√√ Zhang et al. [145] Gosain et al. [407] √√ √√ √√√√√√ Ren et al. [179] Bhardwaj et al. [66] √√√ √√√√ Zhang et al. [146] Renart et al. [202] √√ √√√ √√√ Lorenzo et al. [147] Echeverría et al. [389] √ √√ √√ Chen et al. [434] Ni et al. [349] √√ √√√ √ √ Tang and He [308] Helmer et al. [408] √√ √√ Grewe et al. [100] Morshed et al. [68] √√√ √√ Mohan et al. [267] Prasad et al. [113] √ √√√ √√ √ Cozzolino et al. [245] Li et al. [350] √√ √√ √√ Giri et al. [427] Yoon et al. [351] √√√√ √ √√ √ √√ √ Mondal et al. [214] Zhang et al. [247] √√ √√ √√√√ Zhu and Huang [268] Manzoni et al. [75] √√ √√ √ √ Malandrino et al. [224] Bhardwaj et al. [352] √√√ √√ √√ √ Lyu et al. [90] Jang et al. [353] √√ √√√ Suganuma et al. [148] Naas et al. [157] √√√√ √√ Yuan et al. [149] Qi et al. [354] √ √ √√√√ √√ √ √ Cho et al. [322] Sajjad et al. [355] √ √√√ √√√ Liu et al. [269] Jonathan et al. [76] √√√√ √ √√ √ Kiani and Ansari [110] Lee et al. [356] √√ √ √√ √ √√ √ √√ Sun and Ansari [246] Singh et al. [311] √√ √ Xiang et al. [270] Mei et al. [357] √√ √√ Kamiyama et al. [225] Fricker et al. [312] √ √ Perera et al. [101] Varshney and Simmhan [85] √√ √√√ √ Pahl et al. [401] Hao et al. [51] √√ √ √√ √√√ √ √√ Lin and Shen [323] Chen and Xu [313] ( continued on next page )

320 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Table 7 ( continued )

QoS Cost Energy Bandwidth Security Foundation RAS Mobility Scalability Heterogeneity Management Programmability Paper QoS Cost Energy Bandwidth Security Foundation RAS Mobility Scalability Heterogeneity Management Programmability Paper √ √ √ √ Weinman [77] Rafetseder et al. [391] √√√ √√√√ √ Hou et al. [248] Jeong et al. [205] √√√√ √√ Yang et al. [158] Varadi et al. [114] √√√√ √√ √√ Habak et al. [64] Chen and Hao [286] √√ √ √√ √ Bahreini and Grosu [274] Acharya and Gaur [20] √√√√√ √√√ Ryden et al. [182] Raagaard et al. [164] √ √√ √ Had ž i ćet al. [409] Anglano et al. [228] √√√√√ √ Hagenauer et al. [183] Yannuzzi et al. [115] √√ √√ Ni et al. [102] Nasr et al. [187] √√ √ √ Drolia et al. [358] Moustafa et al. [188] √√ √√ √√ Gia et al. [359] Lera et al. [275] √√ √√√√√ Marín-Tordera et al. [70] Gupta and Ramachandran [206] √ √ √√√√ √√ Chaufournier et al. [249] Mayer et al. [392] √√√√√ √ Mortazavi et al. [184] Pahl et al. [165] √√ √√ Chen et al. [410] Ahn et al. [368] √ √√ √ Chen et al. [360] Harchol et al. [116] √√√√ √√ √√√ Tang et al. [159] Mohan et al. [166] √√ √√ √√ √ √√ Ha et al. [250] Xu et al. [189] √√√ √√ Sha et al. [429] Wang et al. [369] √√√ √√√ Kar et al. [361] Kastanakis et al. [276] √√√√ √√ Yu et al. [314] Zhang et al. [416] √ √√ √ √√ Mahmud et al. [103] Silvestro et al. [167] √ √√ √ Hu et al. [86] Németh et al. [252] √√√ √ √√√ √ Kimovski et al. [450] Körner et al. [393] √√√√ √√√ Naas et al. [390] Zhao et al. [207] √√ √√ √ √√ Lujic et al. [362] Wang et al. [277] √√√√ √√√ Su et al. [203] Hao et al. [318] √√ √√ Buzachis et al. [411] Corneo and Gunningberg [370] √√ √√ √√ Nguyen et al. [227] Wen et al. [190] √ √√√√ Bailas et al. [412] Noghabi et al. [208] √√√√ √√ Machen et al. [251] Bhardwaj et al. [371] √√ √√ Bellavista et al. [185] Ge and Wang [372] √√ √√ √ √√√ He et al. [160] Okada et al. [328] √√ √√√ √ Cheng et al. [204] Giang et al. [394] √√ √√ Zhang et al. [104] Mao et al. [373] √ √√√√ Abbas et al. [81] Nguyen et al. [319] √√ √√ √√ Ali et al. [363] Biookaghazadeh et al. [435] √√√ √√ √ √√ Abeshu and Chilamkurti [364] Garg et al. [168] √√ √ √ √√ √ Ceselli et al. [217] Li et al. [374] √√ √√ √ Amadeo et al. [326] Coutinho et al. [395] √√ √ √ √ √√√√ Cao et al. [365] Mortazavi et al. [253] √√ √√ Shirazi et al. [105] Elias et al. [375] √√ √√√√ Plumb and Stutsman [413] Sathiaseelan and Secker [191] √ √√ Oteafy and Hassanein [161] Zang et al. [117] √√ √√ √√ Muhammad et al. [366] Hung et al. [278] √√√ √√ √ Tanganelli et al. [327] Deyannis et al. [320] √√ √√√ √√ Li et al. [162] Jeong et al. [321] √√√ √ √ √√ Sharma et al. [283] Psaras et al. [118] √√√ √√√ Jia et al. [218] Kim et al. [119] √√√ √√√ √√ √ Cui et al. [54] Yekta and Lu [432] √ √ Shen et al. [430] Cherrueau et al. [120] √√ √√ √ Rausch et al. [284] Venanzi et al. [329] √√ √ √ √√√ √√ Bi et al. [285] Anglano et al. [229] √√√√ √√ √√ √ Yaghmaee and Leon-Garcia [367] Akrivopoulos et al. [417] √√√√√ √√ √ √ Rimal et al. [219] Meiklejohn et al. [169] √√√√ √√√ √√ Puthal et al. [163] Ran et al. [376] √√√ √√ Lyu et al. [315] Li et al. [170] √√ √ √ Bavier et al. [414] Kim et al. [433] √√ √ √√ √√ Lyu et al. [316] Gedeon et al. [220] √ √√ He et al. [431] Talagala et al. [287] √√ √ √√√ Jeong et al. [186] Liu et al. [377] √√ √ √ √√√ √ Li et al. [317] Zeng et al. [230] √√√ √ √√√ Carrega et al. [415] Skarin et al. [418] ( continued on next page )

321 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

Table 7 ( continued )

Supplementary material [23] F. Jalali , K. Hinton , R. Ayre , T. Alpcan , R.S. Tucker , Fog computing may help to save energy in cloud computing, IEEE J. Sel. Areas Commun. 34 (5) (2016) 1728–1739 . [24] Z. Wen , R. Yang , P. Garraghan , T. Lin , J. Xu , M. Rovatsos , Fog orchestration for

Supplementary material associated with this article can be found, in internet of things services, IEEE Internet Comput. 21 (2) (2017) 16–24 . the online version, at doi: 10.1016/j.sysarc.2019.02.009 . [25] S.-C. Hung , H. Hsu , S.-Y. Lien , K.-C. Chen , Architecture harmonization between cloud radio access networks and fog networks, IEEE Access 3 (2015) 3019–3034 . [26] A. Checko , H.L. Christiansen , Y. Yan , L. Scolari , G. Kardaras , M.S. Berger , L. Dittmann , Cloud ran for mobile networks technology overview, IEEE Commun.

References Surv. Tutor. 17 (1) (2015) 405–426. [27] M. Peng , S. Yan , K. Zhang , C. Wang , Fog-computing-based radio access networks: issues and challenges, IEEE Netw. 30 (4) (2016) 46–53 . [1] J. Gantz , D. Reinsel , Extracting value from chaos, IDC iview 1142 (2011) (2011) [28] G.H. Forman , J. Zahorjan , The challenges of mobile computing, Comput. (Long 1–12 . Beach Calif) 27 (4) (1994) 38–47 . [2] A. McAfee , E. Brynjolfsson , T.H. Davenport , D. Patil , D. Barton , Big data: the man- [29] M. Satyanarayanan , Fundamental challenges in mobile computing, in: Proceedings agement revolution, Harv. Bus. Rev. 90 (10) (2012) 60–68 . of the Fifteenth Annual ACM Symposium on Principles of Distributed Computing, [3] D. Evans , The internet of things: how the next evolution of the internet is changing Acm, 1996, pp. 1–7 . everything, CISCO white paper 1 (2011) (2011) 1–11 . [30] H.T. Dinh , C. Lee , D. Niyato , P. Wang , A survey of mobile cloud computing: ar- [4] B. Ravandi , I. Papapanagiotou , A self-learning scheduling in cloud software defined chitecture, applications, and approaches, Wirel. Commun. Mobile Comput. 13 (18) block storage, in: 2017 IEEE 10th International Conference on Cloud Computing (2013) 1587–1611 . (CLOUD), IEEE, 2017, pp. 415–422 . [31] NIST, Mobile cloud computing. [Online]. Available: https://www.nist.gov/ [5] B. Zhang , N. Mor , J. Kolb , D.S. Chan , K. Lutz , E. Allman , J. Wawrzynek , E.A. Lee , programs-projects/mobile-cloud-computing , Technical Report, National Institute J. Kubiatowicz , The cloud is not enough: saving iot from the cloud., HotStorage, of Standards and Technology. 2015 . [32] J. Ren , Y. Zhang , K. Zhang , X. Shen , Exploiting mobile crowdsourcing for perva- [6] OpenFogConsortium, Openfog reference architecture for fog computing, 2017. sive cloud services: challenges and solutions, IEEE Commun. Mag. 53 (3) (2015) [Online]. Available: https://www.openfogconsortium.org/ra/ , February 2017. 98–105 . [7] F. Bonomi , R. Milito , J. Zhu , S. Addepalli , Fog computing and its role in the internet [33] Z. Sanaei , S. Abolfazli , A. Gani , R. Buyya , Heterogeneity in mobile cloud computing: of things, in: Proceedings of the First Edition of the MCC Workshop on Mobile Cloud taxonomy and open challenges, IEEE Commun. Surv. Tutor. 16 (1) (2014) 369–392 . Computing, ACM, 2012, pp. 13–16 . [34] M. Shiraz , A. Gani , R.H. Khokhar , R. Buyya , A review on distributed application [8] C. Mouradian , D. Naboulsi , S. Yangui , R.H. Glitho , M.J. Morrow , P.A. Polakos , A processing frameworks in smart mobile devices for mobile cloud computing, IEEE comprehensive survey on fog computing: state-of-the-art and research challenges, Commun. Surv. Tutor. 15 (3) (2013) 1294–1313 . IEEE Commun. Surv. Tutor. 20 (1) (2018) 416–464 . [35] G. Grassi , K. Jamieson , P. Bahl , G. Pau , Parkmaster: an in-vehicle, edge-based video [9] A.C. Baktir , A. Ozgovde , C. Ersoy , How can edge computing benefit from soft- analytics service for detecting open parking spaces in urban environments, in: Pro- ware-defined networking: a survey, use cases, and future directions, IEEE Commun. ceedings of the Second ACM/IEEE Symposium on Edge Computing, ACM, 2017, Surv. Tutor. 19 (4) (2017) 2359–2391 . p. 16 . [10] M. Mukherjee , L. Shu , D. Wang , Survey of fog computing: fundamental, network [36] C. Liu , Y. Cao , Y. Luo , G. Chen , V. Vokkarane , M. Yunsheng , S. Chen , P. Hou , applications, and research challenges, IEEE Commun. Surv. Tutor. 20 (3) (2018) A new deep learning-based food recognition system for dietary assessment on an 1826–1857 . edge computing service infrastructure, IEEE Trans. Serv. Comput. 11 (2) (2018) [11] C. Perera , Y. Qin , J.C. Estrella , S. Reiff-Marganiec , A.V. Vasilakos , Fog computing 249–261 . for sustainable smart cities: a survey, ACM Comput. Surv. (CSUR) 50 (3) (2017) [37] J.-P. Hubaux , T. Gross , J.-Y. Le Boudec , M. Vetterli , Toward self-organized mo- 32:1–32:43 . bile ad hoc networks: the terminodes project, IEEE Commun. Mag. 39 (1) (2001) [12] C. Li , Y. Xue , J. Wang , W. Zhang , T. Li , Edge-oriented computing paradigms: a 118–124 . survey on architecture design and system management, ACM Comput. Surv. (CSUR) [38] U. Drolia , R. Martins , J. Tan , A. Chheda , M. Sanghavi , R. Gandhi , P. Narasimhan , 51 (2) (2018) 39:1–39:34 . The case for mobile edge-clouds, in: Ubiquitous Intelligence and Computing, 2013 [13] J. Ni , K. Zhang , X. Lin , X. Shen , Securing fog computing for internet of things IEEE 10th International Conference on and 10th International Conference on Au- applications: challenges and solutions, IEEE Commun. Surv. Tutor. 20 (1) (2018) tonomic and Trusted Computing (UIC/ATC), IEEE, 2013, pp. 209–215 . 601–628 . [39] G. Huerta-Canepa , D. Lee , A virtual cloud computing provider for mobile devices, [14] T. Taleb , K. Samdanis , B. Mada , H. Flinck , S. Dutta , D. Sabella , On multi-access in: Proceedings of the 1st ACM Workshop on Mobile Cloud Computing & Services: edge computing: a survey of the emerging 5g network edge cloud architecture and Social Networks and Beyond, ACM, 2010, p. 6 . orchestration, IEEE Commun. Surv. Tutor. 19 (3) (2017) 1657–1681 . [40] Y.-C. Tseng , S.-Y. Ni , Y.-S. Chen , J.-P. Sheu , The broadcast storm problem in a [15] Y. Mao , C. You , J. Zhang , K. Huang , K.B. Letaief , A survey on mobile edge com- mobile ad hoc network, Wirel. Netw. 8 (2–3) (2002) 153–167 . puting: the communication perspective, IEEE Commun. Surv. Tutor. 19 (4) (2017) [41] OpenEdgeConsortium, About - the who, what, and how. [Online]. Available: 2322–2358 . http://openedgecomputing.org/about.html , Technical Report, OpenEdge Comput- [16] P. Mell, T. Grance, et al., The nist definition of cloud computing, 2011. ing. [17] L.M. Vaquero , L. Rodero-Merino , J. Caceres , M. Lindner , A break in the clouds: [42] A. Reale, A guide to edge iot analytics. [Online]. Available: https://www.ibm. towards a cloud definition, ACM SIGCOMM Comput. Commun. Rev. 39 (1) (2008) com/blogs/internet-of-things/edge-iot-analytics , Blog, International Business Ma- 50–55 . chines. [18] T. Dillon , C. Wu , E. Chang , Cloud computing: issues and challenges, in: 2010 24th [43] P. Garcia Lopez , A. Montresor , D. Epema , A. Datta , T. Higashino , A. Iamnitchi , IEEE International Conference on Advanced Information Networking and Applica- M. Barcellos , P. Felber , E. Riviere , Edge-centric computing: vision and challenges, tions (AINA), IEEE, 2010, pp. 27–33 . ACM SIGCOMM Comput. Commun. Rev. 45 (5) (2015) 37–42 . [19] B. Sotomayor , R.S. Montero , I.M. Llorente , I. Foster , Virtual infrastructure manage- [44] M. Chiang , S. Ha , I. Chih-Lin , F. Risso , T. Zhang , Clarifying fog computing and ment in private and hybrid clouds, IEEE Internet Comput. 13 (5) (2009) . networking: 10 questions and answers, IEEE Commun. Mag. 55 (4) (2017) 18– [20] J. Acharya , S. Gaur , Edge compression of gps data for mobile iot, in: 2017 IEEE 20 . Fog World Congress (FWC), IEEE, 2017, pp. 1–6 . [45] What is edge computing?. [Online]. Available: https://www.ge.com/ [21] T. Zhang, Fog computing brings new business opportunities and disrup- digital/blog/what-edge-computing , Blog, General Electric. tions. [Online]. Available: http://internetofthingsagenda.techtarget.com/blog/ [46] F. Giust , G. Verin , K. Antevski , C. Joey , Y. Fang , W. Featherstone , F. Fontes , et al. , IoT-Agenda/Fog-computing-brings-new-business-opportunities-and-disruptions, Mec deployments in 4g and evolution towards 5g, ETSI White Paper 24 (2018) Blog, TechTarget. 1–24 . [22] M. Armbrust , A. Fox , R. Griffith , A.D. Joseph , R. Katz , A. Konwinski , G. Lee , D. Pat- [47] Y.C. Hu , M. Patel , D. Sabella , N. Sprecher , V. Young , Mobile edge computinga key terson , A. Rabkin , I. Stoica , et al. , A view of cloud computing, Commun. ACM 53 technology towards 5g, ETSI white paper 11 (11) (2015) 1–16 . (4) (2010) 50–58 .

322 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

[48] K.P. Kadiyala , J.A. Cobb , Inter-as traffic engineering with sdn, in: 2017 IEEE [79] E. Ahmed , A. Ahmed , I. Yaqoob , J. Shuja , A. Gani , M. Imran , M. Shoaib , Bringing Conference on Network Function Virtualization and Software Defined Networks computation closer toward the user network: is edge computing the solution? IEEE (NFV-SDN), IEEE, 2017, pp. 1–7 . Commun. Mag. 55 (11) (2017) 138–144 . [49] B. Mirkhanzadeh , A. Shakeri , C. Shao , M. Razo , M. Tacca , G.M. Galimberti , [80] W. Shi , J. Cao , Q. Zhang , Y. Li , L. Xu , Edge computing: vision and challenges, IEEE G. Martinelli , M. Cardani , A. Fumagalli , An sdn-enabled multi-layer protection and Internet Things J. 3 (5) (2016) 637–646 . restoration mechanism, Opt. Switch. Netw. (2018) . [81] N. Abbas , Y. Zhang , A. Taherkordi , T. Skeie , Mobile edge computing: a survey, IEEE [50] M. Satyanarayanan , P. Bahl , R. Caceres , N. Davies , The case for vm-based cloudlets Internet Things J. 5 (1) (2018) 450–465 . in mobile computing, IEEE Pervasive Comput. 8 (4) (2009) . [82] I. Stojmenovic , S. Wen , The fog computing paradigm: Scenarios and security is- [51] P. Hao , Y. Bai , X. Zhang , Y. Zhang , Edgecourier: an edge-hosted personal service sues, in: 2014 Federated Conference on Computer Science and Information Systems for low-bandwidth document synchronization in mobile cloud storage services, in: (FedCSIS), IEEE, 2014, pp. 1–8 . Proceedings of the Second ACM/IEEE Symposium on Edge Computing, ACM, 2017, [83] L. Gao , T.H. Luan , B. Liu , W. Zhou , S. Yu , Fog Computing and Its Applications in p. 7 . 5G, in: 5G Mobile Communications, Springer, 2017, pp. 571–593 . [52] Y. Li , W. Wang , Can mobile cloudlets support mobile applications? in: Infocom, [84] M. Chiang , T. Zhang , Fog and iot: an overview of research opportunities, IEEE 2014 Proceedings IEEE, IEEE, 2014, pp. 1060–1068 . Internet Things J. 3 (6) (2016) 854–864 . [53] Y. Jararweh , L. Tawalbeh , F. Ababneh , F. Dosari , Resource efficient mobile com- [85] P. Varshney , Y. Simmhan , Demystifying fog computing: characterizing architec- puting using cloudlet infrastructure, in: 2013 IEEE Ninth International Conference tures, applications and abstractions, in: 2017 IEEE 1st International Conference on on Mobile Ad-hoc and Sensor Networks (MSN), IEEE, 2013, pp. 373–377 . Fog and Edge Computing (ICFEC), IEEE, 2017, pp. 115–124 . [54] Y. Cui , J. Song , K. Ren , M. Li , Z. Li , Q. Ren , Y. Zhang , Software defined coopera- [86] P. Hu , S. Dhelim , H. Ning , T. Qiu , Survey on fog computing: architecture, key tive offloading for mobile cloudlets, IEEE/ACM Trans. Netw. 25 (3) (2017) 1746– technologies, applications and open issues, J. Netw. Comput. Appl. (2017) . 1760 . [87] P. Bellavista , L. Foschini , D. Scotece , Converging mobile edge computing, fog com- [55] A. Davies, Cisco pushes iot analytics to the extreme edge with mist computing, and iot quality requirements, in: 2017 IEEE 5th International Conference puting. [Online]. Available: http://rethinkresearch.biz/articles/cisco-pushes-iot- on Future Internet of Things and Cloud (FiCloud), IEEE, 2017, pp. 313–320 . analytics-extreme-edge-mist-computing-2 , Blog, Rethink Research. [88] A. Yousefpour , G. Ishigaki , R. Gour , J.P. Jue , On reducing iot service delay via fog [56] J.S. Preden , K. Tammemäe , A. Jantsch , M. Leier , A. Riid , E. Calis , The benefits offloading, IEEE Internet Things J. 5 (2) (2018) 998–1010 . of self-awareness and attention in fog and mist computing, Comput. (Long Beach [89] S. Sarkar , S. Chatterjee , S. Misra , Assessment of the suitability of fog computing in Calif) 48 (7) (2015) 37–45 . the context of internet of things, IEEE Trans. Cloud Comput. 6 (1) (2018) 46–59 . [57] P.M.P. Silva , J. Rodrigues , J. Silva , R. Martins , L. Lopes , F. Silva , Using edge-clouds [90] X. Lyu , H. Tian , L. Jiang , A. Vinel , S. Maharjan , S. Gjessing , Y. Zhang , Selective to reduce load on traditional wifi infrastructures and improve quality of experience, offloading in mobile edge computing for the green internet of things, IEEE Netw. in: 2017 IEEE 1st International Conference on Fog and Edge Computing (ICFEC), 32 (1) (2018) 54–60 . IEEE, 2017, pp. 61–67 . [91] X. Masip-Bruin , E. Marín-Tordera , G. Tashakor , A. Jukan , G.-J. Ren , Foggy clouds [58] A. Salem , T. Nadeem , Lamen: leveraging resources on anonymous mobile edge and cloudy fogs: a real need for coordinated management of fog-to-cloud comput- nodes, in: Proceedings of the Eighth Wireless of the Students, by the Students, and ing systems, IEEE Wirel. Commun. 23 (5) (2016) 120–128 . for the Students Workshop, ACM, 2016, pp. 15–17 . [92] L. Tong , Y. Li , W. Gao , A hierarchical edge cloud architecture for mobile computing, [59] R. Morabito , Virtualization on internet of things edge devices with container tech- in: INFOCOM 2016-The 35th Annual IEEE International Conference on Computer nologies: a performance evaluation, IEEE Access 5 (2017) 8835–8850 . Communications, IEEE, IEEE, 2016, pp. 1–9 . [60] V. Bahl , The emergence of micro datacenters (cloudlets) for mobile computing, [93] E.K. Markakis , K. Karras , A. Sideris , G. Alexiou , E. Pallis , Computing, caching, and 2015 . communication at the edge: the cornerstone for building a versatile 5g ecosystem, [61] S. Abdelwahab , B. Hamdaoui , M. Guizani , T. Znati , Cloud of things for sens- IEEE Commun. Mag. 55 (11) (2017) 152–157 . ing-as-a-service: architecture, algorithms, and use case, IEEE Internet Things J. 3 [94] E.K. Markakis , K. Karras , N. Zotos , A. Sideris , T. Moysiadis , A. Corsaro , G. Alexiou , (6) (2016) 1099–1112 . C. Skianis , G. Mastorakis , C.X. Mavromoustakis , et al. , Exegesis: extreme edge re- [62] M. Jang , K. Schwan , K. Bhardwaj , A. Gavrilovska , A. Avasthi , Personal clouds: source harvesting for a virtualized fog environment, IEEE Commun. Mag. 55 (7) Sharing and integrating networked resources to enhance end user experiences, in: (2017) 173–179 . INFOCOM, 2014 Proceedings IEEE, IEEE, 2014, pp. 2220–2228 . [95] C. Chang , S.N. Srirama , R. Buyya , Indie fog: an efficient fog-computing infrastruc- [63] A. Sathiaseelan , A. Lertsinsrubtavee , A. Jagan , P. Baskaran , J. Crowcroft , ture for the internet of things, Comput. (Long Beach Calif) 50 (9) (2017) 92–98 . Cloudrone: Micro clouds in the sky, in: Proceedings of the 2nd Workshop on Micro [96] C.C. Byers , Architectural imperatives for fog computing: use cases, requirements, Aerial Vehicle Networks, Systems, and Applications for Civilian Use, ACM, 2016, and architectural techniques for fog-enabled iot networks, IEEE Commun. Mag. 55 pp. 41–44 . (8) (2017) 14–20 . [64] K. Habak , E.W. Zegura , M. Ammar , K.A. Harras , Workload management for dy- [97] M. Yannuzzi , R. Milito , R. Serral-Gracià, D. Montero , M. Nemirovsky , Key ingredi- namic mobile device clusters in edge femtoclouds, in: Proceedings of the Second ents in an iot recipe: Fog computing, cloud computing, and more fog computing, in: ACM/IEEE Symposium on Edge Computing, ACM, 2017, p. 6 . 2014 IEEE 19th International Workshop on Computer Aided Modeling and Design [65] H. Chang , A. Hari , S. Mukherjee , T. Lakshman , Bringing the cloud to the edge, of Communication Links and Networks (CAMAD), IEEE, 2014, pp. 325–329 . in: 2014 IEEE Conference on Computer Communications Workshops (INFOCOM [98] Z. Hao , E. Novak , S. Yi , Q. Li , Challenges and software architecture for fog com- WKSHPS), IEEE, 2014, pp. 346–351 . puting, IEEE Internet Comput. 21 (2) (2017) 44–53 . [66] K. Bhardwaj , M.-W. Shih , P. Agarwal , A. Gavrilovska , T. Kim , K. Schwan , Fast, [99] S. Yi , Z. Qin , Q. Li , Security and privacy issues of fog computing: a survey, in: Inter- scalable and secure onloading of edge functions using airbox, in: Edge Computing national Conference on Wireless Algorithms, Systems, and Applications, Springer, (SEC), IEEE/ACM Symposium on, IEEE, 2016, pp. 14–27 . 2015, pp. 685–695 . [67] M. Villari , M. Fazio , S. Dustdar , O. Rana , R. Ranjan , Osmotic computing: a new [100] D. Grewe , M. Wagner , M. Arumaithurai , I. Psaras , D. Kutscher , Information-cen- paradigm for edge/cloud integration, IEEE Cloud Comput. 3 (6) (2016) 76–83 . tric mobile edge computing for connected vehicle environments: challenges and [68] A. Morshed , P.P. Jayaraman , T. Sellis , D. Georgakopoulos , M. Villari , R. Ranjan , research directions, in: Proceedings of the Workshop on Mobile Edge Communica- Deep osmosis: holistic distributed deep learning in osmotic computing, IEEE Cloud tions, ACM, 2017, pp. 7–12 . Comput. 4 (6) (2018) 22–32 . [101] C. Perera , Y. Qin , J.C. Estrella , S. Reiff-Marganiec , A.V. Vasilakos , Fog computing [69] I.S. Association, Ieee 1934–2018 - ieee standard for adoption of openfog reference for sustainable smart cities: a survey, ACM Comput. Surv. (CSUR) 50 (3) (2017) architecture for fog computing. [Online]. Available: https://standards.ieee.org/ 32 . standard/1934-2018.html . [102] J. Ni , A. Zhang , X. Lin , X.S. Shen , Security, privacy, and fairness in fog-based ve- [70] E. Marín-Tordera , X. Masip-Bruin , J. García-Almiñana , A. Jukan , G.-J. Ren , J. Zhu , hicular crowdsensing, IEEE Commun. Mag. 55 (6) (2017) 146–152 . Do we all really know what a fog node is? current trends towards an open definition, [103] R. Mahmud , R. Kotagiri , R. Buyya , Fog Computing: A Taxonomy, Survey and Future Comput. Commun. 109 (2017) 117–130 . Directions, in: Internet of Everything, Springer, 2018, pp. 103–130 . [71] L.M. Vaquero , L. Rodero-Merino , Finding your way in the fog: towards a compre- [104] P. Zhang , J.K. Liu , F.R. Yu , M. Sookhak , M.H. Au , X. Luo , A survey on access control hensive definition of fog computing, ACM SIGCOMM Comput. Commun. Rev. 44 in fog computing, IEEE Commun. Mag. 56 (2) (2018) 144–149 . (5) (2014) 27–32 . [105] S.N. Shirazi , A. Gouglidis , A. Farshad , D. Hutchison , The extended cloud: review [72] S. Sarkar , S. Misra , Theoretical modelling of fog computing: a green computing and analysis of mobile edge computing and fog from a security and resilience per- paradigm to support iot applications, IET Netw. 5 (2) (2016) 23–29 . spective, IEEE J. Sel. Areas Commun. 35 (11) (2017) 2586–2595 . [73] A. Sill , Standards at the edge of the cloud, IEEE Cloud Comput. 4 (2) (2017) 63–67 . [106] X. Sun , N. Ansari , Edgeiot: mobile edge computing for the internet of things, IEEE [74] J.C. Guevara , L.F. Bittencourt , N.L. da Fonseca , Class of service in fog computing, Commun. Mag. 54 (12) (2016) 22–29 . in: 2017 IEEE 9th Latin-American Conference on Communications (LATINCOM), [107] A. Brogi , S. Forti , Qos-aware deployment of iot applications through the fog, IEEE IEEE, 2017, pp. 1–6 . Internet Things J. 4 (5) (2017) 1185–1192 . [75] P. Manzoni , E. Hernández-Orallo , C.T. Calafate , J.-C. Cano , A proposal for a pub- [108] C. Dsouza , G.-J. Ahn , M. Taguinod , Policy-driven security management for fog com- lish/subscribe, disruption tolerant content island for fog computing, in: Proceed- puting: Preliminary framework and a case study, in: 2014 IEEE 15th International ings of the 3rd Workshop on Experiences with the Design and Implementation of Conference on Information Reuse and Integration (IRI), IEEE, 2014, pp. 16–23 . Smart Objects, ACM, 2017, pp. 47–52 . [109] S. Yi , Z. Hao , Z. Qin , Q. Li , Fog computing: Platform and applications, in: 2015 [76] A. Jonathan , M. Uluyol , A. Chandra , J. Weissman , Ensuring reliability in geo- Third IEEE Workshop on Hot Topics in Web Systems and Technologies (HotWeb), distributed edge cloud, in: Resilience Week (RWS), 2017, IEEE, 2017, pp. 127– IEEE, 2015, pp. 73–78 . 132 . [110] A. Kiani , N. Ansari , Toward hierarchical mobile edge computing: an auction-based [77] J. Weinman , The 10 laws of fogonomics, IEEE Cloud Comput. 4 (6) (2018) 8–14 . profit maximization approach, IEEE Internet Things J. 4 (6) (2017) 2082–2091 . [78] G. Klas , Edge computing and the role of cellular networks, Comput. (Long Beach [111] Y. Li , Y. Chen , T. Lan , G. Venkataramani , Mobiqor: Pushing the envelope of mo- Calif) 50 (10) (2017) 40–49 . bile edge computing via quality-of-result optimization, in: 2017 IEEE 37th In-

323 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

ternational Conference on Distributed Computing Systems (ICDCS), IEEE, 2017, [140] Y. Nan , W. Li , W. Bao , F.C. Delicato , P.F. Pires , A.Y. Zomaya , A dynamic trade- pp. 1261–1270 . off data processing framework for delay-sensitive applications in cloud of things [112] I. Farris , R. Girau , L. Militano , M. Nitti , L. Atzori , A. Iera , G. Morabito , Social virtual systems, J. Parallel Distrib. Comput. 112 (2018) 53–66 . objects in the edge cloud, IEEE Cloud Comput. 2 (6) (2015) 20–28 . [141] X. Wang , L.T. Yang , X. Xie , J. Jin , M.J. Deen , A cloud-edge computing framework [113] A.S. Prasad , M. Arumaithurai , D. Koll , X. Fu , Raera: A robust auctioning approach for cyber-physical-social services, IEEE Commun. Mag. 55 (11) (2017) 80–85 . for edge resource allocation, in: Proceedings of the Workshop on Mobile Edge Com- [142] M. Ficco , C. Esposito , Y. Xiang , F. Palmieri , Pseudo-dynamic testing of realistic munications, ACM, 2017, pp. 49–54 . edge-fog cloud ecosystems, IEEE Commun. Mag. 55 (11) (2017) 98–104 . [114] S. Varadi , G.G. Varkonyi , A. Kertesz , Law and iot: How to see things clearly in the [143] J. Li , J. Jin , D. Yuan , H. Zhang , Virtual fog: a virtualization enabled fog computing fog, in: 2018 Third International Conference on Fog and Mobile Edge Computing framework for internet of things, IEEE Internet Things J. 5 (1) (2018) 121–131 . (FMEC), IEEE, 2018, pp. 233–238 . [144] T. Yu , X. Wang , A. Shami , A novel fog computing enabled temporal data reduction [115] M. Yannuzzi , R. Irons-Mclean , F. van Lingen , S. Raghav , A. Somaraju , C. Byers , scheme in iot systems, in: GLOBECOM 2017 - 2017 IEEE Global Communications T. Zhang , A. Jain , J. Curado , D. Carrera , et al. , Toward a converged openfog and Conference, 2017, pp. 1–5 . etsi mano architecture, in: Fog World Congress (FWC), 2017 IEEE, IEEE, 2017, [145] W. Zhang , Z. Zhang , H.-C. Chao , Cooperative fog computing for dealing with big pp. 1–6 . data in the internet of vehicles: architecture and hierarchical resource management, [116] Y. Harchol , A. Mushtaq , J. McCauley , A. Panda , S. Shenker , Cessna: Resilient edge– IEEE Commun. Mag. 55 (12) (2017) 60–67 . computing, in: Proceedings of the 2018 Workshop on Mobile Edge Communica- [146] H. Zhang , Y. Qiu , X. Chu , K. Long , V.C.M. Leung , Fog radio access networks: mobil- tions, ACM, 2018, pp. 1–6 . ity management, interference mitigation, and resource optimization, IEEE Wirel. [117] S. Zang , W. Bao , P.L. Yeoh , B. Vucetic , Y. Li , Paying less for more? combo plans Commun. 24 (6) (2017) 120–127 . for edge-computing services, USENIX Workshop on Hot Topics in Edge Computing [147] B. Lorenzo , J. Garcia-Rois , X. Li , J. Gonzalez-Castano , Y. Fang , A robust dynamic (HotEdge 18), USENIX Association, Boston, MA, 2018 . edge network architecture for the internet of things, IEEE Netw. 32 (1) (2018) [118] I. Psaras , O. Ascigil , S. Rene , G. Pavlou , A. Afanasyev , L. Zhang , Mobile data repos- 8–15 . itories at the edge, USENIX Workshop on Hot Topics in Edge Computing (HotEdge [148] T. Suganuma , T. Oide , S. Kitagami , K. Sugawara , N. Shiratori , Multiagent-based 18), USENIX Association, Boston, MA, 2018 . flexible edge computing architecture for iot, IEEE Netw. 32 (1) (2018) 16–23 . [119] D. Kim , H. Lee , H. Song , N. Choi , Y. Yi , On the economics of fog computing: In- [149] Q. Yuan , H. Zhou , J. Li , Z. Liu , F. Yang , X.S. Shen , Toward efficient content delivery ter-play among infrastructure and service providers, users, and edge resource own- for automated driving services: an edge computing solution, IEEE Netw. 32 (1) ers, in: 2018 IEEE International Conference on Communications (ICC), IEEE, 2018, (2018) 80–86 . pp. 1–6 . [150] A. Lebre , J. Pastor , A. Simonet , F. Desprez , Revising openstack to operate fog/edge [120] R.-A. Cherrueau , A. Lebre , D. Pertin , F. Wuhib , J.M. Soares , Edge computing re- computing infrastructures, in: 2017 IEEE International Conference on Cloud Engi- source management system: a critical building block! initiating the debate via open- neering (IC2E), IEEE, 2017, pp. 138–148 . stack, USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), USENIX [151] M. Abderrahim , M. Ouzzif , K. Guillouard , J. Francois , A. Lèbre , A holistic moni- Association, Boston, MA, 2018 . toring service for fog/edge infrastructures: a foresight study, in: The IEEE 5th In- [121] M. Syamkumar , P. Barford , R. Durairajan , Deployment characteristics of “the edge ” ternational Conference on Future Internet of Things and Cloud (FiCloud 2017), in mobile edge computing, in: Proceedings of the 2018 Workshop on Mobile Edge 2017 . Communications, in: MECOMM’18, ACM, 2018, pp. 43–49 . [152] C.A. Garcia-Perez , P. Merino , Enabling low latency services on lte networks, in:

[122] C.-Y. Chang , K. Alexandris , N. Nikaein , K. Katsalis , T. Spyropoulos , Mec architec- IEEE International Workshops on Foundations and Applications of Self ∗ Systems, tural implications for lte/lte-a networks, in: Proceedings of the Workshop on Mo- IEEE, 2016, pp. 248–255 . bility in the Evolving Internet Architecture, ACM, 2016, pp. 13–18 . [153] M. Le , Z. Song , Y.-W. Kwon , E. Tilevich , Reliable and efficient mobile edge [123] S. Abdelwahab , S. Zhang , A. Greenacre , K. Ovesen , K. Bergman , B. Hamdaoui , computing in highly dynamic and volatile environments, in: 2017 Second When clones flock near the fog, IEEE Internet Things J. (2018) . International Conference on Fog and Mobile Edge Computing (FMEC), IEEE, 2017, [124] R. Deng , R. Lu , C. Lai , T.H. Luan , H. Liang , Optimal workload allocation in fog– pp. 113–120 . cloud computing toward balanced delay and power consumption, IEEE Internet [154] I. Lujic , V. De Maio , I. Brandic , Efficient edge storage management based on near Things J. 3 (6) (2016) 1171–1181 . real-time forecasts, in: 2017 IEEE 1st International Conference on Fog and Edge [125] C. Alippi , R. Fantacci , D. Marabissi , M. Roveri , A cloud to the ground: the new Computing (ICFEC), IEEE, 2017, pp. 21–30 . frontier of intelligent and autonomous networks of things, IEEE Commun. Mag. 54 [155] W. Tärneberg , A.V. Papadopoulos , A. Mehta , J. Tordsson , M. Kihl , Distributed ap- (12) (2016) 14–20 . proach to the holistic resource management of a mobile cloud network, in: 2017 [126] O. Skarlat , M. Nardelli , S. Schulte , M. Borkowski , P. Leitner , Optimized iot service IEEE 1st International Conference on Fog and Edge Computing (ICFEC), IEEE, 2017, placement in the fog, Serv. Oriented Comput. Appl. 11 (4) (2017) 427–443 . pp. 51–60 . [127] S. Tomovic , K. Yoshigoe , I. Maljevic , I. Radusinovic , Software-defined fog network [156] S. Shekhar , A.D. Chhokra , A. Bhattacharjee , G. Aupy , A. Gokhale , Indices: exploit- architecture for iot, Wirel. Pers. Commun. 92 (1) (2017) 181–196 . ing edge resources for performance-aware cloud-hosted services, in: 2017 IEEE [128] Y.-J. Ku , D.-Y. Lin , C.-F. Lee , P.-J. Hsieh , H.-Y. Wei , C.-T. Chou , A.-C. Pang , 5G 1st International Conference on Fog and Edge Computing (ICFEC), IEEE, 2017, radio access network design with the fog paradigm: confluence of communications pp. 75–80 . and computing, IEEE Commun. Mag. 55 (4) (2017) 46–52 . [157] M.I. Naas , P.R. Parvedy , J. Boukhobza , L. Lemarchand , ifogstor: an iot data place- [129] R. Moreno-Vozmediano , R.S. Montero , E. Huedo , I.M. Llorente , Cross-site virtual ment strategy for fog infrastructure, in: 2017 IEEE 1st International Conference on network in cloud and fog computing, IEEE Cloud Comput. 4 (2) (2017) 46–53 . Fog and Edge Computing (ICFEC), IEEE, 2017, pp. 97–104 . [130] K. Liang , L. Zhao , X. Chu , H.-H. Chen , An integrated architecture for software de- [158] P. Yang , N. Zhang , Y. Bi , L. Yu , X.S. Shen , Catalyzing cloud-fog interoperation in fined and virtualized radio access networks with fog computing, IEEE Netw. 31 (1) 5g wireless networks: an sdn approach, IEEE Netw. 31 (5) (2017) 14–20 . (2017) 80–87 . [159] B. Tang , Z. Chen , G. Hefferman , T. Wei , H. He , Q. Yang , A hierarchical distributed [131] A. Kapsalis , P. Kasnesis , I.S. Venieris , D.I. Kaklamani , C.Z. Patrikakis , A cooperative fog computing architecture for big data analysis in smart cities, in: Proceedings of fog approach for effective workload balancing, IEEE Cloud Comput. 4 (2) (2017) the ASE BigData & SocialInformatics 2015, ACM, 2015, p. 28 . 36–45 . [160] J. He , J. Wei , K. Chen , Z. Tang , Y. Zhou , Y. Zhang , Multitier fog computing with [132] S. Yangui , P. Ravindran , O. Bibani , R.H. Glitho , N.B. Hadj-Alouane , M.J. Morrow , large-scale iot data analytics for smart cities, IEEE Internet Things J. 5 (2) (2018) P.A. Polakos , A platform as-a-service for hybrid cloud/fog environments, in: 2016 677–686 . IEEE International Symposium on Local and Metropolitan Area Networks (LAN- [161] S.M. Oteafy , H.S. Hassanein , Iot in the fog: a roadmap for data-centric iot develop- MAN)„ IEEE, 2016, pp. 1–7 . ment, IEEE Commun. Mag. 56 (3) (2018) 157–163 . [133] J. Li , J. Jin , D. Yuan , M. Palaniswami , K. Moessner , Ehopes: Data-centered fog [162] J. Li , J. Jin , D. Yuan , H. Zhang , Virtual fog: a virtualization enabled fog comput- platform for smart living, in: 2015 International Telecommunication Networks and ing framework for internet of things, IEEE Internet Things J. 5 (1) (2018) 121– Applications Conference (ITNAC), IEEE, 2015, pp. 308–313 . 131 . [134] F. van Lingen , M. Yannuzzi , A. Jain , R. Irons-Mclean , O. Lluch , D. Carrera , [163] D. Puthal , M.S. Obaidat , P. Nanda , M. Prasad , S.P. Mohanty , A.Y. Zomaya , Secure J.L. Perez , A. Gutierrez , D. Montero , J. Marti , et al. , The unavoidable convergence and sustainable load balancing of edge data centers in fog computing, IEEE Com- of nfv, 5g, and fog: a model-driven approach to bridge cloud and edge, IEEE Com- mun. Mag. 56 (5) (2018) 60–65 . mun. Mag. 55 (8) (2017) 28–35 . [164] M.L. Raagaard , P. Pop , M. Gutiérrez , W. Steiner , Runtime reconfiguration of [135] R. Vilalta , V. López , A. Giorgetti , S. Peng , V. Orsini , L. Velasco , R. Serral-Gracia , time-sensitive networking (tsn) schedules for fog computing, in: Fog World D. Morris , S. De Fina , F. Cugini , et al. , Telcofog: a unified flexible fog and cloud Congress (FWC), 2017 IEEE, IEEE, 2017, pp. 1–6 . computing architecture for 5g networks, IEEE Commun. Mag. 55 (8) (2017) 36–43 . [165] C. Pahl , N. El Ioini , S. Helmer , B. Lee , An architecture pattern for trusted orchestra- [136] H. Zhang , Y. Zhang , Y. Gu , D. Niyato , Z. Han , A hierarchical game framework tion in iot edge clouds, in: 2018 Third International Conference on Fog and Mobile for resource management in fog computing, IEEE Commun. Mag. 55 (8) (2017) Edge Computing (FMEC), IEEE, 2018, pp. 63–70 . 52–57 . [166] N. Mohan , A. Zavodovski , P. Zhou , J. Kangasharju , Anveshak: Placing edge servers [137] P. Borylo , A. Lason , J. Rzasa , A. Szymanski , A. Jajszczyk , Energy-aware fog and in the wild, in: Proceedings of the 2018 Workshop on Mobile Edge Communica- cloud interplay supported by wide area software defined networking, in: 2016 IEEE tions, ACM, 2018, pp. 7–12 . International Conference on Communications (ICC), IEEE, 2016, pp. 1–7 . [167] A. Silvestro , N. Mohan , J. Kangasharju , F. Schneider , X. Fu , Mute: Multi-tier edge [138] B. Amento , R.J. Hall , K. Joshi , K.H. Purdy , Focusstack: orchestrating edge clouds networks, in: Proceedings of the 5th Workshop on CrossCloud Infrastructures & using focus of attention, IEEE Internet Comput. 21 (1) (2017) 56–61 . Platforms, ACM, 2018, p. 1 . [139] M. Yannuzzi , F. van Lingen , A. Jain , O.L. Parellada , M.M. Flores , D. Carrera , [168] S. Garg , A. Singh , K. Kaur , S. Batra , N. Kumar , M.S. Obaidat , Edge-based content J.L. Pérez , D. Montero , P. Chacin , A. Corsaro , et al. , A new era for cities with fog delivery for providing qoe in wireless networks using quotient filter, in: 2018 IEEE computing, IEEE Internet Comput. 21 (2) (2017) 54–67 . International Conference on Communications (ICC), IEEE, 2018, pp. 1–6 .

324 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

[169] C. Meiklejohn , H. Miller , Z. Lakhani , Towards a solution to the red wedding prob- [197] E. Saurez , K. Hong , D. Lillethun , U. Ramachandran , B. Ottenwälder , Incremental lem, USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), USENIX deployment and migration of geo-distributed situation awareness applications in Association, Boston, MA, 2018 . the fog, in: Proceedings of the 10th ACM International Conference on Distributed [170] C.-Y. Li , H.-Y. Liu , P.-H. Huang , H.-T. Chien , G.-H. Tu , P.-Y. Hong , Y.-D. Lin , Mobile and Event-based Systems, ACM, 2016, pp. 258–269 . edge computing platform deployment in 4g LTE networks: a middlebox approach, [198] R. Jain , S. Tata , Cloud to edge: distributed deployment of process-aware iot appli- USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), USENIX Asso- cations, in: 2017 IEEE International Conference on Edge Computing (EDGE), IEEE, ciation, Boston, MA, 2018 . 2017, pp. 182–189 . [171] S. Cirani , G. Ferrari , N. Iotti , M. Picone , The iot hub: a fog node for seamless man- [199] K. Hong , D. Lillethun , U. Ramachandran , B. Ottenwälder , B. Koldehofe , Mobile agement of heterogeneous connected smart objects, in: 2015 12th Annual IEEE fog: a programming model for large-scale applications on the internet of things, in: International Conference on Sensing, Communication, and Networking-Workshops Proceedings of the Second ACM SIGCOMM Workshop on Mobile Cloud Computing, (SECON Workshops), IEEE, 2015, pp. 1–6 . ACM, 2013, pp. 15–20 . [172] X. Hou , Y. Li , M. Chen , D. Wu , D. Jin , S. Chen , Vehicular fog computing: a view- [200] N. Choi , D. Kim , S.-J. Lee , Y. Yi , A fog operating system for user-oriented iot point of vehicles as the infrastructures, IEEE Trans. Veh. Technol. 65 (6) (2016) services: challenges and research directions, IEEE Commun. Mag. 55 (8) (2017) 3860–3873 . 44–51 . [173] C. Prazeres , M. Serrano , Soft-iot: Self-organizing fog of things, in: 2016 30th Inter- [201] N.K. Giang , M. Blackstock , R. Lea , V.C. Leung , Developing IOT applications in the national Conference on Advanced Information Networking and Applications Work- fog: a distributed dataflow approach, in: 2015 5th International Conference on the shops (WAINA), IEEE, 2016, pp. 803–808 . Internet of Things (IOT), IEEE, 2015, pp. 155–162 . [174] N. Kaminski , I. Macaluso , E. Di Pascale , A. Nag , J. Brady , M. Kelly , K. Nolan , [202] E.G. Renart , J. Diaz-Montes , M. Parashar , Data-driven stream processing at the W. Guibene , L. Doyle , A neural-network-based realization of in-network compu- edge, in: 2017 IEEE 1st International Conference on Fog and Edge Computing tation for the internet of things, in: 2017 IEEE International Conference on Com- (ICFEC), IEEE, 2017, pp. 31–40 . munications (ICC), IEEE, 2017, pp. 1–6 . [203] X. Su , P. Li , J. Riekki , X. Liu , J. Kiljander , J.-P. Soininen , C. Prehofer , H. Flores , [175] I. Farris , A. Orsino , L. Militano , M. Nitti , G. Araniti , L. Atzori , A. Iera , Federations Y. Li , Adaptive recovery of incomplete datasets for edge analytics, in: 2018 IEEE of connected things for delay-sensitive iot services in 5g environments, in: 2017 International Conference on Pervasive Computing and Communications (PerCom), IEEE International Conference on Communications (ICC), IEEE, 2017, pp. 1–6 . IEEE, 2018, pp. 1–9 . [176] M. Sookhak , F.R. Yu , Y. He , H. Talebian , N.S. Safa , N. Zhao , M.K. Khan , N. Kumar , [204] B. Cheng , G. Solmaz , F. Cirillo , E. Kovacs , K. Terasawa , A. Kitazawa , Fogflow: easy Fog vehicular computing: augmentation of fog computing using vehicular cloud programming of iot services over cloud and edges for smart cities, IEEE Internet computing, IEEE Veh. Technol. Mag. 12 (3) (2017) 55–64 . Things J. 5 (2) (2018) 696–707 . [177] M. Tao , K. Ota , M. Dong , Foud: integrating fog and cloud for 5g-enabled v2g net- [205] T. Jeong , J. Chung , J.W.-K. Hong , S. Ha , Towards a distributed computing frameworks, IEEE Netw. 31 (2) (2017) 8–13 . work for fog, in: Fog World Congress (FWC), 2017 IEEE, IEEE, 2017, pp. 1–6 . [178] C. Huang , R. Lu , K.-K.R. Choo , Vehicular fog computing: architecture, use case, [206] H. Gupta , U. Ramachandran , Fogstore: A geo-distributed key-value store guaran- and security and forensic challenges, IEEE Commun. Mag. 55 (11) (2017) 105– teeing low latency for strongly consistent access, in: Proceedings of the 12th ACM 111 . International Conference on Distributed and Event-based Systems, ACM, 2018, [179] J. Ren , H. Guo , C. Xu , Y. Zhang , Serving at the edge: a scalable iot architecture pp. 148–159 . based on transparent computing, IEEE Netw. 31 (5) (2017) 96–105 . [207] J. Zhao , T. Tiplea , R. Mortier , J. Crowcroft , L. Wang , Data analytics service com- [180] S. Alonso-Monsalve , F. García-Carballeira , A. Calderón , Fog computing through position and deployment on edge devices, in: Proceedings of the 2018 Workshop public-resource computing and storage, in: 2017 Second International Conference on Big Data Analytics and Machine Learning for Data Communication Networks, on Fog and Mobile Edge Computing (FMEC), IEEE, 2017, pp. 81–87 . in: Big-DAMA ’18, ACM, 2018, pp. 27–32 . [181] H. Li , K. Ota , M. Dong , Learning iot in edge: deep learning for the internet of things [208] S.A. Noghabi , J. Kolb , P. Bodik , E. Cuervo , Steel: Simplified development and de- with edge computing, IEEE Netw. 32 (1) (2018) 96–101 . ployment of edge-cloud applications, 10th USENIX Workshop on Hot Topics in [182] M. Ryden , K. Oh , A. Chandra , J. Weissman , Nebula: Distributed edge cloud for data Cloud Computing (HotCloud 18), USENIX Association, Boston, MA, 2018 . intensive computing, in: 2014 IEEE International Conference on Cloud Engineering [209] H. Gupta , Z. Xu , U. Ramachandran , Datafog: towards a holistic data management (IC2E), IEEE, 2014, pp. 57–66 . platform for the iot age at the network edge, USENIX Workshop on Hot Topics in [183] F. Hagenauer , C. Sommer , T. Higuchi , O. Altintas , F. Dressler , Vehicular micro Edge Computing (HotEdge 18), USENIX Association, Boston, MA, 2018 . clouds as virtual edge servers for efficient data collection, in: 23rd ACM Interna- [210] Q. Fan , N. Ansari , Cost aware cloudlet placement for big data processing at the tional Conference on Mobile Computing and Networking (MobiCom 2017), 2nd edge, in: 2017 IEEE International Conference on Communications (ICC), IEEE, ACM International Workshop on Smart, Autonomous, and Connected Vehicular 2017, pp. 1–6 . Systems and Services (CarSys 2017), 2017 . [211] Y. Xiao , M. Noreikis , A. Ylä-Jaäiski , Qos-oriented capacity planning for edge com- [184] S.H. Mortazavi , M. Salehe , C.S. Gomes , C. Phillips , E. de Lara , Cloudpath: a multi- puting, in: 2017 IEEE International Conference on Communications (ICC), IEEE, -tier cloud computing framework, in: Proceedings of the Second ACM/IEEE Sym- 2017, pp. 1–6 . posium on Edge Computing, in: SEC ’17, ACM, 2017, pp. 20:1–20:13 . [212] H. Guo , J. Liu , H. Qin , Collaborative mobile edge computation offloading for iot [185] P. Bellavista , S. Chessa , L. Foschini , L. Gioia , M. Girolami , Human-enabled edge over fiber-wireless networks, IEEE Netw. 32 (1) (2018) 66–71 . computing: exploiting the crowd as a dynamic extension of mobile edge computing, [213] Y.-Y. Shih , W.-H. Chung , A.-C. Pang , T.-C. Chiu , H.-Y. Wei , Enabling low-la- IEEE Commun. Mag. 56 (1) (2018) 145–155 . tency applications in fog-radio access networks, IEEE Netw. 31 (1) (2017) 52– [186] S. Jeong , O. Simeone , J. Kang , Mobile edge computing via a uav-mounted cloudlet: 58 . optimization of bit allocation and path planning, IEEE Trans. Veh. Technol. 67 (3) [214] S. Mondal , G. Das , E. Wong , A novel cost optimization framework for multi-cloudlet (2018) 2049–2063 . environment over optical access networks, in: GLOBECOM 2017 - 2017 IEEE Global [187] O.A. Nasr , Y. Amer , M. AboBakr , The ǣdroplet ǥ: A new personal device to enable Communications Conference, 2017, pp. 1–7 . fog computing, in: 2018 Third International Conference on Fog and Mobile Edge [215] C. Vallati , A. Virdis , E. Mingozzi , G. Stea , Exploiting lte d2d communica- Computing (FMEC), IEEE, 2018, pp. 93–99 . tions in m2m fog platforms: Deployment and practical issues, in: 2015 IEEE [188] H. Moustafa , E.M. Schooler , J. McCarthy , Reverse cdn in fog computing: the lifecy- 2nd World Forum on Internet of Things (WF-IoT), IEEE, 2015, pp. 585– cle of video data in connected and autonomous vehicles, in: 2017 IEEE Fog World 590 . Congress (FWC), IEEE, 2017, pp. 1–5 . [216] A. Mehta , W. Tärneberg , C. Klein , J. Tordsson , M. Kihl , E. Elmroth , How beneficial [189] C. Xu , K. Wang , G. Xu , P. Li , S. Guo , J. Luo , Making big data open in collaborative are intermediate layer data centers in mobile edge networks? in: IEEE Interna-

edges: a blockchain-based framework with reduced resource requirements, in: 2018 tional Workshops on Foundations and Applications of Self ∗ Systems, IEEE, 2016, IEEE International Conference on Communications (ICC), IEEE, 2018, pp. 1–6 . pp. 222–229 . [190] Z. Wen , P. Bhatotia , R. Chen , M. Lee , et al. , Approxiot: approximate analytics for [217] A. Ceselli , M. Premoli , S. Secci , Mobile edge cloud network design optimization, edge computing, in: 2018 IEEE 38th International Conference on Distributed Com- IEEE/ACM Trans. Netw. 25 (3) (2017) 1818–1831 . puting Systems (ICDCS), IEEE, 2018, pp. 411–421 . [218] M. Jia , J. Cao , W. Liang , Optimal cloudlet placement and user to cloudlet allocation [191] A.R.N.S.A. Sathiaseelan , J.C.A. Secker , Wi-stitch: content delivery in converged in wireless metropolitan area networks, IEEE Trans. Cloud Comput. 5 (4) (2017) edge networks, ACM SIGCOMM Comput. Commun. Rev. 47 (5) (2017) . 725–737 . [192] K.E. Benson , G. Wang , N. Venkatasubramanian , Y.-J. Kim , Ride: A resilient iot data [219] B.P. Rimal , D.P. Van , M. Maier , Mobile-edge computing versus centralized cloud exchange middleware leveraging sdn and edge cloud resources, in: 2018 IEEE/ACM computing over a converged fiwi access network, IEEE Trans. Netw. Serv. Manag. Third International Conference on Internet-of-Things Design and Implementation 14 (3) (2017) 498–513 . (IoTDI), IEEE, 2018, pp. 72–83 . [220] J. Gedeon , J. Krisztinkovics , C. Meurisch , M. Stein , L. Wang , M. Mühlhäuser , A [193] M. Carmo , S. Jardim , A. Neto , R. Aguiar , D. Corujo , J.J. Rodrigues , Slicing wifi multi-cloudlet infrastructure for future smart cities: an empirical study, in: Proceed- wlan-sharing access infrastructures to enhance ultra-dense 5g networking, in: 2018 ings of the 1st International Workshop on Edge Systems, Analytics and Networking, IEEE International Conference on Communications (ICC), IEEE, 2018, pp. 1–6 . ACM, 2018, pp. 19–24 . [194] Q. Zhang , X. Zhang , Q. Zhang , W. Shi , H. Zhong , Firework: big data sharing and [221] M. Aazam , E.-N. Huh , Fog computing micro datacenter based dynamic resource processing in collaborative edge environment, in: 2016 Fourth IEEE Workshop estimation and pricing model for iot, in: 2015 IEEE 29th International Confer- on Hot Topics in Web Systems and Technologies (HotWeb), IEEE, 2016, pp. 20– ence on Advanced Information Networking and Applications (AINA), IEEE, 2015, 25 . pp. 687–694 . [195] K. Dantu , S.Y. Ko , L. Ziarek , Raina: reliability and adaptability in android for fog [222] F. Jalali , A. Vishwanath , J. de Hoog , F. Suits , Interconnecting fog computing and computing, IEEE Commun. Mag. 55 (4) (2017) 41–45 . microgrids for greening iot, in: Innovative Smart Grid Technologies-Asia (ISGT-A- [196] K. Kaur , T. Dhand , N. Kumar , S. Zeadally , Container-as-a-service at the edge: sia), 2016 IEEE, IEEE, 2016, pp. 693–698 . trade-off between energy efficiency and service availability at fog nano data cen- [223] S. El Kafhali , K. Salah , Efficient and dynamic scaling of fog nodes for iot devices, ters, IEEE Wirel. Commun. 24 (3) (2017) 48–56 . J. Supercomput. 73 (12) (2017) 5261–5284 .

325 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

[224] F. Malandrino , S. Kirkpatrick , C.-F. Chiasserini , How close to the edge?: de- in: Proceedings of the Second ACM/IEEE Symposium on Edge Computing, ACM, lay/utilization trends in mec, in: Proceedings of the 2016 ACM Workshop on 2017, p. 12 . Cloud-Assisted Networking, ACM, 2016, pp. 37–42 . [251] A. Machen , S. Wang , K.K. Leung , B.J. Ko , T. Salonidis , Live service migration in [225] N. Kamiyama , Y. Nakano , K. Shiomoto , G. Hasegawa , M. Murata , H. Miyahara , mobile edge clouds, IEEE Wirel. Commun. 25 (1) (2018) 140–147 . Analyzing effect of edge computing on reduction of web response time, in: Global [252] B. Németh , M. Szalay , J. Dóka , M. Rost , S. Schmid , L. Toka , B. Sonkoly , Fast and ef- Communications Conference (GLOBECOM), 2016 IEEE, IEEE, 2016, pp. 1–6 . ficient network service embedding method with adaptive offloading to the edge, in: [226] C.N. Le Tan , C. Klein , E. Elmroth , Location-aware load prediction in edge data cen- IEEE INFOCOM 2018-IEEE Conference on Computer Communications Workshops ters, in: 2017 Second International Conference on Fog and Mobile Edge Computing (INFOCOM WKSHPS), IEEE, 2018 . (FMEC), IEEE, 2017, pp. 25–31 . [253] S.H. Mortazavi , B. Balasubramanian , E. de Lara , S.P. Narayanan , Toward session [227] D.T. Nguyen , L.B. Le , V. Bhargava , Edge computing resource procurement: an on- consistency for the edge, USENIX Workshop on Hot Topics in Edge Computing line optimization approach, in: 2018 IEEE 4th World Forum on Internet of Things (HotEdge 18), USENIX Association, Boston, MA, 2018 . (WF-IoT), IEEE, 2018, pp. 807–812 . [254] R. Bruschi , F. Davoli , P. Lago , J.F. Pajo , Move with me: scalably keeping virtual [228] C. Anglano , M. Canonico , P. Castagno , M. Guazzone , M. Sereno , A game-theoretic objects close to users on the move, in: 2018 IEEE International Conference on Com- approach to coalition formation in fog provider federations, in: 2018 Third In- munications (ICC), IEEE, 2018, pp. 1–6 . ternational Conference on Fog and Mobile Edge Computing (FMEC), IEEE, 2018, [255] L. Gu , D. Zeng , S. Guo , A. Barnawi , Y. Xiang , Cost efficient resource management in pp. 123–130 . fog computing supported medical cyber-physical system, IEEE Trans. Emerg. Top [229] C. Anglano , M. Canonico , M. Guazzone , Profit-aware resource management for Comput. 5 (1) (2017) 108–119 . edge computing systems, in: Proceedings of the 1st International Workshop on Edge [256] V. Souza , W. Ramírez , X. Masip-Bruin , E. Marín-Tordera , G. Ren , G. Tashakor , Han- Systems, Analytics and Networking, ACM, 2018, pp. 25–30 . dling service allocation in combined fog-cloud scenarios, in: 2016 IEEE Interna- [230] M. Zeng , Y. Li , K. Zhang , M. Waqas , D. Jin , Incentive mechanism design for com- tional Conference on Communications (ICC), IEEE, 2016, pp. 1–5 . putation offloading in heterogeneous fog computing: a contract-based approach, [257] F.B. Jemaa , G. Pujolle , M. Pariente , Qos-aware vnf placement optimization in edge– in: 2018 IEEE International Conference on Communications (ICC), IEEE, 2018, central carrier cloud architecture, in: 2016 IEEE Global Communications Confer- pp. 1–6 . ence (GLOBECOM), IEEE, 2016, pp. 1–7 . [231] R. Urgaonkar , S. Wang , T. He , M. Zafer , K. Chan , K.K. Leung , Dynamic ser- [258] O. Skarlat , M. Nardelli , S. Schulte , S. Dustdar , Towards qos-aware fog service place- vice migration and workload scheduling in edge-clouds, Perform. Eval. 91 (2015) ment, in: 2017 IEEE 1st International Conference on Fog and Edge Computing 205–228 . (ICFEC), IEEE, 2017, pp. 89–96 . [232] W. Zhang , Y. Hu , Y. Zhang , D. Raychaudhuri , Segue: Quality of service [259] H.-J. Hong , P.-H. Tsai , C.-H. Hsu , Dynamic module deployment in a fog computing aware edge cloud service migration, in: 2016 IEEE International Conference on platform, in: 2016 18th Asia-Pacific Network Operations and Management Sympo- Cloud Computing Technology and Science (CloudCom), IEEE, 2016, pp. 344– sium (APNOMS), IEEE, 2016, pp. 1–6 . 351 . [260] J. Xu , B. Palanisamy , H. Ludwig , Q. Wang , Zenith: Utility-aware resource allocation [233] E. Yigitoglu , M. Mohamed , L. Liu , H. Ludwig , Foggy: a framework for continuous for edge computing, in: 2017 IEEE International Conference on Edge Computing automated IOT application deployment in fog computing, in: 2017 IEEE Interna- (EDGE), IEEE, 2017, pp. 47–54 . tional Conference on AI & Mobile Services (AIMS), IEEE, 2017, pp. 38–45 . [261] Y.-J. Yu , T.-C. Chiu , A.-C. Pang , M.-F. Chen , J. Liu , Virtual machine placement for [234] I. Farris , T. Taleb , M. Bagaa , H. Flick , Optimizing service replication for mobile backhaul traffic minimization in fog radio access networks, in: 2017 IEEE Interna- delay-sensitive applications in 5g edge network, in: 2017 IEEE International Con- tional Conference on Communications (ICC), IEEE, 2017, pp. 1–7 . ference on Communications (ICC), IEEE, 2017, pp. 1–6 . [262] D. Zeng , L. Gu , S. Guo , Z. Cheng , S. Yu , Joint optimization of task scheduling and [235] S. Abdelwahab , B. Hamdaoui , Flocking virtual machines in quest for responsive iot image placement in fog computing supported software-defined embedded system, cloud services, in: 2017 IEEE International Conference on Communications (ICC), IEEE Trans. Comput. 65 (12) (2016) 3702–3712 . IEEE, 2017, pp. 1–6 . [263] H.-J. Hong , P.-H. Tsai , A.-C. Cheng , M.Y.S. Uddin , N. Venkatasubramanian , [236] R. Morabito , I. Farris , A. Iera , T. Taleb , Evaluating performance of containerized C.-H. Hsu , Supporting internet-of-things analytics in a fog computing platform, in: iot services for clustered devices at the network edge, IEEE Internet Things J. 4 (4) 2017 IEEE International Conference on Cloud Computing Technology and Science (2017) 1019–1030 . (CloudCom)„ IEEE, 2017, pp. 138–145 . [237] B. Ottenwälder , B. Koldehofe , K. Rothermel , U. Ramachandran , Migcep: operator [264] A. Aral , T. Ovatman , A decentralized replica placement algorithm for edge com- migration for mobility driven distributed complex event processing, in: Proceedings puting, IEEE Trans. Netw. Serv. Manage. 15 (2) (2018) 516–529 . of the 7th ACM International Conference on Distributed Event-based Systems, ACM, [265] L. Zhao , J. Liu , Y. Shi , W. Sun , H. Guo , Optimal placement of virtual machines in 2013, pp. 183–194 . mobile edge computing, in: GLOBECOM 2017–2017 IEEE Global Communications [238] N.K. Giang , M. Blackstock , R. Lea , V.C. Leung , Developing iot applications in the Conference, IEEE, 2017, pp. 1–6 . fog: a distributed dataflow approach, in: 2015 5th International Conference on the [266] R.I. Tinini , L.C.M. Reis , D.M. Batista , G.B. Figueiredo , M. Tornatore , B. Mukher- Internet of Things (IOT), IEEE, 2015, pp. 155–162 . jee , Optimal placement of virtualized bbu processing in hybrid cloud-fog ran over [239] H. Gupta, S.B. Nath, S. Chakraborty, S.K. Ghosh, SDFog: A software defined com- twdm-pon, in: GLOBECOM 2017 - 2017 IEEE Global Communications Conference, puting architecture for QoS aware service orchestration over edge devices, 2016 . 2017, pp. 1–6 . arXiv preprint arXiv:1609.01190 . [267] N. Mohan , P. Zhou , K. Govindaraj , J. Kangasharju , Managing data in computational [240] A. Lertsinsrubtavee , A. Ali , C. Molina-Jimenez , A. Sathiaseelan , J. Crowcroft , Pi- edge clouds, in: Proceedings of the Workshop on Mobile Edge Communications, casso: a lightweight edge computing platform, in: 2017 IEEE 6th International Con- ACM, 2017, pp. 19–24 . ference on Cloud Networking (CloudNet), IEEE, 2017, pp. 1–7 . [268] H. Zhu , C. Huang , Availability-aware mobile edge application placement in 5g [241] T.G. Rodrigues , K. Suto , H. Nishiyama , N. Kato , A pso model with vm migration networks, in: GLOBECOM 2017 - 2017 IEEE Global Communications Conference, and transmission power control for low service delay in the multiple cloudlets ecc 2017, pp. 1–6 . scenario, in: 2017 IEEE International Conference on Communications (ICC), IEEE, [269] J. Liu , B. Bai , J. Zhang , K.B. Letaief , Cache placement in fog-rans: from centralized 2017, pp. 1–6 . to distributed algorithms, IEEE Trans. Wirel. Commun. 16 (11) (2017) 7039–7051 . [242] N. Wang, B. Varghese, M. Matthaiou, D.S. Nikolopoulos, Enorm: a frame- [270] H. Xiang , X. Xu , H. Zheng , S. Li , T. Wu , W. Dou , S. Yu , An adaptive cloudlet place- work for edge node resource management, IEEE Trans. Serv. Comput. (2017) ment method for mobile applications over gps big data, in: Global Communications https://ieeexplore.ieee.org/abstract/document/8039523 . Conference (GLOBECOM), 2016 IEEE, IEEE, 2016, pp. 1–6 . [243] W. Bao , D. Yuan , Z. Yang , S. Wang , W. Li , B.B. Zhou , A.Y. Zomaya , Follow me fog: [271] U. Drolia , K. Guo , J. Tan , R. Gandhi , P. Narasimhan , Cachier: Edge-caching for toward seamless handover timing schemes in a fog computing environment, IEEE recognition applications, in: 2017 IEEE 37th International Conference on Dis- Commun. Mag. 55 (11) (2017) 72–78 . tributed Computing Systems (ICDCS), IEEE, 2017, pp. 276–286 . [244] L. Ma , S. Yi , Q. Li , Efficient service handoff across edge servers via docker con- [272] K. Li , J. Nabrzyski , Traffic-aware virtual machine placement in cloudlet mesh with tainer migration, in: Proceedings of the Second ACM/IEEE Symposium on Edge adaptive bandwidth, in: 2017 IEEE International Conference on Cloud Computing Computing, ACM, 2017, p. 11 . Technology and Science (CloudCom), IEEE, 2017, pp. 49–56 . [245] V. Cozzolino , A.Y. Ding , J. Ott , Fades: Fine-grained edge offloading with uniker- [273] O. Ascigil , T.K. Phan , A.G. Tasiopoulos , V. Sourlas , I. Psaras , G. Pavlou , On uncoor- nels, in: Proceedings of the Workshop on Hot Topics in Container Networking and dinated service placement in edge-clouds, in: 2017 IEEE International Conference Networked Systems, ACM, 2017, pp. 36–41 . on Cloud Computing Technology and Science (CloudCom), IEEE, 2017, pp. 41–48 . [246] X. Sun , N. Ansari , Primal: Profit maximization avatar placement for mobile edge [274] T. Bahreini , D. Grosu , Efficient placement of multi-component applications in edge computing, in: 2016 IEEE International Conference on Communications (ICC), computing systems, in: Proceedings of the Second ACM/IEEE Symposium on Edge IEEE, 2016, pp. 1–6 . Computing, ACM, 2017, p. 5 . [247] W. Zhang , J. Chen , Y. Zhang , D. Raychaudhuri , Towards efficient edge cloud aug- [275] I. Lera , C. Guerrero , C. Juiz , Comparing centrality indices for network usage op- mentation for virtual reality mmogs, in: Proceedings of the Second ACM/IEEE Sym- timization of data placement policies in fog devices, in: 2018 Third International posium on Edge Computing, ACM, 2017, p. 8 . Conference on Fog and Mobile Edge Computing (FMEC), IEEE, 2018, pp. 115–122 . [248] I. Hou , T. Zhao , S. Wang , K. Chan , et al. , Asymptotically optimal algorithm for [276] S. Kastanakis , P. Sermpezis , V. Kotronis , X. Dimitropoulos , Cabaret: Leveraging rec- online reconfiguration of edge-clouds, in: Proceedings of the 17th ACM Interna- ommendation systems for mobile edge caching, in: Proceedings of the 2018 Work- tional Symposium on Mobile Ad Hoc Networking and Computing, ACM, 2016, shop on Mobile Edge Communications, in: MECOMM’18, ACM, 2018, pp. 19–24 . pp. 291–300 . [277] L. Wang , L. Jiao , T. He , J. Li , M. Mühlhäuser , Service entity placement for social [249] L. Chaufournier , P. Sharma , F. Le , E. Nahum , P. Shenoy , D. Towsley , Fast virtual reality applications in edge computing, in: INFOCOM 2018-IEEE Conference transparent virtual machine migration in distributed edge clouds, in: Proceedings on Computer Communications, IEEE, IEEE, 2018, pp. 1–9 . of the Second ACM/IEEE Symposium on Edge Computing, ACM, 2017, p. 10 . [278] Y.-H. Hung , C.-Y. Wang , R.-H. Hwang , Combinatorial clock auction for live video [250] K. Ha , Y. Abe , T. Eiszler , Z. Chen , W. Hu , B. Amos , R. Upadhyaya , P. Pillai , M. Satya- streaming in mobile edge computing, in: IEEE INFOCOM 2018-IEEE Conference on narayanan , You can teach elephants to dance: agile vm handoff for edge computing, Computer Communications Workshops (INFOCOM WKSHPS), IEEE, 2018 .

326 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

[279] N.B. Truong , G.M. Lee , Y. Ghamri-Doudane , Software defined networking-based [307] K. Desikan , M. Srinivasan , C. Murthy , A novel distributed latency-aware data vehicular adhoc network with fog computing, in: 2015 IFIP/IEEE International processing in fog computing-enabled iot networks, in: Proceedings of the ACM Symposium on Integrated Network Management (IM), IEEE, 2015, pp. 1202– Workshop on Distributed Information Processing in Wireless Networks, ACM, 2017, 1207 . p. 4 . [280] A. Hakiri , B. Sellami , P. Patil , P. Berthou , A. Gokhale , Managing wireless fog [308] L. Tang , S. He , Multi-user computation offloading in mobile edge computing: a networks using software-defined networking, in: 2017 IEEE/ACS 14th Inter- behavioral perspective, IEEE Netw. 32 (1) (2018) 48–53 . national Conference on Computer Systems and Applications (AICCSA), 2017, [309] R. Beraldi , A. Mtibaa , H. Alnuweiri , Cooperative load balancing scheme for edge pp. 1149–1156 . computing resources, in: 2017 Second International Conference on Fog and Mobile [281] Y. Xu , V. Mahendran , S. Radhakrishnan , Towards sdn-based fog computing: mqtt Edge Computing (FMEC), IEEE, 2017, pp. 94–100 . broker virtualization for effective and reliable delivery, in: 2016 8th International [310] A. Jonathan , A. Chandra , J. Weissman , Locality-aware load sharing in mobile cloud Conference on Communication Systems and Networks (COMSNETS), IEEE, 2016, computing, in: Proceedings of the10th International Conference on Utility and pp. 1–6 . Cloud Computing, in: UCC ’17, ACM, 2017, pp. 141–150 . [282] C. Li , Z. Qin , E. Novak , Q. Li , Securing sdn infrastructure of iot–fog networks from [311] A. Singh , N. Auluck , O. Rana , A. Jones , S. Nepal , Rt-sane: real time secu- mitm attacks, IEEE Internet Things J. 4 (5) (2017) 1156–1164 . rity aware scheduling on the network edge, in: Proceedings of the 10th In- [283] V. Sharma , I. You , F. Palmieri , D.N.K. Jayakody , J. Li , Secure and energy-efficient ternational Conference on Utility and Cloud Computing, ACM, 2017, pp. 131– handover in fog networks using blockchain-based dmm, IEEE Commun. Mag. 56 140 . (5) (2018) 22–31 . [312] C. Fricker , F. Guillemin , P. Robert , G. Thompson , Analysis of an offloading scheme [284] T. Rausch , S. Nastic , S. Dustdar , Emma: Distributed qos-aware mqtt middleware for data centers in the framework of fog computing, ACM Trans. Model. Perform. for edge computing applications, in: 2018 IEEE International Conference on Cloud Eval. Comput. Syst. (TOMPECS) 1 (4) (2016) 16 . Engineering (IC2E), IEEE, 2018, pp. 191–197 . [313] L. Chen , J. Xu , Socially trusted collaborative edge computing in ultra dense net- [285] Y. Bi , G. Han , C. Lin , Q. Deng , L. Guo , F. Li , Mobility support for fog computing: works, in: Proceedings of the Second ACM/IEEE Symposium on Edge Computing, an sdn approach, IEEE Commun. Mag. 56 (5) (2018) 53–59 . ACM, 2017, p. 9 . [286] M. Chen , Y. Hao , Task offloading for mobile edge computing in software de- [314] Y. Yu , X. Li , C. Qian , Sdlb: A scalable and dynamic software load balancer for fined ultra-dense network, IEEE J. Sel. Areas Commun. 36 (3) (2018) 587– fog and mobile edge computing, in: Proceedings of the Workshop on Mobile Edge 597 . Communications, ACM, 2017, pp. 55–60 . [287] N. Talagala , S. Sundararaman , V. Sridhar , D. Arteaga , Q. Luo , S. Subramanian , [315] X. Lyu , W. Ni , H. Tian , R.P. Liu , X. Wang , G.B. Giannakis , A. Paulraj , Optimal S. Ghanta , L. Khermosh , D. Roselli , ECO: harmonizing edge and cloud with ml/dl schedule of mobile edge computing for internet of things using partial information, orchestration, USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), IEEE J. Sel. Areas Commun. 35 (11) (2017) 2606–2615 . USENIX Association, Boston, MA, 2018 . [316] X. Lyu , C. Ren , W. Ni , H. Tian , R.P. Liu , Distributed optimization of collaborative [288] T. Zhao , S. Zhou , X. Guo , Z. Niu , Tasks scheduling and resource allocation in het- regions in large-scale inhomogeneous fog computing, IEEE J. Sel. Areas Commun. erogeneous cloud for delay-bounded mobile edge computing, in: 2017 IEEE Inter- 36 (3) (2018) 574–586 . national Conference on Communications (ICC), IEEE, 2017, pp. 1–7 . [317] W. Li , T. Yang , F.C. Delicato , P.F. Pires , Z. Tari , S.U. Khan , A.Y. Zomaya , On en- [289] X. Guo , R. Singh , T. Zhao , Z. Niu , An index based task assignment policy for achiev- abling sustainable edge computing with renewable energy resources, IEEE Com- ing optimal power-delay tradeoff in edge cloud systems, in: 2016 IEEE International mun. Mag. 56 (5) (2018) 94–101 . Conference on Communications (ICC), IEEE, 2016, pp. 1–7 . [318] Z. Hao , S. Yi , Q. Li , Edgecons: Achieving efficient consensus in edge computing net- [290] B. Li , Y. Pei , H. Wu , B. Shen , Heuristics to allocate high-performance cloudlets for works, USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), USENIX computation offloading in mobile ad hoc clouds, J. Supercomput. 71 (8) (2015) Association, Boston, MA, 2018 . 3009–3036 . [319] P. Nguyen , et al. , BRACELET: Hierarchical Edge-Cloud Microservice Infrastructure [291] L.F. Bittencourt , J. Diaz-Montes , R. Buyya , O.F. Rana , M. Parashar , Mobility- for Scientific Instruments’ Lifetime Connectivity, Technical Report, UIUC, 2018 . aware application scheduling in fog computing, IEEE Cloud Comput. 4 (2) (2017) [320] D. Deyannis , R. Tsirbas , G. Vasiliadis , R. Montella , S. Kosta , S. Ioannidis , Enabling 26–35 . gpu-assisted antivirus protection on android devices through edge offloading, in: [292] A. Kattepur , H.K. Rath , A. Simha , A-priori estimation of computation times in fog Proceedings of the 1st International Workshop on Edge Systems, Analytics and networked robotics, in: 2017 IEEE International Conference on Edge Computing Networking, ACM, 2018, pp. 13–18 . (EDGE), IEEE, 2017, pp. 9–16 . [321] H.-J. Jeong , I. Jeong , H.-J. Lee , S.-M. Moon , Computation offloading for machine [293] A. Kattepur , H. Dohare , V. Mushunuri , H.K. Rath , A. Simha , Resource constrained learning web apps in the edge server environment, in: 2018 IEEE 38th International offloading in fog computing, in: Proceedings of the 1st Workshop on Middleware Conference on Distributed Computing Systems (ICDCS), IEEE, 2018 . for Edge Clouds & Cloudlets, ACM, 2016, p. 1 . [322] J. Cho , K. Sundaresan , R. Mahindra , J. Van der Merwe , S. Rangarajan , Acacia: [294] Y. Nan , W. Li , W. Bao , F.C. Delicato , P.F. Pires , Y. Dou , A.Y. Zomaya , Adaptive context-aware edge computing for continuous interactive applications over mobile energy-aware computation offloading for cloud of things systems, IEEE Access 5 networks, in: Proceedings of the 12th International on Conference on emerging (2017) 23947–23957 . Networking EXperiments and Technologies, ACM, 2016, pp. 375–389 . [295] Y. Xiao , M. Krunz , Qoe and power efficiency tradeoff for fog computing networks [323] Y. Lin , H. Shen , Cloudfog: leveraging fog to extend cloud gaming for thin-client with fog node cooperation, in: INFOCOM 2017-IEEE Conference on Computer Com- mmog with high quality of service, IEEE Trans. Parallel Distrib. Syst. 28 (2) (2017) munications, IEEE, IEEE, 2017, pp. 1–9 . 431–445 . [296] G. Lee , W. Saad , M. Bennis , An online secretary framework for fog network forma- [324] D. Schäfer , J. Edinger , S. VanSyckel , J.M. Paluska , C. Becker , Tasklets: Overcoming tion with minimal latency, in: 2017 IEEE International Conference on Communi- heterogeneity in distributed computing systems, in: 2016 IEEE 36th International cations (ICC), IEEE, 2017, pp. 1–6 . Conference on Distributed Computing Systems Workshops (ICDCSW), IEEE, 2016, [297] X. Chen , J. Zhang , When d2d meets cloud: Hybrid mobile task offloadings in pp. 156–161 . fog computing, in: 2017 IEEE International Conference on Communications (ICC), [325] J. Gedeon , C. Meurisch , D. Bhat , M. Stein , L. Wang , M. Mühlhäuser , Router-based IEEE, 2017, pp. 1–6 . brokering for surrogate discovery in edge computing, in: 2017 IEEE 37th Interna- [298] B. Yin , W. Shen , Y. Cheng , L.X. Cai , Q. Li , Distributed resource sharing in fog-as- tional Conference on Distributed Computing Systems Workshops (ICDCSW), IEEE, sisted big data streaming, in: 2017 IEEE International Conference on Communica- 2017, pp. 145–150 . tions (ICC), IEEE, 2017, pp. 1–6 . [326] M. Amadeo , C. Campolo , A. Molinaro , C. Rottondi , G. Verticale , et al. , Securing the [299] W. Wang , W. Zhou , Computational offloading with delay and capacity constraints mobile edge through named data networking, in: 2018 IEEE 4th World Forum on in mobile edge, in: 2017 IEEE International Conference on Communications (ICC), Internet of Things (WF-IoT), 2018, pp. 80–85 . IEEE, 2017, pp. 1–6 . [327] G. Tanganelli , C. Vallati , E. Mingozzi , Edge-centric distributed discovery and access [300] Y. Sun , S. Zhou , J. Xu , Emm: energy-aware mobility management for mobile edge in the internet of things, IEEE Internet Things J. 5 (1) (2018) 425–438 . computing in ultra dense networks, IEEE J. Sel. Areas Commun. 35 (11) (2017) [328] K. Okada , S. Kashihara , N. Kawanishi , N. Suzuki , K. Sugiyama , Y. Kadobayashi , 2637–2646 . Goedge: A scalable and stateless local breakout method, in: Proceedings of the 2018 [301] N.T. Ti , L.B. Le , Computation offloading leveraging computing resources from edge Workshop on Theory and Practice for Integrated Cloud, Fog and Edge Computing cloud and mobile peers, in: 2017 IEEE International Conference on Communica- Paradigms, ACM, 2018, pp. 29–34 . tions (ICC), IEEE, 2017, pp. 1–6 . [329] R. Venanzi , B. Kantarci , L. Foschini , P. Bellavista , Mqtt-driven sustainable node [302] K. Zhang , Y. Mao , S. Leng , S. Maharjan , Y. Zhang , Optimal delay constrained of- discovery for internet of things-fog environments, in: 2018 IEEE International Con- floading for vehicular edge computing networks, in: 2017 IEEE International Con- ference on Communications (ICC), IEEE, 2018, pp. 1–6 . ference on Communications (ICC), IEEE, 2017, pp. 1–6 . [330] S. Li , M.A. Maddah-Ali , A.S. Avestimehr , Coding for distributed fog computing, [303] H. Tan , Z. Han , X.-Y. Li , F.C. Lau , Online job dispatching and scheduling in edge– IEEE Commun. Mag. 55 (4) (2017) 34–40 . clouds, in: INFOCOM 2017-IEEE Conference on Computer Communications, IEEE, [331] M. Malensek , S.L. Pallickara , S. Pallickara , Hermes: federating fog and cloud do- IEEE, 2017, pp. 1–9 . mains to support query evaluations in continuous sensing environments, IEEE [304] X. He , J. Liu , R. Jin , H. Dai , Privacy-aware offloading in mobile-edge computing, Cloud Comput. 4 (2) (2017) 54–62 . in: GLOBECOM 2017-2017 IEEE Global Communications Conference, IEEE, 2017, [332] S. Yang , Iot stream processing and analytics in the fog, IEEE Commun. Mag. 55 (8) pp. 1–6 . (2017) 21–27 . [305] Z. Chang , Z. Zhou , T. Ristaniemi , Z. Niu , Energy efficient optimization for compu- [333] P.P. Jayaraman , J.B. Gomes , H.L. Nguyen , Z.S. Abdallah , S. Krishnaswamy , A. Za- tation offloading in fog computing system, in: GLOBECOM 2017–2017 IEEE Global slavsky , Cardap: a scalable energy-efficient context aware distributed mobile data Communications Conference, IEEE, 2017, pp. 1–6 . analytics platform for the fog, in: East European Conference on Advances in [306] S. Yi , Z. Hao , Q. Zhang , Q. Zhang , W. Shi , Q. Li , Lavea: latency-aware video ana- Databases and Information Systems, Springer, 2014, pp. 192–206 . lytics on edge computing platform, in: Proceedings of the Second ACM/IEEE Sym- [334] J.K. Zao , T.T. Gan , C.K. You , S.J.R. Méndez , C.E. Chung , Y. Te Wang , T. Mullen , posium on Edge Computing, ACM, 2017, p. 15 . T.P. Jung , Augmented brain computer interaction based on fog computing and

327 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

linked data, in: 2014 International Conference on Intelligent Environments (IE), [363] M. Ali , A. Anjum , M.U. Yaseen , A.R. Zamani , D. Balouek-Thomert , O. Rana , IEEE, 2014, pp. 374–377 . M. Parashar , Edge enhanced deep learning system for large-scale video stream an- [335] G. Ananthanarayanan , P. Bahl , P. Bodík , K. Chintalapudi , M. Philipose , L. Ravin- alytics, in: 2018 IEEE 2nd International Conference on Fog and Edge Computing dranath , S. Sinha , Real-time video analytics: the killer app for edge computing, (ICFEC), IEEE, 2018, pp. 1–10 . Comput. (Long Beach Calif) 50 (10) (2017) 58–67 . [364] A. Abeshu , N. Chilamkurti , Deep learning: the frontier for distributed attack [336] B. Cheng , A. Papageorgiou , F. Cirillo , E. Kovacs , Geelytics: Geo-distributed edge detection in fog-to-things computing, IEEE Commun. Mag. 56 (2) (2018) 169– analytics for large scale iot systems based on dynamic topology, in: 2015 IEEE 2nd 175 . World Forum on Internet of Things (WF-IoT), IEEE, 2015, pp. 565–570 . [365] Y. Cao , H. Song , O. Kaiwartya , B. Zhou , Y. Zhuang , Y. Cao , X. Zhang , Mobile edge [337] P. Hu , H. Ning , T. Qiu , H. Song , Y. Wang , X. Yao , Security and privacy preservation computing for big-data-enabled electric vehicle charging, IEEE Commun. Mag. 56 scheme of face identification and resolution framework using fog computing in (3) (2018) 150–156 . internet of things, IEEE Internet Things J. 4 (5) (2017) 1143–1155 . [366] G. Muhammad , M.F. Alhamid , M. Alsulaiman , B. Gupta , Edge computing with [338] L. Lyu , J. Jin , S. Rajasegarar , X. He , M. Palaniswami , Fog-empowered anomaly cloud for voice disorder assessment and treatment, IEEE Commun. Mag. 56 (4) detection in iot using hyperellipsoidal clustering, IEEE Internet Things J. 4 (5) (2018) 60–65 . (2017) 1174–1184 . [367] M.H. Yaghmaee , A. Leon-Garcia , A fog-based internet of energy architecture for [339] Y. Huo , C. Hu , X. Qi , T. Jing , Lodpd: a location difference-based proximity detection transactive energy management systems, IEEE Internet Things J. (2018) . protocol for fog computing, IEEE Internet Things J. 4 (5) (2017) 1117–1124 . [368] S. Ahn , M. Gorlatova , P. Naghizadeh , M. Chiang , P. Mittal , Adaptive fog-based out- [340] I. Azimi , A. Anzanpour , A.M. Rahmani , T. Pahikkala , M. Levorato , P. Liljeberg , put security for augmented reality, in: Proceedings of the 2018 Morning Workshop N. Dutt , Hich: hierarchical fog-assisted computing architecture for healthcare iot, on Virtual Reality and Augmented Reality Network, ACM, 2018, pp. 1–6 . ACM Trans. Embedded Comput. Syst. (TECS) 16 (5s) (2017) 174 . [369] S. Wang , T. Tuor , T. Salonidis , K.K. Leung , C. Makaya , T. He , K. Chan , When [341] W. Shen , J. Yu , H. Xia , H. Zhang , X. Lu , R. Hao , Light-weight and privacy-preserving edge meets learning: adaptive control for resource-constrained distributed ma- secure cloud auditing scheme for group users via the third party medium, J. Netw. chine learning, in: INFOCOM 2018-IEEE Conference on Computer Communica- Comput. Appl. 82 (2017) 56–64 . tions, IEEE, IEEE, 2018 . [342] P. Hu , H. Ning , T. Qiu , Y. Zhang , X. Luo , Fog computing based face identification [370] L. Corneo , P. Gunningberg , Scheduling at the edge for assisting cloud real-time sys- and resolution scheme in internet of things, IEEE Trans. Ind. Inf. 13 (4) (2017) tems, in: Proceedings of the 2018 Workshop on Theory and Practice for Integrated 1910–1920 . Cloud, Fog and Edge Computing Paradigms, ACM, 2018, pp. 9–14 . [343] Y. Nikoloudakis , S. Panagiotakis , E. Markakis , E. Pallis , G. Mastorakis , C.X. Mavro- [371] K. Bhardwaj , J.C. Miranda , A. Gavrilovska , Towards iot-ddos prevention using edge moustakis , C. Dobre , A fog-based emergency system for smart enhanced living en- computing, USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), vironments, IEEE Cloud Comput. 3 (6) (2016) 54–62 . USENIX Association, Boston, MA, 2018 . [344] S. Teerapittayanon , B. McDanel , H. Kung , Distributed deep neural networks over [372] C. Ge , N. Wang , Real-time qoe estimation of dash-based mobile video applications the cloud, the edge and end devices, in: 2017 IEEE 37th International Conference through edge computing, in: IEEE INFOCOM 2018-IEEE Conference on Computer on Distributed Computing Systems (ICDCS), IEEE, 2017, pp. 328–339 . Communications Workshops (INFOCOM WKSHPS), IEEE, 2018 . [345] G. Kamath , P. Agnihotri , M. Valero , K. Sarker , W.-Z. Song , Pushing analytics to [373] Y. Mao , S. Yi , Q. Li , J. Feng , F. Xu , S. Zhong , A privacy-preserving deep learning the edge, in: Global Communications Conference (GLOBECOM), 2016 IEEE, IEEE, approach for face recognition with edge computing, USENIX Workshop on Hot 2016, pp. 1–6 . Topics in Edge Computing (HotEdge 18), USENIX Association, Boston, MA, 2018 . [346] H.-J. Hong , J.-C. Chuang , C.-H. Hsu , Animation rendering on multimedia fog com- [374] E. Li , Z. Zhou , X. Chen , Edge intelligence: On-demand deep learning model co-in- puting platforms, in: 2016 IEEE International Conference on Cloud Computing ference with device-edge synergy, in: Proceedings of the 2018 Workshop on Mobile Technology and Science (CloudCom), IEEE, 2016, pp. 336–343 . Edge Communications, in: MECOMM’18, ACM, 2018, pp. 31–36 . [347] A. Dhakal , K. Ramakrishnan , Machine learning at the network edge for automated [375] A.R. Elias , N. Golubovic , C. Krintz , R. Wolski , Where’s the bear?-automating home intrusion monitoring, in: 2017 IEEE 25th International Conference on Net- wildlife image processing using iot and edge cloud systems, in: 2017 IEEE/ACM work Protocols (ICNP), IEEE, 2017, pp. 1–6 . Second International Conference on Internet-of-Things Design and Implementation [348] S. Mangiante , G. Klas , A. Navon , Z. GuanHua , J. Ran , M.D. Silva , Vr is on the edge: (IoTDI), IEEE, 2017, pp. 247–258 . how to deliver 360 videos in mobile networks, in: Proceedings of the Workshop on [376] X. Ran , H. Chen , X. Zhu , Z. Liu , J. Chen , Deepdecision: a mobile deep learning Virtual Reality and Augmented Reality Network, ACM, 2017, pp. 30–35 . framework for edge video analytics, in: INFOCOM 2018-IEEE Conference on Com- [349] J. Ni , X. Lin , K. Zhang , Y. Yu , Secure and deduplicated spatial crowdsourcing: A puter Communications, IEEE, IEEE, 2018, pp. 1–9 . fog-based approach, in: Global Communications Conference (GLOBECOM), 2016 [377] Q. Liu , S. Huang , J. Opadere , T. Han , An edge network orchestrator for mobile IEEE, IEEE, 2016, pp. 1–6 . augmented reality, in: INFOCOM 2018-IEEE Conference on Computer Communi- [350] D. Li , T. Salonidis , N.V. Desai , M.C. Chuah , Deepcham: collaborative edge-mediated cations, IEEE, IEEE, 2018, pp. 1–9 . adaptive deep learning for mobile object recognition, in: IEEE/ACM Symposium on [378] Z. Tao , Q. Li , esgd: Communication efficient distributed deep learning on the edge, Edge Computing (SEC), IEEE, 2016, pp. 64–76 . USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), USENIX Asso- [351] J. Yoon , P. Liu , S. Banerjee , Low-cost video transcoding at the wireless edge, in: ciation, Boston, MA, 2018 . IEEE/ACM Symposium on Edge Computing (SEC), IEEE, 2016, pp. 129–141 . [379] Z. Xu , H. Gupta , U. Ramachandran , Sttr: A system for tracking all vehicles all [352] K. Bhardwaj , P. Agrawal , A. Gavrilowska , K. Schwan , A. Allred , Appflux: taming the time at the edge of the network, in: Proceedings of the 12th ACM Interna- app delivery via streaming, in: Proceedings of the Usenix TRIOS, 2015 . tional Conference on Distributed and Event-based Systems, ACM, 2018, pp. 124– [353] M. Jang , H. Lee , K. Schwan , K. Bhardwaj , Soul: an edge-cloud system for mobile 135 . applications in a sensor-rich world, in: IEEE/ACM Symposium on Edge Computing [380] H. Gupta , A. Vahid Dastjerdi , S.K. Ghosh , R. Buyya , Ifogsim: a toolkit for model- (SEC), IEEE, 2016, pp. 155–167 . ing and simulation of resource management techniques in the internet of things, [354] B. Qi , L. Kang , S. Banerjee , A vehicle-based edge computing platform for transit and edge and fog computing environments, Softw.: Pract. Exp. 47 (9) (2017) human mobility analytics, in: Proceedings of the Second ACM/IEEE Symposium on 1275–1296 . Edge Computing, ACM, 2017, p. 1 . [381] D. Bruneo , S. Distefano , F. Longo , G. Merlino , A. Puliafito , V. D’Amico , M. Sapienza , [355] H.P. Sajjad , K. Danniswara , A. Al-Shishtawy , V. Vlassov , Spanedge: towards unify- G. Torrisi , Stack4things as a fog computing platform for smart city applications, ing stream processing over central and near-the-edge data centers, in: IEEE/ACM in: 2016 IEEE Conference on Computer Communications Workshops (INFOCOM Symposium on Edge Computing (SEC), IEEE, 2016, pp. 168–178 . WKSHPS), IEEE, 2016, pp. 848–853 . [356] K. Lee , J. Flinn , B.D. Noble , Gremlin: scheduling interactions in vehicular com- [382] X. Xu , S. Huang , L. Feagan , Y. Chen , Y. Qiu , Y. Wang , Eaaas: Edge analytics as puting, in: Proceedings of the Second ACM/IEEE Symposium on Edge Computing, a service, in: 2017 IEEE International Conference on Web Services (ICWS), IEEE, ACM, 2017, p. 4 . 2017, pp. 349–356 . [357] B. Mei , R. Li , W. Cheng , J. Yu , X. Cheng , Ultraviolet radiation measurement via [383] P. Liu , L. Hartung , S. Banerjee , Lightweight multitenancy at the net- smart devices, IEEE Internet Things J. 4 (4) (2017) 934–944 . work’s extreme edge, Comput. (Long Beach Calif) 50 (10) (2017) [358] U. Drolia , K. Guo , P. Narasimhan , Precog: prefetching for image recognition appli- 50–57 . cations at the edge, in: Proceedings of the Second ACM/IEEE Symposium on Edge [384] M.A. Al Faruque , K. Vatanparvar , Energy management-as-a-service over fog com- Computing, ACM, 2017, p. 17 . puting platform, IEEE Internet Things J. 3 (2) (2016) 161–169 . [359] T.N. Gia , M. Jiang , A.-M. Rahmani , T. Westerlund , P. Liljeberg , H. Tenhunen , Fog [385] M.M. Lopes , W.A. Higashino , M.A. Capretz , L.F. Bittencourt , Myifogsim: a simulator computing in healthcare internet of things: a case study on ecg feature extraction, for virtual machine migration in fog computing, in: Companion Proceedings of in: 2015 IEEE International Conference on Computer and Information Technol- the10th International Conference on Utility and Cloud Computing, ACM, 2017, ogy; Ubiquitous Computing and Communications; Dependable, Autonomic and Se- pp. 47–52 . cure Computing; Pervasive Intelligence and Computing (CIT/IUCC/DASC/PICOM), [386] B. Confais , A. Lebre , B. Parrein , An object store service for a fog/edge computing IEEE, 2015, pp. 356–363 . infrastructure based on ipfs and a scale-out nas, in: 2017 IEEE 1st International [360] N. Chen , Y. Chen , Y. You , H. Ling , P. Liang , R. Zimmermann , Dynamic ur- Conference on Fog and Edge Computing (ICFEC), IEEE, 2017, pp. 41–50 . ban surveillance video stream processing using fog computing, in: 2016 IEEE [387] A.M. Khan , F. Freitag , On edge cloud service provision with distributed home Second International Conference on Multimedia Big Data (BigMM), IEEE, 2016, servers, in: 2017 IEEE International Conference on Cloud Computing Technology pp. 105–112 . and Science (CloudCom), IEEE, 2017, pp. 223–226 . [361] G. Kar , S. Jain , M. Gruteser , F. Bai , R. Govindan , Real-time traffic estimation at [388] C. Sonmez , A. Ozgovde , C. Ersoy , Edgecloudsim: an environment for performance vehicular edge nodes, in: Proceedings of the Second ACM/IEEE Symposium on evaluation of edge computing systems, in: 2017 Second International Conference Edge Computing, ACM, 2017, p. 3 . on Fog and Mobile Edge Computing (FMEC), IEEE, 2017, pp. 39–44 . [362] I. Lujic , V. De Maio , I. Brandic , Adaptive recovery of incomplete datasets for edge [389] S. Echeverría , D. Klinedinst , K. Williams , G.A. Lewis , Establishing trusted identities analytics, in: 2018 IEEE 2nd International Conference on Fog and Edge Computing in disconnected edge environments, in: IEEE/ACM Symposium on Edge Computing (ICFEC), IEEE, 2018, pp. 1–10 . (SEC), IEEE, 2016, pp. 51–63 .

328 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

[390] M.I. Naas , J. Boukhobza , P.R. Parvedy , L. Lemarchand , An extension to ifogsim [417] O. Akrivopoulos , N. Zhu , D. Amaxilatis , C. Tselios , A. Anagnostopoulos , I. Chatzi- to enable the design of data placement strategies, in: 2018 IEEE 2nd International giannakis , A fog computing-oriented, highly scalable iot framework for monitoring Conference on Fog and Edge Computing (ICFEC), IEEE, 2018, pp. 1–8 . public educational buildings, in: 2018 IEEE International Conference on Commu- [391] A. Rafetseder , L. Pühringer , J. Cappos , Practical fog computing with seattle, in: Fog nications (ICC), IEEE, 2018, pp. 1–6 . World Congress (FWC), 2017 IEEE, IEEE, 2017, pp. 1–7 . [418] P. Skarin , W. Tärneberg , K.-E. Årzen , M. Kihl , Towards mission-critical control at [392] R. Mayer , L. Graser , H. Gupta , E. Saurez , U. Ramachandran , Emufog: Extensible the edge and over 5g, in: 2018 IEEE International Conference on Edge Computing and scalable emulation of large-scale fog computing infrastructures, in: 2017 IEEE (EDGE), IEEE, 2018, pp. 1–8 . Fog World Congress (FWC), 2017, pp. 1–6 . [419] C. Esposito , A. Castiglione , F. Pop , K.-K.R. Choo , Challenges of connecting edge [393] M. Körner , T.M. Runge , A. Panda , S. Ratnasamy , S. Shenker , Open carrier interface: and cloud computing: a security and forensic perspective, IEEE Cloud Comput. 4 an open source edge computing framework, in: Proceedings of the 2018 Work- (2) (2017) 13–17 . shop on Networking for Emerging Applications and Technologies, ACM, 2018, [420] Z. Yu , M.H. Au , Q. Xu , R. Yang , J. Han , Towards leakage-resilient fine- pp. 27–32 . grained access control in fog computing, Future Gener. Comput. Syst. 78 (2018) [394] N.K. Giang , R. Lea , M. Blackstock , V. Leung , Fog at the edge: Experiences build- 763–777 . ing an edge computing platform, in: 2018 IEEE International Conference on Edge [421] S. Basudan , X. Lin , K. Sankaranarayanan , A privacy-preserving vehicular crowd- Computing (EDGE), IEEE, 2018, pp. 1–9 . sensing-based road surface condition monitoring system using fog computing, IEEE [395] A. Coutinho , F. Greve , C. Prazeres , J. Cardoso , Fogbed: a rapid-prototyping emu- Internet Things J. 4 (3) (2017) 772–782 . lation environment for fog computing, in: 2018 IEEE International Conference on [422] A. Alrawais , A. Alhothaily , C. Hu , X. Cheng , Fog computing for the internet of Communications (ICC), IEEE, 2018, pp. 1–7 . things: security and privacy issues, IEEE Internet Comput. 21 (2) (2017) 34–42 . [396] P. Liu , B. Qi , S. Banerjee , Edgeeye: an edge service framework for real-time intel- [423] S. Basudan , X. Lin , K. Sankaranarayanan , An efficient compromised node revoca- ligent video analytics, in: Proceedings of the 1st International Workshop on Edge tion scheme in fog-assisted vehicular crowdsensing, in: GLOBECOM 2017–2017 Systems, Analytics and Networking, ACM, 2018, pp. 1–6 . IEEE Global Communications Conference, IEEE, 2017, pp. 1–6 . [397] C. Streiffer , A. Srivastava , V. Orlikowski , Y. Velasco , V. Martin , N. Raval , [424] R. Lu , K. Heung , A.H. Lashkari , A.A. Ghorbani , A lightweight privacy-preserving A. Machanavajjhala , L.P. Cox , eprivateeye: to the edge and beyond!, in: Proceed- data aggregation scheme for fog computing-enhanced iot, IEEE Access 5 (2017) ings of the Second ACM/IEEE Symposium on Edge Computing, ACM, 2017, p. 18 . 3302–3312 . [398] Y. Elkhatib , B. Porter , H.B. Ribeiro , M.F. Zhani , J. Qadir , E. Rivière , On using mi- [425] Q. Wang , D. Chen , N. Zhang , Z. Ding , Z. Qin , Pcp: a privacy-preserving con- cro-clouds to deliver the fog, IEEE Internet Comput 21 (2) (2017) 8–15 . tent-based publish–subscribe scheme with differential privacy in fog computing, [399] R. Muñoz , L. Nadal , R. Casellas , M.S. Moreolo , R. Vilalta , J.M. Fàbrega , R. Martínez , IEEE Access 5 (2017) 17962–17974 . A. Mayoral , F.J. Vílchez , The adrenaline testbed: an sdn/nfv packet/optical trans- [426] X. An , X. Zhou , X. Lü, F. Lin , L. Yang , Sample selected extreme learning machine port network and edge/core cloud platform for end-to-end 5g and iot services, based intrusion detection in fog computing and mec, Wirel. Commun. Mobile Com- in: 2017 European Conference on Networks and Communications (EuCNC), IEEE, put. 2018 (2018) . 2017, pp. 1–5 . [427] D. Giri , M.S. Obaidat , T. Maitra , Sechealth: an efficient fog based sender initiated [400] N. Iotti , M. Picone , S. Cirani , G. Ferrari , Improving quality of experience in future secure data transmission of healthcare sensors for e-medical system, in: GLOBE- wireless access networks through fog computing, IEEE Internet Comput. 21 (2) COM 2017–2017 IEEE Global Communications Conference, IEEE, 2017, pp. 1–6 . (2017) 26–33 . [428] T. He , E.N. Ciftcioglu , S. Wang , K.S. Chan , Location privacy in mobile edge [401] C. Pahl , S. Helmer , L. Miori , J. Sanin , B. Lee , A container-based edge cloud paas clouds: a chaff-based approach, IEEE J. Sel. Areas Commun. 35 (11) (2017) 2625– architecture based on raspberry pi clusters, in: IEEE International Conference on Fu- 2636 . ture Internet of Things and Cloud Workshops (FiCloudW), IEEE, 2016, pp. 117–124 . [429] K. Sha , R. Errabelly , W. Wei , T.A. Yang , Z. Wang , Edgesec: design of an edge layer [402] R. Abdallah , L. Xu , W. Shi , Lessons and experiences of a diy smart home, in: Pro- security service to enhance iot security, in: 2017 IEEE 1st International Conference ceedings of the Workshop on Smart Internet of Things, ACM, 2017, p. 4 . on Fog and Edge Computing (ICFEC), IEEE, 2017, pp. 81–88 . [403] S. Hoque , M.S. de Brito , A. Willner , O. Keil , T. Magedanz , Towards container or- [430] S. Shen , L. Huang , H. Zhou , S. Yu , E. Fan , Q. Cao , Multistage signaling game-based chestration in fog computing infrastructures, in: 2017 IEEE 41st Annual Computer optimal detection strategies for suppressing malware diffusion in fog-cloud-based Software and Applications Conference (COMPSAC), 2, IEEE, 2017, pp. 294–299 . iot networks, IEEE Internet Things J. 5 (2) (2018) 1043–1054 . [404] B.I. Ismail , E.M. Goortani , M.B. Ab Karim , W.M. Tat , S. Setapa , J.Y. Luke , O.H. Hoe , [431] D. He , Y. Qiao , S. Chan , N. Guizani , Flight security and safety of drones in airborne Evaluation of docker as edge computing platform, in: 2015 IEEE Confernece on fog computing systems, IEEE Commun. Mag. 56 (5) (2018) 66–71 . Open Systems (ICOS), IEEE, 2015, pp. 130–135 . [432] N.I. Yekta , R. Lu , Xrquery: achieving communication-efficient privacy-preserving [405] D. Santoro , D. Zozin , D. Pizzolli , F. De Pellegrini , S. Cretti , Foggy: a platform for query for fog-enhanced iot, in: 2018 IEEE International Conference on Communi- workload orchestration in a fog computing environment, in: 2017 IEEE Interna- cations (ICC), IEEE, 2018, pp. 1–6 . tional Conference on Cloud Computing Technology and Science (CloudCom), IEEE, [433] Y. Kim , D. Kim , J. Son , W. Wang , Y. Noh , A new fog-cloud storage framework with 2017, pp. 231–234 . transparency and auditability, in: 2018 IEEE International Conference on Commu- [406] L. Baresi , D.F. Mendonça , M. Garriga , Empowering low-latency applications nications (ICC), IEEE, 2018, pp. 1–7 . through a serverless edge computing architecture, in: European Conference on Ser- [434] C.-H. Chen , M.-Y. Lin , C.-C. Liu , Edge computing gateway of the industrial internet vice-Oriented and Cloud Computing, Springer, 2017, pp. 196–210 . of things using multiple collaborative microcontrollers, IEEE Netw 32 (1) (2018) [407] A. Gosain , M. Berman , M. Brinn , T. Mitchell , C. Li , Y. Wang , H. Jin , J. Hua , 24–32 . H. Zhang , Enabling campus edge computing using geni racks and mobile resources, [435] S. Biookaghazadeh , M. Zhao , F. Ren , Are fpgas suitable for edge computing? in: IEEE/ACM Symposium on Edge Computing (SEC), IEEE, 2016, pp. 41–50 . USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), USENIX Asso- [408] S. Helmer , C. Pahl , J. Sanin , L. Miori , S. Brocanelli , F. Cardano , D. Gadler , D. Moran- ciation, Boston, MA, 2018 . dini , A. Piccoli , S. Salam , et al. , Bringing the cloud to rural and remote areas via [436] D.P. Anderson , Boinc: a system for public-resource computing and storage, in: pro- cloudlets, in: Proceedings of the 7th Annual Symposium on Computing for Devel- ceedings of the 5th IEEE/ACM International Workshop on Grid Computing, IEEE opment, ACM, 2016, p. 14 . Computer Society, 2004, pp. 4–10 . [409] I. Had ž i ć, Y. Abe , H.C. Woithe , Edge computing in the epc: a reality check, in: [437] D.P. Anderson , J. Cobb , E. Korpela , M. Lebofsky , D. Werthimer , Seti@ home: an Proceedings of the Second ACM/IEEE Symposium on Edge Computing, ACM, 2017, experiment in public-resource computing, Commun. ACM 45 (11) (2002) 56–61 . p. 13 . [438] J. Kong , I. Kim , X. Wang , Q. Zhang , H.C. Cankaya , W. Xie , T. Ikeuchi , J.P. Jue , [410] Z. Chen , W. Hu , J. Wang , S. Zhao , B. Amos , G. Wu , K. Ha , K. Elgazzar , P. Pillai , Guaranteed-availability network function virtualization with network protection R. Klatzky , D. Siewiorek , M. Satyanarayanan , An empirical study of latency in an and vnf replication, in: GLOBECOM 2017–2017 IEEE Global Communications Con- emerging class of edge computing applications for wearable cognitive assistance, ference, IEEE, 2017, pp. 1–6 . in: Proceedings of the Second ACM/IEEE Symposium on Edge Computing, in: SEC [439] A.T. Nasrabadi , A. Mahzari , J.D. Beshay , R. Prakash , Adaptive 360-degree video ’17, ACM, 2017, pp. 14:1–14:14 . streaming using scalable video coding, in: Proceedings of the 2017 ACM on Multi- [411] A. Buzachis , A. Galletta , L. Carnevale , A. Celesti , M. Fazio , M. Villari , Towards os- media Conference, ACM, 2017, pp. 1689–1697 . motic computing: analyzing overlay network solutions to optimize the deployment [440] A. Mahzari , A.T. Nasrabadi , A. Samiei , R. Prakash , Fov-aware edge caching for of container-based microservices in fog, edge and iot environments, in: 2018 IEEE adaptive 360 video streaming, in: Proceedings of the 2018 ACM on Multimedia 2nd International Conference on Fog and Edge Computing (ICFEC), IEEE, 2018, Conference, ACM, 2018 . pp. 1–10 . [441] B. Mirkhanzadeh , A. Ferrari , Z. Lu , A. Shakeri , C. Shao , M. Tacca , M. Razo , [412] C. Bailas , M. Marsden , D. Zhang , N.E. O’Connor , S. Little , Performance of video M. Cantono , V. Curri , G. Martinelli , et al. , Two-layer network solution for reliable processing at the edge for crowd-monitoring applications, in: 2018 IEEE 4th World and efficient host-to-host transfer of big data, in: Photonic Networks and Devices, Forum on Internet of Things (WF-IoT), 2018, pp. 482–487 . Optical Society of America, 2018, pp. NeTh2F–5 . [413] J.N. Plumb , R. Stutsman , Exploiting google’s edge network for massively multi- [442] H. Moeini , I.-L. Yen , F. Bastani , Routing in iot network for dynamic service dis- player online games, in: 2018 IEEE 2nd International Conference on Fog and Edge covery, in: 2017 IEEE 23rd International Conference on Parallel and Distributed Computing (ICFEC), IEEE, 2018, pp. 1–8 . Systems (ICPADS), IEEE, 2017, pp. 360–367 . [414] A. Bavier , M. Berman , M. Brinn , R. McGeer , L. Peterson , G. Ricart , Realizing the [443] R. Morabito , V. Cozzolino , A.Y. Ding , N. Beijar , J. Ott , Consolidate iot edge com- global edge cloud, IEEE Commun. Mag. 56 (5) (2018) 170–176 . puting with lightweight virtualization, IEEE Netw. 32 (1) (2018) 102–111 . [415] A. Carrega , M. Repetto , G. Robino , G. Portomauro , Openstack extensions for qos [444] Intel, Intel’s fog reference design overview. [Online]. Available: https:// and energy efficiency in edge computing, in: 2018 Third International Conference www.intel.com/content/www/us/en/internet-of-things/fog-reference-design- on Fog and Mobile Edge Computing (FMEC), IEEE, 2018, pp. 50–57 . overview.html . [416] X. Zhang , Y. Wang , W. Shi , pcamp: performance comparison of machine learning [445] B.A. Martin , F. Michaud , D. Banks , A. Mosenia , R. Zolfonoon , S. Irwan , S. Schrecker , packages on the edges, USENIX Workshop on Hot Topics in Edge Computing (Hot- J.K. Zao , Openfog security requirements and approaches, in: Fog World Congress Edge 18), USENIX Association, Boston, MA, 2018 . (FWC), 2017 IEEE, IEEE, 2017, pp. 1–6 .

329 A. Yousefpour, C. Fung and T. Nguyen et al. Journal of Systems Architecture 98 (2019) 289–330

[446] A. Darki , A. Duﬀ, Z. Qian , G. Naik , S. Mancoridis , M. Faloutsos , (poster) don’t Fatemeh Jalali joined IBM Research as a research scientist

trust your router: detecting compromised router, in: IEEE Proceedings of the 12th in 2015 after ﬁnishing her Ph.D. from The University of Mel-

International Conference on Emerging Networking Experiments and Technologies bourne, Australia. Her main research interests include Inter-

CoNEXT, 16, 2016 . net of Things, Fog/Edge computing, and machine learning,

[447] Z.J. Haas , A. Yousefpour , A Privacy Scheme for Monitoring Devices in the Internet in which she has authored a number of research papers and

of Things, in: Pervasive Computing Paradigms for Mental Health, Springer, 2016, patents. Fatemeh’s technical activities and STEM (Science,

pp. 153–165 . Technology, Engineering, and Math) promotion activities are

[448] J.H. Jafarian , A. Niakanlahiji , E. Al-Shaer , Q. Duan , Multi-dimensional host identity recognized by various professional organizations. anonymization for defeating skilled attackers, in: Proceedings of the 2016 ACM Workshop on Moving Target Defense, ACM, 2016, pp. 47–58 . [449] A.V. Dastjerdi , R. Buyya , Fog computing: helping the internet of things realize its potential, Comput. (Long Beach Calif) 49 (8) (2016) 112–116 . [450] D. Kimovski , H. Ijaz , N. Saurabh , R. Prodan , Adaptive nature-inspired fog architecture, in: 2018 IEEE 2nd International Conference on Fog and Edge Computing (ICFEC), IEEE, 2018, pp. 1–8 .

Ashkan Yousefpour is a Visiting Researcher at the University Amirreza Niakanlahiji is a Ph.D. student and a Provost’s Doc-

of California, Berkeley and a Ph.D. candidate at the Erik Jon- toral Teaching fellow in the College of Computing & Informat-

sson School of Engineering and Computer Science, the Uni- ics at the University of North Carolina at Charlotte, USA. He

versity of Texas at Dallas, USA. His research interests are in is interested in a broad range of security topics from network

the areas of fog computing, cloud computing, Deep Learning, security to web security. Currently, his focus is on data-driven

Internet of Things, and security. He obtained his B.S. in Com- security. He obtained his M.Sc. in Information Security from

puter Engineering from Amirkabir University of Technology, Amirkabir University of Technology, Iran, in 2011. Iran, in 2013, and his M.S. in Computer Science from the Uni- versity of Texas at Dallas, USA, in 2016.

Jian Kong is currently working in Ciena, Austin, USA. He Caleb Fung is an undergraduate student in Computer Science earned his Ph.D. in Telecommunications Engineering from the department at the University of Texas at Dallas, USA. His re- University of Texas at Dallas, in 2018. His research interests search interests include fog computing, graph theory, machine are in the areas of network virtualization, optical network and learning, software-deﬁned networking, and cloud computing. fog/cloud computing. He obtained his B.S. in Automation from Shandong University, China, in 2007, and his M.S. in Con- trol theory and Control Engineering from Beihang University, China, in 2010.

Jason P. Jue received the B.S. degree in Electrical Engineer-

Tam Nguyen is pursuing his master’s in Computer Science at ing and Computer Science from the University of California,

the Georgia Institute of Technology, USA. He earned his un- Berkeley in 1990, the M.S. degree in Electrical Engineering

dergraduate degree in Computer Engineering at the University from the University of California, Los Angeles in 1991, and

of Texas at Dallas, USA. His research interests include fog com- the Ph.D. degree in Computer Engineering from the Univer-

puting, Internet of Things, mobile development, and robotics. sity of California, Davis in 1999. He is currently a Professor in the Department of Computer Science at the University of Texas at Dallas. His current research interests include optical networks and network survivability.

Krishna Kadiyala is a Telecommunications Engineering Ph.D. candidate at the University of Texas at Dallas, USA. Before starting her Ph.D., she worked as a Network Engineer at AT&T Labs, USA. Her research interests are broadly in the areas of networking including SDN, mobile computing, and IoT. She earned her M.S. in Computer Engineering from the University of North Texas, USA.

330