Journal of Sensor and Actuator Networks

Article End-to-End Performance Evaluation of MEC Deployments in Scenarios

Antonio Virdis 1,* , Giovanni Nardini 1 , Giovanni Stea 1 and Dario Sabella 2 1 Dipartimento di Ingegneria dell’Informazione, University of Pisa, 56021 Pisa, Italy; [email protected] (G.N.); [email protected] (G.S.) 2 Intel Deutschland GmbH, Am Campeon 10-12, 85579 Neubiberg, Germany; [email protected] * Correspondence: [email protected]

 Received: 22 October 2020; Accepted: 8 December 2020; Published: 11 December 2020 

Abstract: Multi-access (MEC) promises to deliver localized computing power and storage. Coupled with low-latency 5G radio access, this enables the creation of high added-value services for mobile users, such as in-vehicle infotainment or remote driving. The performance of these services as well as their scalability will however depend on how MEC will be deployed in 5G systems. This paper evaluates different MEC deployment options, coherent with the respective 5G migration phases, using an accurate and comprehensive end-to-end (E2E) system simulation model (exploiting Simu5G for radio access and Intel CoFluent for core network and MEC), taking into account user-related metrics, such as response time or MEC latency. Our results show that 4G radio access is going to be a bottleneck, preventing MEC services from scaling up. On the other hand, the introduction of 5G will allow a considerable higher penetration of MEC services.

Keywords: MEC; new radio; Simu5G; CoFluent; performance evaluation

1. Introduction Edge computing has recently emerged as a promising evolution of . Its benefits include better scalability, especially when large amounts of data are involved (e.g., acquired from sensors), as well as lower latencies, which enable time-critical services, e.g., remote sensing and actuation or distributed gaming, and the preservation of privacy and context information (including that related to user location and access interface). Many standardization efforts and industry groups gravitate around edge computing [1]. In particular, Multi-access Edge Computing (MEC) is the leading international standard introduced by ETSI ISG (Industry Standard Group) MEC to define an architecture and services for edge computing [2,3], with particular (though not exclusive) regard to cellular network access. This standard includes both an MEC framework and architecture [4] and a set of MEC APIs specified following general RESTful API design principles [5]. According to the definition [2], “multi-access edge computing offers application developers and content providers cloud-computing capabilities and an IT service environment at the edge of the network”. This environment is characterized by ultra-low latency and high bandwidth as well as real-time access to radio network information that can be leveraged by applications. MEC provides a new ecosystem and value chain. This includes mobile operators, application developers, over the top players, independent software vendors, telecom equipment & IT platform vendors, system integrators, and technology providers, all of whom are interested in delivering MEC-based services. While MEC is access-agnostic, it is foreseeable that users will conveniently access MEC through cellular networks, i.e., the existing 4G LTE-advanced [6] and mostly the new 5G network [1,7]. Moreover, 5G introduces several innovations, ranging from access to core network side. In particular, 5G New Radio (NR) brings considerable improvements with respect to 4G: access latency is reduced, thanks to

J. Sens. Actuator Netw. 2020, 9, 57; doi:10.3390/jsan9040057 www.mdpi.com/journal/jsan J. Sens. Actuator Netw. 2020, 9, 57 2 of 14 several modifications, e.g., duration of timeslots reduced from 1 ms to 62.5 µs, guaranteeing 16 × faster scheduling, and the available bandwidth is increased accordingly. This aspect (coupled with the network virtualization) makes MEC adoption particularly well suited to support next-generation services requiring very low latencies and/or high bandwidth, such as vehicular ones (from in-vehicle infotainment to assisted/cooperative driving). In addition, operators can exploit their hosting facilities at the edge of mobile networks and can open their radio access network (RAN) edge to authorized third parties, allowing them to flexibly and rapidly deploy innovative applications and services towards mobile subscribers, enterprises, and vertical segments. More specifically, when implementing MEC in mobile networks, MEC hosts are placed at the edge of the network (e.g., close to the radio access network of a 4G or 5G cellular network) and support MEC applications, leveraging edge services exposed through APIs, e.g., radio network information (RNI) API or location API, helping to tailor their computation based on the network conditions or user location. Other APIs can be exposed to MEC applications following the general design principles of RESTful APIs, to ensure interoperability and portability of MEC applications across different domains. For all these reasons, the focus of this paper is on MEC deployment in mobile networks, also due to the huge business and technology impact of 5G systems on future communications. The deployment of the MEC infrastructure should meet different, contrasting needs. On one hand, the infrastructure owner should ensure that its hosts are used efficiently: to this aim, it would be preferable to pool resources relatively far from the users, so that temporal and spatial service usage variability can be met by leveraging statistical multiplexing, aggregating fewer computing resources. On the other hand, low-latency services should, by definition, be hosted as close as possible to the user clients, which pulls towards a capillary distribution of computing resources. Other factors, such as technology constraints (e.g., maximum size of MEC hosts), logistic constraints (e.g., accessibility of sites), or cost aspects (i.e., CAPEX and OPEX), will also play a role. The optimal trade-off point will depend on all these tradeoffs and also on the type of services considered. Finally, the actual MEC penetration will also be influenced by the dominant type of user access technology in a specific area. Accordingly, since 5G access is expected to dominate in the near future, there is a pressing need for a sound evaluation of MEC services and deployments in 5G networks. Such an evaluation should factor in user-related performance metrics, related to both the computation and communication segments of a MEC service. For instance, the round-trip time experienced by a user for a request-response service will include both the communication latency, i.e., both access, core, and backhaul traversing up to a (possibly far away) MEC host, and to the computation time spent by the MEC app on the host itself, which in turn will depend on contention of shared computing resources (e.g., CPU, memory, storage). From this perspective, it is quite clear that deploying MEC hosts at different locations will affect both the communication latency and the contention on resources (since it will alter the number of users sharing those resources). For all these reasons, performance assessment is key for the evaluation of MEC deployment options. In particular, a proper end-to-end (E2E) evaluation should include reliable models of both the network and the computing environment. This is not easy to achieve in practice, due to the presence of many components, from access to edge and core network, and the need to cover not only PHY and MAC layers, but also traffic models in system-level simulations, and a proper workload modeling of the specific MEC app, related to the use-case under evaluation. Some existing works (e.g., refs. [8,9]) rely on numerical evaluations of the system performance, using analytical models. Few simulation tools allow one to simulate applications and services communicating through a 5G network, notably ns3-based 5G Lena [10] and OMNeT++-based Simu5G, both of which quite recent. To the best of our knowledge, ns3 does not include models of MEC hosts. Simu5G, on the other hand, does include a MEC host model [11], but the resource contention at the latter is not modeled satisfactorily. On the other hand, there are cloud/edge computing simulators, e.g., refs. [12,13], which model the computation part but not the communication one. Intel CoFluent [14] is one such tool. However, a unified framework, including both communication and computation, is missing. J.J. Sens.Sens. ActuatorActuator Netw.Netw.2020 2020,,9 9,, 57x FOR PEER REVIEW 33 of 14 16

In this paper, we evaluate the performance of MEC services, specifically in-vehicle infotainment, In this paper, we evaluate the performance of MEC services, specifically in-vehicle infotainment, used by users accessing them through a 5G network in mobility. We focus on the impact of the used by users accessing them through a 5G network in mobility. We focus on the impact of the various various 5G deployment alternatives on the performance of the considered MEC service and their 5G deployment alternatives on the performance of the considered MEC service and their interplay interplay with the capacity provided within the MEC infrastructure. We claim that an effective with the capacity provided within the MEC infrastructure. We claim that an effective performance performance evaluation of MEC services cannot be achieved by analyzing the RAN part through evaluation of MEC services cannot be achieved by analyzing the RAN part through network-only network-only simulation alone. Instead, we advocate that a co-simulation approach, including simulation alone. Instead, we advocate that a co-simulation approach, including models of both models of both network and compute element, would be beneficial for the analysis. However, no network and compute element, would be beneficial for the analysis. However, no simulator that we simulator that we know of models both satisfactorily. For this purpose, we realize a co-simulation know of models both satisfactorily. For this purpose, we realize a co-simulation framework where the framework where the communication part is simulated through Simu5G, and the MEC host is communication part is simulated through Simu5G, and the MEC host is simulated using CoFluent. simulated using CoFluent. The two tools communicate via file exchange, for maximum flexibility. The two tools communicate via file exchange, for maximum flexibility. The above framework is used The above framework is used to assess various deployment options for MEC hosts. Our results show to assess various deployment options for MEC hosts. Our results show that, as one would expect, that, as one would expect, 4G is going to be a major bottleneck, preventing MEC potential to be 4G is going to be a major bottleneck, preventing MEC potential to be unleashed. On the other hand, unleashed. On the other hand, introducing 5G will enable MEC services to scale up to a considerably introducing 5G will enable MEC services to scale up to a considerably larger penetration, challenging larger penetration, challenging the capacity of the MEC infrastructure, and requesting it to scale the capacity of the MEC infrastructure, and requesting it to scale accordingly. One contribution of accordingly. One contribution of this paper also lies in the modularity of our framework: other MEC this paper also lies in the modularity of our framework: other MEC services can be evaluated by services can be evaluated by minimally modifying a setup-namely, the application model in both minimally modifying a setup-namely, the application model in both Simu5G (for what concerns its Simu5G (for what concerns its communication pattern) and CoFluent (for its computation pattern). communication pattern) and CoFluent (for its computation pattern). The rest of the paper is organized as follows. Section 2 describes MEC in the context of 4G and The rest of the paper is organized as follows. Section2 describes MEC in the context of 4G and 5G networks, whereas Section 3 describes our co-simulation framework. Section 4 reports evaluation 5G networks, whereas Section3 describes our co-simulation framework. Section4 reports evaluation results, and Section 5 concludes the paper. results, and Section5 concludes the paper.

2.2. Multi-AccessMulti-Access EdgeEdge ComputingComputing inin 4G4G/5G/5G

2.1.2.1. BackgroundBackground onon Multi-AccessMulti-Access EdgeEdge ComputingComputing InIn FigureFigure1 1 the the MEC MEC architecture, architecture, as as standardized standardized by by ETSI ETSI [ [4],4], isis reported.reported. TheThe MECMEC architecturearchitecture allowsallows MECMEC apps apps to to be be executed executed in a in virtualized a virtuali environment,zed environment, thus enabling thus enabling dynamic dynamic and on-demand and on- allocationdemand allocation of computational of computational resources resources in response in re tosponse users requestto users forrequest a particular for a particular task or service.task or Networkservice. Network operators operators supervising supervising and managing and managing an instance an ofinstance a MEC of system a MEC will system be able will to be optimize able to resourceoptimize utilization resource utilization even dynamically, even dynamically, e.g., by migrating e.g., by migrating and/or relocating and/or relocating user applications user applications between MECbetween hosts, MEC following hosts, following computation computation and communication and communication requirements. requirements.

Device app MEC MEC ... system-level system level Device app management

MEC app MEC ... Platform MEC MEC MEC app host-level host level management Virtualization Infrastructure

Cellular Network External Networks Networks

FigureFigure 1.1. OverviewOverview ofof thethe Multi-accessMulti-access EdgeEdge ComputingComputing Framework.Framework.

AtAt thethe toptop ofof thethe hierarchyhierarchy sitssits thethe MECMEC SystemSystem Level,Level, whichwhich maintainsmaintains aa globalglobal visionvision ofof thethe statestate ofof allall thethe MECMEC HostsHosts withinwithin thethe system.system. TheThe MECMEC SystemSystem LevelLevel handleshandles severalseveral managementmanagement tasks,tasks, includingincluding handlinghandling MECMEC ApplicationApplication InstantiationInstantiation requestsrequests comingcoming fromfrom useruser applications,applications, checkingchecking applicationapplication requirements requirem (e.g.,ents MEC-services(e.g., MEC-services availability, availability, maximum allowed maximum communication allowed communication latency, requested computational resources), selecting and configuring the MEC host

J. Sens. Actuator Netw. 2020, 9, 57 4 of 14 J. Sens. Actuator Netw. 2020, 9, x FOR PEER REVIEW 4 of 16 latency,that will requested host and computationalexecute the MEC resources), application selecting instance. and configuring In a MEC host, the MEC a MEC host platform that will provides host and executemultiple the MEC MEC services, application as defined instance. in ref. In a [15]. MEC The host, latter a MEC can platform be used providesby MEC multipleapps (acting MEC as services, clients asfor defined the MEC in platform), ref. [15]. The to realize latter can complex be used contex by MECt-dependent apps (acting behaviors. as clients Example for the of MEC MEC platform), services toare: realize the smart complex relocation context-dependent service, which behaviors. allows MEC Example apps to ofmigrate MEC servicesamong MEC are: thehosts smart (e.g., relocation to follow service,the least-latency which allows path MECwhen appsusers to are migrate mobile); among the radio MEC network hosts (e.g., information to follow service the least-latency (RNIS), which path whenprovides users information are mobile); on the the radio state networkof the underlyi informationng network service (e.g., (RNIS), how which many provides users are information connected, onwhat the is state the load of the at underlying the base station, network etc.); (e.g., the howbandwidth many users manager, are connected, which sets what data is traffic the load priorities at the basewithin station, the MEC etc.); host; the the bandwidth location service, manager, that which can be sets queried data to tra getffi caccess priorities to the within users’ theposition. MEC MEC host; theapps location run on the service, MEC that host can as virtual be queried machines, to get and access they to can the communicate users’ position. both MEC with apps other run entities on the in MECthe MEC host host as virtual (e.g., the machines, MEC platform, and they to can use communicate its services) bothand with the other outs entitieside world in the (e.g., MEC users’ host (locale.g., theapplications). MEC platform, This is to achieved use its services) by the virtualization and with the outside infrastructure. world (e.g., users’ local applications). This is achieved by the virtualization infrastructure. 2.2. MEC Deployment in Cellular Networks 2.2. MEC Deployment in Cellular Networks The progressive introduction in the market of MEC technology and its use cases are subject to manyThe (technical progressive and business) introduction decision in the factors market [16]. of MEC In particular, technology from and a itsmobile use cases network are subject operator to many(MNO) (technical perspective, and coupling business) MEC decision with factors their plans [16]. fo Inr particular,progressive from introduction a mobile networkof 5G system operator is a (MNO)consolidated perspective, assumption, coupling especially MEC withfor those their who plans have for a progressive migration path introduction to an NFV-based of 5G system network is a consolidated(in fact MEC assumption,is basically based especially on virtualized for those who infrastructure, have a migration and possibly path to andeployed NFV-based also networkin NFV (inenvironments fact MEC is[17]). basically For this based reason, on virtualized it is of the infrastructure, utmost importance and possibly to consider deployed MEC alsodeployment in NFV environmentsoptions jointly [ 17with]). ForMNOs this reason,orientations, it is of and the utmost coherently importance with their to consider plans regarding MEC deployment 5G deployment options jointlyphases. with MNOs orientations, and coherently with their plans regarding 5G deployment phases. A recent survey from GSMA [[18]18] asked operators planning to launch 5G in the next two years which ofof the the 5G 5G Architecture Architecture Options Options (defined (defined in ref. [in19 ])ref. they [19]) were they considering were considering for their deployment for their plans.deployment The survey plans. revealedThe survey that revealed it is widely that expectedit is widely that expected mobile that operators mobile will operators initially will deploy initially 5G usingdeploy Option 5G using 3 (named Option as 3 5G(named NSA, as non-standalone) 5G NSA, non-st allowingandalone) the allowing reuse of the existing reuse evolved of existing packet evolved Core (EPC)packet functionality Core (EPC) functionality (Rel. 15 early (Rel. drop), 15 andearly would drop), require and would from infra-vendorsrequire from infra-vendors that the subsequently that the plannedsubsequently introduction planned of introducti Option 2on (5G of SA, Option standalone) 2 (5G beSA, not standalone) delayed too be much not (possiblydelayed withtoo much some intermediate(possibly with steps, some which intermediate can be foundsteps, which in the can study). be found Overall, in the the study). main phasesOverall, of the the main gradual phases 5G introductionof the gradual (after 5G introduction the legacy LTE (after network, the legacy i.e., Option LTE network, 1) are considered i.e., Option in the1) are sequence considered of Figure in the2, i.e.,sequence starting of fromFigure Option 2, i.e., 3 starting (5G NSA) from and Option then moving 3 (5G towardNSA) and Option then 2 moving (5G SA). toward This paper Option considers 2 (5G thisSA). sequenceThis paper as considers a reference this for sequence performance as a reference comparison for performance when evaluating comparison MEC when deployments evaluating in 5GMEC systems. deployments in 5G systems.

Figure 2. Main phases of 5G architecture opti optionsons [7], [7], as defineddefined by ref. [[19].19].

Another aspectaspect to to be be taken taken into into account account in in the the evaluations evaluations is of is course of course the usethe case.use case.In fact, In fact, KPIs KPIs (key performance(key performance indicators) indicators) and thus and thethus related the related end-to-end end-to-end (E2E)performance (E2E) performance depend depend on the specificon the usespecific case use considered. case considered. In this paper In this we paper analyze we theanalyze infotainment the infotainment use case foruse automotive case for automotive scenarios (alsoscenarios referred (also to referred as in-vehicle to as in-vehicle entertainment), entertainment), where MEC where can MEC support can immersive support immersive high-definition high- entertainmentdefinition entertainment for all vehicle for occupants,all vehicle occupants, including video including streaming, video gaming,streaming, virtual gaming, reality, virtual office reality, work, office work, online education, advertisement, etc. There are several motivations for this choice, from an industrial perspective. First, automotive is a key sector for both MEC and 5G. These technologies

J. Sens. Actuator Netw. 2020, 9, 57 5 of 14

J.online Sens. Actuator education, Netw. 2020 advertisement,, 9, x FOR PEER etc.REVIEW There are several motivations for this choice, from an industrial 5 of 16 perspective. First, automotive is a key sector for both MEC and 5G. These technologies will open willthe open door tothe multiple door to usemultiple cases anduse servicescases and that services can be that monetized can be monetized by 5G Automotive by 5G Automotive Association Associationstakeholders. stakeholders. Second, in addition Second, toin traditionaladdition to use-cases traditional on connecteduse-cases /onautomated connected/automated cars, infotainment cars, is infotainmentalso a promising is also market. a promising According market. to recent According forecasts to recent [20], “Intel forecasts predicts [20], a “Intel new $7 predicts trillion a passenger new $7 trillioneconomy passenger will emerge economy when will passengers emerge be-comewhen pa riders”;ssengers moreover, be-come theriders”; study moreover, showed that the “drivers study showedspend 300that h“drivers a year behindspend 300 the h wheela year andbehind 5G th offe erswheel entertainment and 5G offers opportunities entertainment to opportunities optimize that totime optimize as we that transition time as fromwe transition drivers to from riders”. driver Third,s to riders”. infotainment Third, infotainment interests not interests only automotive not only automotivestakeholders, stakeholders, but also operators but also and operators other content and providers,other content who providers, may be willing who tomay conveniently be willing o fftoer convenientlytheir “legacy” offer multimedia their “legacy” services multimedia for a comfortable services in-vehicle for a comfortable user experience. in-vehicle This user could experience. be easy to Thisoffer, could from be a shorter easy to time-to-market offer, from a perspective.shorter time-to-market From a modeling perspective. point of From view, a infotainment modeling point services of view,(as described infotainment above) services can be (as represented described by above) many can diff beerent represented traffic streams. by many In this different paper, traffic we concentrate streams. Inthe this evaluation paper, we on concentrate a CDN-based the video-streamingevaluation on a CDN-based traffic model. video-streaming traffic model.

3.3. Co-Simulation Co-Simulation Based Based E2E E2E Evaluation Evaluation Framework Framework WeWe now now describe describe the the architecture architecture of of the the co-simulation co-simulation framework framework for for end-to-end end-to-end simulation simulation of of MEC-enabledMEC-enabled cellular cellular networks. networks. The The latter latter is is achi achievedeved by by joining joining the the Simu5G Simu5G network network simulator simulator and and thethe Intel Intel CoFluent CoFluent MEC hosthost simulator.simulator. WeWe first first describe describe both both simulators simulators and and then then the the framework framework that thatjoins joins them. them.

3.1.3.1. Simu5G Simu5G Simu5GSimu5G [21–24] [21–24 ]is is a asystem-level system-level simulator simulator for for th thee data data plane plane of of the the 5G 5G NR NR technology, technology, based based onon OMNeT++ OMNeT++ [25].[25 It]. evolves It evolves from from the the well-known well-known SimuLTE, SimuLTE, which which is a is library a library for forthe thesimulation simulation of 4Gof networks 4G networks [26,27]. [26, 27By]. integrating By integrating the INET the INET framework framework [28], Simu5G [28], Simu5G allows allows the evaluation the evaluation of end- of to-endend-to-end applications, applications, incorporating incorporating the whole the whole TCP/IP TCP /protocolIP protocol stack stack and and standard standard network network devices, devices, e.g.,e.g., routers. routers. It Itis backward-compatible is backward-compatible with with SimuLTE, SimuLTE, and and thus thus it allows it allows one to one simulate to simulate also 4G also and 4G mixedand mixed 4G/5G 4G scenarios./5G scenarios. The Themain main components components of Simu5G of Simu5G areare the the User User Equipment Equipment (UE) (UE) and and the the gNodeBgNodeB (gNB). (gNB). Both Both elements include aa NetworkNetwork Interface Interface Card Card (NIC) (NIC) module module that that implements implements all all the thelayers layers of theof the NR NR protocol protocol stack, stack, as shown as shown in Figure in Figure3. The 3. The gNB gNB can connectcan connect either either directly directly to the to Core the CoreNetwork Network (CN), (CN), i.e., in i.e., a 5G in SAa 5G deployment SA deployment (Option (Option 2), or act 2), as or a secondaryact as a secondary node (SN) node and (SN) connected and connectedto a master to node a master (MN) node implemented (MN) implemented by an LTE eNB by an through LTE eNB the X2through interface. the X2 The interface. latter represents The latter the represents5G NSA deployment the 5G NSA (Option deployment 3) and (Option enables E-UTRA3) and enables/NR dual E-UTRA/NR connectivity dual (ENDC) connectivity [29]. In an(ENDC) ENDC [29].deployment, In an ENDC the UE deployment, can attach tothe either UE thecan gNBattach or theto either eNB, orthe both. gNB Within or the Simu5G, eNB, orthis both. is modeledWithin Simu5G,by endowing this is themodeled UE with by twoendowing protocol the stacks, UE with one two for protocol LTE and stacks, one for one NR, for sharing LTE and the one packet for NR, data sharingconvergence the packet protocol data (PDCP) convergence layer. protocol (PDCP) layer.

FigureFigure 3. 3. MainMain components components of of Simu5G. Simu5G.

Simu5GSimu5G supports supports carrier carrier aggregat aggregation,ion, where where each each component component carrier carrier can can be be configured configured to to use use eithereither frequency- frequency- or or time-division time-division duplexing duplexing (FDD (FDD,, TDD) TDD) and and different different numerologies, numerologies, thus thus allowing allowing maximummaximum flexibility flexibility in inbuilding building simulation simulation scenarios. scenarios. Simu5G Simu5G models models the effects the eofff ectspath ofloss, path fading loss, and inter-cell interference on message transmission, without modeling the transmission of individual symbols, which is instead the focus of link simulators such as ref. [30]. This makes it more scalable, hence apt to evaluate both network- and applications-related aspects in large-scale systems.

J. Sens. Actuator Netw. 2020, 9, 57 6 of 14 fading and inter-cell interference on message transmission, without modeling the transmission of J.individual Sens. Actuator symbols, Netw. 2020 which, 9, x FOR is insteadPEER REVIEW the focus of link simulators such as ref. [30]. This makes it 6more of 16 scalable, hence apt to evaluate both network- and applications-related aspects in large-scale systems. Simu5G can also run as a networknetwork emulator.emulator. In In this this last setting, it can be connected to real applications and carry packets between them in realreal time, with the delay and bandwidth constraint of a 5G5G cellularcellular network.network. This is useful for rapid prototyping of MECMEC applications.applications. For instance, Simu5G cancan be be connected connected to theto the Intel Intel OpenNESS OpenNESS framework framework for MEC for hosting MEC hosting [31]. Work [31]. [22 Work] evaluates [22] evaluatesSimu5G emulation Simu5G emulation capabilities. capabilities.

3.2. CoFluent ® Intel® CoFluent™CoFluent™ studiostudio [14] [14] is is a a modeling modeling and and simulation tool for optimizing, analyzing, and predicting the performance of complex systems. Co CoFluentFluent allows allows one one to model and simulate the interactions amongamong both both hardware hardware and and software software elements, elements, thus speedingthus speeding up the up deployment the deployment of complex of complexsystems. Itsystems. simulates It computingsimulates computing and communication and commu costnication with high cost accuracy with high and highaccuracy efficiency, and high thus efficiency,it can be eff thusectively it can used be effectively to measure used and evaluateto measur thee and latency evaluate and throughputthe latency and of data throughput flows coming of data to flowsand leaving coming from to theand MEC leaving system. from Within the MEC this work,system. CoFluent Within modelsthis work, a MEC CoFluent environment, models including a MEC environment,the User Plane including Functions the (UPFs) User and Plane the Function MEC apps within(UPFs) aand MEC the host. MEC A high-levelapp within view a MEC of the host. latter A high-levelis shown in view Figure of 4the, and latter is composed is shown ofin twoFigure main 4, elements:and is composed the UPF of and two the main MEC elements: app. Data the arrive UPF andat and the leave MEC the app. MEC Data host arrive through at and the leave UPF the element, MEC host which through is further the dividedUPF element, into two which sub-elements, is further dividedrespectively into onetwo for sub-elements, the uplink (UL) respectively and one forone the for downlink the uplink (DL) (UL) directions. and one Thefor the processing downlink time (DL)Tx = directions.required by The a packet processing of size timeS, traversing 𝑇 required the by UPF a packet in direction of size x𝑆,istraversing Tx S/B xthe, where UPF inBx directionis the data 𝑥 isbandwidth 𝑇 𝑆/𝐵 of, where the UPF 𝐵 in is the thex datadirection. bandwidth Packets of processed the UPF in by the the 𝑥 UPF direction. are forwarded Packets toprocessed a MEC app, by whichthe UPF implements are forwarded the service to a MEC required app, by which the user implem and producesents the aservice set of datarequired packets by asthe a response.user and Theseproduces packets a set are of forwardeddata packets back as toa response. the UPF, which These processespackets are them forwarded and sends back them to backthe UPF, to the which RAN. Theprocesses communication them and sends path between them back the to UPF the andRAN. the The MEC communication app is modeled path through between two the packet UPF and queues, the MECone per app direction, is modeled which through have a two configurable packet queues, latency one of traversal. per direction, The behavior which have and structurea configurable of the latencyMEC app of cantraversal. be customized The behavior to model and the structure required of service. the MEC In Sectionapp can4 we be willcustomized describe to the model model the of requireda MEC-based service. infotainment In Section application.4 we will describe the model of a MEC-based infotainment application.

Figure 4. MEC host model on CoFluentTM.

3.3. Co-Simulation Framework Our co-simulation co-simulation architecture architecture enhances enhances the the on onee in inref. ref. [31], [31 based], based on a on 4G a network, 4G network, and incorporatesand incorporates a 5G a 5Gone. one. With With reference reference to toFigure Figure 5,5 ,Simu5G Simu5G and and CoFluentCoFluent areare configuredconfigured independently, each each with with its its own own models, models, configuration configuration files files and and simulation simulation workflows, workflows, and and also alsorun independently.run independently. Coordination Coordination is achieved is achieved via viafile file exchange. exchange. Simu5G Simu5G will will simulate simulate the the 5G 5G RAN, RAN, composed ofof aa configurableconfigurable number number of of UEs UEs and and BSs. BSs. Each Each UE UE runs runs an an application application client, client, modeling modeling the thebehavior behavior of the of selected the selected use case, use and case, initiates and service initiates requests service to therequests MEC infrastructure.to the MEC infrastructure. Requests’ data packetsRequests’ traverse data packets the UL oftraverse the RAN the and UL the of CNthe RAN and get and to the MECCN and host. get The to latterthe MEC generates host. The responses latter generatesand sends responses them back and to the sends UE throughthem back the to DL the path. UE BSsthrough can be the either DL path. eNBs ofBSs gNBs can be depending either eNBs on the of gNBsconsidered depending deployment. on the considered The structure deployment. of the CN depends The structure on the of deployment the CN depends as well. on CoFluent the deployment instead assimulates well. CoFluent the CN instead and the simulate MEC environment.s the CN and Service the MEC requests environment. coming from Service Simu5G requests are coming processed from by Simu5Gthe MEC are host, processed through theby the UPF MEC and host, the MEC through app, andthe UPF CoFluent and the generates MEC app, service and responses, CoFluent generates which are service responses, which are then fed back to Simu5G. Note that Simu5G includes also a model of the EPC, which is not instantiated to avoid duplication.

J. Sens. Actuator Netw. 2020, 9, 57 7 of 14 then fed back to Simu5G. Note that Simu5G includes also a model of the EPC, which is not instantiated toJ. Sens. avoid Actuator duplication. Netw. 2020, 9, x FOR PEER REVIEW 7 of 16

CoFluent Studio TM simu2mec.py

UL Packet trace UL JSON file Collector mec2simu.py

MEC Packet trace MEC JSON file

Uplink Direction Downlink Direction

Figure 5. Implementation of the co-simulation framework. Figure 5. Implementation of the co-simulation framework. The two tools interact via file exchange as follows: Step 1:TheWe two instantiate tools interact a scenario, via file complete exchange with as follows: UE location and mobility, and we run it in Simu5G. Step 1:In theWe latter, instantiate UEs will a scenario, generate complete application-level with UE messages, location and to bemobility, transmitted and we on run the ULit in legSimu5G. of the Inradio the latter, access UEs network, will generate destined application-level to the collector. messages, These messages to be transmitted will accumulate on the delay UL leg on of their the radioway up,access due network, to user contention, destined to protocol the collector delays,. These channel messages quality will impairments, accumulate etc.delay The on tra theirffic waythat arrivesup, due to to the user collector contention, (i.e., thatprotocol exits delays, the RAN) channel leaves quality Simu5G impairments, as well. We timestampetc. The traffic the thattraffi arrivesc, and weto the log collector both timestamp (i.e., that and exits message the RAN) sizes leaves to Logfile#1 Simu5G file. as Morewell. specifically,We timestamp we logthe traffic,both the and simulated we log both time timestamp at which aand message message is generatedsizes to Logfile#1 at the UE, file. and More the specifically, simulated timewe log at bothwhich the it reachessimulated the time collector. at which These a message times are is expressed generated in at Simu5G’s the UE, and units. the simulated time at Step 2:whichWe it run reaches CoFluent, the collector. using Logfile#1 These times as anare input. expressed The in MEC Simu5G’s elements units. modeled in CoFluent Step 2:process We therun requestsCoFluent, generated using Logfile#1 by the UEs, as an and inpu thet. MECThe MEC server elements generates modeled replies in accordingly. CoFluent processThese replies the requests will contain generated a payload, by the which UEs, dependsand the MEC on the server traffi cgenerates model used replies within accordingly. CoFluent. TheseThese replies replies will are sentcontain back a payl to theoad, RAN. which Similar depends to step on the 1, replies traffic aremodel timestamped used within and CoFluent. logged Thesein the replies Logfile#2 are file.sent In back the to latter, the RAN. timestamps Similar are to expressedstep 1, replies in CoFluent’s are timestamped units. Duringand logged step in 2, thetime Logfile#2 is spent file. traversing In the latter, the necessary timestamps MEC are elements, expressed e.g., in queueingCoFluent’s up units. at contention During step points. 2, time Step 3:is spentWe run traversing Simu5G the once necessary more, using MEC the elements same scenario, e.g., queueing as for step up at 1. contention More specifically, points. the UE Step 3:location We andrun Simu5G mobility once are the more, same. using We nowthe same use Logfile#2 scenario toas generatefor step 1. the More replies specifically, to be transmitted the UE locationto the UEs and using mobility the DL are leg ofthe the same. RAN. We As innow step use 1, replies Logfile#2 accumulate to generate delay duethe toreplies interference, to be transmittedchannel issues, to the congestion UEs using and the protocol DL leg mechanisms.of the RAN. As in step 1, replies accumulate delay due to interference, channel issues, congestion and protocol mechanisms. We note that file reads and writes are actually made in C++ within the co-simulation endpoints. This isWe done, note however, that file reads only at and the writes beginning are actually (read) and made at the in endC++ (write)within ofthe the co-simulation simulation. Thisendpoints. allows usThis to is avoid done, unnecessary however, only file-based at the operations beginning during(read) theand execution at the end of (write) the simulations, of the simulation. thus adding This a negligibleallows us to overhead avoid unnecessary to the simulation file-based performance. operations Another during benefit the execution of this approach of the simulations, is that it allows thus usadding to compute a negligible end-to-end overhead delays to the (e.g., simulation the round-trip performance. time Another of a request—RTT) benefit of this without approach having is that to synchronizeit allows us simulatedto compute times end-to-end in the two delays frameworks. (e.g., the Inround-trip fact, the RTTtime canof a be request—RTT) computed as thewithout sum ofhaving the delays to synchronize incurred insimulated steps 1, 2,times and in 3, the each two one frameworks. of which is computedIn fact, the withinRTT can a simulator.be computed Thus, as ifthe messages sum of the keep delays track incurred of the delay in steps accumulated 1, 2, and 3, thus each far, one the of RTT which is obtained is computed automatically within a simulator. at the end ofThus, step if 3. messages Finally, usingkeep track a file-based of the delay interaction accumulate allowsd usthus to far, obtain the aRTT co-simulation is obtained frameworkautomatically with at minimalthe end of modification step 3. Finally, of theusing two a file-based involved simulators,interaction allows and notably us to obtain without a co-simulation any impact on framework their core functionalities.with minimal modification This last characteristic of the two isinvolved particularly simulators, important and innotably the perspective without any of upgradability. impact on their core Thefunctionalities. above co-simulation This last framework characteristic is realized is particularly by minimally important modifying in Simu5G the perspective and CoFluent, of codingupgradability. some software tools to interface the two. In the UL direction the following elements have been implemented: The above co-simulation framework is realized by minimally modifying Simu5G and CoFluent, coding some software tools to interface the two. In the UL direction the following elements have been implemented: 1. An application that issues service requests periodically at a UE, written in C++.

J. Sens. Actuator Netw. 2020, 9, 57 8 of 14

J. Sens. Actuator Netw. 2020, 9, x FOR PEER REVIEW 8 of 16 1. An application that issues service requests periodically at a UE, written in C++. 2. A C++ collector application that collects all the UE messages (after they have got to the respective 2. AC++ collector application that collects all the UE messages (after they have got to the respective BSs) and prints Logfile#1. BSs) and prints Logfile#1. 3. A program written in python, simu2mec.py, that converts Logfile#1 to a structured JSON file, to 3. A program written in python, simu2mec.py, that converts Logfile#1 to a structured JSON file, to be used as an input to CoFluent. be used as an input to CoFluent. Dually, some more apps have been coded to cope with the DL return trip: Dually, some more apps have been coded to cope with the DL return trip: 1. A program written in python, mec2simu.py, that parses the MEC packet trace produced by 1. CoFluent,A program and written generates in python, a structured mec2simu.py, JSON file, that which parses specifies the MEC for each packet packet, trace itsproduced ID, the ID byof theCoFluent, UE that and generated generates it, and a structured the timestamp. JSON file, which specifies for each packet, its ID, the ID of 2. Athe C++ UE application that generated that it, takes and the the timestamp. above JSON file as an input and generates a trace of DL 2. messagesAC++ application accordingly. that takesThese the messages above JSON are filethose as anthat input will and be generatesinjected at a tracestep of3, DLto reach messages the interestedaccordingly. UEs These via their messages serving are BSs. those that will be injected at step 3, to reach the interested UEs 3. Avia C++ their application, serving BSs. running on UEs, that collects responses and generates reports (e.g., by 3. computingAC++ application, the RTT of running each request on UEs, and that assemb collectsling responses it with some and generatesaggregator, reports say the (e.g., mean by value).computing the RTT of each request and assembling it with some aggregator, say the mean value).

4. Performance Evaluation In thisthis section, section, we we assess assess the performance the performance of a MEC-enabled of a MEC-enabled service in variousservice systemin various deployments. system Ourdeployments. aim is to highlight Our aim theis to impact highlight and the contribution impact and of contribution the different of system the different components system (i.e., components RAN, CN, MEC(i.e., RAN, host) CN, on the MEC E2E host) service on performance.the E2E service performance. The application application scenario scenario we we consider consider is an is info antainment infotainment service, service, provided provided in a Cellular in a CellularVehicle- Vehicle-to-Everythingto-Everything (C-V2X) (C-V2X)environment, environment, whose logical whose view logical is shown view is in shown Figure in 6. Figure Each vehicle6. Each carries vehicle a carriesuser client a user that client runs thatwithin runs a UE within and aconnects UE and to connects a MEC host to a MECthrough host the through cellular the network. cellular The network. client Theperiodically client periodically issues a request issues for a requestan infotainment for an infotainment service. The service. server is The realized server as is a realizedMEC app as residing a MEC appin a MEC residing host in and a MEC responds host and to requests responds by to sending requests the by client sending a transcoded the client aversion transcoded of a prerecorded version of a prerecordedvideo stream. video On each stream. request, On each the request,client specifie the clients one specifies of the video one of formats the video reported formats in reported Table 1,in a Tablehigher1, video a higher quality video corresponding quality corresponding to a higher to bandwidth. a higher bandwidth. The MEC Theapp MECis composed app is composed of two parts: of twoa request parts: handler a request and handler a streamer. and aThe streamer. former Thereceiv formeres and receives processes and the processes service requests, the service identifying requests, identifyingthe video and the its video format, and itsand format, notifies and the notifies stream theer. The streamer. latter Thetranscodes latter transcodes and transmits and transmitsthe video thestream, video one stream, frame one each frame 40 ms. each Video 40 ms. frames Video are frames finally arereceived finally at received the client, at thewhich client, collects which them collects and themcomputes and computesmetrics. In metrics. our co-simulation In our co-simulation framework, framework, the client is the implemen client isted implemented as a UDP application as a UDP applicationrunning on runninga UE on Simu5G, on a UE onwhereas Simu5G, the whereas infotainme thent infotainment server is implemented server is implemented as an MEC app as an within MEC appCoFluent. within CoFluent.

Figure 6. Logical view of the infotainment application.

Table 1. Video Formats. Table 1. Video Formats.

VideoVideo Format Format Bitrate Bitrate Frame Frame Size Size (Fixed) (Fixed) 240p240p 400400 kbps kbps 2000 2000 Bytes Bytes 720p720p 2400 2400 kbps kbps 12,000 12,000 Bytes Bytes

We define frame latency as the time between the transmission of a video frame from the MEC app and its reception at the UE. This measures the “age” of the information received by the user, and

J. Sens. Actuator Netw. 2020, 9, 57 9 of 14

J. Sens. Actuator Netw. 2020, 9, x FOR PEER REVIEW 9 of 16 We define frame latency as the time between the transmission of a video frame from the MEC app andcan itsbe receptionfurther divided at the UE.into Thisa MEC measures latency, the i.e., “age” from ofthe the generation information of a received video frame by the at user,the MEC and canapp beto furtherits packets divided leaving into the a MEC server, latency, a i.e.,CN fromlatency, the i.e., generation the timeof it atakes video these frame packets at the to MEC reach app the togNB/eNB, its packets and leaving a RAN the latency, MEC i.e., server, the time a CN between latency, the i.e., arrival the time of ita video takes theseframe packetsat the gNB/eNB to reach and the gNBits reception/eNB, and at athe RAN UE. latency, We consider i.e., the a RAN time composed between the of arrival57 cells, of deployed a video frameaccording at the to gNBthe hexagonal/eNB and itstessellation reception shown at the UE. in the We left consider part of a Figure RAN composed 7 (we show of only 57 cells, the deployedfirst two tiers according of cells to for the readability). hexagonal tessellationEach site hosts shown three in co-located the left part BSs of and Figure each7 (weBS radi showates only towards the first the twocenter tiers of ofthe cells respective for readability). hexagon. EachThe inter-site site hosts distance three co-located is 500 m. BSs UEs and are each deploy BS radiatesed according towards to the an center urban-grid of the respectivevehicular hexagon.scenario, Theshown inter-site in the right distance part isof 500 Figure m. 7, UEs which are deployedcorresponds according to the area to andefined urban-grid by the vehiculardashed rectangle scenario, to shownthe left. in BSs the outside right part that of area Figure are7 simulated, which corresponds as external to cells. the area defined by the dashed rectangle to the left. BSs outside that area are simulated as external cells.

Figure 7. RAN deployment (left) and urban-grid scenario (right). [credits to icograms.com]. Figure 7. RAN deployment (left) and urban-grid scenario (right). [credits to icograms.com]. In order to assess the impact of different 4G/5G and MEC deployments, we consider three scenarios, In order to assess the impact of different 4G/5G and MEC deployments, we consider three coherently with the phases previously described in Section 2.2: scenarios, coherently with the phases previously described in Section 2.2: Baseline LTE deployment (Figure8, top): the BSs are LTE eNBs and the MEC app is co-located •• Baseline LTE deployment (Figure 8, top): the BSs are LTE eNBs and the MEC app is co-located with the S-GW and the P-GW in a centralized EPC site. We assume a one-way latency between with the S-GW and the P-GW in a centralized EPC site. We assume a one-way latency between the RAN and the MEC of 15 ms [17]. the RAN and the MEC of 15 ms [17]. 5G NSA deployment (Figure8, center): the BSs are again LTE eNBs and each of the three central • cells also has a gNB placed in the center of the hexagons, connected to the corresponding eNB via X2 and radiating power according to an omnidirectional pattern. The LTE eNBs are connected to a distributed EPC site in proximity of the RAN. Both S-GW and P-GW, as well as the MEC, are considered as virtual network functions (VNFs). Since the path between the RAN and the MEC involves two VNFs, we assume a one-way latency of 200 µs [32]. For the X2 interface, we assume 5 ms latency [33,34]. 5G SA deployment (Figure8, bottom): the BSs are gNBs connected to a distributed 5G Core (5GC). • Again, the MEC is a VNF connected to (at least) a UPF, acting as PDU session anchor in the 5GC, and terminating to the data network. Since there is only one VNF between the RAN and the MEC, the considered one-way latency is 100 µs.

J. Sens. Actuator Netw. 2020, 9, 57 10 of 14 J. Sens. Actuator Netw. 2020, 9, x FOR PEER REVIEW 10 of 16

Figure 8. LTE deployment with centralized EPC (top), 5G NSANSA deploymentdeployment withwith distributeddistributed EPCEPC (center),), 5G5G SASA deploymentdeployment withwith distributeddistributed 5GC5GC ((bottombottom).).

• The5G NSA main deployment simulation (Figure parameters 8, center): are summarized the BSs are again in Table LTE2. eNBs We first and runeach simulations of the three withcentral a J.varying Sens.cells Actuator number also Netw. has of2020 a UEs gNB, 9, x (24 FORplaced to PEER 40), in REVIEW increasing the center theof the video hexagons, bitrate. connected Figure9 shows to the the corresponding percentage 11 of ofeNB DL 16 framevia occupied X2 and in radiating the 4G RAN, power for theaccording 4G and 5Gto NSAan omnidirectional scenarios. The figure pattern. shows The that LTE this eNBs increases are withconnected the video bitrate,to aNumerology distributed however indexEPC keeping site (5G belowin only) proximity 50%. 2 In(60-kHZ of the the 5G RAN. subcarrier NSA Both case, S-GWspacing) the 4G and DL P-GW, is occupied as well even as less, duethe MEC, to the gNBare Txconsidered o fflPoweroading. as The virtual difference network in user func BS: experiencetions 46 dBm; (VNFs). isUE: clarified Since23 dBm the in Figure path between 10, describing the RAN the frameand latency the MEC and itsFading involves breakdown. effects two ItVNFs, is quite we evident assume Enabledthat, a one-way in the 4G latency scenario, of both 200 theμs RAN[32]. andFor thethe CNX2 introduceinterface, a considerable we assumePath loss delay. 5 modelms latency On the [33,34]. other hand, [35] even high video bitrates are well tolerated in the 5G• NSA5G SA scenario, deploymentUE and mobility the (Figure 5G SA 8, one bottom): exhibits the negligible BSs Linear, are delays.gNBs ~𝑈10,18 connected In all m/s the threeto a distributed scenarios, the5G MECCore delay(5GC). is negligible. Again,CN the Figure latency MEC 11 is (one-way) showsa VNF theconnected standard to LTE:(at deviation le 15ast) ms a ofUPF, the acting frame as latency, PDU session which measuresanchor in the jitter,the 5GC, and isand a meaningful terminating indicator to the data of user-perceivednetwork.5G: Since 200 Qualitythereμs (NSA), is ofonly 100 Experience. one μs (SA)VNF between It is quite the evident RAN that,and although the MEC, theX2 DL the latency is considered far from being one-way congested, latency in 5is ms the100 4G μs. scenarios jitters comparable to twice the inter-packet time andSimulation more can time occur, meaning that packets200 s will need considerable buffering at the UE. The main simulation parameters are summarized in Table 2. We first run simulations with a This does not happenUPF with UL/DL either bandwidth 5G NSA or 5G SA. 100 These Gb/s results show that the 4G network is going varying number of UEs (24 to 40), increasing the video bitrate. Figure 9 shows the percentage of DL to represent the bottleneckUPF-APP for and MEC APP-UPF services delay such as 5.4 this, μs and that only deploying 5G will alleviate frame occupied in the 4G RAN, for the 4G and 5G NSA scenarios. The figure shows that this increases that. In order to findREQ the processing point at which time the MEC host 50 startsμs acting as a bottleneck, we ran simulations with the video bitrate, however keeping below 50%. In the 5G NSA case, the 4G DL is occupied even with a considerablyFrame higher creation number time of users (up to 248) 200 μ ins a 5G SA deployment. This time we used a less, due to the gNB offloading. The difference in user experience is clarified in Figure 10, describing low bitrate video,Frame to avoid size saturating the RAN. The {2000, results 12,000, are shown 24,000} in bytes Figure 12, and they clearly the frame latency and its breakdown. It is quite evident that, in the 4G scenario, both the RAN and show that the MECVideo host becomes duration congested around 2008 s users. However, before this occurs, the bulk of the CN introduce a considerable delay. On the other hand, even high video bitrates are well tolerated the latency is still dueRequest to the period 5G RAN. 12 s in the 5G NSA scenario, and the 5G SA one exhibits negligible delays. In all the three scenarios, the MEC delay is negligible. Figure 11 shows the standard deviation of the frame latency, which measures the jitter, and is a meaningful indicator of user-perceived Quality of Experience. It is quite evident that, although the DL is far from being congested, in the 4G scenarios jitters comparable to twice the inter-packet time and more can occur, meaning that packets will need considerable buffering at the UE. This does not happen with either 5G NSA or 5G SA. These results show that the 4G network is going to represent the bottleneck for MEC services such as this, and that only deploying 5G will alleviate that. In order to find the point at which the MEC host starts acting as a bottleneck, we ran simulations with a considerably higher number of users (up to 248) in a 5G SA deployment. This time we used a low bitrate video, to avoid saturating the RAN. The results are shown in Figure 12, and they clearly show that the MEC host becomes congested around 200 users. However, before this occurs, the bulk of the latency is still due to the 5G RAN. FigureFigure 9. 9. DLDL resource-block resource-block occupancy occupancy in in the the 4G 4G and and 5G-NSA 5G-NSA scenarios. scenarios. Table 2. Main Simulation parameters.

Video Format Bitrate Bandwidth 20 MHz (100 Resource Blocks) Carrier Frequency 2 GHz 6 GHz for gNBs in NSA

J. Sens. Actuator Netw. 2020, 9, 57 11 of 14

Table 2. Main Simulation parameters.

Video Format Bitrate Bandwidth 20 MHz (100 Resource Blocks) Carrier Frequency 2 GHz 6 GHz for gNBs in NSA Numerology index (5G only) 2 (60-kHZ subcarrier spacing) Tx Power BS: 46 dBm; UE: 23 dBm Fading effects Enabled Path loss model [35] UE mobility Linear, ~U(10, 18) m/s CN latency (one-way) LTE: 15 ms 5G: 200 µs (NSA), 100 µs (SA) X2 latency 5 ms Simulation time 200 s UPF UL/DL bandwidth 100 Gb/s UPF-APP and APP-UPF delay 5.4 µs REQ processing time 50 µs Frame creation time 200 µs Frame size {2000, 12,000, 24,000} bytes J. Sens. Actuator Netw. 2020Video, 9, durationx FOR PEER REVIEW 8 s 12 of 16 Request period 12 s

J. Sens. Actuator Netw. 2020, 9, x FOR PEER REVIEW 13 of 16

FigureFigure 10. Frame-latencyFrame-latency breakdown breakdown in in a a scenario scenario with with 24 UEs.

FigureFigure 11. 11. StandardStandard deviation deviation of of the the inter-packet inter-packet times times at the UE.

J. Sens. Actuator Netw. 2020, 9, x FOR PEER REVIEW 14 of 16 J. Sens. Actuator Netw. 2020, 9, 57 12 of 14

Figure 12. Frame-latency breakdown in the 5G SA scenario.

5. Conclusions and Future Work This paper presented a methodology to evaluateevaluate MEC deployments with 4G4G/5G/5G access and the results ofof its its application. application. The The methodology methodology consists consists in using in detailedusing detailed system-level system-level simulators simulators (namely, Simu5G(namely, and Simu5G CoFluent) and CoFluent) to simulate to the simulate communication the communication and computation and computation parts of an E2Eparts service of an withE2E MEC,service and with joining MEC, and them joining into a them flexible into co-simulation a flexible co-simulation framework. framework. We have We used have the used above the tool above to comparetool to compare MEC deployment MEC deployment options, options, namely namely 4G, 5G 4G, non-standalone, 5G non-standalone, and 5G and Standalone, 5G Standalone, as for as their for suitabilitytheir suitability to support to support an infotainment an infotainment service service with MEC. with Our MEC. results Our showresults that show 4G radiothat 4G access radio is access going tois going be the to bottleneck be the bottleneck from a QoE from perspective, a QoE perspective, allowing allowing only a small only number a small ofnumber users, andof users, virtually and leavingvirtually most leaving of the most MEC of computationthe MEC computation power untapped. power untapped. With 4G, large With jitters 4G, large will impairjitters will the serviceimpair eventhe service in a relatively even in uncongesteda relatively uncongested RAN. Deploying RAN. 5G, Deploying already in 5G, NSA already but especially in NSA but in the especially SA option, in isthe going SA tooption, make ais large going di fftoerence. make With a large 5G, thedifference. same service With will 5G, be the able same to accommodate service will considerably be able to moreaccommodate users before considerably MEC hosts more become users a be bottleneck.fore MEC hosts become a bottleneck. Due to the inherentinherent flexibilityflexibility of thethe aboveabove framework,framework, this work can be extended in several directions. ForFor instance, instance, in-car in-car task task offl oadingoffloading from from car to car MEC, to e.g.,MEC, critical e.g., workloadscritical workloads to be offl oadedto be tooffloaded MEC in to the MEC case in of the sudden case of blade sudden failure blade in the failure car. in the car.

Author Contributions:Contributions: Conceptualization, A.V., G.N., and G.S.; soft software,ware, A.V. and G.N.; validation, A.V. and G.N.; writing—original draft preparation, A.V., A.V., G.S., an andd D.S.; writing—review and editing, A.V., A.V., G.N., G.S., and D.S. All authors have readread andand agreedagreed toto thethe publishedpublished versionversion ofof thethe manuscript.manuscript. Funding: Work partially supported by the Italian Ministry of Education and Research (MIUR) in the framework Funding: Work partially supported by the Italian Ministry of Education and Research (MIUR) in the framework of the Cross-Lab project (Departments of Excellence). The subject matter of this paper includes description of resultsof the Cross-Lab of a joint research project (Departments project carried of out Excellence). by Intel Corporation The subject and matter the Universityof this pape ofr Pisa.includes Intel description Corporation of reservesresults of all a proprietaryjoint research rights project in any carried process, out procedure,by Intel Corporation algorithm, and article the of University manufacture, of Pisa. or other Intel results Corporation of said projectreserves herein all proprietary described. rights in any process, procedure, algorithm, article of manufacture, or other results of Conflictssaid project of herein Interest: described.Dario Sabella is affiliated with Intel, which funded some of the activities that are described in this paper. Dario Sabella took part in the writing, contributing in particular to the background description and toConflicts the scenario of Interest: definition. Dario The Sabella funder is affiliated did not influence with Intel, the which claims fu ornded the some results of that the areactivities presented that inare this described paper. in this paper. Dario Sabella took part in the writing, contributing in particular to the background description and Referencesto the scenario definition. The funder did not influence the claims or the results that are presented in this paper.

1.ReferencesIntel White Paper. Edge Computing: From Standard to Actual Infrastructure Deployment and Software Development. Available online: https://intel.ly/2Wx7Gy4 (accessed on 10 November 2020).

2.1. ETSIIntel White MEC (Multi-AccessPaper. Edge Computing: Edge Computing). From Standard Available to online:Actual Infrastructurehttps://www.etsi.org Deployment/technologies and Software/multi- access-edge-computingDevelopment. Available (accessed online: https://intel on 10 November.ly/2Wx7Gy4 2020). (accessed on 10 November 2020).

3.2. Sabella,ETSI MEC D.; Vaillant,(Multi-Access A.; Kuure, Edge P.; Computing). Rauschenbach, Availabl U.; Giust,e online: F. Mobile-Edge https://www.etsi.org/technologies/multi- Computing Architecture: The role ofaccess-edge-computing MEC in the Internet of (accessed Things. IEEEon 10 Consum.November Electron. 2020). Mag. 2016, 5, 84–91. [CrossRef]

4.3. ETSISabella, GS D.; MEC Vaillant, 003 V2.1.1 A.; Kuure, (2019-01). P.; Rauschenbach, Multi-access U Edge.; Giust, Computing F. Mobile-Edge (MEC); Computing Framework Architecture: and Reference The Architecture.role of MEC January in 2019.the AvailableInternet online:of Things. https: //bit.lyIEEE /2yOmYpsConsum. (accessedElectron. on Mag. 10 November 2016, 2020).5, 84–91, 5. ETSIdoi:10.1109/mce.2016.2590118. GS MEC 009 V2.1.1 (2019-01). Multi-Access Edge Computing (MEC); General Principles for MEC 4. ServiceETSI GS APIs. MEC Available 003 V2.1.1 online: (2019-01). https:// Multi-accessbit.ly/2zzYCj6 Ed (accessedge Computing on 10 November (MEC); Framework 2020). and Reference Architecture. January 2019. Available online: https://bit.ly/2yOmYps (accessed on 10 November 2020).

J. Sens. Actuator Netw. 2020, 9, 57 13 of 14

6. ETSI White Paper No. 24. MEC Deployments in 4G and Evolution Towards 5G, First Edition—February 2018. Available online: https://bit.ly/2T5AKe7 (accessed on 10 November 2020). 7. ETSI White Paper No. 28. MEC in 5G Networks. June 2018. Available online: https://bit.ly/3bzCLFI (accessed on 10 November 2020). 8. Emara, M.; Filippou, M.C.; Sabella, D. MEC-Assisted End-to-End Latency Evaluations for C-V2X Communications. In Proceedings of the 2018 European Conference on Networks and Communications (EuCNC), Ljubljana, Slovenia, 18–21 June 2018; pp. 1–9. 9. Pencheva, E.; Atanasov, I.; Velkova, D.; Trifonov, V. Evaluation of Multi-access Edge Computing Deployment Scenarios. In Proceedings of the 2019 9th Annual Information Technology, Electromechanical Engineering and Microelectronics Conference (IEMECON), Jaipur, India, 13–15 March 2019; pp. 155–160. 10. Patriciello, N.; Lagen, S.; Bojovic, B.; Giupponi, L. An E2E simulator for 5G NR networks. Simul. Model. Pract. Theory 2019, 96, 101933. [CrossRef] 11. Nardini, G.; Virdis, A.; Stea, G.; Buono, A. SimuLTE-MEC: Extending SimuLTE for Multi-Access Edge Computing. EPiC Ser. Comput. 2018, 56, 35–42. [CrossRef] 12. Castañé, G.G.; Nuñez, A.; Carretero, J. iCanCloud: A Brief Architecture Overview. In Proceedings of the 2012 IEEE 10th International Symposium on Parallel and Distributed Processing with Applications, Leganes, Spain, 10–13 July 2012; pp. 853–854. 13. Kliazovich, D.; Bouvry, P.; Khan, S.U. GreenCloud: A Packet-Level Simulator of Energy-Aware Cloud Computing Data Centers. J. Supercomput. 2010, 62, 1–5. 14. Available online: https://www.intel.com/content/www/us/en/cofluent/overview.html (accessed on 10 November 2020). 15. ETSI GS MEC 002. Mobile Edge Computing (MEC); Technical Requirements; 2016-03. Available online: https://www.etsi.org/deliver/etsi_gs/MEC/001_099/002/01.01.01_60/gs_MEC002v010101p.pdf (accessed on 10 November 2020). 16. Sabella, D.; Reznik, A.; Frazao, R. Multi-Access Edge Computing in Action; Informa UK Limited: London, UK, 2019. 17. Sabella, D.; Nikaein, N.; Huang, A.; Xhembulla, J.; Malnati, G.; Scarpina, S. A Hierarchical MEC Architecture: Experimenting the RAVEN Use-Case. In Proceedings of the 2018 IEEE 87th Vehicular Technology Conference (VTC Spring), Porto, Portugal, 3–6 June 2018; pp. 1–5. [CrossRef] 18. GSMA. Operator Requirements for 5G Core Connectivity Options. May 2019. Available online: https: //bit.ly/2WOgiiM (accessed on 10 November 2020). 19. 3GPP RP-161266. Architecture Configuration Options for NR. Available online: https://portal.3gpp.org/ ngppapp/CreateTDoc.aspx?mode=view&contributionUid=RP-161266 (accessed on 10 November 2020). 20. Intel 5G Connected Vehicles Webinar, Referring to the Study from Strategy Analytics. Accelerating the Future: The Economic Impact of the Emerging Passenger Economy. June 2017. Available online: https: //intel.ly/2Wwqpdt (accessed on 10 November 2020). 21. Nardini, G.; Stea, G.; Virdis, A.; Sabella, D. Simu5G: A System-level Simulator for 5G Networks. In Proceedings of the 10th International Conference on Simulation and Modeling Methodologies, Technologies and Applications, Paris, France, 8–10 July 2020. 22. Nardini, G.; Stea, G.; Virdis, A.; Sabella, D.; Thakkar, P. Using Simu5G as a Realtime Network Emulator to Test MEC Apps in an End-To-End 5G Testbed. In Proceedings of the 2020 IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications, London, UK, 31 August 2020. 23. Nardini, G.; Sabella, D.; Stea, G.; Thakkar, P.; Virdis, A. Simu5G–An OMNeT++ Library for End-to-End Performance Evaluation of 5G Networks. IEEE Access 2020, 8, 181176–181191. [CrossRef] 24. Simu5g Website. Available online: http://simu5g.org (accessed on 10 November 2020). 25. OMNeT++ Website. Available online: https://omnetpp.org (accessed on 10 November 2020). 26. Virdis, A.; Stea, G.; Nardini, G. Simulating LTE/LTE-Advanced Networks with SimuLTE. In Advances in Intelligent Systems and Computing; Springer Science and Business Media LLC: Heidelberg, Germany, 2015; Volume 402, pp. 83–105. 27. SimuLTE Website. Available online: http://simulte.com (accessed on 10 November 2020). 28. INET Library Website. Available online: https://inet.omnetpp.org (accessed on 10 November 2020). 29. 3GPP TS 37.340 v16.1.0. Evolved Universal Terrestrial Radio Access (E-UTRA) and NR; Multi-Connectivity; Stage 2 (Rel. 16). March 2020. Available online: https://portal.3gpp.org/desktopmodules/Specifications/ SpecificationDetails.aspx?specificationId=3198 (accessed on 10 November 2020). J. Sens. Actuator Netw. 2020, 9, 57 14 of 14

30. Pratschner, S.; Tahir, B.; Marijanovic, L.; Mussbah, M.; Kirev, K.; Nissel, R.; Schwarz, S.; Rupp, M. Versatile mobile communications simulation: The Vienna 5G Link Level Simulator. EURASIP J. Wirel. Commun. Netw. 2018, 2018, 226. [CrossRef] 31. Virdis, A.; Nardini, G.; Stea, G.; Shi, Y.; Bian, Z. A Co-Simulation Framework to Evaluate Edge Deployment Options and Performance. In Proceedings of the 2020 IEEE International Conference on Communications Workshops (ICC Workshops), Dublin, Ireland, 7–11 June 2020. 32. Filippou, M.C.; Sabella, D.; Riccobene, V. Flexible MEC service consumption through edge host zoning in 5G networks. In Proceedings of the 2019 IEEE Wireless Communications and Networking Conference Workshop (WCNCW), Marrakech, Morocco, 15–18 April 2019. 33. Wang, H.; Rosa, C.; Pedersen, K.I. Dual connectivity for LTE-advanced heterogeneous networks. Wirel. Netw. 2015, 22, 1315–1328. [CrossRef] 34. Michalopoulos, D.S.; Maeder, A.; Kolehmainen, N. 5G Multi-Connectivity with Non-Ideal Backhaul: Distributed vs Cloud-Based Architecture. In Proceedings of the 2018 IEEE Globecom Workshops (GC Wkshps), Abu Dhabi, UAE, 9–13 December 2018; pp. 1–6. [CrossRef] 35. 3GPP TR 36.873 v12.7.0. Study on 3D Channel Model for LTE. December 2017. Available online: https:// portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=2574 (accessed on 10 November 2020).

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).