Design and Analysis of Multiple OS Implementation on a Single ARM-Based Embedded Platform

sustainability

Article Design and Analysis of Multiple OS Implementation on a Single ARM-Based Embedded Platform

Byoungwook Kim 1 and Min Choi 2,* 1 Creative Informatics & Computing Institute, Korea University, 145 Anam-ro, Seongbuk-gu, Seoul 02841, Korea; [email protected] 2 Department of Information and Communication Engineering, Chungbuk National University, 1 Chungdae-ro, Seowon-gu, Cheongju, Chungbuk 28644, Korea * Correspondence: [email protected]; Tel.: +82-10-9969-5699

Academic Editor: James J. Park Received: 5 April 2017; Accepted: 15 April 2017; Published: 25 April 2017

Abstract: Recently, with the development of embedded system hardware technology, there is a need to support various kinds of operating system (OS) operation in embedded systems. In mobile processors, ARM started to provide the virtualization extension support technology which was intended for processors in PC processors. Virtualization technology has the advantage of using hardware resources effectively. If the real-time operating system (RTOS) is operated on a hypervisor, there is a problem that RTOS performance is degraded due to overhead. Thus, we need to compare the performance between a single execution of the RTOS and simultaneous execution of multiple OS (RTOS + Linux). Therefore, in this paper, we measure the performance when the RTOS operates independently on the NVidia Jetson TK-1 embedded board supporting virtualization technology. Then, we measure the performance when the RTOS and Linux are operating simultaneously on top of a hypervisor. For this purpose, we implemented and ported such a RTOS, especially FreeRTOS and uC/OS, onto two embedded boards, such as the Arndale board (SAMSUNG, Seoul, South Korea) and the NVidia TK1 board (NVIDIA, Santa Clara, CA, USA).

Keywords: embedded system; real-time operating system; performance evaluation; multiple operating system

1. Introduction Recently, embedded systems can run multiple operating systems through virtualization technology. Through this hypervisor, we implemented an embedded system that simultaneously runs RTOS and Linux on a single platform. The RTOS performs hardware control tasks requiring a real-time property. At the same time, Linux runs a variety of user programs. Therefore, a highly-reliable high-performance embedded system can be realized. Multiple operating systems on a single platform using a hypervisor provides isolation between the operating systems, but the requirement for an embedded hypervisor are distinct from hypervisors targeting servers and applications [1–3]. While desktop and enterprise environments use hypervisors to consolidate hardware and isolate computing environments from one another, in an embedded system, the various components typically function collectively to provide the device’s functionality [4,5]. An implementation of an embedded hypervisor must deal with a number of issues, which include the highly-integrated nature of embedded systems, the requirement for isolated functional blocks within the system to communicate rapidly, the need for real-time/deterministic performance, the resource-constrained target environment, and the wide range of security and reliability requirements [6]. When a hypervisor does not provide a real-time property, the context switching overhead causes a delay, resulting in performance degradation in the guest operating system. This paper compares and analyzes the performance between Linux and a RTOS

Sustainability 2017, 9, 684; doi:10.3390/su9050684 www.mdpi.com/journal/sustainability Sustainability 2017, 9, 684 2 of 14 on a single embedded platform. In order to conduct the experiment, we make use of the Rhealstone benchmark, which was developed by Boger et al. The rest of this paper is organized as follows: Section2 describes background. Sections3 and4 provide system design and experimental results. Finally, we conclude and summarize our work in Section5.

2. Materials and Methods A hypervisor is a middleware that can run multiple guest operating systems on a single type of hardware. The hypervisor monitors and manages the resources used by the guest OS [7,8]. In this environment, each OS is isolated as if it were running alone. The ARM Cortex-A15 processor’s Virtualization Extension features include: (1) Introduction of Privileged Level HYP Mode In the ARM Virtual Extension, a hypervisor can have higher privileges than the guest OS through the Hyp mode. It is not necessary for the guest OS to recognize the existence of the privileged mode, Hyp mode, which is newly introduced in ARM Cortex-15. The guest OS can still operate in supervisor mode and user mode. On the ARMv7 Cortex-A15 processor, the OS kernel uses supervisor mode. On the ARMv7 Cortex-A15 processor, the user application uses the user mode. In addition, the virtualization hypervisor uses Hyp mode. The hypervisor mode has access to special registers that the user mode or supervisor cannot access. For example, the hypervisor mode has its own Stack Pointer (SP), Saved Processor Status Register (SPSR), and Exception Link Register (ELR). Additionally, registers, such as physical counters, are accessible only in hypervisor mode. (2) Interrupt Virtualization: GICv2 Previously, if an interrupt occurred during the operation of the guest OS, the virtual machine would have to generate a trap to switch to hypervisor mode ﬁrst and then process the interrupt. On the other hand, the ARM Cortex-A15 processor introduces the concept of virtual interrupt to eliminate this inconvenience as shown in Figure1. A virtual interrupt allows the virtual machine to receive an interrupt signal directly or to clear an interrupt that has occurred [9,10]. The virtual machine can process its own interrupt at the guest OS level without the hypervisor’s intervention (however, hardware and device drivers must support the virtual CPU interface for this purpose). In the ARM Cortex-A15, the generic interrupt controller can collect/analyze and arbitrate a large number of interrupt sources: (1) interrupt masking; (2) interrupt priority; (3) interrupt distribution for target processor; (4) interrupt status tracking; and (5) interrupt generation for software. In the ARM Cortex-A15, the interrupt can be (1) passed to the currently-running guest OS; (2) passed to the other guest OS; (3) passed to the hypervisor; or (4) passed to the OS or RTOS which runs in the secure TrustZone. In the ARM Cortex-A15, the hypervisor takes priority in physically interrupting. However, if the interrupt is an interrupt to be passed to the guest OS, the hypervisor provides the ability to map the virtual interrupt to the guest OS. In addition, a hypervisor can create a virtual interrupt even if no physical interrupt has occurred [11,12]. Sustainability 2017, 9, 684 3 of 14 Sustainability 2017, 9, 684 3 of 14

FigureFigure 1. 1.Generic Generic interruptinterrupt controller architecture. architecture.

(3) Timer Virtualization Support: GTimer (3) Timer Virtualization Support: GTimer System counter: Gtimer is a shareable “always on” counter. Gtimer has fast read access capabilitySystem counter: for reliable Gtimer time isdistribution. a shareable It “alwayshas a 64-b on”it count counter. value Gtimer and counts has fastfrom read a value access of zero capability at for reliableinitialization. time distribution. The operating It system has a 64-bit reads count the CNTFRQ value and register, counts which from is a the value system of zero control at initialization. register Theat operating system boot system time. reads The theoperat CNTFRQing system register, initializes which the is thesystem system counter control frequency register through at system the boot time.CNTFRQ The operating register system and uses initializes the system the counter. system counter frequency through the CNTFRQ register and uses theVirtual system counter: counter. Virtual counters allow virtualization to measure time on each virtual machine duringVirtual operation. counter: VirtualAll processors counters with allow generic virtualization timers always to measure provide time virtual on eachcounters, virtual even machine if duringvirtualization operation. extension All processors technology with is not generic applied timers(at this time, always the providevirtual counter virtual represents counters, virtual even if time). If the virtualization extension technology is not applied, the virtual time is the same as the virtualization extension technology is not applied (at this time, the virtual counter represents virtual physical time. The virtual counter has the same value as the physical counter. If virtualization time). If the virtualization extension technology is not applied, the virtual time is the same as the extension technology is applied, the virtual counter has a value obtained by subtracting the virtual physical time. The virtual counter has the same value as the physical counter. If virtualization extension offset of 64 bits from the physical counter. The CNTVOFF register is an RW register. It is accessible in technologythe non-secure is applied, hypervisor the virtual mode [13]. counter If the has SCR.NS a value value obtained is set to 1, by it subtractingis accessible even the virtualin the secure offset of 64 bitsmonitor from mode. the physical The current counter. virtual The counter CNTVOFF value is registerstored in is the an CNTVCT RW register. register It isas accessible64 bits. If one in the non-securewants to hypervisor access this register, mode [they13]. need If the special SCR.NS privileges value isto setaccess to 1,the it register is accessible [14,15] even(you can in the access secure monitorthis register mode. on The secure current PL1 virtual mode, non-secure counter value PL1 mode, is stored and in non-secure the CNTVCT PL2 mode). register as 64 bits. If one wants toTimer access function this register, that raises they an need event special after privilegesa certain period to access of time: the The register generic [14 ,timer15] (you provides can access a thistimer register function, on secure which PL1 varies mode, slightly non-secure depending PL1 on mode, the processor and non-secure manufacturer. PL2 mode). Timers provide the abilityTimer to function send signals that to raises the syst anem. event If the after processor a certain is connected period of to time: the GIC, The the generic signaltimer is transferred provides a timerthrough function, the whichPPI. When varies virtualization slightly depending extension ontechno thelogy processor (VE) is manufacturer. applied, the processor Timers provides provide the abilityone to non-secure send signals PL1 to physical the system. timer, If theone processor secure PL1 is connectedphysical timer, to the one GIC, non-secure the signal secure is transferred PL2 physical timer, and one virtual timer. Each timer provides three registers: a 64 bit compare value through the PPI. When virtualization extension technology (VE) is applied, the processor provides register for a 64-bit unsigned up-counter, a 32-bit timer value register for a 32-bit signed one non-secure PL1 physical timer, one secure PL1 physical timer, one non-secure secure PL2 physical down-counter, and a 32-bit control register, which is available as both an up-counter and timer,down-counter. and one virtual timer. Each timer provides three registers: a 64 bit compare value register for a 64-bit unsignedThe RTOS up-counter, kernel requires a 32-bit some timer modifications value register to forsupport a 32-bit ARM signed Cortex-A15 down-counter, virtualization and a on 32-bit controlthe register,hypervisor. which To this is available end, the assystem both anstruct up-counterure and source and down-counter. tree of uC/OS-II and FreeRTOS are brieflyThe RTOS shown. kernel requires some modiﬁcations to support ARM Cortex-A15 virtualization on the hypervisor. To this end, the system structure and source tree of uC/OS-II and FreeRTOS are brieﬂy shown. SustainabilitySustainability 2017, 20179, 684,, 9 , , 684684 4 of 144 of 14

FigureFigure 2 is a 2 conceptual is a conceptual diagram diagram that changesthat changes from fr theom traditional the traditional structure structure to the to one the inone which in which the thehypervisor hypervisorFigure 2is is aused. is conceptual used. The TheAR diagram Mv7ARMv7 Cortex-A15 that Cortex-A15 changes processor from processor the traditionalis implemented is implemented structure by to virtualizationby the virtualization one in which extensionsextensionsthe hypervisor (VE), (VE), a hardware-support is used.a hardware-support The ARMv7ed Cortex-A15virtualizationed virtualization processor to support to issupport implemented virtualization virtualization by virtualizationtechnology. technology. It extensions also It also supports(VE),supports interrupt a hardware-supported interrupt virtualization virtualization virtualizationtechnology technology (GICv2). to (GICv2). support The virtualizationTheGIC GICis a istechnology a technology.technology that thatreduces It also reduces supports the the overheadoverheadinterrupt by handling virtualizationby handling the interrupts the technology interrupts directly (GICv2). directly by th by Thee guest th GICe guest OS is without a OS technology without intervention intervention that reduces of the of the hypervisor the overhead hypervisor by whenwhenhandling the interrupt the theinterrupt interrupts occurs. occurs. Therefore, directly Therefore, bythe the guest the guest guest OS OSneeds OS without needs to operate to intervention operate in the in supervisor the of thesupervisor hypervisor mode mode and when theand the user userinterruptmode mode in occurs.the in samethe Therefore, same manner manner theas guestwhen as when OSit is needs itex isecuted toex operateecuted by itself. by in theitself. There supervisor There are variousare mode various andRTOSs theRTOSs in user the in mode the market.market.in the However, same However, manner in this in as study,this when study, uC/OS-II it is uC/OS-II executed and byFreeRTOSand itself. FreeRTOS There are used, are variousused, for which for RTOSs which the insource the the source market. code code is However, free is free [16]. [16].in this study, uC/OS-II and FreeRTOS are used, for which the source code is free [16].

Figure 2. System architecture. FigureFigure 2. System 2. System architecture. architecture.

3. Design3. Design and Implementationand Implementation In thisIn study, this study, we used wewe usedused NVidia’s NVidia’sNVidia’s Jetson JetsonJetson TK-1 TK-1TK-1 development developmentdevelopment board board with withwith an ARMv7 anan ARMv7ARMv7 Cortex-A15 Cortex-A15Cortex-A15 processor,processor, as shown as shown in Figure in Figure 3. We3 3.. WeinstalledWe installedinstalled the QPlusthethe QPlusQPlus Hypervisor HypervisorHypervisor as a asashypervisor aa hypervisorhypervisor in Jetson inin JetsonJetson TK1. TK1.TK1. LinuxLinux (Ubuntu (Ubuntu(Ubuntu 14.04) 14.04)14.04) and and andFreeRTOS FreeRTOS FreeRTOS were were were used used used asas guestguest as guest OS. OS. We OS.We make makeWe use make use of the useof embeddedthe of embeddedthe embedded hardware hardwarehardwaredebugger debugger Lauterbachdebugger Lauterbach Lauterbach Trace 32,Trace which Trace 32, iswhich 32, a JTAG which is debugger.a JTAGis a JTAG debugger. The debugger. debugger The Thedebugger enables debugger us enables to debugenables us lineto us byto debugdebugline line on lineby the line embeddedby lineon the on embedded system,the embedded dump system, asystem, segment dump dump ofa memory,segment a segment viewof memory, of the memory, content view view and the value contentthe content of aand certain and valuevalueregister, of a certainof anda certain soregister, on. register, and soand on. so on.

(a) (a) (b) (b) Figure 3. Working platform and debugger. Note: (a) TK1 board view from the top; (b) TK1 and Figure 3.3. WorkingWorking platform platform and and debugger. debugger. Note: Note: (a) TK1(a) TK1 board board view view from thefrom top; the (b )top; TK1 ( andb) TK1 Arndale and Arndale board with embedded debugger, the Trace32. Arndaleboard with board embedded with embedded debugger, debugger, the Trace32. the Trace32.

We Weevaluated evaluated the situationthe situation of FreeRTOS of FreeRTOS oper operatingating alone alone and andwith with Linux Linux and andFreeRTOS FreeRTOS We evaluated the situation of FreeRTOS operating alone and with Linux and FreeRTOS operating operatingoperating together together with with QPlus QPlus Hypervisor. Hypervisor. In this In stthisudy, study, we measured we measured the task the tasktransition transition delay delay time time together with QPlus Hypervisor. In this study, we measured the task transition delay time and and andinterrupt interrupt latency latency of FreeRTOS. of FreeRTOS. When When running running on the on Qplusthe Qplus hypervisor, hypervisor, Linux Linux was wasexecuted executed interrupt latency of FreeRTOS. When running on the Qplus hypervisor, Linux was executed with a withwith a background a background process process and noand other no other running running processes. processes. background process and no other running processes. FigureFigure 4 shows 4 shows how howthe uC/OSthe uC/OS porting porting implementation implementation onto onto the ARM-A15the ARM-A15 evaluation evaluation board, board, Figure4 shows how the uC/OS porting implementation onto the ARM-A15 evaluation board, especiallyespecially Arndale Arndale and andTK1, TK1, progress. progress. The TheOS kernelOS kernel starts starts with with startup.S, startup.S, the ARMthe ARM assembly assembly especially Arndale and TK1, progress. The OS kernel starts with startup.S, the ARM assembly start-up start-upstart-up code. code. Then Then it calls it callsthe mainthe main and andOSInit OSInit functions functions in the in leftthe topleft oftop the of Figurethe Figure 4. Then, 4. Then, it it code. Then it calls the main and OSInit functions in the left top of the Figure4. Then, it initializes initializesinitializes and andsets setsseveral several structures structures and andregist registers relateders related to generic to generic timer, timer, general general interrupt interrupt and sets several structures and registers related to generic timer, general interrupt controller, UART, controller,controller, UART, UART, memory, memory, task taskcontrol control blocks, blocks, and soand on. so on. memory, task control blocks, and so on.

Sustainability 2017, 9, 684 5 of 14 Sustainability 2017, 9, 684 5 of 14 Sustainability 2017, 9, 684 5 of 14

Sustainability 2017, 9, 684 5 of 14

Figure 4. uC/OS kernel architecture. FigureFigure 4. 4.uC/OS uC/OS kernel architecture. architecture.

We makeWe make use useof theof theco-processor co-processorFigure interface, interface,4. uC/OS kernel CP15CP15 architecture., to configureconfigure the the generic generic timer. timer. The The generic generic timerWe timeruses make usesvirtual usevirtual timer of the timer control, co-processor control, interrupt interrupt interface, source source CP15, PPI4, PPI4, to IRQ conﬁgure numbernumber the 27. 27. genericWe We set set the timer. the IRQ IRQ Thenumber number generic to 30 timerto 30 foruses the virtualfor physical theWe physical timermake timer. control,use timer. ofThe the Theinterrupt following co-processor following source register register interface, PPI4, settings settings IRQ CP15 number in, to FigureFigure configure 27. 5 5 are We are the required set required generic the IRQ for timer.for the number the GTimer The GTimer togeneric timer 30for timer the interruptphysicaltimerinterrupt timer.to uses be to recognizedvirtual The be recognized following timer ascontrol, aas register GIC a GIC interrupt register. register. settings source Fi First, inrst, Figure PPI4,we we setset IRQ5 theare number ICENABLERICENABLER required 27. We for and set the and theISENABLER GTimer ISENABLER IRQ number timer registers interrupttoregisters 30 forto beGICD'sforfor recognized theGICD's interruptphysical interrupt as atimer. clear GIC clear The register.and and following set. set. First,Then, Then, register we we we set set set settings the th the ICENABLERe ITARGETSRITARGETSR in Figure 5 andregister areregister ISENABLERrequired to to set set forthe the theCPU registers CPUGTimer to recognize to forrecognizetimer GICD 's theinterrupt interrupt.interruptthe interrupt. clear Into and beaddition, In recognized set. addition, Then, we we asinitialize we a initialize setGIC the register. the ITARGETSRthe timer timer First, by by we settingsetting registerset the thethe ICENABLER to bits bits set of of the the the CPUoffset and offset to ISENABLERcorresponding recognizecorresponding registers the to interrupt.IRQ to IRQ for27 inGICD's the IPRIORITYR interrupt clear register and set.to determine Then, we theset thprioritye ITARGETSR of the interrupt register [17–19]. to set the CPU to recognize 27In in addition, the IPRIORITYR we initialize register the timerto determine by setting the thepriority bits ofof thethe offsetinterrupt corresponding [17–19]. to IRQ 27 in the IPRIORITYRthe interrupt. register In addition, to determine we initialize the priority the timer of by the setting interrupt the bits [17 of– 19the]. offset corresponding to IRQ 27 in the IPRIORITYR register to determine the priority of the interrupt [17–19].

Figure 5. Interrupt processing on hardware initialization.

On power-up, orFigureFigure afterFigure 5. 5.a Interrupt5. Interruptreset, Interrupt a processingGIC processing processing implemen on on hardwarehardware hardwaretation that initialization. initialization. initialization. supports interrupt grouping is configured with all interrupts assigned to Group 0, and the FIQ exception request disabled. This Onmeans power-up,On that power-up, Group or 0orafter interrupts after a reset,a reset, are asignaled aGIC GIC implemen implemenusing the tationtationIRQ interrupt thatthat supports supports request. interrupt Figureinterrupt 6grouping shows grouping this is is On power-up, or after a reset, a GIC implementation that supports interrupt grouping is configuredconfiguredconfiguration. with with all interruptsall interrupts assigned assigned to to Group Group 0, and thethe FIQ FIQ exception exception request request disabled. disabled. This This conﬁgured with all interrupts assigned to Group 0, and the FIQ exception request disabled. This means meansmeans that thatGroup Group 0 interrupts 0 interrupts are are signaled signaled usingusing the IRQIRQ interruptinterrupt request. request. Figure Figure 6 shows 6 shows this this that Groupconfiguration. 0 interrupts are signaled using the IRQ interrupt request. Figure6 shows this conﬁguration. configuration.

Figure 6. Co-processor control register for the use of generic timer.

FigureFigure 6. 6. Co-processorCo-processor control control register register for for the the use use of of generic generic timer. timer.

Sustainability 2017, 9, 684 6 of 14

SustainabilitySustainability 2017 2017, 9, ,9 684, 684 6 6of of 14 14 When the timer is initialized, the timer periodically generates an interrupt. At this time, uC/OS-II checksWhenWhen which thethe interrupt timertimer is isis initialized,generated.initialized, The thethe OS timertimer can periodically useperiodically the information generatesgenerates in the anan GIC interrupt.interrupt. register At toAt determinethisthis time,time, uC/OS-IItheuC/OS-II IRQ number. checks checks which Thewhich OS interrupt interrupt executes is is thegenerated. generated. interrupt The The service OS OS can can routine use use the the according information information to the in in IRQ the the number.GIC GIC register register In the to to determineeventdetermine of a timerthe the IRQ interrupt,IRQ number. number. the TimerTheThe OS OS Tick executes executes function the the is called.interrupt interrupt This service service function routine routine calls theaccording according uC/OS-II to to OS the the kernel IRQ IRQ number.functionnumber. OSTimeTick().In In the the event event of of a a timer timer interrupt, interrupt, the the Timer Timer Tick Tick function function is is called. called. This This function function calls calls the the uC/OS-IIuC/OS-IIThe GICOS OS kernel maintainskernel function function a state OSTimeTick(). OSTimeTick(). machine for each supported interrupt on each CPU interface. Figure7 showsTheThe an GIC instanceGIC maintains maintains of this a stateastate state machine machine machine and for for theeach each possible support support stateeded interrupt interrupt transitions. on on each each CPU CPU interface. interface. Figure Figure 77 shows shows an an instance instance of of this this state state machine machine and and the the possible possible state state transitions. transitions.

Figure 7. State transition diagram for interrupt processing. FigureFigure 7. 7. State State transition transition diagram diagram for for interrupt interrupt processing. processing.

TheThe study study builds builds on on the the ARMARM A15A15 embeddedembedded boar boardboardd to to runrun twotwo oror moremore operating operating systemssystems simultaneouslysimultaneously through through the the hypervisor. hypervisor. In In part particular,particular,icular, we we mademade made UbuntuUbuntu Ubuntu LinuxLinux Linux andand and RTOSRTOS RTOS (uC/OS(uC/OS (uC/OS andand FreeRTOS) FreeRTOS) run run simultaneously simultaneously on on the the embedded embedded board. board. Due DueDue to toto the thethe nature naturenature of ofof the thethe hypervisor, hypervisor,hypervisor, twotwo operating operating systems systems share share the the same same UART UART console. console. Therefore, Therefore,Therefore, as asas shown shownshown in inin Figure FigureFigure 88, ,8, thethe the RTOSRTOS RTOS outputsoutputs a a ‘hello ‘hello arndale’ arndale’ message message and and a a ‘Goodbye ‘Goodbye arndale’ arndale’ message message to toto the the screen. screen. At AtAt the thethe same samesame time, time,time, onon Ubuntu Ubuntu Linux, Linux, the the root root user’s user’s Linu Linux Linuxx prompt prompt is is displayed displayed on on the the screen. screen.

FigureFigure 8. 8. Simultaneous Simultaneous execution execution of of mu multiple multipleltiple OS OS on on embedded embedded system. system.

ThisThis studystudy study waswas was performedperformed performed with with with a Tektronixaa Tektronix Tektronix oscilloscope osci oscilloscopelloscope using using using the the GPIOthe GPIO GPIO ports ports ports of the of of Jetson the the Jetson Jetson TK-1 TK-1developmentTK-1 development development board board board to verify to to verify verify the operating the the operating operating characteristics ch characteristicsaracteristics of the of of hypervisor the the hypervisor hypervisor and guestand and guest guest OS. WeOS. OS. alsoWe We performedalsoalso performedperformed some someofsome the Rhealstoneofof thethe RhealstoneRhealstone benchmark benchmbenchm setsark [8ark] for setssets performance [8][8] forfor performanceperformance comparisons. comparisons.comparisons. The Rhealstone TheThe RhealstonebenchmarkRhealstone benchmarkwasbenchmark proposed waswas in proposed 1989.proposed It does inin 1989. not1989. measure ItIt doesdoes the notnot quality measuremeasure of the thethe complete qualityquality of solution,of thethe completecomplete but is, solution,instead,solution, a but measurementbut is,is, instead,instead, targeted aa measurementmeasurement toward a multitaskingtargetedtargeted towardtoward solution. aa multitaskingmultitasking In the task solution. transitionsolution. In delayIn thethe time tasktask transitionmeasurement,transition delay delay we time time repeatedly measuremen measuremen outputt,t, HIGH-LOWwe we repeatedly repeatedly to outputeach output GPIO HIGH-LOW HIGH-LOW port for two to to each tasks.each GPIO GPIO Each port taskport outputsfor for two two tasks.hightasks. to Each Each its GPIO task task outputs portoutputs when high high it isto torunning. its its GPIO GPIO Therefore,port port when when after it it is is running. task running. switching, Therefore, Therefore, the corresponding after after task task switching, switching, GPIO port the the correspondingremainscorresponding low. Task GPIO GPIO 1 and port port Task remains remains 2 use their low. low. own Task Task GPIO 1 1 an an ports.dd Task Task Therefore, 2 2 use use their their Figure own own9 showsGPIO GPIO theports. ports. status Therefore, Therefore, change FigureofFigure each 9 GPIO9 shows shows port the the simultaneously status status change change of throughof each each GPIO GPIO the two-channel port port simultaneously simultaneously input to thethrough through oscilloscope. the the two-channel two-channel input input toto the the oscilloscope. oscilloscope.

Sustainability 2017, 9, 684 7 of 14

Figure 9. Task switching time.

Figure 9. Task switching time. Task switching time is the average time the system takes to switch between two independent and active, notTask suspended switchingswitching time or issleeping, thethe averageaverage tasks timetime of thethe equal systemsystem priority. takestakes toto switchswitchTask betweenswitchingbetween twotwo is independentindependent synchronous and non-preemptiveand active, and not is suspended an important or sleeping, measure tasks of of equalany multitasking priority. Task switchingsystem. isThis synchronous metric is and influenced non-preemptive and is an important measure of any multitasking system. This metric is influenced by the non-preemptivehost CPU's architecture, and is an important instruction measure of se anyt, multitaskingand features, system. and This is metric designed is inﬂuenced to assess by the theby hostthe CPUhost 'sCPU's architecture, architecture, instruction instruction set, and features,set, and andfeatures, is designed and tois assessdesigned the compactnessto assess the of compactness of task control data structures and the efficiency with which the executive manipulates taskcompactness control data of task structures control and data the structures efﬁciency and with the which efficiency the executive with which manipulates the executive the data manipulates structures the data inthestructures saving data structures and in restoring saving in saving and contexts. and restoring restoring Task switching contex contexts. time, TaskTask additionally,switching switching time, measurestime, additionally, additionally, the executive’s measures measures the list the executive'smanagementexecutive's list management list capabilities. management capabilities. Figure capabilities. 10 shows Figure Figure the related1010 shows data the the structures related related data maintained data structures structures by maintained the OS maintained kernel by by the OS kernelduringthe OS duringkernel the task during switching.the task the taskswitching. switching.

Figure 10. Related data structures for switching tasks.

The task-Switching benchmarking sets up two tasks with equal priority. The tasks switch back and forth between the processor. To first determine the time it takes to perform the for loop “work”, the benchmark just measures the loops performing no work and not task switching. FigureFigure 10. 10.RelatedRelated data data structures structures for for switching switching tasks. tasks. As shown in Figure 11, we measured the interrupt latency in the following way. Two tasks output HIGH (Task 1) and LOW (Task 2) status for one GPIO port, respectively. We start the The task-Switching benchmarking sets up two tasks with equal priority. The tasks switch back measurementThe task-Switching when an interrupt benchmarking occurs. sets It outputs up two tasksa signal with to equala specific priority. GPIO The when tasks the switch execution back and forthandflow betweenforth enters between the the interrupt processor. the processor. handler. To ToThus,first first determine we determine can chec k thethe whether time time it an takesit takesinterrupt to perform to handlerperform the is for fromthe loop for Task “work”, loop 1 or “work”, the benchmarktheTask benchmark 2 by just the measureswaveform just measures of the the theloops oscilloscope. loops perf performingorming nono work work and and not not task task switching. switching. As shownAs shownin Figure in Figure 11, 11 we, we measured measured the the interrupt interrupt latency latency in the following in the way.following Two tasks way. output Two tasks HIGH (Task 1) and LOW (Task 2) status for one GPIO port, respectively. We start the measurement output HIGHwhen an (Task interrupt 1) occurs.and LOW It outputs (Task a signal 2) status to a specific for one GPIO GPIO when theport, execution respectively. flow enters We the start the measurementinterrupt when handler. an interrupt Thus, we can occurs. check whetherIt outputs an interrupt a signal handler to a specific is from Task GPIO 1 or when Task 2 bythe the execution flow enterswaveform the interrupt of the oscilloscope. handler. Thus, we can check whether an interrupt handler is from Task 1 or Task 2 by the waveform of the oscilloscope.

Figure 11. Interrupt processing delay.

Sustainability 2017, 9, 684 7 of 14

Figure 9. Task switching time.

Task switching time is the average time the system takes to switch between two independent and active, not suspended or sleeping, tasks of equal priority. Task switching is synchronous and non-preemptive and is an important measure of any multitasking system. This metric is influenced by the host CPU's architecture, instruction set, and features, and is designed to assess the compactness of task control data structures and the efficiency with which the executive manipulates the data structures in saving and restoring contexts. Task switching time, additionally, measures the executive's list management capabilities. Figure 10 shows the related data structures maintained by the OS kernel during the task switching.

Figure 10. Related data structures for switching tasks.

Sustainability 2017, 9, 684 8 of 14 Figure 11. Interrupt processing delay. Figure 11. Interrupt processing delay. 4. Experimental Results 4. Experimental Results Sustainability 2017, 9, 684 8 of 14 In this study, we implemented two real-time operating systems that operate through an ARM In this study, we implemented two real-time operating systems that operate through an ARM A-15 processor with virtualization support: uC/OS and FreeRTOS. We run two ported RTOSs with 4.A-15 Experimental processor Results with virtualization support: uC/OS and FreeRTOS. We run two ported RTOSs with each Linux operating system. These RTOSs generate a digital clock in the form of a wave every eachIn Linux this study, operating we implemented system. These two RTOSs real-time generate operating a digital systems clock in thethat form operate of a through wave every an second.ARM second. We use the generated clock signal to output to the digital timer board, as shown in Figure 12. A-15We useprocessor the generated with virtualizati clock signalon support: to output uC/OS to the digitaland FreeRTOS. timer board, We asrun shown two ported in Figure RTOSs 12. with each Linux operating system. These RTOSs generate a digital clock in the form of a wave every second. We use the generated clock signal to output to the digital timer board, as shown in Figure 12.

FigureFigure 12. 12. DigitalDigital timer timer board board schematic.schematic. Figure 12. Digital timer board schematic. FigureFigure 13 represents13 represents the the waveform waveform simulation simulation results results generated generated by SPICE by toolsSPICE using tools the using above the abovedigital digitalFigure timer timer13 board represents board schematic schematic the ofwaveform Figure of 12Figure .simulation The digital12.The results timer digital board generated timer was implementedboard by SPICE was implementedtools using using digital the ICs using digitalabove(7447, ICs digital 7490,(7447, etc.).timer 7490, We board etc.). connected Weschematic connected the digital of Figure the clock digital 12 output.The clockdigital from output the timer evaluation boardfrom thewas board evaluation implemented to two timer board using boards to two timerdigitalas boards shown ICs as(7447, in shown Figure 7490, 14in etc.).. Figure We 14.connected the digital clock output from the evaluation board to two timer boards as shown in Figure 14.

FigureFigureFigure 13.13. 13. DigitalDigital timer timer timer circuit circuit circuit simulation. simulation.simulation.

Figure 14 shows the environmental setup for this experiment. We make use of an oscilloscope to measure the digital high/low states when multiple OS and hypervisors are working on the evaluation board.

Figure 14. Digital timer circuit simulation environment. Figure 14. Digital timer circuit simulation environment. Figure 14 shows the environmental setup for this experiment. We make use of an oscilloscope to measure the digital high/low states when multiple OS and hypervisors are working on the Figure 14 shows the environmental setup for this experiment. We make use of an oscilloscope evaluation board. to measureThe theserver digital provides high/low the operating states when system multiple image OSin Tftpand onhypervisors the evaluation are workingboard. We on the evaluationcommunicate board. via the Minicom program with the evaluation board and the UART. The evaluation boardThe providesserver provides a clock with the a digitaloperating timer board.system At imthisage time, in we Tftp measure on thethe clockevaluation timing withboard. an We communicateoscilloscope via to measurethe Minicom the clock program waveform, with as theshown evaluation in Figure board15. and the UART. The evaluation board provides a clock with a digital timer board. At this time, we measure the clock timing with an oscilloscope to measure the clock waveform, as shown in Figure 15.

Sustainability 2017, 9, 684 8 of 14

4. Experimental Results In this study, we implemented two real-time operating systems that operate through an ARM A-15 processor with virtualization support: uC/OS and FreeRTOS. We run two ported RTOSs with each Linux operating system. These RTOSs generate a digital clock in the form of a wave every second. We use the generated clock signal to output to the digital timer board, as shown in Figure 12.

Figure 12. Digital timer board schematic.

Figure 13 represents the waveform simulation results generated by SPICE tools using the above digital timer board schematic of Figure 12.The digital timer board was implemented using digital ICs (7447, 7490, etc.). We connected the digital clock output from the evaluation board to two timer boards as shown in Figure 14.

Sustainability 2017, 9, 684 9 of 14 Figure 13. Digital timer circuit simulation.

Figure 14. Digital timer circuit simulation environment.

FigureThe server 14 shows provides the the environmental operating system setup image for this inTftp experiment. on the evaluation We make board. use of We an communicate oscilloscope tovia measure the Minicom the programdigital high/low with the states evaluation when board multiple and theOS UART.and hypervisors The evaluation are boardworking provides on the a evaluationclock with board. a digital timer board. At this time, we measure the clock timing with an oscilloscope to The server provides the operating system image in Tftp on the evaluation board. We Sustainabilitymeasure the 2017 clock, 9, 684 waveform, as shown in Figure 15. 9 of 14 communicate via the Minicom program with the evaluation board and the UART. The evaluation board provides a clock with a digital timer board. At this time, we measure the clock timing with an oscilloscope to measure the clock waveform, as shown in Figure 15.

Figure 15. Digital timer circuit simulation.

In Figure 15,15, the left digital timer board and the right right digital digital timer timer board board may may look look different. different. However, both boards are only show a difference between implementing the same circuit on a breadboard and and implementing implementing it it on on a universal a universal boar board.d. The The left leftboard board in Figure in Figure 15 receives 15 receives the clock the fromclock fromthe Linux the Linux operating operating system. system. The Theright right board board in Figure in Figure 15 15 receives receives the the clock clock from from the the RTOS RTOS operating system. Especially, wewe obtainedobtained thethe followingfollowing four results from this experiment. Task switching latency measurement results are as follows: Figures 16 and 17 show the ﬁrstfirst experimental results, which are captured from thethe oscilloscope.oscilloscope. Of the two waveformswaveforms shown in the ﬁgures,figures, the upper waveform corresponds to Task 1, and the lower waveform correspondscorresponds to Task 2. We set the time axis to 5.00 μµs when FreeRTOSFreeRTOS operates operates alone. alone. Since Since one on scalee scale on on the the oscilloscope oscilloscope is 1.00 is 1.00µs, the μs, task the transitiontask transition delay delaytime was time measured was measured to be approximatelyto be approximately 3.50 µ s3.50 on average.μs on average.

Figure 16. Task switching time on single RTOS execution.

Sustainability 2017, 9, 684 9 of 14

Figure 15. Digital timer circuit simulation.

In Figure 15, the left digital timer board and the right digital timer board may look different. However, both boards are only show a difference between implementing the same circuit on a breadboard and implementing it on a universal board. The left board in Figure 15 receives the clock from the Linux operating system. The right board in Figure 15 receives the clock from the RTOS operating system. Especially, we obtained the following four results from this experiment. Task switching latency measurement results are as follows: Figures 16 and 17 show the first experimental results, which are captured from the oscilloscope. Of the two waveforms shown in the figures, the upper waveform corresponds to Task 1, and the lower waveform corresponds to Task 2. We set the time axis to 5.00 μs whenSustainability FreeRTOS2017, 9, 684operates alone. Since one scale on the oscilloscope is 1.00 μs, the task transition10 of 14 delay time was measured to be approximately 3.50 μs on average.

FigureFigure 16. TaskTask switching time on single RTOS execution. Sustainability 2017, 9, 684 10 of 14

Sustainability 2017, 9, 684 10 of 14

Figure 17. Task switching time on multiple OS (RTOS + Linux) execution. Figure 17. Task switching time on multiple OS (RTOS + Linux) execution. Figure 17. Task switching time on multiple OS (RTOS + Linux) execution. When Linux and FreeRTOS work together through QPlus Hypervisor, the result is as follows. The timeWhen axis Linuxwas set and to FreeRTOS5.00 ms. Since work one together scale on through an oscilloscope QPlus Hypervisor, is 1.00 ms, the the result task istransition as follows. When Linux and FreeRTOS work together through QPlus Hypervisor, the result is as follows. delayThe time time was axis wasmeasured set to 5.00to be ms. about Since 4.00 one ms scale on average. on an oscilloscope There are isthree 1.00 processes, ms, the task which transition have delayall The time axis was set to 5.00 ms. Since one scale on an oscilloscope is 1.00 ms, the task transition thetime same was priorities. measured However, to be about the process 4.00 ms “T” on average.starts with There the highest are three priority processes, at initialization which have time. all the delay time was measured to be about 4.00 ms on average. There are three processes, which have all Aftersame initialization, priorities. However, the process the will process reduce “T” the starts priority with by the adding highest 1 priorityto the tskIDLE_PRIORITY at initialization time. value After the same priorities. However, the process “T” starts with the highest priority at initialization time. of initialization,itself in order the to processyield the will CPU reduce to the priorityprocess by“F” adding and process 1 to the “S”. tskIDLE_PRIORITY Figure 18 shows value how of we itself After initialization, the process will reduce the priority by adding 1 to the tskIDLE_PRIORITY value controlledin order tothe yield GPIO the ports CPU in to the the FreeRTOS process “F” applicatio and processn tasks, “S”. Figurewhich 18are shows used to how evaluate we controlled the task the of itself in order to yield the CPU to the process “F” and process “S”. Figure 18 shows how we switchingGPIO ports time in and the interrupt FreeRTOS delay. application tasks, which are used to evaluate the task switching time and controlled the GPIO ports in the FreeRTOS application tasks, which are used to evaluate the task interrupt delay. switching time and interrupt delay.

Figure 18. GPIO control code example.

Figure 18. GPIO control code example. The taskYIELD()function yields the CPU to other processes, which is in the ready queue. This code does not actually create tasks, but simply determines the execution time of the portions of the The taskYIELD()function yields the CPU to other processes, which is in the ready queue. This code that are not part of the task-switching measurement. The execution of this code segment is code does not actually create tasks, but simply determines the execution time of the portions of the recorded with the oscilloscope and the second portion of the code runs. The second portion utilizes code that are not part of the task-switching measurement. The execution of this code segment is two tasks. This time the two tasks perform task switching for the desired number of iterations. recorded with the oscilloscope and the second portion of the code runs. The second portion utilizes The result of measuring the interrupt latency is as follows: Among the two waveforms shown in two tasks. This time the two tasks perform task switching for the desired number of iterations. the figure, the upper waveforms are alternating between Task 1 and Task 2 and output HIGH and The result of measuring the interrupt latency is as follows: Among the two waveforms shown in LOW states. The waveform below, in Figures 19 and 20, are generated from the waveform for the the figure, the upper waveforms are alternating between Task 1 and Task 2 and output HIGH and interrupt handler. We calculated the time from when the bottom waveform changes from HIGH to LOW states. The waveform below, in Figures 19 and 20, are generated from the waveform for the LOW, and when the top waveform changes from LOW to HIGH (or HIGH to LOW). When interrupt handler. We calculated the time from when the bottom waveform changes from HIGH to operating using FreeRTOS alone, we set the time axis to 2.50 μs. Since the interval is 0.50 μs, the LOW, and when the top waveform changes from LOW to HIGH (or HIGH to LOW). When interrupt latency is measured to be approximately 3.00 μs on average. operating using FreeRTOS alone, we set the time axis to 2.50 μs. Since the interval is 0.50 μs, the interrupt latency is measured to be approximately 3.00 μs on average.

Sustainability 2017, 9, 684 11 of 14

The taskYIELD()function yields the CPU to other processes, which is in the ready queue. This code does not actually create tasks, but simply determines the execution time of the portions of the code that are not part of the task-switching measurement. The execution of this code segment is recorded with the oscilloscope and the second portion of the code runs. The second portion utilizes two tasks. This time the two tasks perform task switching for the desired number of iterations. The result of measuring the interrupt latency is as follows: Among the two waveforms shown

Sustainabilityin the ﬁgure, 2017 the, 9, 684 upper waveforms are alternating between Task 1 and Task 2 and output HIGH11 of and 14 LOW states. The waveform below, in Figures 19 and 20, are generated from the waveform for the interrupt handler. We calculated the time from when the bottom waveform changes from HIGH to LOW, and when the top waveform changes from LOW to HIGH (or HIGH to LOW). When operating using FreeRTOS alone, we set the time axis to 2.50 µs. Since the interval is 0.50 µs, the interrupt latency isSustainability measured 2017 to, be9, 684 approximately 3.00 µs on average. 11 of 14

Figure 19. Interrupt latency on a single RTOS execution.

When Linux and FreeRTOS work together through the QPlus Hypervisor, we set the time axis of the oscilloscope to 250 μs. Since the scale on an oscilloscope is 50 μs, the interrupt latency was measured to be approximately 150 μs on average. Based on the measured results, FreeRTOS operating with Linux throughFigure the19. InterruptQPlus Hypervisor latency on a showedsingle RTOS that execution. the task switching delay time is about 1140 times and the interrupt latency is about 50 times slower than when it is performed alone. When Linux and FreeRTOS work together through the QPlus Hypervisor, we set the time axis of the oscilloscope to 250 μs. Since the scale on an oscilloscope is 50 μs, the interrupt latency was measured to be approximately 150 μs on average. Based on the measured results, FreeRTOS operating with Linux through the QPlus Hypervisor showed that the task switching delay time is about 1140 times and the interrupt latency is about 50 times slower than when it is performed alone.

Figure 20. InterruptInterrupt latency latency on on multiple OS (RTOS + Linux) Linux) execution.

BasedWhen Linuxon the and experimental FreeRTOS work results, together we can through deduce the QPlusthe following. Hypervisor, There we setis theno timesignificant axis of difference in the task transition delay time and interrupt latency time when FreeRTOS operates the oscilloscope to 250 µs. Since the scale on an oscilloscope is 50 µs, the interrupt latency was measured alone.to be approximatelyHowever, the FreeRTOS 150 µs on operating average. Basedon the on QPlus the measuredHypervisor results, shows FreeRTOS a large difference operating in withtask transitionLinux through delay the time.Figure QPlus Additionally, 20. Hypervisor Interrupt the latency showed interrupt on thatmultiple latency the taskOS is (RTOS slowed switching + Linux) down. delay execution. time is about 1140 times and theFigure interrupt 21 show latency our experimental is about 50 times results slower in wh thanich when there itare is performedtwo digital alone. timer boards and they are suppliedBased onon a the clockthe experimental experimentalsignal from results, two results, different we canwe digita deducecan deducel timer the following. boards,the following. respectively. There is There nosigniﬁcant They is nostart significant differencecounting theindifference the digital task transitionintimers the taskat delaythe transition same time time, and delay interruptbut time the timiand latencyngs interrupt are time skewed when latency FreeRTOS occasionally time when operates due FreeRTOS to alone. the operating However,operates systemalone. However, delays [19]. the FreeRTOS operating on the QPlus Hypervisor shows a large difference in task transition delay time. Additionally, the interrupt latency is slowed down. Figure 21 show our experimental results in which there are two digital timer boards and they are supplied a clock signal from two different digital timer boards, respectively. They start counting the digital timers at the same time, but the timings are skewed occasionally due to the operating system delays [19].

Sustainability 2017, 9, 684 12 of 14 the FreeRTOS operating on the QPlus Hypervisor shows a large difference in task transition delay time. Additionally, the interrupt latency is slowed down. Figure 21 show our experimental results in which there are two digital timer boards and they are supplied a clock signal from two different digital timer boards, respectively. They start counting the digital timers at the same time, but the timings are skewed occasionally due to the operating system delaysSustainability [19]. 2017, 9, 684 12 of 14

(a) (b)

Figure 21. Digital timer board execution by clock generated from RTOS and Linux, respectively. Figure 21. Digital timer board execution by clock generated from RTOS and Linux, respectively. Note: (a) at time 00:17; (b) at time 00:55; (c) at time 02:27; (d) at time 03:10 Note: (a) at time 00:17; (b) at time 00:55; (c) at time 02:27; (d) at time 03:10.

Importantly, the Cortex-A15 processor supports interrupt virtualization (GICv2). This reduces the hypervisor’sImportantly, interrupt the Cortex-A15 processing processor overhead supports by virtualizing interrupt virtualization interrupts that (GICv2). occur in This each reduces guest theOS. hypervisor’sDue to these interrupt advantages, processing it can be overhead seen that by the virtualizing interrupt interrupts latency is thatmeasured occur in more each quickly guest OS. in Duethe toguest these operating advantages, system it can beand seen FreeRTOS that the interruptoperation latency than isthe measured hypervisor more shown quickly inabove. the guest The operatingfollowing systemproblems and exist FreeRTOS when guest operation OSs thanrun throug the hypervisorh hypervisor shown that above. do not The guarantee following a problemsreal-time existproperty. when Due guest to OSs the runscheduling through hypervisorof the hypervis that door, notthe guaranteeRTOS does a real-timenot match property. the time Due actually to the schedulingoperated and of the hypervisor,time on the thesystem. RTOS Therefore, does not matchthe system the time causes actually a slight operated performance and the degradation, time on the system.as shown Therefore, in Table 1. the system causes a slight performance degradation, as shown in Table1.

Table 1. Experimental results. Table 1. Experimental results.

SingleSingle RTOS(FreeRTOS) RTOS(FreeRTOS) LinuxLinux and and FreeRTOS FreeRTOS Execution Simultaneously Execution Simultaneously Task switching delay 3.5 µs 4.0 ms Task switching delay 3.5 μs 4.0 ms Interrupt delay 3.0 µs 150 µs Interrupt delay 3.0 μs 150 μs

Sustainability 2017, 9, 684 13 of 14

5. Conclusions There is a problem that RTOS performance is degraded due to overhead. Thus, we need to compare the performance between a single execution of the RTOS and a simultaneous execution of multiple OSs (RTOS + Linux). Therefore, in this paper, we measured the performance when the RTOS operates independently on the NVidia Jetson TK-1 embedded board supporting virtualization technology. Then, we measured the performance when RTOS and Linux are operated simultaneously on top of a hypervisor. To this end, we implemented and ported such a RTOS, especially FreeRTOS and uC/OS, onto two embedded boards, such as an Arndale board and NVidia TK1 board. There are such problems when multiple guest OSs are running on hypervisor that do not guarantee a real-time property. Due to the scheduling of the hypervisor, the RTOS does not match the time actually operated and the time on the system. Therefore, the system causes a slight performance degradation in a different order of measurement time.

Acknowledgments: This work was jointly supported by a Project no. K-17-L01-C01 of Korea Institute of Science and Technology Information, and supported by a Basic Researcher Project No. NRF-2016R1C1B1012189 of National Research Foundation. This work was also jointly supported by a Startup Growth Technology Development Project of Small and Medium Business Administration in Korea. Author Contributions: For the research reported, ByoungWook Kim conceived of and designed the overall architecture and conducted the experiment. Min Choi is the corresponding author, and performed the overall design and the source code implementation for uC/OS II and FreeRTOS porting. Conﬂicts of Interest: The authors declare no conﬂict of interest.

References

1. Kihong, L.; Dongwoo, L.; Young, I.E. A Study on Trend for Hardware-assisted Virtualization Technology. In Proceedings of the Korean Information Science Society, Jeju, Korea, 15–16 November 2013; pp. 1316–1318. 2. Young, G.J.; Byung, J.L.; Hee, Y.Y. The comparative analysis of Hypervisor according to Virtualization Method. In Proceedings of the Korean Society of Computer Information Conference, Cheongju, Korea, 22–24 January 2015; pp. 251–253. 3. Travis, L. Exploring the Design of the Cortex-A15 Processor. Available online: https://www.arm.com/files/ pdf/AT-Exploring_the_Design_of_the_Cortex-A15.pdf (accessed on 15 April 2017). 4. In-Kyu, H. K-Hypervisor: Design and Implementation of Hypervisor Based ARM Cortex-A15. Available online: http://www.dbpia.co.kr/Journal/ArticleDetail/NODE02444451?TotalCount=0&Seq=4& isIdentifyAuthor=1&Collection=0&isFullText=0&specificParam=0&SearchMethod=0&Page=1&PageSize= 20 (accessed on 15 April 2017). 5. Prashant, V.; Gernot, H. Hardware-supported virtualization on ARM. In Proceedings of the Second Asia-Pacific Workshop on Systems, Shanghai, China, 11–12 July 2011. 6. FreeRTOS Project. Available online: http://www.freertos.org (accessed on 15 April 2017). 7. Rabindra, P.K.; Kent, P. Rhealstone: A Real-Time Benchmarking Proposal. Available online: http:// collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/1989/8902/8902a/8902a.htm (accessed on 15 April 2017). 8. ARM Generic Interrupt Controller Architecture Specification. Available online: http://www.cl.cam.ac.uk/ research/srg/han/ACSP35/zynq/arm_gic_architecture_specification.pdf (accessed on 15 April 2017). 9. Simon, D. OS Porting & Analysis for Dual Core ARM Cortex-A9 Based Systems. Available online: http://www.imperas.com/documents/ARM_TechCon_2012_Imperas_Paper_OS_Porting_and_ Analysis_for_Cortex_A9MP_Based_Systems.pdf (accessed on 15 April 2017). 10. Introduction to Basic RTOS Features Using SAM4L-EK FreeRTOS Port. Available online: http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-42247-Introduction-to-Basic%20RTOS- Features-using-SAM4L-EK-FreeRTOS-Port_Training-Manual.pdf (accessed on 15 April 2017). Sustainability 2017, 9, 684 14 of 14

11. Sung-Min, L.; Sang-Bum, S.; Bokdeuk, J.; Sangdok, M.; Brian Myungjune, J.; Jung-Hyun, Y.; Jae-Min, R.; Dong-Hyuk, L. Fine-grained I/O access control of the mobile devices based on the Xen architecture. In Proceedings of the 15th Annual International Conference on Mobile Computing and Networking, Beijing, China, 20–25 September 2009. 12. Gernot, H.; Ben, L. The OKL4 Microvisor: Convergence Point of Microkernels and Hypervisors. Available online: https://ts.data61.csiro.au/publications/papers/Heiser_Leslie_10.slides.pdf (accessed on 15 April 2017). 13. Gernot, H. Hypervisors for consumer electronics. In Proceedings of the 6th IEEE Conference on Consumer Communications and Networking Conference, Las Vegas, NV, USA, 11–13 January 2009. 14. ARM Cortex-A15. Available online: https://en.wikipedia.org/wiki/ARM_Cortex-A15 (accessed on 15 April 2017). 15. Francois, A.; Michel, G. A Practical Look at Micro-Kernels and Virtual Machine Monitors. In Proceedings of the 6th IEEE Conference on Consumer Communications and Networking Conference, Las Vegas, NV, USA, 10–13 January 2009. 16. François, A.; Michel, G.; Gilles, M.; Gregory, M. Shared device driver model for virtualized mobile handsets. In Proceedings of the First Workshop on Virtualization in Mobile Computing (MobiVirt’08), Breckenridge, CO, USA, 17 June 2008. 17. Keir, F.; Steven, H; Rolf, N.; Ian, P.; Andrew, W.; Mark, W. Safe Hardware Access with the xen Virtual Machine Monitor. Available online: http://www.cl.cam.ac.uk/research/srg/netos/papers/2004-oasis-ngio.pdf (accessed on 15 April 2017). 18. Anderson, T.E.; Bershad, B.N.; Lazowska, E.D.; Levy, H.M. Scheduler Activations: Effective Kernel Support for the User-Level Management of Parallelism. Available online: http://ﬂint.cs.yale.edu/cs422/doc/sched- act.pdf (accessed on 15 April 2017). 19. Video Clip for the Digital Timer Board in Experimental Results Section. Available online: https://youtu.be/ 4wMuyhlW97o (accessed on 15 April 2017).

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).