Bring-Up Bhyve on Espressobin Board 1
Total Page:16
File Type:pdf, Size:1020Kb
U.P.B. Sci. Bull., Series C, Vol. 83, Iss. 1, 2021 ISSN 2286-3540 BRING-UP BHYVE ON ESPRESSOBIN BOARD Andrei-Costin Martin1, Darius Mihai2, Maria-Elena Mihailescu3, Mihai Carabas4, Sergiu Weisz5 In order to be sustainable, the cloud architectures must have good scalability and throughput. On the processors market, ARM started to show in- terest for servers and desktop processors. x86 and x86 64 CPU market have a strong opponent, with a lower power consumption and comparable performances. A solution to the scalability and throughput requirements could be a virtualized environment. One example is the FreeBSD’s hypervisor, bhyve, mainly because FreeBSD is often used in server environments, has a strong TCP/IP stack imple- mentation. bhyve’s implementation for ARM was not accepted in the mainline, due the lack of testing on real hardware. This paper presents the steps taken in order to bring-up the hypervisor on a development board, called ESPRESSObin. Keywords: virtualization, ARMv8, bhyve, FreeBSD, EspressoBin, bring-up 1. Introduction Nowadays, there is a great inclination towards Internet, cloud and streaming. These topics raise the issue of scalability in order to obtain a high network traffic from all over the world. One solution can be virtualization. Using virtual machines, all the servers can be duplicated and deployed everywhere, in a safe environment to accept connections. To manage these virtual machines, a hypervisor is needed. The hypervisor has the role of starting, running and managing the execution of vir- tual machines. These cloud and streaming markets saw continuous growth over the last decade, alongside the virtualization solutions in order to increase the ex- change speed of information. Besides that, virtualization aims to the best hardware resources usage and to increase the throughput. At the same time, ARM started to take an interest for the server and desktop processors market. That means that x86 and x86 64 CPUs will have a strong rival, one with a better energy efficiency and 1Master Student, Faculty of Automatic Control and Computer Science, University Politehnica of Bucharest, Romania, e-mail: [email protected] 2Teaching Assistant, Faculty of Automatic Control and Computer Science, University Po- litehnica of Bucharest, Romania, email: [email protected] 3Teaching Assistant, Faculty of Automatic Control and Computer Science, University Po- litehnica of Bucharest, Romania, email: [email protected] 4Associate Professor, Faculty of Automatic Control and Computer Science, University Po- litehnica of Bucharest, Romania, email: [email protected] 5PhD Candidate, Faculty of Automatic Control and Computer Science, University Politehnica of Bucharest, Romania, email: [email protected] 65 66 Andrei-Costin Martin, Darius Mihai, Maria-Elena Mihailescu, Mihai Carabas, Sergiu Weisz comparable performances. This paper presents the steps taken in the ARMv8 bring-up process in order to run a virtual machine using FreeBSD’s bhyve, both in the ARMs Foundation Platform emulator[3] and ESPRESSObin[4], an off-the-shelf board developed by Marvell. The reason for choosing FreeBSD’s hypervisor is because FreeBSD is very often used as operating system on serves, due to its TCP/IP stack implementation and its stability and, with virtualization, would solve the scalability issue. Regarding the virtualization, an ARM implementation for bhyve [2] is not yet available in the FreeBSD’s mainline. There are implementations for ARMv7 and ARMv8 but not yet accepted by the maintainers. The main goal is to have a fully operational hypervisor, that can run virtual ma- chines in the emulator and on the board. After being able to run the hypervisor, it need to be tested, using FreeBSD, Linux and Windows as guests operating systems. The last step is the hypervisor code redesign, both in user-space and kernel-space, requested by the maintainers, in order to accept the code in the FreeBSD’s mainline. 2. State of the art In this section, we will present information regarding the virtualization, FreeBSD and its hypervisor, bhyve, we present a comparison between ARMv7 and ARMv8 and the components used for deploying FreeBSD on a board. 2.1. FreeBSD and bhyve FreeBSD [1] is an operating system designed to run on desktops, servers and embedded platforms. Despite the wide range of compatibility, FreeBSD’s main tar- get is the server market (e.g. web servers or email servers). BSD operating systems have the advantage of stability, offered by a small number of updates that can dam- age the system. Besides that, their implementation of the TCP/IP [7] stack makes this operation system the best for servers that have a huge number of concurrent connections. bhyve is a type-two hypervisor, FreeBSD’s solution regarding the virtualiza- tion problem. It supports a wide range of guest operating systems, such as FreeBSD, Linux or Windows and is available only for FreeBSD on x86 [8] CPU architec- ture. Its implementation contains three user space utilities, bhyveload, bhyve and bhyvectl, which communicate with the kernel driver, vmm, through libvmmapi li- brary. These three utilities have the following roles: bhyveload that has the role of loading a new guest virtual machine, bhyve that runs a virtual machine with a guest operating system, and bhyvectl that has the role of controlling bhyve virtual machines, by creating, destroying or getting and changing statistics/preferences. 2.2. ARMv7 vs ARMv8 A brief comparison between ARMv8 and ARMv7 can be found in Table 1. We can see that one of the improvements is the increased address space, from 32 bits to 64 bits. There is only one exception, the instruction set is 32 bits in both Bring-up bhyve on ESPRESSObin board 67 ARMv7 and ARMv8. That means that the encoding of instructions is not changed. Based on the registers size, ARMv7 and ARMv8 are also called arm32 (or arm) and arm64. TABLE 1. Comparison between ARMv7 and ARMv8 ARMv7 ARMv8 General-purpose registers 13 x 32bit 31 x 64bit Program Counter 32bit 64bit Stack Pointer 32bit 64bit Exception Link Register 32bit 64bit Virtual addressing Support for 32bit Support for 64bit Instruction Set 16/32bit 32bit This new version of the ARM processor, ARMv8 [5], defines two ’Execu- tion States’ [6]: AArch64 and AArch32. AArch64 state uses all registers as 64bit registers, compared to AArch32, which uses them as 32bit registers, to provide backwards compatibility. This is the reason why an ARMv7 application can also work well on ARMv8 CPUs. Besides Execution States, presented in Subsection 2.2, ARMv8 [5] has a total of 4 Execution levels [6]. While the first two levels, EL0 and EL1, represent where the user program and kernel are usually running on any system, the third level, EL2, will be the place where the virtual machine is running. As a requirement, Kernel and Virtualization spaces must communicate with each other. The last level is divided in Secure State where the processor can access the entire memory address space and can access all system control resources, and Non-Secure State that specifies that the CPU can access only the Non-Secure address space and cannot access the system control resources. 2.3. ESPRESSObin, Device Trees and U-Boot In the bring-up phase, this project used an off-the-shelf board, called ESPRES- SObin v5. We chose this board because it has an ARMv8 Cortex-A53 CPU, the same CPU as one version of the ARMs Foundation Platform emulator. The follow- ing lists presents the board components SATA HDD data cable – the data transfer port for a SATA peripheral; SATA HDD power – the power supply for a SATA connected peripheral; textbfMicro USB port (J5) – is a serial (UART) connection; 10-pin ARM JTAG – 10 JTAG pins, dedicated to debugging; 12V DC Jack – is the power supply port; GPIO – General Purpose Input/Output pins, USB 2.0 and 3.0 ports – Two USB ports for peripherals; GbE LAN ports – Two LAN Ethernet ports; WAN port – A WAN port; J3 connector – Three jumpers representing the ESPRESSObin boot mode; Reset switch – A button which resets the board. 68 Andrei-Costin Martin, Darius Mihai, Maria-Elena Mihailescu, Mihai Carabas, Sergiu Weisz Device Trees[9], in the embedded systems, are one of the most important things, describing the hardware specification of a board. Due to the fact that the components (e.g. memory, CPU) are glued to the motherboard, a board specifica- tion is static. It is represented by a human-readable file, containing all the spec- ifications of a board, that will go through a compilation process. The compiled binary, called Device Tree Blob (DTB), will be loaded by the associated board, via a bootloader. U-Boot1 is an open source bootloader, designed for embedded systems. It is versatile, being able to accept FAT32, ext4, even UFS (in newer versions) filesystem formats for different types of storage, such as usb or mmc for MicroSD cards. It also lets the user configure environment variables and command, for example, a variable that contains the path to the Device Tree Blob. For this project, we used a MicroSD card, formatted as FAT32, to store the operating system. This MicroSD card contained the FreeBSD operating system, alongside the DTB. In the compile process, the Marvell’s U-Boot repository was used, found on the wiki page [4], in the Build from source - Bootloader section. When the U-Boot binary has to be made as a valid payload for ARM’s Trusted Firmware (ATF), some values are used, presented in Table 2 . The values are used to set three important variables at the make command, describing the board specifi- cation (CPU and memory frequencies, memory topology and the boot device). TABLE 2. CPU and DDR presets Value CPU frequency (MHz) DDR frequency (MHz) CPU 600 DDR 600 600 600 CPU 800 DDR 800 800 800 CPU 1000 DDR 800 1000 800 3.