Linux Virtual Machine Preparation Guide

For protecting virtual machines to Azure

Version 2.0 Update 0520

Table of Contents

ABOUT THIS GUIDE ...... 2 END STATE OPERATIONS GOALS ...... 2 HOW TO PREPARE A CENTOS 6X OR REDHAT 6X-BASED VIRTUAL MACHINES ...... 3 Assumptions ...... 3 Step 1: Modify Grub Options ...... 3 Step 2: Configure agetty serial console ...... 4 Step 3: Add HyperV drivers to the config ...... 4 Step 4: Check to see if Hyper-V modules are loaded ...... 5 Step 4: Add persistent rule for Azure Nics ...... 5 Step 5: Creating an ifcfg-eth file for the Azure NIC: ...... 6 HOW TO PREPARE A CENTOS 7X OR REDHAT 7X-BASED VIRTUAL MACHINES ...... 8 Assumptions ...... 8 Step 1: Modify Grub Options ...... 8 Step 2: Adding Hyper-V drivers to the initramfs ...... 9 Step 3: Check to see if Hyper-V modules are loaded ...... 10 Step 4: Configuring Networking ...... 10 Step 5: Creating an ifcfg-eth file for the Azure nic ...... 11 HOW TO PREPARE AN OR -BASED VIRTUAL MACHINE ...... 14 Step 1: Create an Azure Netplan profile ...... 14 HOW ZERTO ASSIGNS STATIC IP ADDRESSES TO VMS IN AZURE ...... 16 Step 1: Create or edit the VPG ...... 16 Step 2: Assign the Static IP Address ...... 16 Step 3: Failover Test the Virtual Machine ...... 17 VERSIONS TESTED ...... 18 REFERENCES ...... 19 ENDNOTES ...... 19

Linux Virutal Machine Configuration Guide for Azure 1 of 20

About this Guide Zerto enables migrations and disaster recovery (DR) capabilities between multiple clouds. For the most part, the experience is seamless to the virtual machines (VMs), and there is no impact on the VMs while moving between on-premises VMware or Hyper-V environments and Azure.

Zerto can move any distribution or version of Linux to Azure. To see if the version you have is supported, check the Microsoft documentation. However, when many Linux VMs move to Azure, they may not have the right drivers and will not have network connectivity. Additionally, it is not unusual for older versions of Linux to still be in production, so you need to configure the servers to re-platform between the sites and have specific platform profiles.

This guide is for Linux administrators who want to configure their Linux virtual machines to be able to move between Azure and VMware on-premises sites.

End State Operations Goals DR and migrations using Zerto are effectively re-platforming the Linux virtual machines with any re- platform operation, the challenges are: 1. Adding the necessary drivers to work correctly on-premises and in Azure 2. Automatically detecting and automatically handle the different virtual machine hardware profiles of the underlying platform. 3. Assigning the proper IP Address(s) to the VM

Linux Virutal Machine Configuration Guide for Azure 2 of 20

How to prepare a CentOS 6x or RedHat 6x-based Virtual Machines

Assumptions The following steps assume an on-premises VMware Linux VM with Azure as the target.

If you are only looking to use Azure as a Disaster Recovery platform, and plan to continue running your VM’s in VMware or HyperV, these steps simply prepare a VM to run in Azure. If that never happens, the modification will remain harmless while running on VMware. It is significantly easier to do this before a failover than it is after.

On the production virtual machine, the one running in VMware that you are planning to failover or move to Azure, you will want to do the following steps.

Step 1: Modify Grub Options These steps will enable a serial console so if something goes wrong you can leverage the Microsoft Serial Console for troubleshooting, you should NOT skip this step.

Use VI or nano to edit /boot/grub/grub.conf

Look for the line that starts with “kernel” under your default Kernel boot option, see screenshot below

Usually the default kernel is the first entry. Look for the kernel line.

Linux Virutal Machine Configuration Guide for Azure 3 of 20

Remove the following options:

rhgb quiet crashkernel=auto These options told grub to use the graphical bootloader and to use verbose kernel output to the console. Microsoft says they are not used for public cloud VMs. Crashkernel consumes about 128MB of ram, it can be left according to Microsoft, but they recommend removing it to save RAM.

Now add these options:

rootdelay=300 console=ttyS0 console=tty0 earlyprintk=ttyS0

These options tell grub to wait for 300 seconds for the root volume and turns on both the local console and serial console for boot output messages.

Save and exit your text editor.

Step 2: Configure agetty serial console The configuration above will enable all of the kernel boot debug to go to the serial console. If you want an interactive login prompt to also be on the serial console after boot up, you need to do this step too.

Use a text editor to create a file /etc//ttyS0.conf

Then paste in the following text:

#This service maintains a agetty on ttyS0. stop on [S016] start on [23] respawn exec agetty -h -L -w /dev/ttyS0 115200 vt102

Step 3: Add HyperV drivers to the dracut config Using your favorite text editor, edit /etc/dracut.conf

Paste the following line into the file. (Make sure to leave the spaces before and after the driver names)

add_drivers+=" hv_vmbus hv_netvsc hv_storvsc nvme ena xen_blkfront xen_netfront mptbase mptscsih mptspi "

Update the initramfs by running the following command.

dracut --force -v

Linux Virutal Machine Configuration Guide for Azure 4 of 20

If possible, it is a good idea to reboot the machine to make sure everything is working ok.

Step 4: Check to see if Hyper-V modules are loaded If you are able to reboot the VM after running all the commands above, you should check to see if the Hyper-V modules have been loaded.

lsinitrd | grep hv

You should see something like the following:

All three HyperV kernel modules are now loaded

If those commands worked you should be ready for failover or migration. Perform a Zerto test failover into Azure just to confirm, and repeat the failover test after each kernel upgrade just to make sure all is still working well.

Step 4: Add persistent rule for Azure Nics Next edit /etc//rules.d/70-persistent-net.rules

We want to add a new line that will map the network card using the HyperV driver to an unused “eth” name. I will use eth99, but you can use whatever you would like as long as its not already used.

Add this line at the bottom of the file:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="hv_netvsc", ATTR{type}=="1", KERNEL=="eth*", NAME="eth99"

(If you are using nano, I recommend just copying the line that is already in this file for eth0, and then modifying it to look like the line above. Doing it that way will eliminate Copy/Paste errors of Windows text to Linux text.)

What it should look like now (assuming you only had one rule to start):

Mapping Azure nic to eth99 Linux Virutal Machine Configuration Guide for Azure 5 of 20

Now we can create an ifcfg-eth99 file and set it up for DHCP.

NOTE: If you want to be able to automatically fail back from Azure, and have the VMware NIC reattach to your original eth0 configuration you need to also edit the top line. You need to remove the ‘ATTR(address)==”YOUR:MAC:ADDRESS”, ‘

This will allow non-Azure nics to be eth0.

Step 5: Creating an ifcfg-eth file for the Azure NIC: Create a new file at /etc/sysconfig/network-scripts/ifcfg-eth99

Then add the following to the file:

NAME="eth99" DEVICE="eth99" BOOTPROTO=dhcp ONBOOT=yes

Make sure that the ” ” marks look like the file below. If they do not, manually type them. Sometimes Copy/Paste from a browser does funny things to text in Linux.

It should look like this:

eth99 config

Note: If you want to automatically fail back from Azure and have the VMware NIC reattach to eth0, you will need to edit /etc/sysconfig/network-scripts/ifcfg-eth0 as well.

Find the line that starts with “HWADDR=” and remove the entire line.

Step 6: Enable SSH In Azure you only have access to a serial console, and via the network. So we recommend using SSH. service sshd start chkconfig sshd on

Linux Virutal Machine Configuration Guide for Azure 6 of 20

Step 7: Testing in Azure Once you have your machine prepped, create a VPG to Azure, and then initiate a test failover. After the VM is listed in your Virtual Machine inventory click it and go to Boot Diagnostics then the Serial Log tab.

Azure Serial Log output, available since we turned on the serial console

You can scroll through the serial output, and look for any issues. Assuming routing is setup between where you are and your VM and proper network security rules are configured, you should also now be able to SSH into it, or interact with other services that it has running.

Step 8: Next Steps You will probably notice that we do not install the WAAgent into the VM, it is technically not needed to get the VM up and running in Azure. Therefore, it can be installed later after you have migrated or failed over. For a Zerto customer who only plans to use Azure while their production site is offline, there may be no reason to install it at all.

Linux Virutal Machine Configuration Guide for Azure 7 of 20

How to prepare a CentOS 7x or RedHat 7x-based Virtual Machines

Assumptions The following steps assume an on-premises VMware Linux VM with Azure as the target.

If you are only looking to use Azure as a Disaster Recovery platform, and plan to continue running your VM’s in VMware or HyperV, these steps simply prepare a VM to run in Azure. If that never happens, the modification will remain harmless while running on VMware. It is significantly easier to do this before a failover than it is after.

On the production virtual machine, the one running in VMware that you are planning to failover or move to Azure, you will want to do the following steps.

Step 1: Modify Grub Options These steps will enable a serial console so if something goes wrong you can leverage the Microsoft Serial Console for troubleshooting. We recommend you not skip this step.

Use VI or nano to edit /etc/default/grub.

Look for the line that starts with “GRUB_CMDLINE_LINUX”

Remove the following options

rhgb quiet crashkernel=auto

These options configure grub to use the graphical bootloader and to use verbose kernel output to the console. Microsoft says they are not used for public cloud VMs. Crashkernel consumes about 128MB of ram, it can be left according to Microsoft, but they recommend removing it to save RAM.

Now add these options:

rootdelay=300 console=ttyS0 console=tty0 earlyprintk=ttyS0

These options tell grub to wait for 300 seconds for the root volume and turns on both the local console and serial console for boot output messages. Save and exit your text editor.

Now run this command to update Grub.

grub2-mkconfig -o /boot/grub2/grub.cfg

Linux Virutal Machine Configuration Guide for Azure 8 of 20

Step 2: Adding Hyper-V drivers to the initramfs For Linux to work properly in HyperV as well as Azure you need three modules: hv_bus, hv_netvsc, and hv_storvsc.

To install these into a VM that is running on VMware we need to modify the dracut config file and then run dracut to update the initramfs.

Using VI or nano edit /etc/dracut.conf Add the following line into the file then save the file. add_drivers+=" hv_vmbus hv_netvsc hv_storvsc nvme ena xen_blkfront xen_netfront mptbase mptscsih mptspi "

It is essential to preserve the spaces as well as using the proper ” ” marks. If you use the Azure configuration document, the double quotes that they use are misinterpreted and will cause the drivers not to be loaded.

Now run dracut with the following command.

dracut -force -v

Note: initramfs note and tip The dracut command above will rebuild initramfs for the currently running kernel version. HOWEVER! If you ran yum -y update above and it installed a new kernel, and you haven't rebooted, dracut will build for the WRONG kernel. (It will build for the currently running kernel… not the new kernel that will be used on the next reboot)

You have two options. 1.) reboot the machine before running the dracut command 2.) tell dracut which version of the kernel to build for If rebooting is not an option, we recommend to check your grub file to see what version of the kernel is installed.

cat /boot/grub2/grub.cfg | grep initrd16

Your output will be something like this:

Linux Virutal Machine Configuration Guide for Azure 9 of 20

Output showing kernel versions in grub2 as well as the current running kernel

If the kernel version from the uname command (3.10.0-862.9.1.e17.x86_64 in the example) is also listed as the top line from the first command, dracut will build the initramfs for the proper kernel. If they are not the same then you can force dracut to build for the default kernel by using this command (modify as needed for your kernel)

dracut -f /boot/initramfs-3.10.0-862.el7.x86_64.img 3.10.0-862.el7.x86_64

If you don’t build for the default kernel, your failover test won’t work as you won’t have the Hyper-V drivers in the default kernel.

Step 3: Check to see if Hyper-V modules are loaded If you can reboot the VM after running the dracut command to update initramfs, you should then check to see if the Hyper-V modules have been loaded.

lsinitrd | grep hv

You should see something like the following:

All three HyperV kernel modules are now loaded. If those commands worked you should be ready for failover or migration. We recommend performing a test failover into Azure just to verify, and running a test failover after each kernel upgrade just to make sure all is still working well is a good practice.

Step 4: Configuring Networking Now that the drivers are loaded into the initramfs, next is the network configuration.Azure (and AWS for that matter) require VMs to get their IP address via a DHCP request. So if you want a static address, you can do that via Zerto’s VPG settings, but the VM will still need to be configured like it is getting a DHCP leased address.

Most VMware VMs will use ensXXX for VMware NICs, you can check what your’s is by typing ifconfig (as root or sudo).

Linux Virutal Machine Configuration Guide for Azure 10 of 20

VMware NIC on RHEL 7.4

Azure nics will show up as ethX, most commonly eth0. So you can either assume that it will always be eth0, or you can modify some network rules to make it something more reliable like eth99 (or whatever you would like).

To do that edit /etc/udev/rules.d/70-persistent-net.rules

You may need to create this file if it doesn't exist. Add this line at the bottom of the file:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="hv_netvsc", ATTR{type}=="1", KERNEL=="eth*", NAME="eth99"

What it should look like now (assuming you only had one rule to start):

Mapping Azure nic to eth99

Now we can create an ifcfg-eth99 file and set it up for DHCP.

Step 5: Creating an ifcfg-eth file for the Azure nic Create a new file at /etc/sysconfig/network-scripts/ifcfg-eth99 Then add the following to the file: NAME="eth99" DEVICE="eth99" BOOTPROTO=dhcp ONBOOT=yes

It should look like this:

Linux Virutal Machine Configuration Guide for Azure 11 of 20

eth99 config Step 6: Enable ssh In Azure, you only have access to a serial console, and via the network. So if you don’t already have the ssh server turned on, it is a recommended configuration for troubleshooting.

Run the following commands:

service sshd start chkconfig sshd on

Step 7: Enabling a Serial Console RHEL 7x has an example getty file that we can copy over and enable. To do that run the following commands.

cp /usr/lib//system/[email protected] /etc/systemd/system/serial- [email protected]

Next Symlink it to the systemd “wants” directory

ln -s /etc/systemd/system/[email protected] /etc/systemd/system/getty.target.wants/

Next we will start, and enable the new console. Note, on VMware this won’t do anything, but when you failover to Azure you will be able to use this to login if networking isn’t working.

systemctl -reload systemctl start [email protected] systemctl enable [email protected]

Step 8: Testing in Azure Once the virtual machine is prepped, create a Zerto VPG to Azure, and then kick off a test failover. After the VM is listed in your Virtual Machine inventory, click it and go to Boot Diagnostics then the Serial Log tab

Linux Virutal Machine Configuration Guide for Azure 12 of 20

Azure Serial Log output, available since we turned on the serial console

You can scroll through the serial output, and look for any issues.

Step 9: Next Steps You will probably notice that we do not install the WAAgent into the VM, it is technically not needed to get the VM up and running in Azure. Therefore, it can be installed later after you have migrated or failed over. For a Zerto customer who only plans to use Azure while their production site is offline, there may be no reason to install it at all.

Linux Virutal Machine Configuration Guide for Azure 13 of 20

How to prepare an Ubuntu or Debian-based Virtual Machine

With the release of 18.04LTS, network configuration was moved over to Netplan, from /etc/init.d/network config file.

Netplan reads network configuration from /etc/netplan/*.yaml which are written by administrators, installers, cloud image instantiations, or other OS deployments. During early boot, Netplan generates backend specific configuration files in /run to hand off control of devices to a particular networking daemon.

Netplan currently works with these supported renderers • NetworkManager • Systemd-networkdi

When the VM is on VMware the network adapter type is typically ens160 or ens32. However, the Azure network card shows up as eth0. In order for the machine to pull a dhcp assigned address you will need to add a netplan YAML file that sees the network adapters that are hyperv adapters and tells them to use DHCP.

Step 1: Create an Azure Netplan profile

On your Ubuntu 18.04+ LTS machine, change the working directory to /etc/netplan:

cd /etc/netplan/

Next we want to create/edit a new file called “90-hotplug-azure.yaml”

nano 90-hotplug-azure.yaml

Linux Virutal Machine Configuration Guide for Azure 14 of 20

Paste in the following information:

# This netplan yaml is delivered in Azure cloud images to support # attaching and detaching nics after the instance first boot. # Cloud-init otherwise handles initial boot network configuration in # /etc/netplan/50-cloud-init.yaml network: version: 2 ethernets: ephemeral: dhcp4: true match: driver: hv_netvsc name: '!eth0' optional: true hotpluggedeth0: dhcp4: true match: driver: hv_netvsc name: 'eth0'

Next save the file.

Modify grub, and turn on the serial console, which is invaluable if you need to troubleshoot in Azure.

Edit the /etc/default/grub file and insert the following:

GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300"

Then update grub with the following command: update-grub

Now you can use Zerto to do a test failover of the VM, and you should be able to access the test VM in Azure.

Linux Virutal Machine Configuration Guide for Azure 15 of 20

How Zerto assigns Static IP addresses to VMs in Azure Production servers are commonly configured with static IP addresses. When moving to Azure, the IP addresses are dynamically assigned.

Zerto has a feature that allows you to assign the static IP address to the Linux server when it powers on in Azure. During the Virtual Protection Group (VPG) configuration, assign the static IP address you want the Azure VM to have.

Step 1: Create or edit the VPG

Step 2: Assign the Static IP Address

Linux Virutal Machine Configuration Guide for Azure 16 of 20

Step 3: Failover Test the Virtual Machine In the Azure portal, verify that the failed over virtual machine has the correct IP address and it is marked as a Static IP.

Linux Virutal Machine Configuration Guide for Azure 17 of 20

Linux Versions Tested Linux Version Working Not Working Untested Centos 6.5.6.6, 6.7, 6.8, Yes 6.9, 6.10 Redhat 6.5.6.6, 6.7, Yes 6.8, 6.9, 6-10 Centos 7.0-7.10 Yes Redhat 7.0-7.10 Yes Ubuntu 18.04, 18.10 Yes

Linux Virutal Machine Configuration Guide for Azure 18 of 20

References Preparing Ubuntu Machines for Azure: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-ubuntu

Preparing CentOS Machines for Azure https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-

Preparing RedHat Machines for Azure https://docs.microsoft.com/en-us/azure/virtual-machines/linux/redhat-create-upload-vhd

Netplan https://netplan.io/

Endnotes i https://netplan.io/

Linux Virutal Machine Configuration Guide for Azure 19 of 20