Build Nested Openstack Using as BaseOS on BareMetal Overview

• The following slides outline how to setup your BareMetal HOST (a server) in order to run a Openstack installation using FUEL based Deployer. The overall tasks can be broken down into the following sections: • Host OS installation and preparation • Virtual Machine creation for Openstack Nodes (Compute/Controller and FUEL) • Addition of vRouter (in order to isolate a useable “PUBLIC” network) • Setup of VNC to Nested Host (in order to use UI’s and convienence) • Booting/Configuration of FUEL VM (used to install Openstack to Compute/Controller) • Creation of an Environment and deployment of Openstack (job done) Conceptual Diagram

• Below you can see what the internal Layout of a Nested installation can look like. In this example, we will use 1 bridge for each of the Openstack Networks (5 in total – ADMIN/PXE, PUBLIC, PRIVATE,STORAGE and MANAGEMENT)

• LMCDASM – ADD PICTURE HERE Host OS and preparation

• You should perform the following tasks in this section.

• 1st – Install Ubuntu to your Host and ensure it has connectivity to the internet • Note: For the example used here – we use Ubuntu 14.04.03 LTS – however ,the specific base OS is up to the user (note: Centos/RedHat command will be different to reflect systemd usage and are not reflected herein) • Note: Depending on the size of your host (Disk/CPU/RAM) some of the values you see in the following screen shots may need to be modified to accommodate your system.

Specifications used in this example (this is a small setup for example – you should not use this in production at all) HOST HAS: 12 Cpus (Cores) 16GB RAM 250GB disk 1 Physical NIC – connected to the internet

ISO for booting HOST: ubuntu-14.04.3-server-amd64.iso

Note: in this example the HOST is assigned an IP via DHCP – this might not be the case in your setup, but it is assumed you know how to setup an IP address – if not, then this procedure is going to be a real challenge for you. Host Install

• When you boot your HOST with Ubuntu you should see the “startup page”. In general, we select all the defaults, but below you find the answers to each of the questions. 1st- Lang – English (or whatever you like) 2nd – Install Ubuntu Server (hit enter) 3rd – English 4th – US 5th – No (don’t detect Keyboard – unless you need it) 6th – English 7th – English 8th – Pick a hostname (ex NestedOstack) 9th – Full Name (eg user) 10th – Username (eg. user) 11th – Password (eg system) 12th – Re-enter Password (eg system) note: Answer yes to use password if not complex enough 13th – Answer no to encrypt home dir 14th – Answer “yes” if Timezone is correct 15th – Answer – “Guided – use entire disk and setup LVM” 16th – Select the Disk to install to. (Hit enter if there is only one) 17th – Answer “save Changes and configured LVM” 18th – Hit enter to use the total amount of disk (change if appropriate) 19th – Move to “YES” and hit enter to write all final changes to the disk 20th – Enter Proxy info (if needed). 21st – No to Automatic Updates 22nd – For Software to install – Select SSH only (note: We will add the Virtual Machine packages after the installation along with other tools) 22nd – Yes to Grub loaded and reboot the node.

Ubuntu should install now – wait until it is completed and you have a login prompt and then continue with the HOST-POST-INSTALL steps (next page) Host Post-Install

• Login to the node following the installation and perform the following steps. • 1st – sudo apt-get update && sudo apt-get upgrade • 2nd – sudo apt-get install kvm virt-manager bridge-utils vlan tightvncserver xfce4 xfce4-goodies firefox (note: you can use whatever desktop you like if you don’t like xfce4) • 3rd – Reboot the node (probably don’t “have to”, but a good idea at this point since initramfs was updated in the previous step) • 4th – run “kvm-ok” and ensure you see a positive output – if not – STOP and get HELP) Create Bridges • As outlined, we will create fives bridges on this HOST, they will represent “switches” that will hold each of the OPENSTACK networks required (ADMIN/PXE, MGMT, STOR, PUBLIC and PRIVATE). As well, we will create an “EXT_BR” that will be encapsulate our real network card (so we will be able to route in/out of the Openstack Later on). In this example, we will use 5 “FLAT” networks – no tags - although they can be used just as easily. • Note: sudo is implied below (or do “sudo –I” and execute as root) • Commands (to be done on console – or you will lose connection): • Ifconfig virbr0 down • brctl delbr virbr0 • brctl addbr ext_br • brctl addbr admin • brctl addbr public • brctl addbr private • brctl addbr storage • brctl addbr mgmt

Bring up the bridges Command: ifconfig mgmt 192.168.10.253/24 up ifconfig public 172.16.0.253/24 up ifconfig private up <= Left without IP intentionally ifconfig storage192.168.20.253/24 up ifconfig admin 10.20.0.253/24 upup

Encapsulate external NIC to EXT_BR and assign IP to the BRIDGE. 1st – get your current IP address (assuming you have eth0 as your external NIC) Command: ifconfig –a eth0 2nd – put your eth0 NIC to a null address (leave it up though) and add it to ext_br and set the IP on the bridge Command: ifconfig eth0 0.0.0.0 brctl addif ext_br eth0 ifconfig ext_br

3rd – Test that you can ping your GW still (eg, my GW is 192.168.1.1) IF this doesn’t work – STOP and get HELP, cause nothing else will work going forward. Make Network Setting Permanent

• Login to your node (you can now use SSH through the ext_br you setup in the previous step) and modify /etc/network/interfaces as shown below

• user@NestedOstack:/etc/network$ cat interfaces • # This file describes the network interfaces available on your system • # and how to activate them. For more information, see interfaces(5).

• # The loopback network interface • auto lo • iface lo inet loopback

• # The primary network interface • auto eth0 • iface eth0 inet manual

• auto ext_br • iface ext_br inet static address 192.168.1.115 netmask 255.255.255.0 gateway 192.168.1.1 dns-nameservers 8.8.8.8 bridge_ports eth0

Reboot the node and make sure you can get back in without issue. Setup VNC server and xfce4

• For the VM creation part, I like to use virt-manager (a GUI based tool), you can use the virsh CLI if you like, but the GUI allows to see the CONSOLE of the VMs and edit SPICE and other params on the fly. The following section outlines how to configure tightvncserver and xfce4 for use. • Step 1 – Login to your HOST and start VNCSERVER and set password (this is the password you will use to connect from your VNC client) – DO NOT RUN AS ROOT. • Command: “vncserver: • You will be prompted for a password (you can decide if you want a read-only one as well) • You should see the server started on :1 Setup VNC server and xfce4 /2 • Now kill the running vncserver Command: “vncserver –kill :1” • Modify your users /home//.vnc/xstartup file to look like this (remove everything else) #!/bin/sh unset SESSION_MANAGER unset DBUS_SESSION_BUS_ADDRESS startxfce4 &

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup [ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources xsetroot -solid grey vncconfig -iconic &

• Restart your VNC server (note: :1 is port 5901 for connecting with your client) Command: “vncserver” • Sample Output: • user@NestedOstack:~$ vncserver

• New 'X' desktop is NestedOstack:1

• Starting applications specified in /home/user/.vnc/xstartup • Log file is /home/user/.vnc/NestedOstack:1.log

• user@NestedOstack:~$ Connect with VNC client • I use UltraVNC as a client, but you can use whatever you like. (you can pull X display across if you have good bandwidth) • Open your VNC Client and connect to your Host, screen shot examples below: Connect with VNC Client /2 • Observe a desktop on your HOST is displayed. Select “default Config” Create Your VMs

• In this example, we will create a total of 4 VMs. • 1 VM for a vROUTER – this is used to isolate our Openstack networks and provide and access for FUEL to the Openstack repos • 1VM for a FUEL server – this is the node that will boot and install the Controller and Compute • 1VM for Controller • 1VM for Compute

• Note: You can create as many VM’s as your HARDWARE allows, just repeat the Section for COMPUTE/CONTROLER (note: Max 3 Controllers for HA, Min 1) Create VM - vRouter /1

• With our VNC open, open a terminal and start “virt-manager”, see screen shot below: Create VM – vRouter /2

• For the vRouter, I use (you can get the image at http://dasmlab.org/vyatta-livecd_VSE6.7R6_amd64.iso), or you can use VyOs, or , or whatever you like. • Push the image to your host (SFTP/HTTP/etc) so it is accessible there. • In the virt-manager GUI – create your VM • Details: 1 CPU, 2GB RAM, 5GB HD, 3 NICs (on ext_br, admin, public)

The following SCREEN SHOT outline each STEP Create VM – vRouter /3

• Click “Create New Virtual Machine” Create VM – vRouter /4

• Give your VM a name as shown and select “Local media install”: Create VM – vRouter /5 • Point to the ISO location and set the OS_TYPE and Version (pick any for version – doesn’t matter) click “Forward”. Set RAM/CPU as shown and click “Forward”. Set disk to 5GB and click Forward Create VM – vRouter /6 • On the next page, “CHECK” the customized box, and ensure that the “Advanced Options” is outlined as shown below (must be on ext_br) and click Finish. Create VM – vRouter /7 • On the Virtual Machine Details page (shown) below you will do the following tasks (use the ADD HARDWARE BUTTON for NICs): • 1st – Change “Display VNC” from SPICE to VNC. • 2nd – Add two more NICs – one on bridge “admin” and one on bridge “public” Create VM – vRouter /8 • Picture showing addition of NIC to admin bridge (note: you must “specify the name” Create VM –vRouter /9

• Picture showing adding NIC to “public” bridge Create VM – vRouter /10 • Verify your screen matches below and click “Being Installation” Create VM – vRouter /11

• Console open on your VM, and will boot. Once you are at the login prompt, execute the following commands to setup your vRouter • 1st – Login (vyatta/vyatta) • 2nd – Command: “install image” • Responses: Yes, Auto, vda, Yes, , , , vyatta, • 3rd – Click on the “!” at the top of the Console, details page will be displayed – Select “Boot Options” and change order to boot from Disk. Select Apply and then reboot the VM (see screen on next page for example). IF YOU DON’T – you will only boot from ISO again and nothing will be permanent in your vROUTER. Create VM – vRouter /12

• Screen shot of Changing Boot Order Create VM – vRouter /13 • Now that your is rebooted off its Hard disk, perform the following configurations. Note the difference between config and command mode in vyatta • Command (to be executed in order once you login) • 1st – “config” and press Enter (you should see “edit” above prompt) • 2nd – “set interfaces ethernet eth0 192.168.1.116/24” <= where this is an IP on the external subnet that your hosts physical NIC is on • 3rd – “set interfaces ethernet eth1 10.20.0.1/24” • 4th – “set interface ethernet eth2 172.16.0.1/24” • 5th – “set nat source rule 1 destination address 0.0.0.0/0” • 6th – “set nat source rule 1 outbound-interface eth0” • 7th – “set nat source rule 1 protocol all” • 8th – “set nat source rule 1 source address 10.20.0.0/24” • 9th – “set nat source rule 1 translation address masquerade” • 9.1 – “set service dns forwarding listen-on eth1” • 9.2 – “set service dns forwarding name-server 8.8.8.8” • 10th – “commit” • 11th – “save” Move onto the next page Create VM – vRouter /14

• Continue configuring your vRouter with the following commands: • 1st – “set nat source rule 2 destination address 0.0.0.0/0” • 2nd – “set nat souce rule 2 outbound-interface eth0” • 3rd – “set nat source rule 2 protocol all” • 4th – “set nat source rule 2 source address 172.16.0.0/24” • 5th – “set nat source rule 2 translation address masquerade” • 6th – “set protocols static route 0.0.0.0/0 next-hop 192.168.1.1” <= where the 192 is the IP of my GW – yours might be different • 7th – “set system name-server 8.8.8.8” <= where the IP is of your DNS server • 8th – “set system ntp server 0.ca.pool.ntp.org” <= where NTP is a valid address – you MUST have a good NTP source – LCL is a bad idea in VMs. • 9th – “commit” • 10th – “save”

• YOUR VROUTER IS CONFIGURED AND READY TO GO – NO MORE WORK THERE Create VM - vFUEL

• The following steps will create a FUEL VM – that is used to create/deploy and manage your Openstack Installation. • For this we will use the Brahmaputra (B-release) of FUEL from OPNFV as our ISO to creation – you can fetch this at: http://artifacts.opnfv.org/fuel/brahmaputra/opnfv- brahmaputra.1.0.iso • Download that ISO on your host, it will be need in the following steps Create VM – vFUEL /2

• Login to your HOST, open virt-manager and click “Create New Virtual Machine”. Your first screen should like this – then click “Forward” Create VM – vFuel /3

• On the following page, point to the FUEL ISO you downloaded and set the OS_TYPE and OS_Version as shown below, click Forward. Create VM – vFUEL /4

• Send the RAM/CPU as shown. Set the DISK size as shown. Create VM – vFUEL /5 • Set the final configuration screen as shown below – ensure you put the NIC on the “ADMIN” bridge (specify bridge name) as shown below and click Finish. Create VM – vFUEL /6 • The FUEL VM will boot the splash screen show below – Select “FUEL INSTALL (Static IP)” and press Enter. The node will install – will take some time. Create VM – vFUEL /6 • Once your VM reboots, you will be presented with this screen – press enter to answer the question about password. We will have to modify a couple values in this menu before proceeding. Create VM – vFUEL /7 • Navigate (arrow keys) to DNS & Hostname section and ensure that the DNS entry is valid for your setup. You can modify the hostname and dns suffix to suit your needs (not required for vPOD). Once edited, move to “Check” and press Enter. Create VM –vFUEL /8

• Navigate to Time Sync section and configure the appropriate NTP server – NOTE: for vPOD, a valid NTP server is critical – Click “check” Create VM – vFUEL /9

• Navigate to “Quit Setup” and Select “Save and Quit”. The Fuel VM will test that your configuration is correct and if all is well, finish the installation. Create VM – vFUEL /10

• When you FUEL install is complete the Console will show the following: Create VM – vFUEL /11 • Point your “LOCAL” browser (in your VNC session, or X exported firefox) to http://10.20.0.2 and login (admin/admin). You should see the following. Create VM – vFUEL /12 • After your first login, uncheck “Send Stats” and click “Connect Later”, you should have a page like this. Continue to the next steps Create VM – vController • The following steps outline how to create the vController (your Openstack Controller node). This node will be jumpstarted by the FUEL node, so we only need to create the VM, the NICs and a empty hard disk – no image is supplied. As before, login to your HOST and open virt-manager and your first screen should look like this (note: We use PXE BOOT here): Create VM – vController /2

One the following screen, select the OS type and Version, RAM and CPU Create VM – vController /3 • Ensure that you have the following items checked on the summary screen (specify “admin” for the first bridge) – we will add four more NICs in the Details Screen Create VM – vController /4 • Add four more Network Cards to the Controller, one for each network (bridge) in the setup (mgmt., storage, public, private). Create VM – vController /5 • Screen shots of adding last two NICs Create VM – vController /6 • Below is an example of the layout of the vController, when satisfied, click “Begin Installation”. The VM will boot and the vFUEL will provide PXE boot. Nothing else needs to be done on Controller at this point Create VM –vCompute /1

• The following steps outline how to create the VM for the COMPUTE. These steps are identical to the vController, except that you need to edit the cpu_model in the vCompute configuration to ensure KVM flags are passed from HOST to GUEST (so that the COMPUTE can create VM’s itself). • Notes: • In a virtual environment – the more storage,RAM and CPU you give to the vCompute translates into more resources for Instance (VM) creation under Openstack. Create VM – vCompute /2 • Create the VM as before – ensure you select “PXE BOOT” on the naming page. Create VM – vCompute /3 • Configure you Storage appropriately and make sure you select “admin” as the first bridge for this VM and check to configure options. • Note: Ignore “Network does not support PXE –it Lies! :P) Create VM – vCompute /4 • As we did with the Controller, add 4 more NIC’s to this VM (mgmt, storage, private,public) respectively so that you screen resembles this Create VM – vCompute /5 • There is a bug in the latest kvm/virt-manager and you need to configure CPU information manually. Open the Processor tab of the vCompute and navigate to “Configuration” and click ‘Copy Host CPU Properties”. Example shown below, click Apply and then “Begin Installation” Create VM – vCompute /6

• After you click Begin Installation the VM will boot, PXE boot from the FUEL node and you will end up at a bootstrap login prompt (shown below). You do not need to do anything else on the vCompute at this point. Create Openstack Environment

• The following steps are executed within the FUEL UI (Web Page). • Login to the FUEL Web page at (http://10.20.0.2) (admin/admin) • Observe in the Top right Corner that you have 2 (or more) nodes that are available to be used. Example below Create Openstack Environment /2

• In the Main Page (click “Environments at the top if you get lost). • Click “New Openstack Enviroment” • Provide : • A Name for your Environment e.g – TEST ENV and Click “Next” • Select Hypervisor Type (QEMU/KVM or vCenter) and Click “Next” • Select Networking Type (VLAN or VXLAN) and Click “Next” • Select Storage Backends (LVM or CEPH – note: CEPH needs min 3 nodes) and click Next • Select Additional Openstack Options desired and Click “Next” • Click “Finish” and your Environment will be initialized and you will have a screen presented (see next page) Create Openstack Environment /3

• ENV screen following “FINISH” from last step. Create Openstack Env /4

• Click on “Node” Tab from the Environment Screen • Click “Add Nodes” • Select Role “Controller” (check box) and then select your Controller node from the list at the bottom (use the MAC if you don’t know which is which) • Click “Apply Changes” • Click “Add Nodes” • Select Role “Compute” and “Storage-Cinder” (check box) and then select your COMPUTE node from the list below (check box). • Click Apply Changes Create Openstack Env /5 • You screen should look similar to the following: Create Openstack Env /6 • On the Nodes Screen – Check “Select All” • Then Click on “Configure Interfaces” • Drag and Drop the Networks so that they match the screen shot below and Click “Apply” Create Openstack Env /7

• Click on the Networks Tab on your Environment Screen (next to Nodes) and perform the following modifications. • Under Storage: • change CIDR to 192.168.20.0/24 • Uncheck VLAN tag (not used) • Under Management: • Change CIDR to 192.168.10.0/24 • Uncheck VLAN Tag (not used) • Click Save • On the Left – Click “Neutron Layer 3” and update the DNS with a valid entry (should be the same as you used in the FUEL step) and click “Save”

On the Left – click “Verify Connectivity” and then click the “Verify Networks” button. This is vital and if this fails, your deployment will – correct any errors seen and re-run until you see “Verify Network Succeeded” – see screen shot on next Page Create Openstack Env /8 • Screen shot example of verify networks success Create Openstack Env /9

• Now that Verify has passed, click on the Dashboard TAB (to the right of Nodes) in the FUEL UI and click “Deploy”.

• This will take between 30-60 minutes to deploy and when completed, you will be able to access your Openstack via Horizon (URL will be shown on the screen) or CLI (via the FUEL node): • Note: By default, SSHD to the COMPUTE/CONTROL is disabled except from the FUEL – so you need to SSH to your FUEL (root/r00tme) and then to the openstack nodes. You can run “fuel node” when on the vFUEL and see the admin IP’s of your nodes (or use the hostname) Create Openstack ENV /10 • Screenshot of deployment Tab