Building Appliances Locally with KIWI
Total Page:16
File Type:pdf, Size:1020Kb
TUT 8118 SUSE® Studio Onsite in the Datacenter Andreas Thomas Ralf Dannert Designated Support Engineer Systems Engineer [email protected] [email protected] Agenda • Overview SUSE Studio • SUSE Studio and SUSE Manager Integration • SUSE Studio POCs in Datacenter environments ‒ Challenges in Build Environment ‒ Challenges in Deployment ‒ More Troubleshooting • Q&A 2 Overview SUSE Studio SUSE Studio Onsite: Key Features • Supported Linux in Minutes • Integrated Testing(testdrive) • Multiple Platforms ‒ Live CD/DVD, preload ISO/USB, HDD, PXE, Xen, vmdk, OVF, Amazon EC2) • Supportability Analyzer • Automated Dependency Discovery • Multi-Host Staged Delivery • SUSE Gallery 4 SUSE Studio Onsite (I/II) ‒ Behind the firewall, on-premise, installable and fully supported version ‒ similar core functionalities as SUSE Studio Online ‒ Delivered as software appliance, requires bare metal server to install ‒ Shipped as raw disk image and a bootable CD containing the raw disk image ‒ SUSE Studio is proprietary software and delivered based on a software licensing model 5 SUSE Studio Onsite (II/II) • SUSE Studio needs access to the following types of repositories used to build the appliance: ‒ Installation repositories: Pool repositories ‒ Update repositories: ‒ directly from NCC/SCC ‒ use Subscription Management Tool (SMT) for SLE 11 SP3 to mirror the update repositories from NCC • create appliances using the following base Operating Systems: ‒ SLE{DS} 10 SP4 x86 and x86_64 ‒ SLE{DS} 11 SP1,SP2,SP3 x86 and x86_64 ‒ SLE{DS} 12 with Maintenance Update after SLE 12 Release 6 Challenges Addressed by SUSE Studio Onsite • Reduced Image footprint – JeOS • Standardization and Documentation • Fast iterative development process of a custom distribution ‒ Import changed files from testdrive ‒ Overlay files • Use KIWI for additional functionality later 7 SUSE Studio Architecture 9 SUSE Studio and SUSE Manager Integration SUSE Manager ✔ Optimize ✔ Control ✔ Innovate 11 Background: SUSE Manager Staging ‒ SUSE Manager: move channels errata into the next stage/environment for ongoing phased testing/rollout ‒ natural progression from DEV to QA to PROD ‒ Formerly: spacewalk-clone-by-date ‒ now: spacewalk-channel-patch-lifecycle ‒ spacewalk-manage-channel-lifecycle --promote --phases=DEV,QA,PROD -c sles11-sp3-pool-x86_64 12 SUSE Manager Integration Build appliances from stages Dev/Test/Prod/ ‒ cobbler used as installation source for a distribution ‒ Prerequisite: distribution exists and child channels available ‒ SUSE Manager: ‒ Systems → Autoinstallation → Distributions → lable ‒ parent channel of child channel must be assigned to the distribution ‒ SUSE Studio: use these channels as repositories ‒ http://<susemanager>/ks/dist/child/<childchannel-label>/<dist-label> ‒ Admin User →Advanced → Add Repository ‒ Usecases: ‒ Easy (de)provisioning of repositories possible ‒ Snapshotted, reproducible builds 13 SUSE Manager Integration Troubleshoot ‒ URLs to integrate with SUSE Manager are not “normal URLs” ‒ "grep /ks *" in /etc/apache2/conf.d reveals that /ks/dist gets re-routed to "/rhn/common/DownloadFile.do" ‒ zz-spacewalk-www.conf:RewriteRule ^/ks/dist(.*)$ ‒ /rhn/common/DownloadFile.do?url=/ks/dist$ ‒ every URL starting with /rhn is passed to tomcat as of this rule: ‒ zz-spacewalk-www.conf:RewriteRule ^/rhn(.*) ajp://localhost:8009/rhn$1 [P] 14 Challenges in Build Environment SUSE Studio and Datacenter • Differences between datacenter and “old” purpose of studio usage • Benefits of image deployment vs autoyast ‒ speed up of deploment (less reboots / no hardware probing) ‒ Faster development process (e.g. test drives, overlay files) ‒ Ideally: base image for deployment and customizing via software management stack 16 Using Your Own Repository(I) 17 Using Your Own Repository(II) 18 Using Your Own Repository(III) Declaring Pattern 19 Using Your Own Repository(IV) Updating Repository Data Now we've got a reposory providing “our” RPMs and “our” software patterns 20 Include and Use it in SUSE Studio 21 Challenges in Deployment Challenges in Deployment • Disk-less Servers (boot from SAN, discovery of disks) • Different Network Adapters / IP Address advertising • Struggeling “unknown” Networks (blade center) • Multipathing / host-based mirror requirements • Use of “own” Patterns and custom RPMs • Kiwi version in Studio / Containment ‒ Upstream kiwi with fixes / features not yet in Studio ‒ Awareness of Product Management to update kiwi version in products ‒ There's always more than one way to do it (pxe, initrd, kiwi) 23 Disk-less Server / Boot from SAN • Only plain SCSI disks are being detected / supported out of the box • Need support for /dev/disk/by-*/scsi-XXXX • Solution: ‒ Current Kiwi version ‒ Upcoming Maintenance Update 24 Solving Software Dependencies ‒ Solving dependencies in Studio is based on zypper mechanics ‒ Repositories based on zypper ‒ Refresh of repositories ‒ Priorities of repositories ‒ Custom change in Studio (use old version of RPM as default) ‒ Self created repositories (using createrepo) ‒ Custom templates as FATE 25 Choose Non-default rpm Versions ‒ choose a specific software version from another repository manually ‒ visible in Build → Configuration → Selected software ‒ if version attached to name → have been manually selected ‒ lower priority of repository in /srv/studio/options.yml ‒ add repos_with_lower_priority: according to ‒ Changing Repository Order from the SUSE Studio Onsite Deployment And Administration Guide ‒ example: Add OBS repository ‒ Admin User → Repositories →Add repository ‒ Name: python ‒ URL: http://download.opensuse.org/repositories/devel:/languages:/python/SLE_1 1_SP3/ 26 Software Management • Adjustments in post build script (strip down) • firstboot_script • Adding repositories during bootup/firstboot • Limitations of 3rd party rpms ‒ Scripts in RPMs (e.g. add users/permissions/acls) ‒ Boot-related rpm (missing bootloader file) ‒ Custom templates as FATE 27 RPMs Break Build Process WHY? • Scripts in 3rd party RPMs may use acls ‒ Fixes in the meantime • Scripts in RPMs might require boot-related files ‒ /boot/grub/menu.lst, /etc/fstab etc. Solutions: ‒ Fix RPM script (if possible) ‒ Install after deployment (firstboot script) ‒ Using “wrapper RPMs” if not network accessible (e.g. test drive) ‒ add a repository using “zypper ar” during firstboot and install afterwards 28 No Network... No PXE Deployment • Root cause: Stripping of unnecessary packages after build • Solution: Include kernel-firmware package in bootrequired, specify dedicated 29 ...And Strange Behaviours • First DHCP request is taken for Network configuration • BUT: Some blade centers run their own DHCP server • 169.X.X.X is not routed to “our” TFTP for image rollouts • Possible Solutions: ‒ 1. specify MAC/NIC assingments as append parameter ‒ 2. build initrd having a recent KIWI build environment ‒ 3. Edit initrd (see custom initrd) to limit NICs used for discovery 30 ...Just Specify to Your Demands DEFAULT KIWI-Boot LABEL KIWI-Boot kernel boot/kernel_new append initrd=boot/initrd_new vga=0x314 kiwiserver=4.239.87.130 PXE_IFACE=eth0 lang=de_DE insmod=bnx2 netwait=90 netretry=5 prefer_iface=eth0 BOOTIF=eth0 IPAPPEND 1 31 Building a Custom Image • How to include ‒ Required firmware (e.g. include kernel-firmware package) • Troubleshooting tools (less, vim, util-linux, sshd) • troubeshooting initrd vs roll-out initrd • Use of hooks to extend initrd • Business as usual, integration in process is the key 32 Export – Adapt – Build – Deploy (0) 33 Building Appliances Locally with KIWI ‒ Build tab, scroll down and select Export your appliance's KIWI configuration ‒ have latest version from the Open Build Service repository Virtualization: Appliances ‒ sudo ./create_appliance.sh ‒ specify repository URL for internal (non-public) repositories ‒ <repository type='rpm-md'> ‒ <source path='{SLES 11 SP3 Updates i386}'/> ‒ </repository> ‒ README for Kiwi source from SUSE Studio 34 Export – Adapt – Build – Deploy (I) 35 Export – Adapt – Build – Deploy (II) Kiwi Hooks 36 Export – Adapt – Build – Deploy (III) config.xml for PXE 37 Export – Adapt – Build – Deploy (IV) Pattern in Kiwi 38 Export – Adapt – Build – Deploy (V) toolchain in bootincludes 39 Export – Adapt – Build – Deploy (VI) Adapt Repositories 40 Export – Adapt – Build – Deploy (VII) Kiwi Hooks ‒ RAID 1 (mirror) supported out of the box ‒ Multipath and DM support in recent kiwi ‒ Need to use hook functions for enablement 41 Export – Adapt – Build – Deploy (VIII) ● Copy output to tftp 42 Export – Adapt – Build – Deploy (IX) tftp 43 Export – Adapt – Build – Deploy (X) PXE 44 Export – Adapt – Build – Deploy (XI) 45 Export – Adapt – Build – Deploy (13) 46 More Troubleshooting Some Hints • Use kiwidebug=1 parameter in PXE configuration ‒ Emergency shell ‒ Detailed log in /var/log/boot.kiwi • Build a debug initrd with “your” tools • Use KIWI_FORBID_HOOKS=1 to eleminate “your bugs” • env output helps you to verify what went wrong • /include helps you to understand systems behaviour 48 How to Debug a Containment(I) • What is a containment • %description ‒ Containment appliance to build studio images secured by a VM layer ‒ tar -cjf $RPM_SOURCE_DIR/$NAME-$VERSION- $RELEASE-vmx.tar.bz2 $SOURCE metadata ‒ rpmbuild -ba $FILES_DIR/image.spec ‒ https://github.com/openSUSE/containment-rpm 49 How to Debug a Containment(II) ‒ minor debugging session in containment: ‒ in kiwi-job/lib/containment.rb, set self.debug to 1 ‒ start the build. roughly after 'downloading packages', you'll see something like: ‒ Containment running