The Linux Sysadmins Guide to Virtual Disks i The Linux Sysadmins Guide to Virtual Disks From the Basics to the Advanced Copyright © 2009-2016 Tim Bielawa The Linux Sysadmin’s Guide to Virtual Disks by Tim Bielawa is licensed under the Cre- ative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0). To view a copy of the CC BY-SA 4.0 License, please visit: https://creativecommons.org/licenses/by-sa/4.0/ First Edition - First Printing - 2016 Printed in The United States Published by Scribe’s Guides New York, NY, USA Web Site: https://scribesguides.com/ Editors: Henry Graham Jonathan Connell Jyoti Sabharwal Cover Designer: Tim Bielawa The Disk Components Image (Figure B.1,“Disk Drive Components”[67]) which appears on both covers and in the Disk Drive History appendix of this book is remixed from the Wikipedia image Disk-structure2.svg. The original image ¹ is licensed as a work in the worldwide public domain. Original image created by Wikipedia users MistWiz ² and Heron2 ³ . The image was last updated by Tim Bielawa as displayed in this book. Library of Congress Control Number: 2016903293 Soft Cover ISBN-13: 978-0-692-64121-7 ¹ https://commons.wikimedia.org/wiki/File:Disk-structure2.svg ² https://en.wikipedia.org/wiki/User:MistWiz ³ https://en.wikipedia.org/wiki/User:Heron2 The Linux Sysadmins Guide to Virtual Disks iii Contents Acknowledgments 1 1 Introduction 3 1.1 Introduction .................................. 3 1.2 Typographical Conventions .......................... 3 1.3 Units & Prefixes ................................. 5 1.4 Getting Help/Feedback ............................. 5 1.5 Updates and Alternative Formats ....................... 6 1.6 About The Author ................................ 6 2 The Virtual Disk Cookbook 7 2.1 Creating Simple Images ............................ 7 2.2 Resizing Disk Images .............................. 8 2.2.1 Resizing RAW Images ......................... 8 2.2.2 Resizing QCOW2 Images ........................ 19 2.3 Query an Image for Information ........................ 23 2.4 Converting Between RAW and QCOW2 .................... 25 2.4.1 Convert an Image from RAW to QCOW2 ................ 25 2.4.2 Convert an Image from QCOW2 to RAW ................ 25 2.5 Creating Disks with Backing Images ...................... 26 2.6 Comitting changes to a backing image .................... 28 2.7 Cloning a Physical Disk ............................. 28 3 Disk Concepts 30 3.1 Creating a 1GiB virtual disk from scratch ................... 32 3.1.1 Background on the dd command .................. 32 3.1.2 Running dd .............................. 33 3.1.3 Examining the Created File ...................... 33 3.1.4 Create a Partition Table ........................ 35 3.2 Devices and Partitions ............................. 36 3.2.1 Introduction .............................. 36 3.2.2 Creating a Loop Device ........................ 37 3.2.3 Examine the loop device ........................ 38 3.2.4 Creating partitions ........................... 39 3.2.5 Formatting Partitions ......................... 40 3.2.6 Cleaning Up .............................. 43 4 Helper Utilities 44 4.1 libguestfs .................................... 45 4.1.1 guestmount .............................. 45 4.1.2 virt-filesystems ............................. 45 4.1.3 virt-rescue ............................... 46 4.1.4 virt-resize ................................ 46 4.1.5 virt-sparsify ............................... 46 4.2 virt manager .................................. 48 The Linux Sysadmins Guide to Virtual Disks v 5 Disk Formats 49 5.1 RAW ....................................... 49 5.2 QCOW ...................................... 49 5.3 QCOW2 ..................................... 50 5.4 Other Formats ................................. 50 6 Performance Considerations 52 6.1 I/O Caching ................................... 53 6.1.1 Write-back Caching .......................... 54 6.1.2 Write-through Caching ......................... 55 6.2 I/O Schedulers ................................. 55 6.2.1 Additional Resources ......................... 56 7 Troubleshooting/FAQs 57 8 Glossary 59 A Appendix: Man Pages 64 A.1 UNITS ...................................... 64 B Appendix: Disk Drive History 66 B.1 Disk Drive Components ............................ 66 B.2 Access Modes .................................. 68 B.2.1 CHS Addressing ............................ 68 B.2.2 LBA Addressing ............................. 69 B.3 The Master Boot Record ............................ 69 Colophon 73 Dedication This book is dedicated to the loving memories of Seth Vidal and Donald Brewer. Seth, you were the kind of person who always stuck to what they believed in once they decided what that was. There are few people you can truly say that about. Your creativity and technical prowess changed the world in ways most people couldn’t dream of. It’s a tragedy we lost you so early, but I can promise you one thing: your contributions to the world were inspirational and it’s going to take a lot of people to pick up where you left off. As you used to say, Don, lemme be honest wit’cha.... You were always a stand up gentle- man, sharp as a tack, and loyal as hell to anyone you were responsible for. You may be gone now, but you will never be forgotten by those whose lives you touched. The Linux Sysadmins Guide to Virtual Disks 1 / 73 Acknowledgments This book wouldn’t have been possible without the gracious advice, contributions, and support I received from so many people. In fact, there’s so many people that I can’t remem- ber them all! You know who you are — coworkers, friends, the people reporting errors, the people sending random emails saying they enjoyed the book, the people sending emails describing a tough situation they were in that the guide helped them get out of. Thanks everybody! Andrew “Hoss” Butcher. You’re a badass friend and an even more talented individual. You and I could hack or riff on anything together and have fun with it. Give Hampus and Ripley a kitty treat for me some time, will ya? Thanks to John Eckersberg and Chris Venghaus for copious amounts of feedback early on. It really helped to stoke my fire and get things moving. And a special thanks to Chris for being my biggest word-of-mouth referral. I have no idea how you meet all these people with burning needs to have their virtual disk questions answered, but I’m glad you refer them to me when you do. Alex Wood, your eclectic interests never fail to serve my personal interests in some way. Thanks for the assist with that XSLT a while back. Coincidentally, that XSLT ⁴ pertained to rendering the very Acknowledgements section you’re reading right now. So I guess we all owe you debt of gratitude for that! Jorge Fábregas, you were a fabulous unexpected resource when most of the major writing was happening for this book. Thanks for repeatedly reaching out to me with your feed- back and suggestions and pointing out errors. This book is better because of your unique contributions. ⁴ See my blog post about dblatex+docbook+acknowledgements sections for the interesting details: https: //blog.lnx.cx/2013/03/27/dblatex-docbook-acknowledgements-and-pdf-output/ Thank you Jon Connell, Henry Graham, and Jyoti Sabharwal for editing! Mark Dalrymple, Jason Hibbets, and Christopher Negus, thanks for the reviews and general authoring/pub- lishing advice and encouragement. Obligatory shout-outs to Norman Walsh, the man I consider the living personification of DocBook XML ⁵ , and Bob Stayton, author of DocBook XSL: The Complete Guide ⁶ . Thank you Wikipedia contributor MistWiz for creating the original image, Figure B.1,“Disk Drive Components”[67], used in the appendix. Also, thanks to Wikipedia contributor Heron2 for making later updates. The disk components image featured in this book has subse- quent changes I personally made. Image used and remixed according to permissions de- tailed in the Licensing section of the image’s Wikipedia page ⁷ . To my wife, Alicia, thanks for making me so happy and keeping me sane. Thank you for encouraging me to work on this book when I didn’t want to. And, thank you for everything else. Finally, my biggest “thanks” goes to David Krovich. A mentor, friend, and button-pusher for many years now. You always encouraged me to be better than I was. Without the op- portunities you offered me, and the radical influence you had on my life, this book would never have even reached conception. Truth be told, Chapter 2, The Virtual Disk Cookbook [7] section is mostly a merge and refresh of a lot of the notes I took, and staff documen- tation I wrote, working on one of our provisioning projects. Those notes became the first chapters of this book. It was while working under your instruction that I discovered my passion for documenting everything I learned. This book is a testament to that passion. Thanks, Kro. ⁵ DocBook Homepage: http://www.docbook.org/ ⁶ Read DocBook XSL: The Complete Guide online: http://www.sagehill.net/docbookxsl/ index.html ⁷ Original Disk Components Image: https://en.wikipedia.org/wiki/File: Disk-structure2.svg The Linux Sysadmins Guide to Virtual Disks 3 / 73 Chapter 1 Introduction 1.1 Introduction I was motivated to write this book because I felt the quality of the information regarding commonly used functionality in virtual disk operation was lacking certain specific clear examples. The information that is available is not contained in a central location. Some concepts of the qemu system aren’t covered at all. FAQs lead on to having an answer to