The Definitive Guide to the Xen Hypervisor Prentice Hall Open Source Software Development Series Arnold Robbins, Series Editor “Real world code from real world applications” Open Source technology has revolutionized the computing world. Many large-scale projects are in production use worldwide, such as Apache, MySQL, and Postgres, with programmers writing applications in a variety of languages including Perl, Python, and PHP. These technologies are in use on many different systems, ranging from proprietary systems, to Linux systems, to traditional UNIX systems, to mainframes. The Prentice Hall Open Source Software Development Series is designed to bring you the best of these Open Source technologies. Not only will you learn how to use them for your projects, but you will learn from them. By seeing real code from real applications, you will learn the best practices of Open Source developers the world over. Titles currently in the series include: Linux® Debugging and Performance Tuning SELinux by Example Steve Best Frank Mayer, David Caplan, Karl MacMillan 0131492470, Paper, ©2006 0131963694, Paper, ©2007 The Defi nitive Guide to the Xen Hypervisor UNIX to Linux® Porting David Chisnall Alfredo Mendoza, Chakarat Skawratananond, 013234971X, Hard, ©2008 Artis Walker 0131871099, Paper, ©2006 Understanding AJAX Joshua Eichorn Rapid Web Applications with TurboGears 0132216353, Paper, ©2007 Mark Ramm, Kevin Dangoor, Gigi Sayfan 0132433885, Paper, © 2007 The Linux Programmer’s Toolbox John Fusco Linux Programming by Example 0132198576, Paper, ©2007 Arnold Robbins 0131429647, Paper, ©2004 Embedded Linux Primer Christopher Hallinan The Linux® Kernel Primer 0131679848, Paper, ©2007 Claudia Salzberg, Gordon Fischer, The Apache Modules Book Steven Smolski Nick Kew 0131181637, Paper, ©2006 0132409674, Paper, © 2007 Rapid GUI Programming with Python and Qt Mark Summerfi eld 0132354187, Hard, © 2008 New to the series: Digital Short Cuts Short Cuts are short, concise, PDF documents designed specifi cally for busy technical professionals like you. Each Short Cut is tightly focused on a specifi c technology or technical problem. Written by industry experts and best selling authors, Short Cuts are published with you in mind — getting you the technical information that you need — now. Understanding AJAX: Debugging Embedded Linux Consuming the Sent Data with XML and JSON Christopher Hallinan Joshua Eichorn 0131580132, Adobe Acrobat PDF, © 2007 0132337932, Adobe Acrobat PDF, © 2007 Using BusyBox Christopher Hallinan 0132335921, Adobe Acrobat PDF, © 2007 The Definitive Guide to the Xen Hypervisor David Chisnall Upper Saddle River, NJ • Boston • Indianapolis • San Francisco New York • Toronto • Montreal • London • Munich • Paris • Madrid Capetown • Sydney • Tokyo • Singapore • Mexico City Many of the designations used by manufacturers and sellers to distinguish their products are Editor-in-Chief claimed as trademarks. Where those designations appear in this book, and the publisher was Mark L. Taub aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals. Xen, XenSource, XenEnterprise, XenServer and XenExpress, are either registered trademarks Acquisitions Editor or trademarks of XenSource Inc. in the United States and/or other countries. Debra Williams Cauley The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein. Development Editor The publisher offers excellent discounts on this book when ordered in quantity for bulk pur- Michael Thurston chases or special sales, which may include electronic versions and/or custom covers and con- tent particular to your business, training goals, marketing focus, and branding interests. For more information, please contact: U.S. Corporate and Government Sales, (800) 382-3419, [email protected]. For sales outside the United States please contact: Inter- Managing Editor national Sales, [email protected]. John Fuller Full-Service Production Manager Julie B. Nahil Technical Reviewer Glenn Tremblay Cover Designer Alan Clements Composition Visit us on the Web: www.prenhallprofessional.com David Chisnall Library of Congress Cataloging-in-Publication Data Chisnall, David. The definitive guide to the Xen hypervisor / David Chisnall. p. cm. Includes index. ISBN-13: 978-0-13-234971-0 (hardcover : alk. paper) 1. Xen (Electronic resource) 2. Virtual computer systems. 3. Computer organization. 4. Parallel processing (Electronic computers) I. Title. QA76.9.V5C427 2007 005.4’3—dc22 2007036152 Copyright c 2008 Pearson Education, Inc. All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited repro- duction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding permissions, write to: Pearson Education, Inc., Rights and Contracts Department, 501 Boylston Street, Suite 900, Boston, MA 02116; fax: (617) 671-3447. ISBN-13: 978-0-13-234971-0 ISBN-10: 0-13-234971-X Text printed in the United States on recycled paper at Courier in Westford, Massachusetts. First printing, November 2007 Contents List of Figures xi List of Tables xiii Foreword xv Preface xvii I The Xen Virtual Machine 1 1 The State of Virtualization 3 1.1WhatIsVirtualization?........................ 3 1.1.1 CPUVirtualization...................... 4 1.1.2 I/OVirtualization....................... 5 1.2WhyVirtualize?............................ 7 1.3TheFirstVirtualMachine....................... 8 1.4TheProblemofx86.......................... 9 1.5SomeSolutions............................. 9 1.5.1 BinaryRewriting........................ 10 1.5.2 Paravirtualization....................... 10 1.5.3 Hardware-AssistedVirtualization............... 13 1.6TheXenPhilosophy.......................... 15 1.6.1 SeparationofPolicyandMechanism............. 15 1.6.2 LessIsMore.......................... 15 1.7TheXenArchitecture......................... 16 1.7.1 The Hypervisor, the OS, and the Applications . 16 1.7.2 The RˆoleofDomain0..................... 19 1.7.3 Unprivileged Domains ..................... 22 1.7.4 HVMDomains......................... 22 1.7.5 XenConfigurations....................... 23 v vi Contents 2 Exploring the Xen Virtual Architecture 27 2.1BootingasaParavirtualizedGuest.................. 27 2.2RestrictingOperationswithPrivilegeRings............. 28 2.3ReplacingPrivilegedInstructionswithHypercalls.......... 30 2.4ExploringtheXenEventModel.................... 33 2.5CommunicatingwithSharedMemory................ 34 2.6SplitDeviceDriverModel....................... 35 2.7TheVMLifecycle............................ 37 2.8Exercise:TheSimplestXenKernel.................. 38 2.8.1 TheGuestEntryPoint.................... 40 2.8.2 PuttingItAllTogether.................... 43 3 Understanding Shared Info Pages 47 3.1RetrievingBootTimeInfo....................... 47 3.2TheSharedInfoPage......................... 51 3.3TimeKeepinginXen.......................... 53 3.4Exercise:Implementinggettimeofday()................ 54 4 Using Grant Tables 59 4.1SharingMemory............................ 59 4.1.1 MappingaPageFrame.................... 61 4.1.2 Transferring Data between Domains . .......... 63 4.2DeviceI/ORings............................ 65 4.3GrantingandRevokingPermissions................. 66 4.4Exercise:MappingaGrantedPage.................. 69 4.5Exercise:SharingMemorybetweenVMs............... 71 5 Understanding Xen Memory Management 75 5.1ManagingMemorywithx86...................... 75 5.2Pseudo-PhysicalMemoryModel................... 78 5.3Segmentingon32-bitx86....................... 80 5.4UsingXenMemoryAssists...................... 82 5.5 Controlling Memory Usage with the Balloon Driver . ...... 84 5.6OtherMemoryOperations....................... 86 5.7UpdatingthePageTables....................... 89 5.7.1 CreatingaNewVMInstance................. 93 5.7.2 HandlingaPageFault..................... 94 5.7.3 Suspend,Resume,andMigration............... 94 5.8Exercise:MappingtheSharedInfoPage............... 95 Contents vii II Device I/O 97 6 Understanding Device Drivers 99 6.1TheSplitDriverModel........................ 100 6.2MovingDriversoutofDomain0 ................... 102 6.3UnderstandingSharedMemoryRingBuffers.............103 6.3.1 Examining the Xen Implementation ............. 105 6.3.2 OrderingOperationswithMemoryBarriers......... 107 6.4ConnectingDeviceswithXenBus................... 109 6.5 Handling Notifications from Events . ................. 111 6.6ConfiguringviatheXenStore..................... 112 6.7Exercise:TheConsoleDevice..................... 112 7 Using Event Channels 119 7.1EventsandInterrupts......................... 119 7.2HandlingTraps............................. 120 7.3EventTypes............................... 123 7.4RequestingEvents........................... 124 7.5BindinganEventChanneltoaVCPU................ 127 7.6OperationsonBoundChannels.................... 128 7.7 Getting a Channel’s Status . ..................... 129 7.8MaskingEvents............................. 130 7.9EventsandScheduling......................... 132 7.10Exercise:AFullConsoleDriver.................... 133 8 Looking through the XenStore