Better Utilization of Storage Features from KVM Guest via virtio-scsi Masaki Kimura <[email protected]> Information & Telecommunication Systems Company IT Platform Division Group Hitachi, Ltd. © Hitachi, Ltd. 2013. All rights reserved. Better Utilization of Storage Features from KVM Guest via virtio-scsi Contents 1. Background (Use Cases and Requirements) 2. KVM features for guest SCSI commands 3. Current Status of these features 4. Summary 5. Future work © Hitachi, Ltd. 2013. All rights reserved. 1 Better Utilization of Storage Features from KVM Guest via virtio-scsi 1. Background © Hitachi, Ltd. 2013. All rights reserved. 2 1-1. Background Enterprise systems expect that virtualized environment has the same level of manageability, availability, and reliability achieved in bare-metal. For example: 1. Thin-provisioned Storage for manageability, 2. HA cluster for availability, 3. Backup server for reliability. In bare-metal environment, some of these requirements are achieved by using storage features, such as SCSI commands. In virtualized environment, the same use cases exists for guests. Issuing SCSI commands from guests are required. Three use cases, thin-provisioned storage, HA cluster, and backup server will be explained in the next slides. © Hitachi, Ltd. 2013. All rights reserved. 3 1-2. Use case #1: Thin-provisioned Storage - Many types of enterprise storage have thin-provision function. - For achievement of thin-provision, a disk block is allocated on access. - However, once it is allocated, it can not be reclaimed by storage automatically even when the disk block becomes unused by OS. - To reclaim unused disk block, OS needs to let storage know unused blocks by issuing WriteSame SCSI command. Bare-metal KVM Guest Guest Guest OS OS QEMU OS Host OS Server Server WriteSame WriteSame reclaiming disk blocks This use case exists for both bare-metal and KVM guests. WriteSame is required to be issued to storage from guests. © Hitachi, Ltd. 2013. All rights reserved. 4 1-3. Use Case #2: HA cluster (1/2) - To improve availability, HA cluster is commonly used in bare-metal. - For HA cluster, Persistent Reservation SCSI command is generally used to guarantee an exclusive access from an active system. Bare-metal KVM Guest HA Cluster Active Standby Guest Guest Guest Guest OS OS OS OS HA Cluster QEMU QEMU Active Standby OS OS Host OS Host OS Server Server Server Server Persistent blocked blocked Reservation Persistent LU Reservation LU This use case also exists for KVM guests. Persistent Reservation is required to be issued to storage from guest. © Hitachi, Ltd. 2013. All rights reserved. 5 1-4. Use Case #2: HA cluster (2/2) - Persistent Reservation is held by so-called I_T nexus, the combination of initiator ID and target ID. - I/Os from standby system are blocked, because the I_T nexus is different. - Therefore, I_T nexus is required to be unique for Persistent Reservation to work properly. Bare-metal KVM Guest HA Cluster Active Standby Guest Guest Guest Guest OS OS OS OS HA Cluster initiator_3 initiator_4 Active Standby QEMU QEMU OS OS Host OS Host OS initiator_1 initiator_2 Server Server Server Server Persistent blocked Persistent Reservation Reservation target_B blocked target_A Reserved LU (initiator_1, target_A) LU Reserved (initiator_3, target_B) initiator_* : Initiator ID. target_* : Target ID. © Hitachi, Ltd. 2013. All rights reserved. 6 1-5. Use Case #3: Backup Server - Persistent Reservation is also used by some backup-server products to guarantee an exclusive access from a backup server on backup. - Persistent Reservation to storage and unique I_T nexus are required by these products. Bare-metal KVM Guest Business systems Backup Server Guest Guest Guest OS OS OS initiator_4 initiator_5 initiator_6 Business systems Backup QEMU Server OS OS OS Host OS initiator_1 initiator_2 initiator_3 Server Server Server Server blocked Persistent blocked Persistent Reservation Reservation target_A target_B LU LU LU LU initiator_* : Initiator ID. target_* : Target ID. © Hitachi, Ltd. 2013. All rights reserved. 7 1-6. Summary of Requirements Requirements from the use cases: Requirement #1: SCSI commands to storage from guests. - Thin-provisioned storage requires WriteSame to storage from guests. - HA cluster and backup server require Persistent Reservation to storage from guests. Requirement #2: Unique initiator ID across guests. - HA cluster and backup server require I_T nexus to be unique. © Hitachi, Ltd. 2013. All rights reserved. 8 Better Utilization of Storage Features from KVM Guest via virtio-scsi 2. KVM features for guest SCSI commands © Hitachi, Ltd. 2013. All rights reserved. 9 2-1. KVM features for guest disks This presentation focuses on following three device types and their configurations. Configuration # Device Type Initiator Target Backend 1 File 2 (1) virtio-blk - Device 3 LUN 4 File 5 - (a)qemu Device 6 LUN (2) virtio-scsi 7 block - (b)lio 8 pscsi 9 (c) libiscsi - iSCSI storage 10 (3)PCI device (a)Legacy - PCI device 11 assignment (b)VFIO - PCI device © Hitachi, Ltd. 2013. All rights reserved. 10 2-2. (1) virtio-blk (1/2) (1)virtio-blk Characteristic Block /dev/vdx - Para-virtualized disk. Guest Layer virtio-blk - Shown as /dev/vdX on Linux guests. - Maximum number of disks is limited by kernel SCSI Layer maximum number of PCI devices (32). - Improving performance with virtio-blk plane Device and bio-based I/O. Layer virtio-blk QEMU Block /dev/sdx Layer Host SCSI kernel Layer Device Layer Hardware Device © Hitachi, Ltd. 2013. All rights reserved. 11 2-3. (1) virtio-blk (2/2) Configurations and how SCSI commands are handled. /dev/vdx /dev/vdx /dev/vdx Guest Block virtio-blk virtio-blk virtio-blk SCSI SCSI Command Capability Device virtio-blk virtio-blk virtio-blk QEMU - SCSI command from guest reaches to storage only when Pass- attached as LUN. VFS File through Block Host LUN kernel Block Hardware Device Backend KVM Command Line Libvirt XML SCSI command Disk <disk type=‘file’ device=‘disk’> -device virtio-blk-pci,scsi=off Not Supported (file) <target dev='vda' bus='virtio'/> Disk <disk type=‘block’ device=‘disk’> -device virtio-blk-pci,scsi=off Not Supported (block) <target dev='vda' bus='virtio'/> <disk type=‘block’ device=‘lun’> LUN -device virtio-blk-pci,scsi=on Pass-through <target dev='vda' bus='virtio'/> © Hitachi, Ltd. 2013. All rights reserved. 12 2-4. (2) virtio-scsi virtio-scsi has three types of configurations. (a)Qemu target (b)lio target (c)libiscsi Block /dev/sdx /dev/sdx /dev/sdx Guest Layer kernel SCSI scsi_mod scsi_mod scsi_mod Layer virtio-scsi virtio-scsi virtio-scsi Device virtio-scsi Layer virtio-scsi libiscsi QEMU Block tcm_vhost /dev/sdx LIO Layer /dev/sdx Host SCSI kernel Layer Device Layer Hardware Device © Hitachi, Ltd. 2013. All rights reserved. 13 2-5. (2) virtio-scsi: (a) qemu target (1/2) (a)Qemu target Characteristic Block /dev/sdx Guest Layer [virtio-scsi] - Para-virtualized SCSI transport. kernel SCSI scsi_mod - Shown as /dev/sdX on Linux guests. Layer virtio-scsi Device [Qemu target] Layer virtio-scsi - User space target QEMU Block /dev/sdx Layer Host SCSI kernel Layer Device Layer Hardware Device © Hitachi, Ltd. 2013. All rights reserved. 14 2-6. (2) virtio-scsi (a) qemu-target (2/2) Configurations and how SCSI commands are handled. /dev/sdx /dev/sdx /dev/sdx Guest Block SCSI scsi_mod scsi_mod scsi_mod virtio-scsi virtio-scsi virtio-scsi SCSI Command Capability Device virtio-scsi QEMU virtio-scsi virtio-scsi - SCSI command from guest Emulated reaches to storage only when result Pass- attached as LUN. File through VFS Device - Emulated results return to Host Block LUN guest when attached as file kernel or device. Hardware Device Backend KVM Command Line Libvirt XML SCSI command Disk <disk type=‘file’ device=‘disk’> -device scsi-hd Emulated (file) <target dev='sda' bus='scsi'/> Disk <disk type=‘block’ device=‘disk’> -device scsi-hd Emulated (block) <target dev='sda' bus='scsi'/> <disk type=‘block’ device=‘lun’> LUN -device scsi-block Pass-through <target dev='sda' bus='scsi'/> © Hitachi, Ltd. 2013. All rights reserved. 15 2-7. (2) virtio-scsi: (b) lio target (b)lio target Characteristic /dev/sdx Block [virtio-scsi] Guest Layer - Para-virtualized SCSI transport. kernel SCSI scsi_mod - Shown as /dev/sdX on Linux guests. Layer virtio-scsi [lio target] Device - Kernel space target. Layer QEMU - Using LIO (linux-iscsi.org ) as backend. - LIO supports following back-stores: - block Block tcm_vhost - fileio Layer LIO - pscsi /dev/sdx - ramdisk Host SCSI kernel Layer Device SCSI Command Capability Layer - Not yet evaluated. (pscsi is pass-through SCSI, therefore Hardware Device it is expected to work well.) © Hitachi, Ltd. 2013. All rights reserved. 16 2-8. (2) virtio-scsi: (c) libiscsi (c)libiscsi Characteristic /dev/sdx Block [virtio-scsi] Guest Layer - Para-virtualized SCSI transport. kernel SCSI scsi_mod - Shown as /dev/sdX on Linux guests. Layer virtio-scsi [libiscsi] Device virtio-scsi - User space iSCSI initiator. Layer libiscsi QEMU - Only support iSCSI. - QEMU directly talk to iSCSI storage, therefore host does not see guest disks. SCSI Command Capability Host - SCSI command from guest reaches to storage. kernel Directly talk to iSCSI storage. Hardware Device © Hitachi, Ltd. 2013. All rights reserved. 17 2-9. (3) PCI Device assignment PCI Device assignment has two types: (a) Legacy (b) VFIO Characteristic /dev/sdx /dev/sdx - Assign PCI device to guests. Block - Host PCI device is dedicated to Guest Layer one guest, therefore the number kernel SCSI scsi_mod