Low Cost SAN
Total Page:16
File Type:pdf, Size:1020Kb
Low Cost SAN Date: 25/04/2009 Revision History Version No. Date Prepared by Significant Changes 1. 25/04/2009 Krishna Kumar (NEC HCL ST) 1 Objective The objective of this document is to provide making of Low Cost SAN using FOSS tools. We have tried to set up a SAN which has following features: • Low cost and easily affordable • Ensured Scalability • High Reliability • Easily Manageable • High Performance • Ensured Security • High availability 2 Definitions, Acronyms and Abbreviations This section provides a list of all definitions, acronyms and terms required to properly interpret this document as well as to understand SAN terms and terminology. Abbreviation Description AoE ATA over Ethernet, a open storage protocol ATA Advance Technology Attachment Targets End Point of communication ( normally refers to server side ) Initiators A host that requests access to storage device ( client end ) RHCS Red Hat Clustering Suite Heartbeat A signal periodically sent out by a hardware component in order to inform another component that it is working normally ISCSI Internet Small Computer System Interface SATA Serial ATA, a newer version of ATA interface GFS Global File System, a cluster aware filesystem for Linux SAN Storage Area Networking LVM Logical Volume Manager RAID Redundant Array of Inexpensive disks DRBD Distributed Replicated Block Device NBD Network Block Device ENBD Enhanced Network Block Device GNBD Global Network Block Device HA High Availability, a clustering solution for Linux which provides reliability, availability and serviceability FOSS Free Open Source Software DFS Distributed File System for Windows LVS Linux Virtual Server 3 References This section tells all the references and urls used to prepare this document. SN URLS 1 http://nbd.sourceforge.net/ 2 http://en.wikipedia.org/ 3 http://3ware.com/products/serial_ata2-9650.asp 4 http://www.drbd.org/ 5 http://www.linux-ha.org/ 6 http://www.linuxjournal.com/article/8149 7 http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Cluster Administration/ 4 Layered Architecture of SAN 4.1 Brief Description and Layered Architecture This is the investigation document which touches each aspect of Low Cost SAN making right from hardware, OS and Softwares. The layered architecture of our SAN is shown in following diagrams. Server Architecture: (JavaScript must be enabled in your browser to view the large image as an image overlay.) Client Architecture: (JavaScript must be enabled in your browser to view the large image as an image overlay.) In this diagram, three boxes in red color depicts the solution for windows and DFS is Microsoft distributed file system for windows server. 4.2 SAN Features and available options Having above architecture in mind we have tried to achieve all the features of low cost SAN in terms of speed, reliability, security, scalability and availability. Following table gives us an overview against the features and corresponding available options. All the softwares which we have used to achieve SAN features are available on FOSS: SAN Features Available Options On FOSS Low Cost & Simplicity AoE Protocol and corresponding softwares are available on FOSS. Security No routability provides inherent security Speed of ATA disk 1) Typical 7200-rpm SATA disk drive: 105 MB/s (sustained throughput) 2) Typical 7200-rpm PATA disk drive: 72 MB/s (sustained throughput) Speed of Ethernet 1) Gigabit Ethernet (1000baseT): 125 MB/s 2)10-Gigabit Ethernet: 1,250 MB/s Data Packets AoE simply delivers 48 bytes and data ( only extra 48 bytes ) Full Virtualization Fully Compatible with hypervisors such as Xen, VMware, Support Microsoft Virtual PC to virtualize computers that are used as servers Virtualized disk We can combine multiple 22 TB disk into a single RAID disk. Device access through Remote access to an AoE device through the Internet can be Internet achieved through tunneling, we can use software to convert local packets into routable packets at both ends of a link. Easy management of AoE AoE Tools like CEC provides a terminal interface for AoE Servers and nodes device. All the clusters and nodes can also be managed by RHCS cluster manager Connecting Multiple 24 port SATA controller PCI express card having capacity Disks >=2TB per disk Theoretical limits of AoE AoE has a limitation of 65535 major x 255 minor addresses, so devices you're limited to approximately 16 million block devices on a single broadcast domain / san. For each individual block device, the ATA lba48 addressing restricts you to about 140 PetaBytes. Diskless booting Support Diskless booting (PXE booting) is available in AoE for windows as well as for Linux Fencing RHCS fencing daemon provide fencing against corresponding failover domains Network Load balancing RHCS lvs and piranha provides network load balancing Proper Synchronization RHCS GFS/GFS2 uses DLM to provide this feature among all the nodes Block Level Redundancy DRBD is a tool available on Foss to provide high availability in SAN in terms of block level. If DRBD is used with heartbeat and rhcs, it's a very good solution for HA in storage networking. Directory level NFS fail over and auto mounting is easily handled by RHCS. Redundancy Resource Management CMAN of RHCS and Heartbeat are good solutions against and ensured this. communication among other nodes 4.3 Overview of SAN with HA/Failover The main challenge of a reliable SAN is high availability and zero down time. Thanks to the tools like LVS, RHCS, HEARTBEAT and DRBD by which we can easily restart our applications and can do migration of services. Following diagram shows the failover a node and relocation of services, so that users get their applications running even if corresponding node crashed. 5 Low level Building Block of SAN This section provides brief description of low level building block of SAN. The very first thing comes in our mind is what hard disk we are using and what are the available protocol for proper communication (in terms of read and write operation over network) exists. Of course the corresponding support in OS is also required at protocol level. Choosing of a reliable protocol is necessary according to the corresponding hard disk. (JavaScript must be enabled in your browser to view the large image as an image overlay.) Fiber Channel is much expensive and it has much extra overhead in terms of cost and resources. So, we have three choices to export our block devices over network: AoE, NBD (ENBD/GNBD), and iSCSI. The brief descriptions of these three are as follows: • AoE: ATA over Ethernet ( AoE ) is a network protocol developed by the Brantley Coile Company, designed for simple, high-performance access of SATA storage devices over Ethernet networks. It gives the possibility to build SANs with low-cost, standard technologies. AoE does not rely on network layers above Ethernet, such as IP and TCP. In this regard it is more comparable to Fiber Channel over Ethernet than iSCSI. While the non-routability means AoE cannot be accessed over the Internet or other IP networks, the feature makes AoE more lightweight (with less load on the host), easier to implement, provides a layer of inherent security, and offers higher performance. Its support is available on Linux, Windows, Mac OS X, Free BSD and for plan 9 from Bell Labs. • NBD: The Linux Network Block Device (NBD) is a device driver extension to the Linux kernel. With the NBD device driver you can create a TCP/IP network connection between your local Linux system and a server program on a remote (not necessarily Linux) computer. But NBD has some limitations in terms of read/write operation and using NBD as a root file system. • iSCSI: iSCSI is Internet SCSI (Small Computer System Interface), an Internet Protocol (IP)-based storage networking standard for linking data storage facilities. By carrying SCSI commands over IP networks, iSCSI is used to facilitate data transfers over intranets and to manage storage over long distances. The protocol allows clients (called initiators ) to send SCSI commands (CDBs) to SCSI storage devices ( targets ) on remote servers. ISCSI requires TCP offload Engine and Host Bus Adapter. It has a large number of OS support available like Linux, Windows, HP-UX, AIX, NetWare etc-etc. If we are thinking in terms of Low cost San having less overhead, then AoE is a good choice for us. The advantage of AoE is that you don't have the overhead of translating ATA to SCSI and then back to ATA (if you are using ATA drives). So there is a performance pickup. Server processing load for iSCSI is much higher than AoE for equivalent throughput. AoE can spare processing cycles. iSCSI requires TCP/IP and its requisite complexity. 6 Low level Components (AoE) This section provides brief description of the low level components which are available on FOSS and in market. Since we are focusing on Low Cost SAN and we have to face all the storage networking challenges of clustering, so when it comes to choose an OS, we can go for Centos. Centos 5.2 is almost equivalent to RHEL5 having inherent RHCS (redhat clustering suite) and virtualization facility. We have to attach more and more number of hard disk having much capacity, and we have PCI card/ raid controller card available having 24 ports and 2 TB size of each disk. Therefore we have following table for our low level components: Components Description Disks SATA disks of >= 2TB capacity PCI Card/ Raid Controller 24 port SATA controller PCI express Card, SATA || raid controller card Nic Card / high quality Gigabyte multiport networking switch & gigabyte multiport Switch nic card ( having jumbo frame support ) OS Linux/Windows (Preferably Centos 5.2) Protocol AoE 7 High Level Building Block of SAN This section provides brief description about high level building blocks of SAN.