BeoLink.org Design and build an inexpensive DFS Fabrizio Manfredi Furuholmen FrOSCon August 2008 Agenda BeoLink.org Overview Introduction Old way openAFS New way Hadoop CEPH Conclusion Overview BeoLink.org Why Distributed File system ? • Handle terabytes of data • Transparent to final user • Working in WAN environment • Good level of scalability • Inexpensive • Performance Overview BeoLink.org Software vs Hardware Centralize Storage DFS • Block device (SAN) • Single file system across • Shared file system (NAS) multiple computer nodes • Simple System Management • More complicated System • Single point of failure Management • Scalable • HA (sometime) Overview BeoLink.org DFS Advantages Small number of inexpensive fileservers provides similar performance to client side Increase in capacity are inexpensive Better manageability and redundancy. Overview BeoLink.org Inexpensive Terabyte Cost (SAS/FB) 12 TB Storage (SATA) • 18k $ NAS/SAN • 5k $ DFS Device Device Price Total ($) Price ($) Terabyte Cost (SATA) • 2.5k $ NAS/SAN SAN 1 37,000 37,000 • 0.7 $ DFS File Server 1 23,995 23,995 Disks Type • 500/1000GB SATA Disk reduce >50% DFS 3 2,500 7,500 Installation • space • network • supply 96 TB Storage (SATA) Software Device Device Price Total • Port extension ($) Price ($) • Special software for HA SAN 1 249,995 249,995 Discount • Dumping DFS 16 4,500 72,000 Introduction BeoLink.org DFS Distributed file systems NFS, CIFS, Netware.. Distributed fault tolerant file CODA, MS DFS.. systems Distributed parallel file VPFS2,LUSTRE.. systems Distributed parallel fault tolerant file Hadoop, GlusterFS, systems MogileFS.. Peer-to-peer file systems Ivy, Infinit.. openAFS BeoLink.org Intruduction • Client Caching • Replication Scalability • Load balance among servers while data is in use Transparent Access • Cell • Partitions and volumes and Uniform • Mount Points Namespace • In-use volume moves • Authentication and secure communication • Authorization and flexible access control Security • Single system interface • Administration tasks without system outage System Management • Delegation • Backup openAFS BeoLink.org Main Elements Cell • Cell is collection of file servers and workstation • The directories under /afs are cells, unique tree • Fileserver contains volumes Volumes • Volumes are "containers" or sets of related files and directories • Have size limit • 3 type rw, ro, backup Mount Point Directory • Access to a volume is provided through a mount point • A mount point looks and just like a static directory openAFS BeoLink.org Server Types Fileserver Server • Fileserver, delivers data files from the file server machine to workstations • Volume Server (Vol Server), performs all types of volume manipulation Database Server • Volume Location Server (VL Server), maintains the Volume Location Database (VLDB) • Protection Server (Ptserver), Users can grant access to several other users. • Authentication Server(Kaserver), AFS version of kerberos IV (deprecated). • Backup Server (Buserver), it stores information related to the Backup System. Ubik • Distributed Database openAFS BeoLink.org Implementation Problem: Company file system • Share documents • User home dir • Application file storage • WAN Environment Solution • openAFS • Scalable, HA, good in WAN, inexpensive • More then 20 platforms • Samba (Gateway) • AFS windows client slow and bit unstable • Clientless • Heimdal Kerberos (SSO) • KA emulation • LDAP backend • Openldap • Centralize Identity storage openAFS BeoLink.org Usage Read/Write Volume • Shared development areas • Documentation data storage • User home directories Read-Only Volume • Application deployment • Application executables (binaries, libraries, scripts) • Configuration files • Documentations (Model) openAFS BeoLink.org Design Scalability • Storage scalability (File system layer) • User scalability (Samba Gateway layer) Performance • Load balancing • Roaming user/branch office Clientless • Windows client Centralized Identity • Kerberos • Ldap openAFS BeoLink.org Tricks Cache on Plan the At least 3 separated Directory servers disk Tree Use volume Replicate Use volume name that read only much as explain data possible mount point Replicate “mount 400 clients point” per server volume openAFS BeoLink.org Enviroment 3 AFS Server (3TB) • Disk 6 x 300 SAS RAID 5 • 2 Gigabits Ethernet • 2 Processor Xeon • 2 GB Ram 2 Samba Server • Disk 2 x 73 SAS RAID 1 • 2 Gigabits Ethernet • 2 Processor Xeon • 4 GB Ram 2 Switch (Backbone) • 24 port Users • 400 Concurrent Unix • 250 Concurrent Windows openAFS BeoLink.org Linux Performance • 20-35 MB/s Write • Warm 35-100 MB/s Read • Cold 30-45 MB/s openAFS BeoLink.org Windows through Samba Performance • 18-25 MB/s Write Read • 20-50 MB/s openAFS BeoLink.org Who use it ? Morgan Stanley IT • Internal usage • Storage: 450 TB (ro)+ 15 TB (rw) • Client: 22.000 Pictage, Inc • Online picture album • Storage: 265TB ( planned growth to 425TB in twelve months) • Volumes: 800,000. • Files: 200 000 000. Embian • Internet Shared folder • Storage: 500TB • Server: 200 Storage server • 300 App server RZH • Internal usage 210TB openAFS BeoLink.org Good for.. Good • General purpose • Wide Area Network • Heterogeneous System • Read operation > write operation • Small File Bad • Locking • Database • Unicode • Performance (until OSD) New way BeoLink.org • Object-based storage • Separation of file metadata management (MDS) from OS the storage of file data • Object storage devices • Replace the traditional block-level interface with one named object OSDs New way BeoLink.org • Multiple streams are parallel channels through which data can flow, thus improving the rate at which data can be written to the storage media Stream • Files are striped across a set of nodes in order to facilitate parallel access • Chunk simplify fault tolerance operation. Chunk Hadoop BeoLink.org Introduction Scalable: can reliably store and process petabytes. Economical: It distributes the data and processing across clusters of commonly available computers. “Moving Computation is Cheaper than Moving Data” Efficient: can process data in parallel on the nodes where the data is located. Reliable: automatically maintains multiple copies of data and automatically redeploys computing tasks based on failures. Hadoop BeoLink.org MapReduce MapReduce • it is an associated implementation for processing and generating large data sets. Map • It is a function that processes a key/value pair to generate a set of intermediate key/value pairs, Reduce • It is a function that merges all intermediate values associated with the same intermediate key. Hadoop BeoLink.org Map and Reduce Map • Split and mapped in key- value pairs Combine • For efficiency reasons, the combiner work directly to map operation outputs . Reduce • The files are then merge, sorted and reduced Hadoop BeoLink.org HDFS Architecture Hadoop BeoLink.org Implementation Problem: Log centralization • Centralized log, keep track of all system activity • Search and statistics Solution • HDFS • Scalable, HA, distribution task • Hearbeat+DRDB • HA namenode • Syslog-ng • Flexible and scalable • Grep MapReduce Function • Mail logging • Firewall logging • Webserver logging • Generic Syslog Hadoop BeoLink.org Solution: Scale on demand • Increase syslog concentrator • Hadoop cluster size Performance • Dedicated mapReduce function for report and search • Parallel operation High Availability • Internal replication • Distribution on different shelf Hadoop BeoLink.org Enviroment 2 Log Server • Disk 2 x 143 SAS RAID 1 • 2 Gigabits Ethernet • 2 Processor Xeon • 4 GB Ram 2 Switch (Backbone) • 24 port Gigabit Hadoop • 2 namenode 8gb,300GB, 2Xeon • 5 node 4gb, 2TB, 2 Xeon Hadoop BeoLink.org Tricks Much server Parallel Block size as possible Streams Map / Good No old Reduce/ Network hardware Partitioning (Gigabits) fuctions Simple Software distribution Hadoop BeoLink.org Who use it ? Yahoo! • 2000 nodes (2*4cpu boxes w 3TB disk each) • Used to support research for Ad Systems and Web Search A9.com - Amazon • Amazon's product search Facebook • Internal log storage • Reporting/analytics and machine learning • 320 machine cluster with 2,560 cores and about 1.3 PB raw storage Last.fm • Charts calculation and web log analysis • 25 node cluster (dual Xeon LV 2GHz, 4GB RAM, 1TB/node storage) • 10 node cluster (dual Xeon L5320 1.86GHz, 8GB RAM, 3TB/node storage) Hadoop BeoLink.org Good for.. Good • Task distribution (Basic GRID infrastructure) • Distribution of content (High throughput of data access ) • Read operations >> Write operations Bad • Not General purpose File system • Not Posix Compliant • Low granularity in security setting Ceph BeoLink.org Next Generation Ceph addresses three critical challenges of system storage Scalability Performance Reliability Ceph BeoLink.org Introduction Capabilities • POSIX semantics. • Seamless scaling from a few nodes to many thousands • Gigabytes to Petabytes • High availability and reliability • No single points of failure • N-way replication of all data across multiple nodes • Automatic rebalancing of data on node addition/ removal to efficiently utilize device resources • Easy deployment (userspace daemons) Ceph BeoLink.org Architecture • Client • Metadata Cluster OSD • Object Storage Cluster Ceph BeoLink.org Architecture difference Dynamic Distributed Metadata • Metadata Storage • Dynamic Subtree Partitionin • Traffic Control Reliable Autonomic Distributed Object Storage • Data Distribution • Replication • Data Safety • Failure Detection • Recovery and Cluster Updates Ceph