Ceph: Scaling Storage for the Cloud and Beyond

Ceph: scaling storage for the cloud and beyond

Sage Weil Inktank

● why you should care ● what is it, what it does ● distributed object storage ● ceph fs ● who we are, why we do this

● diverse storage needs – object storage – block devices (for VMs) with snapshots, cloning – shared file system with POSIX, coherent caches – structured data... files, block devices, or objects? ● scale – terabytes, petabytes, exabytes – heterogeneous hardware – reliability and fault tolerance

● ease of administration ● no manual data migration, load balancing ● painless scaling – expansion and contraction – seamless migration

● linear function of size or performance ● incremental expansion – no fork-lift upgrades ● no vendor lock-in – choice of hardware – choice of software ● open

● objects – native – RESTful ● block – thin provisioning, snapshots, cloning ● file – strong consistency, snapshots

RADOSGWRADOSGW RBDRBD CEPHCEPH FSFS LIBRADOSLIBRADOS AA bucket-based bucket-based AA reliable reliable and and AA POSIX-compliant POSIX-compliant AA library library allowing allowing RESTREST gateway, gateway, fully-distributedfully-distributed distributeddistributed file file appsapps to to directly directly compatiblecompatible with with S3 S3 blockblock device, device, with with a a system,system, with with a a accessaccess RADOS, RADOS, andand Swift Swift LinuxLinux kernel kernel client client LinuxLinux kernel kernel client client withwith support support for for andand a a QEMU/KVM QEMU/KVM andand support support for for C,C, C++, C++, Java, Java, driverdriver FUSEFUSE Python,Python, Ruby, Ruby, andand PHP PHP

RADOSRADOS

AA reliable, reliable, autonomous, autonomous, distributed distributed object object store store comprised comprised of of self-healing, self-healing, self-managing, self-managing, intelligentintelligent storage storage nodes nodes

● LGPLv2 – copyleft – ok to link to proprietary code ● no copyright assignment – no dual licensing – no “enterprise-only” feature set ● active community ● commercial support

● data center scale – 10s to 10,000s of machines – terabytes to exabytes ● fault tolerant – no single point of failure – commodity hardware ● self-managing, self-healing

● pools – 1s to 100s – independent namespaces or object collections – replication level, placement policy ● objects – bazillions – blob of data (bytes to gigabytes) – attributes (e.g., “version=12”; bytes to kilobytes) – key/value bundle (bytes to gigabytes)

● more useful than (disk) blocks – names in a single flat namespace – variable size – simple API with rich semantics ● more scalable than files – no hard-to-distribute hierarchy – update semantics do not span objects – workload is trivially parallel

DISKDISK

HUMANHUMAN COMPUTERCOMPUTER DISKDISK

DISKDISK

DISKDISK HUMANHUMAN DISKDISK

HUMANHUMAN COMPUTERCOMPUTER DISKDISK

DISKDISK HUMANHUMAN DISKDISK

DISKDISK

HUMANHUMAN HUMANHUMAN DISKDISK HUMANHUMAN HUMANHUMAN DISKDISK

HUMANHUMAN DISK HUMANHUMAN DISK DISKDISK HUMANHUMAN DISKDISK HUMANHUMAN HUMANHUMAN DISKDISK (COMPUTER(COMPUTER)) HUMANHUMAN DISKDISK HUMANHUMAN HUMANHUMAN DISKDISK HUMANHUMAN HUMANHUMAN DISKDISK

HUMANHUMAN DISKDISK HUMANHUMAN DISKDISK HUMANHUMAN HUMANHUMAN DISKDISK HUMANHUMAN HUMANHUMAN (actually more like this…)

2012 Storage Developer Conference. © Inktank. All Rights Reserved. COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK HUMANHUMAN COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK HUMANHUMAN COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK HUMANHUMAN COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK

FS FS FS FS FS btrfs xfs ext4 DISK DISK DISK DISK DISK

M M M

• Maintain cluster membership and state

• Provide consensus for distributed decision-making via Paxos M • Small, odd number • These do not serve stored objects to clients

Object Storage Daemons (OSDs): • At least three in a cluster • One per disk or RAID group • Serve stored objects to clients • Intelligently peer to perform replication tasks

M M

● all objects are replicated N times

● objects are automatically placed, balanced, migrated in a dynamic cluster

● must consider physical infrastructure – ceph-osds on hosts in racks in rows in data centers

● three approaches – pick a spot; remember where you put it – pick a spot; write down where you put it – calculate where to put it, where to find it

• Pseudo-random placement algorithm

• Fast calculation, no lookup

• Repeatable, deterministic

• Ensures even distribution

• Stable mapping • Limited data migration • Rule-based configuration

• specifiable replication

• infrastructure topology aware

• allows weighting

hash(object name) % num pg

1010 1010 0101 0101 1010 1010 0101 1111 0101 1010

CRUSH(pg, cluster state, policy)

1010 1010 0101 0101 1010 1010 0101 1111 0101 1010

● monitors publish osd map that describes cluster state – ceph-osd node status (up/down, weight, IP) – CRUSH function specifying desired data distribution M ● object storage daemons (OSDs) – safely replicate and store object – migrate data as the cluster changes over time – coordinate based on shared view of reality – gossip! ● decentralized, distributed approach allows – massive scales (10,000s of servers or more) – the illusion of a single copy with consistent behavior

RADOSGWRADOSGW RBDRBD CEPHCEPH FSFS LIBRADOS AA bucket-based bucket-based AA reliable reliable and and AA POSIX-compliant POSIX-compliant A library allowing RESTREST gateway, gateway, fully-distributedfully-distributed distributeddistributed file file apps to directly compatiblecompatible with with S3 S3 blockblock device, device, with with a a system,system, with with a a access RADOS, andand Swift Swift LinuxLinux kernel kernel client client LinuxLinux kernel kernel client client with support for andand a a QEMU/KVM QEMU/KVM andand support support for for C, C++, Java, driverdriver FUSEFUSE Python, Ruby, and PHP

RADOSRADOS

LIBRADOSLIBRADOS native

MM MM MM

● client operations send to the OSD cluster – operate on a single object – can contain a sequence of operations, e.g.

● truncate object ● write new object data ● set attribute ● atomicity – all operations commit or do not commit atomically ● conditional – 'guard' operations can control whether operation is performed

● verify xattr has specific value ● assert object is a specific version – allows atomic compare-and-swap etc.

● store key/value pairs in an object – independent from object attrs or byte data payload ● based on google's leveldb – efficient random and range insert/query/removal – based on BigTable SSTable design ● exposed via key/value API – insert, update, remove – individual keys or ranges of keys ● avoid read/modify/write cycle for updating complex objects – e.g., file system directory objects

● establish stateful 'watch' on an object – client interest persistently registered with object – client keeps session to OSD open ● send 'notify' messages to all watchers – notify message (and payload) is distributed to all watchers – variable timeout – notification on completion

● all watchers got and acknowledged the notify ● use any object as a communication/synchronization channel – locking, distributed coordination (ala ZooKeeper), etc.

watch ack/commit watch ack/commit watch ack/commit

notify

notify notify notify ack ack ack

complete

● radosgw cache consistency – radosgw instances watch a single object (.rgw/notify) – locally cache bucket metadata – on bucket metadata changes (removal, ACL changes)

● write change to relevant bucket object ● send notify with bucket name to other radosgw instances – on receipt of notify

● invalidate relevant portion of cache

● dynamically loaded .so – /var/lib/rados-classes/* – implement new object “methods” using existing methods – part of I/O pipeline – simple internal API ● reads – can call existing native or class methods – do whatever processing is appropriate – return data ● writes – can call existing native or class methods – do whatever processing is appropriate – generates a resulting transaction to be applied atomically

● grep – read an object, filter out individual records, and return those ● sha1 – read object, generate fingerprint, return that ● images – rotate, resize, crop image stored in object – remove red-eye ● crypto – encrypt/decrypt object data with provided key

RADOSGW RBDRBD CEPHCEPH FSFS LIBRADOSLIBRADOS A bucket-based AA reliable reliable and and AA POSIX-compliant POSIX-compliant AA library library allowing allowing REST gateway, fully-distributedfully-distributed distributeddistributed file file appsapps to to directly directly compatible with S3 blockblock device, device, with with a a system,system, with with a a accessaccess RADOS, RADOS, and Swift LinuxLinux kernel kernel client client LinuxLinux kernel kernel client client withwith support support for for andand a a QEMU/KVM QEMU/KVM andand support support for for C,C, C++, C++, Java, Java, driverdriver FUSEFUSE Python,Python, Ruby, Ruby, andand PHP PHP

RADOSRADOS

RADOSGWRADOSGW RBD CEPHCEPH FSFS LIBRADOSLIBRADOS AA bucket-based bucket-based A reliable and AA POSIX-compliant POSIX-compliant AA library library allowing allowing RESTREST gateway, gateway, fully-distributed distributeddistributed file file appsapps to to directly directly compatiblecompatible with with S3 S3 block device, with a system,system, with with a a accessaccess RADOS, RADOS, andand Swift Swift Linux kernel client LinuxLinux kernel kernel client client withwith support support for for and a QEMU/KVM andand support support for for C,C, C++, C++, Java, Java, driver FUSEFUSE Python,Python, Ruby, Ruby, andand PHP PHP

RADOSRADOS

2012 Storage Developer Conference. © Inktank. All Rights Reserved. COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER COMPUTERCOMPUTER DISKDISK DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK

2012 Storage Developer Conference. © Inktank. All Rights Reserved. COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK VMVM COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK VM VM COMPUTERCOMPUTER DISKDISK COMPUTER DISK VMVM COMPUTER DISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK COMPUTERCOMPUTER DISKDISK

VIRTUALIZATIONVIRTUALIZATION CONTAINERCONTAINER

LIBRBDLIBRBD LIBRADOSLIBRADOS

MM MM MM

LIBRBDLIBRBD LIBRBDLIBRBD LIBRADOSLIBRADOS LIBRADOSLIBRADOS

MM MM MM

KRBDKRBD (KERNEL (KERNEL MODULE) MODULE) LIBRADOSLIBRADOS

MM MM MM

2012 Storage Developer Conference. © Inktank. All Rights Reserved. RADOS Block Device: • Storage of virtual disks in RADOS • Decouples VMs and containers • Live migration! • Images are striped across the cluster • Snapshots! • Support in • Qemu/KVM • OpenStack, CloudStack • Mainline Linux kernel • Image cloning • Copy-on-write “snapshot” of existing image

RADOSGWRADOSGW RBDRBD CEPH FS LIBRADOSLIBRADOS AA bucket-based bucket-based AA reliable reliable and and A POSIX-compliant AA library library allowing allowing RESTREST gateway, gateway, fully-distributedfully-distributed distributed file appsapps to to directly directly compatiblecompatible with with S3 S3 blockblock device, device, with with a a system, with a accessaccess RADOS, RADOS, andand Swift Swift LinuxLinux kernel kernel client client Linux kernel client withwith support support for for andand a a QEMU/KVM QEMU/KVM and support for C,C, C++, C++, Java, Java, driverdriver FUSE Python,Python, Ruby, Ruby, andand PHP PHP

RADOSRADOS

metadata 0101 data 1010

MM MM MM

2012 Storage Developer Conference. © Inktank. All Rights Reserved. Metadata Server • Manages metadata for a POSIX-compliant shared filesystem • Directory hierarchy • File metadata (owner, timestamps, mode, etc.) • Stores metadata in RADOS • Does not serve file data to clients • Only required for shared filesystem

● a scaling disaster – name → inode → etc block list → data home usr var vmlinuz – no inode table locality …

– fragmentation hosts mtab passwd ● … bin inode table include lib ● directory … ● many seeks ● difficult to partition

● block lists unnecessary

● inode table mostly useless 100 hosts – APIs are path-based, not 1 mtab passwd etc … inode-based home usr 102 var – no random table access, vmlinuz bin … include sloppy caching lib … ● embed inodes inside directories – good locality, prefetching – leverage key/value object

three metadata servers ??

2012 Storage Developer Conference. © Inktank. All Rights Reserved. 2012 Storage Developer Conference. © Inktank. All Rights Reserved. 2012 Storage Developer Conference. © Inktank. All Rights Reserved. 2012 Storage Developer Conference. © Inktank. All Rights Reserved. 2012 Storage Developer Conference. © Inktank. All Rights Reserved. DYNAMIC SUBTREE PARTITIONING

● scalable ● efficient – arbitrarily partition – hierarchical partition metadata preserve locality ● adaptive ● dynamic – move work from busy – daemons can to idle servers join/leave – replicate hot – take over for failed metadata nodes

● view ceph-mds as cache – reduce reads

● dir+inode prefetching – reduce writes journal

● consolidate multiple writes ● large journal or log – stripe over objects – two tiers

● journal for short term

● per-directory for long term directories – fast failure recovery

● lots of state – journaling is expensive up-front, cheap to recover – non-journaled state is cheap, but complex (and somewhat expensive) to recover ● yes – client sessions – actual fs metadata modifications ● no – cache provenance – open files ● lazy flush – client modifications may not be durable until fsync() or visible by another client

● highly stateful – consistent, fine-grained caching ● seamless hand-off between ceph-mds daemons –

● mount -t ceph 1.2.3.4:/ /mnt

– 3 ceph-mon RT – 2 ceph-mds RT (1 ceph-mds to -osd RT) ceph-mon ceph-osd

● cd /mnt/foo/bar

– 2 ceph-mds RT (2 ceph-mds to -osd RT)

● ls -al

– open – readdir ceph-mds ● 1 ceph-mds RT (1 ceph-mds to -osd RT) – stat each file – close

● cp * /tmp

– N ceph-osd RT

● ceph-mds tracks recursive directory stats – file sizes – file and directory counts – modification time ● virtual xattrs present full stats ● efficient

$ ls alSh | head total 0 drwxrxrx 1 root root 9.7T 20110204 15:51 . drwxrxrx 1 root root 9.7T 20101216 15:06 .. drwxrxrx 1 pomceph pg4194980 9.6T 20110224 08:25 pomceph drwxrxrx 1 mcg_test1 pg2419992 23G 20110202 08:57 mcg_test1 drwxx 1 luko adm 19G 20110121 12:17 luko drwxx 1 eest adm 14G 20110204 16:29 eest drwxrxrx 1 mcg_test2 pg2419992 3.0G 20110202 09:34 mcg_test2 drwxx 1 fuzyceph adm 1.5G 20110118 10:46 fuzyceph drwxrxrx 1 dallasceph pg275 596M 20110114 10:06 dallasceph

● volume or subvolume snapshots unusable at petabyte scale – snapshot arbitrary subdirectories ● simple interface – hidden '.snap' directory – no special tools

$ mkdir foo/.snap/one # create snapshot $ ls foo/.snap one $ ls foo/bar/.snap _one_1099511627776 # parent's snap name is mangled $ rm foo/myfile $ ls -F foo bar/ $ ls -F foo/.snap/one myfile bar/ $ rmdir foo/.snap/one # remove snapshot

● Linux kernel client – mount -t ceph 1.2.3.4:/ /mnt – export (NFS), Samba NFS SMB/CIFS (CIFS) Ganesha Samba ● ceph-fuse libcephfs libcephfs

● libcephfs.so Hadoop your app – your app libcephfs libcephfs – Samba (CIFS) ceph-fuse ceph fuse – Ganesha (NFS) kernel – Hadoop (map/reduce)

RADOSRADOS AWESOME

● limited options for scalable open source storage ● proprietary solutions – expensive – don't scale (well or out) – marry hardware and software

● industry needs to change

● Ceph created at UC Santa Cruz (2007) ● supported by DreamHost (2008-2011) ● Inktank (2012) – Los Angeles, Sunnyvale, San Francisco, remote ● growing user and developer community – Linux distros, users, cloud stacks, SIs, OEMs

sage weil [email protected] @liewegas http://github.com/ceph http://ceph.com/