12/10/2013 Ceph Day London [email protected] SYNNEFO + GANETI + CEPH. VANGELIS KOUKIS, TECHNICAL LEAD, SYNNEFO 1 1 12/10/2013 Ceph Day London Running a public cloud: ~okeanos [email protected] History - Design started late 2010 - Production since July 2011 Numbers - Users: > 3500 - VMs: > 5500 currently active - More than 160k VMs spawned so far, more than 44k networks 2 2 12/10/2013 Ceph Day London Running a public cloud: ~okeanos [email protected] Our choices - Build own AWS-like service (Compute, Network, Storage) - Persistent VMs - Everything open source - Production-quality IaaS - Super-simple UI How? 3 3 12/10/2013 Ceph Day London Running a public cloud: ~okeanos [email protected] The tough stuff - Stability - Persistent VMs: VMs are not cattle, they are pets - Commodity hardware - Scalability - Manageability: Gradual rollout of upgrades and new features 4 4 12/10/2013 Ceph Day London Running a public cloud: ~okeanos [email protected] Our approach - Synnefo - Google Ganeti -DRBD -Ceph -OpenStackAPIs 5 5 ~okeanos VMs 3000 3500 4000 4500 1000 1500 2000 2500 500 0 6/8/2011 6/9/2011 6/10/2011 6/11/2011 6/12/2011 6/1/2012 6/2/2012 6/3/2012 6/4/2012 6/5/2012 6/6/2012 6/7/2012 6/8/2012 6/9/2012 6/10/2012 6/11/2012 6/12/2012 6/1/2013 6/2/2013 6/3/2013 6/4/2013 [email protected] Ceph DayLondon 6 VMs 12/10/2013 6 12/10/2013 Ceph Day London Cluster vs Cloud [email protected] OPENSTACK SYNNEFO UI OPENSTACK OPENSTACK API SYNNEFO CLOUD OPENSTACK CLUSTER GANETI LIBVIRT NODE KVM KVM HYPERVISOR 8 7 12/10/2013 Ceph Day London Google Ganeti [email protected] Mature, production-ready VM cluster management - used for Google’s corporate infrastructure Multiple storage backends out of the box -LVM, DRBD - Files on local or shared directory - RBD (Ceph/RADOS) External Storage Interface for SAN/NAS support Ganeti cluster = masterd on master, noded on nodes Easy to integrate into existing infrastructure - Remote API over HTTP, pre/post hooks for every action! 9 8 12/10/2013 Ceph Day London Architecture [email protected] 10 9 12/10/2013 Ceph Day London [email protected] Spawn my own Ubuntu Freeze 11 10 12/10/2013 Ceph Day London [email protected] Clone Ubuntu + user data Snapshot 12 11 12/10/2013 Ceph Day London [email protected] 13 12 12/10/2013 Ceph Day London [email protected] 14 13 12/10/2013 Ceph Day London [email protected] RADOS block I/O Monitor nodes Volume Composer Storage Mapper and object I/O Blocker Archipelago Object Storage nodes 15 14 12/10/2013 Ceph Day London Volumes: Archipelago [email protected] 16 15 12/10/2013 Ceph Day London Volumes: Archipelago [email protected] Unified storage for Files, Images Volumes Thin layer over the actual storage cluster Storage backend agnostic - NFS, RADOS, … Efficient syncing / sharing of Images as files on the storage service Zero-copy cloning of volumes from Images 17 16 12/10/2013 Ceph Day London Experience [email protected] Operations - Rolling hardware and software upgrades - kernel, Ganeti, RADOS, Synnefo - with no VM downtime - Node evacuations with live VM migrations - Cross-datacenter move, Intel AMD, no VM downtime - On-the-fly migration from NFS-backed storage to RADOS - IP renumbering of all VMs 18 17 12/10/2013 Ceph Day London Experience [email protected] Scalability - From few physical hosts to multiple racks - dynamic addition of Ganeti clusters Diverse workloads - Different network and storage backends - Choice exposed to the user 19 18 12/10/2013 Ceph Day London Try it out! [email protected] http://www.synnefo.org 20 19 12/10/2013 Ceph Day London Thank you! [email protected] 21 20.