Petabytes of Ceph: From Tests to Production Dan van der Ster, CERN IT Storage Group [email protected] Ceph Day Switzerland, 14 June 2016 Ceph Day Suisse - Dan van der Ster 2 Outline • Storage for CERN and Particle Physics • Ceph@CERN, and how we use it • Operating Ceph at Scale • The Ceph Particle Physics Community Ceph Day Suisse - Dan van der Ster 3 Storage for CERN and Particle Physics • Huge data requirements (150PB now, 50PB++/year in future) • Worldwide LHC Grid standards for accessing and moving data • GridFTP, Xrootd to access data, FTS to move data, SRM to manage data NFS RBD CASTOR EOS AFS Ceph • Storage operated by the FDO team at CERN IT • Elsewhere: Hadoop, NFS Appliances, … Ceph Day Suisse - Dan van der Ster 4 Storage for CERN and Particle Physics • Huge data requirements (150PB now, +50PB/year in future) • Worldwide LHC Grid standards for accessing and moving data • GridFTP, Xrootd to access data, FTS to move data, SRM to manage data CERNbox A CVMFS NFS RBD CASTOR EOS F Ceph S Ceph • Shrinking OpenAFS, growing EOS (plus added CERNbox for sync) • Ceph getting its first taste of physics data/apps in CASTOR and CVMFS Ceph Day Suisse - Dan van der Ster 5 Ceph & Cinder @ CERN Ceph Day Suisse - Dan van der Ster 6 Our Ceph Clusters • Beesly + Wigner (3.6 PB + 433 TB, v0.94.7): • Cinder (various QoS types) + Glance + RadosGW • Isolated pools/disks for volumes, volumes++, RadosGW • Hardware reaching EOL this summer. • Dwight (0.5 PB, v0.94.7): • Preprod cluster for development (client side), testing upgrades / crazy ideas. • Erin (2.9 PB, v10.2.1++): • New cluster for CASTOR: disk buffer/cache in front of tape drives • Bigbang (~30 PB, master): • Playground for short term scale tests whenever CERN receives new hardware. Ceph Day Suisse - Dan van der Ster 7 Growth of the beesly cluster From ~200TB total to ~450 TB of RBD + 50 TB RGW Ceph Day Suisse - Dan van der Ster 8 OpenStack Glance + Cinder • OpenStack is still Ceph’s killer app. We’ve doubled usage in the past year. • Very stable, almost 100% uptime*. No data durability issues. • Libnss kvm crashes & 0.94.6->7 broke our record L Ceph Day Suisse - Dan van der Ster 9 Example Use-cases Just a couple… Ceph Day Suisse - Dan van der Ster10 NFS on RBD • ~50TB across 28 servers: • OpenStack VM + RBD • CentOS 7 with ZFS for DR Example: ~25 puppet masters reading node configurations at up to 40kHz • Not highly-available, but… • cheap, thinly provisioned, resizable, trivial to add new filers Ceph Day Suisse - Dan van der Ster 11 /cvmfs on RBD • RO POSIX fs to deliver software (also root fs!) over the WAN • How it works: stratums of preloaded HTTP servers + CDN of Squids + CVMFS FUSE client • Content addressable storage, inherent deduplication • Data compression at publish time. • Same arch as our NFS filers: ZFS on RBD • Stratum 0 on S3 is an future scaling option https://cernvm.cern.ch/ Ceph Day Suisse - Dan van der Ster 12 RadosGW for Volunteer Computing • LHC@Home uses BOINC for volunteer computing • Donate your home CPU cycles to LHC data processing • >10000 volunteer’s cores running in parallel • Data stage-in/out with our Ceph radosgw via Dynafed • Dynafed: The Dynamic Federation project • present a distributed repository as if it were one • redirect GET/PUT requests to the nearest copy • Auth with pre-signed URLs: keep secrets off the desktops • Now in EPEL http://lhcathome.web.cern.ch/ http://lcgdm.web.cern.ch/dynafed-dynamic-federation-project Ceph Day Suisse - Dan van der Ster 13 Operating Ceph at Scale Provisioning Large Clusters • We still use puppet-ceph (originally from eNovance, but heavily modified) • Install software/configuration/tuning, copy in keys, but don’t touch the disks • New: ceph-disk-prepare-all • Inspect the system to discover empty non-system drives/SSDs • Guess an ideal mapping of journals to HDDs • https://github.com/cernceph/ceph-scripts/blob/master/ceph-disk/ceph-disk- prepare-all • Deploying a large cluster takes one afternoon. Ceph Day Suisse - Dan van der Ster 15 Ceph Hardware Replacement • Need to replace 960-3TB OSDs with 1152 new 6TB drives • How not to do it… add new OSDs and remove old OSDs all at once • Would lead to massive re-peering, re-balancing, unacceptable IO latency. • How to do it: gradually add new & remove old OSDs • How quickly? OSD-by-OSD, server-by-server, rack-by-rack? Tweaking weights as we go? • Considerations: • We want to reuse the low OSD id’s (implies add/remove/add/remove/… loop) • We don’t want to have to babysit (need to automate the process) • We want to move rgw pools to another cluster! Ceph Day Suisse - Dan van der Ster 16 RadosGW: One endpoint, many clusters • *.cs3.cern.ch is a DNS load balanced alias • HaProxy (>=1.6) listens on public side • Mapping file from bucket name to cluster • RadosGW listens on loopback https://gist.github.com/cernceph/4a03316a31ce7abe49167c392fc827da Ceph Day Suisse - Dan van der Ster 17 Bigbang: 30PB Ceph Tes ting • To really make an impact, we need Ceph to scale to many 10s of PB’s • At OpenStack Vancouver we presented a 30PB Ceph test: • It worked, but had various issues related to osd->mon messaging volume, pool creation/deletion, osdmap churn and memory usage • We later worked with Ceph devs on further scale testing: • Ceph jewel incorporates these improvements Ceph Day Suisse - Dan van der Ster 18 Bigbang Part II • Bigbang II is a second 30PB test we’ve been running during May 2016 • Previous issues all solved. • Benchmarking: ~30GB/s seems doable • New jewel features: ms type = async • Fewer threads, no tcmalloc thrashing, lower RAM usage • Rare peering glitches, hopefully fixed in next jewel. op queue = wpq • better recovery transparency Ceph Day Suisse - Dan van der Ster 19 (Minor) Pain points Ceph Day Suisse - Dan van der Ster20 Scrubbing • Scrubbing has been a problem historically • Too many concurrent scrubs increases latency • Hammer / jewel randomize the scrub schedule. See plot à • Jewel scrub IOs go via the OSD op queue Minimal scrubbing: • Better fair sharing of disk time J • But still needs tuning L osd scrub chunk max = 1 • Need to throttle on high-BW clusters à osd scrub chunk min = 1 osd scrub priority = 1 osd scrub sleep = 0.1 Ceph Day Suisse - Dan van der Ster 21 Scrubbing • Scrubbing has been a problem historically • Too many concurrent scrubs increases latency • Hammer / jewel randomize the scrub schedule. See plot à • Jewel scrub IOs go via the OSD op queue Minimal scrubbing: • Better fair sharing of disk time J • But still needs tuning L osd scrub chunk max = 1 • Need to throttle on high-BW clusters à osd scrub chunk min = 1 osd scrub priority = 1 osd scrub sleep = 0.1 Ceph Day Suisse - Dan van der Ster 22 Balancing OSD data • We often want to fill a cluster: # PGs per OSD (minimal reweighting) • Imagine not being able to use 10% of a 140 10PB cluster !! 120 • Hammer 0.94.7 and Jewel have a 100 new gradual (test-)reweight-by- 80 utilization feature • This is a good workaround, but it 60 decreases the flexibility of the OSD tree 40 • Proactive reweighting of an empty cluster is much more effective than fixing 20 things later. 0 0 51 5,1 10,2 15,3 20,4 25,5 30,6 35,7 40,8 45,9 56,1 61,2 66,3 71,4 76,5 81,6 86,7 91,8 96,9 Ceph Day Suisse - Dan van der Ster 23 Ceph in High Energy Physics Ceph Day Suisse - Dan van der Ster24 The Broader HEP Community • Ceph is gaining popularity across the Worldwide LHC Grid: • Many OpenStack/RBD deployments + growing usage for physics • U Chicago ATLAS Tier2 (http://cern.ch/go/6T9q): running CephFS + RBD • OSiRIS Project (http://cern.ch/go/F6zS): Three U’s in Michigan building distributed Ceph infrastructure: • transparent & performant access to the same storage from any campus • Orsay/Saclay in France have a similar distributed Ceph project • STFC/RAL in the UK: 12PiB cluster for WLCG Tier1, details next slides. • Meeting monthly to discuss Ceph in HEP • [email protected] ML for discussions: http://cern.ch/go/H6Zh • Thanks to Alastair Dewhurst (STFC/RAL) for the initiative Ceph Day Suisse - Dan van der Ster 25 Echo Cluster at RAL: Ceph for WLCG Data • 12.1PiB RAW disks: 63 storage nodes, 36x5.5TiB HDD each • Platform: SL7, Ceph Jewel, 16+3 EC • Config: Quattor, ncm-ceph, ceph-deploy. • 3 hours to re-deploy from scratch • LHC users: GridFTP and XrootD gateways on libradosstriper • Dev’t: Sebastien Ponce (CERN) and Ian Johnson (STFC) • LHC + other users: RadosGW-S3 endpoints Slide by Bruno Canning (RAL) Ceph Day Suisse - Dan van der Ster 26 Which brings us to Sebastien… … Ceph and Particle Physics Data Ceph Day Suisse - Dan van der Ster27 .