ISTI-CNR, NeMIS laboratory, InfraScience group InfraScience Data Center
OpenStack day 2018
Andrea Dell’Amico
Sep 21th, 2018
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 1/23 ISTI-CNR, NeMIS laboratory, InfraScience group D4Science InfraScience Data Center D-Net CNR-ISTI InfraScience laboratory
From pure PV Xen and AoE to OpenStack and Ceph: a journey
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 2/23 ISTI-CNR, NeMIS laboratory, InfraScience group D4Science InfraScience Data Center D-Net D4Science.org
Integrated technologies that provide elastic access and usage of data and data-management capabilities
D4Science web site: https://www.d4science.org
• Virtual Research Environments (VRE) that give access to multiple services • Data discovery, accessing, analysis, and transformation in standard format • Powered by gCube: https://www.gcube-system.org/
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 3/23 ISTI-CNR, NeMIS laboratory, InfraScience group D4Science InfraScience Data Center D-Net OpenAIRE, https://www.openaire.eu • Operate a pan-European (and global) network for Open Science to articles and research data across countries and across research communities • Definition and dissemination of guidelines for sharing scholarly products and links between them • Provide services for populating and provide to the public an information graph of interlinked scholarly entities • Provide services for assessing research impact of funders (the Commission in primis) and monitoring of open access trends • Powered by D-Net: http://www.d-net.research-infrastructures.eu/
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 4/23 InfraScience Data Center, 2015 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph InfraScience DC: Virtualization servers, 2015
27 servers that host virtual machines PV Xen on all the servers. • Circa 250 Xen PV VMs • Most of the VMs run on the newer and bigger servers. • The older servers did not support hardware virtualization
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 6/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph InfraScience DC: Storage, 2015
10 servers used as storage area network (SAN)
• block devices exported by ATA over Ethernet (AoE) • some block devices locally redundant using software raid • some block devices have no redundancy
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 7/23 InfraScience DC, early 2018 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph InfraScience DC: Virt servers, early 2018 19 servers that host Xen based virtual machines PV Xen on all the servers. • More than 400 Xen PV VMs • 4.8 TB of RAM, 960 CPU cores
5 servers that host oVirt (KVM) based virtual machines
• They host the corporate services • Hyperconverged setup, gluster and hypervisors run on the same hosts • The oVirt Manager runs on top of gluster too
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 9/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph InfraScience Data Center: AoE Storage
11 servers used as storage area network (SAN)
• Block devices exported by ATA over Ethernet (AoE) • Some block devices locally redundant using software raid • Some block devices have no redundancy • Network split into two bonding groups
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 10/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph InfraScience infrastructure, early 2018 - continued
2 OpenStack Regions on the GARR cluster
• 100 VMs • local instances of ldap, DNS resolver and Prometheus • 1.3 TB of RAM in each region • 650 CPU cores in each region
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 11/23 InfraScience architecture when the migration started Resources reserved to the migration experiment when we started
OpenStack hardware resources
• 3 physical hosts reserved to OpenStack
Ceph resources
• Spare disks on the current SANs to be used as Ceph OSD Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph InfraScience Data Center: Ceph Storage
7 servers Ceph cluster • Based on Luminous • Block devices • Object storage (testing) • Posix FS (testing not started yet) • Almost half of the disks still used by AoE
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 14/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph OpenStack: software choices
• RDO as the OpenStack distribution • Ceph as storage • Global GUI cloud management using ManageIQ
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 15/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph First steps, based on PackStack
• Install PackStack • Configure the private network • Configure the floating IPs network • Add a compute node • Repeat
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 16/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph TripleO quickstart
• Test the installation • Experiment with the configuration options • Test deploying the overcloud • Add nagios checks
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 17/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph TripleO production installation
• Install on baremetal on three physical hosts • Undercloud • Overcloud controller • Overcloud compute node • (A fourth node, the block storage, installed on a ceph node)
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 18/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph Starting the migration
WMs that
• Can be reprovisioned from scratch • Without significant local storage • Already redundant (behind a load balancer)
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 19/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph Hardware migration
• When a hypervisor is emptied from the old VMs, convert it to a compute node • Start a new batch of VMs migration Still a work in progress. Some VMs are too old to be migrated, a couple of Xen hypervisors are going to be operational until we can dismiss those VMs
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 20/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph OpenStack upgrade
• Not tested yet • Plan: install a new TripleO undercloud and test on it
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 21/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph Monitoring
• Nagios checks and Prometheus exporters on the VMs • Plan: integrate Nagios and hopefully Prometheus into the OpenStack components
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 22/23 Current Infrastructure when the project started ISTI-CNR, NeMIS laboratory, InfraScience group Infrastructure state at early 2018 InfraScience Data Center Migration to OpenStack and Ceph That’s all
Questions?
Andrea Dell’Amico CNR-ISTI InfraScience laboratory 23/23