Porting Guides

Kunpeng BoostKit for SDS

Porting Guides

Issue 03 Date 2021-03-23

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2021. All rights reserved. No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Contents

1 Bcache Porting Guide (CentOS 7.6)...... 1 1.1 Introduction...... 1 1.2 Environment Requirements...... 1 1.3 Configuring the Compilation Environment...... 2 1.4 Obtaining Source Code...... 2 1.5 Compiling and Installing Bcache...... 3 1.6 Running and Verifying Bcache...... 10 1.6.1 Setting the Default Kernel Version Started by the System...... 10 1.6.2 Installing the Bcache Configuration Tool...... 11 1.6.3 Basic Operations on Bcache...... 11 1.7 More Resources...... 14 2 Ceph 14.2.1 Porting Guide (CentOS 7.6)...... 15 2.1 Introduction...... 15 2.2 Environment Requirements...... 17 2.3 Configuring the Compilation Environment...... 18 2.4 Installing Dependencies...... 19 2.5 Compiling, Installing, Running, and Verifying Ceph...... 20 A Change History...... 23

1 Bcache Porting Guide (CentOS 7.6)

1.1 Introduction 1.2 Environment Requirements 1.3 Configuring the Compilation Environment 1.4 Obtaining Source Code 1.5 Compiling and Installing Bcache 1.6 Running and Verifying Bcache 1.7 More Resources

1.1 Introduction

Bcache Overview Bcache is a cache in the block layer of the Linux kernel. It uses solid-state drives (SSDs) as the cache of hard disk drives (HDDs) for acceleration. The Bcache kernel module is supported only by Linux 3.10 or later. Therefore, before using Bcache, you need to upgrade the kernel to 3.10 or later. For more details, see the following: https://bcache.evilpiepirate.org/ https://evilpiepirate.org/git/linux-bcache.git/tree/Documentation/bcache.txt

1.2 Environment Requirements

Operating System Table 1-1 lists the operating system requirements.

Table 1-1 Operating system requirements

Operating System Version

CentOS 7.6

Kernel 4.14.0

1.3 Configuring the Compilation Environment

Step 1 In the ARM64 platform, the default page size is 64 KB. During the registration of bcache, the superblock information of 4 KB needs to be read from the device, and the bcache-related information needs to be added to the superblock and stored in a memory page. Therefore, you need to recompile the kernel to change the page size to 4 KB and enable bcache.

Step 2 Download the bcache tool package bcache-tools 1.0.8.

https://github.com/g2p/bcache-tools/releases

Step 3 Install the compilation dependency.

yum -y install elfutils-libelf-devel bc openssl-devel ncurses-devel libssl-dev

NO TE

If the dependency package is missing, mount the image package and configure the local image source. For details, see Configuring the Local Yum Repository.

----End

1.4 Obtaining Source Code

Step 1 Create a directory.

mkdir /root/kernel

cd /root/kernel

Step 2 Download the kernel source code of CentOS 7.6.

URL: kernel-alt-4.14.0-115.el7a.0.1.src.rpm

wget http://vault.centos.org/centos/7.6.1810/os/Source/SPackages/kernel- alt-4.14.0-115.el7a.0.1.src.rpm

Step 3 Decompress the source code package.

rpm2cpio kernel-alt-4.14.0-115.el7a.0.1.src.rpm | cpio -div

tar -vxf linux-4.14.0-115.el7a.tar.xz

----End

1.5 Compiling and Installing Bcache

Step 1 Copy the local config file to the source code path. cd /root/kernel/linux-4.14.0-115.el7a/ cp /boot/config-4.14.0-115.el7a.0.1.aarch64 .config Step 2 Compile the .config file and change the page size to 4 KB. 1. make menuconfig 2. Kernel Features > Page size (64KB) > Page size (4KB) Save# Set the page size to 4 KB.

3. Press Enter to go to the next page.

4. Press Enter to display Page size. Press the Up or Down arrow key to select 4KB, 16KB, or 64KB. In this step, select 4KB. Press Enter to exit.

5. Press the Left or Right arrow key to select Exit, and press Enter to exit.

Step 3 Enable the Bcache module. 1. Turn to the outermost menu, choose Device Drivers > Multiple device driver support (RAID and LVM), and press Enter to access the lower-level menu.

2. Enter Y to select Block device as cache.

Step 4 Set a user-defined kernel version. 1. In the outermost menu, choose General setup > Local version – append to kernel release.

2. Enter a user-defined kernel version, such as bcache_kernel.

Step 5 Save the file with the original name.

Step 6 Exit.

Step 7 Delete the CONFIG_SYSTEM_TRUSTED_TRUSTED_KEYS configuration. 1. vi .config 2. Locate CONFIG_SYSTEM_TRUSTED_KEYS and change CONFIG_SYSTEM_TRUSTED_KEYS="certs/centos.pem" to CONFIG_SYSTEM_TRUSTED_KEYS="", as shown in the following example. CONFIG_MODULE_SIG_KEY="certs/signing_key.pem" CONFIG_SYSTEM_TRUSTED_KEYRING=y CONFIG_SYSTEM_TRUSTED_KEYS="" # CONFIG_SYSTEM_EXTRA_CERTIFICATE is not set # CONFIG_SECONDARY_TRUSTED_KEYRING is not set # CONFIG_SYSTEM_BLACKLIST_KEYRING is not set 3. Save the configuration and exit. Step 8 Open the .config file and check whether CONFIG_ARM64_4K_PAGES and CONFIG_BCACHE are set to y.

Step 9 Generate RPM packages.

make rpm

Directories for saving the RPM packages are shown in the red box in the preceding figure.

Step 10 Install the new kernel RPM packages.

cd /root/rpmbuild/RPMS/aarch64/

yum install -y kernel-4.14.0bcache_kernel-1.aarch64.rpm

yum install -y kernel-devel-4.14.0bcache_kernel-1.aarch64.rpm

yum install -y kernel-headers-4.14.0bcache_kernel-1.aarch64.rpm

----End

1.6 Running and Verifying Bcache

1.6.1 Setting the Default Kernel Version Started by the System

Step 1 Check the Bcache kernel version.

cat /etc/grub2-efi.cfg | grep "bcache_kernel"

NO TE

bcache_kernel indicates the self-defined kernel version. For details, see step 4 in 1.5 Compiling and Installing Bcache.

Step 2 Find the kernel version within the single quotation marks ('). In this example, the kernel version is CentOS Linux (4.14.0bcache_kernel) 7 (AltArch). [root@ceph1 aarch64]# cat /etc/grub2-efi.cfg | grep bcache_kernel menuentry 'CentOS Linux (4.14.0bcache_kernel) 7 (AltArch)' --class centos --class gnu-linux --class gnu -- class os --unrestricted $menuentry_id_option 'gnulinux-4.14.0-115.el7a.0.1.aarch64- advanced-9ba3eaee-5c33-4412-8a03-e3ed8a6c2826' { linux /vmlinuz-4.14.0bcache_kernel root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8 initrd /initramfs-4.14.0bcache_kernel.img

NO TE

If no user-defined kernel version name is set, you can run grep "menuentry" to choose a kernel version. The kernel version provided by the system contains information about 4.14.0-115.

Step 3 Set the kernel with Bcache as the default kernel started by the system. In this example, set this parameter to a user-defined version.

grub2-set-default "user-defined version"

The following uses bcache_kernel as an example. Run the following command:

grub2-set-default 'CentOS Linux (4.14.0bcache_kernel) 7 (AltArch)'

Step 4 Verify that the new configuration takes effect.

grub2-editenv list

[root@ceph1 aarch64]# grub2-editenv list saved_entry=CentOS Linux (4.14.0bcache_kernel) 7 (AltArch)

Step 5 Reboot the system.

reboot

After the system is rebooted, perform the compilation based on the preceding kernel configuration. The Bcache module has been built in the kernel and can be directly used. You can check whether Bcache exists by viewing the /sys/fs/bcache directory or by viewing /lib/modules//kernel/drivers/md/ bcache.ko.

----End

1.6.2 Installing the Bcache Configuration Tool

Prerequisites Bcache is created by running the make-bcache command. You need to install the related tool package bcache-tools. Download address: https://github.com/g2p/bcache-tools/releases. Version: v1.0.8

Procedure

Step 1 Use a transfer tool to upload the downloaded bcache-tools-1.0.8.tar.gz package to the /home directory. Step 2 Decompress the package. cd /home/ tar -zxvf bcache-tools-1.0.8.tar.gz cd /home/bcache-tools-1.0.8 Step 3 Install the dependency. yum install libblkid-devel -y Step 4 Perform the installation. make make install Step 5 Run the make-bcache command, as shown in the following figure.

----End 1.6.3 Basic Operations on Bcache ● Create Bcache backend and cache drives. make-bcache -B /dev/sdx1 -C /dev/sdx2

NO TE

-B: specifies a backend drive (data drive). -C: specifies a cache device (for accelerating the data drive). In this example, sdb is a backend drive, and sdc is a cache device.

make-bcache -B /dev/sdb -C /dev/sdc

● Delete a cache drive. a. View cset-uuid of a cache drive. bcache-super-show /dev/sd* The following uses sdc as an example: bcache-super-show /dev/sdc

b. Clear the cache. echo cset-uuid > /sys/block/bcahce/bcache/detach Example: echo 5f50eddf-69d8-45e3-9b67-7386ffdaceb7 > /sys/block/bcache0/ bcache/detach In this case, sdc is detached from bcache0.

NO TE

To re-attach the cache, perform the following command: echo cset-uuid > /sys/block/bcahce/bcache/attach ● Deregister a cache drive. echo 1 > /sys/fs/bcache//unregister Example: echo 1 > /sys/fs/bcache/5f50eddf-69d8-45e3-9b67-7386ffdaceb7/ unregister ● Disable a cache drive. echo 1 > /sys/fs/bcache//stop Example: echo 1 > /sys/fs/bcache/5f50eddf-69d8-45e3-9b67-7386ffdaceb7/stop ● Disable a backend device. echo 1 > /sys/block/bcahce/bcache/stop Example: echo 1 > /sys/block/bcache0/bcache/stop In this case, sdb and sdc are detached from bcache0.

● Uninstall the Bcache module. rmmod bcache

NO TICE

Bcache cannot be used after the Bcache module is uninstalled.

1.7 More Resources

Configuring the Local Yum Repository

Step 1 Create a local repository. cd /etc/yum.repos.d/ touch local.repo Enter the following:

name = local_repo baseurl=file:///mnt enabled=1 gpgcheck=0 Step 2 Mount the OS image source to the /mnt/ directory.

----End

2 Ceph 14.2.1 Porting Guide (CentOS 7.6)

2.1 Introduction 2.2 Environment Requirements 2.3 Configuring the Compilation Environment 2.4 Installing Dependencies 2.5 Compiling, Installing, Running, and Verifying Ceph

2.1 Introduction

Ceph Overview Ceph is a distributed storage system designed for high performance, reliability, and scalability. It provides 3-in-1 interfaces for object-, block- and file-level storage. Figure 2-1 shows the Ceph architecture.

Figure 2-1 Ceph architecture

Table 2-1 describes the modules of Ceph.

Table 2-1 Module functions Module Function

RADOS Reliable Autonomic Distributed Object Store (RADOS) is the heart of a Ceph storage cluster. Everything in Ceph is stored by RADOS in the form of objects irrespective of their data types. The RADOS layer ensures data consistency and reliability through data replication, fault detection and recovery, and data recovery across cluster nodes.

OSD Object storage daemons (OSDs) store the actual user data. Every OSD is usually bound to one physical drive. The OSDs handle the read/write requests from clients.

MON The monitor (MON) is the most important component in a Ceph cluster. It manages the Ceph cluster and maintains the status of the entire cluster. The MON ensures that related components of a cluster can be synchronized at the same time. It functions as the leader of the cluster and is responsible for collecting, updating, and publishing cluster information. To avoid single points of failure (SPOFs), multiple MONs are deployed in a Ceph environment, and they must handle the collaboration between them.

Module Function

MGR The manager (MGR) is a monitoring system that provides collection, storage, analysis (including alarming), and visualization functions. It makes certain cluster parameters available for external systems.

Librados Librados is a method that simplifies access to RADOS. Currently, it supports programming languages PHP, Ruby, Java, Python, C, and C++. It provides RADOS, a local interface of the Ceph storage cluster, and is the base component of other services such as the RADOS block device (RBD) and RADOS gateway (RGW). In addition, it provides the Portable Operating System Interface (POSIX) for the Ceph file system (CephFS). The Librados API can be used to directly access RADOS, enabling developers to create their own interfaces for accessing the Ceph cluster storage.

RBD The RADOS block device (RBD) is the Ceph block device that provides block storage for external systems. It can be mapped, formatted, and mounted like a drive to a server.

RGW The RADOS gateway (RGW) is a Ceph object gateway that provides RESTful APIs compatible with S3 and Swift. The RGW also supports multi-tenant and OpenStack Identity service (Keystone).

MDS The Ceph Metadata Server (MDS) tracks the file hierarchy and stores metadata used only for CephFS. The RBD and RGW do not require metadata. The MDS does not directly provide data services for clients.

CephFS The CephFS provides a POSlX-compatible distributed file system of any size. It depends on the Ceph MDS to track the file hierarchy, namely the metadata.

Recommended Version Ceph 14.2.1

NO TE

This document applies to Ceph 14.2.1. It also provides reference for porting Ceph of other versions.

2.2 Environment Requirements

Hardware Requirements Table 2-2 lists the hardware requirements.

Table 2-2 Hardware requirements

Item Description

Server TaiShan 200 server (model 2280) or TaiShan 200 server (model 5280)

CPU Kunpeng 920 processor

OS Requirements

Table 2-3 lists the operating system requirements.

Table 2-3 OS requirements

Item Version

CentOS CentOS Linux release 7.6.1810 (AltArch)

Kernel 4.14.0-115.el7a.0.1.aarch64

NO TE

If an OS is installed for the first time, select Infrastructure Server > Development Tools rather than Minimal Install to minimize manual operations.

2.3 Configuring the Compilation Environment

Step 1 Install the Software Collections (SCL). yum -y install centos-release-scl

Step 2 Modify the SCL repository source. vi /etc/yum.repos.d/CentOS-SCLo-scl.repo

Add the following:

baseurl=http://mirror.centos.org/altarch/7/sclo/$basearch/rh/

Step 3 Simulate the GCC 7 compilation environment.

yum -y install devtoolset-7 scl enable devtoolset-7 bash gcc --version

----End

2.4 Installing Dependencies

Obtaining the Source Code

Source code URL: https://download.ceph.com/tarballs/

Download the source code package and save it to the /home directory on the server.

Installing Dependencies

Step 1 Go to the /home directory, decompress the source code package and go to the directory generated after decompression. cd /home && tar -zxvf ceph-14.2.1.tar.gz && cd ceph-14.2.1/

Step 2 Change "http" in the scl-rh repo source to "https". vi /etc/yum.repos.d/CentOS-SCLo-scl-rh.repo

Step 3 Configure Yum certificate verification. vi /etc/yum.conf sslverify=false

Step 4 Change x86_64 in install-deps.sh to aarch64. vi /home/ceph-14.2.1/install-deps.sh

Step 5 Run install-deps.sh to install the dependencies. ./install-deps.sh

----End

2.5 Compiling, Installing, Running, and Verifying Ceph

Compiling and Installing Ceph

Step 1 Change -DCMAKE_BUILD_TYPE in do_cmake.sh to RelWithDebInfo. vi do_cmake.sh

Modify the file as follows:

NO TE

By default, do_cmake.sh compiles a debug environment. To avoid affecting the performance, use the -DCMAKE_BUILD_TYPE=RelWithDebInfo parameter to cancel the debug compilation.

Step 2 Install the RPM packaging tool and set the default directory of rpmbuild to / home. 1. Install rpmdevtools. yum install rpmdevtools -y

2. Install rpmbuild. rpmdev-setuptree

NO TE

If the compilation is performed as the root user, the rpmbuild directory is generated in the / root directory. The compilation process occupies about 20 to 30 GB space. If the space of the /root directory is small, you can change the rpmbuild directory to another directory, for example, /home. 1. Modify the .rpmmacros file after installing rpmbuild. vi /root/.rpmmacros Change %_topdir to /home/rpmbuild.

2. Run the following command to install rpmbuild again: rpmdev-setuptree Step 3 Copy the ceph.spec file in the source code package to SPECS. cp /home/ceph-14.2.1/ceph.spec /home/rpmbuild/SPECS/ Step 4 Add fields to the beginning of the ceph.spec file. vi /home/rpmbuild/SPECS/ceph.spec Add the following fields:

%define _binaries_in_noarch_packages_terminate_build 0

Step 5 Go back to the /home directory and compress the source code package into a tar.bz2 file. cd /home/ && tar -cjvf ceph-14.2.1.tar.bz2 ceph-14.2.1 Step 6 Save the tar.bz2 file to the SOURCES directory. cp ceph-14.2.1.tar.bz2 /home/rpmbuild/SOURCES/ Step 7 Run the rpmbuild command to start compilation. rpmbuild -bb /home/rpmbuild/SPECS/ceph.spec

The compilation takes a long time. After the compilation is complete, the aarch64 and noarch directories are generated in the /home/rpmbuild/RPMS/ directory. The directories include the Ceph-related RPM packages.

Step 8 Run the following command to install the RPM packages: yum -y install /home/rpmbuild/RPMS/aarch64/*.rpm

NO TE

● The RPM packages in the aarch64 directory are core components and dependencies of Ceph, covering RBD, CephFS, and RGW. ceph-debuginfo-14.2.1-0.el7.aarch64.rpm and ceph-test-14.2.1-0.el7.aarch64.rpm are optional components. If you do not need to debug Ceph during development, do not install them. ● The RPM packages in the noarch directory contain some additional Ceph functions. You do not need to install them unless otherwise specified. ● During the installation of the Ceph RPM packages, some basic dependencies may be required. You are advised to enable the server to access the Internet. The yum command automatically downloads and installs the dependencies. You may need to configure the Extra Packages for Enterprise Linux (EPEL) source for some dependencies in advance. You can run the following command to configure the EPEL source: yum -y install epel-release

----End

Running and Verifying Ceph Check the Ceph version. If the version is 14.2.1, the installation is successful.

ceph --version

A Change History

Date Description

2021-03-23 This issue is the third official release. Changed the solution name from "Kunpeng SDS solution" to "Kunpeng BoostKit for SDS".

2020-05-09 This issue is the second official release. Added some operations to "Setting the Default Kernel Version for System Startup" in the Bcache Porting Guide (CentOS 7.6).

2020-03-20 This issue is the first official release.