Developer Guide（KAE Encryption & Decryption）

Kunpeng Acceleration Engine

Issue 15 Date 2021-08-06

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2021. All rights reserved. No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Contents

1 Overview...... 1 2 Logical Architecture...... 2 3 Installation, Upgrade, and Uninstallation...... 4 3.1 Installing the KAE Software...... 4 3.1.1 Introduction...... 4 3.1.2 Preparing for Installation...... 6 3.1.3 Installing OpenSSL...... 10 3.1.4 Installing the KAE Software Using an RPM Package...... 12 3.1.5 Installing the KAE Software Using a DEB Package...... 15 3.1.6 Installing the KAE Software Using Source Code...... 19 3.2 Operations After Installation...... 22 3.2.1 Testing the Performance of OpenSSL and the KAE...... 22 3.3 Upgrading the KAE Software...... 25 3.3.1 Upgrading the KAE Software Using an RPM Package...... 25 3.3.2 Upgrading the KAE Software Using a DEB Package...... 26 3.4 Uninstalling the KAE Software...... 28 3.4.1 Uninstalling the KAE Software Installed Using an RPM Package...... 28 3.4.2 Uninstalling the KAE Software Installed Using a DEB Package...... 29 3.4.3 Uninstalling the KAE Software Installed Using Source Code...... 31 4 Application Cases...... 32 4.1 KAE Applications...... 32 4.1.1 KAE Usage Description...... 32 4.1.2 Example Code for the KAE...... 33 4.1.3 Using KAE Through the OpenSSL Configuration File openssl.cnf...... 33 4.1.4 Nginx Applications Accelerated by KAE...... 35 4.1.5 dm-crypt-based Transparent Partition or Drive Encryption...... 35 4.1.6 Using the KAE on a KVM...... 38 4.1.7 Using the KAE on Docker...... 40 4.1.8 MD5 Hardware Acceleration Tuning...... 41 5 Common Operations...... 42 5.1 Querying KAE Log Information...... 42

6 Troubleshooting...... 44 6.1 Initialization Failure...... 44 6.2 Failed to Identify Accelerator Devices After the KAE Installed...... 45 6.3 Failed to Upgrade the Accelerator Driver...... 46 6.4 Failed to Identify the Related Interface Symbols After the OpenSSL of a New Version Is Installed...... 46 A Appendix...... 48 A.1 Terms...... 48 A.2 Acronyms and Abbreviations...... 49 A.3 Change History...... 50

1 Overview

Kunpeng accelerator engine (KAE) is a hardware acceleration solution based on the Kunpeng 920 processor. It includes KAE encryption and decryption as well as KAEzip, which are used to accelerate SSL/TLS applications and data compression, respectively. Therefore, it can significantly reduce processor consumption and improve the processor efficiency. In addition, the KAE shields the internal processing details from the application layer. Users can quickly migrate services through the standard OpenSSL and zlib interfaces.

This document describes KAE encryption and decryption as well as its installation, test procedures, and use cases.

KAE Encryption and Decryption

The KAE encryption and decryption module uses the Kunpeng hardware acceleration engine to implement the RSA, SM3, SM4, DH, MD5, and AES algorithms. It provides high-performance symmetric and asymmetric encryption/ decryption algorithms based on the lossless user-mode driver framework. Compatible with OpenSSL 1.1.1a and later versions, it supports the synchronous and asynchronous mechanisms.

Currently, KAE supports the following algorithms:

● Digest algorithms SM3 and MD5, supporting asynchronous models. ● Symmetric encryption algorithm SM4, supporting asynchronous models and CTR, XTS, CBC, ECB, and OFB modes. ● Symmetric encryption algorithm AES, supporting asynchronous models and ECB, CTR, XTS, and CBC modes. ● Asymmetric algorithm RSA, supporting asynchronous models and key sizes 1024, 2048, 3072, and 4096. ● Key negotiation algorithm DH, supporting asynchronous models and key sizes 768, 1024, 1536, 2048, 3072, and 4096.

NO TE

For details about the operations and applications of KAE encryption and decryption, see the following sections.

2 Logical Architecture

Figure 2-1 shows the logical architecture of the KAE. ● The chip accelerator subsystem, BIOS subsystem, and BMC subsystem are built-in subsystems of the Kunpeng 920 hardware server. ● The accelerator driver subsystem is a core and provides a unified driver interface for each accelerator module to the upper layer. ● The application library subsystem for OpenSSL and zlib libraries provides standard interfaces for upper-layer applications. ● The application system is for upper-layer or user-plane applications, including big data and web applications.

Figure 2-1 Logical architecture

Table 2-1 Subsystem functions Subsystem Function Description

Chip accelerator This subsystem is integrated into the Kunpeng 920 subsystem processor. It provides accelerator capabilities register interfaces for the upper layer. This subsystem is a hardware implemented by the accelerator and is not directly presented to users.

BIOS subsystem The BIOS software system of boards determines the modules to be initialized based on the license and reports the advanced configuration and power interface (ACPI) table of the accelerator to the kernel for the accelerator driver subsystem to process.

BMC subsystem This is a BMC software system, which manages accelerator licenses for servers.

Accelerator driver This is the core part of the system and provides a unified subsystem driver interface for each accelerator module to the upper layer.

Application library This application library subsystem consists of the OpenSSL subsystem: accelerator engine and zlib replacement library, and provides standard interfaces to the upper layer.

Application This is a user system that uses the application library system subsystem or driver subsystem to accelerate applications.

3 Installation, Upgrade, and Uninstallation

3.1 Installing the KAE Software 3.2 Operations After Installation 3.3 Upgrading the KAE Software 3.4 Uninstalling the KAE Software

3.1 Installing the KAE Software

3.1.1 Introduction There are three modes to install the KAE software: Using RPM Packages, Using dpkg Packages, and Using Source Code. Select one of them.

Using RPM Packages

Installation Supported System Pros and Cons Description

To facilitate user CentOS 7.6 4.14.0-115.el7a. ● Advantages: After operations, the 0.1.aarch64 version installation, you RPM installation can use the KAE packages for some SUSE 15.1 4.12.14-195- software without commercial OSs default.aarch64 version compiling source are provided. EulerOS 2.8 4.19.36- code. vhulk1907.1.0.h410.eulerosv2r8.aar ● Disadvantages: ch64 version Only certain OS types are NeoKylin7.6 4.14.0-115.5.1.el7a. supported. 06.aarch64 version

BCLinux-R7-U6-Server-aarch64 version

Installation Supported System Pros and Cons Description

Kylin 4.0.2 (juniper) 4.15.0-70- generic version

Kylin release 4.0.2 (SP2) 4.19.36- vhulk1907.1.0.h403.ky4.aarch64 version

UniKylin Linux release 3(Core) 4.18.0-80.ky3.kb21.hw.aarch64 version

openEuler 20.03 LTS 4.19.90-2003.4.0.0036.oe1.aarch64 version

openEuler 20.03 LTS-SP1 4.19.90-2012.4.0.0053.oe1.aarch64 version

Using dpkg Packages

Installation Supported System Pros and Cons Description

To facilitate user Kylin 4.0.2 (juniper) 4.15.0-70- ● Advantages: After operations, the generic version installation, you dpkg installation can use the KAE packages for some Kylin release 4.0.2 (SP2) 4.19.36- software without commercial OSs vhulk1907.1.0.h403.ky4.aarch64 compiling source are provided. version code. You can run the Ubuntu 18.04.1 LTS 4.15.0-29- ● Disadvantages: dpkg --help generic version Only certain OS command to types are obtain more help supported. information about the dpkg tool.

Using Source Code

Installation Supported System Pros and Cons Description

You can run the CentOS 7.6 4.14.0-115.el7a. ● Advantages: This configure 0.1.aarch64 version method applies to command to most Linux OSs. configure SUSE 15.1 4.12.14-195- The compilation compilation and default.aarch64 version

Installation Supported System Pros and Cons Description

installation, the EulerOS 2.8 4.19.36- and installation make command to vhulk1907.1.0.h410.eulerosv2r8.aa can be performed compile the source rch64 version by modifying code, and the make source code. NeoKylin7.6 4.14.0-115.5.1.el7a. install command to ● Disadvantages: 06.aarch64 version install the KAE. The operation is BCLinux-R7-U6-Server-aarch64 complex and extra version configurations are required. Kylin 4.0.2 (juniper) 4.15.0-70- generic version

Kylin release 4.0.2 (SP2) 4.19.36- vhulk1907.1.0.h403.ky4.aarch64 version

UniKylin Linux release 3(Core) 4.18.0-80.ky3.kb21.hw.aarch64 version

Ubuntu 18.04.1 LTS 4.15.0-29- generic version

openEuler 20.03 LTS 4.19.90-2003.4.0.0036.oe1.aarch64 version

openEuler 20.03 LTS-SP1 4.19.90-2012.4.0.0053.oe1.aarch64 version

3.1.2 Preparing for Installation

Environment Requirements ● The KAE function is enabled on the server (for example, TaiShan 200).

NO TE

To use the accelerator in the non-virtualization scenario, you need to disable the SMMU. For details, see BIOS Parameter Reference (Kunpeng 920 Processor). ● CPU: Kunpeng 920 processor ● The iBMC V365 or later is used. ● The BIOS V105 or later is used. ● One of the following OSs is used: – CentOS 7.6 4.14.0-115.el7a.0.1.aarch64 version – SUSE 15.1 4.12.14-195-default.aarch64 version

– EulerOS 2.8 4.19.36-vhulk1907.1.0.h410.eulerosv2r8.aarch64 version – NeoKylin7.6 4.14.0-115.5.1.el7a.06.aarch64 version – BCLinux-R7-U6-Server-aarch64 version – Kylin 4.0.2 (juniper) 4.15.0-70-generic version – Kylin release 4.0.2 (SP2) 4.19.36-vhulk1907.1.0.h403.ky4.aarch64 version – UniKylin Linux release 3(Core) 4.18.0-80.ky3.kb21.hw.aarch64 version – Ubuntu 18.04.1 LTS 4.15.0-29-generic version – openEuler 20.03 LTS 4.19.90-2003.4.0.0036.oe1.aarch64 version – openEuler 20.03 LTS-SP1 4.19.90-2012.4.0.0053.oe1.aarch64 version ● The SSH remote login tool has been installed on the local PC.

Obtaining a License

Before installing the KAE, you need to install a license. The OS can identify the accelerator device only after the license is installed successfully.

NO TE

The hardware KAE of TaiShan K series servers is enabled by default. You do not need to apply for a license.

For details about how to apply for and use a license, see Huawei Server iBMC License User Guide.

Run the lspci command to check whether an accelerator exists in the OS.

NO TE

The accelerator description queried by running the lspci command varies depending on the OS. In addition to filtering by keywords, you can also check whether the following accelerator SBDF information exists. lspci | grep HPRE 79:00.0 Network and computing encryption device: Huawei Technologies Co., Ltd. HiSilicon HPRE Engine (rev 21) b9:00.0 Network and computing encryption device: Huawei Technologies Co., Ltd. HiSilicon HPRE Engine (rev 21) lspci | grep RDE 78:01.0 RAID bus controller: Huawei Technologies Co., Ltd. HiSilicon RDE Engine (rev 21) b8:01.0 RAID bus controller: Huawei Technologies Co., Ltd. HiSilicon RDE Engine (rev 21) lspci | grep ZIP 75:00.0 Processing accelerators: Huawei Technologies Co., Ltd. HiSilicon ZIP Engine (rev 21) b5:00.0 Processing accelerators: Huawei Technologies Co., Ltd. HiSilicon ZIP Engine (rev 21) lspci | grep SEC 76:00.0 Network and computing encryption device: Huawei Technologies Co., Ltd. HiSilicon SEC Engine (rev 21) b6:00.0 Network and computing encryption device: Huawei Technologies Co., Ltd. HiSilicon SEC Engine (rev 21)

Obtaining Software

NO TE

Download software packages listed in Table 3-1 or Table 3-2.

Table 3-1 OS-specific RPM or DEB software packages for the KAE Software Package Description How to Obtain

uacce-version Unified accelerator 1. Access the KAE number-1.OS framework, including the source code page type.aarch64.rpm uacce.ko and hisi_qm.ko through the uacce-version kernel modules following address: number-1.OS https:// type.aarch64.deb github.com/ kunpengcompute hisi_hpre-version Dependency: uacce RPM /KAE number-1.OS package 2. Find and click type.aarch64.rpm Coverage: hisi_hpre.ko Releases on the hisi_hpre-version kernel module KAE source code number-1.OS Support: RSA and DH page, as shown in type.aarch64.deb algorithms Figure 3-1. The Releases page is hisi_sec2-version Dependency: uacce RPM displayed. number-1.OS package 3. On the Releases type.aarch64.rpm Coverage: hisi_sec2.ko page, select and hisi_sec2-version kernel module download the RPM or DEB package of number-1.OS Support: AES, MD5, SM3, the corresponding type.aarch64.deb and SM4 algorithms versions, as shown hisi_rde-version Dependency: uacce RPM in Figure 3-2. number-1.OS package type.aarch64.rpm Coverage: hisi_rde.ko hisi_rde-version kernel module number-1.OS Support: FLEX EC algorithm type.aarch64.deb

libwd-version Coverage: libwd.so dynamic number-1.OS link library type.aarch64.rpm It provides interfaces for libwd-version the KAE. number-1.OS type.aarch64.deb

libkae-version Dependency: libwd RPM number-1.OS package type.aarch64.rpm Coverage: libkae.so libkae-version dynamic library number-1.OS Support: SM3, SM4, RSA, type.aarch64.deb AES, MD5, and DH algorithms

Figure 3-1 KAE source package page

Figure 3-2 RPM and DEB package download page

Table 3-2 Source packages for the KAE Source Package Description How to Obtain

KAE This package contains all https://github.com/ OpenSSL-related code and kunpengcompute/KA the KAE. E

Source Package Description How to Obtain

KAEdriver This package contains the https://github.com/ KAE driver code, kernel kunpengcompute/ modules such as uacce, KAEdriver hpre, zip, rde, and sec2, and the user-mode driver libwd.

3.1.3 Installing OpenSSL

Prerequisites ● The kernel-devel matching the system version has been installed. Query the current kernel version. uname -r ● perl and bzip2 have been installed. Query the version numbers of perl and bzip2. perl --version bzip2 --version ● The GCC and Make tools have been installed. The performance data varies with the GCC version. Recommended versions are GCC 7.4.1 or later and Make 3.82 or later. Query the versions of GCC and Make. gcc --version make --version ● Automake, Autoconf, and Libtool have been installed. Query the versions of Automake, Autoconf, and Libtool. automake --version autoconf --version libtool --version

For details about how to install the software, see the following NOTE.

NO TE

The default commands used to install software vary depending on the OS used. CentOS and EulerOS use the Yum tool, and SUSE uses the Zypper tool. Ensure that the system has been connected to the network or the image source has been configured. The following uses SUSE as an example to describe how to install GCC. The installation method is the same for other OSs. 1. Mount the operation image. mount -o loop /dev/sr0 /mnt 2. Configure the local Zypper image source. zypper ar file:///mnt local 3. Install GCC. zypper install gcc

Procedure

Step 1 Use the SSH remote login tool to copy the OpenSSL source package to a directory on the server. The OpenSSL version must be 1.1.1a or later.

NO TE

The OpenSSL source package can be downloaded from https://www.openssl.org/ source/old/1.1.1/.

Step 2 Use SSH to log in to the Linux CLI.

Step 3 Compile and install OpenSSL in the directory where the OpenSSL source code is stored. The following uses the default directory /usr/local. ./config -Wl,-rpath,/usr/local/lib

NO TE

This command automatically generates a Makefile based on the compilation platform and environment. You can use ./config --prefix to specify the installation path and use -Wl and -rpath to specify the paths of the libcrypto and libssl libraries on which OpenSSL depends. make make install

The OpenSSL is installed in /usr/local by default. For details, see the README file in the OpenSSL source code directory.

----End

Setting Environment Variables

Run the following command to set the environment variables. If you specified the installation directory, use the actual installation directory instead of /usr/local/lib/ engines-1.1:

export OPENSSL_ENGINES=/usr/local/lib/engines-1.1

NO TE

By default, this environment variable is mounted to the engine path of OpenSSL.

Verifying the Installation

Step 1 Run the cd command to go to the /usr/local/bin directory.

Step 2 View the OpenSSL version. ./openssl version

If information similar to the following is displayed, the installation is complete (OpenSSL 1.1.1a is used as an example).

OpenSSL 1.1.1a 20 Nov 2018

----End

3.1.4 Installing the KAE Software Using an RPM Package

Prerequisites ● You have downloaded the RPM package corresponding to the specified operating system from https://github.com/kunpengcompute/KAE/releases. ● Install the RPM package on an OS of the specified version. If the installation fails, use the source code to install the accelerator software. ● The RPM tool is available. ● OpenSSL 1.1.1a or later has been successfully installed. For details, see 3.1.3 Installing OpenSSL.

Procedure NO TE

The hisi_hpre, hisi_sec2 and hisi_rde driver software packages depend on the uacce software package, and the libkae engine software package depend on the libwd software package. ● If only the RSA and DH algorithms need to be accelerated, you are advised to install only the uacce, hisi_hpre, libwd, and libkae software packages. ● If only AES, MD5, SM3, and SM4 need to be accelerated, you are advised to install only the uacce, hisi_sec2, libwd, and libkae software packages. For details about the algorithm functions provided by RPM, see Table 3-1.

Step 1 Use SSH to copy the KAE software package to a customized directory. Step 2 Use SSH to log in to the Linux CLI as the root user. Step 3 Install the accelerator driver software package.

NO TE

If OpenSSL has been installed using the preceding default method, run the rpm -ivh *.rpm command to install all KAE software packages. Otherwise, go to Step 4 to perform the installation. rpm -ivh *.rpm Preparing... ################################# [100%] checking installed modules Updating / installing... 1:uacce-1.0.1-1.centos7.6 ################################# [ 14%] modules installed 2:libwd-1.0.1-1.centos7.6 ################################# [ 29%] 3:libkae-1.0.1-1.centos7.6 ################################# [ 43%] checking installed modules 4:hisi_hpre-1.0.1-1.centos7.6 ################################# [ 57%] modules installed checking installed modules 5:hisi_rde-1.0.1-1.centos7.6 ################################# [ 71%] modules installed checking installed modules 6:hisi_sec2-1.0.1-1.centos7.6 ################################# [ 86%] modules installed checking installed modules Step 4 Follow the following example to install the uacce-1.0.1 software package. If the RPM software has been installed in Step 3, skip this step. rpm -ivh uacce-1.0.1-1.centos7.6.aarch64.rpm Preparing... ################################# [100%] checking installed modules

Updating / installing... 1:uacce-1.0.1-1.centos7.6 ################################# [100%] modules installed Install the hisi_hpre, hisi_sec2, hisi_rde, and hisi_zip driver software packages and the libwd and libkae engine software packages in sequence. When installing the libkae software package, you need to use --prefix to specify the OpenSSL engine installation path.

rpm -ivh libkae-1.0.1-1.centos7.6.aarch64.rpm --prefix=/usr/local/openssl/lib/engines-1.1

NO TE

--prefix=/usr/local/openssl/lib/engines-1.1: the OpenSSL installation path is /usr/local/ openssl. Preparing... ################################# [100%] Updating / installing... 1:libkae-1.0.1-1.centos7.6 ################################# [100%] Step 5 Check whether the RPM software is installed in the system. rpm -ql uacce /lib/modules/4.14.0-115.el7a.0.1.aarch64/extra/hisi_qm.ko /lib/modules/4.14.0-115.el7a.0.1.aarch64/extra/uacce.ko rpm -ql hisi_sec2 hisi_hpre hisi_rde /lib/modules/4.14.0-115.el7a.0.1.aarch64/extra/hisi_sec2.ko /etc/modproe.d/hisi_sec2.conf /lib/modules/4.14.0-115.el7a.0.1.aarch64/extra/hisi_hpre.ko /etc/modproe.d/hisi_hpre.conf /lib/modules/4.14.0-115.el7a.0.1.aarch64/extra/hisi_rde.ko /etc/modproe.d/hisi_rde.conf ls -al /lib/modules/`uname -r`/extra total 3672 drwxr-xr-x. 2 root root 118 Nov 17 21:56 . drwxr-xr-x. 7 root root 4096 Nov 17 21:56 .. -rw-r--r--. 1 root root 681104 Nov 12 17:32 hisi_hpre.ko -rw-r--r--. 1 root root 618888 Nov 12 17:32 hisi_qm.ko -rw-r--r--. 1 root root 844728 Nov 12 17:32 hisi_rde.ko -rw-r--r--. 1 root root 729304 Nov 12 17:32 hisi_sec2.ko -rw-r--r--. 1 root root 467160 Nov 12 17:32 uacce.ko ls -al /etc/modprobe.d/ total 36 drwxr-xr-x. 2 root root 140 Nov 17 21:56 . drwxr-xr-x. 127 root root 8192 Nov 17 22:14 .. -rw-r--r--. 1 root root 166 Oct 30 2018 firewalld-sysctls.conf -rw-r--r--. 1 root root 44 Nov 17 21:56 hisi_hpre.conf -rw-r--r--. 1 root root 43 Nov 17 21:56 hisi_rde.conf -rw-r--r--. 1 root root 61 Nov 17 21:56 hisi_sec2.conf -rw-r--r--. 1 root root 674 Jul 4 2018 tuned.conf Step 6 Restart the system or manually load the accelerator drivers to the kernel in sequence using commands, and check whether the drivers are successfully loaded. 1. Query the uacce driver module that has been loaded to the kernel. lsmod | grep uacce 2. Load the uacce driver. modprobe uacce 3. Load the hisi_sec2 driver to the kernel based on the configuration file in /etc/ modprobe.d/hisi_sec2.conf. modprobe hisi_sec2 4. Load the hisi_hpre driver to the kernel based on the configuration file in /etc/ modprobe.d/hisi_hpre.conf. modprobe hisi_hpre 5. Load the hisi_rde driver to the kernel based on the configuration file in /etc/ modprobe.d/hisi_rde.conf.

modprobe hisi_rde 6. Query the uacce driver module that has been loaded to the kernel again. lsmod | grep uacce If the following information is displayed, the loading is successful. uacce 36864 3 hisi_sec2,hisi_qm,hisi_hpre,hisi_rde If the SUSE OS is used, perform the following operations.

NO TE

● Before loading external drivers to the SUSE OS, configure /etc/modprobe.d/10- unsupported-modules.conf and set allow_unsupported_modules to 1. ● The KAE code consists of the kernel drivers, user-mode drivers, KAE, and zlib library. The kernel driver and user-mode driver are mandatory. Install the KAE and zlib library based on actual requirements.

----End

(Optional) Setting Environment Variables If the environment variables have been set according to 3.1.3 Installing OpenSSL, skip this step. Otherwise, run the following command to export the environment variables. If you specified the installation path, change /usr/local/lib/engines-1.1 to the actual installation path.

export OPENSSL_ENGINES=/usr/local/lib/engines-1.1

NO TE

By default, this environment variable is mounted to the engine path of OpenSSL. You can also specify a user-defined path.

Verifying the Installation

Step 1 View the KAE software package. rpm -qa uacce hisi_sec2 hisi_hpre hisi_rde If the following information is displayed, the installation is successful. hisi_rde-1.0.1-1.centos7.6.aarch64 hisi_sec2-1.0.1-1.centos7.6.aarch64 uacce-1.0.1-1.centos7.6.aarch64 hisi_hpre-1.0.1-1.centos7.6.aarch64 Step 2 Check the accelerator device in the virtual file system. ls -al /sys/class/uacce/ total 0 lrwxrwxrwx. 1 root root 0 Nov 14 03:45 hisi_hpre-2 -> ../../devices/pci0000:78/0000:78:00.0/0000:79:00.0/ uacce/hisi_hpre-2 lrwxrwxrwx. 1 root root 0 Nov 14 03:45 hisi_hpre-3 -> ../../devices/pci0000:b8/0000:b8:00.0/0000:b9:00.0/ uacce/hisi_hpre-3 lrwxrwxrwx. 1 root root 0 Nov 17 22:09 hisi_rde-4 -> ../../devices/pci0000:78/0000:78:01.0/uacce/hisi_rde-4 lrwxrwxrwx. 1 root root 0 Nov 17 22:09 hisi_rde-5 -> ../../devices/pci0000:b8/0000:b8:01.0/uacce/hisi_rde-5 lrwxrwxrwx. 1 root root 0 Nov 14 08:39 hisi_sec-0 -> ../../devices/pci0000:74/0000:74:01.0/0000:76:00.0/ uacce/hisi_sec-0 lrwxrwxrwx. 1 root root 0 Nov 14 08:39 hisi_sec-1 -> ../../devices/pci0000:b4/0000:b4:01.0/0000:b6:00.0/ uacce/hisi_sec-1 Step 3 Check whether the accelerator has taken effect. The following uses RSA performance verification as an example. cd /usr/local/bin/ ./openssl speed rsa2048

sign verify sign/s verify/s rsa 2048 bits 0.001381s 0.000035s 724.1 28601.0 ./openssl speed -engine kae rsa2048 engine "kae" set. sign verify sign/s verify/s rsa 2048 bits 0.000175s 0.000021s 5730.1 46591.8

NO TE

The command output shows that the RSA performance is greatly improved after the KAE is specified. In addition, during the execution of the RSA performance command, you can view the hardware queue resources of the hisi_hpre device on a new terminal. Similarly, you can view the hardware queue resource usage of the hisi_sec2 device using the SM3/SM4 algorithm. cat /sys/class/uacce/hisi_hpre-*/attrs/available_instances If the value changes from 256 to 255, the RSA algorithm consumes a hardware unit queue of the HPRE accelerator, indicating that the KAE has taken effect. 256 255

----End 3.1.5 Installing the KAE Software Using a DEB Package

Prerequisites ● You have downloaded the DEB package corresponding to the specified operating system from https://github.com/kunpengcompute/KAE/releases. ● Install the DEB packages on an OS of the specified version. If the installation fails, use the source code to install the KAE software. ● The DPKG tool is available. ● OpenSSL 1.1.1a or later has been successfully installed. For details, see 3.1.3 Installing OpenSSL.

Procedure

NO TE

The hisi_hpre, hisi_sec2, hisi_rde, and hisi_zip driver software packages depend on uacce software package, and the libkae and libkaezip engine software packages depend on the libwd software package. ● If only the RSA and DH algorithms need to be accelerated, you are advised to install only the uacce, hisi_hpre, libwd, and libkae software packages. ● If only AES, MD5, SM3, and SM4 (commercial cryptography algorithms mandated by the State Cryptographic Administration to be used within China) need to be accelerated, you are advised to install only the uacce, hisi_sec2, libwd, and libkae software packages. For details about the algorithm functions provided by each software package, see 3.1.1 Introduction.

Step 1 Use SSH to copy the KAE software package to a customized directory.

Step 2 Use SSH to log in to the Linux CLI as the root user.

Step 3 Install the accelerator driver package. For example, you can run the dpkg -i *.deb command to install all KAE software packages. dpkg -i *.deb

Step 4 (Optional) If the DEB software has been installed in Step 3, skip this step. You can install a software package separately. The following example shows how to install the uacce-1.3.1 package. dpkg -i uacce-1.3.1-1.ubuntu18.04.arm64.deb uacce modules installed Install the hisi_hpre, hisi_sec2, and hisi_rde driver software packages and the libwd and libkae engine software packages in sequence. Install the libwd software package before installing the libkae engine package. Step 5 Check whether the DEB packages have been installed in the system. dpkg -L uacce /. /lib /lib/modules /lib/modules/4.15.0-70-generic /lib/modules/4.15.0-70-generic/extra /lib/modules/4.15.0-70-generic/extra/uacce.ko /lib/modules/4.15.0-70-generic/extra/hisi_qm.ko dpkg -L hisi-sec2 hisi-hpre hisi-rde /. /lib /lib/modules /lib/modules/4.15.0-70-generic /lib/modules/4.15.0-70-generic/extra /lib/modules/4.15.0-70-generic/extra/hisi_sec2.ko /etc /etc/modprobe.d /etc/modprobe.d/hisi_sec2.conf

/. /lib /lib/modules /lib/modules/4.15.0-70-generic /lib/modules/4.15.0-70-generic/extra /lib/modules/4.15.0-70-generic/extra/hisi_hpre.ko /etc /etc/modprobe.d /etc/modprobe.d/hisi_hpre.conf

/. /lib /lib/modules /lib/modules/4.15.0-70-generic /lib/modules/4.15.0-70-generic/extra /lib/modules/4.15.0-70-generic/extra/hisi_rde.ko /etc /etc/modprobe.d /etc/modprobe.d/hisi_rde.conf ls -al /lib/modules/`uname -r`/extra total 348 drwxr-xr-x 2 root root 4096 Apr 15 17:12 . drwxr-xr-x 6 root root 4096 Apr 15 17:12 .. -rw-r--r-- 1 root root 49080 Apr 15 09:15 hisi_hpre.ko -rw-r--r-- 1 root root 85728 Apr 15 09:15 hisi_qm.ko -rw-r--r-- 1 root root 59720 Apr 15 09:15 hisi_rde.ko -rw-r--r-- 1 root root 65536 Apr 15 09:15 hisi_sec2.ko -rw-r--r-- 1 root root 40408 Apr 15 09:15 uacce.ko ls -al /etc/modprobe.d/ total 52 drwxr-xr-x 2 root root 4096 Apr 15 17:12 . drwxr-xr-x 94 root root 4096 Apr 15 09:15 .. -rw-r--r-- 1 root root 325 Jan 28 2018 blacklist-ath_pci.conf -rw-r--r-- 1 root root 1667 Nov 13 2018 blacklist.conf -rw-r--r-- 1 root root 210 Jan 28 2018 blacklist-firewire.conf -rw-r--r-- 1 root root 697 Jan 28 2018 blacklist-framebuffer.conf -rw-r--r-- 1 root root 583 Jan 28 2018 blacklist-rare-network.conf -rw-r--r-- 1 root root 45 Apr 15 09:15 hisi_hpre.conf

-rw-r--r-- 1 root root 44 Apr 15 09:15 hisi_rde.conf -rw-r--r-- 1 root root 61 Apr 15 09:15 hisi_sec2.conf -rw-r--r-- 1 root root 347 Jan 28 2018 iwlwifi.conf -rw-r--r-- 1 root root 379 Jul 2 2018 mdadm.conf Step 6 Restart the system or manually load the accelerator drivers to the kernel in sequence using commands, and check whether the drivers are successfully loaded. 1. Query the driver module that has been loaded to the kernel. lsmod | grep uacce 2. Load the uacce driver. modprobe uacce 3. Load the hisi_sec2 driver to the kernel based on the configuration file in /etc/ modprobe.d/hisi_sec2.conf. modprobe hisi_sec2 4. Load the hisi_hpre driver to the kernel based on the configuration file in /etc/ modprobe.d/hisi_hpre.conf. modprobe hisi_hpre 5. Load the hisi_rde driver to the kernel based on the configuration file in /etc/ modprobe.d/hisi_rde.conf. modprobe hisi_rde 6. Query the driver module that has been loaded to the kernel again. lsmod | grep uacce lsmod | grep hisi_qm If the following information is displayed, the loading is successful. uacce 28672 1 hisi_qm hisi_qm 65536 4 hisi_sec2,hisi_hpre,hisi_rde uacce 28672 1 hisi_qm

NO TE

The KAE code consists of the kernel drivers, user-mode drivers, KAE, and zlib library. The kernel driver and user-mode driver are mandatory. Install the KAE and zlib library based on actual requirements.

----End

export OPENSSL_ENGINES=/usr/local/lib/engines-1.1

NO TE

By default, this environment variable is mounted to the engine path of OpenSSL. You can also specify a user-defined path.

Verifying the Installation

Step 1 View the KAE software package. dpkg -s uacce hisi-sec2 hisi-hpre hisi-rde The query result is as follows: Package: uacce Status: install ok installed

Priority: optional Section: utils Installed-Size: 512 Maintainer: HiSilicon Tech. Co., Ltd. Architecture: arm64 Version: 1.3.1 Description: This package contains the Unified/User-space-access-intended Accelerator Framework. URL: https://support.huawei.com

Package: hisi-sec2 Status: install ok installed Priority: optional Section: utils Installed-Size: 512 Maintainer: HiSilicon Tech. Co., Ltd. Architecture: arm64 Version: 1.3.1 Description: This package contains the Huawei Hisilicon SEC Accelerator Driver. URL: https://support.huawei.com

Package: hisi-hpre Status: install ok installed Priority: optional Section: utils Installed-Size: 512 Maintainer: HiSilicon Tech. Co., Ltd. Architecture: arm64 Version: 1.3.1 Description: This package contains the Huawei Hisilicon HPRE Accelerator Driver. URL: https://support.huawei.com

Package: hisi-rde Status: install ok installed Priority: optional Section: utils Installed-Size: 512 Maintainer: HiSilicon Tech. Co., Ltd. Architecture: arm64 Version: 1.3.1 Description: This package contains the Huawei Hisilicon RDE Accelerator Driver. URL: https://support.huawei.com

Step 2 Check the accelerator device in the /sys/ directory. ls -al /sys/class/uacce/

The query result is as follows: total 0 drwxr-xr-x 2 root root 0 Apr 15 17:42 . drwxr-xr-x 65 root root 0 Apr 15 16:49 .. lrwxrwxrwx 1 root root 0 Apr 15 17:42 hisi_hpre-0 -> ../../devices/pci0000:78/0000:78:00.0/0000:79:00.0/ uacce/hisi_hpre-0 lrwxrwxrwx 1 root root 0 Apr 15 17:42 hisi_hpre-1 -> ../../devices/pci0000:b8/0000:b8:00.0/0000:b9:00.0/ uacce/hisi_hpre-1 lrwxrwxrwx 1 root root 0 Apr 15 17:42 hisi_rde-4 -> ../../devices/pci0000:78/0000:78:01.0/uacce/hisi_rde-4 lrwxrwxrwx 1 root root 0 Apr 15 17:42 hisi_rde-5 -> ../../devices/pci0000:b8/0000:b8:01.0/uacce/hisi_rde-5 lrwxrwxrwx 1 root root 0 Apr 15 17:42 hisi_sec2-2 -> ../../devices/pci0000:74/0000:74:01.0/0000:76:00.0/ uacce/hisi_sec2-2 lrwxrwxrwx 1 root root 0 Apr 15 17:42 hisi_sec2-3 -> ../../devices/pci0000:b4/0000:b4:01.0/0000:b6:00.0/ uacce/hisi_sec2-3

Step 3 Run the OpenSSL command to check whether the accelerator takes effect. The following uses the RSA performance as an example. The command is as follows: [root@localhost deb]# cd /usr/local/bin/ [root@localhost bin]# ./openssl speed rsa2048 sign verify sign/s verify/s rsa 2048 bits 0.001381s 0.000035s 724.1 28601.0 [root@localhost bin]# ./openssl speed -engine kae rsa2048 engine "kae" set.

sign verify sign/s verify/s rsa 2048 bits 0.000175s 0.000021s 5730.1 46591.8

NO TE

The command output shows that the RSA performance is greatly improved after the KAE is specified. In addition, during the execution of the RSA performance command, you can view the hardware queue resources of the hpre device on a new terminal. cat /sys/class/uacce/hisi_hpre-*/attrs/available_instances If the value changes from 256 to 255, the RSA algorithm consumes a hardware unit queue of the HPRE accelerator, indicating that the KAE has taken effect. 256 255

----End 3.1.6 Installing the KAE Software Using Source Code

Prerequisites ● You have downloaded the source package. The download path for the KAE is https://github.com/kunpengcompute/ KAE. The download path for the KAEdriver is https://github.com/ kunpengcompute/KAEdriver. ● The system environment meets the requirements described in 3.1.2 Preparing for Installation. ● OpenSSL 1.1.1a or later has been successfully installed. For details, see 3.1.3 Installing OpenSSL.

Procedure

Step 1 Use a remote login tool to copy the source package of the KAE to a specified directory.

NO TE

The code contains the kernel driver, user-mode driver, OpenSSL-based KAE, and zlib library. The kernel driver and user-mode driver are mandatory. Install the KAE and zlib library based on actual requirements.

Step 2 Use SSH to log in to the Linux CLI as the root user.

Step 3 Install the kernel driver.

In the KAEdriver source code directory, go to the kae_driver directory and install the kernel driver.

cd kae_driver make make install

The accelerator driver compile to generate uacce.ko, hisi_qm.ko, hisi_sec2.ko, hisi_hpre.ko, hisi_zip.ko and hisi_rde.ko. The installation path is lib/modules/ `uname -r`/extra.

NO TE

The SUSE and CentOS kernel directory is /lib/modules/ùname -r`/, and the driver installation path is /lib/modules/ùname -r`/extra. Run the ùname -r` command to obtain the current kernel information. If other OSs do not use this directory, modify the kernel directory specified by install in the Makefile file. install: Changed $(shell mkdir -p /lib/modules/ùname -r`/extra) to $(shell mkdir -p kernel directory/extra).

Step 4 Install the user-mode driver.

In the KAEdriver source code directory, go to the warpdrive directory and install the Warpdrive driver development library. cd warpdrive sh autogen.sh ./configure make make install

The --prefix option can be added to the ./configure compilation command to specify the location where the user-mode driver needs to be installed. The dynamic library file of the user-mode driver is libwd.so. The default installation path of warpdrive is /usr/local. The dynamic library file is in /usr/local/lib.

NO TE

The KAE needs to use the OpenSSL dynamic library and warpdrive dynamic library. The installation path of the warpdrive source code must be the same as that of OpenSSL so that the KAE can locate the two dynamic libraries through LD_LIBRARY_PATH.

Step 5 Restart the system or manually load the accelerator drivers to the kernel in sequence using commands, and check whether the drivers are successfully loaded. 1. Query the uacce driver module that has been loaded to the kernel. lsmod | grep uacce 2. Load the uacce driver. modprobe uacce 3. Load the hisi_sec2 driver to the kernel based on the configuration file in /etc/ modprobe.d/hisi_sec2.conf. modprobe hisi_sec2 4. Load the hisi_hpre driver to the kernel based on the configuration file in /etc/ modprobe.d/hisi_hpre.conf. modprobe hisi_hpre 5. Load the hisi_rde driver to the kernel based on the configuration file in /etc/ modprobe.d/hisi_rde.conf. modprobe hisi_rde 6. Query the uacce driver module that has been loaded to the kernel again. lsmod | grep uacce If the following information is displayed, the loading is successful. uacce 36864 3 hisi_sec2,hisi_qm,hisi_hpre,hisi_rde

If the SUSE OS is used, perform the following operations.

NO TE

Before loading external drivers to the SUSE OS, configure /etc/modprobe.d/10- unsupported-modules.conf and set allow_unsupported_modules to 1.

Step 6 Compile and install the KAE. cd KAE chmod +x configure ./configure make clean && make make install You can add the --prefix option to the ./configure compilation command to specify the installation path of the KAE. The dynamic library file of the KAE is libkae.so. You are advised to install the KAE in the default mode. The default installation path is /usr/local. The dynamic library file is in /usr/local/lib/engines-1.1.

NO TE

If libwd and OpenSSL are not installed in default mode, run the following command to specify the installation paths of OpenSSL and libwd: ./configure --openssl_path=/usr/local/openssl –wd_path=/usr/local/libwd In this command, /usr/local/openssl and /usr/local/libwd are examples of OpenSSL and libwd installation paths, respectively.

----End

export OPENSSL_ENGINES=/usr/local/lib/engines-1.1

NO TE

By default, this environment variable is mounted to the engine path of OpenSSL. You can also specify a user-defined path.

Verifying the Installation

Step 1 Run the cd command to go to the /usr/local/lib directory or a user-defined directory. Step 2 Check the soft connection status. 1. Check the status of the libwd soft connection. ls -al /usr/local/lib/ |grep libwd If the command output shows the soft link and the .so files, the libwd is successfully installed. lrwxrwxrwx. 1 root root 14 Jun 25 11:16 libwd.so -> libwd.so.1.0.1 lrwxrwxrwx. 1 root root 14 Jun 25 11:16 libwd.so.0 -> libwd.so.1.0.1 -rwxr-xr-x. 1 root root 137280 Jun 24 11:37 libwd.so.1.0.1 2. Check the soft connection status of the KAE. ls -al /usr/local/lib/engines-1.1/ If the command output shows the soft connection and the SO files, the KAE is successfully installed. lrwxrwxrwx. 1 root root 48 Jun 25 11:21 kae.so -> /usr/local/openssl/lib/engines-1.1/kae.so.1.0.1 lrwxrwxrwx. 1 root root 48 Jun 25 11:21 kae.so.0 -> /usr/local/openssl/lib/engines-1.1/kae.so.1.0.1 -rwxr-xr-x. 1 root root 212192 Jun 24 11:37 kae.so.1.0.1

Step 3 Check the accelerator device in the virtual file system. ls -al /sys/class/uacce/

The output is displayed as follows: total 0 lrwxrwxrwx. 1 root root 0 Nov 14 03:45 hisi_hpre-2 -> ../../devices/pci0000:78/0000:78:00.0/0000:79:00.0/ uacce/hisi_hpre-2 lrwxrwxrwx. 1 root root 0 Nov 14 03:45 hisi_hpre-3 -> ../../devices/pci0000:b8/0000:b8:00.0/0000:b9:00.0/ uacce/hisi_hpre-3 lrwxrwxrwx. 1 root root 0 Nov 17 22:09 hisi_rde-4 -> ../../devices/pci0000:78/0000:78:01.0/uacce/hisi_rde-4 lrwxrwxrwx. 1 root root 0 Nov 17 22:09 hisi_rde-5 -> ../../devices/pci0000:b8/0000:b8:01.0/uacce/hisi_rde-5 lrwxrwxrwx. 1 root root 0 Nov 14 08:39 hisi_sec-0 -> ../../devices/pci0000:74/0000:74:01.0/0000:76:00.0/ uacce/hisi_sec-0 lrwxrwxrwx. 1 root root 0 Nov 14 08:39 hisi_sec-1 -> ../../devices/pci0000:b4/0000:b4:01.0/0000:b6:00.0/ uacce/hisi_sec-1

Step 4 Run the OpenSSL command to check whether the accelerator takes effect. The following uses the RSA performance as an example. The command is as follows: [root@localhost rpm]# cd /usr/local/bin/ [root@localhost bin]# ./openssl speed rsa2048 sign verify sign/s verify/s rsa 2048 bits 0.001381s 0.000035s 724.1 28601.0 [root@localhost bin]# ./openssl speed -engine kae rsa2048 engine "kae" set. sign verify sign/s verify/s rsa 2048 bits 0.000175s 0.000021s 5730.1 46591.8

NO TE

The command output shows that the RSA performance is greatly improved after the KAE is specified. In addition, during the execution of the RSA performance command, you can view the hardware queue resources of the hpre device on a new terminal. cat /sys/class/uacce/hisi_hpre-*/attrs/available_instances If the value changes from 256 to 255, the RSA algorithm consumes a hardware unit queue of the HPRE accelerator, indicating that the KAE has taken effect. 256 255

----End

3.2 Operations After Installation

3.2.1 Testing the Performance of OpenSSL and the KAE

You can run the following commands to test some accelerator functions.

NO TE

The following test case data is from the native CentOS 7.6.

Testing the RSA Performance ● Use the OpenSSL software algorithm to test the RSA performance. ./openssl speed -elapsed rsa2048 The command output is as follows: ... sign verify sign/s verify/s rsa 2048 bits 0.001384s 0.000035s 724.1 28365.8.

● Use the KAE to test the RSA performance. ./openssl speed -elapsed -engine kae rsa2048 The command output is as follows: .... sign verify sign/s verify/s rsa 2048 bits 0.000355s 0.000022s 2819.0 45478.4

NO TE

After the KAE acceleration is used, the signature performance is improved from 724.1 sign/s to 2819 sign/s.

Testing the Asynchronous RSA Performance ● Use the OpenSSL software algorithm to test the asynchronous RSA performance. ./openssl speed -elapsed -async_jobs 36 rsa2048 The command output is as follows: .... sign verify sign/s verify/s rsa 2048 bits 0.001318s 0.000032s 735.7 28555 ● Use the KAE to test the asynchronous RSA performance. The command output is as follows: linux-rmw4:/usr/local/bin # ./openssl speed -engine kae -elapsed -async_jobs 36 rsa2048 .... sign verify sign/s verify/s rsa 2048 bits 0.000018s 0.000009s 54384.1 105317.0

NO TE

After the KAE acceleration is used, the asynchronous RSA signature performance is improved from 735.7 sign/s to 54384.1 sign/s.

Testing the Performance in SM4 CBC Mode ● Use the OpenSSL software algorithm to test the performance of the SM4 CBC mode. ./openssl speed -elapsed -evp sm4-cbc The command output is as follows: You have chosen to measure elapsed time instead of user CPU time. .... Doing sm4-cbc for 3s on 10240 size blocks: 2196 sm4-cbc's in 3.00s .... type 51200 bytes 102400 bytes1048576 bytes2097152 bytes4194304 bytes8388608 bytes sm4-cbc 82312.53k 85196.80k 85284.18k 85000.85k 85284.18k 85261.26k ● Use the KAE to test the performance of the SM4 CBC mode. ./openssl speed -elapsed -engine kae -evp sm4-cbc The command output is as follows: engine "kae" set. You have chosen to measure elapsed time instead of user CPU time. ... Doing sm4-cbc for 3s on 1048576 size blocks: 11409 sm4-cbc's in 3.00s ... type 51200 bytes 102400 bytes1048576 bytes2097152 bytes4194304 bytes8388608 bytes sm4-cbc 383317.33k 389427.20k 395313.15k 392954.73k 394264.58k 394264.58k

NO TE

After KAE acceleration, the performance increases from 82312.53 kbit/s to 383317.33 kbit/s in SM4 CBC mode when the input data block size is 8 MB.

Testing the Performance in SM3 Mode ● Use the OpenSSL software algorithm to test the SM3 mode performance. ./openssl speed -elapsed -evp sm3 The result is as follows: You have chosen to measure elapsed time instead of user CPU time. Doing sm3 for 3s on 102400 size blocks: 1536 sm3's in 3.00s .... type 51200 bytes 102400 bytes1048576 bytes2097152 bytes4194304 bytes8388608 bytes sm3 50568.53k 52428.80k 52428.80k 52428.80k 52428.80k 52428.80k ● Use the KAE to test the SM3 mode performance. ./openssl speed -elapsed -engine kae -evp sm3 The result is as follows: engine "kae" set. You have chosen to measure elapsed time instead of user CPU time. Doing sm3 for 3s on 102400 size blocks: 19540 sm3's in 3.00s .... type 51200 bytes 102400 bytes 1048576 bytes 2097152 bytes 4194304 bytes 8388608 bytes sm3 648243.20k 666965.33k 677030.57k 678778.20k 676681.05k 668292.44k

NO TE

After KAE acceleration, the performance increases from 52428.80 kbit/s to 668292.44 kbit/s using the SM3 algorithm when the input data block size is 8 MB.

Testing the Asynchronous Performance of the AES Algorithm in CBC Mode ● Use the OpenSSL software algorithm to test the asynchronous performance of the AES algorithm in CBC mode. ./openssl speed -elapsed -evp aes-128-cbc -async_jobs 4 The command output is as follows: You have chosen to measure elapsed time instead of user CPU time. Doing aes-128-cbc for 3s on 51200 size blocks: 65773 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 102400 size blocks: 32910 aes-128-cbc's in 3.00s .... type 51200 bytes 102400 bytes1048576 bytes2097152 bytes4194304 bytes8388608 bytes aes-128-cbc 1122525.87k 1123328.00k 1120578.22k 1121277.27k 1119879.17k 1115684.86k ● Use the KAE to test the asynchronous performance of the AES algorithm in CBC mode. ./openssl speed -elapsed -evp aes-128-cbc -async_jobs 4 -engine kae The output is displayed as follows: engine "kae" set. You have chosen to measure elapsed time instead of user CPU time. Doing aes-128-cbc for 3s on 51200 size blocks: 219553 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 102400 size blocks: 117093 aes-128-cbc's in 3.00s .... type 51200 bytes 102400 bytes1048576 bytes2097152 bytes4194304 bytes8388608 bytes aes-128-cbc 3747037.87k 3996774.40k 1189085.18k 1196774.74k 1196979.11k 1199570.94k

NO TE

● The OpenSSL test data length is defined in the lengths_list array of the speed.c file (the speed.c file is in the app directory of the OpenSSL source package, for example, openssl-1.1.1a/apps/speed.c). Testers can modify the data here, compile and install OpenSSL by referring to 3.1.3 Installing OpenSSL and perform the test. That is how the length of the test data (such as 51,200 bytes and 102,400 bytes) is calculated. ● The AES algorithm supports only asynchronous usage when the data length is 256 KB or less. ● After KAE acceleration, the performance increases from 1,123,328.00 kbit/s to 3,996,774.40 kbit/s using the AES algorithm when the input data block size is 100 KB.

3.3 Upgrading the KAE Software

3.3.1 Upgrading the KAE Software Using an RPM Package

NO TE

If the source code is used to install the KAE, skip this section.

Scenarios Run the rpm -Uvh command to upgrade the KAE software.

Procedure

Step 1 Download the latest KAE software package. Step 2 Use SSH to log in to the Linux CLI as the root user. Step 3 In the directory where the software package is stored, upgrade the accelerator driver package and engine library package. rpm -Uvh *.rpm The following command output is displayed:

Preparing... ################################# [100%] checking installed modules uacce modules start to install Updating / installing... 1:uacce-1.2.1-1.centos7.6 ################################# [ 7%] uacce modules installed 2:libwd-1.2.1-1.centos7.6 ################################# [ 14%] 3:libkae-1.2.1-1.centos7.6 ################################# [ 21%] checking installed modules hisi_hpre modules start to install 4:hisi_hpre-1.2.1-1.centos7.6 ################################# [ 29%] hisi_hpre modules installed checking installed modules hisi_rde modules start to install 5:hisi_rde-1.2.1-1.centos7.6 ################################# [ 36%] hisi_rde modules installed checking installed modules hisi_sec2 modules start to install 6:hisi_sec2-1.2.1-1.centos7.6 ################################# [ 43%] hisi_sec2 modules installed checking installed modules Cleaning up / removing... 7:libkae-1.2.0-1.centos7.6 ################################# [ 57%] 8:hisi_sec2-1.2.0-1.centos7.6 ################################# [ 71%] hisi_sec2 modules uninstalled 9:hisi_rde-1.2.0-1.centos7.6 ################################# [ 79%] hisi_rde modules uninstalled 10:hisi_hpre-1.2.0-1.centos7.6 ################################# [ 86%] hisi_hpre modules uninstalled 11:uacce-1.2.0-1.centos7.6 ################################# [ 93%] uacce modules uninstalled 12:libwd-1.2.0-1.centos7.6 ################################# [100%] Step 4 Restart the system or manually uninstall the driver of the old version using the CLI, and then load the driver of the new version. Check whether the driver is successfully loaded.

1. Uninstall the old driver. a. Query the drivers that have been loaded to the kernel. lsmod | grep uacce The following command output is displayed: uacce 262144 5 hisi_zip,hisi_rde,hisi_hpre,hisi_sec2,hisi_qm b. Uninstall the drivers of the old version in sequence. rmmod hisi_hpre rmmod hisi_sec2 rmmod hisi_rde rmmod hisi_qm rmmod uacce c. Run the following command again. If no result is displayed, the old version is uninstalled successfully. lsmod | grep uacce 2. Load the new drivers. a. Load the uacce driver. modprobe uacce b. Load the hisi_sec2 driver to the kernel based on the configuration file in /etc/modprobe.d/hisi_sec2.conf. modprobe hisi_sec2 c. Load the hisi_hpre driver to the kernel based on the configuration file in /etc/modprobe.d/hisi_hpre.conf. modprobe hisi_hpre d. Load the hisi_rde driver to the kernel based on the configuration file in /etc/modprobe.d/hisi_rde.conf. modprobe hisi_rde e. Query the loaded drivers. lsmod | grep uacce If the following information is displayed, the loading is successful. uacce 36864 3 hisi_sec2,hisi_qm,hisi_hpre,hisi_rde

Step 5 Run the rpm -qa | grep command to check the software version after the upgrade. rpm -qa uacce hisi_sec2 hisi_hpre hisi_rde

If the version number after the upgrade is displayed, the upgrade is successful. hisi_rde-1.2.1-1.centos7.6.aarch64 hisi_sec2-1.2.1-1.centos7.6.aarch64 uacce-1.2.1-1.centos7.6.aarch64 hisi_hpre-1.2.1-1.centos7.6.aarch64

----End 3.3.2 Upgrading the KAE Software Using a DEB Package

Scenarios Use a DEB package to upgrade the KAE software.

Procedure

Step 1 Download the latest KAE software package.

Step 2 Use SSH to log in to the Linux CLI as the root user.

Step 3 Uninstall the KAE software package.

NO TE

Due to the dependency relationships, the hisi_hpre, hisi_sec2 and hisi_rde driver packages must be uninstalled before the uacce driver package is uninstalled. The libkae engine package must be uninstalled before the libwd driver package is uninstalled. dpkg -r hisi-sec2 Step 4 In the directory where the software package is stored, reinstall the accelerator driver package and engine library package. dpkg -i *.deb Step 5 Restart the system or manually uninstall the driver of the old version using the CLI, and then load the driver of the new version. Check whether the driver is successfully loaded. 1. Uninstall the old driver. a. Query the drivers that have been loaded to the kernel. lsmod | grep uacce The following command output is displayed: uacce 262144 4 hisi_rde,hisi_hpre,hisi_sec2,hisi_qm b. Uninstall the drivers of the old version in sequence. rmmod hisi_hpre rmmod hisi_sec2 rmmod hisi_rde rmmod hisi_qm rmmod uacce c. Run the following command again. If no result is displayed, the old version is uninstalled successfully. lsmod | grep uacce 2. Load the new drivers. a. Load the uacce driver. modprobe uacce b. Load the hisi_sec2 driver to the kernel based on the configuration file in /etc/modprobe.d/hisi_sec2.conf. modprobe hisi_sec2 c. Load the hisi_hpre driver to the kernel based on the configuration file in /etc/modprobe.d/hisi_hpre.conf. modprobe hisi_hpre d. Load the hisi_rde driver to the kernel based on the configuration file in /etc/modprobe.d/hisi_rde.conf. modprobe hisi_rde e. Query the loaded drivers. lsmod | grep uacce If the following information is displayed, the loading is successful. uacce 36864 3 hisi_sec2,hisi_qm,hisi_hpre,hisi_rde Step 6 Run the dpkg -s command to check the software version after the upgrade. dpkg -s uacce hisi-sec2 hisi-hpre hisi-rde If the version number after the upgrade is displayed, the upgrade is successful. Package: uacce Status: install ok installed Priority: optional Section: utils Installed-Size: 512

Maintainer: HiSilicon Tech. Co., Ltd. Architecture: arm64 Version: 1.3.1 Description: This package contains the Unified/User-space-access-intended Accelerator Framework. URL: https://support.huawei.com

3.4 Uninstalling the KAE Software

3.4.1 Uninstalling the KAE Software Installed Using an RPM Package

Scenarios Uninstall the KAE software that is installed using an RPM package if you do not need to use the KAE software or you want to install a new version of the KAE software.

Procedure Step 1 Use SSH to log in to the Linux CLI as the root user. Step 2 Uninstall the KAE software and check the uninstallation result. 1. Uninstall the KAE software package. Run the rpm -e software package name command. For example, run the following command to uninstall hisi_sec2.

rpm -e hisi_sec2

NO TE

The following uninstalls all KAE software packages: [root@localhost output]# rpm -e hisi_hpre hisi_hpre modules uninstalled [root@localhost output]# rpm -e hisi_sec2 hisi_sec2 modules uninstalled [root@localhost output]# rpm -e hisi_rde hisi_rde modules uninstalled [root@localhost output]# [root@localhost output]# rpm -e uacce uacce modules uninstalling uacce modules uninstalled [root@localhost output]# rpm -e libwd libkae [root@localhost output]# [root@localhost output]# rpm -qa uacce hisi_hpre hisi_sec2 hisi_rde [root@localhost output]# rpm -qa libwd libkae

Step 3 Restart the system or use commands to manually uninstall the drivers loaded to the kernel, and check whether the drivers are successfully uninstalled. [root@localhost rpm]# lsmod | grep uacce uacce 36864 3 hisi_sec2,hisi_qm,hisi_hpre,hisi_rde [root@localhost output]# lsmod | grep uacce uacce 262144 5 hisi_rde,hisi_hpre,hisi_sec2,hisi_qm [root@localhost output]# [root@localhost output]# rmmod hisi_hpre [root@localhost output]# rmmod hisi_sec2 [root@localhost output]# rmmod hisi_rde [root@localhost output]# rmmod hisi_qm [root@localhost output]# rmmod uacce [root@localhost output]# lsmod | grep uacce [root@localhost output]#

----End 3.4.2 Uninstalling the KAE Software Installed Using a DEB Package

Scenarios Uninstall the KAE software that is installed using a DEB package if you do not need to use the KAE software or you want to install a new version of the KAE software.

Procedure

Step 1 Use SSH to log in to the Linux CLI as the root user.

Step 2 Uninstall the KAE software and check the uninstallation result. 1. Uninstall the KAE software package. Run the dpkg -r software package name command. For example, run the following command to uninstall hisi_sec2. dpkg -r hisi-sec2 NO TE

NO TE

The following uninstalls all KAE software packages: [root@localhost output]# dpkg -r hisi-hpre (Reading database ... 79192 files and directories currently installed.) Removing hisi-hpre (1.3.2) ... hisi_hpre modules uninstalling hisi_hpre modules uninstalled [root@localhost output]# dpkg -r hisi-sec2 (Reading database ... 79190 files and directories currently installed.) Removing hisi-sec2 (1.3.2) ... hisi_sec2 modules uninstalling hisi_sec2 modules uninstalled [root@localhost output]# dpkg -r hisi-rde (Reading database ... 79186 files and directories currently installed.) Removing hisi-rde (1.3.2) ... hisi_rde modules uninstalling hisi_rde modules uninstalled [root@localhost output]# dpkg -r uacce (Reading database ... 79184 files and directories currently installed.) Removing uacce (1.3.2) ... uacce modules uninstalling uacce modules uninstalled [root@localhost output]# dpkg -r libwd libkae (Reading database ... 79181 files and directories currently installed.) Removing libkae (1.3.2) ... Removing libwd (1.3.2) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... [root@localhost output]# dpkg -L uacce hisi-hpre hisi-sec2 hisi-rde Package 'uacce' does not contain any files (!) Package 'hisi-hpre' does not contain any files (!) Package 'hisi-sec2' does not contain any files (!) [root@localhost output]# dpkg -L libwd libkae /usr /usr/local /usr/local/include

/usr /usr/local /usr/local/lib /usr/local/lib/engines-1.1

[root@localhost output]# Step 3 Restart the system or use commands to manually uninstall the drivers loaded to the kernel, and check whether the drivers are successfully uninstalled.

[root@localhost output]# lsmod | grep uacce uacce 262144 5 hisi_rde,hisi_hpre,hisi_sec2,hisi_qm [root@localhost output]# [root@localhost output]# rmmod hisi_hpre [root@localhost output]# rmmod hisi_sec2 [root@localhost output]# rmmod hisi_rde [root@localhost output]# rmmod hisi_qm [root@localhost output]# rmmod uacce [root@localhost output]# lsmod | grep uacce [root@localhost output]#

----End 3.4.3 Uninstalling the KAE Software Installed Using Source Code

Scenarios Uninstall the KAE software that is installed using source code if you do not need to use the KAE software or you want to install a new version of the KAE software.

Procedure

Step 1 Use SSH to log in to the Linux CLI as the root user. Step 2 Run the make uninstall command to uninstall the accelerator driver packages and the KAE library packages that are installed using source code. ● Uninstall the driver. cd kae_driver make uninstall ● Uninstall the warpdrive driver development library. cd warpdrive make uninstall ● Uninstall the KAE. cd KAE make uninstall Step 3 Restart the system or use commands to manually uninstall the accelerator drivers loaded to the kernel, and check whether the drivers are successfully uninstalled. [root@localhost output]# lsmod | grep uacce uacce 262144 5 hisi_rde,hisi_hpre,hisi_sec2,hisi_qm [root@localhost output]# [root@localhost output]# rmmod hisi_hpre [root@localhost output]# rmmod hisi_sec2 [root@localhost output]# rmmod hisi_rde [root@localhost output]# rmmod hisi_qm [root@localhost output]# rmmod uacce [root@localhost output]# lsmod | grep uacce [root@localhost output]#

----End

4 Application Cases

4.1 KAE Applications

4.1.1 KAE Usage Description ● If you have not purchased a license for the KAE, do not use the KAE to call the corresponding algorithms. Otherwise, the performance of the OpenSSL encryption algorithm may be affected. ● The SM4-XTS mode can be used only in kernel space. For details, see 4.1.5 dm-crypt-based Transparent Partition or Drive Encryption. ● If the packet size is smaller than 2 KB, the SM4 synchronous mode provides higher performance than the SM4 asynchronous mode. Therefore, the synchronous mode is recommended for small-packet scenarios. ● AES has implemented acceleration of software instruction sets on the AArch64 platform. Hardware acceleration has obvious asynchronous performance advantages over OpenSSL in the medium- or large-packet scenario (packet size: 16 KB to 256 KB). In this scenario, hardware acceleration is recommended. ● The compression and decompression algorithms support only the zlib and Gzip formats, and the refresh modes Z_NO_FLUSH, Z_SYNC_FLUSH, Z_FULL_FLUSH, and Z_FINISH are supported. ● The SM4 and AES asynchronous modes support the data size of 256 KB or less. If the data size is greater than 256 KB, the synchronous mode is used for calculation. ● The MD5 algorithm cannot prevent collision attacks and is not applicable to security authentication, such as SSL public key authentication or digital signature. ● The SM3 and SM4 algorithms are enabled by default. You can enable or disable the two algorithms in the openssl.cnf file.

4.1.2 Example Code for the KAE

#include #include

/* OpenSSL headers */ #include #include #include #include

int main(int argc, char **argv) { /* Initializing OpenSSL */ SSL_load_error_strings(); ERR_load_BIO_strings(); OpenSSL_add_all_algorithms();

/*You can use ENGINE_by_id Function to get the handle of the Huawei Accelerator Engine*/ ENGINE *e = ENGINE_by_id("kae"); /*Enable the KAE asynchronization function. This parameter is optional. The value 0 indicates disabled, and the value 1 indicates enabled. The asynchronization function is enabled by default. */ ENGINE_ctrl_cmd_string(e, "KAE_CMD_ENABLE_ASYNC", "1", 0) ENGINE_init(e); /*Specify the KAE for RSA-based encryption and decryption. If ENGINE_set_default_RSA(ENGINE *e) is used during initialization, e does not need to be transferred.*/ RSA *rsa = RSA_new_method(e); /*The user code*/ ...

ENGINE_free(e);

} You can also specify the KAE for the crypto algorithm during initialization. Other algorithms do not need to use the KAE. In this way, the existing code modification workload is reduced. You only need to set the KAE in a certain initialization phase.

int ENGINE_set_default_RSA(ENGINE *e); int ENGINE_set_default_DH(ENGINE *e); int ENGINE_set_default_ciphers(ENGINE *e); int ENGINE_set_default_digests(ENGINE *e); int ENGINE_set_default(ENGINE *e, unsigned int flags); For details about how to use APIs, visit the official OpenSSL website: https://www.openssl.org/docs/man1.1.0/man3/ ENGINE_set_default_ciphers.html 4.1.3 Using KAE Through the OpenSSL Configuration File openssl.cnf By using the KAE in the configuration file, you can use the KAE functions with only few application modification. As shown in the following figure, you only need to call the initialization API once to complete the corresponding configuration.

OPENSSL_init_crypto(OPENSSL_INIT_LOAD_CONFIG, NULL);

CA UTION

If the openssl req -new -x509 command is used to generate a certificate, configure the openssl.cnf file by referring to Method 2 in What Should I Do If the Certificate Generation by Running the openssl req -new -x509 Command Fails.

Add the following configuration information to openssl.cnf:

openssl_conf=openssl_def [openssl_def] engines=engine_section [engine_section] kae=kae_section [kae_section] engine_id=kae dynamic_path=/usr/local/lib/engines-1.1/kae.so KAE_CMD_ENABLE_ASYNC=1 KAE_CMD_ENABLE_SM3=1 KAE_CMD_ENABLE_SM4=1 default_algorithms=ALL init=1

NO TE

● KAE_CMD_ENABLE_ASYNC is optional. The value 0 indicates that the asynchronization function is disabled, and the value 1 indicates that the asynchronization function is enabled. By default, the asynchronization function is enabled. ● KAE_CMD_ENABLE_SM3 is optional. The value 0 indicates that the SM3 acceleration function is disabled, and the value 1 indicates that the SM3 acceleration function is enabled. By default, the SM3 acceleration function is enabled. ● KAE_CMD_ENABLE_SM4 is optional. The value 0 indicates that the SM4 acceleration function is disabled, and the value 1 indicates that the SM4 acceleration function is enabled. By default, the SM4 acceleration function is enabled. ● default_algorithms=ALL indicates that all algorithms preferentially search for the KAE. If the engine does not support the algorithm, switch to OpenSSL for computing.

Export the OPENSSL_CONF environment variable.

export OPENSSL_CONF=/home/app/openssl.cnf #Path for storing the openssl.cnf file

The following is an example of using the OpenSSL configuration file:

#include #include

/* OpenSSL headers */ #include #include #include int main(int argc, char **argv) { /* Initializing OpenSSL */

ERR_load_BIO_strings(); /* Load openssl configure */ OPENSSL_init_crypto(OPENSSL_INIT_LOAD_CONFIG, NULL); /*The user code*/ ...

4.1.4 Nginx Applications Accelerated by KAE Tengine is a web service engine developed by Alibaba based on the open source Nginx and provides the asynchronization function. The following uses Tengine as an example to describe how the KAE enables Nginx acceleration. For details, see Installing Tengine by Compiling the Source Code in the Tengine 2.2.2 Porting Guide.

Verification For details, see Compilation and Configuration in the KAE RSA Encryption and Decryption Feature Guide. 4.1.5 dm-crypt-based Transparent Partition or Drive Encryption dm-crypt is presented as a target device of the device mapper. After being mapped and mounted, dm-crypt can be used as a transparent encrypted partition or drive. The dm-crypt algorithm is registered in the Crypto module. After the hisi_sec2 driver is installed, the SM4-XTS algorithm is registered in the Crypto module. You can implement hardware encryption and decryption using the Linux Unified Key Setup (LUKS) for configuration. An operation on an encryption drive occupies 24 queues. Currently, the accelerator restricts the number of queues to 256 x 2. If more encryption drives need to be operated, you need to enable all the 1024 x 2 accelerator queues. To enable all the accelerator queues, modify the pf_q_num parameter in the /etc/modprobe.d/ hisi_sec2.conf configuration file and restart the system for the modification to take effect.

Environment Requirements ● The hisi_sec2 driver has been installed. For details about installation, see 3.1 Installing the KAE Software. ● To improve the performance of the SM4-XTS algorithm, upgrade cryptsetup (the LUKS tool) to version 2.2.0. The built-in cryptsetup software of the OS may not be able to use the SM4- XTS algorithm to encrypt drive. You need to upgrade the software. Download the cryptsetup-2.2.0 source code to the environment. The following uses EulerOS 2.8 as an example to describe how to upgrade the cryptsetup software: a. Install the libuuid-devel, device-mapper-devel, popt-devel, json-c-devel, and libblkid-develup2.2.0 dependencies in sequence. yum install libuuid-devel yum install device-mapper-devel yum install popt-devel yum install json-c-devel yum install libblkid-develup2.2.0 b. Perform the compilation and installation in the cryptsetup-2.2.0 source code directory. ./configure make && make install

libuuid-devel, device-mapper-devel, popt-devel, json-c-devel, and libblkid- develup2.2.0 are dependencies of cryptsetup.

Encrypting a Partition or Drive

Step 1 Generate the keyfile in the root directory of the system. dd if=/dev/random of=/home/EncryptKeyFile bs=4k count=1 The command output is displayed as follows: 0+1 records in 0+1 records out 115 bytes copied, 0.00010976 s, 1.0 MB/s Step 2 Encrypt the partition or drive. cryptsetup --batch-mode --cipher sm4-xts-plain64 --key-size 256 --hash sha256 --sector-size=4096 -- type=luks2 --key-file /home/EncryptKeyFile luksFormat /dev/sdb Step 3 Map the partition or drive. cryptsetup --key-file /home/EncryptKeyFile luksOpen /dev/sdb sx_disk Step 4 Check whether the partition or drive is encrypted. lsblk crypt indicates that the partition or drive has been encrypted. NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 5.5G 1 loop /os_lhl sda 8:0 0 2.2T 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi └─sda2 8:2 0 2.2T 0 part ├─vg_os-swap 254:0 0 20G 0 lvm [SWAP] └─vg_os-root 254:1 0 2.2T 0 lvm / sdb 8:16 0 278.5G 0 disk └─sx_disk 254:2 0 278.5G 0 crypt Step 5 Format the partition or drive. mkfs.xfs /dev/mapper/sx_disk The command output is displayed as follows: meta-data=/dev/mapper/sx_disk isize=512 agcount=16, agsize=4562368 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 data = bsize=4096 blocks=72997376, imaxpct=25 = sunit=64 swidth=64 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=35648, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Step 6 Create a mounting directory. mkdir /home/sec_test Step 7 Mount the partition or drive to the directory. mount /dev/mapper/sx_disk /home/sec_test/ df -h The command output is displayed as follows: Filesystem Size Used Avail Use% Mounted on devtmpfs 63G 0 63G 0% /dev tmpfs 63G 0 63G 0% /dev/shm tmpfs 63G 28M 63G 1% /run tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/mapper/vg_os-root 2.2T 18G 2.1T 1% / /dev/sda1 1022M 172K 1022M 1% /boot/efi tmpfs 13G 20K 13G 1% /run/user/472

tmpfs 13G 0 13G 0% /run/user/0 /dev/loop0 5.5G 5.5G 0 100% /os_lhl /dev/mapper/sx_disk 279G 317M 279G 1% /home/sec_test Step 8 Ensure that the directory can be accessed. cd /home/sec_test/;ll Step 9 Check that the partition or drive is encrypted in the /home/sec_test directory and that the partition or drive corresponds to the directory. lsblk The command output is displayed as follows: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 5.5G 1 loop /os_lhl sda 8:0 0 2.2T 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi └─sda2 8:2 0 2.2T 0 part ├─vg_os-swap 254:0 0 20G 0 lvm [SWAP] └─vg_os-root 254:1 0 2.2T 0 lvm / sdb 8:16 0 278.5G 0 disk └─sx_disk 254:2 0 278.5G 0 crypt /home/sec_test Step 10 View the detailed encryption information about the partition or drive in the / home directory. cryptsetup status /dev/mapper/sx_disk The command output is as follows: /dev/mapper/sx_disk is active and is in use. type: LUKS1 cipher: sm4-xts-plain64 keysize: 256 bits key location: dm-crypt device: /dev/sdb sector size: 512 offset: 4096 sectors size: 583979008 sectors mode: read/write Step 11 Perform Step 2 to Step 10 to encrypt multiple partitions or drives.

----End

Deleting an Encrypted Partition or Drive

Step 1 Unmount the partition or drive from the mounting directory.

NO TE

Before running this command, you must exit the directory. If multiple partitions or drives are mounted, you need to run this command for multiple times. umount -l /home/sec_test Step 2 Run the lsblk command to check whether the partition or drive mounting directory is unmounted. lsblk The command output is displayed as follows: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 5.5G 1 loop /os_lhl sda 8:0 0 2.2T 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi └─sda2 8:2 0 2.2T 0 part ├─vg_os-swap 254:0 0 20G 0 lvm [SWAP]

└─vg_os-root 254:1 0 2.2T 0 lvm / sdb 8:16 0 278.5G 0 disk └─sx_disk 254:2 0 278.5G 0 crypt

Step 3 Disable the mapping.

NO TE

You need to run this command for multiple times to disable the mapping of all partitions or drives. cryptsetup luksClose sx_disk

Step 4 Check whether the mapping is disabled. ll /dev/mapper/

The command output is as follows: total 0 crw---- 1 root root 10, 236 Jul 31 22:27 control lrwxrwxrwx 1 root root 7 Jul 31 22:27 vg_os-root -> ../dm-1 lrwxrwxrwx 1 root root 7 Jul 31 22:27 vg_os-swap -> ../dm-0

----End 4.1.6 Using the KAE on a KVM The KAE can be used on a kernel-based virtual machine (KVM). The accelerator device complies with the PCIe specifications. It is presented as a PCIe device on the OS and supports SR-IOV. Each accelerator provides 1024 queues. A PF uses 256 queues by default, and the other 768 queues are reserved for VFs.

Number of VF queues = (1024 – Number of PF queues)/Number of VFs

The remainder queue is added to the last VF.

It is recommended that one PF be virtualized into eight VFs.

Environment Requirements ● A VM has been created on the host OS. ● The accelerator kernel driver has been installed on the host OS, and the KAE software package has been installed on the VM. For details, see 3.1 Installing the KAE Software.

Configuring Virtualization Settings on the Host OS

Step 1 Query the accelerator installed in the host OS environment and the corresponding BDF number. ls -al /sys/class/uacce

Step 2 Configure accelerator VF settings. For example, virtualize three VFs from the hisi_sec devices each, corresponding to hisi_sec-8 to hisi_sec-13.

echo 3 > /sys/devices/pci0000:74/0000:74:01.0/0000:76:00.0/sriov_numvfs echo 3 > /sys/devices/pci0000:b4/0000:b4:01.0/0000:b6:00.0/sriov_numvfs

----End

Configuring Accelerator Settings on the VM

Step 1 Edit the configuration file of the vm1 VM. virsh edit vm1 Step 2 Add the vCPU configuration to the configuration file. For example, configure four cores for the VM. After the configuration, the VM processes run on the physical CPUs of the specified host. Step 3 Configure a VF for the VM.

After the configuration, a VF virtualized by the accelerator is mounted to the VM.

NO TE

● The hisi_sec with the SBDF number of 0000:7x:xx.x corresponds to the device connected to CPU 0. The value starting with 0000:bx:xx.x corresponds to the device connected to CPU 1. ● To ensure stable performance, you are advised to select the core of the corresponding CPU for the VM and select the VF virtualized by the corresponding accelerator. ● The host OS allows a maximum of 11 VFs to be mounted to a VM. Step 4 Start the VM. virsh start vm1

CA UTION

If the VM fails to be started and the message "Unknown PCI header type '127'" is displayed, unbind the mounted VFs and restart the VM.

echo 0000:76:00.1 > /sys/bus/pci/drivers/hisi_sec/unbind echo vfio-pci > /sys/devices/pci0000:74/0000:74:01.0/0000:76:00.1/driver_override echo 0000:76:00.1 > /sys/bus/pci/drivers_probe

Step 5 Log in to the VM to query the device. ls /sys/class/uacce/

If the following information is displayed, the VF is detected on the VM.

hisi_sec-0

----End 4.1.7 Using the KAE on Docker The KAE can be used on Docker. The accelerator device complies with the PCIe specifications. It is presented as a PCIe device on the OS and supports SR-IOV.

Each accelerator provides 1024 queues. A PF uses 256 queues by default, and the other 768 queues are reserved for VFs.

Number of VF queues = (1024 – Number of PF queues)/Number of VFs

The remainder queue is added to the last VF.

It is recommended that one PF be virtualized into eight VFs.

Environment Requirements ● A Docker container has been created on the host OS. ● The accelerator kernel driver has been installed on the host OS. For details, see 3.1 Installing the KAE Software.

Configuring Virtualization Settings on the Host OS

Step 1 Query the accelerator installed in the host OS environment and the corresponding BDF number. ls -al /sys/class/uacce

Step 2 Configure accelerator VF settings. For example, virtualize three VFs from the hisi_sec devices each, corresponding to hisi_sec-8 to hisi_sec-13.

echo 3 > /sys/devices/pci0000:74/0000:74:01.0/0000:76:00.0/sriov_numvfs echo 3 > /sys/devices/pci0000:b4/0000:b4:01.0/0000:b6:00.0/sriov_numvfs

Step 3 Start the Docker container and allocate an accelerator VF to the container. docker run -it -v /usr/:/usr/ --device=/dev/hisi_sec-8:/dev/hisi_sec-2:rwm -m 8192m --cpuset-cpus="4-7" 90b5058926a2 /bin/bash

NO TE

- I: enables Docker to allocate a pseudo terminal and bind it to the standard input of the container. - t: always enables the standard input of the container. - v: mounts the host machine directory to the image. The directory before the colon (:) is the host machine directory, which must be an absolute path. The directory after the colon (:) is the mount path in the image. --device: specifies the host device used by the container. The value before the colon (:) indicates the VF device created on the host. The value after the colon (:) indicates the directory in the container. r, w, m indicates that the container has the permission to read, write, and create files on the device. -m: specifies the maximum memory used by the container. --cpuset-cpus: specifies the CPU cores on which the container runs. 90b5058926a2: indicates the image ID. You can also use the image name. To query the image name, run the docker images command. /bin/bash: indicates the bash that starts the container.

----End 4.1.8 MD5 Hardware Acceleration Tuning For details, see MD5 Hardware Acceleration Tuning in the Ceph Object Storage Tuning Guide.

5 Common Operations

5.1 Querying KAE Log Information

5.1 Querying KAE Log Information Table 5-1 lists log information related to the KAE.

Table 5-1 Log information Directory File Description

/var/log/ kae.log The default level of OpenSSL engine logs is "error". To set the log level, perform the following steps: 1. Configure environment variables. export KAE_CONF_ENV=/var/log/ 2. Create a kae.cnf file in /var/ log/. 3. In the kae.cnf file, configure the content as follows: [LogSection] debug_level=error ● debug_level=error: The value can be none/error/ info/warning/debug ● In normal cases, you are advised not to enable the info or debug log level. Otherwise, the accelerator performance will deteriorate.

Directory File Description

/var/log/ message/syslog ● Kernel logs of OSs such as CentOS, SUSE, and EulerOS are stored in the /var/log/ message directory. ● Kernel logs of OSs such as Ubuntu are stored in the /var/log/syslog directory. Alternatively, you can run the dmesg > /var/log/dmesg.log command to collect driver and kernel logs.

6 Troubleshooting

6.1 Initialization Failure 6.2 Failed to Identify Accelerator Devices After the KAE Installed 6.3 Failed to Upgrade the Accelerator Driver 6.4 Failed to Identify the Related Interface Symbols After the OpenSSL of a New Version Is Installed

6.1 Initialization Failure

Symptom The KAE is not completely loaded due to an initialization failure.

Handling Procedure

Step 1 Check that the accelerator drivers are loaded successfully. Check whether uacce.ko, qm.ko, sgl.ko, hisi_sec2.ko, hisi_hpre.ko, and hisi_rde.ko are loaded. lsmod | grep uacce uacce 262144 2 hisi_hpre,hisi_qm,hisi_sec2,hisi_rde Step 2 Check that the KAE library exists in the software installation directory (/usr/lib64 for RPM installation and /usr/local/lib for source code installation) and OpenSSL installation directory, and that the correct soft connection is established. 1. Check whether the KAE is correctly installed and whether a soft connection is established. ll /usr/local/lib/engines-1.1/ |grep kae If the installation is correct, the following information is displayed: lrwxrwxrwx. 1 root root 22 Nov 12 02:33 kae.so -> kae.so.1.0.1 lrwxrwxrwx. 1 root root 22 Nov 12 02:33 kae.so.0 -> kae.so.1.0.1 -rwxr-xr-x. 1 root root 112632 May 25 2019 kae.so.1.0.1 2. Check whether the WD is correctly installed and whether the soft connection is established. ll /usr/lib64/ | grep libwd If the installation is correct, the following information is displayed:

lrwxrwxrwx. 1 root root 14 Nov 12 02:33 libwd.so -> libwd.so.1.0.1 lrwxrwxrwx. 1 root root 14 Nov 12 02:33 libwd.so.0 -> libwd.so.1.0.1 -rwxr-xr-x. 1 root root 137120 May 25 2019 libwd.so.1.0.1

Step 3 Check whether the path of the OpenSSL engine library can be exported using the export command. echo $OPENSSL_ENGINES export OPENSSL_ENGINES=/usr/local/lib/engines-1.1 echo $OPENSSL_ENGINES /usr/local/lib/engines-1.1

----End

6.2 Failed to Identify Accelerator Devices After the KAE Installed

Symptom

The accelerator devices cannot be identified after the KAE is installed.

Procedure

Step 1 Check whether the devices exist in the virtual file system. ls -al /sys/class/uacce/

Normally, the following accelerator devices are displayed: total 0 lrwxrwxrwx. 1 root root 0 Nov 14 03:45 hisi_hpre-2 -> ../../devices/pci0000:78/0000:78:00.0/0000:79:00.0/ uacce/hisi_hpre-2 lrwxrwxrwx. 1 root root 0 Nov 14 03:45 hisi_hpre-3 -> ../../devices/pci0000:b8/0000:b8:00.0/0000:b9:00.0/ uacce/hisi_hpre-3 lrwxrwxrwx. 1 root root 0 Nov 17 22:09 hisi_rde-4 -> ../../devices/pci0000:78/0000:78:01.0/uacce/hisi_rde-4 lrwxrwxrwx. 1 root root 0 Nov 17 22:09 hisi_rde-5 -> ../../devices/pci0000:b8/0000:b8:01.0/uacce/hisi_rde-5 lrwxrwxrwx. 1 root root 0 Nov 14 08:39 hisi_sec-0 -> ../../devices/pci0000:74/0000:74:01.0/0000:76:00.0/ uacce/hisi_sec-0 lrwxrwxrwx. 1 root root 0 Nov 14 08:39 hisi_sec-1 -> ../../devices/pci0000:b4/0000:b4:01.0/0000:b6:00.0/ uacce/hisi_sec-1

Step 2 If the hpre device is not found in Step 1, check whether the KAE software is correctly installed by referring to 6.1 Initialization Failure.

Step 3 If the KAE software is correctly installed, run the lspci command to check whether the physical devices exist. 1. Check whether HPRE exists. lspci | grep HPRE The command output is as follows: 79:00.0 Network and computing encryption device: Huawei Technologies Co., Ltd. HiSilicon HPRE Engine (rev 21) b9:00.0 Network and computing encryption device: Huawei Technologies Co., Ltd. HiSilicon HPRE Engine (rev 21) 2. Check whether the SEC exists. lspci | grep SEC The command output is as follows: 76:00.0 Network and computing encryption device: Huawei Technologies Co., Ltd. HiSilicon SEC Engine (rev 21) b6:00.0 Network and computing encryption device: Huawei Technologies Co., Ltd. HiSilicon SEC Engine (rev 21)

3. Check whether the RDE exists. lspci | grep RDE The command output is as follows: 78:01.0 RAID bus controller: Huawei Technologies Co., Ltd. HiSilicon RDE Engine (rev 21) b8:01.0 RAID bus controller: Huawei Technologies Co., Ltd. HiSilicon RDE Engine (rev 21) Step 4 If no physical device is found in Step 3, perform the following operations: ● Check whether the accelerator license has been properly imported. If not, import the accelerator license by referring to "License Management" in TaiShan Rack Server iBMC (V300 to V549) User Guide. After the accelerator license is imported, power off and restart the iBMC to enable the license. ● Check whether the iBMC and BIOS versions support the accelerator features. To support KAE, the BIOS version must be later than 1.05, and the iBMC version must be later than 3.65.

----End

6.3 Failed to Upgrade the Accelerator Driver

Symptom After the accelerator driver is upgraded, the driver version is not changed after the system is restarted.

Possible Cause Before the accelerator driver is upgraded, the system upgrades other driver packages. These driver packages may update the boot file system initramfs, and update the accelerator driver to initramfs before upgrade. For example, if the NIC driver is updated or initramfs is manually updated, the system loads the accelerator driver from initramfs first during restart.

Handling Procedure After the accelerator driver is upgraded, run the dracut --force command to update initramfs again.

6.4 Failed to Identify the Related Interface Symbols After the OpenSSL of a New Version Is Installed

Symptom The following error information is displayed when the rpm command is executed:

rpm: relocation error: /lib64/librpmio.so.8: symbol EVP_md2 version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference The following error information is displayed when the OpenSSL command is executed:

/usr/bin/openssl: relocation error: /usr/bin/openssl: symbol EVP_md2 version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference

Possible Cause When the OpenSSL dynamic library path is exported to LD_LIBRARY_PATH or installed in /usr/local/lib, the dynamic library search path in the /etc/ld.so.conf configuration file is set to /usr/local/lib. When the system tool or command attempts to invoke the OpenSSL dynamic library, the library installed by the user rather than the original system library is invoked.

Handling Procedure Install the OpenSSL software by following the procedure described in 3.1.3 Installing OpenSSL, and install the KAE software packages by following the procedures in 3.1.4 Installing the KAE Software Using an RPM Package, 3.1.5 Installing the KAE Software Using a DEB Package, and 3.1.6 Installing the KAE Software Using Source Code. If the system needs to export the /usr/local/lib path to the LD_LIBRARY_PATH environment variable or configure the path to /etc/ld.so.conf, specify the installation path and dynamic library path to install the OpenSSL source code:

./config --prefix=/usr/local/openssl -Wl,-rpath,/usr/local/openssl/lib make make install ● If the accelerator is installed in RPM mode, change the command as follows: rpm -ivh libkae-1.0.1-1.euler2.0.aarch64.rpm --prefix=/usr/local/openssl/lib/engines-1.1 ● If the accelerator is installed in source code mode, run the following compilation and installation commands in sequence. cd KAE chmod +x configure ./configure --openssl_path=/usr/local/openssl make clean && make make install

A Appendix

A.1 Terms

Table A-1 Terms Term Description

AES The Advanced Encryption Standard (AES), also known by its original name Rijndael, is a specification for the encryption of electronic data adopted by the U.S. government. It supersedes the Data Encryption Standard (DES) and is now used worldwide. AES was announced by the U.S. National Institute of Standards and Technology (NIST) as U.S. FIPS PUB 197 (FIPS 197) on November 26, 2001 after a five-year standardization process, and became effective as a federal government standard on May 26, 2002. In 2006, AES has become one of the most popular algorithms in symmetric key encryption.

SM3 A public algorithm mainly used in digital signatures, message authentication codes, and pseudorandom number generators. According to China State Cryptography Administration, the security and efficiency of SM3 are equivalent to SHA-256. SHA-256 in.

Term Description

SM4 A public algorithm mainly used for data encryption in commercial cryptography systems. Both of its key size and block size are 128 bits. The encryption algorithm and key extension algorithm of SM4 use 32 rounds of nonlinear iteration structure. The S-box is fixed for 8-bit input and 8- bit output. In SM4, the length of instructions has been increased up to 64 K (that is, 64 × 1024), which is 128 times the length supported by SM3 (the allowable length of rendering instructions is greater than 512).

RSA An algorithm named after its developers Ron Rivest, Adi Shamir, and Leonard Adleman. RSA is an asymmetric cryptographic algorithm. It was proposed in 1977 and has developed into one of the best asymmetric cryptographic solutions.

DH Diffie-Hellman (DH) key exchange algorithm is a security protocol. It allows both parties to establish a key through an insecure channel without any prior information from each other. This key can be used as a symmetric key to encrypt subsequent communications.

A.2 Acronyms and Abbreviations

Table A-2 List of acronyms and abbreviations Acronyms and Full Spelling Abbreviations

HPRE High Performance RSA Engine

KAE Kunpeng Accelerator Engine

SEC Security Engine

A.3 Change History

Date Description

2021-08-06 This issue is the fifteenth official release. ● Changed the document name to Kunpeng Accelerator Engine Developer Guide (KAE Encryption & Decryption). ● The content related to KAEzip has been migrated to the Kunpeng Accelerator Engine Developer Guide (KAEzip).

2021-07-12 This issue is the fourteenth official release. Added CAUTION in 4.1.3 Using KAE Through the OpenSSL Configuration File openssl.cnf.

2021-06-15 This issue is the thirteenth official release. Added the definition of the test data length in 3.2.1 Testing the Performance of OpenSSL and the KAE.

2021-06-01 This issue is the twelfth official release. Added the setting of OpenSSL environment variables in 3.1.4 Installing the KAE Software Using an RPM Package, 3.1.5 Installing the KAE Software Using a DEB Package, and 3.1.6 Installing the KAE Software Using Source Code.

2021-05-27 This issue is the eleventh official release. Added the method of installing KAE using the RPM package and source code in openEuler 20.03 LTS.

2021-05-14 This issue is the tenth official release. Added the paths for downloading the KAE installation package in 3.1.4 Installing the KAE Software Using an RPM Package, 3.1.5 Installing the KAE Software Using a DEB Package, and 3.1.6 Installing the KAE Software Using Source Code.

2021-01-26 This issue is the ninth official release. Modified the method of obtaining the RPM and DEB packages in Table 3-1.

2021-01-18 This issue is the eighth official release. Changed the document name to Kunpeng Accelerator Engine Developer Guide.

2020-12-14 This issue is the seventh official release. Modified 3.1.6 Installing the KAE Software Using Source Code.

Date Description

2020-11-05 This issue is the sixth official release. ● Modified 1 Overview. ● Modified 4.1.4 Nginx Applications Accelerated by KAE. ● Added 4.1.8 MD5 Hardware Acceleration Tuning. ● Modified the section about the zlib acceleration library.

2020-10-15 This issue is the fifth official release. ● Changed "EBC" to "ECB" in 1 Overview. ● Added the address for downloading the OpenSSL source code in 3.1.3 Installing OpenSSL.

2020-07-22 This issue is the fourth official release. Changed the document name.

2020-06-30 This issue is the third official release. ● Added the description of installation in dpkg mode in 3.1.1 Introduction. ● Modified the environment requirements and software package description in 3.1.2 Preparing for Installation. ● Added the procedure using an OpenSSL command to verify whether the installation has taken effect in 3.1.5 Installing the KAE Software Using a DEB Package. ● Added the upgrade mode using DEB packages in 3.3 Upgrading the KAE Software. ● Added the uninstallation method using DEB packages in 3.4 Uninstalling the KAE Software. ● Added the description of MD5 and SM3/SM4 to 4.1.1 KAE Usage Description. ● Modified the description of configuration file parameters in 4.1.3 Using KAE Through the OpenSSL Configuration File openssl.cnf. ● Modified the description of the link modes in the section about the zlib acceleration library. ● Added the recommended number of VFs virtualized from a PF in 4.1.6 Using the KAE on a KVM and 4.1.7 Using the KAE on Docker.

2020-04-30 This issue is the second official release. Optimized the document.

2020-03-20 This issue is the first official release.