Linux on IBM Z

Pervasive Encryption with Linux on IBM Z: from a performance perspective

Danijel Soldo Software Performance Analyst Linux on IBM Z Performance Evaluation _ danijel.soldo@de.ibm.com

• © 2018 International Business Machines Corporation. No part of • Performance data contained herein was generally obtained in a this document may be reproduced or transmitted in any form controlled, isolated environments. Customer examples are without written permission from IBM. presented as illustrations of how those • U.S. Government Users Restricted Rights — use, duplication • customers have used IBM products and the results they may have or disclosure restricted by GSA ADP Schedule Contract with achieved. Actual performance, cost, savings or other results in IBM. other operating environments may vary. • Information in these presentations (including information relating • References in this document to IBM products, programs, or to products that have not yet been announced by IBM) has been services does not imply that IBM intends to make such products, reviewed for accuracy as of the date of initial publication programs or services available in all countries in which and could include unintentional technical or typographical IBM operates or does business. errors. IBM shall have no responsibility to update this information. This document is distributed “as is” without any warranty, • Workshops, sessions and associated materials may have been either express or implied. In no event, shall IBM be liable for prepared by independent session speakers, and do not necessarily any damage arising from the use of this information, reflect the views of IBM. All materials and discussions are provided including but not limited to, loss of data, business for informational purposes only, and are neither intended to, nor interruption, loss of profit or loss of opportunity. shall constitute legal or other guidance or advice to any individual IBM products and services are warranted per the terms and participant or their specific situation. conditions of the agreements under which they are provided. • It is the customer’s responsibility to insure its own compliance • IBM products are manufactured from new parts or new and used with legal requirements and to obtain advice of competent legal parts. counsel as to the identification and interpretation of any In some cases, a product may not be new and may have been relevant laws and regulatory requirements that may affect the previously installed. Regardless, our warranty terms apply.” customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice • Any statements regarding IBM's future direction, intent or or represent or warrant that its services or products will ensure that product plans are subject to change or withdrawal without the customer follows any law. notice.

• Information concerning non-IBM products was obtained from the suppliers of • IBM, the IBM logo, ibm.com and [names of other referenced those products, their published announcements or other publicly available IBM products and services used in the presentation] are sources. IBM has not tested those products about this publication and trademarks of International Business Machines Corporation, cannot confirm the accuracy of performance, compatibility or any other registered in many jurisdictions worldwide. Other product and claims related to non-IBM products. Questions on the capabilities of non-IBM service names might be trademarks of IBM or other products should be addressed to the suppliers of those products. IBM does companies. A current list of IBM trademarks is available on not warrant the quality of any third-party products, or the ability of any such the Web at "Copyright and trademark information" at: third-party products to interoperate with IBM’s products. IBM expressly www.ibm.com/legal/copytrade.shtml. disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a purpose. • Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. • The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, • Other product and service names might be trademarks of IBM trademarks or other intellectual property right. or other companies.

Sources: https://www.wired.com/story/ibm-z-mainframe-encryption/ https://techcrunch.com/2017/07/16/ibm-dangles-carrot-of-full-encryption-to-lure-buyers-to- new-z14-mainframe/ https://www.cnbc.com/2017/07/17/ibm-unveils-new-mainframe-capable-of-running-more- 4 IBM Z / Danijel Soldo – Pervasive Encryption with Linux on IBM Z: from a performance perspective / © 2018 IBM Corporation than-12-billion-encrypted-transactions-a-day.html Agenda

Pervasive Encryption in Linux Data-in-flight encryption performance Data-at-rest encryption performance End-to-End scenario

Encrypt everything.

- What is the idea?

l Encrypt everything. Shift from selective to pervasive encryption without violating SLAs.

- Why has it become a hot topic recently?

l GDPR, PCI-DSS and other regulations force enterprises to provide data protection.

l Recent improvements introduced with the z14 made it possible. Great performance improvements in critical crypto workloads.

- But how is it implemented with Linux on IBM Z?

Network security or data-in-flight encryption

• TLS, IPSec

• Measured via symmetric encryption throughput and asymmetric handshakes volume

• AES modes measured: CBC, GCM, XTS

Network security or data-in-flight encryption

• TLS, IPSec

• Measured via symmetric encryption throughput and asymmetric handshakes volume

• AES modes measured: CBC, GCM, XTS

Full volume encryption or data-at-rest encryption

• dm-crypt / LUKS

• Measured via disk IO stress tool throughput and CPU consumption

• Different sector sizes and encryption keys compared

Network security or data-in-flight encryption

• TLS, IPSec

• Measured via symmetric encryption throughput and asymmetric handshakes volume

• AES modes measured: CBC, GCM, XTS

Full volume encryption or data-at-rest encryption

• dm-crypt / LUKS

• Measured via disk IO stress tool throughput and CPU consumption

• Different sector sizes and encryption keys compared

Secure Service Container

• Simple and highly secure environment to run an appliance

• Secure boot, encryption and reduced attack surface

TLS/SSL

TLS Handshake Data buffer encryption (asymmetric) (symmetric)

TLS/SSL

TLS Handshake Data buffer encryption (asymmetric) (symmetric)

RSA, DH, ECDHE, ECDSA AES CEX exploitation CPACF exploitation OpenSSL s_time OpenSSL speed

OpenSSL s_time: establishing a max number of TLS/SSL connections with clients in a given time slot. OpenSSL speed: encrypting/decrypting a max possible amount of data in a given time slot.

LPAR z13/z14 Intel x86 Server Client

TLS/SSL

Client – x86 system OpenSSL cipher suites Server – z13/z14 LPAR

Intel(R) Xeon(R) CPU E5-2690 TLS_RSA_WITH_AES_256_CBC_SHA256 4 IFLs + SMT-2

16 cores (2 thread per core) TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 2 GB Memory

2 GB Memory TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 CEX5S/CEX6S Accelerator

OpenSSL 1.0.2h TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 OpenSSL 1.0.2j

Connections per second: 2kB data and 2048 bit RSA Key

TLS_RSA_WITH_AES_256_CBC_SHA256 Crypto Express Card Acceleration l Key Exchange: RSA

l Encryption: AES-256-CBC

l Authentication: RSA

l MAC: SHA256

OpenSSL s_time connection

A connection is defined as: better

the establishment of a TLS session between the client and server, the exchange of fixed size random data and a session disconnect. *All values are normalized to the z13-1client case (100 conn/s)

Connections per second: 2kB data and 2048 bit RSA Key

TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 Crypto Express Card Acceleration l Key Exchange: DH

l Encryption: AES-256-CBC

l Authentication: RSA

l MAC: SHA256

OpenSSL s_time connection

A connection is defined as: better

the establishment of a TLS session between the client and server, the exchange of fixed size random data and a session disconnect. *All values are normalized to the z13-1client case (100 conn/s)

Connections per second: 2kB data and 2048 bit RSA Key

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

l Key Exchange: ECDH

l Encryption: AES-128-GCM

l Authentication: RSA

l MAC: AEAD

OpenSSL s_time connection

A connection is defined as: better

the establishment of a TLS session between the client and server, the exchange of fixed size random data and a session disconnect. *All values are normalized to the z13-1client case (100 conn/s)

Connections per second: 2kB data and 256 bit EC Key

TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

l Key Exchange: ECDH

l Encryption: AES-128-GCM

l Authentication: ECDSA

l MAC: AEAD

OpenSSL s_time connection

A connection is defined as: better

the establishment of a TLS session between the client and server, the exchange of fixed size random data and a session disconnect. *All values are normalized to the z13-1client case (100 conn/s)

Scope of the measurements

Cipher modes OpenSSL Versions AES-256-CBC Buffer sizes AES-256-GCM 1.0.2j AES-256-XTS* 1.1.1x 512B 4096B

* AES-XTS is a recommended data-at-rest encryption mode.

Encryption Throughput 512B buffer size OpenSSL 1.0.2j

2.2x

3.7x

2.8x

better

*All values are normalized to the GCM-256-z13 case IBM Z / Danijel Soldo – Pervasive Encryption with Linux on IBM Z: from a performance perspective / © 2018 IBM Corporation 18 Data-in-Flight: Data buffer encryption

Encryption Throughput 512B buffer size OpenSSL 1.1.1

2.1x

3.8x

5.4x

better OpenSSL 1.1.1 exploiting new CPACF instructions for GCM *All values are normalized to the GCM-256-z13 case IBM Z / Danijel Soldo – Pervasive Encryption with Linux on IBM Z: from a performance perspective / © 2018 IBM Corporation 19 Data-in-Flight: Data buffer encryption

Encryption Throughput 4096B buffer size OpenSSL 1.0.2j

2.3x

8.4x

3.7x

better

*All values are normalized to the GCM-256-z13 case IBM Z / Danijel Soldo – Pervasive Encryption with Linux on IBM Z: from a performance perspective / © 2018 IBM Corporation 20 Data-in-Flight: Data buffer encryption

Encryption Throughput 4096B buffer size OpenSSL 1.1.1

2.3x

8.4x

12.5x

Decryption Throughput 4096B buffer size OpenSSL 1.1.1

8.9x

13.7x

better

With the AES-CBC mode, only decryption benefits from parallel execution.

Asymmetric encryption 2x more RSA/DH engines on the Crypto Express 6S adapter

Symmetric encryption CPACF improvements Huge throughput improvement: up to 12.5x Focus on AES-GCM and AES-XTS

dm-crypt: application I/O system call direct I/O (open, read, § is a mechanism for end-to-end data (bypassing virtual file system write) page cache) encryption. Data only appears in the clear file system (e.g. ext4) when in application. page cache

§ is a kernel component that transparently: text clear direct I/O to standard I/O device (through page cache) – encrypts all data written to disk (e.g. swap) logical block DD – decrypts all data read from disk § uses in-kernel crypto dm-crypt layers of logical – aes_s390 module has to be loaded logical block DD device drivers: – CPACF support logical volumes, physical physical RAID, multipath block DD block DD kernel + dm-crypt encrypted disk disk

§ Clear key performance - Stored in plain text in OS memory § Secure key clear key - Generated and wrapped using a Master Key from a HSM (CEX Adapter) - Every encrypt/decrypt call is executed on the HSM

secure key

security

§ Clear key performance - Stored in plain text in OS memory § Secure key clear key - Generated and wrapped using a Master Key from a HSM (CEX protected key Adapter) - Every encrypt/decrypt call is executed on the HSM § Protected keys – Industry unique! - Stored in cipher text in OS memory secure key - Generated out of a secure key via HSM - Wrapped by a system key accessible to CPU only - Functionally similar to secure keys but much faster (CPACF)

security

• New modules added: pkey & paes_s390

• To generate a secure key: zkey tool (s390 tools) & CEX CCA mode • Supported in cryptsetup Plain and LUKS format ( v2.0.3) What do I need?

Workload – Flexible IO Tester

Direct IO

Synchronous, single thread

LPAR Sequential Read z13/z14

Linux 4.14 Kernel FIO Workload

No Encryption

Clear Key AES-256-XTS

Protected key PAES-256-XTS Server – z13/z14 LPAR Storage Server – DS8886

2 IFLs + SMT-2 512 B 4096 B IBM DS8886 Model 985 16 GB Memory ECKD/DASD Disks Default dev- Since kernel mapper sector 4.12 and size cryptsetup v2.0

27 Data-at-Rest: Results Throughput 120

100 -19.99% -27.08%

dmcrypt volumes -31.05% -35.62% using default 80 512B sector size 60 Throughput Encryption decreases throughput: 40 - 20% clear key - 27% protected key 20 better

0 NoEncryption ClearKey ProtKey NoEncryption ClearKey ProtKey

z13 z14

* Throughput values normalized to the z13 NoEncryption case IBM Z / Danijel Soldo – Pervasive Encryption with Linux on IBM Z: from a performance perspective / © 2018 IBM Corporation 28 Data-at-Rest: Results Throughput 120 dmcrypt volumes using 4096B -8.09% -9.57% sector size 100

-19.63% -20.64% 80 Encryption decreases throughput: - 8% clear key 60 - 9.5% protected key Throughput 40 Sector size 4096B requirements: Kernel > 4.12 20

cryptsetup > 2.0.0 better

0 Available since: NoEncryption ClearKey ProtKey NoEncryption ClearKey ProtKey Ubuntu 18.04 SLES 15 RHEL 7.5 (kernel-alt) z13 z14

dmcrypt volumes 7 using default 512B sector size 6 -27.08% 5

Delta introduced via Avg per CPU [%] encryption: 3 + 4.6% clear key

+ 6.3% protected key 2 better

0 NoEncryption ClearKey ProtKey NoEncryption ClearKey ProtKey

z13 z14

IBM Z / Danijel Soldo – Pervasive Encryption with Linux on IBM Z: from a performance perspective / © 2018 IBM Corporation 30 Data-at-Rest: Results CPU Utilization 10 dmcrypt volumes using 4096B 9 sector size 8

7 Delta introduced via encryption: 6 + 1.7% clear key + 1.9% protected key 5

4 Avg per CPU [%] 3 Sector size 4096B requirements: 2 better Kernel > 4.12 cryptsetup > 2.0.0 1

0 Available since: NoEncryption ClearKey ProtKey NoEncryption ClearKey ProtKey Ubuntu 18.04 SLES 15 RHEL 7.5(kernel-alt) z13 z14

IBM Z / Danijel Soldo – Pervasive Encryption with Linux on IBM Z: from a performance perspective / © 2018 IBM Corporation 31 Data-at-Rest: Results CPU Utilization 10 z14 AES-256-XTS OpenSSL performance dmcrypt volumes using 4096B 9 sector size 8

7 Delta introduced via encryption: 6 + 1.7% clear key + 1.9% protected key 5

4 Avg per CPU [%] 3 Sector size 4096B requirements: 2 better Kernel > 4.12 cryptsetup > 2.0.0 1

0 Available since: NoEncryption ClearKey ProtKey NoEncryption ClearKey ProtKey Ubuntu 18.04 SLES 15 RHEL 7.5(kernel-alt) z13 z14

-35.62% z13 512B sector

+8.2% z13 512B sector

IBM Z / Danijel Soldo – Pervasive Encryption with Linux on IBM Z: from a performance perspective / © 2018 IBM Corporation 33 Data-at-Rest: Summary - A one year journey Relative Throughput 3.5x overhead reduction

-35.62% -9.57% FFWD 1 year z13 z14 512B sector 4kB sector

CPU Utilization 4x overhead reduction

+8.2% +1.9% FFWD 1 year z13 z14 512B sector 4kB sector

FIO Workload • Good to experiment with but no realistic workload

• Low CPU intensive

• Most of the CPU workload is crypto-related 80% AES-XTS • Changing a parameter in the dm-crypt configuration has a huge impact

• ( 512B to 4096B sector size) 15% Kernel 5% 5% dm-crypt

Avg CPU Utilization Workload distribution

FIO Workload HammerDB Workload

80% AES-XTS 80% PostgreSQL 95%

15% Kernel 15% Kernel 5% 5% dm-crypt 5% AES-XTS

Avg CPU Utilization Workload distribution Avg CPU Utilization Workload distribution

LPAR z14 No Encryption No Encryption

Clear Key AES-256-XTS TLS - ECDHE-RSA-AES256 Intel x86 Protected key PAES-256-XTS Load generator

Client – x86 system Server – z14 LPAR Storage Server – DS8886 2 IFLs + SMT-2 Intel(R) Xeon(R) CPU E5-2690 IBM DS8886 Model 985 8 GB Memory ECKD/DASD Disks 16 cores (2 thread per core) Ubuntu 18.04 2 GB Memory PostgreSQL 10 PostgreSQL tuning tips by Marc Beyerle: „Open source on IBM Z: Experiences with PostgreSQL on Linux on IBM Z“ IBM Z / Danijel Soldo – Pervasive Encryption with Linux on IBM Z: from a performance perspective / © 2018 IBM Corporation 37 End to End scenario: Results

Throughput 300 No-encryption dmcrypt-512B dmcrypt-4k TPC-C database 96 Warehouses (19GB) 250

200 No SSL/TLS Clear key dmcrypt: AES-256-XTS 150

100

50 better

0 2 5 25 50 90 Virt. Users * Throughput values normalized to the 2 Usr-NoEncryption case

Encryption impact on throughput

dmcrypt-512B dmcrypt-4096B 18 16,47 16 14 12,52 12 10,69 9,67 9,72 9,73 10 8,88 % 8 better 6,02 6 3,94 4 3,46 2 0 2 5 25 50 90 Virt. Users

CPU 95% Utilization

Encryption impact on throughput

25 User case: Protected key + TLS dmcrypt-512B dmcrypt-4096B 18 16,47 12 16 14 12,52 10 12 10,69 9,67 9,72 9,73 8 10 8,88 % 8 better 6 6,02 6 3,94 4 4 3,46 2 2 0 0 2 5 25 50 90 Clear-512B Clear-4k Prot-4k Prot-4k-TLS Virt. Users

* TLS cipher suite: ECDHE-RSA-AES256 CPU 95% Utilization

12.5x Up to 12.5x higher symmetric encryption throughput due to CPACF improvements Data-in-Flight 2x More RSA handshaking volume 3.5x Throughput overhead reduction Data-at-Rest CPU Utilization overhead reduction 4x

Running a PostgreSQL DB encrypting data end-to-end with TLS and CPACF End-to-End 6% protected keys under 6% throughput cost

41 Pervasive Encryption with Linux on IBM Z: Summary – z14 vs z13

Running a PostgreSQL DB encrypting data end-to-end with TLS and CPACF End-to-End 6% protected keys under 6% throughput cost

42 Acknowledgements

Dr.Reinhard Bündgen Marc Beyerle

Linux on System z – Tuning hints and tips http://www.ibm.com/developerworks/linux/linux390/perf/index.html Live Virtual Classes for z/VM and Linux http://www.vm.ibm.com/education/lvc/ Mainframe Linux blog http://linuxmain.blogspot.com

Danijel Soldo Software Performance Analyst Linux on IBM Z Performance Evaluation _ [email protected]