Ken Cantrell, Netapp Mark Rogov, Dell EMC

Ken Cantrell, NetApp Mark Rogov, Dell EMC

October 20, 2016 SNIA Legal Notice • The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. • Member companies and individual members may use this material in presentations and literature under the following conditions: • Any slide or slides used must be reproduced in their entirety without modification

• The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. • This presentation is a project of the SNIA Education Committee. • Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. • The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information.

NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. About SNIA

About The Speakers

Ken Cantrell Mark Rogov Alex McDonald NetApp Dell EMC NetApp Manager Perf Engineering Advisory Systems Engineer Vice Chair of the SNIA ESF @kencantrelljr @rogovmark @alextangent

Storage Performance Benchmarking

JULY 30, OCT 20, MARCH 8, ONE LEFT! 2015 2015 2016 TODAY

METRICS AND SOLUTION BLOCK FILE WORKLOAD TERMINOLOGY UNDER TEST COMPONENTS COMPONENTS DEFINITIONS

INTRO FS FS OPS PERF FUN END Session 1 – Terminology and Context

OPS IOPS COUNT EVERY COUNT EVERY PROTOCOL IO OPERATION OPERATION PER SECOND PER SECOND CONTEXT MAKES METRICS MATTER RESPONSE MB/S TIME PAYLOAD SUM TIME TARGET OF EVERY TAKES TO REPLY OPERATION TO AN IO PER SECOND System 1 System 2

TERMINOLOGY AND BASIC GRAPH FUN CONCEPTS

INTRO FS FS OPS PERF FUN END Session 2 – The Slowest Component Matters Most

DISK BOUND DO LESS CLIENT WORK BOUND

DO WORK INCREASE FASTER PARALLELISM

SLOW COMPONENT BOTTLENECKS 3 PERFORMANCE MATTERS MOST ALWAYS EXIST PRINCIPLES

INTRO FS FS OPS PERF FUN END Session 3 – Block Is The Foundation

RANDOM VS. SEQUENTIAL FLASH VS DISK DRIVE RAID PERFORMANCE WORKLOADS IMPLICATIONS

INTRO FS FS OPS PERF FUN END Today: File & Filesystem Performance

STORAGE FRONT-END CLIENTS / CONTROLLER CONNECT HOSTS

ILESYSTEM ILESYSTEM F F

NAS LOCAL

INTRO FS FS OPS PERF FUN END Agenda

INTRO INTRODUCTION

FILESYSTEM FILESYSTEM LAYOUT FUNDAMENTALS

FS OPS WHAT ARE FILESYSTEM OPERATIONS?

FILE PERF PERFORMANCE IN A FILE ORIENTED WORLD

FUN GRAPH FUN WITH PERFORMANCE

END SUMMARY What Is A Filesystem? Term “Filesystem” was coined in 1964

etc dev home usr var

ken mark log

dir1 fileA

fileB

INTRO FS FS OPS PERF FUN END Filesystem Metadata

/ METADATA

etc dev home usr var NAME

PERMISSIONS ken mark log ACCESS TIMES dir1 fileA

DATA BLOCK POINTERS fileB DATA BLOCK POINTERS DATA BLOCK POINTERS

INTRO FS FS OPS PERF FUN END Filesystem Basic Metadata Structure

/home/ken/fileA

NAME: / TYPE: DIR SIZE: 2KIB OWNER: ROOT MODE: 0700 …A FEW MORE… FILE & DIR RECORDS

PHYSICAL AREA

INTRO FS FS OPS PERF FUN END Filesystem Basic Metadata Structure

/home/ken/fileA

NAME: / NAME: home TYPE: DIR TYPE: DIR SIZE: 2KIB SIZE: 2KIB OWNER: ROOT OWNER: ROOT MODE: 0700 MODE: 0755 …A FEW MORE… …A FEW MORE… FILE & DIR FILE & DIR RECORDS RECORDS

PHYSICAL AREA

INTRO FS FS OPS PERF FUN END Filesystem Basic Metadata Structure

/home/ken/fileA

NAME: / NAME: home NAME: ken NAME: fileA TYPE: DIR TYPE: DIR TYPE: DIR TYPE: FILE SIZE: 2KIB SIZE: 2KIB SIZE: 2KIB SIZE: 20KIB OWNER: ROOT OWNER: ROOT OWNER: KEN OWNER: KEN MODE: 0700 MODE: 0755 MODE: 0755 MODE: 0755 …A FEW MORE… …A FEW MORE… …A FEW MORE… …A FEW MORE… FILE & DIR FILE & DIR FILE & DIR DATA BLOCK RECORDS RECORDS RECORDS POINTERS

1 3

PHYSICAL AREA 2

INTRO FS FS OPS PERF FUN END Filesystem Metadata

/ METADATA

etc dev home usr var NAME

PERMISSIONS ken mark log ACCESS TIMES dir1 fileA

DATA BLOCK POINTERS fileB DATA BLOCK POINTERS DATA BLOCK POINTERS

INTRO FS FS OPS PERF FUN END Filesystem Metadata

/ METADATA

etc dev FILESYSTEMShome usr var NAME PERMISSIONS ken mark PROVIDElog ACCESS TIMES dir1 fileASTRUCTURE DATA BLOCK POINTERS fileB DATA BLOCK POINTERS DATA BLOCK POINTERS

INTRO FS FS OPS PERF FUN END Agenda

INTRO INTRODUCTION

FILESYSTEM FILESYSTEM LAYOUT FUNDAMENTALS

FS OPS WHAT ARE FILESYSTEM OPERATIONS?

FILE PERF PERFORMANCE IN A FILE ORIENTED WORLD

FUN GRAPH FUN WITH PERFORMANCE

END SUMMARY Local File Operations User Action Converted Into Filesystem Operations

$ echo “test” > data.txt

STAT OPEN WRITE CLOSE

INTRO FS FS OPS PERF FUN END Local File Operations User Action Converted Into Filesystem Operations

$ echo “test” > data.txt

STAT OPEN WRITE CLOSE

$ cat data.txt

STAT OPEN READ CLOSE

INTRO FS FS OPS PERF FUN END Local File Operations User Action Converted Into Filesystem Operations

$ echo “test” > data.txt

STAT OPEN WRITE CLOSE

$ cat data.txt

STAT OPEN READ CLOSE

$ rm data.txt

STAT UNLINK

INTRO FS FS OPS PERF FUN END Local File Operations Are There Only Six File Operations?

$ echo “test” > data.txt READ

STAT OPEN WRITE CLOSE WRITE

OPEN $ cat data.txt STAT STAT OPEN READ CLOSE UNLINK

CLOSE $ rm data.txt …MORE… STAT UNLINK

INTRO FS FS OPS PERF FUN END Local File Operations Relationship Between Filesystem Operations And Block IO Operations

$ echo “test” > data.txt READ

STAT OPEN WRITE CLOSE WRITE BLOCK IO READ OPEN $ cat data.txt STAT BLOCK IO STAT OPEN READ CLOSE WRITE UNLINK

CLOSE $ rm data.txt

STAT UNLINK

INTRO FS FS OPS PERF FUN END NAS File Operations Create A New File Containing Just Three Spaces

SMB2 Operations 120 104 100

20 6 2 0

INTRO FS FS OPS PERF FUN END NAS File Operations Create A New File Containing Just Three Spaces

SMB2 Operations 120 104 100

20 6 2 0 PowerShell CMD.exe Notepad.exe

INTRO FS FS OPS PERF FUN END In The File World…

THERE ARE MORE OPS THAN JUST READS AND WRITES

INTRO FS FS OPS PERF FUN END Agenda

INTRO INTRODUCTION

FILESYSTEM FILESYSTEM LAYOUT FUNDAMENTALS

FS OPS WHAT ARE FILESYSTEM OPERATIONS?

FILE PERF PERFORMANCE IN A FILE ORIENTED WORLD

FUN GRAPH FUN WITH PERFORMANCE

END SUMMARY Performance In a File Oriented World is Affected By …

Performance Data vs Metadata Caching Differences Ratios Between Operations

INTRO FS FS OPS PERF FUN END Performance Characteristics Of File Operations

Time to complete for all files: create (then write) vs read vs delete/unlink 3000 create / write 2500 read 2000

1500

1000

Elapsed Time (seconds) Time Elapsed unlink / delete 500 ls 0 20,000 40,000 80,000 60,000 100,000 120,000 140,000 160,000 180,000 200,000 220,000 240,000 260,000 280,000 300,000 320,000 340,000 360,000 380,000 400,000 420,000 440,000 460,000 480,000 500,000 520,000 540,000 560,000

# of Files in a Single Directory

INTRO FS FS OPS PERF FUN END Performance Characteristics Of File Operations

Time to complete for all files: create (and write) vs read vs delete/unlink 3000 create / write 2500 read 2000

1500

1000

# of Files in a Single Directory

INTRO FS FS OPS PERF FUN END Performance Characteristics Of File Operations

Time to complete for all files: create (then write) vs read vs delete/unlink 3000 create / write 2500 read 2000

1500

1000

# of Files in a Single Directory

INTRO FS FS OPS PERF FUN END Performance Characteristics Of File Operations

Time to complete for all files: create (then write) vs read vs delete/unlink 3000 create / write 2500 read 2000

1500

1000

# of Files in a Single Directory

INTRO FS FS OPS PERF FUN END Performance In a File Oriented World Be Careful … Operation Performance Will Almost Always Be Affected By the SUT

Seconds to List Directory Contents Otherwise idle system, single threaded, single directory 6 SUT-1 SUT-2 5 SUT-3

2 Seconds to Complete to Seconds 1

0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 180,000 200,000 220,000 240,000 260,000 280,000 300,000 320,000 340,000 360,000 380,000 400,000 420,000 440,000 460,000 480,000 500,000 520,000 540,000 560,000

Number of files in the same directory

INTRO FS FS OPS PERF FUN END Performance In a File Oriented World is Affected By …

Performance Data vs Metadata Caching Differences Ratios Between Operations

INTRO FS FS OPS PERF FUN END Performance In a File Oriented World Does The Number Of Files Make A Difference?

CREATE 16,005 SEC 81X 197 SEC MD 8KiB MD READ

MD 1,707 SEC 100 SEC 8KiB 17X

MD 8KiB UNLINK 729 SEC 0.1647 SEC 4.1GiB 4,426X

MD 8KiB

MD LS 8KiB 12 SEC 0.0007 SEC 17,142X

MD 8KiB

STAT 1 SEC 0.0002 SEC 5,000X

INTRO FS FS OPS PERF FUN END Performance In a File Oriented World is Affected By …

Performance Data vs Metadata Caching Differences Ratios Between Operations

INTRO FS FS OPS PERF FUN END Performance in a File Oriented World Client Side Caching and Operation Ordering Forcing the file system to invalidate the cache between operations significantly affects performance 25

20 uncached ‘ls’

10 Seconds to Complete to Seconds

5 cached ‘ls’

0 20,000 40,000 60,000 80,000 200,000 220,000 240,000 260,000 280,000 300,000 320,000 340,000 360,000 380,000 400,000 420,000 440,000 460,000 480,000 500,000 520,000 540,000 100,000 120,000 140,000 160,000 180,000 Files In Directory “Uncached ‘ls’” had a USB unmount just prior to the ‘ls’ command execution

INTRO FS FS OPS PERF FUN END Performance In a File Oriented World is Affected By …

Performance Data vs Metadata Caching Differences Ratios Between Operations

INTRO FS FS OPS PERF FUN END Agenda

INTRO INTRODUCTION

FILESYSTEM FILESYSTEM LAYOUT FUNDAMENTALS

FS OPS WHAT ARE FILESYSTEM OPERATIONS?

FILE PERF PERFORMANCE IN A FILE ORIENTED WORLD

FUN GRAPH FUN WITH PERFORMANCE

END SUMMARY Which Is Better?

Result 1 Result 2

INTRO FS FS OPS PERF FUN END Which Is Better?

4,500

4,000

3,500

3,000

2,500

2,000 Total Ops/s Ops/s Total 1,500

1,000

500

0 1 2 3 4 5 6 7 8 9 Iterations of the test, increasing requested load at each point

Result 1 Result 2

INTRO FS FS OPS PERF FUN END Which Is Better?

20 Average Latency (ms) Average

0 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 Total Ops/s

Result 1 Result 2

INTRO FS FS OPS PERF FUN END Which Is Better?

10 Average Latency (ms) Average

0 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 Total Ops/s

Result 1 Result 2

INTRO FS FS OPS PERF FUN END Which Is Better?

25 READS TOTAL 20 READS WRITES

15 WRITES

10 Average Latency (ms) Average TOTAL 5

0 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 Ops/s

Result 1 - Total Ops/s Result 1 - Read Ops/s Result 1 - Write Ops/s Result 2 - Total Ops/s Result 2 - Read Ops/s Result 2- Write Ops/s

INTRO FS FS OPS PERF FUN END Which Is Better?

WRITES + READS = TOTAL 25 READS TOTAL 20

WRITES 15

10 Average Latency (ms) Average

0 0 500 1,000 1,500 2,000 2,500 Ops/s

Result 2 - Total Ops/s Result 2 - Read Ops/s Result 2- Write Ops/s

INTRO FS FS OPS PERF FUN END Which Is Better?

18 READS 16

8 WRITES 6 Average Latency (ms) Average TOTAL 4

0 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 Ops/s

Result 1 - Total Ops/s Result 1 - Read Ops/s Result 1 - Write Ops/s

INTRO FS FS OPS PERF FUN END Which Is Better?

20 READS WRITES 18 + READS + STAT + OPEN/CLOSE FILE 16 + RMDIR/CREATEDIR + CREATE/DELETE FILES 14 + ACCESS + CHMOD + READDIR

12 = TOTAL

8 WRITES 6 Average Latency (ms) Average STAT CALLS TOTAL 4

0 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 Ops/s

Result 1 - Total Ops/s Result 1 - Read Ops/s Result 1 - Write Ops/s Result 1 - Stat Ops/s

INTRO FS FS OPS PERF FUN END Which Is Better?

25 DATABASE WORKLOAD 20

10 SOFTWARE BUILD WORKLOAD Average Latency (ms) Average

0 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 Total Ops/s

INTRO FS FS OPS PERF FUN END Which Is Better?

25 DATABASE WORKLOAD 20 THE ONLY 15 WORKLOAD 10 SOFTWARE BUILD WORKLOAD Average Latency (ms) Average THAT MATTERS 5

0 IS YOURS 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 Total Ops/s

INTRO FS FS OPS PERF FUN END Agenda

INTRO INTRODUCTION

FILESYSTEM FILESYSTEM LAYOUT FUNDAMENTALS

FS OPS WHAT ARE FILESYSTEM OPERATIONS?

FILE PERF PERFORMANCE IN A FILE ORIENTED WORLD

FUN GRAPH FUN WITH PERFORMANCE

END SUMMARY Storage Performance Benchmarking

JULY 30, OCT 20, MARCH 8, TODAY ONE LEFT! 2015 2015 2016

METRICS AND SOLUTION BLOCK FILE WORKLOAD TERMINOLOGY UNDER TEST COMPONENTS COMPONENTS DEFINITIONS

INTRO FS FS OPS PERF FUN END After This Webcast

A PDF and a PPT of the slides for this and all previous parts of this Webcast series will be posted to the SNIA Ethernet Storage Forum (ESF) website and available on-demand

PPT and PDF: http://www.snia.org/forums/esf/knowledge/webcasts

Presentation Recording: https://www.brighttalk.com/webcast/663/219127

A full Q&A from this webcast, including answers to questions we couldn't get to today, will be posted to the SNIA-ESF blog

http://sniaesfblog.org/

Final Webcast – First Half of 2017

“Storage Performance Benchmarking: Part 5”

Client Caching Affects Operation Mixes

Example SPEC SFS 2014 SWBUILD Operation Mix Across Multiple Load Points As Observed From the NAS System 100% write 90% setattr 80% rmdir

70% remove

60% Lookup calls increase from 4% to 42% readdir+

50% read

40% mkdir

lookup 30% Getattr calls decrease from 65% to 14% % of Overall Operation Mix OverallOperation %of getattr 20% create 10% access 0% INCREASING # OF FILES, ACTIVE DATASET AND WORKLOAD INTENSITY

INTRO FS FS OPS PERF FUN END Operation Mixes For Different Applications SPEC SFS® 2014 (proposed 2014+) Application-Level Operation Mixes

100% chmod 90% stat

80% access create 70% unlink readdir 60% Metadata IO rmdir 50% mkdir read/modify/write 40% random write 30% write_file write 20%

% of Overall Requested Operation Mix OverallRequested Operation %of random read Data IO 10% read_file read 0% SWBUILD VDI VDA-1 VDA-2 DB-1 DB-2 EDA-1 EDA-2 PROPOSED

INTRO FS FS OPS PERF FUN END