Dell EMC Powerscale Onefs: Macos Network Storage User Experience and Performance Optimizations

Home , Finder (software), Resource fork, Spotlight (software), Streams

Best Practices Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations

Abstract This document describes performance and user-experience optimizations for Apple® macOS® 10.14 (Mojave), 10.15 (Catalina), and 11 (Big Sur) with the Dell EMC™ PowerScale™ OneFS operating system version 9.2.

June 2021

H17613.2 Revisions

Revisions

Date Description February 2019 Initial release March 2019 Minor updates April 2019 Updates on .DS_Store behavior and flow control recommendations June 2021 Updated for macOS 10.15 and 11. Added recommendations and troubleshooting advice.

Acknowledgments

Special Thanks: Thomas Berglund

The following members of Dell EMC storage engineering staff produced this paper:

Author: Gregory Shiff

Support: Simon Haywood, Rafal Szczesniak

The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable software license.

This document may contain certain words that are not consistent with Dell's current language guidelines. Dell plans to update the document over subsequent future releases to revise these words accordingly.

This document may contain language from third party content that is not under Dell's control and is not consistent with Dell's current guidelines for Dell's own content. When such third party content is updated by the relevant third parties, this document will be revised accordingly.

Copyright © 2019–2021 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. [6/4/2021] [Best Practices] [H17613.2]

2 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2 Table of contents

Table of contents

Revisions...... 2 Acknowledgments ...... 2 Table of contents ...... 3 Executive summary ...... 5 Audience ...... 5 1 Introduction ...... 6 2 Network storage protocols and macOS ...... 7 2.1 SMB multichannel in macOS 11.3 ...... 7 3 Editing sysctl.conf ...... 9 3.1 TCP delayed acknowledgment (delayed_ack) ...... 9 3.1.1 Delayed ack: macOS 10.14 and 10.15 ...... 9 3.1.2 Delayed ack: macOS 11 ...... 9 3.1.3 Adjusting delayed ack ...... 9 4 Defaults and com.apple.desktopservices ...... 11 4.1 .DS_Store files on network shares ...... 11 5 Editing /etc/nsmb.conf ...... 12 5.1 SMB signing ...... 12 5.2 SMB session signing ...... 13 5.3 SMB change notifications ...... 13 5.4 Force SMB protocol version ...... 14 5.5 Enabling alternate data streams (named streams) ...... 14 5.6 Apple SMB extensions ...... 15 5.6.1 Disabling Apple extensions: macOS 10.14 ...... 15 5.6.2 Disabling Apple SMB extensions: macOS 10.15 and 11 ...... 15 5.7 Prefer wired connections for SMB multichannel ...... 16 5.8 Example /etc/nsmb.conf ...... 16 6 Flow control, jumbo frames, and macOS ...... 17 7 Setting boot arguments for improved network performance ...... 19 7.1 Increasing network memory allocation ...... 19 7.2 Configuring ServerPerfMode ...... 19 7.3 Appending Boot Arguments ...... 20 8 Spotlight search of network volumes ...... 22 8.1 Enabling Spotlight search ...... 22 8.2 Performing Spotlight searches of network volumes ...... 22

3 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2 Table of contents

8.3 Performance implications of Spotlight and how to disable ...... 23 9 Using Apple internal 10 GbE NICs with OneFS ...... 24 10 Recommended settings for PowerScale and macOS ...... 25 11 Troubleshooting Apple performance issues ...... 26 11.1 Reduce complexity ...... 26 11.2 Simplify the client ...... 26 11.3 Observe macOS network statistics ...... 27 12 Examining SMB mounts with smbutil ...... 28 13 Conclusion ...... 29 A Technical support and resources ...... 30

4 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2 Executive summary

Executive summary

Many environments and workflows that feature Dell EMC™ PowerScale™ systems as the primary shared storage platform are highly reliant on systems running Apple® macOS® operating system. This guide provides performance-tuning and user-experience configurations for macOS 10.14, 10.15, and 11 with the PowerScale OneFS operating system, version 9.2.

The tunings in this guide provide advice for controlling macOS behavior when interacting with PowerScale storage. It is meant to allow administrators the ability squeeze the maximum performance out of the macOS client. For the most part, the settings in this guide will not “fix” an environment where performance is unstable or unusable. In those cases, there are likely issues going on that are beyond the tuning advice in this guide. See troubleshooting advice in section 11.

One notable exception is the poor performance of the onboard 10GBASE-T NICs in modern Apple professional systems. There were significant driver issues with those NICs in macOS releases before macOS 11. When running earlier macOS releases, storage performance of macOS clients connecting to OneFS using the onboard 10GBASE-T NICs is poor. A setting change in OneFS immediately fixes this issue, and Apple resolved the driver problems in macOS 11. See section 9 for details. Audience

This guide assumes the reader is proficient with macOS, the macOS command-line interface, and UNIX style operating systems in general.

5 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2 Introduction

1 Introduction This paper focuses on the SMB protocol with macOS 10.14, 10.15, and 11 when connecting to PowerScale OneFS 9.2.

Historically, many optimizations were necessary on macOS to achieve maximum performance from network storage. These included adjustments such as Send and Receive Windows, Window Scaling, Inflight Bandwidth Calculation, TCP Slow Start, and others. Fortunately, many such optimizations are no longer necessary due to improvements in how macOS dynamically autotunes its network performance.

The following performance optimizations and user experience changes are covered in this document:

• Edits to /etc/sysctl.conf, • Settings in com.apple.desktopservices

- DSDontWriteNetworkStores

• Edits to /etc/nsmb.conf

- SMB signing - SMB session signing - SMB notifications - Force protocol version - Alternate data streams (named Streams) - Apple SMB extensions - Force SMB multichannel to prefer wired connections

• Jumbo Frames • Memory allocation to network stack, including changes to default nvram boot-args • Server Performance mode

No single optimization is right for every environment. To take advantage of the tuning parameters outlined in this paper, testing is critical to determine how each change affects the performance of storage clients and their unique workloads.

6 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

2 Network storage protocols and macOS Three network storage protocols are supported by macOS: AFP, SMB, and NFS. There is some support for FTP with macOS, but that is out of scope for this document.

AFP is unsupported by PowerScale, and since macOS 10.9, AFP is not the default file-sharing protocol for macOS. As such, it is not covered in this document.

Historically, NFS was the protocol of choice for connecting macOS systems to PowerScale. NFS is no longer the best choice for macOS. Testing described in this paper concluded that SMB is as performant, and often more performant, than NFS on macOS.

SMB is the protocol of choice for macOS clients connecting to PowerScale storage. This paper shows that SMB consistently outperformed NFS on macOS, and describes additional topics:

• Alternate data streams (named streams) support

- PowerScale has robust support for macOS resource fork data stored through alternate data streams over SMB. - OneFS does not support macOS clients writing resource fork data to alternate data streams over NFS.

• Full ACL support

- SMB fully supports file system ACLs for fine-grained control of file access permissions. - OneFS supports NFSv4 ACLs which can be used on macOS, but can introduce further complexity, especially in multiprotocol, multiplatform environments (and are not recommended).

• PowerScale support for Apple-specific extensions to the standard SMB protocol

- The Apple SMB implementation (SMBX) has added a few macOS-specific features. OneFS supports some of these Apple-specific extensions to SMB.

2.1 SMB multichannel in macOS 11.3 SMB multichannel is a major advantage of SMB3. PowerScale OneFS 7.1.1+ supports SMB3 multichannel by default. Apple has introduced SMB3 multichannel support as of macOS 11.3. Apple describes this functionality in the following KB article: Configure SMB Multichannel behavior

SMB multichannel provides two features, failover and additional throughput.

Testing for this paper indicates that failover works as expected in macOS 11.3. A macOS client was configured with two 40GbE links to a OneFS file system (using an ATTO™ Thunderlink™ adapter). While the macOS client was playing back several streams of video from a OneFS storage volume, one of the network links was pulled from the switch. The video continued playing smoothly with little to no dropped frames.

However, when that network link was reestablished, the macOS client did not recognize it. The network file system had to be unmounted, and the user had to log out and back into their account for the macOS client to reestablish the second link.

Multiple Ethernet connections to the storage only resulted in a small increase of aggregate throughput. Testing with 2x 10 GbE links to a OneFS file system showed approximately a 10% read improvement and 20% write improvement. Testing with 2x 40 GbE links to the same file system showed little to no read and

7 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

write throughput improvement. SMB multichannel support is new in macOS, and it is likely that over time the throughput benefits will improve.

The above Apple KB article on SMB multichannel behavior includes several /etc/nsmb.conf settings that impact how macOS uses this feature. Since SMB multichannel is enabled by default in OneFS, it is recommended to set macOS to use wired Ethernet links for SMB Multichannel. Using wireless connections for SMB multichannel could incur significant performance penalties.

To force macOS 11.3 to prefer wired connections for SMB multichannel, add the following line to /etc/nsmb.conf.

mc_prefer_wired=yes

See Section 5 in this document for more information on creating and editing /etc/nsmb.conf.

8 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

3 Editing sysctl.conf On macOS, the /etc/sysctl.conf file can be edited to change some of the default settings in the macOS kernel at boot time. There is also a CLI command, sysctl, that when run as root will make temporary changes to those same settings. To make the changes persist across reboots, editing the /etc/sysctl.conf file is required. If the file is not present on a system, a text editor can be used to create it.

3.1 TCP delayed acknowledgment (delayed_ack) There are many explanations for TCP delayed acknowledgment (delayed_ack), such as the description in the article TCP Performance problems caused by interaction between Nagle’s Algorithm and Delayed ACK. For most uses of TCP, delayed acknowledgment is a good thing and makes network communication more efficient. TCP acknowledgments are added to subsequent data packets. However, in practice on macOS, particularly with network connections greater than 10 GbE, TCP delayed acknowledgment can cause performance degradation.

There are four options in macOS for setting the characteristics of delayed_ack:

• delayed_ack=0: Responds after every packet (OFF) • delayed_ack=1: Always employs delayed_ack; 6 packets can get 1 ack • delayed_ack=2: Immediate ack after 2nd packet; 2 packets per ack (Compatibility Mode) • delayed_ack=3: Should auto-detect when to employ delayed ack; 3 packets per ack

3.1.1 Delayed ack: macOS 10.14 and 10.15 The default setting for macOS 10.14 and 10.15 is delayed_ack=3. Testing has shown that in some environments, particularly where the client is reading and writing simultaneously, delayed_ack=0 is typically the ideal value. Rendering a video sequence is an example of such behavior.

3.1.2 Delayed ack: macOS 11 In macOS 11, the default value is delayed_ack=0. While it is possible to temporarily change this value with the below sysctl -w command, the change does not persist across reboots. Values set for delayed_ack in the sysctl.conf file are ignored in macOS 11. Thus it would appear that Apple has settled on delayed_ack=0 as being the ideal value.

3.1.3 Adjusting delayed ack To query the current setting of the client, enter the following at the macOS command line:

$ sudo sysctl net.inet.tcp.delayed_ack

Example response:

net.inet.tcp.delayed_ack: 3

To change this setting, enter the following at the macOS command line:

$ sudo sysctl -w net.inet.tcp.delayed_ack=0

Example response:

net.inet.tcp.delayed_ack: 3 -> 0

9 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

To make the setting persist over a reboot (in macOS 10.14 and 10.15), edit the /etc/sysctl.conf file on the macOS client:

• Create or edit the file at /etc/sysctl.conf • Add the line net.inet.tcp.delayed_ack=0

10 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

4 Defaults and com.apple.desktopservices The way macOS Finder treats directories accessed through SMB can be changed by using the CLI defaults command to update preferences in com.apple.desktopservices.

Apple has outlined this behavior in the support article Adjust SMB browsing behavior in macOS High Sierra 10.13 and later.

Users must log out and log in again for changes to take effect.

4.1 .DS_Store files on network shares The macOS Finder automatically creates .DS_Store files in every folder it accesses. This file stores metadata about how to display that directory’s contents. The reading and writing of these files can slow down performance when listing the contents of directories with high file counts.

It is possible to prevent macOS from creating .DS_Store files on network shares in macOS 10.13. As of macOS 10.14, it is no longer possible to stop these files from being created on network shares. However, it is possible to prevent macOS from reading the .DS_Store file before listing a directory’s contents. In the absence of a .DS_Store file or if reading the .DS_Store file is suppressed (as in 10.14 and higher), macOS will list the contents of a folder in alphanumeric order only upon initial open. Listing folders in this way has been shown to significantly reduce the time it takes for macOS to display the contents of directories with large numbers of files, such as those containing image sequences.

The following macOS CLI command prevents macOS 10.14 and higher from reading .DS_Store files on network shares:

defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool TRUE

After running this command, the user will need to log out and log in for the changes to take effect.

For more information about .DS_Store files, see the .DS_Store Wikipedia article.

11 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

5 Editing /etc/nsmb.conf The /etc/nsmb.conf file is a plain text file that offers configuration settings for how macOS treats SMB shares. If this file is not present on a system, it will need to be created with a text editor. The /etc/nsmb.conf file is divided into sections as outlined on its man page. For these edits, defaults for all SMB shares need to be set. The first line in the file should be [default] followed by the parameters and their values:

[default] parameter=value parameter2=value2 parameter3=value3

After updating /etc/nsmb.conf, unmount and remount SMB shares from the macOS system for changes to take effect.

5.1 SMB signing Packet signing increases the security of SMB connections by helping prevent man-in-the-middle attacks. Essentially, a digital signature attached to each packet helps the client system confirm that data has not been tampered with while in transit.

Packet signing is enabled by default in OneFS and is also the default behavior for SMB2 and SMB3 connections on macOS versions 10.11.5 through 10.13.3. Later versions of Apple’s documentation are a little murky as to when SMB signing is enabled. This archived KB article HT205926 states:

Packet signing for SMB 2 or SMB 3 connections turns on automatically when needed if the server offers it.

The extra overhead of packet signing can cause significant performance degradation on the latency-sensitive, high-performance workloads common to PowerScale OneFS. The security risks posed by man-in-the-middle attacks need to be assessed for each environment. When possible, it is recommended that these workflows take place on private, secure networks, and that packet signing is disabled on the macOS client.

Testing for this paper showed that SMB signing was automatically enabled on the macOS client when connecting to a native Apple file server. SMB signing did not get enabled automatically when connecting to PowerScale OneFS (which also offers SMB signing).

Given the inconsistent nature of the SMB signing behavior in macOS it is recommended to manually disable the feature using the /etc/nsmb.conf file.

Apple outlines disabling SMB signing in the support article Turn off packet signing for SMB 2 and SMB 3 connections.

To disable SMB signing on macOS, add the following entry to /etc/nsmb.conf:

signing_required=no

After updating /etc/nsmb.conf, unmount and remount SMB shares from the macOS system for the changes to take effect.

12 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

5.2 SMB session signing SMB3 introduces security enhancements that help prevent man-in-the-middle attacks during the initiation of an SMB client connection to an SMB server. SMB session Signing is different from SMB signing which adds a digital signature to each packet. SMB session signing can be described as a way to protect an SMB session from being tampering with as it commences. When macOS client systems are bound anonymously to a directory server, SMB session signing may cause authentication errors when a system tries to connect to an SMB share.

In situations where proper network credentials are not working from macOS systems running version 10.13+, disabling SMB session signing may resolve the issue. As with disabling SMB signing, disabling SMB session signing reduces the security of an SMB connection and is recommended on systems running on private, secure networks.

Apple describes session signing and how to disable it in the support article If you can't mount SMB share hosted by a Mac bound to Open Directory.

Add the following line to the /etc/nsmb.conf file:

validate_neg_off=yes

After updating /etc/nsmb.conf, unmount and remount SMB shares from the macOS system for the changes to take effect.

5.3 SMB change notifications The SMB server provides macOS with updates or changes to mounted file shares. SMB notifications provided by PowerScale provide the macOS client with this information in a OneFS environment. With a busy file share, this may result in the macOS Finder refreshing itself frequently and users may notice their file listings fluctuating or Finder windows reset to the top level directory while browsing. To avoid this behavior, it is possible to disable the Finder from requesting SMB change notifications.

Applying this setting can break workflows that require SMB notifications for folder listings to be current. For example, the collaborative Productions feature in Adobe™ Premiere Pro™ relies on change notifications to be active on all clients participating in the Production. Disabling change notifications can also lead to data corruption and other issues where multiple users are accessing the same files and directories. Therefore, as with all macOS optimizations, great care and testing is required to make sure that enabling this setting does not cause workflow problems.

To disable SMB notification, add the following line to the /etc/nsmb.conf file:

notify_off=yes

After updating /etc/nsmb.conf, unmount and remount SMB shares from the macOS system for the changes to take effect.

13 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

5.4 Force SMB protocol version Under certain circumstances, it may be desirable to force macOS to connect through a particular version of SMB. The following edits to /etc/nsmb.conf force a particular SMB version. Note that macOS uses a binary bitmap to specify which version of SMB to use. Add protocol_vers_map= and the appropriate value to force protocol types. See the following examples.

# Protocol version is specified using binary bitmap # 1 => 0001 => SMB 1 only # 2 => 0010 => SMB 2 only # 3 => 0011 => SMB 1 or 2 # 4 => 0100 => SMB 3 only # 6 => 0110 => SMB 2 or 3

# To Force SMB 3 only: protocol_vers_map=4

# To Force SMB 2 or 3 only protocol_vers_map=6

5.5 Enabling alternate data streams (named streams) PowerScale OneFS supports storing Apple resource fork data transparently in an alternate data stream, also called a named stream. This prevents the creation of AppleDouble files, increases macOS performance when writing to OneFS, and helps prevent file corruption.

To better understand this feature, refer to the following background information.

When macOS stores a file, that file is composed of two forks. There is a resource fork which contains extended attributes about the file itself, such as the file’s icon. There is also a data fork which contains the file data itself. When macOS writes data to local file systems, such as HFS+ or APFS-formatted volumes, the resource fork data is stored in an alternate data stream and is not visible to users or systems accessing those files; only the file itself is shown with all of its associated metadata.

When macOS writes to a storage system that does not support alternate data streams, the resource fork data still needs to be stored somewhere. In this case, macOS stores the resource fork data in AppleDouble files. These are files beginning in ._ followed by the file name. For example, saving a file called file.dat results in two files being created on the file system: file.dat and ._file.dat. This degrades performance because it increases the file operations on the target storage system. File corruption can also become an issue if a non- macOS system overwrites or deletes the AppleDouble files.

As mentioned previously in this section, OneFS supports storing resource fork data in an alternate data stream which prevents the creation of AppleDouble files. It is the default behavior of macOS 10.5+ to make use of an alternate data stream on an SMB server if the feature is available (as it is on OneFS). When writing through NFS, macOS will revert to storing resource fork data as AppleDouble files (another good reason to always use SMB with macOS).

OneFS intelligently manages resource fork data so even non-Apple systems connected by NFS can interact with files created or used by SMB-connected macOS systems with no corruption of the resource fork data and no need for AppleDouble files.

14 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

While it is not necessary to force macOS to use alternate data streams (the default since macOS version 10.5), some administrators feel more comfortable explicitly enabling this feature in /etc/nsmb.conf by adding the following line:

streams=yes

The following archived support document outlines this functionality: Mac OS X v10.5, v10.6: About named streams on SMB-mounted NAS, Mac OS X, and Windows servers; "-36" or "-50" alerts may appear.

5.6 Apple SMB extensions Apple has added several macOS-specific features to their implementation of SMB. Primarily, these features address the handling of file metadata stored in alternate data streams as described previously. OneFS supports these features:

ReadDirAttr: This feature changes how macOS handles reads of file metadata stored in alternate data stream when listing the contents of large directories. Finder info, access rights, and resource fork size are returned more efficiently for the files in the directory.

OsxCopyFile: With the SMB2 protocol, Microsoft implemented server-side optimizations when copying files between directories on the file share. The extension introduced by Apple ensures that all Apple-specific file metadata is properly copied along with the file itself. The copy process is also simplified as it is performed in one request as opposed to splitting the requests into logical chunks. When performing a large server-side copy, the macOS Finder ignores updates from the server. During these copies, the Finder copy progress bar does not update and may appear to have stopped responding. When the copy finishes, the progress bar snaps to complete.

In versions 10.13+, macOS has changed how it interacts with SMB servers to pull directory file listings. Unintuitively, the Apple SMB extensions can slow performance in workflows that have directories with high file counts or elaborate project files (such as complicated video edit projects). Disabling Apple extensions in OneFS significantly improves performance in those situations.

5.6.1 Disabling Apple extensions: macOS 10.14 It is possible to disable Apple-specific SMB features on macOS clients running 10.14. To disable the macOS client from using these extensions, add the following line to /etc/nsmb.conf:

aapl_off=true

5.6.2 Disabling Apple SMB extensions: macOS 10.15 and 11 It is not possible to disable Apple-specific SMB extensions in macOS 10.15 and 11. Instead, the Apple extensions must be disabled on the OneFS storage cluster itself. To disable Apple SMB extensions in OneFS, run the following command from the OneFS CLI:

isi_gconfig \ registry.Services.lwio.Parameters.Drivers.srv.smb2.EnableAppleExtensions=0

The following screen captures show the share information before and after Apple extensions were disabled on the macOS client. Notice that before they are disabled, macOS identifies PowerScale as a fully native file share, while afterwards, PowerScale shows up as a more generic share.

15 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

5.7 Prefer wired connections for SMB multichannel macOS 11.3 adds SMB3 multichannel support as outlined in Apple KB HT212277 and Section 2.1 of this document. Apple outlines a couple of settings that can be changed with /etc/nsmb.conf. Apple outlines how to disable multichannel all together and force multichannel to prefer wired connections. Due to the possible negative performance impacts of using Wi-Fi with SMB multichannel, it is recommended to force macOS to prefer wired connections.

To force macOS 11.3 to prefer wired connections for SMB multichannel, add the following line to /etc/nsmb.conf.

mc_prefer_wired=yes

5.8 Example /etc/nsmb.conf The following shows an example of the /etc/nsmb.conf. Notice that the disabling of Apple-specific SMB extensions is present in the file but has been commented out.

16 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

6 Flow control, jumbo frames, and macOS Using jumbo frames (9,000 MTU) can increase macOS SMB performance on network connections greater than 1 GbE. However, implementation of jumbo frames requires more than just setting a NIC port to use 9,000 MTU. The macOS client, network switch, and PowerScale storage must all be properly configured.

Keep in mind that changes to network settings on the switch, clients, and storage will cause connections to drop. As such, these changes should not be undertaken during active production.

If jumbo frames are used, the network on which the macOS client connects to the PowerScale storage should be dedicated to production. The network should not be used for non-storage traffic (such as email or Internet browsing). Essentially, MTU sizes should be consistent on the network and that network should not be cluttered with non-storage-related traffic. There should be no 1500 MTU traffic on the storage network running at 9000 MTU.

The ports on the network switch itself must be configured to use jumbo frames, which means setting those ports to greater than 9,000 MTU. On the switch used for testing in this document, the ports were set to 9,412 MTU. See to the documentation for the specific switch in use for optimal port MTU settings. The extra overhead makes room for any additional data attached to the network packets, such as VLAN header information. In testing, if the switch ports were set to exactly 9,000 MTU, macOS clients stopped responding when connecting to network shares, even on a flat network. Administrators should refer to their switching documentation for proper port configuration, since some switches may transparently add additional MTU to a port and others need to be explicitly configured.

Another critical setting for using macOS with jumbo frames is to enable flow control on the switch. Flow control for both RX and TX should be enabled on the network switch for all ports that are connected to both the macOS client and PowerScale storage. In absence of switch side flow control, macOS jumbo frame performance was lower than the standard 1,500 MTU. Other platforms tested were not as sensitive to flow control. It made little to no difference in Windows or Linux performance but was required on macOS to achieve good performance using jumbo frames.

Enabling flow control on switch ports in use by macOS and PowerScale is recommended, even when running at 1,500 MTU. PowerScale storage is fast and can overflow the macOS network buffers causing dropped packets and throughput degradation.

Once the ports are properly configured on the network switch, the external network ports on the PowerScale storage must be set for 9,000 MTU. See the OneFS documentation on the PowerScale Info Hubs for setting the external network ports on the PowerScale storage to 9,000 MTU.

Now that OneFS and the switch are set to jumbo frames, the NIC ports used by the macOS client can be configured. There are two settings that must be configured on the macOS client: Jumbo (9000) MTU and Duplex: full-duplex, flow-control. Setting Duplex to full-duplex with no flow-control resulted in poor performance, even when flow control was disabled on the switch. To realize the benefit of jumbo frames on macOS, flow control and MTU settings must be properly configured on both the client and the network switch.

17 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

In the macOS UI, this can be configured in System Settings > Network. Select the appropriate NIC port, select Advanced Settings, and select the Hardware tab.

In the hardware tab, configure the port Manually and set the Duplex and MTU controls.

18 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

7 Setting boot arguments for improved network performance Several boot arguments (boot-args) can be configured to increase the amount of system resources allocated to network performance in macOS. These include increasing network buffer size and enabling ServerPerfMode.

Configuring boot-args must be done from the macOS terminal while the system is booted into Recovery Mode. Apple describes the process of booting to recovery mode in the article Mac startup key combinations.

To view currently set boot-args, run the following terminal command:

nvram boot-args

Setting boot-args explicitly from the command line will overwrite existing boot-args. Apple provides a sample code snip-bit in the KB article about ServerPerfMode that uses sed and cut to append existing boot-args to newly set ones. The example in that article is for ServerPerfMode, and can be adapted for other settings such as ncl.

Client systems have a finite amount of resources. Allocating additional system resources to the network stack means that there is less RAM and CPU available for other purposes. Administrators need to be aware of the implications changing the defaults (as outlined below), and be prepared to roll the changes back if the client systems becomes constrained as a result.

7.1 Increasing network memory allocation Increasing the amount of memory that macOS allocates for network buffering can increase throughput for network connections greater than 1 GbE.

Depending on the total system memory and the network connection type, various values can be set. 512 MB may be the logical maximum.

Once booted into recovery mode, launch the Terminal application and use the following commands to add a boot argument. Note this will overwrite any existing boot-args:

nvram boot-args="ncl=262144"

This setting is the number of 2 KB clusters assigned to the network buffer and is calculated as follows:

(262,144 * 2) / 1,024 = 512 MB

After setting this nvram boot argument, reboot the macOS system for the changes to take effect. Once rebooted, run the following terminal command to verify the boot argument:

nvram boot-args

To remove this boot-args setting, run the following terminal command:

nvram -d boot-args

7.2 Configuring ServerPerfMode ServerPerfMode updates various sysctl settings in macOS to increase system resources available to the network stack. While the name suggests that this setting is aimed at Apple systems being used as servers, testing has shown that client systems also benefit by enabling ServerPerfMode. In one particular example, a

19 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

video file in Canon’s™ C200 codec at 50 MBps (400mbps) would not play back smoothly until ServerPerfMode was enabled. This was true despite both the Apple hardware and PowerScale storage providing more than enough performance and other heavier codecs playing fine.

Apple does not provide specific documentation about all the changes enabled by ServerPerfMode. Only documentation about how to enable it: Turn on performance mode for macOS Server

There is some discussion online about some of the sysctl changes that take place, such as this discussion on apple.stackexchange.com. The exact changes may largely depend on the total amount of RAM in the Apple system. One way to check would be to run: sysctl -a from the CLI before and after the change, dumping the output to a text files to compare.

To enable ServerPerfMode, boot the Apple client system into Recovery Mode. Once in Recovery Mode, run the following command from the Terminal application (existing boot arguments will be overwritten):

nvram boot-args=”serverperfmode=1”

7.3 Appending Boot Arguments Apple provides terminal commands to add new boot-args without overwriting existing boot-args in this KB article about enabling ServerPerfMode. Below are those commands adapted for both ServerPerfMode and ncl. All commands assume the Apple system is booted into Recovery Mode.

To set both ServerPerfMode and ncl simultaneously, overwriting existing boot-args. Run the following command from macOS Terminal:

nvram boot-args=”serverperfmode=1 ncl=262144”

To remove all boot-args simultaneously:

nvram -d boot-args

To enable ServerPerfMode and keep existing boot-args:

nvram boot-args="serverperfmode=1 $(nvram boot-args 2>/dev/null | cut -f 2-)"

To disable ServerPerfMode and keep existing boot-args:

nvram boot-args="$(nvram boot-args 2>/dev/null | sed -e \ $'s/boot-args\t//;s/serverperfmode=1//')"

To update ncl setting and keep existing boot-args:

nvram boot-args="ncl=262144 $(nvram boot-args 2>/dev/null | cut -f 2-)"

To remove ncl setting and keep existing boot args:

nvram boot-args="$(nvram boot-args 2>/dev/null | sed -e \ $'s/boot-args\t//;s/ncl=262144//')"

To update both ServerPerfMode and ncl simultaneously while keeping existing boot-args:

nvram boot-args="serverperfmode=1 ncl=262144 $(nvram boot-args 2>/dev/null \ | cut -f 2-)"

To remove both ServerPerfMode and ncl settings simultaneously while keeping other boot-args:

20 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

nvram boot-args="$(nvram boot-args 2>/dev/null | sed -e \ $'s/boot-args\t//;s/serverperfmode=1//;s/ncl=262144//')"

21 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

8 Spotlight search of network volumes Spotlight search of network volumes (including PowerScale OneFS) is supported in macOS 10.14, 10.15 and 11. However there are some significant performance implications that need to be considered. Each individual Apple client maintains its own local database of the network share. Having multiple Apple systems simultaneously index a file system will degrade performance for all users until the index is complete.

To verify status of Spotlight search of the PowerScale OneFS volume, run the following terminal command from the Apple client: mdutil -s /path/to/network/share/

8.1 Enabling Spotlight search To enable Spotlight search on the network share, follow this process from the Apple client:

From the macOS UI, verify that the network volume is not listed:

System Preferences → Spotlight → Privacy

Next from the macOS terminal disable Spotlight for the network share with:

sudo mdutil -i off /path/to/network/share

Erase any existing Spotlight index for that share:

sudo mdutil -E /path/to/network/share

Reenable Spotlight indexing for the share:

sudo mdutil -i on /path/to/network/volume

If enabling Spotlight search on the network volume fails, examine the permissions on the share, they may need to be made more permissive.

Once Spotlight indexing has begun, keep an eye on the mdworker process running on the Apple client.

The Apple client system stores a local index of the network volume on its internal storage. Separate indexes are maintained for each network storage login and IP address. As a result, it is conceivable that multiple indexes of the same network volume could be resident on a single Apple client. These indexes can grow large depending on the size and complexity of the volume being scanned. The Spotlight index files for network storage are located here:

/var/db/Spotlight-V100/LocalVolumes/

Occasionally, users may find that Spotlight search of a network volume stops returning fresh results. The only fix for a “broken” Spotlight index is to disable search, erase the database, and reenable search, as outlined above.

8.2 Performing Spotlight searches of network volumes Spotlight search of network volumes is slightly different from how macOS users are used to using Spotlight.

• Spotlight search results for network volumes do not show up using the macOS key command “Command+Space bar”.

22 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

• Spotlight search results for network volumes do not show up in searches using the Magnifying Glass icon in the upper right of the macOS Desktop. • Spotlight search results only show up when searching from the search field in the upper right of a macOS Finder window with the network share selected.

Below is a screenshot of a Spotlight search for the term “autodesk” on the network volume “Space”.

8.3 Performance implications of Spotlight and how to disable The macOS process which maintains the Spotlight index, mdworker, is aggressive in building its database. Mdworker will begin creating this database quickly after being enabled. Even a few macOS systems could cause significant storage traffic and degrade performance for other users. Spotlight search on Apple clients should be initially enabled during off hours and in small batches in order to minimize performance impact during initial database build.

When the OneFS file system is used for latency sensitive workflows, such as video editing, Spotlight search should be disabled on network storage. For fast file system search in those environments, alterative tools such as Dell DataIQ™ maybe more appropriate.

Spotlight indexing of a network volume can be disabled in several ways.

From the macOS UI, add the network volume to the list at:

System Preferences → Spotlight → Privacy

From the macOS command line:

mdutil -i off /path/to/network/volume

mdutil -d /path/to/network/volume

23 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

9 Using Apple internal 10 GbE NICs with OneFS Before macOS 11, there were significant driver issues with the integrated 10GBASE-T NICs on professional Apple hardware. A change in OneFS to protect against certain kinds of Denial of Service (DOS) attacks resulted in poor performance for Apple systems using integrated 10GBASE-T NICs to connect to OneFS. Fortunately, the macOS driver issues are resolved in macOS 11.

However, for users of macOS 10.14 and 10.15, updating the PowerScale cluster setting of net.inet.tcp.reass.maxqueulen to 2048 resolves these performance problems. This change is unnecessary for macOS 11 users. The procedure for making the change is outlined in the following Dell KB article (login required): Sysctl Security Change Can Affect TCP Performance On Degraded Networks OneFS v8.1.0+

Increasing net.inet.tcp.reass.maxqueulen can make the PowerScale cluster more vulnerable to certain DOS attacks, but if the PowerScale cluster is on a secure, private network, the risk is minimal.

To view the current setting of net.inet.tcp.reass.maxqueuelen, on the PowerScale cluster run the following CLI command (default value is 100):

sysctl net.inet.tcp.reass.maxqueuelen

Temporarily change the value on a particular node (the node to which the macOS 10.14 or 10.15 client is connected):

sysctl net.inet.tcp.reass.maxqueuelen=2048

Have the clients check the network to confirm if the performance issue has been resolved. The change should be immediate, the macOS clients do not need to unmount/remount the volume.

If the issue is resolved, change the value for the whole cluster on reboot by editing the following file:

vi /etc/mcp/defaults/sysctl.conf

If the sysctl exists, edit that line. If it is not in the file, add these lines to the bottom of the file:

net.inet.tcp.reass.maxqueuelen=2048

Push the change to all nodes in the cluster:

isi_sysctl_cluster net.inet.tcp.reass.maxqueuelen=2048

24 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

10 Recommended settings for PowerScale and macOS The following recommendations are a good starting point. Having a second macOS system with a fresh install of the operating system (and no optimizations) to compare against is ideal.

On the PowerScale OneFS cluster:

• Disable Apple extensions o It is counterintuitive, but the current state of Apple extensions causes large directory listings and opening complex projects to be slow. Disabling Apple extensions on the PowerScale cluster disables them for all clients, avoiding the individual operating system behavior of 10.14 vs 10.15+. • Increase the net.inet.tcp.reass.maxqueuelen setting to 2048. o This change assumes the PowerScale cluster is on a private secure network. While technically this change is unnecessary for macOS 11, at least at the time of this writing, most macOS users are still running 10.14 or 10.15. Updating this setting does not negatively impact macOS 11.

On the Apple client system:

• Boot into Recovery Mode, enable ServerPerfMode and increase network buffer allocation. o The default settings of the macOS network stack are not optimized for network storage performance. Making these changes means the Apple system will devote more resources to networking. • Use the defaults command to force macOS to ignore.DS_Store files on network shares o In combination with disabling Apple extensions on the PowerScale cluster, this greatly improves listing large directories in the macOS Finder. • If connecting by 10 GbE or higher, change delayed_ack to 0 (macOS 10.14 and 10.15) o The default setting of 3 can degrade performance with networks greater than 1 GbE o macOS 11 defaults to delayed_ack=0 • Configure the following changes to /etc/nsmb.conf o Disable SMB signing ▪ It is the default, but given the performance implications, best to hard code this setting o Disable SMB Session signing ▪ Reduce incidence of password and connectivity problems o Force SMB protocol version to SMB3 only ▪ SMB3 is the preferred protocol for macOS. If for some reason the client cannot connect by SMB3, it may be better to have the mount fail and trigger further investigation. o Set SMB3 multichannel to prefer wired connections ▪ Having a macOS system try to connect by Wi-Fi when a wired connection is available will negatively impact performance. • Disable Spotlight search on the network share o Unless necessary, Spotlight search can lead to performance degradation on network shares with large numbers of macOS clients.

On the network switch:

• Enable flow control both RX and TX for all macOS clients and PowerScale front-end ports. o This is a good idea for both 1500 MTU and 9000 MTU networks when used with macOS.

25 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

11 Troubleshooting Apple performance issues Administrators often arrive at this tuning guide when trying to solve macOS performance problems. The settings outlined in this document can be the difference between average macOS performance and exceptional performance. However the recommendations in this paper are not intended to “fix” environments where macOS systems are in an unusable state. An exception being the change of net.inet.tcp.reass.maxqueuelen for older versions of macOS, which does make a huge difference.

The good news is that a fresh install of macOS 10.14 and higher connected to a PowerScale cluster works reasonably well with no tuning whatsoever. This means browsing the file system, opening, reading, and writing files and playing many video formats.

The question then arises of where to start when macOS systems are slow to browse the file system, will not open or save files, and video playback is dropping frames. The trick here is to reduce complexity in the system, get back to a known good state and work backwards to the user’s end state. Determining where the problem lies (storage, network, client) is nontrivial in the best of circumstances. The following suggestions have helped administrators track down their problems and answer those challenging questions.

11.1 Reduce complexity The network itself is often the biggest source of complexity in a production environment. Removing as much of the network as possible between the client and storage is a good first step.

• Connect the macOS client directly into one of the PowerScale front-end network ports. o Otherwise connect the macOS system into the same switch as the PowerScale front-end ports. o If plugged through a switch, verify that flow control (RX and TX) is enabled for all macOS and storage ports • Verify that the macOS system is on the same subnet and VLAN as the PowerScale and that no routing is required for the client system to access the storage. • Use standard 1500 MTU size, and that ping times are between the macOS and PowerScale are low (and conversely). o 9000 MTU can provide extra performance in macOS for the most demanding workflows. Unfortunately, 9000 MTU often causes more problems than it solves due to MTU mismatch on the network. Having all devices work well using 1500 MTU first and then experimenting with 9000 MTU is the recommended path.

If good performance is achieved with these steps, try advancing one network hop at a time to determine where functionality breaks down and efforts must be focused.

11.2 Simplify the client Did the system ever work as intended? Or did something change to “break” performance? Reimaging the macOS client back to a known good state is another thing to try:

• Using Recovery Mode, reimage the client back to a base install of macOS • Reinstall any third-party network drivers o Verify the latest driver version and macOS compatibility. Apple is notorious for breaking compatibility with new versions of macOS. • Verify speed of Thunderbolt cables and which Thunderbolt ports are in use.

26 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

o Thunderbolt 1 and Thunderbolt 2 cables look physically identical but provide different types of performance. Use of Thunderbolt 1 cables with Thunderbolt 2 compatible devices will not realize the full performance capabilities of PowerScale or the Apple client. ▪ Thunderbolt speeds can be checked from Apple menu → About this mac → System Report → Thunderbolt o This ATTO technical bulletin shows which Thunderbolt ports to use on various Apple hardware to avoid disconnects and performance issues: ATTO Technical Bulletin January 2020

11.3 Observe macOS network statistics From the macOS terminal, the nettop command is an easy way to observe periodically updated network statistics (default interval 1 second). The feedback provided by nettop can be helpful to understanding more about how the macOS client is communicating with network storage during certain application workloads. The following command show macOS TCP behavior for the kernel_task process, which is the parent process for all SMB connections on TCP port 445:

nettop -m tcp -p kernel_task -J interface,bytes_in,bytes_out,rx_dupe,rx_ooo,re- tx,rtt_avg,rcvsize,unacked,tx_win

Key metrics to look at, when analyzing network statistics with nettop are:

• rtt_avg – Round-Trip Time average (latency) o Should be as low as possible. If there are spikes above 10 milliseconds, bandwidth and performance will be negatively impacted. TCP and especially SMB traffic over TCP are sensitive to latency. • rx_ooo – Out of order TCP packets o Can be a sign of storage server not being able to reassemble packets. • re-tx – TCP Retransmits o Can be a sign of latency or packet loss • rx_dupe – TCP Duplicate ACKs o The higher the latency is, there more Duplicate ACKs will appear.

High latency or large amounts of out of order packets will degrade performance. Either of these issues could be the results of operating system or driver issues on the Apple client, network issues between the client and storage, or latency issues within the storage file system on a software or hardware-related layer like intermittently busy or queued up disks.

To confirm if the latency issues are related to SMB read or write requests, you can try capturing Wireshark/tcpdump trace files with a capture filter for port 445, while reproducing issues. When the capture is stopped, look at Statistics > Service Response Time > SMB2, and look out for high MAX SRT values. Details about Wireshark and tcpdump are out of scope for this paper.

Before going down the rabbit hole of analyzing TCP packets, try to simplify the environment (see Section 11.1 above). Reducing the amount variables between the client and the storage system is a good starting point.

27 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

12 Examining SMB mounts with smbutil The macOS CLI utility smbutil can provide useful information about what features are available and enabled on SMB-mounted file shares. The following command displays information about mounted SMB shares and their supported features:

smbutil statshares -a

Running this command helps administrators verify that edits made to /etc/nsmb.conf have taken effect. It shows detail about the SMB mount unavailable elsewhere in macOS.

The following shows sample output of this command on a macOS 11 system while connecting to an PowerScale cluster running OneFS 9.2. Apple Extensions have been disabled on the PowerScale cluster and the macOS client has been forced to use SMB3.

28 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2

13 Conclusion The tight integration of Apple hardware and macOS remain a favorite of many professional users. As such, delivering the highest performance is a priority for macOS administrators. By understanding how macOS interacts with network storage, performance can be optimized for each environment.

Choosing correct network storage protocol, MTU settings, security features, and macOS Finder functionality are critical to this process. Apple provides a manageable mix of parameters to make meaningful changes in each category. As with any complex system, testing is critical to ensure that changes to the default behavior have the intended consequences.

PowerScale treats macOS as a first-class citizen. By tuning client system interaction with the storage, users can take advantage of PowerScale performance across diverse workflows.

29 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2 Technical support and resources

A Technical support and resources

Dell.com/support is focused on meeting customer needs with proven services and support.

Storage technical documents and videos provide expertise that helps to ensure customer success on Dell EMC storage platforms.

30 Dell EMC PowerScale OneFS: macOS Network Storage User Experience and Performance Optimizations | H17613.2