Reference Architecture Dell EMC Isilon and Cloudera Reference Architecture and Performance Results
Abstract This document is a high-level design, performance results, and best-practices guide for deploying Cloudera Enterprise Distribution on bare-metal infrastructure with Dell EMC’s Isilon scale-out NAS solution as a shared storage backend.
August 2020
H18523 Revisions
Revisions
Date Description August 2020 Initial release
Acknowledgments
Author: Kirankumar Bhusanurmath, Analytics Solutions Architect, Dell EMC
Support:
The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Use, copying, and distribution of any software described in this publication requires an applicable software license.
Copyright © 2020 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. [9/22/2020] [Reference Architecture] [H18523]
2 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Table of contents
Table of contents
Revisions ...... 2 Acknowledgments ...... 2 Table of contents ...... 3 Executive summary ...... 5 Disclaimer ...... 5 Audience and scope ...... 6 Glossary of terms ...... 6 1 Dell EMC Isilon distributed storage array for HDFS and bare-metal nodes as compute nodes ...... 8 1.1 Physical cluster component list ...... 9 1.2 Logical cluster topology ...... 9 1.3 Sizing recommendation for physical nodes ...... 10 1.4 Sizing recommendation for storage allocation ...... 11 1.5 Supportability and compatibility matrix ...... 11 2 Performance testing and platform positioning ...... 13 2.1 DFSIO testing with Dell EMC Isilon F800 ...... 13 2.1.1 Overview ...... 13 2.1.2 Test environment setup ...... 13 2.1.3 Testing with DFSIO ...... 13 2.1.4 Hadoop cluster configurations ...... 14 2.1.5 Performance results ...... 14 2.2 Dell EMC Isilon model for Hadoop use cases ...... 14 3 Platform tuning recommendations ...... 16 3.1 CPU ...... 16 3.1.1 CPU BIOS settings ...... 16 3.1.2 CPUfreq governor ...... 16 3.2 Memory ...... 17 3.2.1 Minimize anonymous page faults ...... 17 3.2.2 Disable transparent huge-page compaction ...... 17 3.2.3 Disable transparent huge-page defragmentation...... 17 3.3 Network...... 17 3.3.1 OneFS TCP tuning ...... 17 3.3.2 Compute nodes network tuning ...... 18 3.3.3 Verify NIC advanced features ...... 19 3.3.4 NIC ring buffer configurations ...... 19
3 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Table of contents
3.4 Storage ...... 19 3.4.1 Disk/FS mount options ...... 19 3.4.2 FS Creation Options ...... 20 A Technical support and resources ...... 21 A.1 Related resources ...... 21
4 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Executive summary
Executive summary
Dell EMC works closely with Cloudera, understanding the needs of Apache Hadoop customers, validating hardware and software-based solutions, empowering organizations with deeper insights and enhanced data- driven decision making by using the right infrastructure for the right data.
Dell EMC Isilon and Cloudera - Combines a powerful yet simple, highly efficient, and massively scalable storage platform with integrated support for Hadoop analytics. Dell EMC Isilon is the first, and only, scale-out NAS platform to incorporate native support for the HDFS layer. With your unstructured data on Isilon, these capabilities allow you to quickly implement an in-place data analytics approach and avoid unnecessary capital expenditures, increased operational costs, and time-consuming replication of your big data to a separate infrastructure.
This document describes the high-level design, performance results, and best-practices guide for deploying Cloudera Enterprise Distribution on bare-metal infrastructure with Dell EMC’s Isilon scale-out NAS solution as a shared storage backend. Disclaimer
The documentation is and contains Cloudera proprietary information protected by copyright and other intellectual property rights. No license under copyright or any other intellectual property right is granted herein.
Copyright information for Cloudera software may be found within the documentation accompanying each component in a particular release.
Cloudera software includes software from various open source or other third-party projects, and may be released under the Apache Software License 2.0 (“ASLv2”), the Affero General Public License version 3 (AGPLv3), or other license terms. Other software included may be released under the terms of alternative open-source licenses. Review the license and notice files accompanying the software for additional licensing information.
Please browse the Cloudera software product page for more information about Cloudera software. For more information about Cloudera support services, please browse either the Support or Sales page. Feel free to contact us directly to discuss your specific needs.
Cloudera reserves the right to change any products at any time, and without notice. Cloudera assumes no responsibility nor liability arising from the use of products, except as expressly agreed to in writing by Cloudera.
Cloudera, Cloudera Altus, HUE, Impala, Cloudera Impala, and other Cloudera marks are registered or unregistered trademarks in the United States and other countries. All other trademarks are the property of their respective owners.
Disclaimer: EXCEPT AS EXPRESSLY PROVIDED IN A WRITTEN AGREEMENT WITH CLOUDERA, CLOUDERA DOES NOT MAKE NOR GIVE ANY REPRESENTATION, WARRANTY, NOR COVENANT OF ANY KIND, WHETHER EXPRESS OR IMPLIED, IN CONNECTION WITH CLOUDERA TECHNOLOGY OR RELATED SUPPORT PROVIDED IN CONNECTION THEREWITH. CLOUDERA DOES NOT WARRANT THAT CLOUDERA PRODUCTS NOR SOFTWARE WILL OPERATE UNINTERRUPTED NOR THAT IT WILL BE FREE FROM DEFECTS NOR ERRORS, THAT IT WILL PROTECT YOUR DATA FROM LOSS, CORRUPTION NOR UNAVAILABILITY, NOR THAT IT WILL MEET ALL OF CUSTOMER’S BUSINESS REQUIREMENTS. WITHOUT LIMITING THE FOREGOING, AND TO THE MAXIMUM EXTENT PERMITTED
5 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Audience and scope
BY APPLICABLE LAW, CLOUDERA EXPRESSLY DISCLAIMS ANY AND ALL IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY, QUALITY, NON- INFRINGEMENT, TITLE, AND FITNESS FOR A PARTICULAR PURPOSE AND ANY REPRESENTATION, WARRANTY, OR COVENANT BASED ON COURSE OF DEALING OR USAGE IN TRADE. Audience and scope
This guide is for IT architects who are responsible for the design and deployment of infrastructure and a shared storage platform in the data center, as well as for Hadoop administrators and architects who will be data center architects or engineers and/or collaborate with specialists in that space.
This document describes Dell EMC and Cloudera recommendations on the following topics:
1. Storage array considerations 2. Data network considerations 3. Hardware/platform considerations
Glossary of terms
Below table the lists of glossaries and their description.
Glossary terms and descriptions Term Description NameNode The NameNode is a separate server that holds metadata for every file that is stored on the DataNodes. On an Isilon OneFS cluster, every node in the cluster acts as a NamNode. DataNode Worker node of the cluster to which the HDFS data is written. On an Isilon OneFS cluster, every node in the cluster acts as a DataNode HDD Hard disk drive HDFS Hadoop Distributed File System. In an Isilon OneFS cluster with Hadoop deployment, OneFS serves as the file system for Hadoop compute clients. ISL Interswitch link JBOD Just a Bunch of Disks (this is in contrast to disks configured using software or hardware with redundancy mechanism for data protection. Job History Server Process that archives job metrics and metadata. One per cluster. NIC Network interface card. NodeManager The process that starts application processes and manages resources on the DataNodes. PDU Power distribution unit.
6 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Glossary of terms
QJM Quorum Journal Manager. Provides a fencing mechanism for high availability in a Hadoop cluster. This service is used to distribute HDFS edit logs to multiple hosts (at least three are required) from the active NameNode. The standby NameNode reads the edits from the JournalNodes and constantly applies them to its own namespace. If there is a failover, the standby NameNode applies all the edits from the JournalNodes before promoting itself to the active state. QJN Quorum JournalNodes. Nodes on which the journal services are installed. RM ResourceManager. The resource management component of YARN. This initiates application startup and controls scheduling on the DataNodes of the cluster (one instance per cluster). ToR Top of rack. ZK Zookeeper. A centralized service for maintaining configuration information, naming, and providing distributed synchronization and group services.
7 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
1 Dell EMC Isilon distributed storage array for HDFS and bare- metal nodes as compute nodes In this model Isilon replaces HDFS shipped in Cloudera Enterprise.
In this architecture, Isilon acts as the HDFS/storage layer and the bare-metal nodes only provide the compute resources needed. Considerations for a storage component are not required, but care has to be taken to ensure reasonable oversubscription ratio between Isilon switches and the compute node switches.
The graphic below depicts a cluster deployed across several racks (Rack1, Rack 2, … Rack n).
Physical Cluster Topology
Each host is networked to two top-of-rack (TOR) switches which are in turn connected to a collection of spine switches which are then connected to the enterprise network. This deployment model allows each host maximum throughput, minimize of latency, while encouraging scalability. The specifics of the network topology are described in the subsequent sections.
8 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
1.1 Physical cluster component list Below table describes the cluster components, configurations, and quantity.
Physical cluster component list Component Configuration Description Quantity Physical servers DELL R740 servers Hosts that house the Minimum 3 master + various 5 compute (8 nodes) • 2 x Intel® Xeon Gold NodeManager and 6148 CPU @ 2.4Ghz 20 compute instances. core • 768 GB RAM
NICs Dual-port 10 Gbps Ethernet Provide the data At least two per NICs. network services server. Internal disk drives 2 SSD 480 GB(RAID) These ensure Two per physical continuity of service server configured as on server resets. a RAID 1 volume (mirrored). Ethernet ToR/leaf switches Minimally 10 Gbps switches Although most At least two per rack. with sufficient port density to enterprises have accommodate the compute mature data network cluster. These require practices, consider enough ports to create a building a dedicated realistic spine-leaf topology data network for the providing ISL bandwidth Hadoop cluster. above a 1:4 oversubscription ratio (preferably 1:1). Ethernet spine switches Minimally 10 Gbps switches Same considerations Depends on the with sufficient port density to as for ToR switches. number of racks. accommodate incoming ISL links and ensure required throughput over the spine (for inter traffic).
1.2 Logical cluster topology For the YARN NodeManager instances, data protection at the HDFS level is not required, because the physical nodes are running only the compute part of the cluster.
The minimum requirements to build out the cluster are:
1. Three Master Nodes 2. Five Compute nodes
The following table identifies service roles for different node types. For detailed information about this topic see recommended cluster hosts and role distribution.
9 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
Service roles and node types YARN
Master Node Master Node Master Node NodeManager nodes 1..n ZooKeeper ZooKeeper ZooKeeper ZooKeeper YARN Resource Manager Resource Manager History Server Node Manager Hive Meta-Store, WebHCat, HiveServer2 Management(misc) Cloudera Agent Cloudera Agent Cloudera Agent, Cloudera Agent Cloudera Manager, Management Services Ambari Agent Ambari Agent Ambari Server, Ambari Agent Ambari Agent Navigator Navigator, Key Management Services HUE HUE HBASE HMaster HMaster HMaster RegionServer Impala StateStore, Catalog Impala Daemon
NOTE: Low-latency workloads are subject to network latency, because all data traffic between compute nodes and HDFS (Isilon-based) is north-south traffic.
1.3 Sizing recommendation for physical nodes The following table provides size recommendations for the Physical nodes.
Physical nodes size recommendations Component Configuration Description Quantity Master Nodes: 2RU 2-socket These nodes house the Three (for scaling up to 100 nodes with at Master services and serve as cluster nodes). least 256 GB the gateway/edge device that RAM connects the rest of the customer’s network to the two-socket with 6-10 Hadoop cluster. cores/socket > 2 GHz; minimally 128 GB RAM; 8-10 disks compute instances: Have at least 8 These nodes will house the Results in the field show that SATA or SAS YARN node managers, and 1:2 ratio of Isilon nodes to two-socket with 6-10 Drives, or 2 additional required services. compute nodes is cores/socket > 2 GHz; SSD drives for reasonable for most use minimally 256 GB RAM intermediate cases. If there is Heavy storage. Impala workloads, use 1:1.5
10 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
2 x OS disks, 8 SATA or ratio. So if Isilon has 5 SAS drives or 2x SSDs nodes, have 8 compute nodes.
1.4 Sizing recommendation for storage allocation The following table provides recommendations for storage allocation.
Storage size recommendation Node/Role Disk Layout Description Management/Master • 2 x 500 GB OS (RAID 1) Avoid fracturing the file system layout into • Swap partition <= 2 GB multiple smaller file system. Instead, keep • 4 x 500 GB RAID 10 a separate “/” and “/var”. (database) • 1 x 500 GB RAID 0 - Zookeeper
Compute nodes • 2 x 500 GB OS (RAID 1) Avoid fracturing the file system layout into • Approximately 20% of total multiple smaller file system. Instead, keep a DFS storage (in this case separate “/” and “/var”. Isilon storage) needs to be provisioned as intermediate For example, for 10 TB of total storage in storage on these nodes. The Isilon, 2 TB is needed for intermediate storage can Direct-attached storage. SAS/SATA drives, or a pair of Having more or faster local spindles will SSD drives of sufficient speed up the intermediate shuffle stage of capacity. Distribute the 20% of MapReduce. capacity evenly across all the NodeManager nodes, with its own mount-point and file system.
1.5 Supportability and compatibility matrix The following table provides Dell EMC Isilon support for Hadoop distributions and their features.
Supportability and compatibility matrix Cloudera HDP or Apache Hadoop features Cloudera CDH Hadoop
Hive, MapReduce, Hbase, Spark ✔ Currently supported ✔ Currently supported
Hadoop 3.0 ✔ HDP 3 currently supported ✔ CDH 6 currently supported
WebHDFS ✔ Currently supported ✔ Currently supported
Knox ✔ Currently supported ✔ Currently supported
11 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
Transparent Data Encryption ✔ Supported in OneFS 8.2 ✔ Supported in OneFS 8.2 HDFS Snapshots Not currently supported, NFS Snapshots are supported HDFS ACLs Not currently supported, File ACLs are supported
Ambari, Falcon, Atlas ✔ Currently supported N/A
LLAP ✔ Currently supported N/A
Apache Ranger ✔ Authorization currently N/A supported
Audit not currently supported*
Cloudera Navigator N/A ✔ Lineage, Metadata, Search supported
Audit not currently supported*
Cloudera Manager N/A ✔ Currently supported
Sentry N/A Not currently supported*
Kudu N/A Not supported
For version specific compatibility, please refer to Hadoop distributions and products supported by OneFS web page.
See Cloudera Quality Assurance Test Suite (QATS) certification summary report for service-specific features supported on OneFS.
DellEMC Isilon: Cloudera CDH / QATS Execution Summary Report DellEMC Isilon: Cloudera HDP / QATS Execution Summary Report
12 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
2 Performance testing and platform positioning
2.1 DFSIO testing with Dell EMC Isilon F800
2.1.1 Overview Objective is to run a series of industry standard benchmark tests to analyze the performance throughput and reliability of F800.
2.1.2 Test environment setup The performance lab environment was configured with eight Isilon F800 nodes, and eight Dell R740 servers, below are the compute server details.
Specifications per server:
• 768 GB RAM • 2 x Intel® Xeon® Gold 6148 CPU @ 2.4 GHz 20 Core • Intel® 2P X710/2 I350 rNDC • MLNX 40 Gb 2P ConnectX3Pro adapter • Two SSD 480 GB (RAID 1) • CentOS Linux release 7.5.1804
The backend network between compute nodes and Isilon is 40 Gbps with Jumbo Frames set (MTU-9162) for the NICs and the switch ports.
Performance environment architecture diagram
2.1.3 Testing with DFSIO This performance testing uses the DFSIO utility that comes with HDP and CDH, while it is not a perfect benchmark it is useful in providing relative results within the environment. What we were striving for here was a test that would keep as many cores in the NodeManagers running as continuously as possible. Since
13 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
DFSIO creates a task for each file this was simple to deploy. With the compute side flooded we are showing the effect this has on Isilon as an HDFS datastore.
Example DFSIO command
yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-client- jobclient-2.6.0-cdh5.14.2-tests.jar TestDFSIO -write -nrFiles 10000 -fileSize 1000
2.1.4 Hadoop cluster configurations The major configuration applied here is the size of each mapper and reducer. Using some simple yarn math, we used 2 vCores of each container since each server had 8- vCores this would give 40 containers per server. Then allocated memory 8 GB per container. That used 320 GB of the total 768 GB for each run.
2.1.5 Performance results F800 provides good throughput results that reflect 18GBps for read and 16GBps for writes.
Isilon Throughput 20 Isilon Network Write Isilon Network Read 18 16 14 12 10 8
6 Throughput (GBps) Throughput 4 2 0 8 7 6 5 4 3 2 1 Number of Isilon Nodes
For more detailed results, please refer to the DFSIO testing with Isilon F800 blog post
2.2 Dell EMC Isilon model for Hadoop use cases Below table shows right Isilon model for Hadoop use cases.
Isilon model for Hadoop use cases Model Use cases
14 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
F800 • Multi-PB High-Density Hadoop Clusters (with 15.4 TB Drives). • Highly Latency sensitive Spark, Hbase, Cloudera, Impala, Hive/SQL-on-Hadoop • High-Performance Data Discovery and Visualization
H5600 • ” Performance Centric” Hadoop: Spark, Hbase, Cloudera Impala, Hive/SQL-on-Hadoop • Data Discovery and Visualization
H500 • “Storage Centric” Hadoop: MapReduce, Hive, Spark • Hadoop Tiered Storage Solution
A2000 • Archive for Hadoop-on-DAS
15 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
3 Platform tuning recommendations In this section we cover tuning recommendations only for the compute nodes, these are generic recommendations, but they should be applied only after sufficient testing. For detailed information, see Cloudera enterprise reference architecture for bare metal deployments.
For Isilon performance tuning and best practices refer OneFS best practice guide
3.1 CPU
3.1.1 CPU BIOS settings In compute nodes’ BIOS, set CPU to Performance mode for best performance.
3.1.2 CPUfreq governor Following CPUfreq governor types are available in RHEL 7, check other OS-specific governors if you are not using CentOS or RHEL 7.
Governor Type Description cpufreq_performance • Forces the CPU to use the highest possible clock frequency. It is meant for heavy workloads. This is best fit for interactive workloads.
Cpufreq_powersave • Forces the CPU to stay at the lowest clock frequency possible.
Cpufreq_ondemand • This allows CPU frequency to scale to maximum under heavy load, but drops down to lowest frequency under light or no load. This is the ideal governor and subject to appropriate testing, and can be used (as it will reduce power consumption under low load/idle conditions)
Cpufreq_userspace • This allows user space programs to set the frequency. This is used with the cpuspeed daemon.
Cpufreq_conservative • Similar to the cpufreq_ondemand but it switches frequencies more gradually
Find the appropriate kernel modules for available on the system. And then use modprobe to add the driver needed.
# modprobe cpufreq_performance
Once a particular governor is loaded into the kernel, it can be enabled by using –
# cpupower frequency-set –governor cpufreq_performance
16 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
3.2 Memory
3.2.1 Minimize anonymous page faults Minimize anonymous page faults by setting vm.swappiness = 1 , thereby freeing from page cache before “Swapping” application pages (this reduces the OOM-killer invocation)
Edit /etc/sysctl.conf in editor of choice and add following line –
vm.swappiness=1
After that, run –
# sysctl –p # sysctl –a|grep "vm.swappiness"
3.2.2 Disable transparent huge-page compaction
echo "never" > /sys/kernel/mm/redhat_transparent_hugepage/enabled
3.2.3 Disable transparent huge-page defragmentation
echo "never" > /sys/kernel/mm/redhat_transparent_hugepage/defrag
Add these commands to /etc/rc.local to ensure that transparent hugepage compaction and defragmentation remain disabled across reboots.
3.3 Network The backend network between compute nodes and Isilon should be 40Gbps with Jumbo Frames set (MTU=9162) for the NICs and the switch ports.
3.3.1 OneFS TCP tuning The default TCP stack of OneFS needs some tuning for Hadoop and 40GbE connectivity. The tuning needs to be done within the CLI directly on Isilon. A tcptune.sh script is available at Github.
Simply run sh ./tcptune.sh Max to make the changes, an example script run is shown below:
Before changes:
isilon# sh ./tcptune.sh Max Tuning TCP stack to Max TCP sysctls before... kern.ipc.maxsockbuf=2097152 net.inet.tcp.sendbuf_max=2097152 net.inet.tcp.recvbuf_max=2097152 net.inet.tcp.sendbuf_inc=8192 net.inet.tcp.recvbuf_inc=16384 net.inet.tcp.sendspace=131072 net.inet.tcp.recvspace=131072
17 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
efs.bam.coalescer.insert_hwm=209715200 efs.bam.coalescer.insert_lwm=178257920
After Changes:
Apply tuning... Value set successfully Value set successfully Value set successfully Value set successfully Value set successfully Value set successfully Value set successfully Value set successfully
TCP sysctls after... kern.ipc.maxsockbuf=104857600 net.inet.tcp.sendbuf_max=52428800 net.inet.tcp.recvbuf_max=52428800 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_inc=32768 net.inet.tcp.sendspace=26214400 net.inet.tcp.recvspace=26214400 efs.bam.coalescer.insert_hwm=209715200 efs.bam.coalescer.insert_lwm=178257920 net.inet.tcp.mssdflt=8948
3.3.2 Compute nodes network tuning Following parameters are to be added in /etc/sysctl.conf.
Disable TCP timestamps to improve CPU utilization (this is an optional parameter and will depend on your NIC vendor) –
net.ipv4.tcp_timestamps=0
Enable TCP sacks to improve throughput –
net.ipv4.tcp_sack=1
Increase the maximum length of processor input queues –
net.core.netdev_max_backlog=250000
Increase the TCP max and default buffer sizes using setsockopt() –
net.core.rmem_max=4194304 net.core.wmem_max=4194304 net.core.rmem_default=4194304 net.core_wmem_default=4194304 net.core.optmem_max=4194304
Increase memory thresholds to prevent packet dropping –
18 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
net.ipv4.tcp_rmem=4096 87380 4194304 net.ipv4.tcp_wmem=4096 65536 4194304
Set the socket buffer to be divided evenly between TCP window size and application buffer –
net.ipv4.tcp_adv_win_scale=1
3.3.3 Verify NIC advanced features Check what features are available with your NIC by using “ethtool” –
$ sudo ethtool -k
There are various offload capabilities modern NICs (and especially high-performance NICs) have and it is always recommended to enable them.
A few features such as tcp-segmentation-offload (TSO), scatter-gather (SG) and generic-segmentation- offload (GSO) are good features to enable (if not enabled by default).
3.3.4 NIC ring buffer configurations Check existing ring buffer sizes by running.
$ ethtool -g
After checking the pre-set maximum values, and the current hardware settings, the following command can be used to resize the ring buffers –
# ethtool –G
Or
# ethtool –G
NOTE: The ring buffer sizes will depend on the network topology to a certain degree and might need to be tuned depending on the nature of the workload. We have seen for 10G NICs, setting the RX and TX buffers to maximum is reasonable. This will of course have to be tuned based on the network architecture and type of traffic.
3.4 Storage
3.4.1 Disk/FS mount options Disable “atime” from the data disks (and root fs for that matter) - use “noatime” option during mounting the FS.
In the /etc/fstab file, ensure that the appropriate file system have the “noatime” mount option specified -
LABEL=ROOT / xfs noatime 0 0
19 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
Create separate mount points for separate disk drives, and provision all of them for the scratch space, for example “yarn.nodemanager.local-dirs” is a comma separated list of local-directories that one can configure to be used for copying files during localization. The idea behind allowing multiple directories is to use multiple disks for localization – it helps both fail-over (one/few disk(s) going bad does not affect all containers) and load balancing (no single disk is bottlenecked with writes). Thus, individual directories should be configured if possible, on different local disks.
3.4.2 FS Creation Options FS creation - enable journal mode, reduce superuser block reservation from 5% to 1% for root (using the -m1 option), the sparse_super,dir_index,extent options (minimize number of super block backups and use b-tree indexes for directory trees, extent-based allocations).
# mkfs.xfs –t xfs –m1 –O sparse_super,dir_index,extent,has_journal /dev/sdb1
20 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523 Technical support and resources
A Technical support and resources
Dell.com/support is focused on meeting customer needs with proven services and support.
A.1 Related resources
1. Apache Ambari Install Guide 2. HDP 3.1.5 Install Guide 3. OneFS 8.2.0 HDFS Reference Guide 4. OneFS 8.2.0 Web Admin Guide 5. Using CDH with Isilon Storage 6. Using Hadoop with OneFS info hub 7. Dell EMC Isilon best practice guide for Hadoop data storage
21 Dell EMC Isilon and Cloudera Reference Architecture and Performance Results | H18523