DOCSLIB.ORG

MEETING BIG DATA CHALLENGES with EMC ISILON STORAGE SYSTEMS Anuj Sharma

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
MEETING BIG DATA CHALLENGES with EMC ISILON STORAGE SYSTEMS Anuj Sharma MEETING BIG DATA CHALLENGES WITH EMC ISILON STORAGE SYSTEMS Anuj Sharma Contents Abstract...................................................................................................................................... 3 Big Data Challenges and EMC Isilon Storage Systems .............................................................. 6 Big Data Value-Add To Business ............................................................................................... 9 OneFS Architecture ..................................................................................................................11 Which EMC Isilon Cluster to choose? .......................................................................................16 EMC Isilon Cluster Networking Best Practices ..........................................................................17 EMC Isilon Smart Connect Internals .........................................................................................20 SmartConnect Architecture Example ........................................................................................27 EMC Isilon Smart Quotas Internals ...........................................................................................28 EMC Isilon and vSphere Integration Best Practices ..................................................................31 EMC Isilon SyncIQ Architecture and Tips ..................................................................................36 EMC Isilon NDMP Backup Configuration for EMC NetWorker ...................................................37 Cluster Performance Tuning .....................................................................................................41 EMC Isilon Cluster Maintenance ...............................................................................................43 References ...............................................................................................................................47 Glossary....................................................................................................................................48 Figures Figure 2: Big Data Sources ........................................................................................................ 5 Figure 3: Isilon Node Types ....................................................................................................... 8 Figure 5: Big Data Enabled Property and Casualty Insurance Policy Premium Factors .............. 9 Figure 6: OneFS vs. Traditional File Systems ...........................................................................11 Figure 7: OneFS vs. Traditional File Systems ...........................................................................12 Figure 8: Isilon Cluster ..............................................................................................................13 Figure 9: OneFS Protection.......................................................................................................15 Figure 10: 10GigE Networking with Accelerator Node ...............................................................19 Figure 11: Redundant Internal Network Topology .....................................................................20 Figure 12: SmartConnect Communication .................................................................................22 Figure 14: SmartConnect Configuration ....................................................................................27 Figure 15: Optimizing Isilon NFS for VM I/O Operations ...........................................................33 Figure 16: Isilon NFS Architecture .............................................................................................34 Figure 17: Isilon iSCSI Architecture ...........................................................................................35 Figure 19: Direct NMDP Method ...............................................................................................37 Figure 20: Remote NDMP Model ..............................................................................................38 Disclaimer: The views, processes, or methodologies published in this article are those of the author. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies. 2012 EMC Proven Professional Knowledge Sharing 2 Abstract What is Big Data? Big Data does not refer to a specific type of data; every kind of unstructured data can be considered big data when a single file size is in terabytes. Digital data is growing at an exponential rate today, and “big data” is the new buzzword in IT circles. According to International Data Corporation (IDC), the amount of digital data created and replicated will surpass 1.8 Zettabytes (1.8 trillion GB) in 2011, having grown by a factor of nine in just five years. The information we deal with today is very different from the information that we used to deal with 20-30 years ago. Chip manufacturers render terabyte files, oil and gas exploration companies deal with terabytes of data to be analyzed, advancements in healthcare has led to the creation of high-definition 4-D imaging files ranging up to terabytes, and NASA deals with a large number of files with sizes in terabytes. Social networking sites and online communities generate data in huge numbers. It seems no industry is safe from massive data growth and the storage implications are profound. Figure 1: Data Growth The massive amount of rich unstructured file data generated by richer file formats and Internet Era computing is creating a demand for new and innovative scale-out file storage solutions to economically scale bandwidth and performance to previously unheard of capacities. 2012 EMC Proven Professional Knowledge Sharing 3 I have seen many organizations relying on scale-up storage for Big Data storage and analytics. However, in the long run, companies are bound to encounter performance and storage problems using scale-up storage systems for Big Data storage. Scale-up storage are monolithic storage systems where lots of storage sits behind one or two file server heads and is designed to scale to multi-TB range behind those file server heads. Once the storage and performance limitations are reached, a new monolithic storage system must be added. As the existing file systems residing on the previous scale-up storage system cannot be expanded to leverage the storage of the new scale-up storage system, a new file system needs to be managed, even if there is only the need to add minimal incremental storage capacity. This is one of the problems that enterprises run into while dealing with Big Data using scale-up storage systems. A single file system is limited to TB’s in the case of scale-up storage systems and file system migration is often a painful exercise and requires downtime. Also, traditional analysis tools cannot be used to analyze Big Data. Data needs to be mined real-time and results need to be published. For example, a retail store can see instantaneously which stores are most profitable, which item is in demand, the consumer choices per region, and so on. Big Data analysis is a critical factor that plays a significant role in future business decisions of the organization. Parallel processing is required to mine data of such huge volumes simultaneously. Consequently, scale-out storage systems are the best candidates for high parallel processing power. Scale-out storage architectures are significantly different than monolithic scale-up storage architectures (e.g. traditional NAS or SAN systems) that were developed to meet distributed computing needs. “Scale-out NAS” are systems designed from the ground up for economically dynamic scale and for supporting extremely high bandwidth applications dealing with multi- terabyte files often referred to as Big Data. The EMC Isilon® storage system is the world leader in the scale-out NAS category. 2012 EMC Proven Professional Knowledge Sharing 4 Figure 2: Big Data Sources In this article we will discuss why scale-up storage is not able to meet the performance, cost, and storage requirements of Big Data efficiently and how scale-out storage successfully meets all the requirements of Big Data. We also discuss an example of how Big Data can add value to business. Additionally, the following areas will be covered: • Big Data Challenges • EMC Isilon Storage Systems and OneFS® Architecture • EMC Isilon Storage Systems Features such as SyncIQ, SmartConnect, SmartQuotas • Best practices for the Implementation of EMC Isilon Storage Clustered Systems • Best Practices for VSphere 4 integration with EMC Isilon Storage Systems • Cluster Maintenance • EMC Isilon NDMP Backup Configuration And much more … 2012 EMC Proven Professional Knowledge Sharing 5 Big Data Challenges and EMC Isilon Storage Systems • Unstructured Data is being generated at exponential rates The pace of data growth requires storage that can scale on demand. Typically, storage is purchased with a view on future or peak requirements. Most often, we end up spending more as the cost of the equipment declines over time. EMC Isilon provides the benefit of adding the storage on demand instead of buying at once; start with minimal nodes and then scale out over time. • Seismic applications, NASA satellite imaging, and high performance video rendering applications require storage that support huge IOPS and data transfer throughput EMC Isilon has a scale-out architecture; whenever a node is added, storage and compute is also added. Hence, compute increases linearly
Load more

Recommended publications
  • RECOVER Recover Systems Directly from Networker Backups
    NBMR RECOVER Recover systems directly from NetWorker backups With Cristie NBMR you can protect critical servers from the consequences of failure or disaster by recovering them directly from your EMC NetWorker backups. Recover your protected servers, including; operating systems, applications, user configuration and data to any point in time provided by NetWorker. Highlights Features Benefits 9 Leverage existing NetWorker 9 Saves storage space, network Integrated system backups without extra infrastructure, bandwidth, processing power and recovery or management management resources Point in time 9 Recover to any point in time 9 Recoveries can be rolled back to a recoveries provided by the NetWorker backups time before the disaster occurred 9 Supports Windows, Linux, AIX and Support for multiple 9 Most systems can be protected Solaris and dissimilar hardware, OS and cloud using NBMR virtual and cloud environments 9 Recover entire workloads including; 9 The recovered machine does not Recovered system OS, applications, data and need to be calibrated and tuned to identical configuration information run as it did before 9 Improve recovery time objectives 9 Meet aggressive service level by restoring your server in as little as Fast recovery times agreements recovery time and point ten minutes. Multiple servers can be objectives recovered simultaneously Flexible recovery 9 Recover to a range of system 9 Flexibility to suit most environments targets recovery targets Remote 9 Manage multiple, simultaneous 9 Easy, secure, mobile access management system recoveries on site, or remotely 9 NBMR is re-sold globally by EMC Recommended 9 NBMR is a recommended recovery and its channel partners as a solution solution for NetWorker recommended recovery solution for NetWorker Support is very cooperative and give a timely response to any technical issues raised with them.
    [Show full text]
  • Networker Data Domain Integration Guide
    PROTECTING IT WITH NETWORKER Aaron Kleinsmith Principal Technical Instructor EMC [email protected] Table of Contents Protecting the Modern Information Technology Environment .......................................... 5 NetWorker Modules......................................................................................................... 7 Diagram 1: NetWorker Module Backups .................................................................. 7 NetWorker Module for Microsoft .................................................................................. 8 NetWorker Module for Databases and Applications ..................................................... 8 Other NetWorker Modules ........................................................................................... 8 Configuring the NetWorker Modules ............................................................................ 9 NetWorker Snapshot Management and Block Based Backup ......................................... 9 When would these different snapshot technologies be used? ...................................... 9 NetWorker Snapshot Management for NAS .................................................................. 10 NetWorker and Windows Bare Metal Recovery ............................................................. 10 Data Domain ................................................................................................................. 12 Data Domain CIFS and NFS ..................................................................................... 12
    [Show full text]
  • Commvault Backup Appliance with Netapp a Powerful Combination of Commvault Simpana® Software and Netapp E-Series Storage
    CommVault Backup Appliance with NetApp A powerful combination of CommVault Simpana® software and NetApp E-Series storage Two leaders in data protection software and hardware have partnered to bring you a best-of-breed solution that closes the scalability, flexibility and manageability gaps in current offerings. You get powerful scalability • Plug-and-play scaling for capacity and compute power • Scale without reconfiguration • Software AND hardware resiliency • Rapid restores across simultaneous jobs You get broad flexibility • Choose a single appliance for smaller environments • Deploy multiple appliances for medium to large environments • Improve performance and resiliency through load balancing You get simple, comprehensive management • Central management, reporting and software deployment • Self-service access • Integration with NetApp E-Series node management COMMVAULT.COM | 888.746.3849 | [email protected] © 2015 COMMVAULT SYSTEMS, INC. ALL RIGHTS RESERVED. Two industry leaders. One truly integrated solution. COMMVAULT EMC AVAMAR/ SYMANTEC NETBACKUP FEATURE BENEFIT SIMPANA DATA DOMAIN APPLIANCES VM LIFECYCLE Manage the complete Y PARTIAL PARTIAL MANAGEMENT VM lifecycle from provisioning and monitoring to backup and replication to VM archiving and retirement DEDUPLICATION Inexpensively delivers Y PARTIAL N RESILIENCY WITH software load balancing PARALLEL and resiliency to DEDUPLICATION complement hardware availability SIMPANA ONEPASS™ Saves time, money Y N N SINGLE PASS BACKUP, and resources FILE ARCHIVING AND REPORTING ENDPOINT
    [Show full text]
  • High Availability and Data Protection with Dell Emc Isilon Scale-Out Nas
    WHITE PAPER HIGH AVAILABILITY AND DATA PROTECTION WITH DELL EMC ISILON SCALE-OUT NAS Abstract This white paper details how Dell EMC Isilon scale-out NAS storage and the Isilon OneFS operating system architecture provide high availability and data protection needed to meet the challenges organizations face as they deal with the deluge of digital content and unstructured data and the growing importance of data protection. July 2018 High1 | High Availability Availability and andData Data Protection Protection with withDell DellEMC EMC Isilon Isilon Scale Scale-out -NoutAS NAS © 2018 Dell Inc. or its subsidiaries. TABLE OF CONTENTS Introduction .......................................................................................................................................... 4 High availability and reliability at the petabyte scale ............................................................................ 4 Areal density and drive rebuild times ................................................................................................................................. 4 Silent data corruption ......................................................................................................................................................... 4 Data protection continuum ................................................................................................................................................. 4 High availability with Isilon ..................................................................................................................
    [Show full text]
  • Dell EMC Vxflex Family Overview Transformation Without Trade-Offs
    Dell EMC VxFlex Family Overview Transformation without trade-offs Abstract This whitepaper provides an overview of the Dell EMC™ VxFlex family, powered by Intel® Xeon® Scalable Processors, that offers “Transformation without Trade-offs,” explaining the compelling performance, resilience and flexibility of VxFlex systems. May 2019 Dell EMC VxFlex Family Overview | H17693 © 2019 Dell Inc. or its subsidiaries Revisions Revisions Date Description May 2019 Initial Publication Acknowledgements This paper was produced by the VxFlex product and technical marketing teams and sponsored by Intel®. Content Owner: Donna Brasure The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying and distribution of any software described in this publication requires an applicable software license. © 2019 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. Dell believes the information in this document is accurate as of its publication date. The information is subject to change without notice. 2|Dell EMC VxFlex Family Overview | H17693 © 2019 Dell Inc. or its susidiaries Table of contents Table of contents Revisions............................................................................................................................................................................
    [Show full text]
  • Backup & Recovery Manager 1.3 User Guide
    EMC Backup & Recovery Manager Version 1.3 User Guide 302-002-003 REV 08 Copyright © 2012-2017 Dell Inc. or its subsidiaries. All rights reserved. Published May 2017 Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS-IS.“ DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners. Published in the USA. EMC Corporation Hopkinton, Massachusetts 01748-9103 1-508-435-1000 In North America 1-866-464-7381 www.EMC.com 2 EMC Backup & Recovery Manager 1.3 User Guide CONTENTS Figures 9 Tables 13 Preface 15 Chapter 1 Getting Started 19 Product architecture.................................................................................. 20 IPv6 support............................................................................................... 21 Requirements..............................................................................................21 Backup & Recovery Manager server software requirements......... 22 Operating system software requirements for the Avamar and NetWorker adaptors.....................................................................
    [Show full text]
  • Dell EMC Not Just Vendor Convergence World
    E-guide In this e-guide Introduction Breaking News The show that originally started as the EMC Enterprise Wizards Conference in 2001 is running May 8 to May 11 in Las Vegas under a new name, Dell EMC Not Just Vendor Convergence World. All-Flash with Substance Well, it's not entirely a new name. Dell World rebranded as Dell EMC World in October 2016, in the wake of Dell's $67 billion acquisition of the world's largest storage vendor. But the May 2017 show is the lineal descendant of the Wizards Head in the Clouds Conference and the long-running EMC World, and marks the seventh straight year the event is being held in Las Vegas. Dell-Emc integration Many of the themes at Dell EMC World 2017 will be familiar for an IT/storage conference these days. You'll hear a lot about hyper-converged infrastructure, cloud storage and flash arrays. More traditional storage topics, such as data protection, disk arrays and storage management will also be covered, although pushed into the background a bit. And because Dell Technologies extends beyond storage, the conference will cover servers, virtualization, DevOps, security, internet of things, software- defined data centers and cloud computing. The speaker list includes Michael Dell; Dell EMC storage leaders David Goulden, Chad Sakac and Jeff Boudreau; VMware CEO Pat Gelsinger; and industry gurus Andy Bechtolsheim and Tim Berners-Lee. E-guide We'll be reporting the latest news from the show on SearchStorage and our In this e-guide other TechTarget sites. To prepare for Dell EMC World 2017, check out what's been happening with the IT giant since the merger, as well as all the news from EMC World 2016.
    [Show full text]
  • Backup Optimization 'Networker Inside'
    BACKUP OPTIMIZATION ‘NETWORKER INSIDE’ Shareef Bassiouny Mohamed Sohail EMC EMC Giovanni Gobbo Senior IT Consultant Table of Contents Executive summary .................................................................................................................... 3 Introduction ................................................................................................................................ 4 Part 1 ......................................................................................................................................... 5 How much Data Storage could be gained? How could it be maximized? ................................ 7 What is the penalty of this gain? ............................................................................................. 8 Classic design example .......................................................................................................... 8 Advantages/disadvantages of the new DD Boost over Fibre Channel (DFC) .........................15 Part II ........................................................................................................................................16 Journey to an optimized backup environment ........................................................................16 The Journey ..........................................................................................................................18 Steps to the solution .................................................................................................................23
    [Show full text]
  • EMC Corporation Fundamental Company
    +44 20 8123 2220 [email protected] EMC Corporation Fundamental Company Report Including Financial, SWOT, Competitors and Industry Analysis https://marketpublishers.com/r/EB2FE1B37EABEN.html Date: September 2021 Pages: 50 Price: US$ 499.00 (Single User License) ID: EB2FE1B37EABEN Abstracts EMC Corporation Fundamental Company Report provides a complete overview of the company’s affairs. All available data is presented in a comprehensive and easily accessed format. The report includes financial and SWOT information, industry analysis, opinions, estimates, plus annual and quarterly forecasts made by stock market experts. The report also enables direct comparison to be made between EMC Corporation and its competitors. This provides our Clients with a clear understanding of EMC Corporation position in the Computers and Electronic Equipment Industry. The report contains detailed information about EMC Corporation that gives an unrivalled in-depth knowledge about internal business-environment of the company: data about the owners, senior executives, locations, subsidiaries, markets, products, and company history. Another part of the report is a SWOT-analysis carried out for EMC Corporation. It involves specifying the objective of the company's business and identifies the different factors that are favorable and unfavorable to achieving that objective. SWOT-analysis helps to understand company’s strengths, weaknesses, opportunities, and possible threats against it. The EMC Corporation financial analysis covers the income statement and ratio trend-charts with balance sheets and cash flows presented on an annual and quarterly basis. The report outlines the main financial ratios pertaining to profitability, margin analysis, asset turnover, credit ratios, and company’s long- EMC Corporation Fundamental Company Report Including Financial, SWOT, Competitors and Industry Analysis +44 20 8123 2220 [email protected] term solvency.
    [Show full text]
  • Emc Networker Delegation Model for Robo Environment
    EMC NETWORKER DELEGATION MODEL FOR ROBO ENVIRONMENT Puneet Goyal Technical Specialist HCL Comnet Table of Contents Introduction ................................................................................................................................ 4 Goal and Objective .................................................................................................................... 6 Audience .................................................................................................................................... 6 What is Delegation Model? ........................................................................................................ 6 The Concept .............................................................................................................................. 7 Understanding the customer’s backup environment and architecture ......................................... 9 Collecting features which can be used in the solution ...............................................................10 Authentication-related feature ................................................................................................10 Backup/Cloning-related feature .............................................................................................11 Recovery-related feature .......................................................................................................13 VMware-related Feature ........................................................................................................16
    [Show full text]
  • SAP HANA Infrastructure Made Simple DELL EMC Platforms and Solutions for SAP
    SAP HANA Infrastructure Made Simple DELL EMC Platforms and Solutions for SAP GLOBAL SPONSORS Agenda • SAP auf DellEMC IT Infrastruktur • Global SAP Center of Excellence • Infrastructure Platforms | Solutions for SAP • Data Protection | Availability • SAP HANA | Big Data (Core DC) • DELL IT SAP Deployment • Summary 3 SAP HANA is Driving Disruption for IT 67% 33% IT budgets New tech operations growth and and cost investment CIO IMPERATIVE Legacy SAP Next generation business suite Balancing risk SAP HANA platform versus innovation platform 4 Planning your journey Common IT Goals for SAP Simplifying IT for SAP Reduce Cost | Increase Productivity Consolidation of "siloed" IT On-premise: managed services Automate maintaining & refreshing complex Off-premise: enterprise cloud and SAP landscapes managed services On-Premise: IaaS Investing in HANA Enterprise Hybrid Cloud with VMware for SAP Make HANA part of standard datacenter architecture with DELL EMC Services Deploy data mart | ERP Accelerators | SOH | BWoH Cloud “Core to edge” platforms to run Data Analytics with SAP HANA and big data Ready bundles Ready nodes Enabling the Real-time Business Ready systems Migrating to SAP S4HANA Deliver SAP as a Service enabling faster time to new insights | business models Infrastructure 5 Building blocks Accelerating your transformation journey Data Analytics: Industry | Departmental | LoB Simplifying IT for SAP Business generated data meets “IoT” On-premise: managed services Off-premise: enterprise cloud and managed services On-Premise: IaaS Enterprise
    [Show full text]
  • EMC Networker 7.2 (And Later) Hardware Compatibility Guide
    EMC NetWorker 7.2 (and Later) Hardware Compatibility Guide Last Updated: July 1, 2008 WARNING: A A device is supported only on the NetWorker release in which it was added or later versions. B New compatibility criteria have been implemented to dramatically expand the supportability matrix, as follows: 1 Criteria #1 - From a NetWorker perspective the Tape Drive is the essential element in qualification not the Tape Library. The Tape Library supports all the OSes supported by a Tape Drive. Criteria #1 applies to Virtual Tape Libraries (VTL), with the exclusion of the VTL libraries with robotics driven through a ADIC DAS or STK ACSLS/ Silo Api, etc. Criteria #1 is NOT applicable to Silos. Silos require qualification with every OS. NOTE: Rule #1 has been applied in this document to all Tape Libraries configured with LTO-1, LTO-2, LTO-3, and LTO-4 Tape Drives with an intent for updating the rest of the document in future updates. The library must be officially qualified on at least one NetWorker platform, after which we will consider the library to be supported on all platforms. It does need to be tested at least once, Example 1: An HP LTO-3 Tape Drive is qualified with Windows 2003 x64. A Tape Library configured with the HP LTO-3 or a VTL emulating an LTO-3 is qualified to work with Windows 2003 x64 (AMD64, EM64T) without testing the Tape/Library combination. 2 Criteria #2 - If a Tape Drive is qualified with the latest release of OS, previous releases of the same OS are supported without qualification tests (see Example 2 below).
    [Show full text]