Cloud Storage: Key to Future of IT Infrastructure

Home , Cloud computing, Cloud storage, Nirvanix

Anil Vasudeva President & Chief Analyst [email protected] 408-268-0800 Agenda

(1) The Need for Cloud Storage Data Centers Infrastructure IT Roadmap to Clouds Cloud Computing Ecosystem & Architectural Stack Storage Usage in Data Centers & in Cloud Mega-Centers Applications mapped by Key Workload & Storage Metrics (2) Cloud Infrastructure Technology Enablers Virtualization: Key to Cloud Storage Infrastructure SSDs: A New SCM Filling Price Performance Gaps Networks: Fast March to 100Gb/sec Hadoop: Meeting Big Data/BI/Data Base Paradigm (3) Drivers for Cloud Storage Adoption Storage Efficiencies and Costs Reduction Tier 4 Storage for Data Protection Real-Time Online Analytics Technology Innovations HW, SW 2005-2015 (3) Cloud Segments: Challenges & Solutions Adopting Public/Private/Hybrid Clouds: Pros & Cons (4) Cloud Storage Requirements Adopting Public/Private/Hybrid Clouds: Pros & Cons Best Practices: Getting Ready for the Cloud (5) Key Take-Aways Data Centers & Cloud Infrastructure

Public CloudCenter© Enterprise VZ Data Center On-Premise Cloud

Vertical Clouds Servers VPN Switches: Layer 4-7, IaaS, PaaS Layer 2, 10GbE, FC Stg SaaS

Supplier/Partners ISP

ISP Internet ISP FC/ IPSANs ISP Core Optical ISP ISP Edge Caching, Proxy, Database Servers, Remote/Branch Office ISP FW, SSL, IDS, DNS, LB, Middleware, Data Web Servers Mgmt

Tier-1 Application Servers Tier-3 Edge Apps HA, File/Print, ERP, Data Base Web 2.0 SCM, CRM Servers Servers Social Ntwks. Facebook, Tier-2 Apps Cellular Twitter, YouTube… Directory Security Policy Management Wireless Cable/DSL… Middleware Platform Home Networks

Analytics – BI IT Industry Predictive Analytics - Unstructured Data Roadmap From Dashboards Visualization to Prediction Engines using Big Data. IMEX Research Cloudization On-Premises > Private Clouds > Public Clouds DC to Cloud-Aware Infrast. & Apps. Cascade migration to SPs/Public Clouds. Automation Automatically Maintains Application SLAs (Self-Configuration, Self-Healing©IMEX, Self-Acctg. Charges etc) Virtualization Pools Resources. Provisions, Optimizes, Monitors Shuffles Resources to optimize Delivery of various Business Services Integration/Consolidation

Integrate Physical Infrast./Blades to meet CAPSIMS ®IMEX Cost, Availability, Performance, Scalability, Inter-operability, Manageability & Security Standardization

Standard IT Infrastructure- Volume Economics HW/Syst SW (Servers, Storage, Networking Devices, System Software (OS, MW & Data Mgmt SW) Cloud Computing Ecosystem

Cloud Services Providers Examples Service Public - BT, Telstra, T-Systems France Telecom Public – Mutitenancy,OnDemand Private – Providers Private - On Premises, Enterprise Hybrid – IBM/Cloudburst, Hybrid – Interoperable P2P

Software-as-a-Service Examples eMail - Yahoo!,Google… SaaS - Servers, Network, Storage Collaboration - Facebook,Twitter … - Management, Reporting Bus.Apps - SalesForce, GoogleApps, Intuit…

Examples Platform Tools & Services Amazon EC2 PaaS - Deploy developed platforms Force.com ready for Application SW on Navitaire Cloud Aware Infrastructure

Examples Infrastructure HW & Amazon S3 IaaS Services Nirvanix - Servers, Network, Storage - Management, Reporting Virtualized Cloud Infrastructure

Cloud Computing Private Cloud Hybrid Public Cloud Service Enterprise Cloud Providers

SaaS Applications

SaaS

SLA SLA SLA SLA App App SLA App App App

Platform Tools & Services

….. …..

EJB

.Net

PHP Ruby

PaaS Python

Operating Systems Management

Virtualization IaaS Resources (Servers, Storage, Networks)

Application’s SLA dictates the Resources Required to meet specific requirements of Availability, Performance, Cost, Security, Manageability etc. Data Storage Usage – In Corporate Data Centers

I/O Access Frequency vs. Percent of Corporate Data 100% 95%

75% Disk Tape 65% Arrays Libraries • Back Up Data • Tables

• Archived Data • Indices • Offsite DataVault Cache • Hot Data • Logs • Journals • Temp Tables

• Hot Tables

SSD % of I/O Accesses I/O of %

1% 2% 10% 50% 100% % of Corporate Data

• Temporary Storage – Test & Dev Data, Excess Storage from Data Centers • Internet Web Content – Web 2.0 user-generated content, Digital Catalogs, Public Files, iPhone App Content • Data Archiving & Records Retention – Regulatory compliance, Corporate governance, Audits & searches, Legal, Intellectual Property, Risk Management • Data Back Up & Restore – Offsite data protection, Historical Versioning, Online Access, Automated Retrieval • Intranet File Storage – Employee Portals, Digital Libraries, Document Collaboration, Training Media, File Shares, Historical Versioning, Online Access, Automated Retrieval

NextGen Data Storage Usage in Cloud

I/O Access Frequency vs. Percent of Corporate Data

95% 75% Cloud 65% FCoE/ Storage

SAS SATA

Arrays • Back Up Data

SSD • Tables • Archived Data • Logs • Indices • Offsite DataVault • Journals • Hot Data • Temp Tables • Hot Tables

% of I/O Accesses I/O of %

1% 2% 10% 50% 100% % of Corporate Data

1000 K OLTP Transaction Processing eCommerce

100 K

- Business (RAID - 0, 3) (RAID - 1, 5, 6) Data Intelligence Warehousing 10K (*Latency OLAP

IOPS* IOPS* Scientific Computing 1K HPC Imaging TP 100 Audio Web 2.0 HPC Video

Data Protection Storage Efficiency DRAM Back Up/Archive/DR Auto Tiered Storage Flash RAID – 0,1,5,6,10 Storage Virtualization SSD Virtual Tape Thin Provisioning Performance Replication Deduplication Disk MAID Capacity Disk

Tape

Auto Tiering System using SSDs Data Class (Tiers 0,1,2,3) Storage Media Type (Flash/Disk/Tape) Policy Engines (Workload Mgmt) Transparent Migration (Data Placement) File Virtualization (Uninterrupted App.Opns.in Migration)

• A single server 1.5x larger than standard 2-way server will handle consolidated load of 6 servers. • VZ manages the workloads + important apps get the compute resources they need automatically w/o operator intervention. • Physical consolidation of 15-20:1 is easily possible • Reasonable goal for VZ x86 servers – 40-50% utilization on large systems (>4way), rising as dual/quad core processors becomes available • Savings result in Real Estate, Power & Cooling, High Availability, Hardware, Management

Source: Dan Olds & IMEX Research 2009 TCO Savings with Virtualization

995 Pre-Virtualization (VZ) Servers  78 VZ

$16,000 Servers

0800 - Provisioning Downtime

(408)268 $12,000

Disaster Recovery DC Real Estate Power & Cooling $8,000 Network SAN

Hardware

Cost over Cost over 3 years imex@ imexresearch.comimex@

$4,000

VZ SW & $- For TCO Analysis,EM: TCOFor Support w/o VZ w VZ Cloud Storage Requirements

Tape

Auto Tiering System using SSDs Data Class (Tiers 0,1,2,3) Storage Media Type (Flash/Disk/Tape) Policy Engines (Workload Mgmt) Transparent Migration (Data Placement)

Price $/GB CPU SDRAM

DRAM getting DRAM NOR Faster (to feed faster CPUs) & Larger (to feed Multi-cores & NAND Multi-VMs from Virtualization) PCIe SCM SSD

HDD SSD segmenting into SATA  PCIe SSD Cache Tape SSD - as backend to DRAM &  SATA SSD HDD becoming - as front end to HDD Cheaper, not faster

16 SSD in SAN Storage - Perf & TCO

SAN StorageSAN Performance – Performance & TCO Improvements using SSD Improvements using SSD SAN TCO using HDD vs. Hybrid Storage 300 10 250 IOPS $/IOPS 9 250 Improvement Improvement 8 200 475% 800% 7 200 145 6 150 HDD-FC 0

150 5 IOPS

$/IOP HDD- 36 Cost $K 4 100 SATA 100 0 3 SSD 64 2 50 50 75 1 RackSpace 28 0 0 14.2 Pwr/Cool 0 5.2 FC-HDD Only SSD/SATA-HDD HDD Only HDD/SSD

Performance (IOPS) $/IOP Power & Cooling RackSpace SSDs HDD SATA HDD FC

17 SSD Challenges & Solutions: Goals & Best Practices

Concerned about SSD Adoption in your Enterprise ? 20%

Be aware of Tools & Best Practices … And you should be OK !! 15% Best Practices 10% – By leveraging Error Avoidance Algorithms, and Best Practices of Verification Testing, to keep total functional failure rate <=3% (with defects and 5% wear-outs issues combined) – In practice, endurance ratings are likely to be 0% % of Drives Failing (AFR Failing % of (AFR Drives %) significantly higher than typical use, so data errors 0 1 2 3 4 5 JEDEC and failures will be even less. Years of Use SSD – Capacity Over-provisioning will provide large Std. increases in random performance and endurance. <=3% – Select SSD based on confirmed EVT Ratings – Use MLC within requirements of Endurance Limits

Using Best-of-Breed Controllers to achieve <=3% AFR and JEDEC Endurance Verification Testing should allow Enterprise Capabile SSDs

Source: Intel IDF’10 & IMEX Research SSD Industry Report 2011 ©IMEX 2010-12 I/O Forensics for Auto Storage-Tiering

Storage-Tiered Virtualization Storage-Tiering at LBA/Sub-LUN Level Physical Storage Logical Volume

SSDs Hot Data Arrays

Cold Data HDDs Arrays

LBA Monitoring and Tiered Placement • Every workload has unique I/O access signature • Historical performance data for a LUN can identify performance skews & hot data regions Automatic by LBAs Migration (Policy Based)

19 Source: IBM & IMEX Research SSD Industry Report 2011 ©IMEX 2010-12 AutoSmart Storage-Tiering SW: Enhancing Database Throughput

• DB Throughput Optimization • Productivity (Response Time) Improvement – Every workload has unique I/O access – Using automated reallocation of hot spot data to SSDs signature and historical behavior (typically 5-10% of total data), significant performance – identify hot “database objects” and smartly improvements is achieved : placed in the right tier. • Response time reduction of around 70+% or – Scalable Throughput Improvement - 300% • IOPS increase of 200% for any I/O intensive loads – Substantial IO Bound Transaction Response – Verticals benefitting from Online Transactions : • Airlines Reservations,  Investment Banking  Wall St. Stock time Improvement - 45%-75% Transactions  Financial Institutions Hedge Funds etc. plus Low Latency seeking HPC Clustered Systems etc.

Source: IBM & IMEX Research SSD Industry Report 2011 ©IMEX 2010-12 20 Big Data – Driving New Storage

Data Stored by Large US Enterprises

Big Data Cloud Storage Potential Big Data Storage PotentialData Stored by Large US Enterprises

Data Stored byStored Large US Data Enterprise by Industry Stored Data TB/Firm (in US 2009 PB) (>1K Employees US)

Discrete Manafacturing 14% 231 Government 12% 150 Communications and Media 10% 825 Process manufacturing 10% 1,507 Banking 9% 536 Health Care Providers 6% 801 Securities & Investment Srvcs 6% 870 Professional Services 6% 319 Retail 5% 697 Education 4% 278 Insurance 4% 3,866 Transportation 3% 370 Wholesale 3% 1,931 Utilities 3% 831 Resource Industries 2% 1,792 Consumer and Recreational 2% 1,312 Services Construction 1% 967 Harnessing Big Data for Business Insights

Data Big Data Business Sources Infrastructure Insights

Information is at the center of Majority of data growth is being 80% of world’s data is unstructured New Wave of opportunity driven by unstructured data driven by rise in Mobility devices, and billions of large objects collaboration machine generated data. Type of Data Organizations Analyze Types of Data Organizations Analyze 59% Customer/member data 68% 44% Transactional data from applications 68% 69% Application Logs 37% 64% Other Types of Event Data 23% 41% Network Monitoring/Network Traffic 33% 33% Online Retail Transactions 34% 51% Other Log Files 26% 28% Call Data Records 32% 46% Web Logs 21% 36% Text data from social media and online 15% 36% Search logs 11% Trade/quote data 18% 15% Hadoop 18% Intelligence/defense data 11% Non Hadoop 21% Multimedia (audio/video/images) 9% 8% Weather 3% 3% Smartmeter data 6% 3% Other (please specify) 5% NextGen Drivers: Big Data Cloud

Smart Mobile Commercial Bio/Health Predictive Bus. Entertainment-

Devices Visualization Analytics Intelligence On Demand

24 Data: IMEX Research & Panasas & Research IMEX Data:

Instant On Boot Ups Rendering (Texture & Polygons) Data Warehousing Most Accessed Videos Rugged, Low Power Very Read Intensive, Small Block I/O Random IO, High OLTPM Very Read Intensive 1GB/s, __ms 10 GB/s, __ms 10 GB/s, __ms 4 GB/s, __ms Big Data – Architectural Goals

Meet Requirements of V3 Meet Enterprise Criterion

Big Data Platform

Analyze Data in Native Format Rise of Big Data Analytics Demand for Real Time Analytics Technology Innovations - HW Technology Innovations: DB SW

Tech Innovation 1985 1990 1995 2000 2005 2010 2015

OLTP Transactions OpenSource / Rows Locking Optimizer Parallel Query Clustering XML Grid DB SW Hadoop

OLAP- Analytics Materialized Bit Mapped Indexing Partitioning Columnar In-Memory Query Binding DB SW View Index

Multi-core/ Hardware 32 bit SMP NUMA 64 bit Flash MPP Blades

Columnar Big Data Multi-core In-Memory MPP Visualization

OLTP Database Innovation Progress Big Data: Analytics DB Technology Impact

100000 10000 100,000 100000 10000 10000 1000 10,000 $/GB 1000 1,000 DW Size TB 1000 $/TPMc 100 100 100 10 100 TPMc/Processor 10

$ / TPMc / $ 10 1 10 1 0.1 1 0.01

1 0.1 Processor / TPMc .10 0.001 .01 0.1 0.01 0 0.0001 1985 1990 1995 2000 2005 2010 2015 1985 1990 1995 2000 2005 2010 2015 Big Data – Product Segments

PB Data Set Size TB GB Transaction Machine Data Structure Unstructured Other Transaction Access/Use Search Big Data - Analytics Appliance Product Parallel Metrics Cluster < 1K Processing Cluster > 1K In-Memory Memory Flash Columnar DB Technique Zero Sharing No SQL Data Text Image Cataloging Audio SW Video Big Data Ecosystem Big Data Stack

BI Framework - Interoperable with Enterprise Data Warehousing

EDW BI APPLICATIONS (Query, Analytics, Reporting, Statistics)

Big Data Big Data Orchestration Framework

(Connectors) HBase Avro Flume ZooKeeper

Big Data Access Framework Backup &Recovery

Management

Pig Hive Sqoop Security

Network

Big Data Storage Framework Big Data Processing Framework

(HDFS) (MapReduce)

Hypervisor/VMs Operating System Servers Cloud Types - Benefits & Risks

Private Cloud Public Cloud Hybrid Cloud (On Premises) (Remote) (Co-Oper)

. IT acts as provider of services . Owned and delivered by 3rd Party . Agreed upon Cooperation between Charact- . Available to Internal Users only . Application on User Side & App Services OnPremises user owned by User and . Bidirectional Scalability and Resources on Cloud Srvcs Provider side Cloud Services Provider (CSP) on eristics . Reservationless Auto Allocation of . Cloud Provider sells Srvcs to anyone, Cloud Side Resources . Data stored in Shared Facility serving multi- . Onsite Gateway Appliance with . Pay/Use – Metering Mechanism tenants - impact on Perf., BW, & abstraction layer between usr . Application & SW Agnostic Availability of Apps served. location & public cloud site of CSP

. Instant Provisioning Agility . Lower Costs using Shared Offsite Stg . Improves Performance. by serving . Full Physical Control & Security + Instant Provisioning Hot I/Os On-Premise User’s access to Infrastructure . Infinite Scalability . Deduped Data sent to Cloud > Benefits . Customizable Srvcs & Solns to meet . Efficient Data Sharing by user particularly if Lowers reqd. BW & Monthly User’s specific Needs info shared by users dispersed in multi- Networking Costs . Predictable & Lower Costs solutions locations . Only Encrypted Data Sent > ensures for large/midrange sites . Storage Costs CAPEX >> OPEX security for MC Apps • Demands extra capacity be available for unforeseen demand in . SLA managed as outsourced contract provisioning and management . Full understanding of Performance, Availability, Costs required per App before • Building Private Cloud Infrast. migration difficult, requires new expertise . Shared Infrastructure awareness for Risks • New Cos.now moving as Private sensitive data despite encryption Cloud Infras. Builders vs CSPs . Infrastructures Failures/Hacked • Requires Monitoring, DR Solns, SensitiveData Exposure visible in Press Change Mgmt, Testing, SLA/ Charge . Cloud Provider can abruptly exit Biz Back mechanisms etc. . Lower HA than achievable in Pvt Cloud Cloud Storage Requirements

Expect more from your storage . Hyper-efficient Storage . Storage that analyzes, adapts, and improves performance . Cloud ready storage, with features you can use today Technology & Best Practices enable storage efficiency . Real-time Compression … up to 80% less space . Auto Tiering … 3x more performance with just 2% solid-state . Storage Virtualization … up to 30%more utilization . Thin Provisioning … up to 35%more utilization Data Protection/HA: . Fileset level snapshots and writeable file clones . Capability for remote mirror . File replication and file level snapshots for business continuity and DR Manageability: . Unified SAN & NAS (block and file) storage system with a tightly integrated management console . Support for NFS/CIFS/FTP/HTTPS/SCP file protocols . Storage administration with a single user interface . Policy-based management of files with user-defined policies! Cloud Storage – Issues & Solns.

Virtualization (VZ) requires Shared Storage for - VMotion - Storage VMotion - HA/DRS - Fault Tolerance Additional Capacity Consumed for - VZ snapshots, - VM Kernel etc

Following techniques Reduce Storage Costs

Traditional Thin Replication ~ 95% Data Growth Snapshots ~ 75%

Virtual Clones ~80% Thin Provisioning ~30% DeDuplication ~ 25-95% CAPACITY Auto-Tiering 65-95% Storage Storage CostsReduction SAVINGS ~ xx % RAID*DP ~ 40% vs R10 Capacity Requirements Cloud Storage Requirements

Efficient Storage Back Up Automated DR Replication

Thin Provisioning & Others Simplifying the increase capacity utilization Automatic avoidance of disk hotspots Extent-based volume management Efficient cache management at disk system (controller & disk arrays) Optimized performance through automated data placement Dynamic QoS management Private Cloud - Requirements

Public Cloud Storage Private Cloud Storage Service Catalog VZ . Choose pre-defined IT Services/user/dept. App . Define SLA to efficiently meet services

Silos Operational Flexibility Costs Service Self- Private Service Cloud Analytics . Access Resources on demand to Storage . Monitor & Analyze Usage for speed deployment and delivery Charge back . Scale Resource Up/down . Interactively auto-tune . to optimize their usage, performance . Release when not needed Availability with SLA Automation requirements . Automated Provisioning - Moving Data & Processes Seamlessly . Allocation, Self Tuning of Resources to meet Workload Requirements Service Catalog by User/Workload

Service Catalog by Workloads Class Metric Initial Capacity TB Storage Storage Protocol (NFS/SCSI …) Downtime/Year (Max Unplanned) Mts Availability Copy Retention Time Hrs/Yrs RPO Recovery Point RTO Recovery Time Operational Recovery Consistency Recovery Recovery Assurance % Min # Recovery Points RPO Recovery Point RTO Recovery Time Disaster Recovery Consistency Recovery Recovery Assurance % Min # Recovery Points I/O Response Time ms (Max/Avg) Performance IOPs (Min/Assured) Bandwidth MB/sec (Min/Assured) Encryption DLP Security Copy Retention Time Hrs/Yrs Copy Destruction Method -Quick/Secure Erase Authencity Guarantee Cloud Storage - Requirements

Enable Policy Based Efficient File Management Locally - Identify files for backup or replication - Move to right tier of storage hierarchy including cloud - Delete expired or unwanted files Globally Globally - Localize data Locally > Improve file access performance and reduce network costs. - Allow multiple sites to collaborate on information exchange - Virtualize to a single/global namespace > Multiple users can view same files from universal locations Collaboration in Cloud - Local or long distance users sharing a dedicated home file

System but with individual home directory. - Each site/system has own file set at their local cluster. -Data Center has all Home dirs and manages BUs/hsm - Remote sites periodically prefetch (via policy) or pull on demand Data - Data Center maintains BUs & R-kive Copy of data. Center - Each location maintains logs/records for that Geography IT Infrastructure in the Cloud

SLA

• Business Priorities • Cost of IT Ops/Charge Back Methods • Response Time/Availability/Throughput, QoS • Transactions/Sessions/Events/Analysis/Reporting • Business Services Managed & Charged

Cloud IT Services Provider Usage Assets Dept/User Location • Usage Profiles • Cloud Services Hosting • Users/Services/Workloads • Platform – OS/Processors/#/Speed/Type • Applications (OLTP/BI/HPC/Data Streaming) • Pooled Infrastructure Resources by • Execution: Rules Driven, Adaptive Provisioning Application Metrics • Services Abstraction, Adaptive Provisioning • Pooled Capacity Provisioning: Processing, Bandwidth, Storage, Repository

For copy of case study on how a major financial institution implemented Cloud Infrastructure email [email protected] Key Takeaways

• Cloud Storage is increasing being adopted – IT-as-a-Service is being delivered through a virtualized, Shared IT infrastructure, automation and Service Efficiencies using Private and Hybrid Cloud Infrastructure – Savings through Storage Efficiency using automation that is being integrated – Due diligence in careful planning, a three year roadmap, selection and testing of vendors/products required before starting migration from existing DC operations • IT Infrastructure to meet needs of your Business Applications/SLA – Let your business needs determine your Cloud Strategy. – Identify and define specific SLA metrics per user/dept/division. – Strategize for Long Term but take incremental approach to implementing Clouds – Identify and screen promising Cloud Service Providers – their Enterprise Class, Security and Compliance adherence, Change Controls, Referenced High Availability & Performance Results. • Be aware of technology and best practices available – Engage the expertise of consultants and system integrators to validate your strategy and implementation plans