DOCSLIB.ORG

Systems for Cloud Data Analytics

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Systems for Cloud Data Analytics Peter Boncz SYSTEMS FOR CLOUD DATA ANALYTICS www.cwi.nl/~boncz/badsCloud Data Systems Credits • David DeWitt & Willis Lang (Microsoft) – cloud DW material • Stratis Viglas (Google) – extreme computing course (University Edinburgh) • Marcin Zukowski (Snowflake) • Ippokratis Pandis (Amazon Redshift/Spectrum) • Spark Team – Matei Zaharia, Xiangrui Meng (Stanford), – Ion Stoica, Xifan Pu (UC Berkeley) – Reynold Xin, Alex Behm (Databricks) www.cwi.nl/~boncz/badsCloud Data Systems Is it safe to have enterprise data in the Cloud? 2005: No way! Are you crazy? 2012: Don’t think so... But wait, we store our email where? 2018: Of course! www.cwi.nl/~boncz/badsCloud Data Systems Getting a database in a cloud Hi! I'm a Data Scientist! Hello! I am your account manager at X! I'm looking for a database for our cloud system Sure thing! Let's install our product, DBMS X for you! Awesome! It seems to work! Great. Let me send you that invoice! Just a sec… How much does the storage cost ? Hold on, let me check that Wait, what? And the system is elastic, right? Mommy!!! And I only pay for what I use, right? www.cwi.nl/~boncz/badsCloud Data Systems Traditional DB systems and the cloud • Designed for: –Small, fixed, optimized clusters of machines –Constrained amount of data and resources • Can be delivered via the Cloud –Reduce the complexity of hardware setup, software installation –No elasticity –No cheap storage –Not designed for cloud's poor stability –Not easy to use –Not "always on" –... www.cwi.nl/~boncz/badsCloud Data Systems Data in the Cloud • Data traditional DW systems are built for –Assume predictable, slow-evolving internal data –Complex ETL (extract-transform-load) pipelines and physical tuning –Limited number of users and use-cases –OK to cost $100K per TB • Data in the cloud –Dynamic, external sources: web, logs, mobile devices, sensor data… –ELT instead of ETL (data transformation inside the system) –Often in semi-structured form (JSON, XML, Avro) –Access required by many users, very different use cases –100TBs volume common www.cwi.nl/~boncz/badsCloud Data Systems 10,000 ft. view: Complexity vs Cost Only 2 options 5 years ago! Complexity (deployment & Roll-your-own (RYO) operational) ▪ Buy & install a cluster of servers ▪ Buy, install, & configure software (Vertica, High RYO Asterdata, Greenplum, …) ▪ High complexity ▪ Medium capex and opex Medium Buy an appliance ▪ Teradata, Microsoft APS, Netezza Appliance ▪ High capex, low opex ▪ Low complexity Low ▪ Gold standard for performance Cost (capex + opex) Low Medium High www.cwi.nl/~boncz/badsCloud Data Systems 10,000 ft. view: Complexity vs Cost Complexity Use a SAAS DW in the cloud (deployment & ▪ AWS Redshift, MSFT SQL DW, Snowflake, BigQuery operational) ▪ Low complexity ▪ No capex, low opex High RYO Roll-your-own-Cloud (RYOC) RYOC ▪ Rent a cluster of cloud servers Medium ▪ Buy, install, & configure software (Spark, Hive, Vertica, Asterdata, Greenplum, …) Appliance ▪ Medium to high complexity ▪ Low capex Low CLOUD DW ▪ Medium opex Cost (capex + opex) Low Medium High www.cwi.nl/~boncz/badsCloud Data Systems Scalability and the price of agility Time to make an adjustment Months Appliance Weeks RYO RYOC Minutes CLOUD DW Cost of making an adjustment Low Mediu High m www.cwi.nl/~boncz/badsCloud Data Systems Why Cloud DW? • No CapEx and low OpEx • Go from conception to insight in hours • Rock bottom storage prices (Azure, AWS S3, GFS) • Flexibility to scale up/down compute capacity • Simple upgrade process www.cwi.nl/~boncz/badsCloud Data Systems Parallel Processing in Analytical DBs ▪ Alternative architectures –Shared-memory –Shared-disk/storage –Shared-nothing “The Case for Shared Nothing,” ▪ Partitioned tables Stonebraker, HPTS ‘85 ▪ Partitioned parallelism www.cwi.nl/~boncz/badsCloud Data Systems Shared-Nothing • Commodity servers connected via commodity networking • DB storage is ”strictly local” to each node Node 1 Node 2 Node K Co-located CPU CPU CPU … compute and MEM MEM MEM storage Interconnection Network • Design scales extremely well www.cwi.nl/~boncz/badsCloud Data Systems Shared Disk/Storage • Commodity servers connected to each other and storage using commodity networking Local• DB disks is stored for caching on DB“remote storage” (e.g. a SAN, S3, Azure pages,Storage) temp files, … Node 1 Node 2 Node K Network can limit scaling as CPU CPU CPU it must carry I/O traffic MEM MEM … MEM Storage Area Network www.cwi.nl/~boncz/badsCloud Data Systems Table Partitioning What? Shared Storage Distribute rows of each table across multiple storage devices Why? • spread I/O load • parallel query execution • data lifecycle management How? Hash, Round Robin, Range www.cwi.nl/~boncz/badsCloud Data Systems Shared-Nothing Ex. Application Select Name, Item from Orders O, Customers C Parser where O.CID = C.ID Join can be done “locally” Optimizer Catalogs Execution Coordinator NODEJOIN No data movementNODEJOIN O.CID1 = C.ID O.CID2 = C.ID CID OID Item CID OID Item Example of 602 10 Tivo Orders Table 933 20 Surface 633 21 TV 752 31 iPhone hash partitioned “partitioned parallelism” 633 21 DVD 602 10 Xbox on CID 602 11 iPod 19 51 TV ID Name AmtDue Customers ID Name AmtDue 602 Larry $13K 933 Mary $49K 752 Anne $75K Table 633 Bob $19K hash partitioned 322 Jeff $20K 19 George $83K on ID www.cwi.nl/~boncz/badsCloud Data Systems Shared-Nothing Ex. Application Select Name, Item from Orders O, Customers C Parser where O.CID = C.ID JoinJoin cannot can be be done done “locally” “locally” Optimizer Catalogs Execution Coordinator data movement needed NODEJOIN No data movementNODEJOIN Biggest table (orders)O.CID 1 = C.ID O.CID2 = C.ID needs to be shuffled: all-to- all communications CID OID Item CID OID Item Example of 602 10 Tivo Orders Table 933 20 Surface 633 21 TV 752 31 iPhone hash partitioned “partitioned parallelism” 633 21 DVD 602 10 Xbox on OID 602 11 iPod 19 51 TV ID Name AmtDue Customers ID Name AmtDue 602 Larry $13K 933 Mary $49K 752 Anne $75K Table 633 Bob $19K hash partitioned 322 Jeff $20K 19 George $83K on ID www.cwi.nl/~boncz/badsCloud Data Systems Shared-Storage Ex. Application Select Name, Item from Parser Orders O, Customers C Optimizer where O.CID = C.ID Execution Coordinator NODE NODE 1 2 LAN Both tables are remote CID OID Item CID OID Item 602 10 Tivo Orders Table 933 20 Surface 633 21 TV 752 31 iPhone hash partitioned 633 21 DVD 602 10 Xbox on CID 602 11 iPod 19 51 TV ID Name AmtDue Customers ID Name AmtDue 602 Larry $13K 933 Mary $49K 752 Anne $75K Table 633 Bob $19K hash partitioned 322 Jeff $20K 19 George $83K on ID www.cwi.nl/~boncz/badsCloud Data Systems For 30+ years • Shared-nothing has been “gold standard” –Teradata, Netezza, DB2/PE, SQL Server PDW, ParAccel, Greenplum, … • Simplest design • Excellent scalability • Minimizes data movement –Especially for DBs with a star schema design • The “cloud” has changed the game –shared nothing: 2017 www.cwi.nl/~boncz/badsCloud Data Systems Outline • Part 1: Intro • Part 2: Databases in the Cloud – Amazon Redshift – Snowflake – Microsoft Azure Synapse Analytics – Google BigQuery – Databricks • Part 3: Cloud Research Challenges www.cwi.nl/~boncz/badsCloud Data Systems Amazon (AWS) Redshift • Classic shared-nothing design w. locally attached storage –Engine is ParAccel database system (classic MPP, JIT C++) • Leverages AWS services –EC2 compute instances –S3 storage system –Virtual Private Cloud (VPC) • Leader in market adoption www.cwi.nl/~boncz/badsCloud Data Systems A Redshift Instance Application Single Leader Node LEADER Catalogs NODEOne or more compute One slice/core Memory,nodes storage, (EC2 instance) & data partitioned among slices ) NODE 1 Hash & round-robin table partitioning AmtDue SLICE 1 NODE 1 SLICE 2 SLICE 3 NODE 2 SLICE 4 ID` , Name, , ID ID Name Amt ID Name Amt ID Name Amt ID Name Amt HashPartitionon Customers( 22 www.cwi.nl/~boncz/badsCloud Data Systems Within a slice Two sort options: 1) Compound sort key 2) “Interleaved” sort key (multidimensional sorting) NAM ID E AMT Columns stored in 1MB blocks Min and Max value of each block retained in a “zone” map Rich collection of compression options (RLE, dictionary, gzip, …) www.cwi.nl/~boncz/badsCloud Data Systems Unique Fault Tolerance Approach Catalogs LEADER NODE Each 1MB block gets replicated on a different compute node SLICE NODE 1 SLICE SLICE NODE 2 SLICE 1 2 3 4 ID Name Amt ID Name Amt ID Name Amt ID Name Amt ID Name Amt ID Name Amt ID Name Amt ID Name Amt And also on S3 S3, in turn, triply replicates S3 ID Name Amt ID Name Amt ID Name Amt ID Name Amt each block www.cwi.nl/~boncz/badsCloud Data Systems Handling Node Failures Alternative #1: Node 2 processes load until Node 1 is restored Alternative #2: New node instantiatedCatalogs LEADER NODE Assume Node 1 fails Node 3 processes SLICE NODENODE 1 3 SLICE SLICE NODE 2 SLICE workload using data in S3 1 2 3 4 ID Name Amt ID Name Amt ID Name Amt ID Name Amt ID Name Amt ID Name Amt ID Name Amt ID Name Amt Until local disks are restored S3 ID Name Amt ID Name Amt ID Name Amt ID Name Amt www.cwi.nl/~boncz/badsCloud Data Systems Redshift Summary • Highly successful cloud SAAS DW service • Classic shared-nothing design • Leverages S3 to handle node and disk failures • Key strength: performance through use of local storage • Key weaknesses: compute cannot be scaled independent of storage (and vice versa) www.cwi.nl/~boncz/badsCloud Data Systems Redshift Spectrum www.cwi.nl/~boncz/badsCloud Data Systems Redshift Spectrum Spectrum www.cwi.nl/~boncz/badsCloud Data Systems Outline • Part 1: Intro • Part 2: Databases in the Cloud
Load more

Recommended publications
  • Data Warehouse Fundamentals for Storage Professionals – What You Need to Know EMC Proven Professional Knowledge Sharing 2011
    Data Warehouse Fundamentals for Storage Professionals – What You Need To Know EMC Proven Professional Knowledge Sharing 2011 Bruce Yellin Advisory Technology Consultant EMC Corporation [email protected] Table of Contents Introduction ................................................................................................................................ 3 Data Warehouse Background .................................................................................................... 4 What Is a Data Warehouse? ................................................................................................... 4 Data Mart Defined .................................................................................................................. 8 Schemas and Data Models ..................................................................................................... 9 Data Warehouse Design – Top Down or Bottom Up? ............................................................10 Extract, Transformation and Loading (ETL) ...........................................................................11 Why You Build a Data Warehouse: Business Intelligence .....................................................13 Technology to the Rescue?.......................................................................................................19 RASP - Reliability, Availability, Scalability and Performance ..................................................20 Data Warehouse Backups .....................................................................................................26
    [Show full text]
  • Magic Quadrant for Data Warehouse Database Management Systems
    Magic Quadrant for Data Warehouse Database Management Systems Gartner RAS Core Research Note G00209623, Donald Feinberg, Mark A. Beyer, 28 January 2011, RV5A102012012 The data warehouse DBMS market is undergoing a transformation, including many acquisitions, as vendors adapt data warehouses to support the modern business intelligence and analytic workload requirements of users. This document compares 16 vendors to help you find the right one for your needs. WHAT YOU NEED TO KNOW Despite a troubled economic environment, the data warehouse database management system (DBMS) market returned to growth in 2010, with smaller vendors gaining in acceptance. As predicted in the previous iteration of this Magic Quadrant, 2010 brought major acquisitions, and several of the smaller vendors, such as Aster Data, Ingres and Vertica, took major strides by addressing specific market needs. The year also brought major market growth from data warehouse appliance offerings (see Note 1), with both EMC/Greenplum and Microsoft formally introducing appliances, and IBM, Oracle and Teradata broadening their appliance lines with new offerings. Although we believe that much of the growth was due to replacements of aging or performance-constrained data warehouse environments, we also think that the business value of using data warehouses for new applications such as performance management and advanced analytics has driven — and is driving — growth. All the vendors have stepped up their marketing efforts as the competition has grown. End-user organizations should ignore marketing claims about the applicability and performance capabilities of solutions. Instead, they should base their decisions on customer references and proofs of concept (POCs) to ensure that vendors’ claims will hold up in their environments.
    [Show full text]
  • Government Contracting M&A Update
    Government Contracting M&A Update “Market Intelligence for Business Owners” Q3 2013 Capstone Partners Investment Banking Advisors BOSTON | CHICAGO | LONDON | LOS ANGELES | PHILADELPHIA | SAN DIEGO | SILICON VALLEY Government Contracting Coverage Report MERGERS & ACQUISITIONS UPDATE With the nation’s attention focused on reducing government spending and sequestration, one would expect mergers & acquisitions in the government contracting space to come CAPSTONE PARTNERS LLC to a standstill. But such is not the case, with the number of acquisitions announced 200 South Wacker Drive through June totaling more than 250. 31st Floor Chicago, IL 60606 M&A Activity: Government Contractors www.capstonellc.com 1000 964 900 852 800 786 772 786 732 751 700 568 Ted Polk 600 521 Transactions Managing Director 500 of 398 (312) 674‐4531 400 [email protected] 300 256 Number 200 100 Lisa Tolliver 0 Director 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 YTD (312) 674‐4532 2013 [email protected] YTD 2013 through June 30, 2013 Source: Capital IQ, Capstone Partners LLC research While the year’s activity is currently on‐track to come in under the 2012 figure, that trend is reflective of what we are seeing in mergers and acquisitions in general. M&A activity across the board has been down in early 2013 compared to 2012, primarily the result of the market continuing to absorb the rash of transactions that were closed at the end of 2012 in anticipation of rising capital gains tax rates. But, while the number of closed transactions has slowed this year, M&A activity continues to be supported by strong market fundamentals, namely reasonably high transaction valuations; strategic acquirers with strong balance sheets; abundant private equity capital; an accessible and affordable debt market; and a modestly expanding U.S.
    [Show full text]
  • Nexus User Guide (Pdf)
    The Best Query Tool Works on all Systems When you possess a tool like Nexus, you have access to every system in your enterprise! The Nexus Query Chameleon is the only tool that works on all systems. Its Super Join Builder allows for the ERwin Logical Model to be loaded, and then Nexus shows tables and views visually. It then guides users to show what joins to what. As users choose the tables and columns they want in their report, Nexus builds the SQL for them with each click of the mouse. Nexus was designed for Teradata and Hadoop, but works on all platforms. Nexus even converts table structures between vendors, so querying and managing multi-vendor platforms is transparent. Even if you only work with one system, you will find that the Nexus is the best query tool you have ever used. If you work with multiple systems, you will be even more amazed. Download a free trial at www.CoffingDW.com. The Tera-Tom Video Series Lessons with Tera-Tom Teradata Architecture and SQL Video Series These exciting videos make learning and certification much easier Four ways to view them: 1. Safari (look up Coffing Studios) 2. CoffingDW.com (sign-up on our website) 3. Your company can buy them all for everyone to see (contact [email protected]) 4. YouTube – Search for CoffingDW or Tera-Tom. The Tera-Tom Genius Series The Tera-Tom Genius Series consists of ten books. Each book is designed for a specific audience, and Teradata is explained to the level best suited for that audience.
    [Show full text]
  • TOP-OF-MIND TIME for IN-MEMORY DATABASES PERFORMANCE 2 In-Memory Databases Help Meet Need for IT Speed
    VIRTUALIZATION CLOUD DEVELOPMENT APPLICATION IT HEALTH NETWORKING ARCHITECTURE STORAGE CENTER MANAGEMENT DATA BI/APPLICATIONS RECOVERY/COMPLIANCE DISASTER SECURITY 1 EDITOR’S NOTE Top-of-Mind Time for 2 IN-MEMORY DATABASES HELP In-Memory Databases MEET NEED FOR IT SPEED For all the promise they hold, in-memory databases and the process of implementing them require heavy investments in 3 IBM GIVES DB2 MORE GAS WITH company resources and skills. Is it worth it? IN-MEMORY ACCELERATOR 4 ADD-ON SOFTWARE TAKES ORACLE 12C IN NEW DIRECTION EDITOR’S NOTE 1 In-Memory’s Moment in the Database Sun In-memory databases used to be terri- 2014 report that in-memory databases could Home tory for niche technology vendors and equally provide “transformational performance im- niche applications. But today vendors of all da- provements” in operational and analytical ap- Editor’s Note tabase stripes—SQL, NoSQL, NewSQL—now plications. But in a video posted on YouTube offer in-memory technology, some as stand- the following month, Rosen said the heavily In-Memory Databases Help alone products and others as add-ons to disk- hyped technology also has “the potential to be Meet Need for IT based database management systems. That the next failed silver bullet from IT.” Challenges Speed includes relational database market leaders Or- abound, he cautioned, including data migration acle, IBM and Microsoft as well as business ap- issues and the proliferation of data silos that IBM Gives DB2 plications bigwig SAP with its HANA system. make it hard to do real-time analytics. More Gas With In an interview with SearchDataManage- This guide explores in-memory database In-Memory Accelerator ment’s Jack Vaughan, data management consul- trends and offers advice to help you get started tant William McKnight said that as the price of on deciding whether the technology is right for Add-On Software RAM declines, “memory in a lot of ways is be- your organization.
    [Show full text]
  • Next Generation Data Warehouse Platforms
    fourth quarter 2009 TDWI besT pracTIces reporT Next geNeratioN Data Warehouse Platforms By Philip Russom www.tdwi.org Research Sponsors Aster Data Systems HP IBM Infobright Kognitio Microsoft Oracle/Intel Sybase Teradata fourth QuArtEr 2009 TDWI besT pracTIces reporT Next geNeratioN Data Warehouse Platforms By Philip Russom Table of Contents Research Methodology and Demographics . 3 Introduction to Next Generation Data Warehouse Platforms . 4 Definitions of Terms and Concepts. 4 Why Care about Data Warehouse Platforms Now? . 5 The Evolving State of Data Warehouse Platforms . 6 Technology Drivers for New Generations of Data Warehouses . 6 Business Drivers for New Generations of Data Warehouses . 9 Your Data Warehouse Today and Tomorrow. 10 Quantifying Data Warehouse Generations . 13 Growth or Decline of Usage versus Breadth or Narrowness of Commitment . 14 Trends for Next Generation Data Warehouse Platform Options . 16 Next Generation Data Warehouse Platform Options . 17 Real-Time Data Warehousing. 17 Data Management Practices . 19 Cloud Computing and Software-as-a-Service (SaaS). .20 In-Memory Processing and 64-Bit Computing . 21 Open Source Software . .22 Advanced Analytics . .23 Services . 24 Processing Architectures. .25 Data Warehouse Appliances and Similar Platforms . .26 New Database Management Systems as Alternative Options. .28 Recommendations . 31 © 2009 by TDWI (The Data Warehousing InstituteTM), a division of 1105 Media, Inc. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. E-mail requests or feedback to [email protected]. Product and company names mentioned herein may be trademarks and/or registered trademarks of their respective companies. www.tdwi.org 1 NENERATIONE x T G DATA WAREHOUSE Pl ATfORMS About the Author PHILIP RUSSOM is the senior manager of TDWI Research at The Data Warehousing Institute (TDWI), where he oversees many of TDWI’s research-oriented publications, services, and events.
    [Show full text]
  • Big Data Landscape for Databases
    Big Data Landscape for Databases Bob Baran Senior Sales Enginee [email protected] ! May 12, 2015 Typical Database Workloads OLTP Applications Real-Time Web, Real-Time, Ad-Hoc Analytics Enterprise Data Mobile, and IoT Operational Warehouses Applications Reporting Typical • MySQL • MongoDB • MySQL • Greenplum • Teradata Databases • Oracle • Cassandra • Oracle • Paraccel • Oracle • MySQL • Netezza • Sybase IQ • Oracle Use Cases • ERP, CRM, Supply • Web, mobile, social • Operational • Exploratory • Enterprise Chain • IoT Datastores Analytics Reporting • Crystal Reports • Data Mining Workload • Real-time updates • Real-time updates • Real-time updates • Complex • Parameterized Strengths • ACID transactions • High ingest rates • Canned, queries reports against • High concurrency • High concurrency of parameterized requiring full historical data of small reads/ small reads/ writes reports table scans writes • Range queries • Range queries • Append only • Range queries Operational Analytical 2 Recent History of RDBMSs ▪ RDBMS Definition ▪ Relational with joins ▪ ACID transactions ▪ Secondary indexes ▪ Typically row-oriented ▪ Operational and/or analytical workloads ▪ By early 2000s ▪ Limited innovation ▪ Looked like Oracle and Teradata won… 3 Hadoop Shakes Up Batch Analytics ▪ Data processing framework ▪ Cheap distributed file system ▪ Brute force, batch processing through MapReduce ▪ Great for batch analytics ▪ Great place to dump data to look at later 4 NoSQL Shakes Ups Operational DBs ▪ NoSQL wave ▪ Companies like Google, Amazon and
    [Show full text]
  • A Technical Overview the Paraccel Analytic Database
    THE PARACCEL ANALYTIC DATABASE A TECHNICAL OVERVIEW The ParAccel Analytic Database: A Technical Overview The ParAccel Analytic Database: A Technical Overview Version 2.5 February 10, 2010. www.paraccel.com © 2010 ParAccel, Inc. All Rights Reserved. ParAccel product names are trademarks of ParAccel, Inc. Other product names are trademarks of their respective owners. © 2010 ParAccel, Inc. All rights reserved. i The ParAccel Analytic Database: A Technical Overview TABLE OF CONTENTS Introduction ............................................................... 1 System Architecture ................................................... 2 LEADER NODE.......................................................................................2 COMPUTE NODES .................................................................................3 COMMUNICATION FABRIC ....................................................................4 OPTIONAL STORAGE AREA NETWORK (SAN) .....................................4 PADB Features ............................................................ 4 PERFORMANCE.....................................................................................4 Columnar Orientation ................................................................................... 5 Column vs. Row Example ........................................................................... 6 How Does Columnar Orientation Impact Design Considerations?.............. 6 Extensible Analytics ....................................................................................
    [Show full text]
  • The “Tech”Tonic Shift Dale Wickizer Chief Technology Officer U.S
    The “Tech”tonic Shift Dale Wickizer Chief Technology Officer U.S. Public Sector NetApp Confidential — Limited Use Today I want to talk to you about the “Tech”tonic shift occurring to traditional enterprise applications as well as the IT organizations that manage them. © 2011 NetApp. All rights reserved. 1 World Data Explosion Growth Over the Next Decade: Servers (Phys/VM): 10x Data/Information: 50x #Files: 75x IT Professionals: <1.5x Source: Revisited: The Rapid Growth in Unstructured Data « Wikibon Blog http://bit.ly/oRSdXm • Growing 9x in 5 yrs! (1.8 ZB in 2011) • > 90% unstructured data Source: Gantz, John and Reinsel, David, “Extracting Value from Chaos”, • End user and machine generated IDC IVIEW, June 2011, page 4. 2 That shift is being driven by an explosion of data being generated and consumed in the world. Data has grown by a factor of 9 over the past 5 years, crossing 1.2 ZB for the first time! (If anyone wonders what 1.2 ZB is, Wikibon has this great graphic, showing it is the equivalent of 75 billion fully loaded iPads, stacked end-to-end and side-by-side, covering Wembley stadium, in a column more than 4 miles high). This year it will grow to 1.8 ZB. More than 90% of this data was unstructured and much of machine generated, in response to data stored by end users. Over the next decade, this data growth is expected to accelerate, increasing by a factor of 50. Over the same time, the number of files is expected to increase by more than a factor of 75, which will break most traditional file systems.
    [Show full text]
  • TPC Benchmark H Full Disclosure Report Vmware® ESX
    TPC Benchmark H Full Disclosure Report VMware® ESX™ Using ParAccel Analytic Database™ Submitted for Review Report Date: April 11, 2010 TPC Benchmark H Full Disclosure Report Pricing revision: August 24, 2010 TPC Benchmark H Full Disclosure Report Page 1 First Edition – April 2010 Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware, ESX, and ESXi are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. TPC-H Benchmark™ is a trademark of the Transaction Processing Performance Council. ParAccel Analytic Database™ is a registered trademark of ParAccel, Inc. VMware, Inc., the Sponsor of this benchmark test, believes that the information in this document is accurate as of the publication date. The information in this document is subject to change without notice. The Sponsor assumes no responsibility for any errors that may appear in this document. The pricing information in this document is believed to accurately reflect the current prices as of the publication date. However, the Sponsor provides no warranty of the pricing information included in this document. Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, the TPC Benchmark H should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated.
    [Show full text]
  • Big Data: Challenges, Opportunities and Realities
    Big Data: Challenges, Opportunities and Realities (This is the pre-print version submitted for publication as a chapter in an edited volume “Effective Big Data Management and Opportunities for Implementation”) Recommended Citation: Bhadani, A., Jothimani, D. (2016), Big data: Challenges, opportunities and realities, In Singh, M.K., & Kumar, D.G. (Eds.), Effective Big Data Management and Opportunities for Implementation (pp. 1-24), Pennsylvania, USA, IGI Global Big Data: Challenges, Opportunities, and Realities Abhay Kumar Bhadani Indian Institute of Technology Delhi, India Dhanya Jothimani Indian Institute of Technology Delhi, India ABSTRACT With the advent of Internet of Things (IoT) and Web 2.0 technologies, there has been a tremendous growth in the amount of data generated. This chapter emphasizes on the need for big data, technological advancements, tools and techniques being used to process big data are discussed. Technological improvements and limitations of existing storage techniques are also presented. Since, the traditional technologies like Relational Database Management System (RDBMS) have their own limitations to handle big data, new technologies have been developed to handle them and to derive useful insights. This chapter presents an overview of big data analytics, its application, advantages, and limitations. Few research issues and future directions are presented in this chapter. Keywords: Big Data, Big Data Analytics, Cloud Computing, Data Value Chain, Grid Computing, Hadoop, High Dimensional Data, MapReduce INTRODUCTION With the digitization of most of the processes, emergence of different social network platforms, blogs, deployment of different kind of sensors, adoption of hand-held digital devices, wearable devices and explosion in the usage of Internet, huge amount of data are being generated on continuous basis.
    [Show full text]
  • Citus Data Prepares Citusdb 4.0, Now a Massively Parallel Postgresql Analytic Database
    Citus Data prepares CitusDB 4.0, now a massively parallel PostgreSQL analytic database Analyst: Matt Aslett 9 Mar, 2015 Citus Data has changed its positioning since our last update, evolving CitusDB from being a scalable analytics database predominantly designed to bring SQL analytics to Hadoop to offering a stand-alone massively parallel columnar analytics database that is PostgreSQL-compatible. The 451 Take We noted that Citus Data was entering a crowded market in 2013, and therefore see the change of direction as a good thing. While the MPP analytic-database market is no less crowded, Citus Data is differentiated by its focus on extending, rather than forking, PostgreSQL. Making the cstore_fdw and pg-shard projects open source should grow the company's profile in the PostgreSQL user community and lay the foundation for potential CitusDB adoption. The competitive situation is likely to heat up, given that Pivotal's open source Greenplum strategy appears to be dependent on making it the default MPP choice for PostgreSQL, but we agree with Citus Data that Greenplum, having forked from PostgreSQL several years ago, will be a challenge. Either way, the PostgreSQL community will decide. Context The first time we encountered Citus Data, almost two years ago, the company had just launched CitusDB 2.0, bringing real-time SQL analytics to the Apache Hadoop data-processing framework. The SQL-on-Hadoop party was already in full swing, and got very crowded very quickly. While Citus Data had planned to differentiate itself by bringing SQL-based analytics to other nonrelational data Copyright 2015 - The 451 Group 1 platforms, including NoSQL databases, the company instead took a more radical change of direction, turning CitusDB into a stand-alone PostgreSQL-compatible massively parallel columnar analytics database.
    [Show full text]