SAP Sybase IQ 16 Product Overview What was the motivation for SAP Sybase IQ 16? Marketplace Business Challenges SAP Sybase IQ 16… Lost revenues due to lack Cost-effective petabyte-scale EDW of insight platform Exploding Data Volumes Quickly handles and shares all the Slow Performance data in your world The need for Speed High performance and efficiency for user-driven analytics workloads High Costs & Complexities Rising IT Costs and Secure Complexity Data Management Challenges Ensures your data is always available, day or night © 2013 SAP AG. All rights reserved. Public 2 SAP Sybase IQ 16 SAP Sybase IQ transforms the way companies compete and win through actionable intelligence delivered at the speed of business to more people and processes. © 2013 SAP AG. All rights reserved. Public 3 Value of SAP Sybase IQ 16 1 Exploits the value of Big Data 2 Transforms businesses through deeper insights 3 Extends the power of analytics across the entire enterprise © 2013 SAP AG. All rights reserved. Public 4 SAP Sybase IQ 16 Path to Actionable Intelligence 2009 2009 2010 2011 2011 1H 2013 XLDB Analytics MapReduce API PlexQ MPP Foundation Full text search; Web 2.0 API In-database analytics Very large database (VLDB) platform foundation © 2013 SAP AG. All rights reserved. Public 5 SAP Sybase IQ 16 What’s New! SAP Sybase IQ 16 Engine Comprehensive web- Web-enabled analytics Web based administration and monitoring and administration based Web based monitoring and administration Communications and security Resilient Multiplex grid Resilient Information lifecycle management lifecycle Information withstands network Role-based access control interrupts and server LDAP authentication failures Enterprise-grade RBAC authorization and LDAP Query engine authentication Aggressive scale out architecture grid Multiplex Loading engine Hash partitioned tables Aggressively scaled out and data affinity High performance, fully query engine Fully parallel bulk loading parallel Text search In-database analytics Intelligent query engine New indexing technology for with data affinity for “shared increased compression and Column indexing N-bit and nothing” performance on a fast, incremental batch loads subsystem tiered indexing flexible “shared everything” Column store architecture New Generation Re-engineered column PETABYTE SCALE store store for extreme data Low latency, write optimized store Low latency, concurrent volumes write optimized delta store Storage area network © 2013 SAP AG. All rights reserved. Public 6 SAP Sybase IQ A comprehensive analytics platform Sybase Bradmark, SAS, SPSS, PowerDesigner, Symantec, KXEN, Fuzzy BMMSoft, Sybase Replication Whitesands, Logix, Zementis, SOLIX, and PBS Server, Quest, and and Visual SAP and Panopticon ZEND Numerics Optimized BI and Development and Packaged ILM EIM to model and Predictive analytics Ecosystem administration tools applications replicate Application services Hadoop DBMS and R Comprehensive In-database analytics Built-in full- Web 2.0 Big Data open ANSI SQL with with MapReduce and text search APIs source APIs OLAP simulator Most mature Comprehensive MPP queries, virtual High-speed Structured and column store lifecycle tiering marts, and user scaling loads unstructured store © 2013 SAP AG. All rights reserved. Public 7 SAP Sybase IQ is a Key Component in the SAP Real-Time Data Platform Unified open software platform for real-time business SAP Real-time Data Platform SAP Real-Time Data Platform foundations Common programming ● Cross-paradigm data access for new APIs models of value discovery. SAP Sybase Sybase SAP ● Hyper-performance on all classes of SAP Sybase ASE SAP Sybase IQ application and usage scenarios transactions EDW ● Price-Performance value across all use Control cases In-memory/real-time C Benefits odeling SAP HANA enter enter m ● Execute, record, analyze, and optimize without system limitations m SAP Sybase ESP SAP Sybase SQL onitoring Hadoop ● Embrace and extend across variations of streams SAP Sybase PowerDesigner Sybase SAP Big Data Anywhere data forms and processing models mobile and embedded ● Common modeling, integrated development environment, shared SAP Data Services / SAP Information Steward systems management infrastructure, and information management deployment-independent solutions ● Trusted and unified data environment © 2013 SAP AG. All rights reserved. Public 8 SAP Sybase IQ 16 Architectural Details SAP Sybase IQ 16 Innovations for extremely large databases (XLDB) Storage Architecture Loading Engine • New generation column store • Fully parallel bulk loading • New partitioning and compression • Real-time loading into delta store SAP Sybase IQ Petabytes XLDB Real-time Analytics System Reliability Query Processing • Grid resiliency • Data affinity • LDAP and role-based security • Aggressively parallel and distributed © 2013 SAP AG. All rights reserved. Public 10 SAP Sybase IQ 16 Architecture © 2013 SAP AG. All rights reserved. Public 11 SAP SYBASE IQ 16 NEW COLUMN STORE ARCHITECTURE Before … Value proposition Logical Enhanced compression page of fixed sized Storage savings cells Improved I/O bandwidth Architectural considerations offset = ((rowID – start rowID ) – 1 / Support variable number of cells per page cellsPerPage Support various page formats within a column High performance access paths NOW… Even with variable length data, insert/update/delete Logical cell values efficiently into an existing page page of Richer metadata variable sized cells directory © 2013 SAP AG. All rights reserved. Public 12 SAP SYBASE IQ 16 N-BIT DICTIONARY COMPRESSION Value proposition Reduced memory footprint Raw Data = 400 MB; 1 Billion 4-byte integer values fn (N) Improved effective I/O rates N Token Size Savings More efficient table scans 2 (1B * 2 ) / 8 = 250MB 93.75% 3 (1B * 3 ) / 8 = 375MB 90.6% Reduced disk space 4 (1B * 4 ) / 8 = 500MB 87.5% ……. 24 (1B * 24 ) / 8 = 3000MB 25% Architectural considerations N-bit FP, instead of 1, 2 and 3-byte FPs Different data pages for same column can have different values of “N” for N-bit No more requirement to rollover FP format for all column data 2->3 Column is N-bit by default, unless otherwise specified to be flat 3->4 4->5 5->6 Options provided to set threshold for rollover to flat (to prevent large dictionaries) 6->8 8->10 Options provided to prevent rollover to flat (to prevent long rollover time) 10->12 12->16 16->21 Compatibility mode allows the database to mimic IQ 15 rollover behavior 21->24 © 2013 SAP AG. All rights reserved. Public 13 SAP SYBASE IQ 16 SMALL BATCH LOAD PERFORMANCE Value proposition Improved performance of frequent, small batch loads Predictable performance of small batch loads: performance is proportional to the size of the data being loaded, not the table being updated Small tree Periodic component Merge Architectural considerations Btree- Inserting into a large High Group (HG) b-tree Trickle based index is costly insert index HG index will have a tiered structure with a small tree component and a large tree component Small, batch loads into the HG index are written to the small tree component quickly and synchronously The small tree component is periodically merged into the large tree component as a background task © 2013 SAP AG. All rights reserved. Public 14 SAP SYBASE IQ 16 FULLY PARALLEL BULK LOAD Before… Value proposition Serial and parallel two phase load process Raw data and bitmapped indexes – partly parallel: mixture of horizontal Improved performance and vertical processing Maximize use of existing cores on the machine B-Tree based indexes (HG, TEXT. WD) – fully parallel Dynamic load balancing Numerous bottlenecks: Architectural considerations • Complex thread scheduling Load an index/column concurrently with multiple threads • Expensive synchronization points Remove all bottlenecks which contribute to • Steps that are executed too infrequently to keep threads busy inefficient use of CPU and storage Dynamically scale up and down degree of parallelism depending on the workload NOW … Fully parallel two phase load process Raw data– fully parallel All secondary indexes – fully parallel © 2013 SAP AG. All rights reserved. Public 15 SAP SYBASE IQ 16 HIGH VELOCITY DATA LOADING Value proposition Continuous analytics over operational data Hybrid Table High velocity, concurrent data modifications Storage Insert/Update/Delete IQ DML Write ahead/commit Exploit large memory and core footprints engine In- RLV log on Architectural considerations memory shared RLV rows RLV Store store Write optimized in-memory In-memory RLV (Row- Lock Manager View Consistent periodic merge level versioned) store global row space Row level locking, and statement snapshot IQ query isolation engine IQ main on main rows shared store Low latency micro operations In-memory RLV store has reduced compression, no sorting, no indexing Transaction Consistent View Version Manager Fully recoverable with dedicated transaction log Manager Manager Asynchronous data transfer from In-memory RLV store to IQ main store Users choose which tables are In-memory RLV tables © 2013 SAP AG. All rights reserved. Public 16 SAP SYBASE IQ 16 QUERY SCALE OUT – HASH PARTITIONING Value proposition Gives best of both worlds of shared everything and shared nothing Decreases hardware needs and localizes processing Architectural considerations Data is automatically partitioned during loading with built- in hash algorithms Data is divided into persistent subsets Reduces results sharing More efficient CPU usage Reduces instantaneous temp usage Optimizer will use hash partitions