© 2013 SAP AG. All rights reserved. Public 1 Agenda
Welcome
Agenda • Introduction to Dobler Consulting • SAP IQ Roadmap – What to Expect • Q&A
Presenters • Courtney Claussen - SAP IQ Product Management • Peter Dobler - CEO Dobler Consulting
Closing
© 2013 SAP AG. All rights reserved. Public 2 Introduction to Dobler Consulting
Dobler Consulting is a leading information technology and database services company, offering a broad spectrum of services to their clients; acting as your Trusted Adviser, Provide License Sales, Architectural Review and Design Consulting, Optimization Services, High Availability review and enablement, Training and Cross Training, and lastly ongoing support and preventative maintenance. Founded in 2000, the Tampa consulting firm specializes in SAP/Sybase, Microsoft SQL Server, and Oracle. Visit us online at www.doblerconsulting.com, or contact us at 813 322 3240, or [email protected].
© 2013 SAP AG. All rights reserved. Public 3 Your Data is Your DNA, Dobler Consulting Focus Areas
Strategic Database Consulting
SAP VAR D&T
DBA Database Managed Training Services Programs
Cross-Platform Expertise SAP Sybase® SQL Server® Oracle®
© 2013 SAP AG. All rights reserved. Public 4 What’s Ahead
ISUG-TECH Annual Conference April 14-17, Atlanta
• Register at http://my.isug.com/conference/registration
• Early Bird ending 2/28/14 (free hotel room with registration)
SAPPHIRENOW Annual Conference June 3-5, Orlando
• Come visit our kiosk in the exhibition hall
© 2013 SAP AG. All rights reserved. Public 5 SAP IQ Roadmap Dobler Events Webinar
Courtney Claussen / SAP IQ Product Management February 27, 2014 Disclaimer
This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 2 Table of contents
Product Overview SAP IQ product description Strategic asset in SAP’s database portfolio SAP IQ architecture Product Road Map Shared nothing Extreme data volume Enhanced administration Summary
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 3 Product Overview Capabilities and positioning SAP IQ product description High performance analytics with low TCO
Trusted open analytics Performance & scalability Integrated ecosystem server
10x – 1000x faster queries over #1 disk-backed column DB for Best-of-breed partners certified traditional DBMS reporting and analytics with SAP IQ Supports heavy ad-hoc loads with Platform of algorithms, services & Tighter integration with SAP 1000s of users concurrently APIs for big data analysis HANA, BusinessObjects BI Store & analyze terabytes to Highly flexible deployment options; Supports 3rd party BI tools and petabytes of structured & supports all major hardware & OS analytic applications unstructured data platforms
© 2014 SAP AG or an SAP affiliate company. All rights reserved. See Appendix for abbreviations 5 SAP IQ in SAP’s database portfolio Strategic asset
SAP ESP, SAP Replication Server, SAP ASE SAP IQ SQL Anywhere PowerDesigner & EIM
Transactional Petabyte scale Mobile Unified database with low analytics database and embedded EIM platform for TCO with low TCO database real-time SAP HANA
Open for partners SAP HANA Data Platform
© 2014 SAP AG or an SAP affiliate company. All rights reserved. See Appendix for abbreviations 6 SAP IQ architecture A comprehensive analytics platform
SAP PowerDesigner, Bradmark, SAS, SPSS, KXEN, SAP Replication Server, Symantec, BMMSoft, Fuzzy Logix, SAP Data Services, Whitesands, Quest, SOLIX, and PBS Zementis Panopticon and ZEND
Optimized BI and EIM to Development and Packaged ILM Predictive analytics Ecosystem model and replicate administration tools applications Application services Hadoop and DBMS R
Comprehensive ANSI Built-in full-text In-database analytics with Big Data open Web 2.0 APIs SQL with OLAP search MapReduce and simulator source APIs
Most mature Comprehensive MPP queries, virtual marts, High-speed Structured and column store lifecycle tiering and user scaling loads unstructured store
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 7 Product Road Map
Improvements planned through the end of 2014 SAP Sybase IQ 16 SP08 and SP09 Path to Extreme Scale 2009 2009 2010 2011 2011 2013 2014
New generation column store • Extended storage for MapReduce API SAP HANA PlexQ MPP Foundation • Shared nothing Full text search; Web 2.0 API architecture
In-database analytics
Very large database (VLDB) platform foundation
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 9 SAP IQ 16.0 SP08 and SP09 Planned improvements through 2014
Shared Nothing Extreme Data Volume • Extended storage for HANA • DAS (Direct Attached Storage) Cache • Distributed load • DAS Files • RLV (row level version) store on • DAS Files elasticity Multiplex
SAP IQ Enhanced XLDB
Enhanced Administration
• Point in time recovery • Integrated system management
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 10 SAP IQ 16 Key building blocks
SAP IQ 16: engine
Web based administrationmonitoring and Web based Web-enabled analytics
Resilient Information lifecycleInformation management Communications and security Database, column, and wire encryption
LDAP, Kerberos authentication, and RBAC
Multiplex Query engine Loading engine Distributed queries with data affinity Hash partitioned tables Fully architecture grid parallel Text search bulk loading
N-bit and tiered Schema indexing Column store
In-memory, write-optimized delta store
Re-engineered petascale disk store
Shared store
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 11 SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
SAP IQ 16: engine
Web based administrationmonitoring and Web based Web-enabled analytics
Resilient Information lifecycleInformation management • Integrated system Communications and security management Database, column, and wire encryption
LDAP, Kerberos authentication, and RBAC
Multiplex Query engine Loading engine Distributed queries with data affinity Hash partitioned tables Fully architecture grid parallel Text search bulk loading
N-bit and tiered Schema indexing Column store
In-memory, write-optimized delta store
Re-engineered petascale disk store
Shared store
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 12 SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
SAP IQ 16: engine
Web based administrationmonitoring and Web based Web-enabled analytics
Resilient Information lifecycleInformation management • Integrated system Communications and security management Database, column, and wire encryption
LDAP, Kerberos authentication, and RBAC
Multiplex Query engine Loading engine Distributed queries with data affinity Hash partitioned tables • Distributed load Fully architecture grid parallel Text search bulk loading
N-bit and tiered Schema indexing Column store
In-memory, write-optimized delta store
Re-engineered petascale disk store
Shared store
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 13 SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
SAP IQ 16: engine
Web based administrationmonitoring and Web based Web-enabled analytics
Resilient Information lifecycleInformation management • Integrated system Communications and security management Database, column, and wire encryption
LDAP, Kerberos authentication, and RBAC
Multiplex Query engine Loading engine Distributed queries with data affinity Hash partitioned tables • Distributed load Fully architecture grid parallel Text search bulk loading
N-bit and tiered Schema indexing
• HANA extended tables Column store In-memory, write-optimized delta store • PITR – Point in Time Recovery
Re-engineered petascale disk store
Shared store
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 14 SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
SAP IQ 16: engine
Web based administrationmonitoring and Web based Web-enabled analytics
Resilient Information lifecycleInformation management • Integrated system Communications and security • DAS Cache extends data management Database, column, and wire encryption affinity
LDAP, Kerberos authentication, and RBAC • DAS Files “shared nothing”
Multiplex architecture with elasticity Query engine Loading engine Distributed queries with data affinity Hash partitioned tables • Distributed load Fully architecture grid parallel Text search bulk loading
N-bit and tiered Schema indexing
• HANA extended tables Column store In-memory, write-optimized delta store • PITR – Point in Time Recovery
Re-engineered petascale disk store
Shared store
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 15 SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
SAP IQ 16: engine
Web based administrationmonitoring and Web based Web-enabled analytics
Resilient Information lifecycleInformation management • Integrated system Communications and security • DAS Cache extends data management Database, column, and wire encryption affinity
LDAP, Kerberos authentication, and RBAC • DAS Files “shared nothing”
Multiplex architecture with elasticity Query engine Loading engine Distributed queries with data affinity Hash partitioned tables • Distributed load Fully architecture grid parallel Text search bulk loading
N-bit and tiered Schema indexing
• HANA extended tables Column store In-memory, write-optimized delta store • PITR – Point in Time Recovery
Re-engineered petascale disk store
Shared store
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 16 SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
SAP IQ 16: engine
Web based administrationmonitoring and Web based Web-enabled analytics
Resilient Information lifecycleInformation management • Integrated system Communications and security • DAS Cache extends data management Database, column, and wire encryption affinity
LDAP, Kerberos authentication, and RBAC • DAS Files “shared nothing”
Multiplex architecture with elasticity Query engine Loading engine Distributed queries with data affinity Hash partitioned tables • Distributed load Fully architecture grid parallel Text search bulk loading • RLV on Multiplex N-bit and tiered Schema indexing
• HANA extended tables Column store In-memory, write-optimized delta store • PITR – Point in Time Recovery
Re-engineered petascale disk store
Shared store
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 17 Shared Nothing
• DAS (Direct Attached Storage) Cache • DAS Files • DAS Files elasticity
IQ 16.0 – “Shared Everything” Multiplex with Data Affinity
SAP IQ Multiplex
IQ Server IQ Server IQ Server IQ Server IQ Server
Scale out Scale out Data affinity Data affinity Data affinity Data affinity Data affinity Elastic Logical Server Elastic Logical Server
High Speed Interconnect to Shared Store
Scale out Scale out
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 19 DAS cache extends data affinity
SAP IQ Multiplex
DAS cache DAS cache DAS cache • DAS (direct attached storage) Cache is a mechanism for maintaining one or more IQ Server IQ Server IQ Server copies of primary data in direct attached storage on different physical servers. Data affinity Data affinity Data affinity • The DAS cache is populated in a pull model as new versions of data are accessed Shared Store • The DAS cache can offload bandwidth from the SAN for subsequent data access
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 20 DAS Files – IQ adopts a “shared nothing” architecture
Multiplex with high speed interconnect • DAS (direct attached storage) files are primary direct attached storage devices that are being DAS DAS DAS shared across the cluster using the high- cache cache cache speed interconnect. IQ Server IQ Server IQ Server • Primary data may exist in only one DAS file.
• These can be used in place of the SAN for primary storage
DAS Files DAS Files DAS Files • When data is loaded, it will be automatically written to the appropriate DAS file that the data has affinity to.
• With DAS files, DAS cache will still have value for dealing with data access that cannot be affinitized.
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 21 DAS Files – mirrored for write safety
Logical Server (Server_1, Server_2, Server_3, Server_4)
Server_1 Server_2 Server_3 Server_4
L1 M4 L2 M1 L3 M2 L4 M3
Primary Mirror
• DAS files are managed within a logical server
• Primary files are mirrored on a different server for data redundancy
• Optionally, user may create a DAS file without a redundant copy
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 22 DAS Files elasticity
High speed interconnect High speed interconnect
NEW IQ IQ IQ IQ NEW IQ IQ Server_1 Server_2 Server_3 Server_1 Server_3 IQ Server_4 Server_4
Empty
The IQ server redistributes data when nodes or files are added or removed
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 23 DAS Files – does not preclude a shared store
High speed interconnect • SAP Sybase IQ continues its flexible
DAS DAS DAS architecture. cache cache cache • An IQ database may be comprised of both IQ Server IQ Server IQ Server DAS DBSpaces and shared DBSpaces.
• Shared store could be used for data that is needed within each node, without duplication. DAS Files DAS Files DAS Files
Optional shared store
DBFile DBFile DBFile
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 24 Extreme Data
• Extended storage for SAP HANA • Distributed load • RLV (row level version) store on Multiplex SAP IQ is the preferred solution for warm and cold data archiving capabilities for SAP BW, Business Suite, and HANA
NLS (Near Line Store) for SAP BW • Optimized NLS data transfer throughput using IQ Loader functionality • SAP HANA and IQ share the same columnar paradigm, and similar data compression rates • Ready for large data volumes • Suitable for ad-hoc queries with long history • Minimum administration effort
ILM for SAP Business Suite • Fast archive index read w/o additional secondary DB indexes • Increased search capabilities • Faster archive I/O – fewer layers (software, network, storage hardware) • ERP archive files as well as archive indexes stored in SAP IQ
Warm archive for HANA using smart data access (SDA) • Using SDA, HANA customers can access IQ as a federated store • Store warm data in IQ, and real time data in HANA • Push query processing down to IQ
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 26 HANA extended storage brings integrated warm data management to SAP HANA
Cold/warm data archival Warm archive for NLS Store for ILM for SAP SAP BW HANA, federated Business Suite via SDA
Warm Data Management for SAP HANA Extended storage for SAP HANA • Handle all data seamlessly via HANA tables with Calc View support • Transformative ROI – enable new insights from data processing at speed and scale not possible before • Steady performance across data volumes, variety, velocity with scalable/elastic capacity • Practical for both on-premise and hosted models • Native Big Data solution allows customers to gain real-time insights by cost- effectively managing and analyzing ALL enterprise data
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 27 HANA extended storage Requirements from our customers
• Manage data cost effectively, yet with desired performance based on SLAs • Application defines which data is “hot”, and which data is “warm” • Handle very large data sets – terabytes to petabytes • Update and query all data seamlessly via HANA extended tables – IQ store is transparent to applications • Provide a native Big Data solution that covers most enterprise use cases without Hadoop
Table Queries and SAP IQ updates initiated Table from HANA Extended table (access via HANA) SAP HANA
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 28
BW on HANA with extended storage in IQ
Bulk load
BW Process and Data Management
NLS NLS interface SAP HANA Master Data, InfoCubes, DataStore Objects (DSO)
SDA SDA Smart Data Access (SDA) Persistent Corporate Staging Area Memory (PSA) Reads
SAP IQ Near Line Store (NLS) SAP Extractors, Data Services, SLT Replication SAP and non-SAP data SAP IQ sources
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 29 DISTRIBUTED LOADS SCALED OUT DATA LOADING
Value proposition Optimize performance of massive bulk loads Client Exploit large memory and core footprints Bulk load to IQ table Architectural considerations Multiplex SAP IQ base table data ingestion is distributed. Bulk loads scale out for the best possible load performance. IQ Server IQ Server IQ Server IQ Server Load speed should scale almost linearly when more Logical server cores / nodes are added. User may control whether a load gets distributed or not. Graphical plans for loads will be enhanced for Shared Store distributed loads.
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 30 ROW LEVEL VERSION DELTA STORE FOR MULTIPLEX HIGH VELOCITY DATA LOADING ON IQ CLUSTER
Value proposition Clients Clients Continuous analytics over operational data High velocity, High velocity, concurrent writes High velocity, concurrent data modifications concurrent writes across all writer nodes in a Multiplex Multiplex Exploit large memory and core footprints in a clustered deployment IQ Writer IQ Writer IQ Server Server Reader Architectural considerations Memory RLV Memory RLV Server Each writer node in a Multiplex may manage its store store own RLV store periodic periodic All RLV stores are fully recoverable with merge merge dedicated transaction logs on each writer node IQ Store Query engine sees a consistent view of data across RLV stores and main IQ store
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 31 Enhanced Administration
• Point in time recovery • Integrated system management POINT IN TIME RECOVERY
Backup at Server crashed midnight on and database is Monday corrupt
Thursday Sunday Monday Tuesday Wednesday Thursday 3:00pm
Op I D U C I D K B I U K D C I K I C I D C K U I C Crash
Time 2a 6a 11a 4p 3a 2p 5p 6p 11p 1a 8a 3p 9p 3a 10a 7p 11p 4a 10a 1p 8p 9a 11a 1p 3p U = Update B = Backup D = Delete C = Commit Database transaction log I = Insert K = Checkpoint Recover to noon Point in time recovery: • Database backup performed at 6:00pm on Monday • Disk failure occurs on Thursday at 3:00pm • User wants to recover database state on Thursday at noon: • Restore Monday backup • Replay transaction log from time of Monday backup until Thursday at noon – apply committed transactions, and roll back uncommitted transactions
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 33 INTEGRATED SYSTEMS MANAGEMENT
SAP Control Center (SCC) supports up to 100 servers of: HANA* SAP ASE ASE RTDP* SAP IQ SAP Replication Server (SRS) SAP Event Stream Processor (ESP) SAP Mobile Platform (SMP) SCC IQ >Admin SA* >Monitor >Alert * Emerging in future: SAP HANA SAP Real-Time Data Platform (RTDP) SRS SMP SAP SQL Anywhere (SA) Support up to 250 servers ESP
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 34 SAP CONTROL CENTER PLANNED ARCHITECTURAL IMPROVEMENTS
• Creation of a new datacenter management feature which will easily allow an administration group to identify the overall health of the servers within the datacenter and quickly drill into those servers.
• Provide integration of the datacenter features with existing landscape components registered to the LMDB
• Modify the existing Flex based management console to UI5 to enable a better user experience for ‘on board’ features.
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 35 Landscape view
Show nodes or server statuses, for example SCC Heat Chart:
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 36 Topology view
Show connectivity or relationship, for example SCC Topology View (future and current):
ASE RS HANA
ASE RS IQ
ESP HANA
SA ASE
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 37 Summary Summary
SAP IQ is a strategic technology asset for SAP’s database portfolio, and continues its innovative roadmap with:
Shared Nothing MPP – Optional shared nothing MPP configuration supports cloud deployments with more elastic architecture. Extreme Data Volume – New extended store feature for HANA, distributed loads, and RLV store on Multiplex address the extreme data volume and velocity requirements of big data. Enhanced Administration – Improved management features help today’s database administrators perform their jobs more efficiently and confidently.
This is the current state of planning and may be changed by SAP at any time.
© 2014 SAP AG or an SAP affiliate company. All rights reserved. See Appendix for abbreviations 39 SAP IQ Resources Where to get information on SAP IQ
• SAP IQ SCN Developer Site: http://scn.sap.com/community/developer-center/analytic-server
• Free Express and Enterprise Edition Downloads https://www.sap.com/iq-downloads
• “How to” Tutorials http://scn.sap.com/docs/DOC-43300
• SAP IQ 16 (video): https://www.youtube.com/watch?v=Bbcl8uY3tj4
• SAP IQ for Big Data(video): https://www.youtube.com/watch?v=XRNgVPxiyVw
• SAP IQ as Near-line Storage (video): https://www.youtube.com/watch?v=FdtdAhTQYSg
• SAP IQ for BusinessObjects (video): http://www.youtube.com/watch?v=A4hR0n2y8zQ
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 40
Thank you Courtney Claussen, Director of Product Management, SAP IQ [email protected]
© 2014 SAP AG or an SAP affiliate company. All rights reserved. Acronym glossary (1 of 2)
Acronym Full Text
API Application Programming Interface
ANSI American National Standards Institute
BI Business Intelligence
DAS Direct Attached Storage
EDW Enterprise Data Warehouse
EIM Enterprise Information Management
XLDB Extreme Large Database
ILM Information Lifecycle Management
LDAP Lightweight Directory Access Protocol
PAM Linux Pluggable Authentication Module
MPP Massively Parallel Processing
NLS Near Line Storage
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 42 Acronym glossary (2 of 2)
Acronym Full Text
OS Operating System
RTDP Real-Time Data Platform
RDBMS Relational Database Management System
RBAC Role Based Access Control
RLV Row Level Versioned
ASE SAP Adaptive Server Enterprise
ERP SAP Enterprise Resource Planning
ESP SAP Event Stream Processor
SAN Storage Area Network
SMP Symmetric multiprocessing (slide 21); SMP is also the acronym for ‘Service Marketplace’
TCO Total Cost of Ownership
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 43