Real-time Insights and Decision Making using Hybrid Streaming, In-Memory Computing Analytics and Transaction Processing Applying Gartner’s Hybrid Transaction/Analytical Processing Architecture to Cybersecurity 1 Welcome Welcome 3 Businesses are processing exponentially more The financial services industry has been Hybrid Transaction/Analytical data from clicks, swipes, micropayments, cyber successfully dealing with similar challenges for Processing Will packets, social feeds and meter readings. The over two decades. Banks and trading firms have Foster Opportunities for Dramatic volume of streaming data is overwhelming coped with steadily increasing data volumes Business Innovation IT systems, as a result many businesses are over that period using a simple scalable data “blind and deaf” for minutes to hours at a time. architecture composed of a real-time database 8 Traditional data solutions are struggling to (RDB) and an historical database (HDB). Their Proven Performance and Flexibility – Kx HTAP Solution update, organize and index their data so it can approach is based on a hybrid of streaming, Cybersecurity Use Case be queried in real-time to see the true state of in-memory computing (IMC) RDBs and persistent their business. HDBs to provide real-time services to traders, 9 regulators and risk managers. Analyst for Kx Today’s businesses need fast interactive access to tens or hundreds of terabytes of data extracted Analysts in other industries like energy, telecom from huge data lakes and legacy systems. They operations systems support (OSS) and cybersecurity also need to combine multiple data sources, also need to maintain a total view of the current and both in-flight and historical data, to dynamically past state of their grids and networks. aggregate their data along different dimensions and visualize it in both tabular and graphical One of the major benefits of this architecture is form. In particular, analysts at data driven that it is easy to scale up or out, simply by adding businesses must be able to “live in their data,” additional RDBs and HDBs. Multiple RDBs simultaneously interacting with both historical and HDBs can be provisioned for specific time and real-time streaming data. periods, types of transactions, etc. In financial services it is common to find very sophisticated data architectures that combine multiple instances of streaming with complex event processing (CEP), RDBs and HDBs with publish/subscribe capabilities, and with today’s memory solutions enables businesses to query all their data business processing using micro-services. in near real-time providing a consistent picture directly assembled from the raw transactional data as required. The in-memory RDB addresses real-time business needs while the HDB is an immutable record of all past transactions. By querying both the RDB This widely used RDB/HDB architecture has lately been re-discovered by and the HDB one can always get a consistent view of the world and the the Big Data community. Variations on the model have been described as state of the business. Increasingly, RDBs are being placed in huge non- Lambda Architecture, Gartner’s Hybrid Transaction/Analytical Processing volatile memory and HDBs are being stored on fast SSDs. Working smartly (HTAP) 1, Forrester’s Translytical DB2 and Event Sourcing. We feel the Gartner report below provides valuable guidance on how to realize business value using HTAP. 1 Gartner, Predicts 2016: In-Memory Computing-Enabled Hybrid Transaction/Analytical Processing Supports Dramatic Digital Business Innovation by Fabrizio Biscotti, Massimo Pezzini, Nigel Rayner, Joseph Unsworth, Roxane Edjlali, Susan Tan, Errol Rasit, Andrew Norwood, Andrew Butler, W. Roy Schulte ; 17 December 2015 2 Forrester, Emerging Technology: Translytical Databases Deliver Analytics At the Speed of Transactions by Noel Yuhanna and Mike Gualtieri; December 10, 2015 Real-time Insights and Decision Making using Hybrid Streaming, In-Memory Computing Analytics and Transaction Processing is published by KX. Editorial supplied by KX is independent of Gartner analysis. All Gartner research is © 2016 by Gartner, Inc. All rights reserved. All Gartner materials are used with Gartner’s permission. The use or publication of Gartner research does not indicate Gartner’s endorsement of KX’s products and/ or strategies. Reproduction or distribution of this publication in any form without prior written permission is forbidden. The information contained herein has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. Gartner shall have no liability for errors, omissions or inadequacies in the information contained herein or for interpretations thereof. The opinions expressed herein are subject to change without notice. Although Gartner research may include a discussion of related legal issues, Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company, and its shareholders may include firms and funds that have financial interests in entities covered in Gartner research. Gartner’s Board of Directors may include senior managers of these firms or funds. Gartner research is produced independently by its research organization without input or influence from these firms, funds or their managers. For further information on the independence and integrity of Gartner research, see “Guiding Principles on Independence and Objectivity” on its website, http://www.gartner.com/technology/about/ombudsman/omb_guide2.jsp. 2 Research from Gartner: 3 Hybrid Transaction/Analytical Processing Will Foster Opportunities for Dramatic Business Innovation Hybrid Transaction/Analytical Processing • Plan for coexistence and interoperability Analytical processing systems (see Note 2) Will Foster Opportunities for Dramatic of IMC and traditional technology, mixing were created separately to optimize data Business Innovation products from different vendors, if needed. structures for massive queries, as well as to avoid access concurrency and the Hybrid transaction/analytical processing will • Revisit your information management associated performance and scalability empower application leaders to innovate via strategy, including governance, SLAs impact of analytical queries over business- greater situation awareness and improved and life cycle management, to ensure critical transactional systems. Such a business agility. This will entail an upheaval proper organizational alignment and separation also allowed the combining of in the established architectures, technologies ownership of the information in support data from multiple sources into analytical and skills driven by use of in-memory of using HTAP architectures along with databases to measure enterprise computing technologies as enablers. traditional approaches. performance across applications. Impacts Analysis Therefore, transaction processing and • The emergence of HTAP means IT leaders The notion of running transactions and analytical systems are usually based must identify the value of advanced real- analytics on the same database of record on distinct architectures, thus adding time analytics, and where and how these has been around since the early days of complexity to the information architecture enable process innovation. computing, but has not fully materialized so and relevant infrastructure and introducing far because of a variety of issues, including delay in data analysis (see Figure 1). • By eliminating analytic latency and data technology limitations. synchronization issues, HTAP will enable Because of technology advances such IT leaders to simplify their information Transaction processing systems (see Note 1) as in-memory computing (IMC), hybrid management infrastructure, if they can are usually supported by operational DBMSs transaction/analytical processing (HTAP) overcome the challenges of adopting this (formerly referred to as online transaction architectures, which enable applications new approach. processing [OLTP] DBMSs; storing data in to analyze “live” data as it is created structures optimized for fast access to single and updated by transaction-processing • Technology immaturity and established data items (“rows”). functions, are now realistic and possible. application environment value and Although not every business process equally FIGURE 1 complexity will force IT leaders to plan for Traditional Transaction and Analytical Processing long-term coexistence between HTAP and traditional approaches. Recommendations • Educate business leaders about HTAP and IMC concepts, and why they are important. Brainstorm with them to identify concrete opportunities to rethink business processes. • Discuss with your strategic information management and business application providers their vision, road map and technology for HTAP implementation in their products. • Pilot the use of HTAP architectures in individual “system of innovation” projects. • Balance the costs of HTAP adoption (hardware and software infrastructure, migration, operational processes, and skills) with the anticipated business and IT benefits. Source: Gartner (January 2014) 3 benefits by such architectures (for example, Gartner believes less than 1% of user architecture, information management and in certain cases latency is characteristic of organizations have implemented HTAP- business analytic strategies. the process itself and not just an induced based applications, and expects