In-Memory Computing Platform: Data Grid Deep Dive

Rachel Pedreschi Matt Sarrel Director of Solutions Architecture Director of Technical Marketing GridGain Systems GridGain Systems [email protected] [email protected] @rachelpedreschi @msarrel

• Introduction • In-Memory Computing • GridGain / Apache Ignite Overview • Survey Results • Data Grid Deep Dive • Customer Case Studies

Digital Transformation is Driving Companies Closer to Their Customers • Driving a need for real-time interactions

Internet Traffic, Data, and Connected Devices Continue to Grow • Web-scale applications and massive datasets require in-memory computing to scale out and speed up to keep pace • The Internet of Things generates huge amounts of data which require real-time analysis for real world uses

The Cost of RAM Continues to Fall • In-memory solutions are increasingly cost effective versus disk-based storage for many use cases

Data Growth and Internet Scale Declining DRAM Cost Driving Demand Driving Attractive Economics Growth of Global Data

35.

26.3

17.5

ZettabytesData of 8.8 DRAM

0. Flash Disk

8 zettabytes in 2015 growing to 35 in 2020 Cost drops 30% every 12 months

• Supports data caching, massive parallel processing, in-memory SQL, streaming and much more

Does Not Replace Existing Databases

• Slides in between the existing application and data layers

Supports OLTP and OLAP Use Cases

• Offers ACID compliant transactions as well as analytics support

Multi-Platform Integration

• Works with all popular RDBMS, NoSQL and Hadoop databases and offers a Unified API with support for a wide range of languages

Deployable Anywhere

• Can be deployed on premise, in the cloud, or in hybrid environments

© 2015 GridGain Systems, Inc. GridGain Company Confidential Apache Ignite Project • 2007: First version of GridGain • Oct. 2014: GridGain contributes Ignite to ASF • Aug. 2015: Ignite is the second fastest project to | graduate after Spark • Today: • 60+ contributors and rapidly growing • Huge development momentum - Estimated 192 years of effort since the first commit in February, 2014 [Openhub] • Mature codebase: 1M+ lines of code

In-Memory Always Available Scalable SQL 99 / ACID / No Rip & Replace MapReduce

Monitoring and Security High Availability Management

Services OLTP OLAP GaaS and Streaming Caching Caching Messaging

Data Grid Compute Grid

Apache Ignite / GridGain Professional

Software Banking IT Consulting Other Telecommunications Insurance Investment Management Healthcare FinTech Government Retail eCommerce Other Financial Services Other Online Services Manufacturing Online Gaming Social Media Media Logistics Other Electronics Online Business Services Computers Hospitality Pharmaceuticals Utilities

Database Caching

Application Scaling

High Speed Transactions

Real Time Streaming

Faster Reporting

RDBMS Scaling

HTAP

Apache Spark Acceleration

Web Session Clustering

Hadoop Acceleration

MongoDB Acceleration

0 18 35 53 70 Column1

Data Grid

Compute Grid

In-memory File System

Service Grid

Streaming Grid

ANSI SQL-99 Compliance

Container Support

In-Memory Hadoop File System

Spark Shared RDDs

Hadoop Acceleration

Integration with Zeppelin

0. 1.2 2.3 3.5 4.6 5.8 Column1

Private Cloud

On Premise

AWS

Microsoft Azure

Google Cloud Platform

Softlayer

Another Public Cloud

0 15 30 45 60 Column1

Java

SQL

MapReduce

.NET

C++

Scala

Node.js

PHP

Groovy

0 23 45 68 90 Column1

Oracle

MySQL

HDFS

PostresSQL

MongoDB

Microsoft SQL Server

Cassandra

DB2

Other

0 13 25 38 50 Column1

A A B A A B

Primary Primary Primary Primary B B

C B D C D C A C D C

Local Client Backup Backup Local Client Backup Backup

JVM 1 JVM 2 JVM 1 JVM 2

A A D C A A D C

B B Primary Primary B B Primary Primary

C C A B C B A D C C B A

Near Cache Near Cache Remote Client Backup Backup Remote Client Backup Backup

Client JVM JVM 3 JVM 4 Client JVM JVM 3 JVM 4

Replicated Cache Partitioned Cache

Data Cache Zone 1 Data & Compute Zone 2 Compute Grid Zone 3

Data read Data write

Server Server Client

Server Server Server Server Client Client

Server Server Client

Map/Reduce Data read Data write Job

Data read Compute Job

Client Client Client

GridGain / Ignite Cluster

• Tiered Memory Latency • On-Heap -> • Off-Heap -> Swap Space • Disk (Swap)

• Persistent On-Disk Store Off-Heap • Fast Recovery On-Heap • Local Data Reload

• Eliminate Network and Db impacts Capacity when reloading in-memory store Local Recoverable Store

• Unlimited Vertical Scale • Avoid Java Garbage Collection Pauses • Small On-Heap Footprint • Configurable eviction policies • Off-Heap Indexes • Full RAM Utilisation • Simple Configuration

DC1 DC2 • Multiple (up to 32) Data Centres • Complex Replication Technologies • Active-Active & Active-Passive • Smart Conflict Resolution • Durable Persistent Queues Active • Automatic Throttling Transactional or Bi-Directional Eventually • GridGain Enterprise Replication Consistent DC3

Write Through • Read-through & Write- through • Support for Write-behind DB External • Configurable eviction policies Persistent • DB schema mapping wizard: Ignite Clients Store • Generates all the XML configuration and Read Java POJOs Through

Ignite Cache Nodes

• Predicate-based Scan Queries • Text Queries based on Lucene indexing • Query configuration using annotations, Spring XML or simple Java code • SQL Queries: Automatic Group By, Aggregations, Sorting, Cross-Cache Joins, Unions • Memcached (PHP, Java, Python, Ruby) • HTTP REST API • JDBC & ODBC

• ANSI-99 SQL • In-Memory Indexes (On and Off- Heap) • Automatic Group By, Aggregations, Sorting • Cross-Cache Joins, Unions • Use local H2 engine

• Fully ACID • Support for Transactional & Atomic • Cross-cache transactions • Optimistic and Pessimistic concurrency modes with multiple isolation levels • Deadlock protection • JTA Integration

© 2016 GridGain Systems, Inc. 1000’s of Deployments Automated Trading Systems • Real time analysis of trading positions • Real time market risk assessment • High volume transactions • Ultra low latencies trading

Financial Services • Fraud Detection • Risk Analysis Mobile & IoT • Insurance rating and modeling • Real-time streaming processing • Complex event processing Big Data Analytics • Real time analysis of inventory Biotech • Operational up-to-the-second BI • High performance genome data matching • Drug discovery

© 2016 GridGain Systems, Inc. Case Study: • Connects over 650 lifestyle and consumer brands (more than 4.5 million products) with brand advocates • Brands use the network to identify experts and engage them in their sales channel • Member profiles (interests, experience, content viewed, purchases) • Real Time Analytics on "We need it to work fast and work at scale, thousands of active user and GridGain does. We couldn't do what we sessions for personalized do without GridGain, and we're confident we content can scale to meet the growth we anticipate.“ -Jeremy Knudsen, CTO

• Background: • The Challenge – Intelligentpipe is a big data – Collect and analyze massive software company serving amounts of mobile user the global traffic data in real time telecommunications industry – Tens of millions of users by developing solutions for – Consumption of network mobile operators to improve resources their business and – Type of network traffic operational processes (voice or data)

• GridGain Professional Edition used to build a high performance low latency analysis platform • “GridGain ensures responsiveness regardless of how much information we need to search through.” Sakari Paloviita, CTO, Intelligentpipe • Collect and analyze multiple terabytes per day

• Real-time analytics provides fast insight • Easy integration with existing systems due to GridGain’s Unified API and ANSI SQL-99 support • Linear scaling across deployed server to keep up seamlessly as the business grows

We’ll want to use technology GridGain offers so we can focus on our core business ourselves.” - Jari Kuusela, Director of Product Management

• Background: • The Challenge – Cyber Dust is a platform for text – To build a real-time, reliable messages: “A safer place to and highly available server text.” infrastructure to support a • Untraceable mobile messaging platform • Encrypted – More than 500K users • Disappearing – Millions of messages a day • Screenshot blocking – Available for Andoid and IoS – Avoid writing messages to – Mark Cuban funded disk

• GridGain Professional Edition used to build a messaging platform • Runs completely on Amazon EC2 • All user account data, configurations, and messages held in memory • Messages deleted without a trace because they were never written to disk • Extensive use of Unified API

• “Blast” feature performance: capable of broadcasting disappearing messages to all of a user’s contacts • Real-world performance of 300,000 messages sent and disappeared in 30 seconds

I was pleasantly surprised by I keep learning about the GridGain solution and additional capabilities performance. GridGain offers. It’s what I -Igor Shpitalnik, CTO expected and more. -Igor Shpitalnik, CTO

• Financial services software • The Challenge: Eliminating – Retail and corporate banking Data Processing Bottlenecks – Lending – Huge amounts of trade and – Treasury accounting data – Capital markets – Customers need – Investment management • High-speed transactions – Enterprise risk • More than 2,000 customers in • Real-time reporting 130 countries – New Java-based IT stack • Used by 48 of the world’s 50 with data lake support largest banks – Global regulatory compliance

• Commodity servers (256GB RAM) • Data stored in memory – Transactions – Market data • Parallel processing across cluster – Calculation heavy reporting for regulatory compliance

© 2016 GridGain Systems, Inc. Case Study: FusionFabric.cloud • Integrates trading systems with cloud-based components – OTC derivatives “With GridGain, we have achieved real- – Exchange traded derivatives time processing of massive amounts of – Inflation trade and transaction data, eliminating – Fixed income bottlenecks and enabling us to offer next- – FX/MM generation financial services to our customers.“ – Hybrids -Felix Grevy, Director of – Developing additional Product Management for modules FusionFabric.cloud at Misys

Thank you for joining us. Follow the conversation. www.gridgain.com www.gridgain.com/resources/blog

@gridgain #gridgain #inmemorycomputing @msarrel [email protected]