Getting Started with Apache® Ignite™ as a Distributed Database
VALENTIN KULICHENKO Lead Architect GridGain Systems, Inc.
© 2018 GridGain Systems, Inc. Agenda
• Apache Ignite as a Distributed Database • Connectivity • Data Definition Language • Data Manipulation Language • Collocated and Non-collocated Joins • Under the Hood • Durable Memory Architecture • Native Persistence • Data Storage Configuration • Data Regions • Data Eviction and Page Replacement • Performance Tips • Demo • Q&A
© 2017 GridGain Systems, Inc. What is Apache Ignite?
Financial Telco Travel & E-Commerce Pharma & IoT Services Logistics Healthcare
SQL Key/Value Transactions Compute Services Streaming ML
Memory-Centric Storage
Ignite Native Persistence Third-Party Persistence (Flash, SSD, Intel 3D XPoint) (RDBMS, HDFS, NoSQL)
© 2017 GridGain Systems, Inc. Ignite as a Distributed Database
© 2017 GridGain Systems, Inc. Distributed SQL
Cross-platform Java .NET C++ BI Tools Compatibility
SELECT, UPDATE, DDL, DML Support JDBC ODBC SQL API INSERT, MERGE, DELETE, CREATE and ALTER
Apache Ignite Cluster
Indexes in DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY Dynamic RAM or Disk Scaling
Server Node Server Node Server Node
© 2017 GridGain Systems, Inc. Connectivity
• JDBC // Register JDBC driver. • ODBC Class.forName("org.apache.ignite.IgniteJdbcThinDriver"); // Open the JDBC connection. • REST Connection conn = DriverManager.getConnection("jdbc:ignite:thin://192.168.0.50"); • Java, .NET and C++ APIs
./sqlline.sh --color=true --verbose=true -u jdbc:ignite:thin://127.0.0.1/
© 2017 GridGain Systems, Inc. Data Definition Language
• CREATE/DROP TABLE CREATE TABLE `city` ( `ID` INT(11), • CREATE/DROP INDEX `Name` CHAR(35), `CountryCode` CHAR(3), • ALTER TABLE `District` CHAR(20), `Population` INT(11), • Changes Durability PRIMARY KEY (`ID`, `CountryCode`) ) WITH "template=partitioned, backups=1, affinityKey=CountryCode"; • Ignite Native Persistence
https://apacheignite-sql.readme.io/docs/ddl
© 2017 GridGain Systems, Inc. Data Manipulation Language
• ANSI-99 specification
• Fault-tolerant and consistent SELECT country.name, city.name, MAX(city.population) as max_pop FROM country JOIN city ON city.countrycode = country.code • INSERT, UPDATE, DELETE WHERE country.code IN ('USA','RUS','CHN') • SELECT GROUP BY country.name, city.name ORDER BY max_pop DESC LIMIT 3; • JOINs • Subqueries
https://apacheignite-sql.readme.io/docs/dml
© 2017 GridGain Systems, Inc. Collocated Joins
Ignite Node
Toronto Montreal 2 Canada Ottawa Calgary 1
Ignite Node 3 2
Mumbai India New Delhi 1. Initial Query 2. Query execution over local data 3. Reduce multiple results in one
© 2017 GridGain Systems, Inc. Non-Collocated Joins
Ignite Node
Toronto Mumbai 2 Canada Calgary 1
Montreal 3 Mumbai Ottawa
4 2 Ignite Node
Montreal India Ottawa 1. Initial Query 2. Query execution (local + remote data) New Delhi 3. Potential data movement 4. Reduce multiple results in one
© 2017 GridGain Systems, Inc. Under the Hood
© 2017 GridGain Systems, Inc. Durable Memory
Automatic Defragmentation Ignite Cluster
Predictable memory Off-heap Removes DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY consumption noticeable GC pauses
Fully Transactional Stores Superset (Write-Ahead Log) of Data
Instantaneous Server Node Server Node Server Node Restarts
© 2017 GridGain Systems, Inc. Ignite Native Persistence
Server Node
1. Update
2. Persist Write-Ahead Log
3. Ack
RAM 4. Checkpointing Partition File 1
Partition File N
© 2017 GridGain Systems, Inc. Data Storage Configuration
© 2017 GridGain Systems, Inc. Single Data Region
Default Data Region (4 GB)
Cache 1 Cache 2
Cache 3 Cache 4
© 2017 GridGain Systems, Inc. Multiple Data Regions
DATA_REGION_4_GB Large Cache 1
Small Cache
Large Cache 2
© 2017 GridGain Systems, Inc. Enabling Data Eviction
• Eviction Modes: • DISABLED • RANDOM_LRU • RANDOM_2_LRU
© 2017 GridGain Systems, Inc. Enabling Native Persistence
• WAL Modes: • DEFAULT (full sync) • LOG_ONLY • BACKGROUND • NONE
© 2017 GridGain Systems, Inc. Performance Tips
• Leave memory space for heap and OS • Use production ready SSDs • Have separate physical disk for WAL • Advanced data storage configuration: • Change page size • Enable write throttling • Increase checkpoint buffer size
https://apacheignite.readme.io/docs/durable-memory-tuning
© 2017 GridGain Systems, Inc. Demo
© 2017 GridGain Systems, Inc. Resources • GridGain in-memory computing resources – Learning: https://www.gridgain.com/resources/in-memory-computing-resources – In-Memory Computing Summit: https://www.imcsummit.org/ – Local Events: https://www.gridgain.com/company/news/events – Customer Stories: https://www.gridgain.com/customers/customers-and-industries • Getting started with Apache Ignite: – https://ignite.apache.org/ – Getting Started doc: https://apacheignite.readme.io/docs/getting-started – GridGain Web Console (free): https://www.gridgain.com/resources/manage – Apache Ignite User Forum: http://apache-ignite-users.70518.x6.nabble.com/ – Stack Overflow: https://stackoverflow.com/questions/tagged/ignite • Getting started with GridGain: – https://www.gridgain.com/ – https://www.gridgain.com/resources/download
© 2018 GridGain Systems, Inc. HAPPY VALENTIN’S DAY!
VALENTIN KULICHENKO Lead Architect GridGain Systems, Inc.
© 2018 GridGain Systems, Inc.