Be.Er Together – Apache Ignite & Apache Spark

Be.Er Together – Apache Ignite & Apache Spark

Be#er Together – Apache Ignite & Apache Spark Fast Data Meets Open Source DMITRIY SETRAKYAN GridGain Founder & Chief Product Officer Apache Ignite PMC VALENTIN KULICHENKO GridGain Lead Architect Apache Ignite PMC hp://ignite.apache.org @apacheignite @dsetrakyan Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Agenda • Apache Ignite(tm) Overview • Data Grid • Par<<oning Schemes • SQL • Shared Memory Layer • Share Spark RDDs • In-Memory File System • DevOps: Yarn and Mesos • Faster MapReduce & Hive • Ignite MapReduce • Demo - Shared Ignite RDDs • Demo - SQL using Apache Zeppelin • Q & A Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Apache Ignite - We Are Hiring! • Very Ac<ve Community • Great Way to Learn Distributed Compu<ng • How To Contribute: – hRps://ignite.apache.org/community/ contribute.html#contribute – hRps://cwiki.apache.org/confluence/ display/IGNITE/How+to+Contribute Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Apache IgniteTM In-Memory Data Fabric: Strategic Approach to IMC • Supports Applications of various types and languages • Open Source – Apache 2.0 • Simple Java APIs • 1 JAR Dependency • High Performance & Scale • Automatic Fault Tolerance • Management/Monitoring • Runs on Commodity Hardware • Supports existing & new data sources • No need to rip & replace Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Apache Ignite In-Memory Data Fabric Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Why Share State in Spark? • Long Running Applicaons – Passing State Between Jobs • Disk File System (HDFS?) – Convert RDDs to Disk Files and Back – Argh#$% • Share RDDs In-Memory – Nave Spark API – Nave Spark Transformaons Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Why Ignite Data Grid? • In-Memory Key-Value Store – Good for Caching Tuples • Foundaon for Shared Memory State – IgniteRDD is based on Data Grid – Ignite File System is based on Data Grid • On-Heap & Off-Heap Memory • In-Memory Indexes – Fast SQL • Built for High Throughput and Low Latencies Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Data Grid: JCache (JSR 107) • Key-Value Store (JCache, JSR 107) – In-Memory Key-Value Store – Basic Cache Operaons – ConcurrentMap APIs – Collocated Processing (EntryProcessor) – Events and Metrics – Pluggable Persistence • Data Grid – ACID Transac<ons – SQL Queries (ANSI 99) – In-Memory Indexes – On-Heap & Off-Heap Memory – Automac RDBMS Integraon Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Data Grid: Distributed Caching Par<<oned Cache Replicated Cache Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Data Grid: Ad-Hoc SQL (ANSI 99) • ANSI-99 SQL • Always Consistent • Fault Tolerant • In-Memory Indexes (On-Heap and Off-Heap) • Automac Group By, Aggregaons, Sor<ng • Cross-Cache Joins, Unions, etc. • Ad-Hoc SQL Support Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. SQL Cross-Cache GROUP BY Example Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Apache Ignite for Spark and Hadoop Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. DevOps: IntegraZon with Yarn and Mesos • Automac Resource Management • Easy Data Center Installaon • Easy Data Center Configuraon • On-Demand Elas<city Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Share RDDs Across Spark Jobs • IgniteRDD Deployment Modes – Share RDD across tasks on the host – Share RDD across tasks in the applicaon – Share RDD globally – Embedded vs External Deployments • Faster SQL – In-Memory Indexes – SQL on top of Shared RDD Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. IgniteContext • Main Entry Point from Spark to Ignite • Specify Different Ignite Configuraons • Embedded vs External Deployments – Client vs Server Modes Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. IgniteRDD • Implementaon of SparkRDD • Mutable (unlike nave RDDs) • Par<<oned over Ignite Par<<oned Caches • Indexed SQL – Spark only does Full Scans – Indexes are 1000x faster Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Ignite In-Memory File System • Ignite In-Memory File System (IGFS) – Hadoop-compliant – Easy to Install – On-Heap and Off-Heap – Caching Layer for HDFS – Write-through and Read-through HDFS – Performance Boost Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. Apache Ignite Roadmap • Non-Collocated Joins (released in 1.7) • Data Modificaon Language (DML in 2.0) – INSERT, UPDATE, DELETE • Data Defini<on Language (DDL in 2.1) – CREATE, ALTER, DROP • More IGFS Performance • Nave Data Frame Integraon Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. InteracZve SQL with Apache Zeppelin Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. ANY QUESTIONS? Thank you for joining us. Follow the conversaon. hRp://www.ignite.apache.org @apacheignite @dsetrakyan Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache So8ware Foundaon in the United States and/or other countries. .

View Full Text

Details

  • File Type
    pdf
  • Upload Time
    -
  • Content Languages
    English
  • Upload User
    Anonymous/Not logged-in
  • File Pages
    20 Page
  • File Size
    -

Download

Channel Download Status
Express Download Enable

Copyright

We respect the copyrights and intellectual property rights of all users. All uploaded documents are either original works of the uploader or authorized works of the rightful owners.

  • Not to be reproduced or distributed without explicit permission.
  • Not used for commercial purposes outside of approved use cases.
  • Not used to infringe on the rights of the original creators.
  • If you believe any content infringes your copyright, please contact us immediately.

Support

For help with questions, suggestions, or problems, please contact us