Hazelcast In-Memory Platform Terry Walters Sr Solutions Architect
Total Page:16
File Type:pdf, Size:1020Kb
THE LEADING IN-MEMORY COMPUTING PLATFORM Hazelcast In-Memory Platform Terry Walters Sr Solutions Architect !1 Hazelcast In-Memory Computing Platform Payment Fraud Customer Edge eCommerce BI ETL/Ingest Use Cases Processing Detection Loyalty Processing … Microservices IoT Cache AI/ML Hazelcast In-Memory Computing Platform Stream & Batch Analytical Data Store Monitoring AI/ML Processing Processing Processing Distributed Data Distributed Streaming Data-at-rest Data-in-motion System of Sources Record APIs Sensors Streams Hadoop Data Lakes … !2 In-Memory Data Grid !3 IMDG Evolution Through Time Open Client Protocol | Java | .NET | C++ | Python | Node.js | Go | HTTP/2 Clients RingBuffer | HyperLogLog | CRDTs | Flake IDs | Event Journal | CP RAFT Subsystem Data Structures Cloud Discovery SPI | Azure | AWS | PCF | OpenShift | IBM Cloud Private | Managed Services Open Cloud Platform JCache | HD Memory | Hot Restart | HotCache Caching j.u.c. | Performance | Security | Rolling Upgrades | New Man Center In-Memory Data Grid 2016 2017 2018 2019 !4 Roadmap: IMDG 3.11 New Enterprise Features Features Description Use Merkle trees to detect inconsistencies in map and cache data. Sync only the different entries after a WAN Replication Consistency consistency check, instead of sending all map and cache entries. Fine-Grained Control over Wan Allow the user finer grained control over the types of Map/Cache events that are replicated over WAN, and also Replication Events provide control as to how they should be processed when received. License Enforcement - Warnings, Grace Different ways on alerting customers/users about expiration, renewal approach. Periods, Flexible Cluster Sizes Members only License Installation Remove license check from Hazelcast IMDG clients for simplification. !5 Roadmap: IMDG 3.11 New Features Features Description Ability to Set TTL for Map and Cache Ability to update TTL without updating the value. Entries Hazelcast Works on Java 8 through to Make sure Hazelcast works on Java 8/9/10/11. 11. Index Statistics Index statistics including index definition, memory cost, index hit stats, creation time etc. Map: Max Idle Timeout per Entry The IMap API should allow to set the max idle timeout per entry, same as we provide for TTL. Exponential Back off on Client Retry Allow a back off strategy for client re-connection attempts. Strategy I/O Pipeline Prevent I/O Threads from becoming blocked. Improve scheduling of 'batch' Operations so the Operation thread will not process multiple operations from the Queries scheduling fairness same batch without other operations getting executed. !6 Hazelcast 3.12 Roadmap !7 Hazelcast IMDG 3.12 In-Memory Computing Platform Java Scala C++ C#/.NET Python Node.js Go Near Cache HD Near Cache OperationsOperations Open Client Network Protocol Clients REST Memcached Clojure (Backward & Forward Compatibility, Binary Protocol) Management Center" Serialization" (JMX/REST) (Serializable, Externalizable, DataSerializable, IdentifiedDataSerializable, Portable, Custom) Web Sessions" Continuous " Security Suite java.util.concurrent Hibernate 2nd Level Cache JCache (Tomcat/Jetty/Generic) HD HD HD Query (Connection, Encryption, Authentication, Authorization, JAAS LoginModule, SocketInterceptor, TLS, OpenSSL, Map MultiMap Replicated Map Set List Queue Topic ReliableTopic Ringbuffer Mutual Auth) APIs HD HyperLogLog Lock/Semaphore AtomicLong AtomicRef ID Gen. Flake ID Gen. CRDT PN Counter WAN Replication Predicate &" Executor Service" (Socket, Solace Systems, One-way, Multiway, Init New Data SQL Query Entry Processor Aggregation Center, Partition Predicate HD & Scheduled Exec Service DR Data Center Recovery, Discovery SPI, Delta Synchronization)xx Low-Level Services API Node Engine" Rolling Upgrades (Threads, Instances, Eventing, Wait/Notify, Invocation) (Rolling Client Upgrades, Rolling Member Upgrades, No Downtime, Compatibility Test Suite) Partition Management" Engine (Members, Lite Members, Master Partition, Replicas, Migrations, Partition Groups, Partition Aware) Cluster Management with Cloud Discovery SPI" Blue/Green Deployments (AWS, Azure, Consul, Eureka, etcd, Heroku, IP List, Apache jclouds, Kubernetes, Multicast, Zookeeper) Networking" (IPv4, IPv6) Automatic Disaster Recovery Fail-Over On-Heap Store High-Density Memory Store (Intel, Sparc) Storage HD Enterprise PaaS Deployment Environments Hot Restart Store (Pivotal Cloud Foundry, Red Hat OpenShift Container Platform, (SSD, HDD) HD IBM Cloud Private) JVM (JDK: 6,7,8,9,10,11 Vendors: Oracle JDK, OpenJDK, IBM JDK, Azul Zing & Zulu) HD Operating" Operating System Hazelcast Striim Hot Cache Env (Linux, Oracle Solaris, Windows, AIX, Unix) (Sync Updates from Oracle DB, MS SQL Server, MySQL and NonStop DB) On Premise AWS Azure Kubernetes Docker VMware Professional Support Enterprise Edition Enterprise HD Edition HD Enterprise HD Edition-Enabled Feature Hazelcast Solution Integrates with Jet !8 Roadmap: IMDG 3.12 New Enterprise Features Features Description Harden the Kubernetes support for all IMDG Enterprise features, and increase First Class Kubernetes Support the operational experience. Hot Restart: Remove IP Address Dependence Hot Restart must be completely independent of the member IP address. Option for Removing Hot Restart Data Add possibility to remove hot-restart data automatically when a Member re-joins Automatically to an active cluster. Add and remove WAN targets (publishers) in Ability to add and remove WAN Targets to a running cluster. running cluster Automatically switch cluster version after Switch to newer cluster version automatically instead of manually change the Rolling Upgrade cluster version. FIPS-140 Validation Make Hazelcast Enterprise FIPS-140 validated. !9 Roadmap: IMDG 3.12 New Enterprise Features Features Description Blue/Green Deployments reduce downtime and risk by running two identical Hazelcast® Enterprise IMDG clusters called Blue and Green. One of the clusters provides production services to clients whilst the other cluster can be upgraded with new application Blue/Green Deployments code. Hazelcast Blue/Green Deployment functionality allows clients to be migrated from one cluster to another without client interaction or cluster downtime. All clients of a cluster may be migrated or groups of clients can be moved with the use of label filtering and black/white lists. Automatic disaster recovery fail-over provides clients connected to Hazelcast® Enterprise Automatic Disaster Recovery Fail-Over IMDG clusters with the ability to automatically fail-over to a Disaster Recovery cluster should there be an unplanned outage of the primary production Hazelcast cluster. !10 Roadmap: IMDG 3.12 - New Features Features Description JSON Serializers Make Hazelcast IMDG to understand JSON format. Also includes fast query over JSON data. YAML Configuration Support Play well with the needs of Cloud environments. Composite Indexes Support Improve performance for queries such as `foo = 1 and bar = 2 and foobar = 3` Attributes for Client Ability to set attributes per Client as we can do it for members. Allow to disable scripting on members Allow users to disable scripting on members to increase security of the product. We introduce a CP sub-system that provides strict correctness at all times even during split brain. This is therefore CP and not AP as with the rest of Hazelcast. The data structures that will be put on top of it are: CP Sub-system • IAtomicLong • ISemaphore • ICountDownLatch • ILock • IAtomicReference !11 Hazelcast Clients Roadmap !12 Clients: Recently Released New Features Features Description .NET Client: Eventually Consistent Near Cache Eventually Consistent near-cache for .NET client. Discovery mechanism for Hazelcast Cloud, enterprise-grade Hazelcast software in the .NET Client: Hazelcast Cloud Discovery Cloud, deployed and managed by the experts. .NET Client: TLS with Mutual Authentication TLS and Mutual Auth support for Hazelcast .NET Client. Ability to redirect the C++ client logs to a file programmatically as in Java client via Logger C++ Client: Logger Redirection implementations like Log4j etc. C++ Client: Statistics C++ Client statistics to be showed in Management Center. C++ Client: ReplicatedMap ReplicatedMap for Hazelcast C++ Client. Python Client: TLS with Mutual Authentication TLS and Mutual Auth support for Hazelcast Python Client. Discovery mechanism for Hazelcast Cloud, enterprise-grade Hazelcast software in the Python Client: Hazelcast Cloud Discovery Cloud, deployed and managed by the experts. Node.js Client: Statistics Ability to provide Node.js client statistics to be showed in Management Center. !13 Clients: Recently Released New Features (cont’d) Features Description Discovery mechanism for Hazelcast Cloud, enterprise-grade Hazelcast software in the Go Client: Hazelcast Cloud Discovery Cloud, deployed and managed by the experts. Go Client: TLS Support with Mutual Authentication TLS and Mutual Auth support for Hazelcast Go Client. Provide Go Client related info and statistics so that Management Center can show them Go Client: Statistics on the Clients page. Go Client: Reliable Topic Reliable Topic for Hazelcast Go Client. Go Client: Custom Credentials Object for Authentication Support for custom credentials object for authentication. !14 Clients: Planned New Features Features Description .NET Client: Reliable Topic Reliable Topic for Hazelcast .NET Client. .NET Client: Set TTL for Map Ability to set TTL without updating the value. C++ Client: Distributed Executor Service Distributed Executor Service from C++