Intro to Redis Streams IMCSUMMIT - NOVEMBER 2019 | DAVE NIELSEN • What is a data stream? • What are the typical challenges you face while managing data streams? • What is Redis Stream? • How Redis Stream addresses the challenges? • How to use Redis Streams? 2 Who We Are Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source home and commercial provider of Redis Enterprise technology, platform, products & services. 3 Redis Top Differentiators 1 2 3 Performance Simplicity Extensibility NoSQL Benchmark Redis Data Structures Redis Modules Strings Sets Bitmaps Sorted Sets Bit field Geospatial Indexes Hashes Hyperloglog Lists Streams 4 Data Stream Use Cases 5 What is a Data Stream? 6 Simplest form of a Data Stream Producer Consumer 7 But Often You Have M Producers and N Consumers Producer 1 Consumer 1 Producer 2 Consumer 2 Producer 3 Consumer 3 Producer m Consumer n 8 It’s a bit more complex then that 9 Data Stream Components Message Broker Ingest Stream or Output Stream Analytics Stream Processor Store the data and Collect large perform analytical IoT volume of data Transform Push the data to operations such as arriving in high incoming data the consumers predictive analytics, velocity into one or ML training, ML more formats Allow backward classification and Store the data as required by looking queries to categorization, Activity logs temporarily the consumers pull data recommendations, pattern matching, etc. Messages 10 11 Redis Supports All the Functions of a Data Stream Message Broker Ingest Stream or Output Stream Analytics Stream Processor Redis • Streams • Streams • Streams • Sorted Sets • Pub/Sub • Pub/Sub • Sorted Sets • Sets • Lists • Lists + • Geo • Spark Connector • Bitfield • ODBC/JDBC Connector + • RedisGraph • RediSearch 12 13 Understanding Redis Streams 14 Let’s look at the challenges first 15 How to Support a Variety of Consumers Messaging Real-time Producer Analytics Real-time or Periodic Lookup Data Backup Periodic Read Consumers 16 The Catch Up Problem Producer Image Processor Arrival Rate: 500/sec Consumption Rate: 100/sec Redis Stream Fast Slow Backlog 17 How to Address The Catch Up Problem? Image Processor Image Processor Producer Image Processor Arrival Rate: 500/sec Redis Stream Image Processor Image Processor Consumption Rate: 500/sec Scale Out New Problem: Mutual Exclusivity 18 Data Recovery from Consumer Failures Image Processor Image Processor Producer Image Processor Arrival Rate: 500/sec Redis Stream Image Processor Image Processor Consumption Rate: 500/sec Scale Out Yet Another Problem: Failure Scenarios 19 Connection Failures Image Processor Image Processor Producer Image Processor Redis Stream Image Processor Image Processor The solution must be resilient to failures 20 Here comes Redis Streams 21 What is Redis Streams? Pub/Sub Lists Sorted Sets It is like Pub/Sub, but It is like Lists, but decouples It is like Sorted Sets, but with persistence producers and consumers asynchronous + • Lifecycle management of streaming data • Built-in support for timeseries data • A rich choice of options to the consumers to read streaming and static data • Super fast lookback queries powered by radix trees • Automatic eviction of data based on the upper limit 22 1. It enables asynchronous data exchange between producers and consumers Producer Messaging Consumer 23 2. You can consume data in real-time as it arrives or lookup historical data Messaging Producer Analytics Data Backup Consumers 24 3. With consumer groups, you can scale out and avoid backlogs Image Processor Image Processor Producer Image Processor Arrival Rate: 500/sec Redis Stream Image Processor Image Processor Consumption Rate: 500/sec 25 4. Simplify data collection, processing and distribution to support complex scenarios Deep Learning-based Classification Classifier 1 XREADGROUP Classifier 2 Producer 1 XADD XACK Producer 2 Classifier n Producer 3 Consumer Group Producer m Messaging Analytics XREAD Data Backup 26 Consumers Redis Streams Commands Quick Reference Guide https://redislabs.com/docs/getting-started-redis-streams/ 27 Commands: XADD, XREAD Asynchronous producer-consumer message transfer Producer Messaging Consumer XADD XREAD XADD mystream * name Anna XREAD COUNT 100 STREAMS mystream 0 XADD mystream * name Bert XADD mystream * name Cathy XREAD BLOCK 10000 STREAMS mystream $ 28 Commands: XRANGE, XREVRANGE Lookup historical data Messaging Producer Analytics XADD Data Backup XREAD Consumers XRANGE XREVRANGE XRANGE mystream 1518951123450-0 1518951123460-0 COUNT 10 XRANGE mystream - + COUNT 10 29 Scaling out using consumer groups Image Processor Image Processor Producer Image Processor Arrival Rate: 500/sec Redis Stream Image Processor Consumption Rate: 500/sec 30 Scaling out using consumer groups Consumer Group Consumer 1 Image Processor Consumer 2 Image Processor Producer Consumer 3 Image Processor Arrival Rate: 500/sec Redis Stream Unconsumed List Consumer n Image Processor Consumers Consumption Rate: 500/sec XADD XGROUP XREADGROUP XACK XPENDING XCLAIM 31 Command: XGROUP CREATE Create a consumer group mygroup mystream Alice App A a b c d e a b c d e Bob App B Unconsumed List XGROUP CREATE XGROUP CREATE mystream mygroup $ MKSTREAM 32 Command: XREADGROUP Read the data mygroup d e Alice App A a c b Bob App B Unconsumed List XREADGROUP XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream > XREADGROUP GROUP mygroup COUNT 2 Bob STREAMS mystream > 33 Command: XACK Consumers acknowledge that they consumed the data mygroup XACK Alice App A a c b Bob App B Unconsumed List XACK mystream mygroup 1526569411111-0 1526569411112-0 34 Command: XREADGROUP Repeat the cycle mygroup a Alice App A c b Bob App B Unconsumed List XREADGROUP XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream > 35 How to claim the data from a consumer that failed while processing the data? mygroup Alice App A c b Bob App B Unconsumed List XCLAIM mygroup b c Alice App A Bob App B Unconsumed List 36 Commands: XPENDING, XCLAIM Claim pending data from other consumers mygroup b c Alice App A Bob App B Unconsumed List XPENDING mystream mygroup - + 10 Bob XCLAIM mystream mygroup Alice 0 1526569411113-0 1526569411114-0 37 Now is the time to 38 Sample Use Case: Social Media Analytics Sentiment analyZers Consumer is x times slower than the a producer b XREADGROUP x Language classifier Redis Streams Influencer classifier Location classifier XREAD Real-time reports 39 Demo Twitter Ingest Stream Influencer Classifier Redis Streams https://github.com/redislabsdemo/IngestRedisStreams 40 Redis Clients that Support Redis Streams Java Jedis https://github.com/xetorthio/jedis Lettuce https://lettuce.io/ Python redis-py https://github.com/andymccurdy/redis-py JavaScript ioredis https://github.com/luin/ioredis node_redis https://github.com/NodeRedis/node_redis .NET StackExchange.Redis https://github.com/StackExchange/StackExchange.Redis Go redigo https://github.com/gomodule/redigo go-redis https://github.com/go-redis/redis C Hiredis https://github.com/redis/hiredis PHP predis https://github.com/nrk/predis phpredis https://github.com/phpredis/phpredis Ruby redis-rb https://github.com/redis/redis-rb 41 Questions ? ? ? ? ? ? ? ? ? ? ? Thank you! redislabs.com 43.