EXPERT ANALYSIS BY MARCOS ALBE, SUPPORT ENGINEER, PERCONA Beyond Relational Databases: A Focus on Redis, MongoDB, and ClickHouse Many of us use and love relational databases… until we try and use them for purposes which aren’t their strong point. Queues, caches, catalogs, unstructured data, counters, and many other use cases, can be solved with relational databases, but are better served by alternative options. In this expert analysis, we examine the goals, pros and cons, and the good and bad use cases of the most popular alternatives on the market, and look into some modern open source implementations. Beyond Relational Databases Developers frequently choose the backend store for the applications they produce. Amidst dozens of options, buzzwords, industry preferences, and vendor offers, it’s not always easy to make the right choice… Even with a map! !# O# d# "# a# `# @R*7-# @94FA6)6 =F(*I-76#A4+)74/*2(:# ( JA$:+49>)# &-)6+16F-# (M#@E61>-#W6e6# &6EH#;)7-6<+# &6EH# J(7)(:X(78+# !"#$%&'( S-76I6)6#'4+)-:-7# A((E-N# ##@E61>-#;E678# ;)762(# .01.%2%+'.('.$%,3( @E61>-#;(F7# D((9F-#=F(*I## =(:c*-:)U@E61>-#W6e6# @F2+16F-# G*/(F-# @Q;# $%&## @R*7-## A6)6S(77-:)U@E61>-#@E-N# K4E-F4:-A%# A6)6E7(1# %49$:+49>)+# @E61>-#'*1-:-# @E61>-#;6<R6# L&H# A6)6#'68-# $%&#@:6F521+#M(7#@E61>-#;E678# .761F-#;)7-6<#LNEF(7-7# S-76I6)6#=F(*I# A6)6/7418+# @ !"#$%&'( ;H=JO# ;(\X67-#@D# M(7#J6I((E# .761F-#%49#A6)6#=F(*I# @ )*&+',"-.%/( S$%=.#;)7-6<%6+-# =F(*I-76# LF6+21+-671># ;G';)7-6<# LF6+21#[(*:I# @E61>-#;"# @E61>-#;)(7<# H618+E61-# *&'+,"#$%&'$#( .761F-#%49#A6)6#@EEF46:1-# !"#"$ ./0-1)H(18-)#M(7# $%&#$:M(;E>-7-#;)7-6<+# NKF-:)5# ##@E61>-#J6I((E# JK#$A.'# LF6+21+-671># D((9F-# @R*7-#;)7-6<#@:6F521+## A6)6#@72+6:+# ##@E61>-#[F4:8# @R*7-# =F(*I## )&'"( .761F-## A6)6c(X# L+9:A%U# %&"#'()*+, L:I-16#;-73-7# ;-671># @Q;# @E61>-# @E61>-# $%&# @126:# S7-6+*7-# @]34(# W4:-+4+# '-6:V16F-#S76M(I4(:# ;EF41-#&61>4:-# &6<<()>A%# A74FF# K7-+)(# %49#;G'# T(7)-N# A6)6# ;14A%# JK==# @+)-74NA%# $%&#$:M(;E>-7-## !"#$%&'( '*14IQ(78+## ;)671(*:)-7# ;G'4)-# S-76I6)6## A6)6#LNEF(7-7# [47-/47I# @E61>-# @E61>-# @E61>-# B-)>7(A6)6# K43()6F#JAU# @E61>-#=6R-:6# =4)*+A%# -"., 45)6( %49#A6)6# S60(# J43-# $<E6F6# @E61>-#J@QG# W*I*# @+)-7# '(99F5# @126:#$:97-+# ;*<(# =F(*I-76# ;@K#;5/6+-#@;L# $%&#K*7-A6)6# /"01")2$3456$ '(941# ;-671># M(7#@:6F521+UI6+>A%# '(9-:)74-+# ;@K#;5/6+-#;G'#@:5X>-7-# 3"45(( % S$%=.# ;EF*:8# &66:6# % '(9'(941# L:)-7E74+-A%# !"#$%&'$#()&'"( ;G7-6<# D-:-76F#E*7E(+-# K(+)97-+ZV'# &417(+(\# H5\# V!a#;(\X67-# .761F-# $%&## ;@K## ;G'#;-73-7# .761F-# S-76I6)6# ;E-146F4+)#6:6F521# K(+)97-;G'# LN6I6)6#K*7-A6)6# J@,@# KAQ# LN6F521+# A6)6/6+-# Z6+Z6Z;-7341-# .71>-+)76)-# K-71(:6#;-73-7# &5;G'# %5#=-:)*75'4:8# &678'(941# =(7)-NA%# @76:9(A%# # @126:#K;G'# V)7-<-A6)6# %49S6/F-+# .74-:)A%# &6746A%## &6746A%# .761F-# $%&# $%&# ;G'## JK#,(:;)(E#;G'# A7*4I# ;b77F# L:)-7E74+-# A6)6/6+-# A%O#$:M(7<4N# ;-73-7# K7(97-++#.E-:LI9-# D76E># L:)-7E74+-# S4/-7(# @126:#T-1)(7# $E-I(#V&'# LN6+(F# A(1*<-:)# &5;G'# $:<-<(75P:-)# .761F-#S4<-+S-:# A6)6/6+-# [6/741# W(9:42(# W-5#36F*-#+)(7-+# @-7(+E48-# T(F)A%# &5;G'#=F*+)-7#=F*+)74N# ;16F-A%# ;E4I-7# S-+(76# +(F4IA%# '*14IA%# W-5#36F*-#I47-1)## S6<4:(# [647=(<# .E-:;)618#S7(3-# A6)6;)6N# J6:IF-7+(18-)# &-<;G'# [F4:9*6F# ,*(A%# A%66;# WN#;5+)-<+# 611-++# = V&'#;-73-7# L:)-7E74+-# $:C:4;G'# J-7(8*# @126:#&6)74N# = A(1*<-:)*<# $:M(/749>)# J6I((E# K(+)97-+# H618+E61-# $%&#$:M(;E>-7-# NA%# =4)74N#K67;)7-6<# Y7486ZDA# ;15FF6A%# 0;(:67# =76)-# A6)(<41# ;16F-@71# =F(*I#A6)6/6+-+# &5;G'#-1(+5+)-<# Y:4A6)6# @E61>-#=6++6:I76# K-71(:6#S(8*A%# ;@K#;5/6+-#$G# =(187(61>A%# S-+(76#ATL# JK#T-7216# @I36:1-I## Y:4T-7+-# ,-("B#J5E-7)6/F-# H468# D((9F-#=F(*I#;G'# ;E678+--# B*+).:-A%# K43()6F#D7--:EF*<U# 1F*+)-74:9U+>67I4:9# T(FI-<(7)# D7--:EF*<#A6)6/6+-# @I6/6+# @E61>-#J%6+-# H-)>4:8A%# &6746A%#&6N;16F-# ,-X#;G'#I6)6/6+-+# @E61>-#@11*<*F(# '-3-FA%# !!"# JK#=F(*I# &(:-)A%## @E61>-#=(*1>A%# H-F62(:6F#A6)6/6+-# D7(3-;)7-6<+# D6f-7# %75)F5)# A6)6#161>4:9# $%&#$&;# D76E>-:-A%# ##S(7(A%# %-78-F-5A%# @F2/6+-#JA%# T&X67-#=(:2:*-:)# ##H63-:A%# J5E-7A-N# @Q;#HA;# @Q;#@*7(76# &6EA# Q686:I6A%# $:+)61F*+)7# @R*7-#;G'# A6)6#974I# K-71(:6## G*6+67A%# @F2/6+-#VA%# D6F-76# A6)6#Q67->(*+-# D((9F-#=F(*I# ;-73-7#M(7## A--E# =F-67A%# A L:94:-# @Q;# A ;-671># ./0-1);)(7-# S4)6:# %49S6/F-# &(:9(A%# H-I4+# Q-/;16F-;G'# @R*7-#;G'# H-I+>4\# D((9F-#=F(*I# ./0-1)H(18-)# A6)6/6+-# $:c*NA%# S-<E($G# ;:(Xc68-# @EEF46:1-+# A6)6+)(7-# &6EHZA%# X4)>#H-I4+# &1./0-1)# A6)6/6+-P1(<# ;)67I(9# &(:9(A%#H-I4+D7--:# D((9F-## !_!_I6)6# ;E61-=*73-# $:Z<-<(75# %49G*-75# &(I*F*+# H-I4+Z)(Z9(# ;)7-6<#E7(1-++4:9# &417(+(\# &(:9(A47-1)(7# H-I4+#'6/+# @Q;# @126:# D76E># $%&## @Q;#LF6+2=61>-# &-<161>-I#=F(*I# $7(:=61>-# LF6+2=61>-# T-7+6:)# L:94:-# =(<E(+-# X4)>#H-I4+# # &(:9('6/# &-<=61>4-7# $:)-7;5+)-<+# .:)()-N)#D76E>A%# H-I4+#'6/+# H-I4+#'6/+# =61>?# L:)-7E74+-#=F*+)-7# H-I4+#=F(*I# -,./01$12"()&'"( 7##.+899 J5E-7976E>A%# @R*7-#H-I4+## L>161>-# $:C:4;E6:# @FF-97(976E># ./0-1)H(18-)# =61>-# %49&-<(75# H-I#J6)#B%(++# :;5)<+<")=7>=(*9 @R*7-#A(1*<-:)A%# L# $:C:4)-D76E># &-<161>-I# T67:4+>#=61>-# ,=61>-# A6)6#D74I# L# ./0-1234)5# $%&#=F(*I6:)# $%&#=F(*I6:)#'(16F# =(*1>/6+-# +#"#<?('?#7<? D74ID64:#$:Z&-<(75# K43()6F# ;16F-.*)# A6)6#[6/741# S$%=.# .761F-## $%&## @Q;#A5:6<(A%# ;(\X67-# D-<[47-# @123-;E61-+# J6R-F16+)#=(>-7-:1-# -V)7-<-#;16F-# @"#"A"+<?&"0@+=".<$ $%&#'()*+#,()-+# &69:-)(A%#.761F-# @Q;## ,(;G'# ;4<EF-A%# @E61>-#$9:4)-# @E61>-#D-(I-# D496;E61-+#V@K# $:C:4=61>-# S6R5D74I# ^#O_!`#/5#"a!#H-+-671>#''=P## !# O# d# "# a# `# @FF#749>)+#7-+-73-I## Relational databases are generally the go-to option as they are flexible and well-known. But, when used for the wrong purpose, they can end up being a bottleneck down the road. To help you make the right choice for your business we have focused on the three most popular database alternatives: Key-value, Document, and Columnar, and have given an overview of situations when relational databases might be a poor choice. In this paper, we will consider alternatives to relational databases, and examine Key-Value databases in more details. Why Not Relational Databases? MySQL, PostgreSQL, and all other relational databases work well for general purposes. They can represent any data model and this flexibility is one of the reasons that they are very popular. You can represent the necessary data structures required for a social graph, for an OLAP cube, for a work queue, for an Entity-Key-Value model, and many more. And these will work… but the relational model is not always the best fit for structures such as these. Relational databases don’t shine here due to the way the data is represented, because of performance overhead, or for other reasons that are inherent to the relational paradigm. The bottomline is that they will fail to scale for many workloads that are commonplace in the industry. Another reason for relational databases popularity is SQL. This follows a well-defined standard, so a single expert can apply the same knowledge through many systems. But, much as I love SQL, I recognize that it is sometimes hard to grasp for developers. It is not hard to end up with JOIN bombs, and the rigidity of the schema is a pain- point most of us are likely to have suffered during early development stages. In some environments, employing a query language more naturally integrated with the programming language, and with a flexible schema, would be highly valued. These are two key features of Document stores. Another feature that helped popularize MySQL is that it provides ACID properties and referential integrity, which guarantees strong consistency for our data. But, ACID and referential integrity are concepts that imply paying a performance price for. This is because all the locking mechanisms, and the intrinsic waits for IO that are needed to provide such guarantees, can become extremely costly. This is one of the main reasons that relational databases fail to scale for some workloads. So, in cases where data can be rebuilt, or is known to have few relationships, an in-memory Key-Value database might be a better fit. Finally, we know for a fact that MySQL can store 70TB of data, or more in a single instance. We even have a customer with 34TB in a single table! So, yes: it can handle these volumes of data… as long as you don’t really need to read all of it! In which case, I question why you stored all that data in the first place? The problem with relational databases is the way that they store data on disk, because to read a single column you must either index it (paying the associated maintenance costs), or read all of the columns.