SED 863 Transcript EPISODE 863 [INTRODUCTION] [00:00:00] JM: FoundationDB is a multi-model distributed key-value store. It is fully ACID compliant and horizontally scalable. FoundationDB is not usually used directly by an application developer. FoundationDB is a foundational building block for higher-level distributed systems, such as the metadata storage system for a data warehousing tool like Snowflake. Ryan Worl is a software engineer who specializes in FoundationDB. He joins the show to discuss the architecture of FoundationDB including the roles of different server components and the read and write path of FoundationDB. We also talk about applications of FoundationDB and how it compares to storage engines such as RocksDB and databases such as CockroachDB and Spanner. If you want to find all of our episodes about distributed systems and large-scale databases, you can check out the Software Daily app for iOS. It includes all 1,000 of our old episodes as well as related links, and greatest hits, and topics, and reading material. You can comment on episodes. You can have discussions with members of the community and you can become a paid subscriber. You can get ad-free episodes of Software Engineering Daily by going to softwareengineeringdaily.com/subscribe. If you're looking for help with mobile and web development, I recommend checking Altalogy. They’re the company that has helped us build the newest version of the Software Daily app for iOS, and the Android app is on its way. It’s another 2 to 4 weeks away, and I'm really excited to have two awesome apps for Software Engineering Daily that will soon be in the App Store. The one for iOS is already there and it's quite good. So please check it out if you're interested. [SPONSOR MESSAGE] [00:02:02] JM: DigitalOcean is a simple, developer friendly cloud platform. DigitalOcean is optimized to make managing and scaling applications easy with an intuitive API, multiple storage options, integrated firewalls, load balancers and more. With predictable pricing and © 2019 Software Engineering Daily !1 SED 863 Transcript flexible configurations and world-class customer support, you’ll get access to all the infrastructure services you need to grow. DigitalOcean is simple. If you don’t need the complexity of the complex cloud providers, try out DigitalOcean with their simple interface and their great customer support, plus they’ve got 2,000+ tutorials to help you stay up-to-date with the latest open source software and languages and frameworks. You can get started on DigitalOcean for free at do.co/sedaily. One thing that makes DigitalOcean special is they’re really interested in long-term developer productivity, and I remember one particular example of this when I found a tutorial in DigitalOcean about how to get started on a different cloud provider. I thought that really stood for a sense of confidence, and an attention to just getting developers off the ground faster, and they’ve continued to do that with DigitalOcean today. All their services are easy to use and have simple interfaces. Try it out at do.co/sedaily. That’s the D-O.C-O/sedaily. You will get started for free with some free credits. Thanks to DigitalOcean for being a sponsor of Software Engineering Daily. [INTERVIEW] [00:04:02] JM: Ryan Worl, welcome to the show. [00:04:04] RW: Nice to be here, Jeff. Thank you. [00:04:05] JM: We’re going to talk about FoundationDB today, and I'd like to start by discussing the applications of FoundationDB. What would I use FoundationDB to build? [00:04:16] RW: That is a very tough question, because it's so applicable to so many different problem domains and you’d be able to use it in so many different ways. But just the most basic use case for it would be you need some key-value storage that you need to be able to transact across any of the keys at any time. So, it presents itself as a single system to you and you can do transactions over any keys that you want, but it's really important to know that you’re not designed to use it just in that way. You're supposed to build stuff on top that takes advantage of that to do fancier things. © 2019 Software Engineering Daily !2 SED 863 Transcript [00:04:52] JM: Is that where the name FoundationDB comes from, the fact that it is meant to be used as a foundation to build more abstract systems? [00:05:00] RW: Yes, as far as I know, that is the conceit there where the name comes from. The company, it would be 10-years-old today, or not today, but this year if it were still a private company, and that's the origin story as far as I know. But that was a long time ago, and I wasn't there. [00:05:19] JM: Indeed. So, there are many different key-value stores in the world. People who are listening to this are probably familiar with in-memory systems like Redis. Well, it’s not entirely in-memory, or systems like MongoDB. It's a document storage system. There're things like Spanner and CockroachDB. Well, I guess those are more SQL transactionality. You’ve got RocksDB and maybe we can go through these comparisons a little bit later. But before we get to the comparisons, can you give a more narrow definition of when and why I would use FoundationDB even as a building block for a different system? [00:05:59] RW: Yeah. So the important criteria you would need to evaluate whether or not FoundationDB is good for your specific use case is if you are expecting the storage to exceed the capacity of a single machine. That would be a good place to start, because it's a distributed key-value store designed to run on multiple machines. You'd also want to know that you need transactions. There are certain use cases, like an obvious one is purely read-only data where transactions will not be particularly useful. But once you get past those two things, basically, you need something that has the capacity to grow beyond a single machine and you need transactions. Those would be about the two things I would say. There are specific different issues when it comes to performance and the cost of using FoundationDB in terms of maybe it's not as efficient as some other system that you could compare it against. But those more in the weeds, and I think that if you know that you need both scale out storage and a number of requests per second that you can process and you need transactions, those would be two good criteria to start with. © 2019 Software Engineering Daily !3 SED 863 Transcript [00:07:08] JM: All right. Well, it's clear to me at this point that we’re going to need to delve into the subtleties of FoundationDB in order to truly understand when and why we should be using it. But just so we can anchor our understanding around some prototypical use case, can you tell me one or two places where you have seen FoundationDB would be particularly useful. I don’t know if it’s a shopping cart application, or a videogame, data management, just some concrete application for us to anchor our understanding around. [00:07:43] JM: Yeah. So, I think a good example that has been talked about relatively publicly, including at the FoundationDB Summit, is the use case from Snowflake, which is the cloud data warehouse company. They use FoundationDB to manage the metadata for their cloud data warehouse. By metadata, I basically mean – This is a bit in the weeds of their specific architecture. But they store the data for the tables and the data warehouse in S3, and the metadata about what files belonged to what table is in FoundationDB. Alongside that, they also store all of the kind of generic OLTP type data that people are familiar with, like user records and accounts and permissions and things like that. Those are all stored in FoundationDB as well. So it's a combination of the metadata for the actual data warehouse itself along with all of the supporting cloud services on top that are basically access control and management, things like that. [00:08:44] JM: Wow! Now I'm seeing Snowflake in a totally different light. We did a show about Snowflake a while ago, and if I recall, one thing that makes Snowflake pretty good, and this is something you want out of a data warehouse, is you've got your piles and piles of data and you want fast access to that data. One way to get fast access to your data is to index it in a bunch of different ways. An index gives you faster access, faster lookup time. I think an index is kind of one form of metadata, data about the data. Like the data that you're actually indexing and building metadata around is the data that you're throwing in your data warehouse, but you need a way to comprehend that data, and that is the metadata store that will exist in a data warehouse. So it makes sense to me that a fast metadata store that is eventually – I think there’s kind of an eventual consistency element to FoundationDB. It's not maybe a strongly consistent as – [00:09:53] RW: Oh, no.