Appears in the proceedings of the 12th ACM Interna6onal Systems and Storage Conference (ACM SYSTOR '19) Kurma: Secure Geo-Distributed Multi-Cloud Storage Gateways Ming Chen and Erez Zadok Stony Brook University {mchen,ezk}@fsl.cs.sunysb.edu ABSTRACT 1 INTRODUCTION Cloud storage is highly available, scalable, and cost-ecient. Cloud storage has obvious security challenges since tenants Yet, many cannot store data in cloud due to security concerns do not control the physical media of their data. Cloud users and legacy infrastructure such as network-attached storage also suer from long latency when data need to be frequently (NAS). We describe Kurma, a cloud storage gateway system transferred between branch oces and remote clouds. On- that allows NAS-based programs to seamlessly and securely premises cloud storage gateways alleviate these problems by access cloud storage. To share les among distant clients, keeping sensitive data in private storage media, safeguarding Kurma maintains a uniedle-system namespace by replicat- against attacks in public clouds, and caching hot data locally. ing metadata across geo-distributed gateways. Kurma stores Kurma is a cloud-based le system designed for organiza- only encrypted data blocks in clouds, keeps le-system and tions that have several to dozens of physical branch oces security metadata on-premises, and can verify data integrity and want to share data securely among the oces. Kurma and freshness without any trusted third party. Kurma uses uses geo-distributed cloud-storage gateways (one per oce) multiple clouds to prevent cloud outage and vendor lock-in. to collectively provide a uniedle-system namespace for Kurma’s performance is 52–91% that of a local NFS server all oces. Each Kurma gateway is physically a coordinated while providing geo-replication, condentiality, integrity, cluster of on-premises machines, and provides NFS services and high availability. to local clients. Kurma gateways use multiple public clouds as back-ends and use on-premises storage to cache hot data. CCS CONCEPTS Kurma considers the on-premises gateways trusted, and the public clouds untrusted. Kurma stores only encrypted • Security and privacy → Management and querying of and authenticated le data blocks on clouds; it keeps all sen- encrypted data; • Computer systems organization → sitive metadata in trusted gateways, including le-system Cloud computing; metadata, encryption keys, and integrity metadata of data blocks. Many cloud object stores are eventually consistent, KEYWORDS meaning they may return stale data [4, 6, 59] as with replay Multi-cloud, cloud storage gateways, storage security attacks. Kurma eciently detects stale data using timestamps and versions. Kurma has a simple and secure key manage- ACM Reference format: ment scheme that does not need any trusted third parties. Ming Chen and Erez Zadok. 2019. Kurma: Secure Geo-Distributed Kurma stores data in multiple clouds to tolerate cloud fail- Multi-Cloud Storage Gateways. In Proceedings of The 12th ACM In- ures. Kurma increases data availability across clouds using ternational Systems and Storage Conference, Haifa, Israel, June 3–5, 2019 (SYSTOR’19), 12 pages. replication [61], erasure coding [43, 44], or secret sharing [35, https://doi.org/10.1145/3319647.3325830 46]. With secret sharing, Kurma provides an additional level of security such that one compromised cloud cannot recover any part of the data. Kurma supports AWS S3, Azure Blob Permission to make digital or hard copies of all or part of this work for Store, Google Cloud Storage, and Rackspace Cloud Files. personal or classroom use is granted without fee provided that copies are not Each Kurma gateway maintains a copy of the whole le- made or distributed for prot or commercial advantage and that copies bear system metadata, so that it can still serve local clients after this notice and the full citation on the rst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with a network partition. Kurma minimizes its metadata size by credit is permitted. To copy otherwise, or republish, to post on servers or to using large data blocks and compression. Metadata changes redistribute to lists, requires prior specic permission and/or a fee. Request made by a Kurma gateway are asynchronously replicated to permissions from [email protected]. all other gateways using Hedwig [54], a publish-subscribe SYSTOR’19, June 3–5, 2019, Haifa, Israel system that provides guaranteed-delivery of large amounts of © 2019 Association for Computing Machinery. data across the Internet. Kurma provides NFS close-to-open ACM ISBN 978-1-4503-6749-3/19/06...$15.00 https://doi.org/10.1145/3319647.3325830 consistency among clients connected to a common Kurma gateway. For clients across geo-distributed gateways, Kurma Clients provides FIFO consistency [36], trading oconsistency for Secure higher performance and availability. Thus, operations in dif- Region Gateway 1 ferent gateways may be conicting. Kurma detects conicts storage and provides resolution for common types of conicts. metadata Kurma uniquely combines many advanced and industry- metadata proven techniques from prior studies [8, 16, 22, 25, 35, 45, 54] S3 to achieve high security and availability in a simple way. Azure The rest of this paper is organized as follows. §2 discusses Region Google Region 2 3 the design. §3 describes the implementation of our Kurma Rackspace prototype. §4 evaluates its performance. §5 discusses related Untrusted work. §6 concludes and discusses future work. Public Clouds metadata 2 DESIGN Figure 1: Kurma architecture with three gateways. We present Kurma’s threat model, design goals, architecture, Each dashed oval represents an oce in a region, metadata management, security, multi-cloud redundancy, where there are clients and a Kurma gateway. Each consistency, and persistent caching. gateway is a cluster of coordinated machines rep- resented by three inter-connected racks. The green 2.1 Threat Model arrows connecting gateways are private secret chan- nels for replicating le-system and security metadata. Our threat model reects the settings of an organization with Each gateway has local storage for cache. Clocks of all oces in multiple regions, and employees in these oces machines are synchronized using NTP [41]. store and share les via Kurma gateways. Public clouds are not trusted. Data stored in cloud may be leaked or tampered by malicious tenants and compro- mised providers. Transferring data to clouds is vulnerable 2.3 Architecture to man-in-the-middle attacks. Eventually-consistent clouds may return stale data. Cloud outage happens [3, 57]. Figure 1 shows Kurma’s architecture. Kurma uses trusted Clients are trusted. Clients represent internal employees on-premises machines and storage to build gateways that and are generally trustworthy with proper access control. seamlessly protect data in clouds. For strong security, Kurma Kurma supports NFSv4 with advanced ACLs [50]. uses public clouds to store only encrypted le blocks, but Kurma gateways are trusted. They provide consolidated not any metadata. Instead, Kurma store all metadata, includ- security services. Kurma gateways can authenticate each ing le block mapping and le keys, in trusted on-premises other and establish trusted secret channels. Each gateway is machines; and uses a secret channel between each pair of a cluster of computers that ts in one access-controlled room. gateways for distributing metadata among gateways. For high availability, Kurma uses multiple clouds as back- ends. Kurma stores le-system metadata in ZooKeeper [25], 2.2 Design Goals which is distributed and highly available. Each Kurma gate- Kurma has four goals in descending order of importance: way runs a separate ZooKeeper instance that stores a full replica of the whole le-system metadata; thus, outage in (1) Strong security: Kurma should ensure condential- one region will not bring down gateways in other regions. ity, integrity, and freshness to both le data and meta- For high performance, each Kurma gateway uses a per- data while outsourcing storage to clouds. sistent write-back cache to avoid the long latency of cloud (2) High availability: Kurma should have no single accesses. Kurma replicates metadata asynchronously among point of failure, and be available despite network gateways detecting and resolving conicts as needed. partitions and outage of a small subset of clouds. Tobeexible, Kurma supports three redundancy mech- (3) High performance: Kurma should minimize the anisms when storing data in multiple clouds: replication, performance penalty of its security features, and erasure coding, and secret sharing. They enable a wide range overcome the high latency of remote cloud storage. of trade-os among availability, performance, and costs. (4) High exibility: Kurma should be congurable in Figure 2 shows the Kurma servers and their components. many aspects to support exible trade-oamong Each gateway has three types of servers. NFS Servers export security, availability, performance, and cost. les to clients via NFS; each NFS Server has a Cache Module 2 Public Google S3 Azure Rackspace Clouds Other Clients Kurma Gateway Gateways Figure 2: Kurma gateway components. A gateway con- Figure 3: Simplied Kurma data structures in Thrift. sists of three types of servers as separated by dashed i16 is a 16-bit integer. Thrift does not have a native lines: NFS, Gateway, and Metadata Servers. Each NFS i128; we emulated it using two i64s. list and map Server has a persistent Cache Module and a Gateway are builtin linear and associative containers, respec- Module. Each Gateway Server has six modules: le tively. We omit common attributes such as mode, uid, system (FS), conguration (Cong), metadata, security, and other data structures for directories and volumes. cloud, and garbage collection (GC). Each Metadata Server has a ZooKeeper Module and a Hedwig Module. metadata in clouds. Each Kurma gateway asynchronously NFS Servers and Metadata Servers have local storage replicates metadata changes to all other gateways. for data cache and metadata backups, respectively. Figure 3 shows Kurma’s le-system metadata format.