[ VOLUME 5 I ISSUE 3 I JULY– SEPT 2018] E ISSN 2348 –1269, PRINT ISSN 2349-5138 EFFECTIVE CLOUD STORAGE IN DISTRIBUTED SYSTEM USING KEY- VALUE STORES G.BHAVANA1 & P.BALAKISHAN2 & Dr.M.SUJATHA3 1M.Tech Student, 2Associate Professor, 3Associate Professor Department Of CSE, JYOTHISHMATHI INSTITUTE OF TECHNOLOGICAL SCIENCES, KARIMNAGAR T.S.INDIA. Received: May 22, 2018 Accepted: July 08, 2018 ABSTRACT Nowadays, cloud-based storage services are rapidly growing and becoming an emerging trend in data storage field. There are many problems when designing an efficient storage engine for cloud-based systems with some requirements such as big-file processing, lightweight meta-data, low latency, parallel I/O, deduplication, distributed, high scalability. Key-value stores played an important role and showed many advantages when solving those problems. This work reports on the realization of a key manager that uses an untrusted distributed key-value store (KVS) and offers consistent key distribution over the Key-Management Interoperability Protocol (KMIP). To achieve confidentiality, it uses a key hierarchy where every key except a root key itself is encrypted by the respective parent key. The hierarchy also allows for key rotation and, ultimately, for secure deletion of data. The design permits key rotation to proceed concurrently with key-serving operations. A prototype was integrated with IBM Spectrum Scale, a highly scalable cluster file system, where it serves keys for file encryption. Linear scalability was achieved even under load from concurrent key updates. The implementation shows that the approach is viable, works as intended, and suitable for high throughput key serving in cloud platforms. Keywords: Cloud Storage, Key-Value, NoSQL, Big File, Distributed Storage. I. INTRODUCTION smaller chunks, storing them on disks or Cloud-based storage services commonly distributed nodes and then managing them using serves millions of users with storage capacity for a meta-data system [1], [8], [24], [4]. Storing each user can reach to several gigabytes to chunks and meta-data efficiently and designing a terabytes of data. People use cloud storage for the lightweight meta-data are significant problems daily demands, for example backing-up data, that cloud storage providers have to face. After a sharing file to their friends via social networks long time of investigating, we realized that such as Facebook [3], Zing Me [2]. Users also current cloud storage services have a complex probably upload data from many different types meta-data system, at least the size of metadata is of devices such as computer, mobile phone or linear to the file size for every file. Therefore, the tablet. After that, they can download or share space complexity of these meta-data system is them to others. System load in a cloud storage is O(n) and it is not well scalable for big-file. In this usually really heavy. Thus, to guarantee a good research, we propose new big-file cloud storage quality of service for users, the system has to face architecture and a better solution to reduce the many difficult problems and requirements: space complexity of meta-data. Serving intensity data service for a large number Key-Value stores have many advantages for of users without bottle-neck; Storing, retrieving storing data in data-intensity services. They often and managing big-files in the system efficiently; outperform traditional relational databases in the Parallel and resumable uploading and ability of heavy load and largescale systems. In downloading; Data deduplication to reduce the recent years, key-value stores have an waste of storage space caused by storing the unprecedented growth in both academic and same static data from different users. In industrial field. They have low-latency response traditional file systems, there are many time and good scalability with small and medium challenges for service builder when managing a key-value pair size. Current key-value stores are huge number of bigfile: How to scale system for not designed for directly storing big-values, or big the incredible growth of data; How to distribute file in our case. We executed several experiments data in a large number of nodes; How to replicate in which we put whole file-data to key-value data for load-balancing and fault-tolerance; How store, the system did not have good performance to cache frequently accessed data for fast I/O, etc. as usual for many reasons: firstly, the latency of A common method for solving these problems put/get operation for big-values is high, thus it which is used in many Distributed File Systems affects other concurrent operations of key-value and Cloud Storages is splitting big file to multiple store service and multiple parallel accesses to 1716x IJRAR- International Journal of Research and Analytical Reviews Research Paper [VOLUME 5 I ISSUE 3 I JULY – SEPT 2018] e ISSN 2348 –1269, Print ISSN 2349-5138 http://ijrar.com/ Cosmos Impact Factor 4.236 different value reach limited. Secondly, when the management can be seen as an essential service value is big, there is no more space to cache of an IT infrastructure and especially for cloud another objects in main memory for fast access platforms, similar to network connectivity, operations. Finally, it is difficult to scale-out computing, and storage. The most prominent system when number of users and data increase. standard for distributed key management today This research is implemented to solve those is the OASIS Key Management Interoperability problems when storing big-values or big-file Protocol (KMIP) [2], which specifies operations using key-value stores. It brings many advantages for managing, storing, and retrieving keys at a of key-value store in data management to design remote server. For local key management using a cloud-storage system called Big File Cloud library-style access PKCS #11 [3] is the prevalent (BFC). interface. In the context of cloud services, where These are our contributions in this research: – service interactions are REST calls, the open- Propose a light-weight meta-data design source Barbican [4] key manager provides keys for big file. Every file has nearly the same to all services of OpenStack. Commercial cloud size of meta-data. BFC has O(1) space platforms use proprietary protocols inside their complexity of meta-data of a file, while infrastructure. Key managers differ according to size of meta-data of a file in Dropbox[1], the operations they support and in terms of their HDFS[4] has space complexity of O(n) performance, resilience, and security. Prominent where n is size of original file. See Fig 9 commercial key servers often put emphasis on Propose a logical contiguous chunk-id of the needs of enterprise environments, such as chunk collection of files. That makes it fine-grained authentication and support for easier to distribute data and scale-out hardware security modules (HSMs). For example, the storage system. governmental standards for handling health data Bring the advantages of key-value store dictate a reliable audit trail to reconstruct all into big-file data store which is not operations accessing cryptographic keys. default supported for big-value. ZDB is Enterprise key managers are also designed for used for supporting sequential write, high availability to allow uninterrupted service. small memory-index overhead. They must support the complete lifecycle of These contributions are implemented and cryptographic secrets, with operations for evaluated in Big File Cloud (BFC) that serve creating, importing, storing, reading, updating, storage for Zing Me Users. Disk Image files of exporting, and deleting keys. Designing and VNG’s CSM Boot diskless system are stored in Big operating a key-management service in a File Cloud. distributed system with many entities running Encryption plays a fundamental role for realizing cryptographic operations is challenging because secure networked computing environments. Key it must balance between the conflicting goals of management ensures reliable and secure performance and security. distribution of cryptographic keys to legitimate In this paper we present a solution for scaling a clients, which are then able to encrypt data or to key management service to cloud applications establish secure communication channels. Key with thousands of clients and possibly millions of management for cloud scale distributed keys. This includes the capability to scale installations poses additional challenges over dynamically while maintaining security. Our classical, centralized systems, due to the vastly distributed key management solution should bigger systems and the higher demands for handle all core key lifecycle-management tasks resilience and security. Maintaining the and scale in a linear way. Existing enterprise- confidentiality of encryption keys is extremely grade key managers do not provide such important, especially for encrypting data in scalability because they rely too heavily on storage systems, where losing access to the centralized components, such as HSMs or encryption key implies losing the data itself. A strongly consistent relational databases. In communication system, in contrast, may just particular, we address key management for an restart the session if a key is lost. As key enterprise environment with a scalable cloud management is critical for many environments, platform. The clients or endpoints accessing the industry standards have been introduced to key manager primarily perform dataat-rest separate key management