Eudat D7.1.2
Total Page:16
File Type:pdf, Size:1020Kb
European Data Grant agreement number: RI-283304 D7.1.2: Towards a Globally Scalable Archive Federation Technology (update) Author(s) Shaun de Witt (STFC), Maciej Brzeźniak (PSNC), Martin Hellmich (CERN) Status Final Version v1.1 Date 28.11.2013 Abstract: This document presents the work done in the second year of the Assessing Scalability task of the Addressing Scalability and Preservation task of EUDAT. This document is an update to deliverable D7.1.1 looking at alternative components which could be used either within the current, or as part of an alternative, CDI. Based on the experience of projects discussed in the original deliverable, we propose an alternative scalable architecture and provide a migration path from the current CDI to this new system. EUDAT – 283304 D7.1.2: Towards a Globally Scalable Archive Federation Technology (update) Document identifier: EUDAT-DEL-WP7-D7.1.2 Deliverable lead PSNC Related work package WP7 Author(s) Shaun de Witt (STFC), Maciej Brzeźniak (PSNC), Martin Hellmich (CERN) Contributor(s) Participants of Task 7.1.1 Due date of deliverable 30/09/2013 Actual submission date 28/11/2013 Reviewed by John Kennedy (RZG), Ari Lukkarinen (CSC) Approved by PMO Dissemination level PUBLIC Website www.eudat.eu Call FP7-INFRA-2011-1.2.2 Project number 283304 Instrument CP-CSA Start date of project 01/10/2011 Duration 36 months Disclaimer: The content of the document herein is the sole responsibility of the publishers and it does not necessarily represent the views expressed by the European Commission or its services. While the information contained in the document is believed to be accurate, the author(s) or any other participant in the EUDAT Consortium make no warranty of any kind with regard to this material including, but not limited to the implied warranties of merchantability and fitness for a particular purpose. Neither the EUDAT Consortium nor any of its members, their officers, employees or agents shall be responsible or liable in negligence or otherwise howsoever in respect of any inaccuracy or omission herein. Without derogating from the generality of the foregoing neither the EUDAT Consortium nor any of its members, their officers, employees or agents shall be liable for any direct or indirect or consequential loss or damage caused by or arising from any information advice or inaccuracy or omission herein. Copyright © The EUDAT Consortium PUBLIC 2 / 58 EUDAT – 283304 D7.1.2: Towards a Globally Scalable Archive Federation Technology (update) TABLE OF CONTENT EXECUTIVE SUMMARY ................................................................................................................. 5 1. INTRODUCTION ................................................................................................................ 6 1.1. Purpose an Structure .................................................................................................................. 6 1.2. Aims and Objectives .................................................................................................................... 6 2. NEW AND ALTERNATIVE TECHNOLOGIES ........................................................................... 8 2.1. Technology analysis methodology .............................................................................................. 8 2.2. New evaluations .......................................................................................................................... 8 2.2.1. HTTP-based Dynamic Federation ........................................................................................ 9 2.2.2. XRootD .............................................................................................................................. 10 2.2.3. Object storage ................................................................................................................... 12 2.2.4. WOS................................................................................................................................... 12 2.2.5. Open source object storage implementations: Swift and CEPH ....................................... 14 2.2.6. Federation capabilities of object storage systems............................................................ 14 2.3. Further Developments .............................................................................................................. 15 3. SCALABLE ARCHITECTURE DESIGN ................................................................................... 17 3.1. Issues effecting scalability ......................................................................................................... 17 3.2. Other consideration in design ................................................................................................... 18 3.3. Strawman design ....................................................................................................................... 19 4. CONCLUSIONS AND RECOMMENDATIONS ....................................................................... 24 4.1. Conclusions ............................................................................................................................... 24 4.1.1. Summary of technologies considered ............................................................................... 24 4.1.2. Scalable architecture highlights ........................................................................................ 24 4.2. Recommendations .................................................................................................................... 25 4.2.1. Short-term recommendations .......................................................................................... 25 4.2.2. Long-term recommendations ........................................................................................... 26 ANNEX A. SCALABILITY TEST RESULTS ...................................................................................... 27 A.1. HTTP Dynamic Federation tests results .................................................................................... 27 A.2. WOS Test Results ...................................................................................................................... 28 A.3. XRootD Test Results .................................................................................................................. 29 A.4. iRODS test results ...................................................................................................................... 31 ANNEX B. IRODS-FEDERATED NAMESPACES ............................................................................. 35 ANNEX C. DATA DISCOVERY AT THE BACK-END OF THE FEDERATION LAYER .............................. 37 ANNEX D. SCALABLE ARCHITECTURE APPLICATION TO EUDAT CDI ............................................ 39 D.1. Data ingest ................................................................................................................................ 39 D.2. Data management .................................................................................................................... 41 D.3. Data access ................................................................................................................................ 42 D.4. Discussion on Architecture and Migration ............................................................................... 43 D.5. Long term vision ........................................................................................................................ 48 D.6. Technology and vendor lock-in, modularity ............................................................................. 51 ANNEX E. GLOSSARY ............................................................................................................... 54 Copyright © The EUDAT Consortium PUBLIC 3 / 58 EUDAT – 283304 D7.1.2: Towards a Globally Scalable Archive Federation Technology (update) LIST OF FIGURES Figure 1: Request and Transfers During XRootD Testing ...................................................................... 11 Figure 2: Proposed Scalable Architecture Detailing Data Ingest .......................................................... 21 Figure 3: Details of the Proposed EUDAT Service Layer ....................................................................... 21 Figure 4: Scalable Architecture for Data Access ................................................................................... 22 Figure 5 Simple bombard test performed on HTTP federation ............................................................ 27 Figure 6 Parallel get test results acquired from single and HTTP-federated endpoints ....................... 27 Figure 7: Time to Write 5000 100Byte Objects into WOS .................................................................... 28 Figure 8: Peak Server Read and Write Rates in WOS ............................................................................ 28 Figure 9: Aggregate Server Read and Write Rates in WOS ................................................................... 29 Figure 10: Single Client Read/Write Rates in XRoot ............................................................................. 29 Figure 11: Multi Client Read/Write Rates in XRoot .............................................................................. 30 Figure 12: iRODS parallel upload / download throughput ................................................................... 32 Figure 13: Comparison IOZone tests results for disk subsystem .........................................................