DOCSLIB.ORG

LOCKSS Boxes in the Cloud

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
LOCKSS Boxes in the Cloud LOCKSS Boxes in the Cloud David S. H. Rosenthal Daniel L. Vargas LOCKSS Program, Stanford University Libraries, CA Abstract These and other libraries asked whether they could use “afford- The LOCKSS system is a leading technology in the able cloud storage” for their LOCKSS boxes. The Library of field of Distributed Digital Preservation. Libraries run Congress funded the LOCKSS team to investigate this ques- tion, and this is the final report of this work. LOCKSS boxes to collect and preserve content published on the Web in PC servers with local disk storage. They We first describe the relevant features of typical cloud ser- vices, then discuss the various possible technical architectures form nodes in a network that continually audits their by which a LOCKSS box could use such a service, ruling content and repairs any damage. Libraries wondered some out for performance reasons. Applying the typical ser- whether they could use cloud storage for their LOCKSS vice’s charging model to the remaining architectures rules oth- boxes instead of local disks. We review the possible ers out for economic reasons. We then describe an experiment configurations, evaluate their technical feasibility, assess in which we implemented the most cost-effective architecture their economic feasibility, report on an experiment in and ran a production LOCKSS box preserving GLN content which we ran a production LOCKSS box in Amazon’s in Amazon’s cloud service for several months. We report on cloud, and describe some simulations of future costs of the pricing history of cloud storage, using it and the costs from cloud and local storage. We conclude that current cloud the experiment to compare the projected future costs of stor- storage services are not cost-competitive with local hard- ing content preserved by LOCKSS boxes in cloud and local ware for long-term storage, including LOCKSS boxes. storage. We conclude with discussion of future work, some of which is already under way. Copyright c 2012 David S. H. Rosenthal 1 Introduction 2 Features of Cloud Technology “I really worry about everything going to the Amazon, as a typical cloud service, provides the following cloud,” [Wozniak] said. “I think it’s going to be components relevant here: horrendous. I think there are going to be a lot of horrible problems in the next five years. ... With the • A compute service in which a customer can run a num- cloud, you don’t own anything. You already signed ber of virtual machine instances. These instances have it away. ... I want to feel that I own things, ... A the normal resources of a physical PC but their (virtual) lot of people feel, ’Oh, everything is really on my disk storage is limited in size and is evanescent, vanish- ing without trace when the instance stops for any rea- computer,’ but I say the more we transfer everything 2 onto the web, onto the cloud, the less we’re going to son . Amazon calls this service Elastic Cloud Computing have control over it.” [12] (EC2). About 14 years ago the LOCKSS1 Program at the Stanford • An object storage service in which a customer can store University Libraries started building tools for libraries to col- arbitrary-sized byte strings. These objects persist, are lect and preserve content published on the Web. About 150 named by URLs, and are accessed via an API using HTTP libraries currently run LOCKSS boxes preserving e-journals, e- PUT, GET and HEAD requests. Depending on the ser- books, databases, special collections and other content. These vice, the object store may be more or less reliable. Ama- boxes are typically modest PC servers with substantial amounts zon calls this service Simple Storage Service (S3), and of local disk. offers two levels of reliability. The standard level is de- signed for 11 nines reliability, and Reduced Reliability Some of these libraries are members of the US National Digital Stewardship Alliance run by the Library of Congress. Storage (S3-RRS) is designed for 4 nines reliability. 1Lots Of Copies Keep Stuff Safe, a trademark of Stanford Univer- 2This is true for instances backed by S3, not by EBS. But EBS is sity. not reliable enough; see Section 5. • A block storage service, in which a customer can store computer that accesses the repository to perform the preser- a file system that can be mounted by virtual machine in- vation functions described in the OAIS Reference Model [9] stances running in the compute service. These file sys- including ingest, dissemination. integrity checking, damage re- tems persist across virtual machine restarts but are de- pair and management as part of a peer-to-peer network [13]. signed for performance rather than extreme reliability. The following architectures are feasible using current com- Facilities are normally provided by which snapshots of pute and storage service capabilities: the file systems can be reliably preserved as objects in the • A LOCKSS daemon running in a local machine with stor- object storage service. Amazon calls this service Elastic age in an object storage service. We have modified the Block Storage (EBS). LOCKSS daemon’s repository to use the S3 object stor- 2.1 Technical Aspects age interface and run experiments against S3, and also against Eucalyptus and the Internet Archive’s object stor- It is important to understand the interfaces between these com- age service, both of which are mostly compatible with ponents. Their performance characteristics determine the tech- S3. The performance we observe disqualifies this archi- nical viability of the various system architectures possible for tecture; extracting the content from S3 via its Web inter- LOCKSS boxes in the cloud: face for integrity checking and dissemination is too slow. • The interface between the outside world and the compute • A LOCKSS daemon running in a compute service virtual service, which is typically as slow as the corresponding machine instance with storage in the object storage ser- interface for a local computer. vice. The performance of this architecture led us to reject • The interface between the outside world and the storage it; even from the compute service the I/O performance of service, which is typically as slow as the corresponding the object storage service is much slower than local disk. interface for a local computer. • A LOCKSS daemon running in a compute service virtual • The interface between the compute service and the object machine instance with storage in the virtual machine’s storage service which is typically much slower than the disk. We implemented this architecture in EC2 with an interface between a local computer and its disk, or a local S3-backed virtual machine. It is disqualified for two rea- storage area network. sons. If the instance stopped for any reason the preserved • The interface between the compute service and the block content would be entirely lost, and the total space avail- storage service which is typically about the same speed as able is too small for practical use. the interface between a local computer and its disk. • A LOCKSS daemon running in a compute service vir- • The interface between the block storage service and the tual machine instance with storage in the block storage object storage service whose performance is typically not service. We implemented this architecture in EC2 with critical. EBS storage, but concerns about the unknown reliability of EBS and the cost of recovering from a total failure of 2.2 Economic Aspects EBS over the network from other LOCKSS boxes led us The charging models for the various services determine the to prefer the next architecture. economic viability of the various architectures possible for • A LOCKSS daemon running in a compute service virtual LOCKSS boxes in the cloud: machine instance with storage in the block storage ser- • The compute service typically levies charges based on the vice and a snapshot preserved in the object storage service size of the virtual machine and the time for which it is at regular intervals. We implemented this architecture in running. EC2 with EBS storage and snapshots in S3 and ran it for several months, see Section 5. • The object storage service typically levies charges based on the total size of the objects stored, the time for which We have examined some other architectures that require en- they are stored, the GET, PUT and other HTTP operations hanced capabilities from the object storage service. They are invoked on them, and the reliability level at which they discussed briefly in Section 7.2. are stored. 4 Economic Considerations • The block storage service typically levies charges based We now apply the Amazon charging model to the preferred ar- on the total amount of storage consumed by the file sys- chitecture: tems, the time for which it is stored, and the number of I/O transactions between it and the compute service. • A LOCKSS daemon running in a compute service virtual machine instance with storage in the block storage service • The service typically also levies charges based on the and a snapshot preserved in the object storage service at amounts of data transferred in each direction across the regular intervals. interface between the compute and storage services and the outside world. The following costs could be incurred: 3 Technical Architectures • EC2 charges for the virtual machine running the box. A LOCKSS box consists of some storage holding the preserved • EBS charges for storing the content. content in a repository, and a daemon, software running in a • EBS charges for I/Os to and from the content. • EBS charges for storing the backup snapshot. Week Compute Storage Bandwidth Total $ $ $ $ • Charges for inbound network usage for collecting the content. In Amazon’s case, inbound network usage is 2 53.76 2.43 0.16 56.36 free.
Load more

Recommended publications
  • The Persistence of Open Access Electronic Journals Elizabeth A
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by DigitalCommons@Florida International University Florida International University FIU Digital Commons Works of the FIU Libraries FIU Libraries 2016 The persistence of open access electronic journals Elizabeth A. Lightfoot Florida International University, [email protected] Follow this and additional works at: http://digitalcommons.fiu.edu/glworks Part of the Collection Development and Management Commons, and the Scholarly Publishing Commons Recommended Citation Elizabeth A. Lightfoot , (2016),"The persistence of open access electronic journals", New Library World, Vol. 117 Iss 11/12 pp. 746 - 755 This work is brought to you for free and open access by the FIU Libraries at FIU Digital Commons. It has been accepted for inclusion in Works of the FIU Libraries by an authorized administrator of FIU Digital Commons. For more information, please contact [email protected]. The Persistence of Open Access Electronic Journals Author Details: Elizabeth A. Lightfoot University Libraries; Florida International University; Miami; Florida; USA [email protected] Biographical Details: Elizabeth Lightfoot is the electronic resources librarian and serials coordinator at Florida International University. She has a background in serials acquisitions, with primary interest in the management and sustainability of serials and e-resources. Structured Abstract: Purpose – Open access (OA) electronic journals have been identified as potentially at risk of loss without more coordinated preservation efforts. The purpose of this paper is to test the current availability of OA electronic journals indexed in the Directory of Open Access Journals (DOAJ). Design/methodology/approach – Using publicly available journal metadata downloaded from DOAJ, individual journal URLs were tested for validity and accessibility using a Microsoft Excel Visual Basic for Applications macro.
    [Show full text]
  • A Guide to Distributed Digital Preservation
    A Guide to Distributed Digital Preservation Copyright © 2010 This collection is covered by the following Creative Commons License: Attribution-NonCommercial-NoDerivs 3.0 License You are free to copy, distribute, and display this work under the following conditions: Attribution. You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). Specifically, you must state that the work was originally published in A Guide to Distributed Digital Preservation (2010), Katherine Skinner and Matt Schultz, eds., and you must attribute the individual author(s). Noncommercial. You may not use this work for commercial purposes. No Derivative Works. You may not alter, transform, or build upon this work. For any reuse or distribution, you must make clear to others the license terms of this work. Any of these conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author’s moral rights. The above is a summary of the full license, which is available at the following URL: http://creativecommons.org/licenses/by-nc-nd/3.0/legalcode Publication and Cataloging Information: ISBN: 978-0-9826653-0-5 Editors: Katherine Skinner Matt Schultz Copyeditor: Susan Wells Parham Publisher: Educopia Institute Atlanta, GA 30309 TABLE OF CONTENTS Acknowledgements ................................................................................................... vii Road Map ....................................................................................................................ix
    [Show full text]
  • States of Sustainability:A Review of State Projects
    States of Sustainability: A Review of State Projects funded by the National Digital Information Infrastructure and Preservation Program (NDIIPP) May 2012 Christopher A. Lee University of North Carolina at Chapel Hill Lee, Christopher A. “States of Sustainability: A Review of State Projects funded by the National Digital Information Infrastructure and Preservation Program (NDIIPP).” Chapel Hill, NC: University of North Carolina, 2012. Attribution-NonCommercial CC BY-NC Revision History Final review draft distributed to all project partners February 6, 2012 Final report completed March 22, 2012 Final report published April 2, 2012 Final report updated - two sentences revised to clarify PeDALS May 8, 2012 project licensing arrangements and implementation of LOCKSS 1 Contents Executive Summary ........................................................................................................................ 5 Overview of the Four NDIIPP States Projects ................................................................................ 8 Geospatial Multistate Archive and Preservation Partnership (GeoMAPP) ................................ 9 A Model Technological and Social Architecture for the Preservation of State Government Digital Information (MTSA) ....................................................................................................... 9 Multi-State Preservation Partnership (MSPP)........................................................................... 10 Persistent Digital Archives and Library System (PeDALS) ....................................................
    [Show full text]
  • LOCKSS: Lots of Copies Keep Stuff Safe
    LOCKSS: Lots Of Copies Keep Stuff Safe David S. H. Rosenthal LOCKSS Program, Stanford University Libraries, CA Abstract Journal of Biological Chemistry. The sudden advent and im- Over the last 12 years the LOCKSS Program at Stanford mediate popularity of the Web versions of expensive and impor- has developed and deployed an open source, peer-to-peer tant academic journals forced an unexpected and unwelcome transition upon librarians: from purchasing a copy of the jour- system now comprising about 200 LOCKSS boxes in li- nal to renting access to the publisher’s copy. While rental sat- braries around the world preserving a wide range of web- isfied the librarians’ responsibility to current readers, it did not published content. Initially supported by NSF, and sub- satisfy their responsibility to future readers. It did not provide sequently by the Mellon Foundation, Sun Microsystems “perpetual access”. and NDIIPP, the program has since 2004 been sustain- able, funded by the libraries using it. The program won 2 Design an ACM award for breakthrough research in fault and at- One way to satisfy the goal of providing libraries with perpet- tack resistance in peer-to-peer systems. ual access to the e-journal content to which they subscribed Since it was designed initially for e-journals, the sys- was commonly suggested. This involved the concept of a third- tem’s design is unusual; it is driven primarily by copy- party archive, which would acquire e-journal content from the right law. The design principles were: publisher and re-publish it. In addition to subscribing to the journal itself, libraries would subscribe to the archive.
    [Show full text]
  • A Model for Integrating the Publication and Preservation of Journal Articles
    A Model for Integrating the Publication and Preservation of Journal Articles © Kevin S. Hawkins University of Michigan, Ann Arbor [email protected] Abstract [4], and Portico [5] in order to guarantee that one or more copy of the content will become available if it is There are policy, technical, and workflow gaps no longer available from the publisher. Similarly, the in library efforts to preserve online journal Koninklijke Bibliotheek and Elsevier reached an literature. Since libraries are increasingly agreement in 2002 whereby the KB will preserve involved in journal publishing, HathiTrust, a Elsevier journals under terms similar to those governing shared preservation-quality digital repository, journals that use LOCKSS, CLOCKSS, and Portico [6]. is a natural place to archive and provide access Still, there are problems with these models. LOCKSS to journal literature to ensure its long-term and CLOCKSS use web crawling, which captures only preservation and discoverability. The U-M the appearance of webpages but not their underlying Library is funding the creation of mPach, an structure or search functionality. Portico and the KB, on open-source, end-to-end publishing system in the other hand, rely on publishers to deliver journal which archiving in HathiTrust happens as a articles in valid file formats, and not just the version byproduct of publication rather than being first published but also any corrected versions of these carried out after the fact. The architecture of articles. mPach, its envisioned workflow, and plans for One way to ensure that a library always has access creating a shared infrastructure for publishing to the latest content is for the library to operate the very open-access journals are all summarized.
    [Show full text]
  • Abstract Systems Interoperability and Collaborative Development for Web
    Abstract Systems Interoperability and Collaborative Development for Web Archiving As the Web becomes the medium of record for many types of information, web archiving is an increasingly vital function of memory institutions. In furtherance of the National Digital Platform priority of IMLS, this project seeks funding to expand national web archiving capacity by undertaking research that will build a foundation for collaborative technology development, improved systems interoperability, and an Application Programming Interface (API) based model for enhanced access to, and research use of, web archives. The project team consists of Internet Archive (Archive-It) as lead institution partnering with Stanford University Libraries (both DLSS and LOCKSS), University of North Texas, and Rutgers University. This two-year research project will outline successful community models for cooperative technology development work, prototype and test API-based interoperability, and explore how interoperability can enable new access models, improve discoverability, and expand shared digital services. The practice of archiving web content poses severe programmatic and technical challenges to institutions, given its considerable infrastructure and engineering requirements; the dynamic, evolving nature of the Web; and the sheer volume and complexity of born-digital materials being collected. These high barriers to entry help explain why Archive-It (https://archive-it.org) is the overwhelming web archiving solution of choice. Yet, the ease of entry into web archiving via Archive-It has not spurred investment in local preservation or sustained collaborative technology development. Given this landscape, the assembled project partners would pursue design-based research and development to prototype API-enabled interoperability and research and test economic and community models to support collaborative development.
    [Show full text]
  • Maintaining the Integrity of Digital Archives
    Maintaining the Integrity of Digital Archives June M. Besek, Philippa S. Loengard and Jane C. Ginsburg Kernochan Center for Law, Media and the Arts 435 West 116th Street, Box A-17 New York, NY 10027 (212) 854-9869 MAINTAINING THE INTEGRITY OF DIGITAL ARCHIVES June M. Besek, Philippa S. Loengard and Jane C. Ginsburg1 Table of Contents Acknowledgements iii Executive Summary v 1.0 Introduction 1 2.0 Material Removed from Databases and Archives 4 3.0 Archives and Digital Archives 9 4.0 The Legal Landscape 12 Copyright Law and Archives Legislation 12 Moral Rights 43 Defamation 46 Mistake of Fact 55 Invasion of Privacy 57 Censorship 61 Internet Service Provider Limitation of Liability 64 Summary and Conclusions; International Law Considerations 65 5.0 How Libraries and Archives Encounter Removal 66 6.0 Development of Guidelines to Govern Publisher Retraction and Removal 70 7.0 Libraries’ Contracts with Publishers 73 8.0 Digital Archives: Reducing the Risk of Removal 80 9.0 Other Challenges to the Integrity of the Scholarly Record 89 10.0 Recommendations 90 Appendices App.-1 i ii Acknowledgements We would also like to acknowledge the many people who generously shared their time and expertise with us during the course of our study, including Joseph Branin, Ohio State University Libraries; Beverly Brown, NRC-CISTI; Mimi Calter, Stanford University Libraries; Stephen Chapman, Preservation Librarian for Digital Initiatives, Harvard University Library; Kenneth Crews, Professor of Law and Director, Copyright Management Center, IUPUI; Patricia Cruse,
    [Show full text]
  • Building and Preserving Library Digital Collections Through Community Collaboration
    Building and Preserving Library Digital Collections Through Community Collaboration Victoria Reich Executive Director, LOCKSS Program, Stanford University Libraries Abstract Traditionally, libraries in the paper world were responsible for the safekeeping of much of society’s cultural memory. They did this by building local collections, primarily to provide current local readers with rapid, convenient access to content. However, an important side effect of these local collections was a robust system for preserving the printed, and paper-based record for future readers. There were many copies of the content, on a durable, tamper-evident medium, under many separate administrations, with catalogue systems capable of directing a reader to a nearby copy. Over the last two decades, the transition from paper-based publishing to Web publishing changed models for access and preservation for libraries. Instead of purchasing a copy, libraries now lease access to the publisher's copy. This has made preserving the contemporary published record more difficult, in these ways: Unlike paper, the publisher maintains the only legal long-term copies, and the publisher can alter or destroy them at will; Copyright law makes it problematic, if not impossible, to maintain other copies without specific permission from the publisher; Unlike paper, the copies are on a fragile, easily alterable medium with a short service life (magnetic disk or tape); Unlike paper, the format of the copies is expected to become rapidly obsolete. Preparations for this eventuality are expensive. The paper reviews these challenges, describing how the collaboration between libraries underlying the LOCKSS program enables each challenge to be addressed successfully. Author Victoria Reich is Executive Director of the LOCKSS Program, Stanford University Library, [www.lockss.org].
    [Show full text]
  • Architectural Choices in LOCKSS Networks
    Architectural Choices in LOCKSS Networks David S. H. Rosenthal Stanford University Libraries Abstract 2 Existing Networks The LOCKSS digital preservation technology collects, To illustrate the differences between the various net- preserves and disseminates content in peer-to-peer net- works we describe three examples, the GLN and the works. Many such networks are in use. The Global CLOCKSS network, both preserving e-journals and e- LOCKSS Network (GLN) is an open network with many books, and Data-PASS, a PLN preserving social science nodes in which libraries preserve academic journals and data. Social science data has to be preserved in confor- books that they purchase. The CLOCKSS network is mity with one set of laws, governing for example the a closed network, managed by a non-profit consortium storage and dissemination of personal information. E- of publishers and libraries to form a dark archive of e- journals and e-books have to be preserved in conformity journal content. There are also many Private LOCKSS with a different legal framework, copyright law. This is Networks (PLNs) preserving various genres of content. one major reason for the differences between the config- Each of these networks is configured to meet the urations of these networks. specific requirements of its community and the content it preserves. This paper describes these architectural 2.1 The Global LOCKSS Network choices and discusses a development that could enable other configurations. The LOCKSS technology was invented in 1997 as a re- sponse to librarians’ concerns about long-term access to the journal content to which they subscribe [10].
    [Show full text]
  • Preserving Our Digital Heritage: the National
    Preserving Our Digital Heritage: The National Digital Information Infrastructure and Preservation Program 2010 Report A Collaborative Initiative of The Library of Congress First issued, January 2011 ii Contents I. Executive Summary ....................................................................................... 1 A. What Is at Stake ...................................................................................... 1 B. NDIIPP Legislation ................................................................................ 1 C. Key Outcomes and Findings ................................................................ 2 D. Moving Forward .................................................................................... 4 E. Securing Knowledge for the Future .................................................... 7 II. Understanding the Digital Information Needs of the Nation .............. 9 A. The Preservation Challenge for the Nation ...................................... 10 B. Congressional Charge to the Library ................................................ 13 C. National Plan for Preservation ........................................................... 15 D. Implementing the Plan ........................................................................ 16 III. Building the National Digital Preservation Network .......................... 21 A. Stewardship Network ......................................................................... 21 B. National Digital Collection ................................................................
    [Show full text]
  • Open Is Not Forever: a Study of Vanished Open Access Journals
    Received: 3 September 2020 Revised: 28 December 2020 Accepted: 25 January 2021 DOI: 10.1002/asi.24460 RESEARCH ARTICLE Open is not forever: A study of vanished open access journals Mikael Laakso1 | Lisa Matthias2 | Najko Jahn3 1Information Systems Science, Hanken School of Economics, Helsinki, Finland Abstract 2Department of Political Science, John The preservation of the scholarly record has been a point of concern since the F. Kennedy Institute, Freie Universität beginning of knowledge production. With print publications, the responsibility Berlin, Berlin, Germany rested primarily with librarians, but the shift toward digital publishing and, in 3Göttingen State and University Library, particular, the introduction of open access (OA) have caused ambiguity and University of Göttingen, Göttingen, Germany complexity. Consequently, the long-term accessibility of journals is not always guaranteed, and they can even disappear from the web completely. The focus Correspondence Mikael Laakso, Information Systems of this exploratory study is on the phenomenon of vanished journals, some- Science, Hanken School of Economics, thing that has not been carried out before. For the analysis, we consulted sev- Arkadiankatu 22, 00100 Helsinki, eral major bibliographic indexes, such as Scopus, Ulrichsweb, and the Finland. Email: [email protected] Directory of Open Access Journals, and traced the journals through the Inter- net Archive's Wayback Machine. We found 174 OA journals that, through lack of comprehensive and open archives, vanished from the web between 2000 and 2019, spanning all major research disciplines and geographic regions of the world. Our results raise vital concern for the integrity of the scholarly record and highlight the urgency to take collaborative action to ensure contin- ued access and prevent the loss of more scholarly knowledge.
    [Show full text]