DOCSLIB.ORG

A Guide to Distributed Digital Preservation

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A Guide to Distributed Digital Preservation A Guide to Distributed Digital Preservation Copyright © 2010 This collection is covered by the following Creative Commons License: Attribution-NonCommercial-NoDerivs 3.0 License You are free to copy, distribute, and display this work under the following conditions: Attribution. You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). Specifically, you must state that the work was originally published in A Guide to Distributed Digital Preservation (2010), Katherine Skinner and Matt Schultz, eds., and you must attribute the individual author(s). Noncommercial. You may not use this work for commercial purposes. No Derivative Works. You may not alter, transform, or build upon this work. For any reuse or distribution, you must make clear to others the license terms of this work. Any of these conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author’s moral rights. The above is a summary of the full license, which is available at the following URL: http://creativecommons.org/licenses/by-nc-nd/3.0/legalcode Publication and Cataloging Information: ISBN: 978-0-9826653-0-5 Editors: Katherine Skinner Matt Schultz Copyeditor: Susan Wells Parham Publisher: Educopia Institute Atlanta, GA 30309 TABLE OF CONTENTS Acknowledgements ................................................................................................... vii Road Map ....................................................................................................................ix Chapter 1: Preserving Our Collections, Preserving Our Missions ...............................1 Chapter 2: DDP Architecture .....................................................................................11 Chapter 3: Technical Considerations for PLNs..........................................................27 Chapter 4: Organizational Considerations..................................................................37 Chapter 5: Content Selection, Preparation, and Management....................................49 Chapter 6: Content Ingest, Monitoring, and Recovery for PLNs...............................73 Chapter 7: Cache and Network Administration for PLNs..........................................85 Chapter 8: Copyright Practice in the DDP: Practice makes Perfect...........................99 Appendix A: Private LOCKSS Networks ................................................................113 Glossary of Terms ....................................................................................................127 Author Biographies ..................................................................................................133 ACKNOWLEDGEMENTS We wish to express our appreciation to the institutions and departments that provided various kinds of support to enable this publication. These include the Library of Congress through its National Digital Information Infrastructure and Preservation Program (NDIIPP), which has provided financial support for the founding and development of the MetaArchive Cooperative, and the National Historical Publications and Records Commission (NHPRC), which has provided funding for the Cooperative’s transition from a successful project to a sustainable business. We are also grateful to Vicky Reich and the LOCKSS team who provided the technical framework for the Cooperative’s network and who number among the most important pioneers of our generation, not just in digital preservation technology development, but in cultivating new ways for cultural memory organizations to collaborate to preserve our digital collections as community-based, community-owned, and community-led endeavors. We would like to thank all of the contributors to this volume, first for their work in digital preservation, and second for their contributions to the substance of this book. This has been a group effort in every way and this volume depended upon the distributed knowledge and expertise of our community of contributors. Special thanks also goes to our copyeditor, Susan Wells Parham and the editorial expertise and input she has brought to this publication, and to Bill Donovan, whose fresh eyes relieved our own rather bleary ones as we finalized our work. We also greatly appreciate the efforts (and patience!) of our reviewer-editors—Vicky Reich and Tom Lipkis of the LOCKSS team, Richard Pearce-Moses of PeDALS, and Aaron Trehub of ADPNet and MetaArchive. Finally, we would like to extend a special thanks to the distributed community of readers who have assisted us along the way and to our future readership, from whom we hope to hear in later editions of this book. The distributed digital preservation field is quickly expanding and as we state repeatedly herein, other technical solutions and organizational approaches are emerging and maturing rapidly. We look forward to working with you all in the growing community of cultural stewardship that we will collectively foster. Katherine Skinner and Matt Schultz February 2010 ROAD MAP A Guide to Distributed Digital Preservation is intentionally structured such that every chapter can stand on its own or be paired with other segments of the book at will, allowing readers to pick their own pathway through the guide as best suits their needs. This approach has necessitated that the authors and editors include some level of repetition of basic principles across chapters, and has also made the Glossary (included at the back of this guide) an essential reference resource for all readers. This guide is written with a broad audience in mind that includes librarians, curators, archivists, scholars, technologists, lawyers, and administrators. Any resourceful reader should be able to use this guide to gain both a philosophical and practical understanding of the emerging field of distributed digital preservation (DDP), including how to establish or join a Private LOCKSS Network (PLN). Please note that our title is “A Guide,” rather than “The Guide,” to distributed digital preservation. We are practitioners, and as such, are covering in this book the territory that we know well. While the LOCKSS software is certainly not the only approach to DDP, it has reached a high level of maturity in the library, archives, and museum communities. We have chosen here to focus some of our attention on this technological approach because we use it ourselves and because we believe that it is readily available to groups of institutions that wish to quickly begin preserving their collections in geographically distributed preservation networks. We are excited to see other promising technologies reach similar levels of maturity, and we hope that similar documentation of these methodologies and approaches will be offered to the extended community in the future, either through other books and articles or through future editions of this guide. The Chapters Chapter 1: Preserving Our Collections, Preserving Our Missions Martin Halbert and Katherine Skinner provide a philosophical base for cultural memory organizations’ need to participate in distributed digital preservation solutions as community-owned and community-led initiatives. This chapter will be useful for all readers, particularly those with questions about the value of collaborative engagement in the digital arena for cultural memory organizations. Chapter 2: DDP Architecture Katherine Skinner and Monika Mevenkamp explore the architecture of DDP networks, focusing primarily on the Private LOCKSS Network (PLN) and its central elements. This chapter provides a foundation for all of the chapters that follow and as such, is highly recommended for all readers. Chapter 3: Technical Considerations for PLNs Beth Nicol focuses on some of the core technical decisions that different PLNs have made based on the needs of their members. This chapter will be most useful to administrators and technologists who are thinking about producing or joining a DDP, especially one that uses the LOCKSS software. Chapter 4: Organizational Considerations Tyler O. Walters provides an overview of the administrative and organizational apparatuses that are available to DDP networks and uses a series of case studies to explore the operational decisions made by existing PLNs. Administrators who are considering hosting or joining a DDP network of any type will find this chapter a helpful guide. Chapter 5: Content Selection, Preparation, and Management Gail McMillan and Rachel Howard offer a set of best practices for policies regarding the selection, preparation, and management of digital content for preservation purposes. Though they draw upon PLNs in their featured examples, the chapter may be applied more broadly to many DDP initiatives. As such, it will be of great use to librarians, curators, and archivists, as well as administrators, who are seeking to ready their collections for preservation. Chapter 6: Content Ingest, Monitoring, and Recovery for PLNs Katherine Skinner, Matt Schultz, and Monika Mevenkamp describe the central elements of the PLN-based strategy: content ingest, monitoring, and recovery. This detailed chapter will be most useful to librarians, archivists, curators, and technologists who are participating as content contributors and as a preservation site for a PLN. Chapter 7: Cache and Network Administration for PLNs Matt Schultz and Bill Robbins provide an overview of the different network responsibilities incurred through two different approaches to
Load more

Recommended publications
  • The Persistence of Open Access Electronic Journals Elizabeth A
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by DigitalCommons@Florida International University Florida International University FIU Digital Commons Works of the FIU Libraries FIU Libraries 2016 The persistence of open access electronic journals Elizabeth A. Lightfoot Florida International University, [email protected] Follow this and additional works at: http://digitalcommons.fiu.edu/glworks Part of the Collection Development and Management Commons, and the Scholarly Publishing Commons Recommended Citation Elizabeth A. Lightfoot , (2016),"The persistence of open access electronic journals", New Library World, Vol. 117 Iss 11/12 pp. 746 - 755 This work is brought to you for free and open access by the FIU Libraries at FIU Digital Commons. It has been accepted for inclusion in Works of the FIU Libraries by an authorized administrator of FIU Digital Commons. For more information, please contact [email protected]. The Persistence of Open Access Electronic Journals Author Details: Elizabeth A. Lightfoot University Libraries; Florida International University; Miami; Florida; USA [email protected] Biographical Details: Elizabeth Lightfoot is the electronic resources librarian and serials coordinator at Florida International University. She has a background in serials acquisitions, with primary interest in the management and sustainability of serials and e-resources. Structured Abstract: Purpose – Open access (OA) electronic journals have been identified as potentially at risk of loss without more coordinated preservation efforts. The purpose of this paper is to test the current availability of OA electronic journals indexed in the Directory of Open Access Journals (DOAJ). Design/methodology/approach – Using publicly available journal metadata downloaded from DOAJ, individual journal URLs were tested for validity and accessibility using a Microsoft Excel Visual Basic for Applications macro.
    [Show full text]
  • States of Sustainability:A Review of State Projects
    States of Sustainability: A Review of State Projects funded by the National Digital Information Infrastructure and Preservation Program (NDIIPP) May 2012 Christopher A. Lee University of North Carolina at Chapel Hill Lee, Christopher A. “States of Sustainability: A Review of State Projects funded by the National Digital Information Infrastructure and Preservation Program (NDIIPP).” Chapel Hill, NC: University of North Carolina, 2012. Attribution-NonCommercial CC BY-NC Revision History Final review draft distributed to all project partners February 6, 2012 Final report completed March 22, 2012 Final report published April 2, 2012 Final report updated - two sentences revised to clarify PeDALS May 8, 2012 project licensing arrangements and implementation of LOCKSS 1 Contents Executive Summary ........................................................................................................................ 5 Overview of the Four NDIIPP States Projects ................................................................................ 8 Geospatial Multistate Archive and Preservation Partnership (GeoMAPP) ................................ 9 A Model Technological and Social Architecture for the Preservation of State Government Digital Information (MTSA) ....................................................................................................... 9 Multi-State Preservation Partnership (MSPP)........................................................................... 10 Persistent Digital Archives and Library System (PeDALS) ....................................................
    [Show full text]
  • LOCKSS: Lots of Copies Keep Stuff Safe
    LOCKSS: Lots Of Copies Keep Stuff Safe David S. H. Rosenthal LOCKSS Program, Stanford University Libraries, CA Abstract Journal of Biological Chemistry. The sudden advent and im- Over the last 12 years the LOCKSS Program at Stanford mediate popularity of the Web versions of expensive and impor- has developed and deployed an open source, peer-to-peer tant academic journals forced an unexpected and unwelcome transition upon librarians: from purchasing a copy of the jour- system now comprising about 200 LOCKSS boxes in li- nal to renting access to the publisher’s copy. While rental sat- braries around the world preserving a wide range of web- isfied the librarians’ responsibility to current readers, it did not published content. Initially supported by NSF, and sub- satisfy their responsibility to future readers. It did not provide sequently by the Mellon Foundation, Sun Microsystems “perpetual access”. and NDIIPP, the program has since 2004 been sustain- able, funded by the libraries using it. The program won 2 Design an ACM award for breakthrough research in fault and at- One way to satisfy the goal of providing libraries with perpet- tack resistance in peer-to-peer systems. ual access to the e-journal content to which they subscribed Since it was designed initially for e-journals, the sys- was commonly suggested. This involved the concept of a third- tem’s design is unusual; it is driven primarily by copy- party archive, which would acquire e-journal content from the right law. The design principles were: publisher and re-publish it. In addition to subscribing to the journal itself, libraries would subscribe to the archive.
    [Show full text]
  • A Model for Integrating the Publication and Preservation of Journal Articles
    A Model for Integrating the Publication and Preservation of Journal Articles © Kevin S. Hawkins University of Michigan, Ann Arbor [email protected] Abstract [4], and Portico [5] in order to guarantee that one or more copy of the content will become available if it is There are policy, technical, and workflow gaps no longer available from the publisher. Similarly, the in library efforts to preserve online journal Koninklijke Bibliotheek and Elsevier reached an literature. Since libraries are increasingly agreement in 2002 whereby the KB will preserve involved in journal publishing, HathiTrust, a Elsevier journals under terms similar to those governing shared preservation-quality digital repository, journals that use LOCKSS, CLOCKSS, and Portico [6]. is a natural place to archive and provide access Still, there are problems with these models. LOCKSS to journal literature to ensure its long-term and CLOCKSS use web crawling, which captures only preservation and discoverability. The U-M the appearance of webpages but not their underlying Library is funding the creation of mPach, an structure or search functionality. Portico and the KB, on open-source, end-to-end publishing system in the other hand, rely on publishers to deliver journal which archiving in HathiTrust happens as a articles in valid file formats, and not just the version byproduct of publication rather than being first published but also any corrected versions of these carried out after the fact. The architecture of articles. mPach, its envisioned workflow, and plans for One way to ensure that a library always has access creating a shared infrastructure for publishing to the latest content is for the library to operate the very open-access journals are all summarized.
    [Show full text]
  • Abstract Systems Interoperability and Collaborative Development for Web
    Abstract Systems Interoperability and Collaborative Development for Web Archiving As the Web becomes the medium of record for many types of information, web archiving is an increasingly vital function of memory institutions. In furtherance of the National Digital Platform priority of IMLS, this project seeks funding to expand national web archiving capacity by undertaking research that will build a foundation for collaborative technology development, improved systems interoperability, and an Application Programming Interface (API) based model for enhanced access to, and research use of, web archives. The project team consists of Internet Archive (Archive-It) as lead institution partnering with Stanford University Libraries (both DLSS and LOCKSS), University of North Texas, and Rutgers University. This two-year research project will outline successful community models for cooperative technology development work, prototype and test API-based interoperability, and explore how interoperability can enable new access models, improve discoverability, and expand shared digital services. The practice of archiving web content poses severe programmatic and technical challenges to institutions, given its considerable infrastructure and engineering requirements; the dynamic, evolving nature of the Web; and the sheer volume and complexity of born-digital materials being collected. These high barriers to entry help explain why Archive-It (https://archive-it.org) is the overwhelming web archiving solution of choice. Yet, the ease of entry into web archiving via Archive-It has not spurred investment in local preservation or sustained collaborative technology development. Given this landscape, the assembled project partners would pursue design-based research and development to prototype API-enabled interoperability and research and test economic and community models to support collaborative development.
    [Show full text]
  • Maintaining the Integrity of Digital Archives
    Maintaining the Integrity of Digital Archives June M. Besek, Philippa S. Loengard and Jane C. Ginsburg Kernochan Center for Law, Media and the Arts 435 West 116th Street, Box A-17 New York, NY 10027 (212) 854-9869 MAINTAINING THE INTEGRITY OF DIGITAL ARCHIVES June M. Besek, Philippa S. Loengard and Jane C. Ginsburg1 Table of Contents Acknowledgements iii Executive Summary v 1.0 Introduction 1 2.0 Material Removed from Databases and Archives 4 3.0 Archives and Digital Archives 9 4.0 The Legal Landscape 12 Copyright Law and Archives Legislation 12 Moral Rights 43 Defamation 46 Mistake of Fact 55 Invasion of Privacy 57 Censorship 61 Internet Service Provider Limitation of Liability 64 Summary and Conclusions; International Law Considerations 65 5.0 How Libraries and Archives Encounter Removal 66 6.0 Development of Guidelines to Govern Publisher Retraction and Removal 70 7.0 Libraries’ Contracts with Publishers 73 8.0 Digital Archives: Reducing the Risk of Removal 80 9.0 Other Challenges to the Integrity of the Scholarly Record 89 10.0 Recommendations 90 Appendices App.-1 i ii Acknowledgements We would also like to acknowledge the many people who generously shared their time and expertise with us during the course of our study, including Joseph Branin, Ohio State University Libraries; Beverly Brown, NRC-CISTI; Mimi Calter, Stanford University Libraries; Stephen Chapman, Preservation Librarian for Digital Initiatives, Harvard University Library; Kenneth Crews, Professor of Law and Director, Copyright Management Center, IUPUI; Patricia Cruse,
    [Show full text]
  • Building and Preserving Library Digital Collections Through Community Collaboration
    Building and Preserving Library Digital Collections Through Community Collaboration Victoria Reich Executive Director, LOCKSS Program, Stanford University Libraries Abstract Traditionally, libraries in the paper world were responsible for the safekeeping of much of society’s cultural memory. They did this by building local collections, primarily to provide current local readers with rapid, convenient access to content. However, an important side effect of these local collections was a robust system for preserving the printed, and paper-based record for future readers. There were many copies of the content, on a durable, tamper-evident medium, under many separate administrations, with catalogue systems capable of directing a reader to a nearby copy. Over the last two decades, the transition from paper-based publishing to Web publishing changed models for access and preservation for libraries. Instead of purchasing a copy, libraries now lease access to the publisher's copy. This has made preserving the contemporary published record more difficult, in these ways: Unlike paper, the publisher maintains the only legal long-term copies, and the publisher can alter or destroy them at will; Copyright law makes it problematic, if not impossible, to maintain other copies without specific permission from the publisher; Unlike paper, the copies are on a fragile, easily alterable medium with a short service life (magnetic disk or tape); Unlike paper, the format of the copies is expected to become rapidly obsolete. Preparations for this eventuality are expensive. The paper reviews these challenges, describing how the collaboration between libraries underlying the LOCKSS program enables each challenge to be addressed successfully. Author Victoria Reich is Executive Director of the LOCKSS Program, Stanford University Library, [www.lockss.org].
    [Show full text]
  • Architectural Choices in LOCKSS Networks
    Architectural Choices in LOCKSS Networks David S. H. Rosenthal Stanford University Libraries Abstract 2 Existing Networks The LOCKSS digital preservation technology collects, To illustrate the differences between the various net- preserves and disseminates content in peer-to-peer net- works we describe three examples, the GLN and the works. Many such networks are in use. The Global CLOCKSS network, both preserving e-journals and e- LOCKSS Network (GLN) is an open network with many books, and Data-PASS, a PLN preserving social science nodes in which libraries preserve academic journals and data. Social science data has to be preserved in confor- books that they purchase. The CLOCKSS network is mity with one set of laws, governing for example the a closed network, managed by a non-profit consortium storage and dissemination of personal information. E- of publishers and libraries to form a dark archive of e- journals and e-books have to be preserved in conformity journal content. There are also many Private LOCKSS with a different legal framework, copyright law. This is Networks (PLNs) preserving various genres of content. one major reason for the differences between the config- Each of these networks is configured to meet the urations of these networks. specific requirements of its community and the content it preserves. This paper describes these architectural 2.1 The Global LOCKSS Network choices and discusses a development that could enable other configurations. The LOCKSS technology was invented in 1997 as a re- sponse to librarians’ concerns about long-term access to the journal content to which they subscribe [10].
    [Show full text]
  • Preserving Our Digital Heritage: the National
    Preserving Our Digital Heritage: The National Digital Information Infrastructure and Preservation Program 2010 Report A Collaborative Initiative of The Library of Congress First issued, January 2011 ii Contents I. Executive Summary ....................................................................................... 1 A. What Is at Stake ...................................................................................... 1 B. NDIIPP Legislation ................................................................................ 1 C. Key Outcomes and Findings ................................................................ 2 D. Moving Forward .................................................................................... 4 E. Securing Knowledge for the Future .................................................... 7 II. Understanding the Digital Information Needs of the Nation .............. 9 A. The Preservation Challenge for the Nation ...................................... 10 B. Congressional Charge to the Library ................................................ 13 C. National Plan for Preservation ........................................................... 15 D. Implementing the Plan ........................................................................ 16 III. Building the National Digital Preservation Network .......................... 21 A. Stewardship Network ......................................................................... 21 B. National Digital Collection ................................................................
    [Show full text]
  • LOCKSS Boxes in the Cloud
    LOCKSS Boxes in the Cloud David S. H. Rosenthal Daniel L. Vargas LOCKSS Program, Stanford University Libraries, CA Abstract These and other libraries asked whether they could use “afford- The LOCKSS system is a leading technology in the able cloud storage” for their LOCKSS boxes. The Library of field of Distributed Digital Preservation. Libraries run Congress funded the LOCKSS team to investigate this ques- tion, and this is the final report of this work. LOCKSS boxes to collect and preserve content published on the Web in PC servers with local disk storage. They We first describe the relevant features of typical cloud ser- vices, then discuss the various possible technical architectures form nodes in a network that continually audits their by which a LOCKSS box could use such a service, ruling content and repairs any damage. Libraries wondered some out for performance reasons. Applying the typical ser- whether they could use cloud storage for their LOCKSS vice’s charging model to the remaining architectures rules oth- boxes instead of local disks. We review the possible ers out for economic reasons. We then describe an experiment configurations, evaluate their technical feasibility, assess in which we implemented the most cost-effective architecture their economic feasibility, report on an experiment in and ran a production LOCKSS box preserving GLN content which we ran a production LOCKSS box in Amazon’s in Amazon’s cloud service for several months. We report on cloud, and describe some simulations of future costs of the pricing history of cloud storage, using it and the costs from cloud and local storage.
    [Show full text]
  • Open Is Not Forever: a Study of Vanished Open Access Journals
    Received: 3 September 2020 Revised: 28 December 2020 Accepted: 25 January 2021 DOI: 10.1002/asi.24460 RESEARCH ARTICLE Open is not forever: A study of vanished open access journals Mikael Laakso1 | Lisa Matthias2 | Najko Jahn3 1Information Systems Science, Hanken School of Economics, Helsinki, Finland Abstract 2Department of Political Science, John The preservation of the scholarly record has been a point of concern since the F. Kennedy Institute, Freie Universität beginning of knowledge production. With print publications, the responsibility Berlin, Berlin, Germany rested primarily with librarians, but the shift toward digital publishing and, in 3Göttingen State and University Library, particular, the introduction of open access (OA) have caused ambiguity and University of Göttingen, Göttingen, Germany complexity. Consequently, the long-term accessibility of journals is not always guaranteed, and they can even disappear from the web completely. The focus Correspondence Mikael Laakso, Information Systems of this exploratory study is on the phenomenon of vanished journals, some- Science, Hanken School of Economics, thing that has not been carried out before. For the analysis, we consulted sev- Arkadiankatu 22, 00100 Helsinki, eral major bibliographic indexes, such as Scopus, Ulrichsweb, and the Finland. Email: [email protected] Directory of Open Access Journals, and traced the journals through the Inter- net Archive's Wayback Machine. We found 174 OA journals that, through lack of comprehensive and open archives, vanished from the web between 2000 and 2019, spanning all major research disciplines and geographic regions of the world. Our results raise vital concern for the integrity of the scholarly record and highlight the urgency to take collaborative action to ensure contin- ued access and prevent the loss of more scholarly knowledge.
    [Show full text]