Principles and Good Practice for Preserving Data

IHSN

International Household Survey Network

About the IHSN Principles and Good Practice for In February 2004, representatives from developing countries and development agencies participated in the Second Roundtable on Development Results held in Marrakech, Morocco. They reflected on how donors can better coordinate support to strengthen the statistical systems and monitoring and Preserving Data evaluation capacity that countries need to manage their development process. One of the outcomes of the Roundtable was the adoption of a global plan for statistics, the Marrakech Action Plan for Interuniversity Consortium for Political and Statistics (MAPS). Social Research (ICPSR) Among the MAPS key recommendations was the creation of an International Household Survey Network. In doing so, the international community acknowledged the critical role played by sample surveys in supporting the planning, implementation and monitoring of development policies and programs. Furthermore, it provided national and international agencies with a platform to better coordinate and manage socioeconomic data collection and analysis, and to mobilize support for more efficient and effective approaches to conducting surveys in developing countries.

The IHSN Working Paper series is intended to encourage the exchange of ideas and discussion on topics related to the design and implementation of household surveys, and to the analysis, dissemination and use of survey data. People who whish to submit material for publication in the IHSN Working Paper series are encouraged to contact the IHSN secretariat via [email protected].

IHSN Working Paper No 003 December 2009 www.ihsn.org E-mail: [email protected] www.ihsn.org

WP-Cover-003.indd 1 8/29/10 8:46 AM

Principles and Good Practice for Preserving Data

Interuniversity Consortium for Political and Social Research (ICPSR)

December 2009

IHSN Working Paper No 003

Abstract

This document provides basic guidance for managers in statistical agencies who are responsible for preserving data using the principles and good practice defined by the digital preservation community. The guidance in this paper defines the rationale for preserving data and the principles and standards of good practice as applied to data preservation, documents the development of a digital preservation policy and uses digital archive audit principles to suggest good practice for data.

About ICPSR

The Interuniversity Consortium for Political and Social Research (ICPSR) is an international consortium of about 700 academic institutions and research organizations. ICPSR is a unit within the Institute for Social Research at the University of Michigan and maintains its office in Ann Arbor.

The Consortium provides leadership and training in data access, curation, and methods of analysis for the social science research community. It maintains a data archive of more than 500,000 files of research in the social sciences, and hosts 16 specialized collections of data in education, aging, criminal justice, substance abuse, terrorism, and other fields.

ICPSR’s educational activities include the Summer Program in Quantitative Methods of Social Research, a comprehensive curriculum of intensive courses in research design, statistics, data analysis, and social methodology. ICPSR also leads several initiatives that encourage use of data in teaching, particularly for undergraduate instruction.

ICPSR-sponsored research focuses on the emerging challenges of digital curation and data science.

iii Acknowledgments

The document was developed by the Inter-university Consortium for Political and Social Research (ICPSR) for the International Household Survey Network (IHSN), with financial support from the World Bank Development Grant Facility, Grant No 4001009-06, administered by the PARIS21 Secretariat at OECD.

It was prepared by Nancy Y. McGovern, Digital Preservation Officer, ICPSR, with contributions from: Lance Stuchell, Digital Preservation Projects Coordinator, and research assistants Heather Backman, Annelise Doll, Michael Perry and Lindsey Williams.

Olivier Dupriez (World Bank), François Fonteneau (PARIS21), Gaye Parcon (consultant, PARIS21) and a team of reviewers have provided comments on and suggestions for successive versions of the paper.

The guidance refers to Digital Preservation Management: Implementing Short-term Strategies for Long-term Problems, with acknowledgement of contributions by Anne R Kenney and the rest of the workshop-development team at Cornell University Library in Ithaca, NY, USA; and funding from the US National Endowment for the Humanities.

© Copyright of this document is shared by OECD, the World Bank and the Regents of the University of Michigan (for ICPSR). The parts of the document defined as background intellectual property (ie, those developed as part of the Digital Preservation Management workshop that have pre-existing copyright held by Cornell University, for which the Regents of the University of Michigan have a non-exclusive licence to use) are licensed for use in this document under Creative Commons - specifically the Attribution, Non-commercial, No Derivative Works licence.

This paper (or a revised copy of it) is available on the web-site of the International Household Survey Network at www.ihsn.org.

Citation

Inter-university Consortium for Political and Social Research (ICPSR). 2009. “Principles and Good Practice for Preserving Data”, International Household Survey Network, IHSN Working Paper No 003, December 2009.

The findings, interpretations, and views expressed in this paper are those of the author(s) and do not necessarily represent those of the International Household Survey Network member agencies or secretariat.

iv Table of Contents

Page Overview of this Guidance...... vi Acronyms ...... vii

1. Introduction ...... 1 2. Principles and Standards...... 5 3. Putting the Principles into Practice...... 13 4. Formulating a Preservation Policy...... 19 5. Evaluation and Audit for Data Preservation...... 27 Annexes

A. Glossary ...... 32 B. References...... 34 C. Milestones of Data Preservation...... 37 D. Programme and Policy Examples...... 39 E. Survey of Institutional Readiness ...... 41 F. Technological Obsolescence and Other Threats...... 45 List of Tables

Table 1 Good practice for integrity features of digital content...... 5 Table 2 OAIS roles for data preservation...... 7 Table 3 OAIS functions and implications for data preservation...... 7 Table 4 OAIS information packages and implications for data...... 8 List of Figures

Figure 1 The high-level OAIS Reference Model...... 6 Figure 2 The overlap between OAIS and PAIMAS...... 10 Figure 3 The four phases of PAIMAS...... 11 Figure 4 The TDR attributes applied to an organisational setting...... 18 Figure 5 Three-legged stool for digital preservation...... 18

v Overview of this Guidance

1. Introduction

This document provides basic guidance for managers in statistical agencies who are responsible for preserving data using the principles and good practice defined by the digital preservation community. The guidance defines the rationale for preserving data and the principles and standards of good practice as applied to data preservation; documents the development of a digital preservation policy; and uses digital archive audit principles to suggest good practice for data. This section defines the purpose and scope of the guidance.

2. Principles and Standards

Data archives have been preserving data in digital form since the 1960s and the digital preservation community has been emerging as an international, inter-disciplinary domain since the 1996 report Preserving Digital Information. This section reviews the current framework of community principles and standards and the relevance of these for data preservation.

3. Putting the Principles into Practice

The seven attributes of a trusted digital repository provide a framework for an organisation to preserve data that is adaptable to the mandate of the organisation -- a framework that will preserve the data and their nature.

4. Formulating a Preservation Policy

The development and implementation of a policy is essential for effective data preservation. A good policy reflects the mandate of the organisation to preserve data; provides an explicit commitment by the organisation to preserving data in accordance with community principles and standards; and fulfils a fundamental requirement of good preservation practice.

5. Evaluation and Audit for Data Preservation

The digital preservation community is developing a standard for self-assessment, audit and certification of digital archives. These principles provide a framework of recommended practice for any organisation involved in preserving data.

Annexes

A. Glossary B. References C. Preservation Milestones for Managing Data D. Programme and Policy Examples E. Survey of Institutional Readiness F. Technological Obsolescence and Other Threats

vi List of Acronyms

AIC Archival Information Collection (OAIS) AIP Archival Information Package (OAIS) ASCII American Standard Code for Information Interchange CCSDS Consultative Committee for Space Data Systems CLIR Council on Library and Information Resources CRL Centre for Research Libraries DCC Digital Curation Centre DDI Data Documentation Initiative DIP Dissemination Information Package (OAIS) DRAMBORA Digital Repository Audit Method Based on Risk Assessment EU European Union IASSIST International Association for Social Science Information Service and Technology ICPSR Inter-university Consortium for Political and Social Research IHSN International Household Survey Network iPres International Conference on the Preservation of Digital Objects ISO International Organisation for Standardisation JISC Joint Information Systems Committee METS Metadata Encoding and Transmission Standard NASA National Aeronautics and Space Agency NDIIPP National Digital Information Infrastructure Preservation Program, Library of Congress Nestor Network of Expertise in Long-term STOrage and Availability of Digital Resources NSF National Science Foundation OCLC On-line Computer Library Consortium OECD Organisation for Economic Cooperation and Development OAIS Open Archival Information System Reference Model PADI Preserving Access to Digital Information PAIMAS Producer-Archive Interface – Methodology Abstract Standard (OAIS) PDI Preservation Description Information PLANETS Preservation and Long-term Access through Networked Services RAC Digital Repository Audit and Certification RLG Research Libraries Group SIP Submission Information Package (OAIS) SPSS Statistical Package for the Social Sciences TDR Trusted Digital Repository TRAC Trustworthy Repositories Audit and Certification UK United Kingdom US United States UNESCO United Nations Educational, Scientific and Cultural Organisation XML Extensible Mark-up Language

vii viii

IHSN Working Paper No. 003 December 2009

1. Introduction

This guidance aims to assist official data producers enduring or at least continuing value, and continue to (national statistical offices and line ministries) in do so until that commitment is formally and explicitly defining and meeting their digital preservation ended. The decision to end that commitment could requirements and obligations. The recommendations have political, economic, cultural and potentially legal address core preservation requirements. They identify ramifications, so sound selection criteria are needed to steps for developing an effective preservation approach enable sound preservation planning.1 suited to the needs and requirements of individual official data producers and complying with standards A sustainable preservation programme addresses of good practice within the digital preservation organisational issues, technological concerns and community, the latter being committed to the long- funding questions. term management of digital content of all types. This section defines the purpose and scope of the guidance. Organisational infrastructure includes the policies, procedures, practices and people Purpose — the elements that any programme needs to thrive but specialised to address digital Producers of official data (national statistical agencies preservation requirements. It addresses this and line ministries) generate vast amounts of data key development question: and information in digital form each year, including microdata from surveys and censuses, the content What are the requirements and parameters of administrative recording systems, databases of for the organisation’s digital preservation indicators, sample frames, registers, methodological programme? and analytical reports and publications and maps and other types of content. These digital assets represent Technological infrastructure consists of significant investment by producers and have the requisite equipment, software, hardware, considerable value for present and future users. This a secure environment and the skills to digital content is part of each country’s heritage of its establish and maintain the digital preservation statistical agency’s institutional memory and needs to programme. It anticipates and responds wisely be preserved. to changing technology. It addresses this key development question: Increasingly, this content is ‘born’ digitally, though print and other analogue formats are digitised for How will the organisation meet defined digital easier storage, discovery and use. The challenges of preservation requirements? maintaining digital content over successive generations of technology have been recognised since the 1960s Resources framework addresses the when the first data archives were established. There requisite start-up, ongoing and contingency are numerous examples of important data being lost in funding to enable and sustain the digital the absence of an effective preservation approach -- for preservation programme. It addresses this key example data have been stored on outdated, unreadable development question: mainframe tapes and other obsolete technology. Also, organisations have expected staff to preserve digital What resources will be needed to develop and information without providing adequate support maintain the digital preservation programme?2 and resources to do so, and data often lacks adequate documentation to correctly identify valid versions. The preservation requirements of these three Statistical and other data-producing agencies have organisational components are discussed in the a responsibility not only to collect and disseminate guidance. data but also to guarantee their usability in the long term, either by preserving the data themselves or by 1 DPM tutorial, “Digital Assets”, available at http://www.icpsr.umich. entrusting specialist organisations with a mandate for edu/dpm/dpm-eng/program/assets.html preserving data. Data producers must commit resources 2 DPM Workshop, “Conclusion”, available at http://www.icpsr.umich. to the management of data and related information of edu/dpm/dpm-eng/conclusion.html

1 Principles and Good Practice for Preserving Data

Scope organisation to demonstrate compliance with good practice, the guidance distils the current standards and In defining the scope of the guidance, it is necessary principles of the digital preservation community into to distinguish between digital preservation, ensuring recommendations for managers to use in framing their long-term access to data, and data curation, value- organisation’s approach. The desired outcome of this added activities to make specific data as understandable guidance is to encourage organisations to achieve and and usable as possible to their community. Data maintain a state of well-managed data as organisational curation practices are not nearly as generalised as requirements and technology evolve. This guidance preservation practices because they are applied to -- ideally in conjunction with relevant training and content type rather than format type. In practice, access to the growing set of suitable organisational different domains might curate data differently, but examples of approaches and practice (examples of every domain should preserve data using common which are included in the guidance) -- should achieve practices. For example, common data content types, eg, this outcome. census and surveys, refer to content types that might be created and managed in a wide range of technological Rationale for Preserving Data environments and might be stored in a wide range of file formats. The content type does not determine The rationale for effective and compliant preservation the file format, but the file format does inform the of data encompasses a range of incentives. A primary selection of an appropriate preservation strategy. incentive is to meet the obligations when an organisation Digital curation is a term that encompasses both data is required by legislation or another mandate to keep curation and digital preservation. There are many good data. A second incentive is to maximise the investment sources for data curation and more recently for digital of resources in data production when there is an curation, eg, the UK Digital Curation Centre resources expectation that they will continue to be available into and the ICPSR Data Preparation Guide. There are fewer the foreseeable future due to the cost of producing sources that connect good digital preservation practice them. A third incentive is to demonstrate the capability, to the management of data, so this guidance focuses on in relation to peer organisations, to maintain status filling this gap. when there is an implicit or explicit expectation that an organisation is preserving data. A fourth incentive The digital preservation community has been is to avoid the embarrassment of losing data when an developing and promulgating standards and organisation is identified as responsible for managing principles of good practice since the mid-1990s. This them. Finally, a key incentive is to provide ongoing community has defined the fundamentals of good access to the historical and cultural heritage of a nation. practice for organisations through a set of standards and principles for preserving digital content. This In early 2010 the Blue Ribbon Task Force on guidance incorporates the current set of community Sustainable Preservation and Access -- convened by the requirements and directs users of the guidance US National Science Foundation, with international to preservation standards and principles that are support, to examine the cost of long-term preservation emerging as the current set are extended and revised. -- is expected to complete and publish its report. Its The principles of good digital preservation practice results might offer additional incentives to the rationale are generalised, i.e. applicable to any organisation for digital preservation costs. For now, these are the that preserves digital content of any kind – and local most common incentives for preservation and this set practice is individualised: an organisation is expected has been sufficient for many organisations to justify to demonstrate how the principles apply to its specific their programmes. organisational setting based upon local factors, e.g. relevant mandates, requirements, available human and Digital Preservation Terminology technological resources and the extent of digital content to be preserved. As a starting point for discussing good practice for digital preservation, a brief review of preservation All organisations that produce and manage data terminology in use within the community provides a are expected to adhere to these principles. However, context for the guidance (see Annex A for definitions all such organisations will not apply the principles of terms used). A challenge in discussing and sharing in the same way or develop the same approach examples of digital preservation practice has been a lack to preserving data. To address the need for each of standardised language. There is yet no international

IHSN Working Paper No. 003 December 2009

authoritative source that provides the community with to remain meaningful over time. For digital content to approved definitions of digital preservation and related be usable and meaningful, authorised users must be terms, as they emerges and evolve. Therefore, this provided with the means — human or automated — to guidance uses a working definition reflecting discussion find, open, read and use the material maintained by a within the community about the scope and meaning of digital preservation programme. In this guidance, the the term. The term digital preservation refers to all the digital content referred to is primarily social, economic activities undertaken by a steward (an organisation or and other human-related data, e.g. population censuses, individual) to ensure that the digital content for which labour force surveys, macroeconomic databases and the steward has responsibility is maintained in usable related publications, although the recommendations formats and can be made available in meaningful ways are relevant to data produced in other contexts. for current and future uses over time. To understand the term, it is necessary to define activities in the context Digital information is at risk of loss in two important of preservation and to consider the implications of ways: obsolescence of enabling technologies and managing data over time. physical damage. Obsolescence might affect software, hardware and even the arrangement of the data in a In practical terms, preservation activities include stored file, and it might occur at an alarming pace. explicitly identifying the digital content that needs to be preserved; taking responsibility for bringing that -- A file format might be superseded by digital content into a sustainable environment with newer versions, which might no longer appropriate policies and procedures; and ensuring be supported by the current vendor or that the digital content can be made available over relevant standards body. time to authorised users. Organisations that are responsible for managing data might include data -- A storage medium might be superseded by archives, national statistical offices and line ministries, newer and denser versions of that medium research programmes that produce data, and any other or by new types of media — smaller, organisation with responsibility for producing and denser, faster and easier to read. managing data. Users within these environments might be internal or external and preservation is intended -- The device needed to read a storage medium to ensure that data remain useful for any required or might no longer be manufactured. desirable use. -- Software used to create, manage or access In this context, over time refers to the length of digital content might be superseded by time the digital content should remain usable for newer versions or newer generations with legal, financial, cultural, business or other purposes. more capability using the most current This might be measured in months, years, decades or technology. centuries. In practice, over time refers to the need to manage data for five years or more because the speed -- Computers of every size and scale are and impact of continual technological change requires continually superseded by faster and more that organisations take action to ensure the longevity powerful machines that can store and of data within their care. If an organisation is required process more and more content. or wishes to ensure that digital content is usable and meaningful for five years or more, that equates to -- Vendors of all technology compete, preserving the data because they will have to remain emerge, merge and fade, making it even useful for one or more generations or iterations of the more difficult to maintain digital content supporting technology, eg, hardware, software). over time.

Digital content consists of a growing range and Digital information is also vulnerable to physical combination of information types, e.g. text, images, threat. Like obsolescence, physical damage might affect geospatial data, audio and video. Data for social science one of the multiple components required to access digital and other domains increasingly refer to all these types information, namely hardware and media. Computer of digital content. Throughout its lifecycle, digital components and media can physically fail due to content relies upon information technology of all kinds human error, natural events and even just the passing

3 Principles and Good Practice for Preserving Data

of time.3 One option for an organisation wishing to be (see Annex B for a list of resources referenced in the alert to relevant changes is to monitor technological guidance). developments and systematically consider potential preservation implications. Currently, there is no central Good practice does not change whatever type of place within the community where all the necessary content to be preserved. The principles of digital information has been brought together, but Annex B preservation define what should be done for any identifies a number of community-based resources organisation. Each organisation determines how that provide updates of relevant developments, e.g. preservation will suit its requirements, technological Preserving Access to Digital Information (PADI), and setting and content. In practice, specific preservation the guidance notes emerging developments of interest plans are developed to reflect the nature of the materials, at appropriate points in the document. Annex F offers a especially the file formats; the usage requirements for more detailed discussion of the nature and implications long-term management, e.g. authenticity and legal of technological obsolescence and other threats to requirements; and the technology environment in digital information from a preservation perspective. The which the content was created and will be managed over challenges noted in Annex F are addressed throughout time. Most common file formats have been associated the guidance. with one or more preservation strategies devised within the digital preservation community. Throughout the Defining Good Practice for Digital Preservation guidance, there are examples illustrating the principles of good practice. The digital preservation community has been developing principles and standards for good practice This document guides organisations towards an over the past dozen years, beginning in 1996 with the understanding of the standards and principles of the release of Preserving Digital Information. Since then, digital preservation community and defining attainable the documents developed by the community have objectives and stages of development for achieving provided a comprehensive framework of good practice alignment and compliance with those standards and for managing data. The framework is presented in this principles. Whenever relevant, the recommendations guidance and should be achievable by any organisation consider good practice for the long-term management of with preservation responsibilities. The specific data in terms of minimum requirements, recommended community documents in the framework, which are practice and ideal (or fully compliant) levels, dependent referenced, include: Open Archival Information upon the needs and mandate of the organisation. Systems (OAIS) Reference Model, a standard approved by the International Standards Organisation (ISO) For organisations not familiar with preservation in 2003 that defines the technological framework for and which plan to use the guidance to develop a preserving data; Trusted Digital Repositories (TDR): digital preservation approach, a comprehensive self- Attributes and Responsibilities, a report released assessment of requirements, resources and capabilities in 2002 that provides the organisational context is a good starting point. For this purpose, Annex E for preserving data; the Data Seal of Approval, a provides a survey of institutional readiness form set of principles for data archives released in 2008 developed for the Digital Preservation Management that presumes the use of both TDR and OAIS in workshop series. The survey provides a checklist to demonstrating good practice for producers, archives, assist an organisation in evaluating its digital assets and users of data; and Core Requirements: Ten Basic in terms of scope, priorities, resources and overall Characteristics of Digital Preservation Repositories, a readiness to address digital preservation concerns. set of high-level principles released in 2007 that reflects consensus on audit and control requirements and practice for digital archives and provides a framework for implementing good practice. The guidance refers to other relevant community documents where applicable

3 A R Kenney, NY, McGovern and Entlich et al, Digital Preservation Management: Implementing Short-term Strategies for Long-term Problems, an on-line tutorial developed and maintained for the Digital Preservation Management workshop series since 2003. See the “Obsolescence and Physical Threats” section of the tutorial at: http://www.icpsr.umich.edu/dpm/dpm-eng/oldmedia/ index.html

IHSN Working Paper No. 003 December 2009

2. Principles and Standards

Data archives have been preserving data in digital as a problem to be solved by the community. It form since the 1960s (see Annex C for a complete provided a framework for the community documents set of milestones for data preservation). The digital that followed, many of which incorporated components preservation community has been emerging as an of the 1996 report. international, inter-disciplinary domain for more than a decade. References in this guidance to the digital The report identified these four challenges faced by preservation community include any institution or organisations in managing digital collections over time. individual in any organisational context in which participants have acknowledged formally or informally As technology upon which digital content relies the intention of preserving digital content across one or is superseded or replaced by new technological more generations of technology. The publication of the developments, effective preservation requires avoiding 1996 report Preserving Digital Information represented the impact of this cycle of obsolescence a first step in formalising good practice for organisations engaged in digital preservation. A series of community -- As technology evolves, digital content documents followed that report, in particular a set of needs to be migrated over generations of approved and emerging international standards that technology began with the Open Archival Information System (OAIS) Reference Model in 2003. This section defines -- In addition to technological concerns, the current framework of community practice for digital organisations need to address relevant preservation using these documents and indicates how legal and organisational issues to community documentation offers practical guidelines accomplish digital preservation objectives for preserving data. -- To address the challenges of digital Founding Document: Preserving Digital preservation, there is a need for Information, 1996 community-wide infrastructure

The 1996 Preserving Digital Information: Report of the These challenges remain relevant and continue to Task Force on Archiving of Digital Information report shape the development of the community, as well as marked the starting point of the digital preservation providing priorities for organisations that preserve community. This proved to be a seminal report digital content. identifying the core challenges of digital preservation for the community that manages digital collection -- a The report identified five integrity features for digital set of essential criteria for well-formed digital content, content that guide preservation efforts. Table 1 indicates roles and responsibilities for preserving it and a set of good practice associated with each feature. principles. The report remains useful to organisations in developing an understanding of digital preservation

Table 1 Good Practice for Integrity Features of Digital Content Integrity Feature Related Good Practice Content: ensures that essential elements of digital content are preserved An organisation is expected to explicitly identify and actively manage data to be preserved Fixity: requires that changes to content are recorded, ideally from the At minimum, this feature might be addressed through routine use of a checksum (a computed value generated moment of creation onward by widely-available utilities that uniquely identify a digital file) to detect intentional or unintentional changes to data and notify data managers for action Reference: ensures content is uniquely and specifically identifiable in For example, an organisation is required to adopt and maintain a persistent identifier approach – a system for relation to other content across time assigning and managing enduring identifiers that allows digital objects to be consistently and uniquely referred to over time Provenance: requires digital content be traceable to its origin (point of This feature requires that an organisation records information (captured as metadata) on the creation and creation) or, at minimum, from deposit in a trusted digital repository action that have affected the content since its creation (e.g. data deposited in an archive, migrated from one file format to another) Context: documents and manages relationships of digital content An organisation that preserves data document relationships between its own digital content and, to the extent possible, to data managed by other organisations

5 Principles and Good Practice for Preserving Data

It is essential that an organisation responsible for -- Data producers (or the digital archives preserving digital content is able to demonstrate how they engage) should identify a technology it addresses each of these five integrity features. Apart environment (software and hardware from the content feature, these features are addressed combination) sufficient in scale and through the capture, creation and management security to manage the data of metadata for preservation, as indicated in the descriptions of each feature. Preserving metadata is -- Data producers (or the digital archives initially addressed in the next section on OAIS. they engage) should adopt and implement migration and other preservation -- The core principles for data preservation strategies to ensure that data are usable that the report described included: over time

-- Data producers (creators, providers and These principles define preservation-related owners) are responsible for archiving their responsibilities for digital content that informed the digital content to ensure the objects are development of community standards following the preserved release of the 1996 report.

-- Data producers might meet this obligation Open Archival Information System (OAIS) by depositing data to be preserved in a Reference Model and Related Standards certified digital archive (one that meets the current definition of a trusted digital The Open Archival Information System (OAIS) repository) Reference Model represents the most formal and comprehensive expression of preservation as a process -- Data producers (or the digital archives that is available for community use. OAIS describes the they engage) should select data for roles, functions and content components involved in preservation so that sufficient resources preserving digital content. can be designated to maintain the data over time The international space community initiated the development of the Open Archival Information System -- Data selected to be preserved must be (OAIS) Reference Model in 1995 to address its own need packaged and prepared for preservation to manage data better. It convened an international and with sufficient metadata and in formats representative group of experts. OAIS was approved by that are able to be preserved (e.g. avoiding the International Standards Organisation (ISO) as ISO software-dependent formats where 14721 in 2003. Prior to the development of the OAIS possible, embracing ASCII and now XML standard, there was no language or practice in place formats) within the community to enable productive discussion of good practice across organisations, domains or

Figure 1 The High-level OAIS Reference Model

Preservation Planning P C RODUCER Descriptive Data Descriptive O Info Management Info N queries SU Ingest result sets Access MER SIP Archival orders Storage AIP AIP DIP

Administration

IHSN Working Paper No. 003 December 2009

national boundaries. The promulgation of the OAIS In practice, a producer or consumer might be an standard has provided a common language for the individual, an organisation or a system authorised to digital community. The OAIS model was developed to provide or use data. These roles are useful for discussing be applicable in any organisational context in which responsibilities that pertain to preserving data and for digital content is managed for the long term. Most raising awareness of it. Organisations that preserve organisations that manage digital collections have data are required to ensure these roles are assigned and indicated an intention to design and implement their implemented effectively. digital repositories in accordance with the OAIS. Any organisation that preserves data should be aware of OAIS Functions the OAIS Reference Model, commit to developing its technological approaches in accordance with The OAIS identifies seven areas of activity (defined in OAIS, and monitor the community for OAIS-related Table 3) performed in managing digital content over developments. Section 4 of this guidance, on the time: Ingest, Archival Storage, Data Management, technological infrastructure for preserving digital Administration, Preservation Planning, Access and content, uses the OAIS as a framework. Common Services. Each area consists of individual functions for performing that activity. OAIS Roles OAIS Content Packages and Metadata OAIS defines three explicit roles (Producer, Manager and Consumer) and one implicit role (Archive) that The OAIS defines three types of information packages participate in an OAIS environment, as illustrated in (described in Table 4) that contain digital content over Table 2. time as data are preserved.

Table 2 OAIS Roles for Preserving Data Table 4. OAIS information packages and implications OAIS Role Implications of Roles Producer generates or is responsible for data to be preserved and provides the for data preservation data to the archive or unit responsible for preservation Manager has direct authority for the archive (or unit responsible for preserving These packages represent stages of life-cycle data), specifically the approval of the budget and development of high- level policy for preservation initiatives management for data: acquiring (SIP), storing and Consumer is authorised to use the data, either for internal purposes or general archiving (AIP) and making available to authorised public use users (DIP). An organisation that preserves digital Archive is the unit responsible for coordinating OAIS functions and managing the content packages defined by the OAIS standard (see Table 4) content must interpret and develop the SIP, AIP, and

Table 3 OAIS Functions OAIS Function Implications for Data Preservation Ingest: enables secure acceptance and quality Documents the identification of data to be preserved, whether by a unit within an organisation that creates data or by a designated control of data for preservation data archive. The preserving unit needs to determine what types of data (e.g. file formats) will be preserved, what kinds of validation or quality control will be performed, what steps need to be taken to ensure the data are able to be preserved (e.g. conversion to common file formats or to ASCII-based formats) Archival Storage: ensures the secure storage, Organisations might be inclined to equate system back-up with archival storage. For preservation purposes, archival storage is management, and retrieval of preserved data replaced by multiple copies, preferably in more than one media to avoid massive failure, e.g. on-line and off-line, distributed copies stored in more than one location, and enough copies to avoid loss. The trend within the community is towards managing most copies on-line. When that option is not possible, the most effective and feasible means for managing multiple copies should be adopted and the procedures updated as technology and requirements evolve. Data Management: supports ongoing An organisation requires complete and current information about data as units (e.g. data files) and aggregates (collections, studies accumulation and availability of administrative or series), as well as information about the repository and its management (e.g. policies, standards, procedures and performance data on the operation of and documentation of measures). More standardised approaches to this administrative documentation are slowly beginning to emerge. the content of an archive Administration: develops, maintains, and Administration is where organisational and technological responsibilities intersect. Preservation decisions are captured in policies and applies policies and procedures to operate and procedures that inform repository operations, e.g. policies on acquisition, storage management, disaster recovery and access. coordinate OAIS functions Preservation Planning: develops and Provides advice and procedural support for data preservation, e.g. preservation strategies for specific data content types. This should recommends standards, policies, procedures be a shared responsibility within the community: no organisation reinvents the wheel, no single organisation is required to do it all and means for preserving digital content and all organisations are able to benefit from lessons learned by others and share their own experiences. Access: finds and delivers content in an archive Preservation enables long-term access. The OAIS requires a repository to be able to find and deliver the data over time in effective to authorised users and efficient ways, whether access is provided for internal or external use. Common Services: refers to the operating An organisation needs to ensure these common services are robust and reliable enough to sustain data over time. Adhering to system, network services and security preservation requirements might require supplementing the common services used for current activities, e.g. more levels of security services needed to implement and manage a or different approaches to encryption to ensure that data are able to be read and used. preservation repository

7 Principles and Good Practice for Preserving Data

Table 4 OAIS Information Packages and Implications for Data Preservation OAIS Information Package Implications for Data Preservation Submission Information Package (SIP): the state (format and structure) of the digital Effective preservation should make it possible always to be able to demonstrate the state of content when it is provided to the repository or unit responsible for preservation data as submitted (the original) Archival Information Package (AIP): the state of digital content when it is placed into An AIP accumulates information on preservation activities (e.g. data are migrated to a new archival storage for preservation format, preservation metadata are updated) so anything that might affect preservation or use of data is known and documented Dissemination Information Package (DIP): the state of the digital content when it is The format and structure of the DIP reflects technology as it evolves. Access is expected to made available to users over time use current technology to deliver data; preservation uses proven and reliable technology

DIP to meet its own requirements in alignment with metadata. Documentation that conforms to the Data community practice, especially metadata requirements Documentation Initiative (DDI) generally meets the when an organisation is responsible for providing its requirements for descriptive and structural metadata data to other organisations. (see below for more information about DDI). In addition, organisations that preserve data should Preservation Metadata and OAIS identify a minimal set of metadata for data files (e.g. data, file name, checksum, file format, content type, The Open Archival Information Systems (OAIS) such as data or documentation) to ensure ongoing Reference Model incorporated the five integrity access to data. features identified in the 1996 report into its definition of Preservation Description Information (PDI) within These are other metadata developments that pertain the Archival Information Package, the formal definition to or might be useful for data preservation: of digital content for preservation. The 2009 revision of OAIS acknowledged the importance of documenting -- The minimal set of Dublin Core metadata and managing rights associated with digital content elements is a simple set of metadata widely by adding access rights’ information (e.g. copyright used internationally to describe resources, information for data, licensing information for data, including data. right to copy and otherwise preserve data) to this list of integrity features for digital content. -- The Data Documentation Initiative (DDI) has produced DDI 2.0, a fixed format for Preservation metadata include all the information, metadata that might be used to document including descriptive, structural, administrative and data for preservation purposes, and DDI technical metadata, required by an organisation to 3.0, a new version of DDI that addresses preserve data. As with other preservation terms, there the whole life-cycle for data management, is no authorised definition of preservation metadata, including preservation, and shows although common approaches to creating and managing promise for broad on-line use when tools metadata for preservation are beginning to emerge to implement it are in production for use within the community. There is also no international by any organisation. standard for preservation metadata. -- Statistical Data and Metadata Exchange The Preservation Metadata Implementation (SDMX), a technical specification for Strategies (PREMIS) data dictionary is the only the exchange of data and metadata, was metadata definition specifically developed for approved as an ISO standard in 2005 (ISO preservation. The most recent version of the PREMIS 17369) and is increasingly used for time- data dictionary was released in 2008 and, since the series and other data. initial version in 2005, tools and examples for using PREMIS have emerged, making PREMIS more likely to -- Information Technology – Metadata be used by organisations for preserving digital content. Registries (ISO/IEC 11179), a specification PREMIS is a community initiative that recommends for standardising and registering data good practice for preservation metadata and it might elements to make data understandable be formalised as a standard. and shareable.

For data, metadata play a very large role in These metadata initiatives are valuable community preservation because the documentation required developments that also support preservation through to understand and use data is a specialised form of improved documentation of the structure and content of

IHSN Working Paper No. 003 December 2009

data. All these developments are typically implemented Repository Architecture) repository using Extensible Mark-up Language (XML), a textual software packages (available through data format designed for simplicity, interoperability the Fedora Commons) is one example and generalised use. XML tags contain internal of open source4 repository software that documentation and description that make XML-based is increasingly used and has a solution documents easier to use and preserve. community to help organisations adapt Fedora for preservation; Often, metadata schemas are used in combination because each was developed for a specific purpose -- Purchase a software package and adapt and each has strengths and limitations. One step for it for use: one example of a software integrating multiple metadata schemas is a crosswalk package available for purchase and which that maps one schema to another, e.g. this element on acknowledge the OAIS standard is the schema A is the same as or relates to this element on Rosetta software from Ex Libris, developed schema B. In the preservation context, the Towards as a result of partnership with the National Interoperable Preservation Repositories (TIPR) project Library of New Zealand; is an example of a project focusing on the packaging and exchange of metadata for preservation. TIPR is a two- -- Develop repository software, alone or year project funded by the Institute for Libraries and in collaboration with partners: building Museum Services (IMLS) in the USA and scheduled to a repository requires ongoing support conclude in 2010. for programming time, equipment and software tools for developing the repository Managing Data Using OAIS that might require licensing.

The OAIS Reference Model can be a powerful Each of these options requires an understanding communication and planning tool for organisations of the OAIS Reference Model standard and related responsible for preserving data. The model can be preservation requirements identified in this guidance used to illustrate the assignment of responsibilities, and an ongoing commitment to maintaining the the costs associated with various stages of life-cycle repository. management, the completion of policies and other required documents and the priorities for preservation- If an organisation implements a repository to manage related developments. There is a community its digital content, a technological platform might need expectation that an organisation that preserves digital to be assembled, scaled appropriately, documented, content will be aware of and working to comply with the monitored for key developments and upgraded or OAIS standard. enhanced as necessary (note the discussion of cloud- computing services under OAIS Compliance in the next Although the community trend is towards using section of the guidance). There are four likely scenarios repository software to preserve digital content, this for developing a technical infrastructure. Each has its section does not presume its use for the effective strengths but the correct solution will depend on the management of data by an organisation over time. An capabilities and priorities of the organisation. The four organisation is not required to implement software options are: repository to preserve content, although doing so might make preservation decisions and actions more Build Opt to build on internal IT for DP consistent and scalable. There are no currently- program. available and in-production software packages fully Focus on developing capacity and compliant with the OAIS, particularly in the functional storage options. areas of Administration and Preservation Planning, though that is an objective of many developers. There Extend Define based upon gap analysis. are several options available to organisations for Add capacity and storage. repository software to enable the preservation of digital Ensure adequate redundancy in content: back-up.

-- Adopt an open-source package and adapt 4 Open source software is freely available, although there are costs the package to suit requirements: the to the organisation in implementing it in terms of programming Fedora (Flexible Extensible Digital Object time, technical skills that might need to be acquired through consulting, and other costs.

9 Principles and Good Practice for Preserving Data

Join Evaluate alternatives, then Producer-Archive Interface Methodology outsource or subscribe to Abstract Standard (PAIMAS) service. Ensure longevity through effective In 2003 the OAIS working groups released the contract/service management. Producer-Archive Interface – Methodology Abstract Standard (PAIMAS), which was approved as an ISO Collaborate Specialise in core areas of expertise standard in 2006 (ISO 20652: 2006). The PAIMAS or capacity. standard addresses the relationship between producers Extend capacity through of data and an archive from the point of deposit collaboration. through the ingestion of the data into the archival Share well-defined responsibilities. storage component of the archive, including the Leverage extra-repository benefits.5 specification of deposit agreements, requirements for and frequency of deposits and responsibilities of The factors an organisation will need to consider the producer and the archives (e.g. specifying what in determining a workable digital repository option happens if there are questions about or difficulties with for its situation include: the extent of digital content the deposit). As Figure 2 illustrates, PAIMAS overlays to be managed (scale), the degree of access the the OAIS Reference Model, encompassing the roles of organisation has to technical support for implementing the producer (the organisation or individual producing the repository, the resources available to implement or owning the data with responsibility for its long- and sustain a repository, and the time-frame the term management) and the archive (the entity that organisation needs to have a repository in place. takes on long-term management of the data) -- from the identification of the content for submission to The OAIS Reference Model identified a road map for the archive to the ingestion of the data into the OAIS the development of other related standards including: system.

Figure 2 The Overlap Between OAIS and PAIMAS

Preservation Planning P C RODUCER Descriptive Data Descriptive O Info Management Info N queries SU Ingest result sets Access MER SIP Archival orders Storage AIP AIP DIP

Administration

a preservation metadata standard; submission and The standard delineates the interaction between the dissemination standards, for identification of digital producer that creates (or produces) the digital content content (persistent identifiers); preservation methods; and the archive which accepts long-term responsibility accreditation (or certification); and the need for a for preserving this digital content. Managing data over producer-archive interface standard. These standards time might involve placing data in a data archive or an continue to be developed and maintained by the organisation establishing its own archival function for community preserving its data.

5 DPM Tutorial, “Program Elements”, available at http://www.icpsr. umich.edu/dpm/dpm-eng/program/techinf.html

IHSN Working Paper No. 003 December 2009

Figure 3 The Four Phases of PAIMAS

Develop agreement (data Define the information to be delivered, complementary Validate the transferred to be archived elements, schedule Actual transfer of the objects objects Phase objectives

Preliminary Formal Definition Transfer Validation Phase Phase Phase Phase Preliminary Dictionary Anomalies Validation Agreement Formal model Agreement Submission Agreement

AIP creation

PAIMAS identifies four phases (Figure 3) to These phases make explicit the steps an organisation coordinate the transition of digital content from current must take when archiving data. use to archival storage: Formalising the interaction between the producer and -- Preliminary Phase: identifies and the archive might make the process of identifying data specifies digital content to be preserved, to be preserved easier and more efficient. When the data including the initial contact between the are managed internally, the PAIMAS stages serve as a producer and the archive, a preliminary reminder to make the process of identifying content for definition of the scope of the interaction preservation and placing content into archival storage and drafts of the Submission Agreement. implicit and intentional. An inventory of data for which a data-producing agency is responsible is a first step -- Formal Definition Phase: defines in addressing its digitalpreservation obligations. The requirements and expectations for Digital Curation Centre in the UK recently developed preserving data and finalises the a Digital Audit Framework to assist in identifying submission agreement stipulating the and assessing data for long-term retention and use; terms and means of depositing the digital this might be helpful to data-producing agencies in content into the archive completing this required step.6

-- Transfer Phase: agrees upon the Ongoing Community Developments of requirements for sending data to an Standards and Principles archive (or moving it to archival storage for management) and defines a workflow Community documents, of which the documents identifying who will be responsible if discussed in this section are examples, continue to there are questions about or issues with a emerge as good digital preservation practice is defined transfer and standardised. The Blue Ribbon Task Force on the costs of digital preservation, introduced in section 1 of -- Validation Phase: defines the quality this guidance, is one example of emerging good practice checking (validation steps) that will be within the community. The pending approval of an completed for data to be preserved ISO standard for the certification of digital archives,

6 Digital Curation Centre, Data Audit Framework, 2008: http:// www.data-audit.eu/.

11 Principles and Good Practice for Preserving Data

discussed in Section 3, is another example. This section international delegates since 2004. The International discusses the current core set of community documents. Conference on the Preservation of Digital Objects It can be difficult for organisations to keep up with (iPres) has been held annually since 2004. iPres is the developments in the digital preservation community, first regularly-held international conference entirely but it is essential that conferences and community devoted to digital preservation. literature – both ‘grey’ and more formally published – are employed as primary ways for them to keep track. In addition to periodical articles about the challenges Increasingly, conferences offer proceedings that enable of preserving digital content that have appeared in the organisations to learn about developments reported at professional literature of relevant domains – since meetings. the late 1960s for archival literature, the 1980s for library literature and the 1990s for museum literature Since 2004, several professional conferences – increasing numbers of publications since 1996 either have provided ongoing sources of information about highlight or are devoted to digital preservation issues, current developments and practice by featuring or including Ariadne in the UK and Preserving Access including digital preservation. The Society for Image to Digital Information (PADI) in Australia. These are Science and Technology (IS&T) Archiving Conference community-based publications providing information has been held since 2004. This conference series was and updates on research and developments pertaining initiated by the digital-imaging domain and includes to digital content. The International Journal of Digital both general digital preservation sessions and image- Curation was launched in 2006. Digital curation specific preservation sessions. The Digital Curation includes both data curation and digital preservation, Centre (DCC) has hosted conferences that include making it particularly relevant to organisations that digital preservation topics in the programme, with manage data.

IHSN Working Paper No. 003 December 2009

3. Putting the Principles into Practice

The OAIS Reference Model describes the roles, The most common way for an organisation to functions and content types of an effective digital demonstrate compliance with this attribute is through archive. The OAIS standard does not provide much the promulgation of a mission statement encompassing guidance on developing the organisational side of data preservation. The Inter-university Consortium for operating an OAIS system. The Attributes of a Trusted Political and Social Research (ICPSR) has a mission Digital Repository: Roles and Responsibilities report, statement that serves as a good example of for data released in 2002, defined and described expectations preservation: “ICPSR provides leadership and training and requirements for an organisation intent on being in data access, curation and methods of analysis a trusted digital repository (TDR). The document for a diverse and expanding social science research acknowledged that the OAIS Reference Model did community”. It has become clear, based on accumulated not provide specific guidance for organisations on examples within the digital preservation community, conforming to the standard, on managing the data that it is difficult or impossible for an organisation to from an organisational rather than a technological sustain a digital preservation initiative if there is no perspective and on achieving and maintaining trust in overt support for the programme at top organisational an organisation that manages data over time. levels or by key stakeholders.

The TDR document has become a de facto community Organisational Viability standard for good practice by organisations that manage digital content over time. The seven attributes The organisational viability attribute requires that an of a trusted digital repository provide the framework organisation that accepts responsibility for preserving for applying the standards and principles in Section 2 of digital content has the wherewithal to undertake digital this guidance. The TDR attributes are adaptable to the preservation – e.g. the legal status, the mandate, mandate and scope of any organisation that preserves the resources, the expertise, the capacity and the data, and every organisation that preserves data is equipment. Additional characteristics of this attribute expected to reflect the attributes of a trusted digital include a commitment to demonstrating the viability repository. and trustworthiness of the organisation’s preservation programme; establishing well-documented This section uses practical recommendations to preservation policies and practices (e.g. procedures illustrate the application of the TDR attributes to and work-flows); defining comprehensive written an organisation responsible for managing data over agreements with depositors; reviewing and updating time. The seven TDR attributes – administrative policies and procedures on a regular basis; and engaging responsibility, organisational viability, financial in contingency and succession (trusted inheritors’) sustainability, technological and procedural planning, in case the organisation is no longer able to suitability, system security, procedural accountability preserve the data. and OAIS compliance – are each addressed in the following sections. The most important way for an organisation to demonstrate conformity with this attribute is by Administrative Responsibility developing and promulgating a digital preservation policy framework that defines the purpose, objectives, The administrative responsibility attribute requires an scope and desired outcome of an organisation’s organisation with responsibility for preserving data to preservation approach. The digital preservation make an explicit commitment to that end. Additional policy consolidates the organisation’s documentation characteristics of this attribute include a commitment addressing the characteristics of this attribute by to meet or exceed community standards; to share explicitly referring to the organisation’s relevant performance measurements with depositors; to involve policies and related documents, e.g. job descriptions, external community experts in periodic reviews to training plans, submission policies, access policies and validate and certify the organisation’s processes and policy-development procedures. The development of procedures; and to thoroughly document and be a compliant digital preservation policy is the focus of accountable for preservation-related actions. Section 4 of this guidance.

13 Principles and Good Practice for Preserving Data

Financial Sustainability required for the digital preservation programme at many levels to: The financial sustainability attribute requires an organisation responsible for preserving data to designate -- create digital content adequate resources for implementing and maintaining a data-preservation programme. An organisation should -- capture content and associated metadata be able to demonstrate that ongoing funding is available as digital objects and allocated for preserving its data. Additional characteristics of this attribute include the ability of the -- transfer those objects to and within a organisation to demonstrate good business practices digital repository through comprehensive documentation and regular audits by outside accounting professionals, to invest -- process or otherwise interact with digital limited resources wisely to support the preservation objects in the repository programme, and to seek funding sources to cover data- preservation expenses. -- find and deliver stored objects

There is currently no specific community document -- build and maintain the repository that addresses in detail the cost of preservation. One study of digital preservation challenges noted that: -- define and implement the policies and protocols that pertain to the repository “Digital preservation is an essentially distributed process including a range of -- integrate the repository and the digital different (and often differently interested) material into the broader organisational stakeholders who become involved with digital environment and the context in which it resources at particular phases of their life- operates8 cycle. To increase the prospects for digital preservation and reduce the costs, different This attribute presumes that organisations will groups of stakeholders need to become more comply with the OAIS standard and that adequate and aware of how their particular involvement appropriate technology support is in place to manage the with a digital resource ramifies across its life- data. Additional characteristics of this attribute include cycle.”7 a commitment by the organisation to consider and adopt appropriate preservation strategies; ensure there The report of the US National Science Foundation’s is appropriate infrastructure (e.g. hardware, software, Blue Ribbon Task Force on Sustainable Digital facilities) for the acquisition, storage and access to Preservation and Access is expected to provide data over time; establish a technology- management community-wide recommendations for addressing policy for preservation (e.g. procurement, replacement this attribute. Until the recommendations from the and enhancement of requisite technology, funding task force can be promulgated within the community, for technology investment); comply with relevant reasonable documentation for this attribute includes standards and best practices supported by adequate appropriate references to data-preservation expenses in technical expertise; and undergo regular external audits an organisation’s budget documentation and narrative of technology components and performance. reports that discuss data-preservation funding, e.g. the organisation’s annual report. Implementing effective preservation strategies is at the core of the technological and procedural suitability Technological and Procedural Suitability attribute. There are three dominant preservation strategies: The technological and procedural suitability attribute requires an organisation that preserves data to adopt -- Migration typically refers to the appropriate technology in accordance with community conversion of a file format from a less standards, practice and expectations. Technology is common or obsolete format to a current file format, e.g. converting data from an older 7 Arts and Humanities Data Service (AHDS), A Strategic Policy Framework for Creating and Preserving Digital Collections, 2001, 8 Kenney and McGovern, DPM Tutorial, http://www.icpsr.umich. pg 3. edu/dpm/dpm-eng/program/techinf.html

IHSN Working Paper No. 003 December 2009

version of an SPSS file format to a newer Organisations need to identify the type, quantity one or converting a documentation file and risk status (e.g. more at risk due to a decrease from one form of PDF to PDF/A, a format in use) of file formats that need to be preserved, and defined specifically for preservation.9 the technology and processes that support them. One prominent data archive had this recommendation -- Emulation refers to the ability of a regarding their preservation- management approach: computer to mimic the capabilities and functionality of older or obsolete “Routinely review, identify and improve the computers; this technique might be used use of hardware, software, data formats and to enable a computer to read older file standards to reduce the risk of storage media formats, then save them as current file deterioration or technological obsolescence… formats (which combines emulation and Review and improve policies, plan, and migration), or to enable a computer in the procedures regularly to ensure completeness future to read and use an older, obsolete and coverage for contingencies such as changes file format. in technology, standards and/or project requirements and capabilities, as well as to -- Normalisation might refer to limiting proactively engage in risk management.”11 file formats for preservation to a small set of common formats (e.g. limiting text The range of file formats in use within the files to a version of the Word format or organisation should be consolidated to minimise Open Document format), or converting duplication and eliminate problem formats. As noted software-dependent file formats (e.g. SPSS above, this process is a form of normalisation. Those system files) to less software-dependent or formats most at risk, such as those created by obsolete software-independent file formats (e.g. software or by obsolete versions of current software, ASCII or XML-based formats).10 should be targeted first.

These strategies were initially developed by the It is common for an organisation to use one or more information-technology community to solve common preservation strategies to adequately preserve its data. problems caused by technological change. The strategies ICPSR, for example, uses a combination of migration, have been adapted for preservation use and do not to convert data and documentation to accepted formats, yet have community standards defining the protocols and normalisation, to both limit the number of formats for implementing them, and do not have automated that are preserved and convert data and documentation protocols for quality control to determine the success of to software-independent formats that are easier to applying one of the strategies. preserve.

The digital preservation community continues 9 Migration might also be used to refer to migrating content from to invest significant energy and resources in the one storage media to another, e.g. from one computer disk to development and promulgation of effective digital another when one disk fails, from an off-line storage device to preservation strategies. For example, the Preservation on-line storage for managing multiple copies of digital content. and Long-term Access through NETworked Services This form of migration is more commonly referred to as media refreshing. File- format migration is a preservation strategy with (PLANETS) project is a five-year project funded by potentially significant impact because applying the strategy might the EU committed to developing an automated way of introduce changes to the content. Media refreshing is a tactic matching an appropriate preservation strategy to file for managing copies of digital content that should not introduce formats using the characteristics of the formats and changes to the content. If digital content is stored on storage media that become obsolete, extreme actions (sometimes referred rules defined by an organisation. The PLANETS work is to as digital archaeology) might have to be utilised to rescue the not yet available for production use beyond the project content and save it on current media, but this threat to digital members, but the project is one to watch for useful content due to storage media obsolescence occurs outside an developments (see Farquhar and Helen Hockx-Yu in effective digitalpreservation programme, e.g. before digital content is included in a preservation programme. Annex B for more information on PLANETS). 10 Discussion of preservation strategies often present the options as either a migration or emulation choice, leaving out normalisation entirely. Some members of the digital preservation community 11 Centre for International Earth Science Information Network believe that normalisation is not a correct term for this preservation (CIESIN). http://www.ciesin.columbia.edu/documents/ strategy, but no widely-used alternative has been proposed. CIESINpreservationpolicy.pdf

15 Principles and Good Practice for Preserving Data

There are several high-profile initiatives in the US, for preservation strategies; and establish the means UK and EU that are extending good practice for digital for feedback from all users of procedures and systems preservation to the management of data. The Digital to support problem resolution and negotiate evolving Curation Centre, for example, has developed the Data requirements between data producers and data users. Audit Framework, which consists of a four-step process to identify and assess data from a long-term management The most important documentation for preservation and use perspective. The Data Information Specialists is the digital preservation policy framework. Section Committee in the UK has produced guidance for life- 4 defines the components of and demonstrates the cycle management of research data, including a brief development of a digital preservation policy and section on preservation. Both these documents provide Annex D identifies additional policy and procedural relevant information for preserving data and are listed documents to serve as examples of creating adequate along with other useful resources in Annex B. documentation to meet preservation requirements.

System Security There is yet no community standard providing a specification for digital preservation documentation, The system-security attribute requires an organisation although there are increasing examples to use as that is preserving data to implement adequate controls a starting point for developing an organisation’s to appropriately restrict access to data and address documentation. Section 4 refers to proposed potential vulnerabilities in systems that manage components of a digitalpreservation policy standard and preserve data. Additional characteristics of this that might be used by the community. attribute include a commitment by the organisation to assure the security of systems used to manage and The TRAC requirements, introduced in Section preserve data; establish policies and procedures to 2, are built upon the OAIS and TDR. TRAC provides meet security requirements (e.g. policies that address specific examples of the kinds of documentation needed copying data, authentication of system users, firewalls, by an organisation to meet each requirement. A revised back-ups, disaster preparedness, emergency response, version of TRAC is waiting to become an ISO standard disaster recovery and training); and stress processes -- some time in 2010 if there are no extensive delays. that will detect, avoid and repair loss and document TRAC is further discussed in Section 5. and notify changes and resulting action. OAIS compliance There are yet no community standards for the comprehensive implementation of system security for The OAIS compliance attribute requires an organisation preservation, but the digital preservation community that preserves data to commit to designing and is able to benefit from the policies and practices of the implementing its preservation programmes in information-security field. The integrity and security accordance with the OAIS standard. OAIS compliance of digital content an organisation preserves need to be may be expressed in very minimal terms using a assured, and measures to protect the digital content simple narrative statement of the ways in which an must be comprehensive and well-documented. organisation’s practice corresponds with components of the OAIS Reference Model (e.g. for ingestion, our Procedural Accountability organisation has established a policy for accepted file formats; defined quality-assurance measures for The procedural accountability attribute requires an content to be preserved; and also defined the elements organisation that preserves data to fully document the to be included in each Archival Information Package), approval and implementation of decisions, policies, or in more meaningful terms by using the OAIS procedures and practices. This documentation serves as Reference Model to inform and guide the development the basis for self-assessment, audit and certification of and enhancement of an organisation’s preservation the organisation’s preservation programme. Additional practice. characteristics of this attribute include: a commitment by the organisation to enact all relevant policies and A challenge for organisations in being OAIS- procedures for specified tasks and functions and compliant is to establish a reliable technological document all preservation practices; establish a means infrastructure to sustain an OAIS system in the face of of monitoring systems and procedures to ensure their rapid and continuing change. Meeting that challenge continued operation; document and provide a rationale requires adequate funding, appropriate expertise,

IHSN Working Paper No. 003 December 2009

systematic monitoring of relevant technology and an -- Organisational Viability encompasses established process for informed decision-making the repository but relies upon aspects of in technology investment. It is not possible for an Administrative Responsibility organisation to incorporate every technological change, enhancement and upgrade, but it should -- Financial Sustainability is the establish the means of identifying essential changes most critical of the components within to implement over time that are needed to sustain the the repository, which cannot exist in its digital preservation programme. Cloud-computing is absence; this attribute relates to other an option becoming available to organisations as an financial commitments in the organisation alternative or supplement to maintaining an internal technological infrastructure. Cloud-computing is a -- Technological and Procedural form of technological infrastructure in which a provider Suitability is the next most essential of scalable services is made available over the Internet. component within the repository and DuraCloud is a recently-launched provider of cloud- determines the success of the preservation computing services aimed at archives, libraries and programme; this attribute should be museums.12 A lack of Internet access, legal requirements, influenced by and adapted from external jurisdictional constraints on where and how data can experience and practice be stored and security concerns for confidential data -- all might prohibit organisations from using cloud- -- System Security is critical to the computing. But it is becoming a viable and currently- success of the implementation, but there is affordable alternative to managing all or some portion known methodology for establishing and of technological infrastructure internally.13 maintaining system security; this attribute will be part of larger system-security Managing a Trusted Digital Repository practices, both within the organisation and externally To be a trusted digital repository, an organisation must provide continuing evidence that it adheres to -- Procedural Accountability cuts community standards and principles and continually across and underpins the trusted nature of demonstrate that it manages its preservation the repository; this attribute is dictated by programme in competent and effective ways. Doing and based upon external authorities so requires an organisation to address each of the seven TDR attributes. Having looked at the individual The TDR document contains little discussion of and attributes, an organisation needs to consider the no diagrams to illustrate these relationships between relationship between the attributes as it strives to the seven attributes or the relative significance of each, become a trusted digital repository: an absence that Figure 4 is intended to fill.14

-- OAIS Compliance is an underlying First and foremost, an organisation needs to make an principle that is an integral part of the explicit commitment to preserving data (administrative other framework components responsibility); and, secondly, an organisation needs to have the requisite mandate, training, equipment and -- Administrative Responsibility other resources (organisational viability) to preserve encompasses all the other components and data effectively. Prior to the release of the TDR document lays the foundation of a trusted repository; in 2002, the cost of managing digital content (financial this attribute is influenced by and based sustainability) had not been identified so explicitly for upon larger organisational factors (e.g. digital preservation. Technology (technological and legislation, mandates) procedural suitability and system security) was typically the focus of digital preservation discussion, rather than being the means of achieving preservation objectives; 12 DuraCloud is offered by DuraSpace, a partnership of the dSpace Federation and Federa Commons, two repository-software providers. More information about DuraCloud is available at http://www.duraspace.org/duracloud.php 14 The model was developed at Cornell University Library by Kenney and McGovern for the Digital Preservation Management (DPM) 13 One group investigating security implications in the cloud is the Workshop http://www.library.cornell.edu/iris/dpworkshop/, a Cloud Security Alliance. More information is available at http:// workshop series funded by the US National Endowment for the www.cloudsecurityalliance.org/ Humanities (NEH) http://www.neh.gov/ since 2003.

17 Principles and Good Practice for Preserving Data

and the need to demonstrate through organisational The TDR attributes reflect the three requisite evidence (procedural accountability) the adequacy of components of a sustained programme for preserving an organisation’s operations had not been explicitly digital content -- organisational infrastructure required. OAIS compliance is implicit in the attributes (“what?”), technological infrastructure (“how?”) and and thus not included in the diagram. requisite resources (“how much?”) -- as illustrated by the three-legged stool for digital preservation (Figure The TDR diagram stresses the importance of a strong 5). The organisational leg defines the scope of a digital and well-planned organisational setting for enabling a preservation programme (what data will be preserved?). sustainable approach to long-term preservation, and The technological leg encompasses the approach of the places technology as an important – but not singular digital preservation programme enabled by technology – component within that context. The placement of the as it emerges and evolves (how will the data be attributes in the diagram emphasises that technology preserved?). The resources’ leg determines the cost of should be suited to the scope and requirements of each sustaining digital preservation, based upon the scope organisation. The TDR diagram added a digital archives and approach (how much will it cost to preserve the border to the TDR attributes because one organisation data?). might maintain more than one repository, in which case the outer layers might be coordinated across the The three-legged stool provides an essential structure organisation; or a group of organisations might come for a digital preservation programme. For a programme together to manage one repository, e.g. in a partnership to be viable and sustainable, the three legs must be or other collaborative effort, when that is relevant. equally strong, and therefore balanced, to sustain data over time.

Figure 4 The TDR Attributes Applied to an Organisational Setting Figure 5 Three-Legged Stool for Digital Preservation1

Trusted Digital Repository Model 1. Administrative Responsibility 2. Organizational Viability

rchive l A s B ta o $$$$$$$$$$$$$$$$$$$$$$$$$gi rd i e D r $$$$$$$$$$$$$$$$$$$$$$$$$ Technology Resources $$$$$$$$$$$$$$$$$$$$$$$$$3. Financial $$$$$$$$$$$$$$$$$$$$$$$$$Sustainability $$$$$$$$$$$$$$$$$$$$$$$$$ $$$$$$$$$$$$$$$$$$$$$$$$$ Organisation 101010101010101010101010110104. Technological and 1 The three-legged stool concept was developed by McGovern and Kenney 10101010101010101010110101010Procedural Suitability to support the DPM workshop series. 10101010101010101101010101010 101010101010110101010101010105. System Security 10101010110101010101010101010 10101101010101010101010101010

6. Procedural Accountability (Certification) 1101010101010101010101010

IHSN Working Paper No. 003 December 2009

4. Formulating a Preservation Policy

The development and implementation of a policy is an Financial Sustainability [required] essential step to implementing an effective approach Institutional Commitment [required] to preserving data. A good policy reflects the mandate Cooperation and Collaboration [optional] of the organisation to preserve data, makes an explicit Technological and commitment by the organisation to preserve data in Procedural Suitability [required] accordance with community principles and standards, System Security [required] and fulfils a fundamental requirement of good Procedural Accountability [required] preservation practice. The process of developing the Audit and Transparency [required] policy is especially important because it consolidates an Framework Administration [required] organisation’s understanding of its digital preservation Definitions and References [desirable] commitment and brings together the group that will be involved in the process. OAIS Compliance

Every organisation that preserves data requires a The OAIS Compliance section of the policy consists policy, but every organisation should not be required of an explicit statement of the intent of the digital to reinvent the policy’s scope. There is no standard or preservation programme to comply with the Open even common format for a preservation policy, but Archival Information System (OAIS) Reference Model. there is an increasing number of policies that provide For example: good examples of formulating one. The outline should produce a digital preservation policy of no more than “In achieving its digital preservation objectives, five pages. It should address the attributes of a Trusted ICPSR recognises the need to comply with the Digital Repository; present a high-level overview of an prevailing standards and practice of the digital organisation’s digital preservation programme; reflect preservation community. ICPSR is committed current not future capabilities of the digital preservation to developing its digital preservation policies, programme; provide links to more detailed and repository and strategies in accordance frequently-updated documents, e.g. lower-level policies with the Open Archival Information System and procedures; point to the organisation’s plan (OAIS) Reference Model (ISO 14721:2003). for priorities, time-frames and future development; ICPSR tracks and responds to related OAIS and document the policy-approval and maintenance initiatives, including developments in digital process. This section defines the recommended archives certification, persistent identifiers, components of such a policy and provides data-specific preservation metadata and the producer- examples for each component.15 The policy outline archive interface. The mapping of ICPSR’s includes the following sections: preservation process to OAIS is synthesised in Digital Preservation Requirements Applied to OAIS Compliance [required] ICPSR and explained in greater detail in OAIS Administrative Responsibility [required] Conformance at ICPSR.” [ICPSR] Purpose [required] Mandate [required] The more detailed plan for making the organisation Objectives [desirable] OAIS-compliant would be contained in a separate Organisational Viability [required] document referred to in this section. In addition, Scope [required] this section might refer to the strategic plan for Operating Principles [optional] the organisation or the preservation plan, if these Roles and Responsibilities [required] documents have been developed. Selection and Acquisition [desirable] Access and Use [desirable] Administrative Responsibility

15 The policy components were developed for the curriculum of The administrative responsibility section of the policy the Digital Preservation Management Workshop (http://www. makes an explicit commitment to preserving the data icpsr.umich.edu), then vetted by the more than 500 workshop participants who have attended the workshop since 2003; they that fall within the scope of responsibility for the represent more than 250 organisations in more than 25 countries organisation and to comply with prevailing standards on five continents. and practice for digital preservation.

19 Principles and Good Practice for Preserving Data

Purpose This component might also address requirements not specifically identified as preservation, e.g. The purpose component of the policy makes legal admissibility, authenticity, Data Protection explicit the intentions of an institution regarding Acts, copyright legislation, public records acts, the preservation of data and the role preservation E-Government, National Grid for Learning (UK). The plays in fulfilling the organisation’s mission. This mandate component might refer to laws, legislation, component defines the rationale for the policy, contracts, related policies, the organisation’s mission identifies responsible parties and stakeholders, statement, regulations or other relevant documents. indicates the intended audience for the document, and places the document in the context of Objectives organisation-wide efforts. The purpose statement might range from broad to narrow, reflecting The objectives’ component of the policy states the variations in intention for different types of digital high-level aims and targets of the organisation for archives. For example: collecting, managing, preserving and sustaining access to its data. This component identifies the “This policy statement sets out Statistics benefit of the programme to an institution and its New Zealand’s approach to the long-term relationship to other objectives, goals and policies. retention and preservation of unit record For example: and aggregate datasets held by Statistics New Zealand and gives guidance on implementing “The digital preservation function at ICPSR is those principles. This statement has been organised to address these objectives: developed from international best practice, in consultation with Statistics New Zealand’s -- Maintain a comprehensive and responsive subject matter area.” [Statistics New Zealand] digital preservation programme that identifies, acquires, verifies, archives and distributes core The purpose might refer to the mission social science digital assets statement for the organisation, other high-level -- Adapt preservation strategies to incorporate policies, and the strategic plan for the organisation, the capabilities afforded by new and emerging if these documents have been developed. technologies in cost-effective and responsible ways Mandate -- Serve the needs of membership organisations by enabling uninterrupted access to digital The mandate component of the policy stipulates content over time as the technology for digital the authority, jurisdiction or governance based content creation and distribution evolves upon which the organisation has undertaken the -- Meet the archival requirements of funding preservation of the data, e.g. laws, legislation, agencies and contracting entities committed policies and mission. For example: to the long-term preservation of designated digital content “This policy follows from the overarching -- Demonstrate auditable compliance with and principle that the statistical datasets created contribute to the development of the standards by Statistics New Zealand and other agencies and practice of the digital preservation in the Official Statistics System are valuable community and irreplaceable, with their utility maximised -- Foster collaborative partnerships with social through ongoing use. The Statement of science researchers and other digital archives Principles for the Official Statistics System to make the best use of available resources and states this under the maximisation principle avoid duplicated efforts” [ICPSR] as ‘Statistical data are treated as an enduring national resource, with their value increasing The objectives’ component might refer to the through widespread and long-term use’. The organisation’s strategic plan, goals and objectives, utility of official statistics lies in their ability to budget, preservation plan and technology plans, if provide snapshots of society, the economy and these documents have been developed. the environment, and to show patterns and change over time.” [Statistics New Zealand]

IHSN Working Paper No. 003 December 2009

Organisational Viability cycle management, interoperability, evidence- based requirements and preferred methods of The organisational viability section of the policy preservation. For example: addresses the legal status as well as human and other resources needed to establish and maintain a digital “The ICPSR digital preservation function preservation programme. This section might include operates in accordance with an established set the following components: of principles. ICPSR will strive to:

Scope -- comply with the OAIS and other digital preservation standards and practice The scope component of the policy establishes -- ensure that digital content at ICPSR can be the overall time-frame, levels of responsibility, provided to users and exchanged with partner boundaries, extent, limitations and priorities of the and other digital archives so that it remains digital preservation programme. This component readable, meaningful and understandable delineates what the organisation’s preservation -- participate in the development and will include and, as importantly, will not include. promulgation of digital preservation The scope statement might be brief or extensive, community standards, practice and research- depending on the nature and scale of the content based solutions preserved. For example: -- develop a scalable, reliable, sustainable and auditable digital preservation repository “The policy covers all data captured in the -- manage the hardware, software and storage Process, Analysis and Dissemination phases media components of the digitalpreservation of the statistical process. It covers unit record function in accordance with environmental and aggregate datasets, both published and standards, quality control specifications and unpublished. It covers pilot surveys, sampling security requirements.” [ICPSR] frames and customised datasets. Statistical datasets created from administrative data and The operating principles’ component might used to create official statistics, or integrated to refer to community and organisational standards, create official statistics, are also covered by the documents that define good practice, work-flow policy. The treatment of integrated personal and process documents and procedures. data is consistent with the guidelines in the Data Integration Protocol.” [Statistics New Roles and Responsibilities Zealand] The roles and responsibilities’ component of The scope component of the policy might refer the policy describes key stakeholders and their to the organisation’s strategic plans, collection- respective roles in digital preservation, including development policies, preservation plan and creators, producers, digital repository staff, definitions of roles, if these documents have been administrators, financial managers, user groups, developed. advisors, other repositories and collaborators. The “policy should define who will undertake the Operating Principles work and assign responsibility. By connecting policies with current activities, proposed tasks The operating principles’ component of the policy can be aligned with existing work-flows and defines the key principles, models, processes and staff skills. When assigning new responsibilities, assumptions upon which the digital preservation changes should be thought through to ensure they programme is developed and implemented. This are practical and implementable. Defining roles section is particularly important in establishing will also be important for accountability.”16 This system-wide benchmarks for distributed section makes an explicit statement that digital programmes when multiple operational and preservation is a shared responsibility requiring technical processes are implemented. Common participants within and beyond the organisation. principles include adherence to standards (in particular the OAIS) and other accepted 16 Jones, S DCC Curation Policies Report. Digital Curation Centre indicators of good practice, support for life- (DCC), 2009. http://www.dcc.ac.uk/docs/reports/DCC_Curation_ Policies_Report.pdf

21 Principles and Good Practice for Preserving Data

It describes broad categories of roles and The selection and acquisition component responsibilities and cites documents containing might refer to the organisation’s collection- more specific descriptions. For example: development policy, submission guidelines and the workflow for acquiring data or other digital “As an organisation acting for its member content, if these documents are relevant or have institutions, funding bodies, and depositors, been developed. ICPSR has accepted responsibility for preserving its digital assets. Within ICPSR, Access and Use the Director, the Digital Preservation Officer, the Computer and Network Services unit, The access and use component of the policy the Collection Development unit, the topical identifies the designated communities for the archive managers and the Collection Delivery digital preservation programme and the barriers unit all contribute to the management of the and/or restrictions to use of the digital content digital preservation function and the life- for which the programme is responsible. Specific cycle of digital content at ICPSR. The ICPSR policies should be developed to further articulate Council, an elected advisory board, evaluates access and use requirements and restrictions. high-level policy documents and reviews Preservation requires demonstrable proof that an programmatic plans and progress. Roles and organisation is able to provide meaningful long- Responsibilities for Digital Preservation at term access to the content it is responsible for ICPSR provides descriptions of the roles and preserving. For example: current assignments.” “The designated community at ICPSR, as The roles and responsibilities’ component described by the OAIS, includes traditional might refer to role definitions with explicit users, e.g. social science researchers and specification of responsibilities, documentation graduate students at member institutions; and of current role assignments, job descriptions and newer categories of users, e.g. undergraduates, organisational charts. policy-makers, practitioners and journalists. To protect the identity of human subjects Selection and Acquisition who might be represented in the deposited data, ICPSR devotes significant resources to The selection and acquisition component of the developing and implementing the means to policy provides the rationale and processes for ensure confidentiality. preserving data based on specific parameters (e.g. formats, types of records, geographic scope). A “ICPSR uses current technology and tools to clear articulation of the data preserved is critical provide a range of access services. The ICPSR in ensuring that preservation supports the Data Access Policy defines the principles organisation’s mission and priorities and that and criteria for access to data in the ICPSR requisite funding is made available for preserving collections. ICPSR has developed lower-level data. For example: policies and procedures to manage access to digital content, including Procedures for “The ICPSR Collection Development Policy Processing Requests for Restricted Data, the sets forth the priorities and criteria for Privacy Policy for handling information about acquiring digital content. The ICPSR Deposit users, Release Management Procedures that Form reflects the priorities and criteria that specifies the preparation of digital content for are defined in the policy. The Guide to Social release, and the Requests for Permission to Science Data Preparation and Archiving Redistribute ICPSR Data policy that addresses provides guidance and templates for depositors the use of ICPSR digital content by other data to encourage complete and well-documented archives and distributors.” [ICPSR] deposits. The Collection Development Policy and the Guide are available on the ICPSR web- The access and use component might refer site and the related policies are available upon to the organisation’s access policy, deposit request.” [ICPSR] agreements, digital rights’ management rules and

IHSN Working Paper No. 003 December 2009

practice and user agreements, if these documents responsibility. ICPSR has long-standing and are relevant or have been developed. emerging partnerships with other data archives, other digital repositories, data producers, Financial Sustainability and data providers in the United States and internationally for digital preservation The financial sustainability section of the policy cooperation and collaboration.” [ICPSR] documents the tangible basis for sustaining the digital preservation programme and might include these The cooperation and collaboration component components: might refer to partnership agreements, operating principles and good practice for collaborative Institutional Commitment projects.

The institutional commitment component Technological and Procedural Suitability confirms and synthesises the support for the programme and the resources available to sustain The technological and procedural suitability component digital preservation. For example: of the policy summarises the preservation approach, strategy and techniques employed by the digital “To sustain its digital preservation function, preservation programme to achieve stated objectives. ICPSR has allocated a portion of its This section states the general philosophy of the membership support to digital preservation digital preservation programme and points to relevant services. In addition, ICPSR continually seeks requirements, policies, standards, guidelines and external research funding to extend its digital practice. It makes a tangible link with the preservation- preservation scope and capabilities and has planning component of the digitalpreservation secured contracts to fund specific initiatives. programme and to the organisation’s preservation plan. Detailed information about digital preservation funding is available in the ICPSR Annual “The majority of digital content in the Report and in the annual budget of ICPSR.” collections at ICPSR currently consists [ICPSR] of social science research data, requisite documentation to use and understand the The institutional commitment component data, and associated files. Upon receipt of a might refer to budgets, financial reports, fiscal deposit, ICPSR processes the digital content policies, annual reports, succession plans and to ensure that confidential information has contracts. not been included in the data; corrects errors; fills gaps in the documentation; and produces Cooperation and Collaboration distribution versions of the data. Technicians digitise documentation that is received only in The cooperation and collaboration component of hard copy format. ICPSR archives the original the policy acknowledges that the organisation’s digital content received, the normalised effort exceeds or will exceed available resources versions of processed data and superseded and might not guarantee the safety of all vital versions of data that have been distributed. The assets. This section places digital preservation archived files will enable ICPSR to retain the programmes in a broader context that recognises ability to regenerate distribution formats over their dependency on other partners and the time. For files submitted on physical storage community at large. Collaboration and partnership media, ICPSR makes archival copies but does might require formal, legally-binding agreements not preserve the original transmission media. that delineate the explicit roles and responsibilities In terms of documentation, ICPSR has been of each party. For example: an active participant in the development of and is an adherent to the Data Documentation “The active and collaborative research Initiative (DDI) standard. programme at ICPSR integrates digital preservation requirements and competencies “ICPSR has adopted normalisation and into its priorities and acknowledges digital migration as its primary digital preservation preservation as a shared community strategies. Normalisation produces file formats

23 Principles and Good Practice for Preserving Data

for data and documentation that are as close as specific aspects of the long-term protection of possible to ASCII for text, or TIFF for images, its digital assets. For example, Secure Package to enable preservation, and reduces the range Handling Procedures at ICPSR stipulates the of file formats to be preserved to ensure that procedure for receiving and sending physical the digital preservation load is manageable. packages containing sensitive data; Procedures Migration converts digital content to current for Processing Requests for Restricted Data file formats as software and related technology stipulates the steps for responding to requests evolve and copies digital content from for restricted data; and ICPSR Policy on older to newer storage media as part of a Receiving Data with Direct Identifiers defines systematic programme. ICPSR is investigating appropriate actions when data received appropriate preservation strategies for the contain direct identifiers. Disaster Planning expanding range of digital content types in its for Digital Assets at ICPSR provides a set of collections. The ICPSR Digital Preservation principles ICPSR has embraced for protecting Plan identifies the priorities, objectives, and digital assets within the broader context of action plans for the next three years.” [ICPSR] disaster planning for ICPSR.” [ICPSR]

The technological and procedural suitability The system security component might refer to component might refer to preservation strategies, security policies and procedures or to the organisation’s preservation plans for the organisation and for specific disaster-planning documentation. Disaster planning content, and deposit agreements. is an essential component of digital preservation. Examples of the policies and documentation needed System Security by an organisation are available on the ICPSR web- site (http://www.icpsr.umich.edu/icpsrweb/ICPSR/ The system-security component of the policy specifies curation/disaster/index.jsp) and Annex F contains the organisation’s commitment and approach to additional information on disaster planning. ensuring the accuracy, completeness, authenticity, integrity and long-term protection of the organisation’s Procedural Accountability digital assets. The procedural accountability section of the policy “The processing procedures for digital acknowledges the need and stipulates the means content at ICPSR actively address the need for ensuring the transparency and accountability for ensuring the accuracy and completeness of of the digital preservation programme’s policies digital content through the careful comparison and operations. This section might include these of documentation and data submitted and the components: generation of metadata and documentation for data. The implementation at ICPSR of Audit and Transparency an automated deposit form addressed the need for ensuring the authenticity of digital The audit and transparency component of the assets by requesting detailed information and policy explicitly commits the organisation to signatures for submission. ICPSR ensures periodic self-assessment and audit to evaluate, the authenticity and integrity of its digital measure and adjust the policies, procedures, content through the active and ongoing use of preservation approaches and practices of the checksums from receipt of the digital content digital preservation programme. Transparency onward. In addition, ICPSR conducts periodic enables self-assessment and audit. Self- reviews and audits of its digital content assessment and audit improve internal operations, in archival storage. Information Security facilitate external reviews and contribute to for Digital Assets at ICPSR provides more the development of effective partnerships and detailed information about the principles and collaboration. good practice for ICPSR’s approach to systems security. “ICPSR participated as a test audit in the Certification of Digital Archives research project “ICPSR has developed several lower-level conducted by the Centre for Research Libraries. policy and procedural documents that address ICPSR is committed to a two-year cycle of

IHSN Working Paper No. 003 December 2009

self-assessment and a five-year audit cycle The section might either provide or point to to evaluate, measure and adjust the policies, requisite definitions. procedures, preservation approaches and practices of the digital preservation function. A “Digital Preservation and Data Archiving complete set of current policies is available on Terms for ICPSR provides a current set the ICPSR web-site and procedural documents of definitions of terms used in this digital are available upon request.” [ICPSR] preservation policy framework and employed by the digital preservation function at ICPSR.” The audit and transparency component might [ICPSR] refer to audit and self-assessment schedules and results, strategic plans and preservation plans. The definitions’ component might refer to definitions developed by the organisation and Framework Administration glossaries adopted by it.

The framework administration component of the References policy describes the organisation’s policies and practice pertaining to the development, approval The references’ component of the policy provides and maintenance of the policy framework over citations for or pointers to key resources that time, e.g. frequency of updates and reviews, informed the development and application of maintenance roles and expiration dates. The the framework. This section identifies more framework has little value if it has not received detailed documents, both internal and external, the appropriate approval and has not been that provide a deeper expression of the mission, implemented. At a minimum, the date and underlying principles, illustrative processes source of approval and the review cycle should be and sustaining roles. It might contain citations provided. for these documents or point to a current list of relevant community standards and guidance. “The digital preservation policy framework was completed in April 2007; approved by the “Citations for Digital Preservation at ICPSR ICPSR Directors Group on May 1, 2007; and provides a list of ICPSR, digital preservation approved by the ICPSR Council at the June community and other documents that have 2007 meeting. The March 2006 predecessor informed the development and maintenance of document, ICPSR Preservation Policy, the framework.” [ICPSR] served as a starting point for developing the framework. ICPSR will review the framework The references component cited resources, every two years to ensure that it remains current community lists of standards and practice. and comprehensive as the digital preservation function at ICPSR evolves.” [ICPSR] Developing the Policy

The framework administration component This section emphasises the process of developing the might refer to the organisation’s policy policy because it not only fulfils a core requirement administration procedures and policy-approval of preservation but also provides a focal point for documentation. establishing the organisation’s data-preservation initiative. The importance of a preservation policy Definitions cannot be overemphasised. If the organisation carefully considers the mandatory, desirable and optional policy The definitions’ component of the policy components and develops a policy that reflects the identifies terms and concepts that might be priorities and context of the organisation, the result needed to understand the framework and will be a solid foundation on which the organisation can could be instrumental in strategies for securing build and sustain its data-preservation programme. institutional commitment. This is an optional section but one that can be very important. It is There are a number of factors for the organisation particularly important to include legally-required to consider in initiating and completing the policy- and other mandated terminology and definitions. development process, including:

25 Principles and Good Practice for Preserving Data

-- Who should lead and coordinate the on the time-frame for developing the policy, specific development of the policy? milestones to be met and the roles of the group in developing the policy. -- Who should be involved in developing the policy, both internally and externally? It is worth endeavouring to develop a sound and comprehensive preservation policy because it is -- Who needs to be aware of the policy essential documentation to demonstrate that an initiative before, during and after it is organisation is engaged in preserving data, and it developed? enables the organisation to, at least in part, address a number of the requirements to provide evidence of its -- Who will need to approve the policy when preservation action. The policy-development process it is completed? should be explicit, intentional and transparent within the organisation, because a key benefit of the process -- How will the organisation make the policy is raising awareness of preservation, its importance available and to whom? and its requirements. The policy-development process is not complete until the policy is finalised, approved In developing the policy, the ideal is for one or two and implemented. It is necessary for an organisation people to draft the policy document or sections of not only to have a preservation policy but also to it, then share it with a larger group for feedback and demonstrate that it is acting in accordance with it. The suggestions. Attempting to actually write the policy policy – both the process to create it and the completion as a group is often not successful and adds to the of that process – is instrumental in achieving good time required to develop the policy. When the group digital preservation practice. convenes, some effort should be devoted to agreeing

IHSN Working Paper No. 003 December 2009

5. Evaluation and Audit for Data Preservation

This guidance defines the expectations for organisations defined in sections 2, 3 and 4 of this guidance. These that preserve data. This section concludes the guidance sections provided examples and descriptions of what by considering two approaches an organisation an organisation is expected to do to meet community might consider as alternatives in demonstrating good requirements as defined by TDR and the OAIS. practice: the Data Seal of Approval, which is particularly Organisations are encouraged to read and refer to the useful for organisations that are data archives, and TRAC document as a useful resource, even if they find the Characteristics of a Trusted Digital Repository, one of the suggested options to be more helpful as a which is useful for any organisation responsible for beginners’ guide. preserving data. This section also defines a set of stages that have proven useful to organisations in achieving Option 1: Data Seal of Approval incremental but continuing progress in developing their preservation approach. The first community framework that might aid organisations in developing their data- preservation As a starting point for this final portion of the discussion, approach is the Data Seal of Approval, a concise set it is useful to provide some context for the emergence of of higher-level principles that defines expectations for performance measures for digital repositories. Both the producers of digital content, repositories that manage 1996 Preserving Digital Information report and the digital content and consumers of digital content. The OAIS Reference Model identified the need for digital principles were proposed by the National Data Archives archives to demonstrate compliance with standards in the Netherlands (DANS) in 2008. The principles and practice through a commitment to ongoing audit reflect the fundamentals of both TDR and OAIS and and certification. To address this need, the Research represent a workable approach to digital-content Libraries Group (RLG) and the US. National Archives management. The Data Seal of Approval has since been and Records Administration convened a working accepted for international use by data archives and the group that produced the Trustworthy Repositories purpose of the principles is being extended to apply Audit & Certification (TRAC): Criteria and Checklist to digital content of any kind. There is an increasing in 2007. In early-2007, the certification of digital community expectation that data archives will adhere archives became the focus of an international working to the principles and seek the Data Seal of Approval. group to develop an ISO standard. The ISO working The principles are summarised here. The principles’ group used TRAC as a starting point for its work. The aim is to be simple and straightforward, so no additional work on the certification standard is also informed discussion of them is provided. Organisations should by the Digital Repository Audit Method Based on the refer to the recommendations provided in preceding Risk Assessment (DRAMBORA) toolkit, it is a risk- sections and review the guidelines and assessment management approach for assessing digital collections, process provided for the Data Seal of Approval at developed by the Digital Curation Centre and Digital http://www.datasealofapproval.org/ Preservation Europe (DPE), and also embracing the work of the Nestor project in Germany, which produced The data-producer deposits the research data in a a catalogue of criteria for a digital archive audit and a data repository with sufficient information for others coaching approach for assisting smaller organisations to assess the scientific and scholarly quality of the in addressing such criteria. research data and compliance with disciplinary and ethical norms. The TRAC requirements have been a focal point for the community and formed the basis of most 1. The data-producer provides the research data in community efforts in audit and certification. The formats recommended by the data repository. requirements are, however, extensive and might prove daunting to organisations just starting to develop a data- 2. The data-producer provides the research data preservation approach. Therefore, this section presents together with the metadata requested by the two more concise options. Both options presume that data repository. organisations adhere to both TDR and OAIS. Each of the audit and certification approaches builds upon 3. The data repository has an explicit mission in the accumulated good practice of the community, as the area of digital archiving and promulgates it.

27 Principles and Good Practice for Preserving Data

4. The data repository uses due diligence to ensure for digital archives: the Digital Curation Centre, compliance with legal regulations and contracts DigitalPreservationEurope and Nestor met at the including, when applicable, regulations Center for Research Libraries in Chicago to consider governing the protection of people. core criteria based on this cumulative set of audit and certification developments. The purpose of the meeting 5. The data repository applies documented was to compare the progress of the groups’ work to processes and procedures to managing data that point and avoid redundant or conflicting results storage. that might complicate the process of defining good practice for audit and certification of digital archives or 6. The data repository has a plan for long-term confuse the community. They identified these ten basic preservation of its digital assets. characteristics of digital preservation repositories; these provide a convenient and manageable set of 7. Archiving takes place according to explicit work- criteria for an organisation to address. flows across the data life-cycle 1. The repository commits to continuing 8. The data repository assumes responsibility from maintenance of digital objects for the data producers for access and availability of identified community/communities. the digital objects. This characteristic represents an organisation’s fundamental commitment to preserving its 9. The data repository enables the users to utilise digital content over time. the research data and refer to them. 2. The repository demonstrates 10. The data repository ensures the integrity of the organisational fitness (including digital objects and the metadata. financial, staffing structure and processes) to fulfil its commitment. 11. The data repository ensures the authenticity of This characteristic equates to demonstrating the digital objects and the metadata the organisational viability to preserve digital content. 12. The technical infrastructure explicitly supports the tasks and functions described in 3. The repository acquires and maintains internationally-accepted archival standards like requisite contractual and legal the OAIS. rights and fulfils responsibilities. This characteristic addresses the need to take full 13. The data consumer complies with access responsibility for the digital content that is regulations set by the data repository. preserved.

14. The data consumer conforms to and agrees 4. The repository has an effective and with any codes of conduct that are generally efficient policy framework. This accepted in higher education and research for characteristic reflects the need to document the exchange and proper use of knowledge and and share the intention of the organisation to information. preserve digital content.

15. The data consumer respects the applicable 5. The repository acquires and ingests licences of the data repository regarding the use digital objects based upon stated of the research data. criteria that correspond to its commitments and capabilities. This Option 2: Ten Characteristics of a Digital characteristic addresses the need to accurately Preservation Repository define and maintain the scope of preservation by an organisation. These characteristics were agreed at a meeting convened just before the start of the ISO working group 6. The repository maintains/ensures the in 2007. The meeting included representatives of the integrity, authenticity and usability of major international audit and certification initiatives digital objects it holds over time. This

IHSN Working Paper No. 003 December 2009

characteristic reflects the essential purpose of areas for development. Some organisations might also preservation. At a minimum, an organisation need or want to undergo an external audit, either an needs to have control over its digital content informal process with a peer organisation or a more (know where it is and who has or is able to gain formal one with a recognised authority, if there is one access to it), associate minimal metadata data for the organisation. Certification of digital archives with the content and create and control more is only known as a current option for organisations in than one copy of the digital content. Germany, but the efforts of the ISO working group might result in broader certification options. In the meantime, 7. The repository creates and maintains organisations have the set of self-assessment and audit requisite metadata about actions taken tools developed by the community and identified in this on digital objects during preservation as guidance. well as about the relevant production, access support, and usage process None of the audit and certification initiatives contexts before preservation. This presumes that an organisation is able to meet all the characteristic reflects the development of an requirements the first time it completes a self-assessment appropriate preservation approach for the scope or audit process. Organisations all go through stages of and content to be preserved. incremental development as they continue to progress their capacity to achieve preservation objectives. Five 8. The repository fulfils requisite stages of development for digital preservation have dissemination requirements. This been identified and formalised:17 characteristic ensures that appropriate access is provided to preserved digital content and that Stage 1. Acknowledge: understanding intended users are aware of access policies and that digital preservation is a local concern–an practices. organisation acknowledges and determines to address preservation 9. The repository has a strategic programme for preservation planning Stage 2. Act: initiating digital preservation and action. This characteristic acknowledges projects–an organisation takes action to the need for organisations to plan for the address preservation, often through a project impact of technological change, in order to avoid loss. Stage 3. Consolidate: moving smoothly from projects to programmes –an organisation 10. The repository has technical recognises that projects are limited so develops infrastructure adequate to continuing a basic programme maintenance and security of its digital objects. This characteristic refers to the need Stage 4. Institutionalise: incorporating for each organisation to adopt technology suited the larger environment and rationalising to their requirements and resources for storing programmes –an organisation formalises the media used for preservation, storage options preservation programme as a mainstream and file formats preserved. initiative

The purpose of these characteristics is to provide a Stage 5. Externalise: embracing inter- basic guide for repositories that are preserving digital institutional collaboration and dependency – content, regardless of type and size. The principles an organisation might choose to partner and acknowledge that an organisation’s preservation collaborate with others approach will be developed at an appropriate scale to address relevant requirements and resources. These stages provide a means for the organisation to identify next steps for developing its digital- Building Capacity and Measuring Progress

17 Anne R Kenney and Nancy Y McGovern, “The Five Organisational Every organisation should engage in a thorough Stages of Digital Preservation”, in Digital Libraries: A Vision for self-assessment process using one or more of the the Twenty First Century , a festschrift to honour Wendy Lougee, community frameworks to identify its strengths and 2003.

29 Principles and Good Practice for Preserving Data

preservation programme, to provide a way of Although there is not yet a comprehensive set of communicating progress and plans with peers and standards for digital preservation, there is a known and others in the community, and to enable measuring growing base of good practice to guide an organisation’s progress towards goals. efforts. As with the development of a preservation policy, organisations should use the opportunity of Organisations that preserve data can learn several self-assessment via a working group process. Some lessons from the audit and certification developments organisations, especially those with a national or in the digital preservation community to date. Every international mandate for managing data over time, organisation responsible for preserving data should have community examples to follow in conducting an regularly employ self-assessment to demonstrate audit that will demonstrate adherence to good practice. alignment with good practice, document progress Any organisation responsible for preserving data should towards objectives, and identify priorities for be able to use the framework of good practice outlined further development. Changes within or outside the in this guidance to develop a better understanding of organisation might require it to revisit its preservation digital preservation, and use that understanding to programme status, for example: establish a system that meets their needs, requirements and scope and nature of data to be preserved. There is no -- a change in leadership in the organisation, single approach to preserving data but there is a range especially a change that removes an of documents and examples to guide organisations to advocate for preservation good practice.

-- a major shift in technology (e.g. the impact of the World-Wide Web on computing and communication practice)

-- a change in accepted preservation practice (e.g. the shift from off-line storage of preservation copies to on-line management of preservation copies)

IHSN Working Paper No. 003 December 2009

Annexes

A. Glossary

B. References

C. Milestones for Data Preservation

D. Programme and Policy Examples

E. Survey of Institutional Readiness

F. Technological Obsolescence and Other Threats

31 Principles and Good Practice for Preserving Data

Annex A. Glossary

This glossary defines terms used throughout the meaningful to users; and makes the content available guidance and provides a stand-alone reference to to authorised users. Most archives now commit relevant terms and concepts for ongoing use by readers. themselves to designing their digital repositories in accordance with the OAIS Reference Model. Access: The act of making information available. Digital preservation ensures long-term access to digital Source: ICPSR. content. The OAIS requires that an archive should supply and deliver digital content to authorised users Checksum: “A checksum or hash sum is a fixed- (consumers); delivery might be to an individual or to an size datum computed from an arbitrary block of digital access-delivery system. data for the purpose of detecting accidental errors that may have been introduced during its transmission Source: OAIS: CCSDS 650.0-B-1, p. 1-7. or storage. The integrity of the data can be checked at any later time by re-computing the checksum and Administration: The OAIS entity that coordinates comparing it with the stored one. If the checksums and controls day-to-day operations of an OAIS system. do not match, the data were certainly altered (either The OAIS identifies the policies and other documents intentionally or unintentionally).” that are the responsibility of Administration and are required by an OAIS. Source: http://en.wikipedia.org/wiki/Checksum

Source: OAIS: CCSDS 650.0-B-1, p. 1-7. Common Services: The OAIS services that include the operating system, network services and security Archival Information Collection (AIC): An services. AIC is an Archival Information Package (AIP) that represents an aggregation of other Archival Information Source: OAIS: CCSDS 650.0-B-1, p. 1-8 Packages. Consumer: The OAIS role that refers to individuals, Source: OAIS: CCSDS 650.0-B-1, p. 1-7. organisations or other systems that interact with an OAIS system to retrieve and use preserved content in Archival Information Package (AIP): An that system. AIP is an Information Package composed of Content Information and related Preservation Description Source: OAIS: CCSDS 650.0-B-1, p. 1-8 Information (PDI). An AIP is a core concept of OAIS and the basis for storing and preserving digital content. Content Information: Content Information There is yet no standard package, but active research comprises a Data Object (one or more files or and development within the digital preservation data-streams) and its Representation Information community has produced examples for organisations to (information that allows the object to be rendered by a use. computer). For example, Content Information could be a single table of numbers representing the responses to Source: OAIS: CCSDS 650.0-B-1, p.1-7. a survey but excluding the documentation that would explain its history and origin, how it relates to other Archival Storage: The OAIS entity responsible surveys, etc. for the storage and retrieval of Archival Information Packages. Source: Trusted Digital Repositories: Attributes and Responsibility Source: OAIS: CCSDS 650.0-B-1, p. 1-8. Context Information: The information that Archive: An organisation that identifies information documents the relationship of the Content Information it is committed to preserving; takes responsibility to its environment, including why the Content for that content for a specified period of time; uses Information was created and how it relates to other approaches that ensure the preserved content remains Content Information.

IHSN Working Paper No. 003 December 2009

Source: Trusted Digital Repositories: Attributes OAIS: The Open Archive Information System and Responsibility (OAIS) Reference Model, an ISO standard that formally expresses the roles (producer, management, consumer Data Management: The OAIS entity that captures and, implicitly, archives), functions (common and coordinates the information required to operate services, ingest, archival storage, data management, the OAIS system, including information on individual administration, preservation planning and access), objects, aggregations of objects and the repository and content (submission information package, archival system itself. Data Management receives updates from information collection, archival information package other functions and provides reports to other functions and dissemination information package) of an archive. in response to requests. It was approved as an ISO standard in 2003. OAIS underwent a five-year review beginning in 2007. Source: OAIS: CCSDS 650.0-B-1, p. 1-9. Source: The Consultative Committee for Space Designated Community: An OAIS concept Data Systems (CCSDS) of NASA coordinated the describing the constituency for which the archived development of the OAIS. Its final version, produced information should be relevant and understandable. by CCSDS before the OAIS became an ISO standard was CCSDS 650.0-B-1. This is the version often cited Source: ICPSR and referenced because it is in the public domain. The OAIS is an extensive document full of descriptions and Dissemination Information Package (DIP): examples of additional concepts. It was accepted by the “The Information Package derived from one or more International Standards Organisation in 2003 (ISO AIPs, received by the Consumer in response to a request 14721: 2003) and updated in 2009. to the OAIS.” An archive works with Consumers over time to ensure DIPs remain useful. Persistent Identifier: A persistent identifier refers to enduring identifiers that allow digital objects Source: OAIS: CCSDS 650.0-B-1, p. 1-10. to be consistently and uniquely referred to over time. For preservation purposes, over time refers to the life Information Package: Content Information of the object. and associated Preservation Description Information needed to aid preservation of Content Information. Notes: often, persistent identifiers are discussed in The Information Package has associated Packaging relation to resources that are available via the World Information used to specify the Content Information Wide Web, but even off-line digital objects need to have and Preservation Description Information. persistent identifiers for preservation purposes. There are a number of options for implementing persistent Source: Trusted Digital Repositories: Attributes identifiers – this article is one concise source that and Responsibilities. reviews available options.

Ingest: The OAIS entity that accepts Submission Source: http://www.ariadne.ac.uk/issue56/ Information Packages from Producers; ensures the tonkin/ quality of the content adheres to established criteria; generates the Archival Information Packages for storage; Preservation Planning: The OAIS entity that and confirms that AIPs and additional information to monitors the environment of the OAIS to ensure it is find and use the AIPs (Descriptive Information) are able to be maintained across generations of technology received and secured by Archival Storage. and provides recommendations to Administration regarding standards and policies; the implication of Source: OAIS: CCSDS 650.0-B-1, p. 1-11. technological developments; migration plans; and other strategies for maintaining preserved content, Management: The OAIS role referring to information packaging appropriate to requirement and individuals or organisations that establish policies and content types, test plans and associated prototypes for approve the budget for an OAIS system. implementing recommendations.

Source: OAIS: CCSDS 650.0-B-1, p. 1-11. Source: OAIS: CCSDS 650.0-B-1, p. 4-2.

33 Principles and Good Practice for Preserving Data

Producer: The OAIS role that refers to individuals, Submission Information Package (SIP): An organisations and systems that provide information to Information Package provided by the Producer to the be preserved by the OAIS. OAIS for long-term management.

Source: OAIS: CCSDS 650.0-B-1, p. 1-12. Source: OAIS: CCSDS 650.0-B-1, p. 1-13.

Annex B. References

This set of references identifies sources used in Observations Ocean Report. 2005. http:// developing the guidance, sources for additional dmac.ocean.us/dacsc/docs/march2005_dmac_ information on digital preservation and specific topics plan/dmac_partI_3.15.05.pdf within the scope of the guidance, and sources of Data Management Policy. Rural Economy and information for ongoing community updates. Land Use (RELU, UK). 2006. http://www. relu.ac.uk/about/data.htm Good Practice – Data-specific or related Data Policy. Earth Observing Laboratory (EOL). Undated. http://www.eol.ucar.edu/data/data- Atkins, D E. Revolutionizing Science and Engineering policy through Cyberinfrastructure. US National Data Policy. Economic and Social Research Science Foundation (NSF). 2003. http:// Council (ESRC, UK). 2001. http://www. www.nsf.gov/od/oci/reports/atkins.pdf esrcsocietytoday.ac.uk/ESRCInfoCentre/ Beagrie, N, Chruszcz, J and Lavoie, B. Keeping Research Images/DataPolicy2000_tcm6-12051.pdf Data Safe: A Cost Model and Guidance for UK The Dissemination of Government Geographic Universities. Joint Information Systems Data in Canada: Guide to Best Practices. Committee (JISC). 2008. http://www. Geo Connections. 2008. http://www. jisc.ac.uk/media/documents/publications/ geoconnections.org/publications/Best_ keepingresearchdatasafe0408.pdf practices_guide/Guide_to_Best_Practices_ CUGIR Data Management and Distribution Summer_2008_Final_EN.pdf Policy. Cornell University Geospatial Dusa, A. “A Data Archive for Social Sciences in Information Repository (CUGIR), Romania.” IASSIST Quarterly. Summer 2001. 2006. http://cugir.mannlib.cornell.edu/ http://iassistdata.org/publications/iq/iq25/ CugirDataMgmtPolicy.20060828.pdf iqvol252dusa.pdf Data Archive Policy. US National Radio Astronomy Establishment of a National Policy to Archive Surface- Observatory (NRAO). Undated. http://www. Geophysical Data Memo. United States nrao.edu/admin/do/dataarchive.shtml Geological Survey (USGS). 2009. http:// Data Archiving and Networked Services (DANS, water.usgs.gov/admin/memo/GW/gw09.02. Netherlands). Data Seal of Approval. Undated. html http://www.datasealofapproval.org/sites/ Gold, A. “Cyberinfrastructure, Data, and Libraries: default/files/Data_Seal_of_Approval_1-4.pdf Part 1 and Part 2 - A Cyberinfrastructure Data Archiving Policy. Statistics New Zealand. Primer for Librarians.” D-Lib Magazine. Undated. http://www.stats.govt.nz/about_us/ September/October 2007. http://www.dlib. policies-and-guidelines/data-archiving-policy. org/dlib/september07/gold/ aspx Govett, M W, M Doney and P Hyder. The Grid: Data Audit Framework. Digital Curation Centre (DCC, An IT Infrastructure for NOAA in the UK). 2009. http://www.dcc.ac.uk/tools/daf/ 21st Century. National Oceanic and Data Bank Role. Greek Social Data Bank (GSDB). Atmospheric Administration (NOAA). Undated. http://www.gsdb.gr/databank_role_ 2004. http://nomads.ncdc.noaa.gov/docs/ en.html noaagridcomputing.pdf Data Documentation Initiative (DDI). http://www. Guide for IOSS Data Providers. Integrated icpsr.umich.edu/DDI/ Ocean Observing System (IOOS). 2006. Data Management and Communications Plan http://www.sccoos.org/interactive/dmac/ for Research and Operational Integrated bestpractices/Guide_for_IOOS_Data_ Ocean Observing Systems. National Office Providers_060206.doc for Integrated and Sustained Ocean

IHSN Working Paper No. 003 December 2009

ICPSR Guide to Social Science Data Preparation and National Science Foundation Cyberinfrastructure Archiving Guide: Best Practice Throughout the Council. Cyberinfrastructure Vision for Data-Like Cycle (4th ed). Inter-university 21st Century Discover. National Science Consortium for Political and Social Foundation (NSF). 2007. http://www.nsf. Research (ICPSR). 2009. http://www.icpsr. gov/od/oci/ci_v5.pdf umich.edu/icpsrweb/ICPSR/access/dataprep. NewDISS - A 6- to 10-Year Approach to Data pdf Systems and Services for NASA’s Earth Science ICPSR Digital Preservation Policy Framework. Inter- Enterprise. National Aeronautics and university Consortium for Political and Space Administration (NASA). 2002. Social Research (ICPSR). 2007. http://www. http://esdswg.eosdis.nasa.gov/pdf/ND_Reprt. icpsr.umich.edu/DP/policies/dpp-framework. pdf html Paris, J. “Collaboration In Africa.” IASSIST Quarterly. Jones, S and M Donnelly. Data Management Plan Summer 2002. http://iassistdata.org/ Template. Digital Curation Centre (DCC). publications/iq/iq26/iqvol262paris.pdf 2009. http://www.dcc.ac.uk/docs/templates/ Preston, J and S Moses. A Public Digital Repository DMP_checklist.pdf for Jamaica. eJamaica.org. 2007. http://pubs. Jones, S, S Ross and R Ruusalepp. The Data Audit or08.ecs.soton.ac.uk/121/3/ejamaica2.pdf Framework: A Toolkit to Identify Research Policy-making for Research Data in Repositories: A Assets and Improve Data Management in Guide. Data Information Specialists Committee Research-led Institutions. Presentation at UK. 2009. http://www.disc-uk.org/docs/guide. iPRES, 2008. http://www.data-audit.eu/docs/ pdf DAF_iPRES_paper.pdf Rural Economy and Land Use: Guidance Data Kanyengo, C. Managing Digital Information Management. Economic and Social Data Resources in Africa: Preserving the Integrity Service. 2006. http://www.data-archive.ac.uk/ of Scholarship. University of Zambia. relu/reluaug2006.pdf 2006. http://www.ascleiden.nl/pdf/ Stebe, J and I Vipavc. “The Social Science Data Archive elecpublconfkanyengo.pdf in Slovenia.” IASSIST Quarterly. Summer Krejci, J. “The Czech Sociological Data Archive.” 2001. http://iassistdata.org/publications/iq/ IASSIST Quarterly. Summer 2001. http:// iq25/iqvol252stebe.pdf iassistdata.org/publications/iq/iq25/ Stewardship of digital research data: a framework of iqvol252krejci.pdf principles and guidelines. January 2008: http:// Laaksonen, H, S Borg and J Stebe. “Setting up www.rin.ac.uk/our-work/data-management- Acquisition Policies for a New Data Archive.” and-curation/stewardship-digital-research- IASSIST Quarerly. Spring 2006. http:// data-principles-and-guidelines www.iassistdata.org/publications/iq/iq30/ A Strategic Policy Framework for Creating and iqvol301laaksonen.pdf Preserving Digital Collection. Arts and Lesaoana, M A. “Data Archiving in Africa: The South Humanities Data Service (AHDS). 2001. African Experience.” IASSIST Quarterly. http://ahds.ac.uk/strategic.pdf Winter 1998. http://iassistdata.org/ Treloar, A, D Groenewegen and C Harboe-Ree. “The publications/iq/iq21/iqvol211lesaoana.pdf Data Curation Continuum.” D-Lib Magazine. Murakas, R and A Rämmer. “Estonian Social Science 2007. http://www.dlib.org/dlib/september07/ Data Archives: Past and Future Perspectives.” treloar/09treloar.html IASSIST Quarterly. Summer 2001. http:// Woolfrey, L. “The Establishment of the African iassistdata.org/publications/iq/iq25/ Association of Statistical Data Archivists iqvol252murakas.pdf (AASDA).” IASSIST Quarterly. Summer 2007. National Research Council (NRC). Environmental http://www.iassistdata.org/publications/iq/ Data Management at NOAA: Archiving, iq31/iqvol312woolfrey.pdf Stewardship, and Access. 2007. http://www. Woolfrey, L. “A Survey Data Archive Network in Africa nap.edu/catalog.php?record_id=12017#toc - Possibilities and Practicalities.” IASSIST National Science Board (NSB). Long-Lived Quarterly. 2007. http://www.iassistdata.org/ Digital Data Collections Enabling Research publications/iq/iq31/iqvol311woolfrey.pdf and Education in the 21st Century. National Woollard, M. UK Data Archive Preservation Policy. Science Foundation (NSF, US). 2005. United Kingdom Data Archive. 2009. http://www.nsf.gov/pubs/2005/nsb0540/ http://www.data-archive.ac.uk/news/ nsb0540.pdf publications/preservationpolicy.pdf

35 Principles and Good Practice for Preserving Data

Good Practice – relevant, but not data-specific Keakopa, S M Trends in Long-Term Preservation of Digital Information: Challenges and Possible Beagrie, N. “Digital Preservation: Best Practice and Solutions for Africa. University of Botswana. its Dissemination,” Ariadne (43). 2005. http:// 2008. http://www.codesria.org/Links/ www.ariadne.ac.uk/issue43/beagrie/ conferences/el_publ08_eng/segomotso_ British Library Digital Preservation Strategy. British keakopa.pdf Library. 2006. http://www.bl.uk/aboutus/ LIFE: Life Cycle Information for E-Literature http:// stratpolprog/ccare/introduction/digital/ www.life.ac.uk/ digpresstrat.pdf NSF Blue Ribbon Task Force on Sustainable Digital Consultative Committee for Space Data Systems, Preservation and Access, Interim Report. Reference Model for an Open Archival December 2008. http://brtf.sdsc.edu/ Information System (OAIS), draft recommended bibliography.html standard. October 2009. http://public.ccsds. National Library of Australia. Preserving Access to org/sites/cwe/rids/Lists/CCSDS%206500P11/ Digital Information (PADI) since 1995. http:// CCSDSAgency.aspx www.nla.gov.au/padi/ Core Requirements: Ten Basic Characteristics of Digital Parse Insight Deliverable D2.1. Draft Road Map. Preservation Repositories. Centre for Research Permanent Access to the Records of Science in Libraries (CRL). 2007. http://www.crl.edu/ Europe (PARSE) Insight. 2009. http://www. archiving-preservation/digital-archives/ parse-insight.eu/downloads/PARSE-Insight_ metrics-assessing-and-certifying/core-re D2-1_DraftRoadmap_v1-1_final.pdf Digital Repository Audit Method Based on Risk Peters, D “DISA: Insights of an African Model for Assessment (DRAMBORA). Digital Curation Digital Library Development.” D-Lib Magazine. Centre and DigitalPreservationEurope (DPE). 2001. http://www.dlib.org/dlib/november01/ Revisions 2006-2009. http://www.dcc.ac.uk/ peters/11peters.html tools/drambora/ Preservation Metadata Implementation Strategies Farquhar, A and H Hockx-Yu. Planets: Integrated (PREMIS). http://www.loc.gov/standards/ Services for Digital Preservation. International premis/ Journal of Digital Curation 2.2 (2007): 88-99. Preserving Digital Information: Report of the Task http://www.ijdc.net/ijdc/article/view/46/59 Force on Archiving of Digital Information. Guidelines for the Preservation of Digital Commission on Preservation and Access Heritage. National Library of Australia. and The Research Libraries Group. 2003. http://unesdoc.unesco.org/ 1996. http://www.clir.org/pubs/reports/ images/0013/001300/130071e.pdf pub63watersgarrett.pdf Invest to Save: Report and Recommendations of RLG-OCLC, Trusted Digital Repositories: Attributes the NSF-DELOS Working Group on Digital and Responsibilities, An RLG-OCLC Report. Archiving and Preservation. 2003. http://delos- May 2002. noe.iei.pi.cnr.it/activities/internationalforum/ Resource Centre. Digital Curation Centre. http://www. Joint-WGs/digitalarchiving/Digitalarchiving. dcc.ac.uk/resource/ pdf Ribes, D, et al. “The Long Now of Technology It’s About Time, a joint report by NSF and LC on Infrastructure: Articulating Tensions research challenges in digital archiving. 2003. in Development.” National Science http://www.digitalpreservation.gov/library/ Foundation (NSF), Journal of the Association resources/pubs/docs/about_time2003.pdf for Information Systems (JAIS) 10 (5). 2009. Jones, M, and N Beagrie. Preservation Management Suleman, H. An African Perspective on Digital of Digital Materials: A Handbook. First edition Preservation. University of Cape Town, 2001, current version maintained by the Digital 2008. http://www.husseinsspace.com/ Preservation Coalition. http://www.dpconline. research/publications/iwdph_2007_african. org/graphics/handbook/index.html pdf Jones, S. DCC Curation Policies Report. Digital Webb, C. UNESCO Guidelines for the Preservation of Curation Centre (DCC). 2009. http:// Digital Heritage. 2003. http://unesdoc.unesco. www.dcc.ac.uk/docs/reports/DCC_Curation_ org/images/0013/001300/130071e.pdf Policies_Report.pdf

IHSN Working Paper No. 003 December 2009

Annex C. Milestones of Data Preservation

These milestones in the development of standards 1985-1986: UK Data Archive and the BBC are partners and practice provide context and background for the in producing a new Domesday Survey, released guidance, and complete the comprehensive community on interactive video disc landscape for preserving data. 1991: Australian Centre for Remote Sensing (ACRES) rescues aging space data from disintegration by 1881: J S Billings, then Director of what was to become migrating from high-density magnetic tapes to the US National Library of Medicine, suggests optical tape to Herman Hollerith that a mechanical system based on cards be used to tabulate the Census. -- The “Bromley Principles” on full and open Hollerith develops a punch-card system used access to global change data are published with the 1890 Census, resulting in the first data http://www.gcrio.org/USGCRP/ in an automatically-stored format DataPolicy.html 1928: IBM introduces a rectangular hole punch-card that becomes the industry standard -- The UK Data Archive releases its first 1939: A committee at the US National Archives compact disk (CD) determines that federal agencies (rather than archivists) can determine whether records 1992: GenBank launched by the US National Centre stored in punch-cards have historical value and for Biotechnology Information should be preserved. Following this decision, 1993: The Aboriginal Studies Electronic Data Archive few agencies retain any punch- card records for (ASEDA) is launched by the Australian Institute historical purposes of Aboriginal and Torres Strait Islander Studies 1950: The US Census Bureau begins using the first non-military computer -- The South African Data Archive (SADA) 1960s: The establishment of data archives and national is established by the Centre for Science data programs for preserving data Development (CSD) of the Human 1961: National Archives of Australia Sciences Research Council (HSRC) 1962: Inter-University Consortium for Political and Social Research (ICPSR) 1994: UK Data Archive becomes web-based resource 1968: National Archives and Records Administration 1996: The US Commission on Preservation and (US), custodial program for electronic records Access (CPA)/Research Library Group (RLG) 1968: Statistical Package for the Social Sciences (SPSS) publishes a seminal report on preserving digital released information http://www.oclc.org/programs/ 1970: General Household Survey (UK) started ourwork/past/digpresstudy/final-report.pdf 1971: Statistical Analysis System (SAS) released 1997: The Social Science Data Archive (ADP) is 1975: UK Data Archive begins affiliation with ICPSR established in Slovenia 1976: Council of European Social Science Data Archives 1998: The Sociological Data Archive (SDA) of the (CESSDA) established Institute of Sociology in Prague opens to the 1980: US Census Bureau establishes the State Data public Center Program, a system to facilitate public 1999: The UK’s Arts and Humanities Data Service access to data stored on computer tapes. All US (AHDS) begins “Preservation Management states joined the program by mid-1980s of Digital Materials,” a project to develop a 1985: Ronald Reagan’s White House issues National handbook giving guidance on digital preservation Security Decision Directive 189, stating: 2000: The US Library of Congress receives funding for “[i]t is the policy of this Administration that, to the National Digital Information Infrastructure the maximum extent possible, the products of and Preservation Program (NDIIPP) to “provide fundamental research remain unrestricted.” a national focus on important policy, standards and technical components necessary to preserve -- The US Census Bureau becomes the first digital content” government agency to make information available on CD-ROM

37 Principles and Good Practice for Preserving Data

-- The Dutch Digital Preservation Testbed 2007: The US National Science and Technology is established as a part of the Digitale Council establishes the Inter-agency Working Duurzaamheid programme with the goal Group on Digital Data of achieving lasting accessibility of digital government information -- An interest group for people interested in data management in Africa is formed at -- The Internet becomes the principal mode IASSIST’s conference of dissemination for the 2000 US census data 2008: Data Seal of Approval launches http://www. datasealofapproval.org/ 2001: The UK Data Archive begins providing downloadable data from the Internet -- First meeting of AASDA, the African 2002: Trusted Digital Repositories: Attributes and Association of Statistical Data Archivists, Responsibilities and Preservation Metadata and at the DataFirst Survey Data Archive at the the OAIS Information Model are both published University of Cape Town, South Africa by RLG/OCLC 2009: CCSDS publishes a revision of the Reference -- Initial Open Archival Information System Model for an Open Archival Information System (OAIS) standards are released, providing (OAIS) for public examination and comment a framework for long-term digital http://cwe.ccsds.org/moims/docs/MOIMS- information preservation and access, DAI/Draft%20Documents/OAIS-candidate-V2- including terminology and concepts markup.pdf for describing and comparing archival architectures [where, by whom?] -- CCSDS publishes Metrics for Digital Repository Audit and -- The Globus Toolkit is released by the Certification: http://wiki. Globus Alliance, establishing grid digitalrepositoryauditandcertification. computing as a major component of the org/pub/Main/WebHome/ scientific computing infrastructurehttp:// MetricsForDigitalRepository www.globus.org/ AuditAndCertificationWBv03a.doc

2003: National Academy of Science releases an Sources for Milestones assessment of the US National Archives and Records Administration’s proposed digital- Gold, Anna. “Cyberinfrastructure, Data, and Libraries: archiving plan Part 1 - A Cyberinfrastructure Primer for Librarians.” D-Lib Magazine. September/October -- The US National Institutes of Health (NIH) 2007. http://www.dlib.org/dlib/september07/ adopts a data-sharing policy http://grants. gold/09gold-pt1.html. nih.gov/grants/policy/data_sharing/ ———, “Cyberinfrastructure, Data, and Libraries: Part 2 - Libraries and the Data Challenge: Roles 2004: Digital Curation Centre launched in the UK, and Actions for Libraries.” D-Lib Magazine. introducing the term digital curation as the September/October 2007. http://www.dlib.org/ combination of data curation and digital dlib/september07/gold/09gold-pt2.html. preservation The Inter-university Consortium for Political and Social 2005: The US National Science Board (NSB) issues the Research and Cornell University. report Long-Lived Data Collections: Enabling “Digital Preservation Management Workshops and Research and Education in the 21st Century Tutorial.” http://www.icpsr.umich.edu/dpm/ http://www.nsf.gov/pubs/2005/nsb0540/ index.html. 2006: The Accelerated Data Programme (ADP) is Krejci, Jindrich. “The Czech Sociological Data Archive.” launched. The project “provides training and IASSIST assistance in data curation to National Statistics Quarterly. Summer 2001. http://iassistdata.org/ Offices (NSOs) in developing countries” publications/iq/iq25/iqvol252krejci.pdf.

IHSN Working Paper No. 003 December 2009

Lesaoana, Maseka A. “Data Archiving in Africa: The The Task Force on Archiving of Digital Information. South African Experience.” IASSIST “Preserving Digital Information.” The Quarterly, Winter 1998. http://iassistdata.org/ Commission on Preservation and Access and the publications/iq/iq21/iqvol211lesaoana.pdf. Research Libraries Group. 1996. http://www. McDonald, John. “Towards a National Digital clir.org/pubs/reports/pub63watersgarrett.pdf Information Strategy: A Review of Relevant Woolfrey, Lynn. “The Establishment of the African International Initiatives.” Libraries and Archives of Association of Statistical Data Archivists Canada. 2005. http://www.collectionscanada. (AASDA).” IASSIST Quarterly. Summer 2007. http:// gc.ca/obj/012033/f2/012033-400-e.pdf www.iassistdata.org/publications/iq/iq31/ Stebe, Janez and Irena Vipavc. “The Social Science Data iqvol312woolfrey.pdf. Archive in Slovenia.” IASSIST UK Data Archive. “Across the Decades of the Archive.” Quarterly. Summer 2001. http://iassistdata.org/ 2009. publications/iq/iq25/iqvol252stebe.pdf http://www.data-archive.ac.uk/ukda40/about/ Suber, Peter. “Timeline of the Open Access Movement.” timeline.asp. 2009. http://www.earlham.edu/~peters/fos/ United States Census Bureau. “History.” 2009. timeline.htm. http://www.census.gov/history/www/through_the_ decades/overview/.

Annex D. Programme and Policy Examples

These examples of policy and practice from a range Cornell University Geospatial Information of programmes, including data archives and other Repository (CUGIR). CUGIR Data Management institutions, identify appropriate size and scale and Distribution Policy. 2006. http://cugir.mannlib. for organisations that preserve data. The guidance cornell.edu/CugirDataMgmtPolicy.20060828.pdf itself addresses organisations of any size or with any complexity of structure (e.g. a programme that produces Note: This document provides a good overview data, a unit that preserves data, a data archive). of policies and procedures with respect to the institution’s management and access policies, Arts and Humanities Data Service (AHDS). including data and metadata management, A Strategic Policy Framework for Creating and data security, distribution, use and rights. Preserving Digital Collection. 2001. http://ahds. ac.uk/strategic.pdf Data Archiving and Networked Services (DANS). Data Seal of Approval. Undated. http:// Note: This report provides an overview of a www.datasealofapproval.org/sites/default/files/Data_ high-level approach, case studies and best Seal_of_Approval_1-4.pdf practice for creating strong digital preservation policies. Note: This document provides “guidelines for the application and verification of quality Centre for International Earth Science aspects with regard to creation, storage and Information Network (CIESIN). CIESIN (re)use of digital research data in the social Policy for Preservation of Digital Resources. 2002. sciences and humanities”, and identifies what http://www.ciesin.columbia.edu/documents/ each stakeholder must address to ensure data CIESINpreservationpolicy.pdf are safeguarded.

Note: CIESIN is a well-known programme that DataFirst has provided a number of policy documents for the community. This policy is brief but http://www.datafirst.uct.ac.za/home/ offers an example of concise language and the identification of roles and duties of the Note: DataFirst is a survey data archive and CIESIN staff. training facility in South Africa.

39 Principles and Good Practice for Preserving Data

Earth Observing Laboratory (EOL). Data National Centre for Atmospheric Research Policy. Undated. http://www.eol.ucar.edu/data/data- (NCAR). The Computational and Information policy Systems Laboratory Strategic Plan. 2009. http://www. cisl.ucar.edu/dir/StrategicPlan/CISLSP2009-2014.pdf Note: This policy outlines how EOL will deal with data from differing sources, access and Note: This document provides an example levels of service provided. of an organisation enumerating its strategic goals and identifying areas of growth and Economic and Social Research Council development. (ESRC). Data Policy. 2001. http://www. esrcsocietytoday.ac.uk/ESRCInfoCentre/Images/ Rural Economy and Land Use (RELU) DataPolicy2000_tcm6-12051.pdf Data Management Policy. 2006. http://www.relu. ac.uk/about/data.htm and http://relu.esds.ac.uk/ Note: This policy integrates subjects such as reluaug2006.pdf Principles, Collaboration with other Agencies, Charging Policy, Ethics and Intellectual Note: This high-level overview provides a Property. Although this might be too broad gateway to a more detailed data-management for an overall data-archive policy, it could be plan with an extremely comprehensive useful in constructing subordinate policies. identification of roles and responsibilities.

Greek Social Data Bank (GSDB). Data Bank Statistics New Zealand. Data Archiving Policy, Role. Undated. http://www.gsdb.gr/databank_role_ undated. http://www.stats.govt.nz/about_us/policies- en.html and-guidelines/data-archiving-policy.aspx

Note: Although this is not a policy, it could UK Data Archive. UK Data Archive Preservation be helpful in drafting an organisation’s role, Policy. 2009. http://www.data-archive.ac.uk/news/ goals and founding principles. publications/preservationpolicy.pdf

Laaksonen, Helena, Sami Borg & Janez Note: This comprehensive policy also serves Stebe. “Setting up Acquisition Policies for a New as a guide to some procedural aspects of the Data Archive.” IASSIST Quarterly. Spring 2006. archive such as IT architecture and security. http://www.iassistdata.org/publications/iq/iq30/ iqvol301laaksonen.pdf University of Zambia. Christine W Kanyengo. Managing Digital Information Resources in Africa: Note: Provides a detailed framework for preserving the integrity of scholarship. 2006. http:// establishing acquisition policies based on www.ascleiden.nl/pdf/elecpublconfkanyengo.pdf experience from the Finnish Social Science Data Archive (FSD) and the Slovene Social Note: Identifies issues facing digital Science Data Archives (ADP). preservation in Africa, specifically a lack of policies and infrastructure.

IHSN Working Paper No. 003 December 2009

Annex E. Survey of Institutional Readiness

The Digital Preservation Management: Implementing 2a. Do you have a written agreement with principal Short-term Strategies for Long-term Problems workshop stakeholders on defined roles and responsibilities? series developed this checklist to help organisations consider their digital assets in terms of scope,  Yes priorities, resources and overall readiness to address  No digital preservation concerns. The survey is available  Don’t know in the Digital Preservation Management tutorial that supplements the workshop curriculum http://www. 2b. Do you have policies and guidelines covering icpsr.umich.edu/dpm/dpm-eng/foundation/tdr/ selection, de-selection and acquisition? readiness.pdf The checklist incorporates the themes of organisational, technological and resource concerns,  Yes which are introduced in the guidance.  No  Don’t know At the start of planning, it is very common for many organisations to answer “No” or “Don’t know” 2c. Have you defined and promulgated quality- to many of the questions. This survey is intended to creation requirements and procedures? identify requisite components of a digital preservation programme to enable effective planning.  Yes  No A. Organisational Infrastructure  Don’t know

Organisational readiness is best reflected in the 2d. Are comprehensive deposit guidelines in place? development and adoption of explicit policies that address digital preservation commitments and  Yes decisions. Often an organisation undertakes a  No programme without first ensuring necessary policies  Don’t know and controls are in place. 2e. Do you have written transfer requirements? Mission  Yes 1. Can your institution’s mission statement be  No interpreted as supporting a long-term commitment  Don’t know to the preservation of valuable digital materials that your agency has acquired or created? 2f. Have you explicitly defined preservation strategies appropriate to digital collections and objects that  Yes you have committed to preserving?  No  Don’t know  Yes  No Policies and Procedures  Don’t know

2. Do you have written policies and procedures that If yes, please describe: address long-term access (as opposed to those covering digitization)? If “No” or “Don’t know”, skip to section B. If “Yes”, continue to 2a.

 Yes — go to question 2a  No — skip to section B  Don’t know — skip to section B

41 Principles and Good Practice for Preserving Data

Authority Implementation

3. If you answered “Yes” to any part of question 2, 4. Has your organisation implemented the policies were these digital preservation documents vetted and practices contained in these documents? by senior management?  Fully  Fully  Partially  Partially  No but in progress  No but in progress  No  No  Don’t know  Don’t know

B. Technological Infrastructure

Organisations tend to rely on or create digital content PDF files first and address long-term access later. This section addresses current and planned digital objects and  have now  will have  no collections, storage management and depositories. Numeric data files Digital Collections  have now  will have  no 5. Do you currently have the following types of digital objects that your institution has committed Databases and spreadsheets to maintaining over time? (Some objects might fall into multiple categories. Please check all that  have now  will have  no apply.) Geographic information systems (GIS) Licensed e-journal files (articles, issues, journals)  have now  will have  no  have now  will have  no Audio-visual files Institutional records (in any format)  have now  will have  no  have now  will have  no Other (please list) Web-sites

 have now  will have  no 6. Is there any digital material in your holdings for E-mail which you lack the operational and/or technical capacity to mount, read and access?  have now  will have  no  Yes Word-processing files  No  Don’t know  have now  will have  no If “Yes”, please describe: Digital-image files

 have now  will have  no

IHSN Working Paper No. 003 December 2009

Archival Storage

7. Are you using any of these kinds of file storage? Other (Please describe.) (Please check all that apply.)

On-line (i.e. spinning magnetic disk) Obsolescence  access copies  master files  backup 9. Have you undertaken any action to extend the Magnetic tape life of digital content threatened by obsolescence of file formats, storage media and the supporting  access copies  master files  backup hardware to access it or other associated hardware and software? CD, DVD or other optical or magneto-optical disk File Formats  Yes  No  access copies  master files  backup Storage Media  Yes  No Solid state Storage Drive  Yes  No  access copies  master files  backup 10. What action have you taken? Other (please list)

Depository Storage Practice 11. Have you established any kind of digital depository 8. Does your storage programme include (please arrangements for managing your digital collections check all that apply): over time?

 multiple copies of digital content managed  Yes online?  No  Don’t know  the use of high-quality storage media? Depository Development  on-line or off-line copies stored in geographically- distributed locations? 12. If yes, have you:

 an access-controlled area for the machines and  developed a depository in-house? media on which files are stored?  acquired proprietary software to implement the  an environmentally-controlled area for storage depository? media?  acquired open-source software to implement the  a disaster-recovery plan? depository?

 a media-testing programme?  outsourced the development/maintenance of the depository?  a media-refreshing/migration plan?  contracted with a third party organisation for  back-up? depository services?

 off-site storage for back-ups?  joined a consortium for developing/delivering depository services?

43 Principles and Good Practice for Preserving Data

 made other arrangements? (describe) OAIS Compliance

If you have ticked any of these, please bring whatever 14. Is your organisation committed to the development documentation you have on the arrangements. or use of an OAIS-compliant depository for the long-term preservation of digital objects that are Security or will be within the scope of your preservation programme? 13. Does your depository have security and other mechanisms in place to ensure the integrity of  Yes objects against intentional or accidental security  No threats?  Don’t know

 Yes  No  Don’t know

C. Resources

Once the need to establish a digital preservation 17. Is there adequate organisational expertise to programme is recognised and there is the will to do so, develop a digital preservation programme? the organisation must be ready to build and sustain the programme. This section covers resources: financial,  Yes human and technical.  No  Don’t know Sustainable Funding 18. Is there adequate technical expertise to develop a 15. Does your institution currently have funding digital preservation programme? dedicated to the long-term maintenance of your digital collections?  Yes  No  Yes  Don’t know  No  Don’t know 19. Does senior management view digital preservation as a key priority? If “Yes”, please describe the extent and nature of funding.  Yes  No If “No”, please outline potential sources of and plans  Don’t know for acquiring funding. 20. Is there adequate support for staff training in Staffing digital preservation?

16. Are there staff at your institution specifically  Yes charged with digital preservation responsibility?  No  Don’t know  Yes  No 21. Does your institution currently use outside  Don’t know sources of expertise for digital preservation (e.g. consultants, contracts)? 16a. If “Yes”, how many?  Yes 16b. If “Yes”, please list their titles.  No  Don’t know

IHSN Working Paper No. 003 December 2009

Technological Infrastructure Administrative Structure

22. Is the current technological infrastructure of your 24. Would the digital preservation programme be: organisation adequate to build and/or sustain a digital preservation programme, with requisite  incorporated into existing units? upgrading and enhancement over time?  a separate unit?  Yes  No 25. How would you rank the following factors as threats  Don’t know to the loss of digital material at your institution within the next three years? 1=greatest threat, Please describe the status of the infrastructure and 5=smallest threat: potential concerns.  Technological obsolescence

 Insufficient policies or plans for preservation

 Insufficient resources for preservation 23. Has your institution dedicated funds annually for technology development, replacement and  Inadequate support from senior staff upgrading?  Lack of expertise  Yes  No  Other; please describe:  Don’t know

Annex F. Technological Obsolescence and Other Threats

This annex includes a discussion of file format Although some file format specifications are largely obsolescence, hardware obsolescence and physical independent of specific software (for example, encoding threats to digital content. The content was extracted schemes such as ASCII and Unicode), most are tied to from section 3 of the Digital Preservation Management: individual or related groups of software. The software Implementing Short-term Strategies for Long-term and its related file format specification usually evolve Problems on-line tutorial. The full content of the tutorial together and their fates are closely linked. Therefore, it is available at http://www.icpsr.umich.edu/dpm/ The makes sense to discuss software obsolescence and file primary author of this text was Richard Entlich at format obsolescence together. Cornell University Library, with periodic additions and updating since the tutorial’s launch in 2003. What Factors Contribute to File-Format Obsolescence? Obsolescence: File Formats and Software File formats can become obsolete for a number of Computer files, the objects normally thought of as reasons: the main target of digital preservation, are presented according to pre-defined structural and organisational -- Software upgrades fail to support legacy principles. These principles, usually referred to as a files. file format, are typically laid out in a document called a format specification. A format specification provides -- The format itself is superseded by another the details necessary to construct a valid file of a or evolves in complexity. particular type and to develop software applications that can decode and render such files. -- The format “take-up” is low or industry fails to create compatible software.

45 Principles and Good Practice for Preserving Data

-- The format fails, stagnates or is no longer schemes, and data-transfer protocols). So users must be compatible with the current environment. watchful of files that either rapidly evolve or stagnate, since both are prone to obsolescence. -- Software supporting the format fails in the marketplace or is bought by a competitor To decode an old file format, the format specification and withdrawn. must be available. Therefore, the degree of control the creator of a format specification exerts over its Why are File Formats a Challenge to Digital publication has a significant impact on the format’s Preservation? vulnerability to obsolescence. Specifications tend to fall into one of three categories: A number of factors have contributed to the challenge presented by digital file formats. During the early decades Proprietary, closed specifications. Proprietary of computing, the threat of file format obsolescence and closed specifications represent some of the most to the long-term maintenance of digital objects was enduring and successful software in use. However, not widely recognised. No systematic efforts were these also tend to evolve quickly and exist in many made to collect software documentation or file format different versions for different platforms, with only specifications. Without proper documentation, the limited backward compatibility provided. In fact, there task of trying to interpret an old file, or even determine is substantial commercial incentive to avoid good what format it was written in, is daunting. Thousands backward compatibility, since the need to share files of file formats and their variants have been created. ultimately forces all users, including those who would Only recently has an effort been made to catalogue and prefer to keep using older versions, to upgrade to newer document them and understand their relationships and versions. Commercial vendors must regularly release variations. Tools are beginning to emerge to automate new versions of their software with added features the process of identifying and characterising files by and functions in order to entice users to upgrade and their formats. provide a revenue stream. Unfortunately, experience has shown that even very old specifications for versions Most software is upgraded on a regular basis. of commercial file formats withdrawn from the market Although most applications can read files created in the might never be released. Also, as one might expect, previous version and perhaps the one before that, the proprietary and closed file formats are interpreted ability to read older versions is often lost. Files that have with the highest accuracy by the manufacturer’s not been migrated might not be readable by the latest own software. Therefore, such formats are the most software version, and the older-version software might vulnerable to obsolescence since they face the dual risk no longer be available or run on a current computer, or of rapid specification change and being tied to a single under a current version of the operating system. product or company.

Also, due to the complexity and dynamic nature Furthermore, today’s wildly successful software of many file formats, it can be extremely difficult to can be tomorrow’s also-ran or distant memory. There determine whether a file moved from one format to has been tremendous consolidation in the commercial another (or to a newer version of the same format) has software industry and many products have disappeared retained all its characteristics and functions. following mergers and acquisitions. Others have succumbed to competition from superior or more Are Some File Formats Less Vulnerable to cleverly-marketed products. Obsolescence than Others? Proprietary, open specifications. Some Since all software is subject to obsolescence, all file proprietary formats have a lower risk because the formats used by that software are also vulnerable. On specification has been publicly released, allowing other the surface, it might seem that the files used by software companies (and non-commercial entities) to produce that is more stable (i.e. not undergoing much change) software that can read them. However, commercial would be less subject to obsolescence, and that is true in entities can and sometimes do change their minds the short term. But software that stands still inevitably about leaving specifications open. For example, the becomes obsolete also, because it fails to adapt to the DjVu image format was an open specification for a changing computing environment that it must operate while before its owner decided to make changes and not in (e.g. CPU architecture, operating systems, encoding release them to the public.

IHSN Working Paper No. 003 December 2009

Such formats can provide a compromise between -- good metadata support (in open format closed specifications and international standards by such as XML) combining commercial clout with some degree of openness. There is even the potential for middle ground, -- good range of functionality but not overly as when a subset of a proprietary format is adopted complex as a standard, such as the case of PDF/A, an archival version of PDF grounded in Adobe’s proprietary -- available interchange format with usable but open specification. PDF/A differs from PDF in target the requirement of XML-based metadata and the elimination of elements likely to complicate decoding -- built-in error checking and accelerate obsolescence, such as audio and video, JavaScript, unembedded fonts and device-dependent -- reasonable upgrade cycle colour spaces. Obsolescence: Hardware and Media Most proprietary but open specifications are still vulnerable to the whims of market forces. In addition Rapid obsolescence of computer hardware has been to being subject to arbitrary withdrawal, they can be a particular characteristic of the industry since its abandoned for commercial reasons. inception more than 50 years ago. A one-or-two order of magnitude improvement in power, speed, efficiency Non-proprietary, open specifications. In or cost per value has occurred every several years in terms of guaranteed long-term availability, published areas such as CPU speed, memory chip density, storage specifications produced by international standards device capacity, video-processing rate and data- bodies are the safest. Generally, representatives from transmission rate. many different constituencies are involved in creating the standard, helping to ensure that it balances the Such monumental changes have a powerful needs of a wide variety of users and is not beholden to obsolescent effect. New computers replace older ones any particular commercial interest. Broad participation not just because they are quantitatively faster, more also helps provide an incentive for wide support once productive or higher in capacity (though this impact the standard is completed. Backward compatibility alone is a considerable incentive to upgrade), but with older, related standards is usually a priority because they enable qualitative changes in the function and there are no commercial pressures for rapid of the device. Entire classes of computer uses, and obsolescence. One of the most recent examples of this the software and file formats with which they were is the standardisation of the OpenDocument Format implemented, would not exist today had computing (ODF) as an open-source format. ODF stemmed from hardware not advanced to such a degree. These include the open XML-based OpenOffice.org specification and uses such as CAD, digital-imaging, audio and video was approved by ISO as a standard in 2006. production, simulation, and graphic web surfing.

On the other hand, not all standard formats should Thus, new computing hardware opens the door to be assumed to be the best choices. Standards must new and improved software, leading to software and become widely adopted by both user and developer file-format obsolescence. The new software will not communities to acquire reduced vulnerability to run on old hardware, further exacerbating hardware obsolescence. That does not always happen. obsolescence. At the same time, the new hardware introduces other new technology, such as peripheral Choosing File Formats for Reduced connections (e.g. Firewire and USB have replaced RS- Vulnerability to Obsolescence 232 serial and Centronics parallel ports) and storage devices (e.g. USB keys and CD drives have replaced The following factors should be considered in assessing floppy disks). These changes force older peripherals a file format’s long-term stability: into retirement along with their compatible computers.

-- wide adoption

-- history of backward compatibility

47 Principles and Good Practice for Preserving Data

Types of Digital Storage Media Declining cost per unit of storage -- hard drives (most rapid) The nature of the physical media on which digital data -- tape cartridges are stored presents a major challenge to the preservation -- optical media (least rapid) of digital content. The great variety of media types, their often rapid obsolescence due to technological Other trends are less uniform for all media changes and their vulnerability to physical degradation -- reliability (generally improving) all contribute to the problem. -- fragility (variable) -- stability (generally improving) There are three commonly-used categories of digital- -- time to obsolescence (variable) storage media: disk, tape and solid state. Within each category are many levels of sub-categories, representing Physical Threats to Storage Media and both integrated storage (drive and media as a single Hardware unit) as well as removable media. Digital storage media and hardware are subject to numerous internal and external forces that can damage Magnetic (fixed hard drive) or destroy their readability: Magnetic (removable) -- material instability Hard disk packs -- unsuitable storage conditions Floppy Zip, Jaz, etc (temperature, humidity, light, dust) Magneto-optical (write-once, read/write) -- over-use (mainly for physical contact Optical (read-only, write-once, recordable, read/write) media) -- natural disaster (fire, flood, earthquake) Open reel -- infrastructure failure (plumbing, electrical, Cassette climate control) Cartridge -- inadequate hardware maintenance -- hardware malfunction CompactFlash, Memory Stick, Smart Media (digital camera memory) -- human error (including improper USB memory key or stick, pen drives, keychain drives (portable handling) storage up to 2 GB) -- sabotage (theft, vandalism) Flash drives (IDE and SCSI) using standard hard disk form factors; often for industrial or military use in adverse temperatures or extreme or dust conditions (capacity up to 61 GB) Storage Issues

Improper storage might be the most common reason for Trends Contributing to Hardware premature media failure. Moderation of temperature Obsolescence and humidity are well known to extend the usable life of most storage media, but many other factors can help, Several technological trends encourage storage media too. obsolescence. These trends include: Suggested Preservation Action for Managing Decrease in physical size Storage Media -- hard drives (24” —> 1” over a 40 year -- Maintain consistent temperature ~ 20 deg period) C (68 deg F) (see the IPI Media Storage -- floppy disks (8” —> 5.25” --> 3.5” over a Quick Reference for specific guidelines). 10 year period) -- Maintain relative humidity around 40%. -- optical media (14” —> 2” over a 20 year -- Avoid large and rapid fluctuations in period) temperature/humidity. -- Control dust (maintain a slight positive Increase in storage capacity pressure environment). -- hard drives (5 MB —> 400 GB, 2 TB in -- Avoid exposure to magnetic fields (for 2009) magnetic media). -- tape cartridges (1 TB cartridges coming) -- Avoid exposure to fumes. -- 12 cm optical media (650 MB —> 50 GB in 2009)

IHSN Working Paper No. 003 December 2009

-- Establish a “No food, drink or smoking” -- CDs should be labelled only on the top policy in media-storage areas. surface and with approved markers. -- Store media in closed metal cabinets, -- Avoid flexing CDs and DVDs. electrically grounded. -- Do not leave media in drive after use. -- Shelve media vertically (not stacked). -- Re-tension magnetic tapes after use and -- Store media in their original cases. every 1-3 years even if not used. -- Minimise exposure to sunlight and UV -- Limit media access to properly-trained from light fittings. staff. -- Allow media to acclimatise to new -- Use write-once or recordable (rather than temperatures and humidity before using. rewritable) media. -- Return to controlled storage immediately -- Set write-protect tabs, if available. after use. Life Expectancy Preparation Matters A lot of ink has been spilt over the issue of media Physical threats from natural disasters, infrastructure longevity. Media consumers and producers have placed failure and malicious destruction cannot usually be a great deal of emphasis on seeking and promoting predicted, but it is possible to lessen their occurrence long-lasting media. Ultimately, however, for a great and minimise the damage they cause by proper many reasons, longevity is overrated as a desirable preparation. Awareness the moment a hazardous media characteristic. condition arises allows the fastest possible response. Sensors and alarm systems to detect and report Media life expectancy claims are statistical averages the presence of fire, heat, smoke, water leaks and based on accelerated aging tests and can only provide unauthorised entry are available. Fire-suppression a rough estimate of how long any particular piece of systems, floor drains and use of heat-resistant and/or media will last under certain storage and handling waterproof storage can all help minimise damage to conditions.. sensitive media and equipment. Media storage areas should be locked and accessible only to properly-trained Longevity provides no protection against many personnel. All media, no matter how reliable, need to be media threats, including theft, natural disasters, backed up. Creation of multiple back-ups and the use of infrastructure failures and accidental handling damage. off-site storage for one set of copies provides the best protection against catastrophic loss. Media technology changes so rapidly that long- lasting media are likely to be threatened by obsolescence Handling Issues before their useful life is over.

Another major threat to storage media comes from Suggested Preservation Action improper handling. Though many digital media give -- A better strategy is to take steps to the impression of sturdiness and durability, they can maximise the intrinsic longevity of be damaged by too casual an approach to use. Observe standard-longevity media: these recommendations: -- Adhere to good storage and handling practices. Suggested Preservation Action -- Buy quality media. -- Don’t open shutters designed to protect -- Take note of media manufacturer and media in cartridges. batch numbers so performance and quality -- Handle media with lint-free gloves to trends can be tracked. minimise dust. -- Do not over-buy. -- Clean and dry hands before handling -- Remember that some unrecorded media media. have a shorter shelf-life than recorded -- Do not touch exposed media surfaces (e.g. media (optical and magneto-optical in handle CDs at edges). particular). -- Keep media in their cases except when in -- Buy media designed for the speed and use. capacity of the drives in which it will be -- Place labels only in approved areas; write used. on label before applying.

49 Principles and Good Practice for Preserving Data

All media need periodic testing to confirm data -- The top surface (label area) of CDs requires integrity. At a minimum this should include procedures extra care, as it is more vulnerable than the to: bottom layer -- Confirm fidelity of all media immediately -- CD/DVD cleaning should be done axially after recording. (i.e. outer edge to inner edge), not radially -- Once recorded, read samples (by batch or along the tracks code and manufacturer and/or storage -- DVDs are more vulnerable to flexing location) of the entire media and samples damage due to closer track spacing. Use of files from several media periodically.. special DVD carriers to minimise flexing -- Detect problems with specific batches, upon removal manufacturers or storage conditions and test them more extensively. Disaster Recovery -- Test blank media (can be expensive and time-consuming). The methods and procedures already mentioned are -- Monitor use of error correction and replace designed to minimise casual loss of data and maximise media before errors become impossible to media longevity. However, even if you had perfect correct.. storage conditions and impeccable handling protocols, some media still fail. Therefore, valuable data must be Specific Media Issues stored redundantly, that is, backed up, on more than one piece of media. In addition, back-ups and disaster- Hard disk drives recovery plans are needed to avoid catastrophic media -- Highly commoditised. Expect to pay more loss from causes such as: for quality drives with above-average -- sabotage (theft, vandalism, malicious reliability. modification/erasure, viruses, terrorist -- Do not buy excess capacity. Prices and attack, etc) technology change extremely rapidly. -- natural disaster (fire, flood, earthquake, -- Do not expect more than five years’ use hurricane, tornado, infrastructure failure) from any hard-disk drive. -- High temperature can dramatically reduce A disaster-recovery plan that deals specifically life expectancy. Control the environment with information technology infrastructure is needed. and make sure fans are working and not Developing such a plan is not a one-off process; it has clogged with dust. to be tested and modified as changing circumstances dictate. Revisit the disaster recovery plan for occurrences Magnetic tape such as new staff, new or reorganised physical plant or -- Still the least expensive back-up medium new equipment. Once a disaster-recovery plan is in per unit. place, take steps to prevent catastrophes and minimise -- New technology offers extremely high- damage from them. density storage. -- High-density cartridge technology (SDLT, NIST (National Institute of Standards and LTO, AIT) considered the most reliable. Technology) publishes an excellent guide to developing -- New generations of tape formats appear and implementing plans to cope with disaster entitled regularly. Backward compatibility is “Contingency Planning Guide for Information usually only offered for 1-2 generations. Technology Systems”.

Optical (CD/DVD variants) Back-ups -- These media have several possible failure modes The maintenance of redundant copies of valued digital  Dye layer (for recordable media) content is an essential component of any digital-  Reflective layer preservation programme, and a key element in the  Substrate separation prevention of catastrophic loss. A great variety of back- -- Unrecorded media have a 5-10 year shelf- up solutions is available. Which to use depends on: life -- quantity of data -- rate of change

IHSN Working Paper No. 003 December 2009

-- degree of automation desired warehousing. Generic storage facilities are unlikely to -- available budget know how to store and handle digital media properly. Environmental controls and handling protocols should In addition to backing up data files, application be at least as rigorous as those at the main facility. software and operating systems might also need back- up. In some cases, it might be necessary to purchase An important consideration in selecting a back-up additional licences or obtain special permission from strategy is the possibility of disastrous loss of the entire the software vendor in order to back up applications. primary facility -- all equipment and data. In such a situation, it will be necessary to replace the entire IT In addition to testing back-up media periodically infrastructure and to restore the secondary data store to ensure the data are still readable and have not been to new equipment. If new equipment that can handle altered, restore procedures should also be tested to the back-up media, the restoration software and the ensure the hardware, software and any outside vendors applications software and operating environment involved in maintaining back-up are all functioning as needed to access the data cannot be purchased, it might expected. be time to think again about the back-up plan.

A prudent back-up strategy places at least one copy Emergency Rescue of all critical data at a sufficient distance from the main data store so both are not likely to succumb to the same Poor planning or circumstances beyond one’s control disaster. This is called off-site storage. Institutions can lead to emergency rescue of critical data. Anything should check with regulatory agencies for their from budget woes to bad luck can provide the trigger. records-retention requirements. Medical and financial Companies exist that specialise in salvaging data records might have more rigorous requirements for the from badly-damaged media, when no backup exists, distance of the off-site storage facility from the main and reading data from obsolete storage technology. one. Cooperating on a reciprocal storage arrangement These services can often be quite expensive but with a another institution could be an inexpensive also be a lifesaver. A web search for “data recovery” way of managing off-site storage. If outsourcing, make should produce a plethora of links to such specialised sure you are getting true data management, not just companies.

51 Principles and Good Practice for Preserving Data

IHSN

International Household Survey Network

About the IHSN Principles and Good Practice for In February 2004, representatives from developing countries and development agencies participated in the Second Roundtable on Development Results held in Marrakech, Morocco. They reflected on how donors can better coordinate support to strengthen the statistical systems and monitoring and Preserving Data evaluation capacity that countries need to manage their development process. One of the outcomes of the Roundtable was the adoption of a global plan for statistics, the Marrakech Action Plan for International Consortium for Political and Statistics (MAPS). Social Research (ICPSR) Among the MAPS key recommendations was the creation of an International Household Survey Network. In doing so, the international community acknowledged the critical role played by sample surveys in supporting the planning, implementation and monitoring of development policies and programs. Furthermore, it provided national and international agencies with a platform to better coordinate and manage socioeconomic data collection and analysis, and to mobilize support for more efficient and effective approaches to conducting surveys in developing countries.

IHSN Working Paper No 003 December 2009 www.ihsn.org E-mail: [email protected] www.ihsn.org