DOCSLIB.ORG

Research Data Management Best Practices

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Research Data Management Best Practices Research Data Management Best Practices Introduction ............................................................................................................................................................................ 2 Planning & Data Management Plans ...................................................................................................................................... 3 Naming and Organizing Your Files .......................................................................................................................................... 6 Choosing File Formats ............................................................................................................................................................. 9 Working with Tabular Data ................................................................................................................................................... 10 Describing Your Data: Data Dictionaries ............................................................................................................................... 12 Describing Your Project: Citation Metadata ......................................................................................................................... 15 Preparing for Storage and Preservation ............................................................................................................................... 17 Choosing a Repository .......................................................................................................................................................... 19 Glossary ................................................................................................................................................................................. 22 Research Data Management Best Practices 2018-02-28 INTRODUCTION The following best practices are intended for use by Smithsonian researchers and affiliated staff who plan for, create, and/or work with digital research data. Additional information about available tools, policies, and resources for managing research data can be found on https://library.si.edu/research/data-management . There are many phases in the research data lifecycle and they do not always occur in the tidy order pictured in the diagram (left). These best practices are designed to improve overall management of data at each point in the lifecycle, resulting in published data that are not only easy to care for long after the project is complete, but that are also findable, accessible, interoperable, and reusable. The Smithsonian has other resources that can contribute to effective data management including software and high performance computing provided by OCIO, and training and planning consultation services from the Libraries. Contact [email protected] for more information. SI also has two locally managed repositories that accept research data for publication and/or archiving: SIdora – is best for larger, or more complicated datasets, including actively updated datasets. To deposit data in SIdora contact Beth Stern or email [email protected] Smithsonian Research Online (SRO) – is best for smaller (<50GB), fixed (inactive) datasets that accompany or support publications deposited in SRO. To deposit data and publications in SRO, you can self-deposit using the forms found here http://staff.research.si.edu/input_forms.cfm or contact [email protected] Actively managing your data throughout the research process enables reproducibility, reusability, and discovery, and can help maximize the impact of your research into the future. Research Data Management Best Practices 2018-02-28 PLANNING & DATA MANAGEMENT PLANS Many granting agencies, such as NSF and the Alfred P. Sloan Foundation, require a formal data management plan (DMP) as part of a grant proposal. Even if a granting agency does not require a DMP, SI strongly recommends that PIs create a planning document before starting any project that will create digital research data. DMPs are valuable tools for addressing issues that affect not only collection and use, but also the long-term viability of your data. A written data management plan can: provide continuity on projects if staff join or leave allow for future validation or reproduction of results enable reuse of your data in potentially novel ways SI Libraries staff are available for consultation on creating DMPs and are happy to review draft DMPs before submission with a proposal. Contact [email protected] for more information. Proposals The Smithsonian Office of Sponsored Projects (OSP) provides administrative and financial services for externally funded grants and contracts, and is available to assist PIs with technical and procedural questions related to managing grants and awards. OSP also provides training in proposal development, writing and editing, and compliance oversight for areas such as Institutional Animal Care & Use, Export Control, Human Subjects in Research, and Responsible Conduct of Research. Their list of online and in person learning opportunities are available on their PRISM site. Planning checklist Any plan should at a minimum answer the following questions in bold for each stage in the data management lifecycle. More specific guidance for questions in the data collection, publishing, and archiving stages is available at https://library.si.edu/research/data-management PROPOSAL/PLANNING STAGE ⃞ What type of data is being collected/generated? ⃞ Who is involved in data collection? ⃞ Who "owns" the rights to the data? ⃞ Are there restrictions on sharing and reuse? ⃞ Are there applicable institutional policies on how the data is handled, shared, or archived? ⃞ Who will be using the data? ⃞ If a collaborative project, are there MOUs that define roles and responsibilities? ⃞ How do the outcomes need to be reported, e.g., to a sponsor or publisher? DATA COLLECTION STAGE ⃞ How will data be acquired/collected? ⃞ What metadata standards and schema will be used? ⃞ What are the file and data field naming conventions? ⃞ What are the temporary storage requirements (size,cost,media)? ⃞ How, where, and how frequently will data be backed up? ⃞ Are there existing standards for data structure and vocabularies, or will they be developed? Research Data Management Best Practices 2018-02-28 ⃞ Are there existing workflows for collecting, processing, describing, and storing the data, or will they need to be developed? ⃞ Is there a data model for the project? ⃞ Will your data be versioned, and if so, how will versioning be handled? ⃞ What is your quality assurance/quality control process? PUBLISHING STAGE ⃞ What repository or platform will be used to share the data? ⃞ Who will be responsible for deposit and archiving after the project ends? ⃞ If the data is to be shared publicly, what license should be applied? Are there any use restrictions? ⃞ If the data is embargoed, what is the embargo period, and who will manage it? ⃞ If the data is not public, how will access be restricted? ⃞ What costs are associated with publishing? ⃞ What unique identifier will be assigned to the data (DOI, etc.)? ARCHIVING STAGE ⃞ Who is responsible for preserving the datasets in the future? ⃞ What data should be retained? ⃞ Where will the data be archived? ⃞ How much storage will be needed? ⃞ How long should the data be maintained, and why? ⃞ What are the risks for future access to the data, i.e., proprietary file formats, specialty software needed to interpret, password-protected systems? ⃞ How should the data be maintained in the future? ⃞ Is there a cost associated with archiving the data? ⃞ How will the data be found? Funder-specific DMP Requirements Some funding agencies require that plans submitted with grant proposals include specific elements or specific formatting. Below is a list of links to those requirements, alphabetical by funder, for selected granting organizations. * = sample plans available on their site Alfred P. Sloan Foundation BCO-DMO NSF OCE: Biological and Chemical Oceanography Department of Energy – DOE: Generic * Gordon and Betty Moore Foundation (pdf) Institute for Museum and Library Services IMLS : guidelines for datasets (Word doc) * National Aeronautics and Space Administration NASA * National Endowment for the Humanities NEH-ODH (pdf) National Oceanic and Atmospheric Administration NOAA * National Science Foundation NSF-Generic DMP o NSF-Atmospheric and Geo o NSF-Astronomy (pdf) o NSF-Biology (pdf) o NSF-Earth Sciences o NSF-Education and Human Resources (pdf) Research Data Management Best Practices 2018-02-28 United States Geological Survey USGS Tools and templates SMITHSONIAN SPECIFIC TEMPLATES The Data Management Team has developed boilerplate (temporarily located on an internal Confluence site) that can be used when applying for an NSF grant if you plan to deposit data either in SRO or SIDora. The boilerplate address the specifics of data archiving, dissemination, policies, and roles and responsibilities within the SI data management ecosystem. DMPTOOL One of the major tools for creating data management plans is the DMPTool, hosted by the University of California Curation Center (UC3 The Smithsonian was one of the original partner institutions involved in creating the DMPTool. The DMPTool website includes templates and requirements for a large number of granting bodies, including NSF, DOE and NIH. Any researcher at the Smithsonian can create an account and login to the DMPTool by selecting "Smithsonian Institution" from the list of institutions
Load more

Recommended publications
  • An Overview of the 50 Most Common Web Scraping Tools
    AN OVERVIEW OF THE 50 MOST COMMON WEB SCRAPING TOOLS WEB SCRAPING IS THE PROCESS OF USING BOTS TO EXTRACT CONTENT AND DATA FROM A WEBSITE. UNLIKE SCREEN SCRAPING, WHICH ONLY COPIES PIXELS DISPLAYED ON SCREEN, WEB SCRAPING EXTRACTS UNDERLYING CODE — AND WITH IT, STORED DATA — AND OUTPUTS THAT INFORMATION INTO A DESIGNATED FILE FORMAT. While legitimate uses cases exist for data harvesting, illegal purposes exist as well, including undercutting prices and theft of copyrighted content. Understanding web scraping bots starts with understanding the diverse and assorted array of web scraping tools and existing platforms. Following is a high-level overview of the 50 most common web scraping tools and platforms currently available. PAGE 1 50 OF THE MOST COMMON WEB SCRAPING TOOLS NAME DESCRIPTION 1 Apache Nutch Apache Nutch is an extensible and scalable open-source web crawler software project. A-Parser is a multithreaded parser of search engines, site assessment services, keywords 2 A-Parser and content. 3 Apify Apify is a Node.js library similar to Scrapy and can be used for scraping libraries in JavaScript. Artoo.js provides script that can be run from your browser’s bookmark bar to scrape a website 4 Artoo.js and return the data in JSON format. Blockspring lets users build visualizations from the most innovative blocks developed 5 Blockspring by engineers within your organization. BotScraper is a tool for advanced web scraping and data extraction services that helps 6 BotScraper organizations from small and medium-sized businesses. Cheerio is a library that parses HTML and XML documents and allows use of jQuery syntax while 7 Cheerio working with the downloaded data.
    [Show full text]
  • Data and Computer Communications (Eighth Edition)
    DATA AND COMPUTER COMMUNICATIONS Eighth Edition William Stallings Upper Saddle River, New Jersey 07458 Library of Congress Cataloging-in-Publication Data on File Vice President and Editorial Director, ECS: Art Editor: Gregory Dulles Marcia J. Horton Director, Image Resource Center: Melinda Reo Executive Editor: Tracy Dunkelberger Manager, Rights and Permissions: Zina Arabia Assistant Editor: Carole Snyder Manager,Visual Research: Beth Brenzel Editorial Assistant: Christianna Lee Manager, Cover Visual Research and Permissions: Executive Managing Editor: Vince O’Brien Karen Sanatar Managing Editor: Camille Trentacoste Manufacturing Manager, ESM: Alexis Heydt-Long Production Editor: Rose Kernan Manufacturing Buyer: Lisa McDowell Director of Creative Services: Paul Belfanti Executive Marketing Manager: Robin O’Brien Creative Director: Juan Lopez Marketing Assistant: Mack Patterson Cover Designer: Bruce Kenselaar Managing Editor,AV Management and Production: Patricia Burns ©2007 Pearson Education, Inc. Pearson Prentice Hall Pearson Education, Inc. Upper Saddle River, NJ 07458 All rights reserved. No part of this book may be reproduced in any form or by any means, without permission in writing from the publisher. Pearson Prentice Hall™ is a trademark of Pearson Education, Inc. All other tradmarks or product names are the property of their respective owners. The author and publisher of this book have used their best efforts in preparing this book.These efforts include the development, research, and testing of the theories and programs to determine their effectiveness.The author and publisher make no warranty of any kind, expressed or implied, with regard to these programs or the documentation contained in this book.The author and publisher shall not be liable in any event for incidental or consequential damages in connection with, or arising out of, the furnishing, performance, or use of these programs.
    [Show full text]
  • 2020 Global Data Management Research the Data-Driven Organization, a Transformation in Progress Methodology
    BenchmarkBenchmark Report report 2020 Global data management research The data-driven organization, a transformation in progress Methodology Experian conducted a survey to look at global trends in data management. This study looks at how data practitioners and data-driven business leaders are leveraging their data to solve key business challenges and how data management practices are changing over time. Produced by Insight Avenue for Experian in October 2019, the study surveyed more than 1,100 people across six countries around the globe: the United States, the United Kingdom, Germany, France, Brazil, and Australia. Organizations that were surveyed came from a variety of industries, including IT, telecommunications, manufacturing, retail, business services, financial services, healthcare, public sector, education, utilities, and more. A variety of roles from all areas of the organization were surveyed, which included titles such as chief data officer, chief marketing officer, data analyst, financial analyst, data engineer, data steward, and risk manager. Respondents were chosen based on their visibility into their organization’s customer or prospect data management practices. Page 2 | The data-driven organization, a transformation in progress Benchmark report 2020 Global data management research The data-informed organization, a transformation in progress Executive summary ..........................................................................................................................................................5 Section 1:
    [Show full text]
  • Value of Data Management
    USGS Data Management Training Modules: Value of Data Management “Data is a precious thing and will last longer than the systems themselves.” -- Tim Berners-Lee …. But only if that data is properly managed! U.S. Department of the Interior Accessibility | FOIA | Privacy | Policies and Notices U.S. Geological Survey Version 1, October 2013 Narration: Introduction Slide 1 USGS Data Management Training Modules – the Value of Data Management Welcome to the USGS Data Management Training Modules, a three part training series that will guide you in understanding and practicing good data management. Today we will discuss the Value of Data Management. As Tim Berners-Lee said, “Data is a precious thing and will last longer than the systems themselves.” ……We will illustrate here that the validity of that statement depends on proper management of that data! Module Objectives · Describe the various roles and responsibilities of data management. · Explain how data management relates to everyday work. · Emphasize the value of data management. From Flickr by cybrarian77 Narration:Slide 2 Module Objectives In this module, you will learn how to: 1. Describe the various roles and responsibilities of data management. 2. Explain how data management relates to everyday work and the greater good. 3. Motivate (with examples) why data management is valuable. These basic lessons will provide the foundation for understanding why good data management is worth pursuing. 2 Terms and Definitions · Data Management (DM) – the development, execution and supervision of plans,
    [Show full text]
  • Research Data Management: Sharing and Storage
    Research Data Management: Strategies for Data Sharing and Storage 1 Research Data Management Services @ MIT Libraries • Workshops • Web guide: http://libraries.mit.edu/data-management • Individual assistance/consultations – includes assistance with creating data management plans 2 Why Share and Archive Your Data? ● Funder requirements ● Publication requirements ● Research credit ● Reproducibility, transparency, and credibility ● Increasing collaborations, enabling future discoveries 3 Research Data: Common Types by Discipline General Social Sciences Hard Sciences • images • survey responses • measurements generated by sensors/laboratory • video • focus group and individual instruments interviews • mapping/GIS data • computer modeling • economic indicators • numerical measurements • simulations • demographics • observations and/or field • opinion polling studies • specimen 4 5 Research Data: Stages Raw Data raw txt file produced by an instrument Processed Data data with Z-scores calculated Analyzed Data rendered computational analysis Finalized/Published Data polished figures appear in Cell 6 Setting Up for Reuse: ● Formats ● Versioning ● Metadata 7 Formats: Considerations for Long-term Access to Data In the best case, your data formats are both: • Non-proprietary (also known as open), and • Unencrypted and uncompressed 8 Formats: Considerations for Long-term Access to Data In the best case, your data files are both: • Non-proprietary (also known as open), and • Unencrypted and uncompressed 9 Formats: Preferred Examples Proprietary Format
    [Show full text]
  • Data Management, Analysis Tools, and Analysis Mechanics
    Chapter 2 Data Management, Analysis Tools, and Analysis Mechanics This chapter explores different tools and techniques for handling data for research purposes. This chapter assumes that a research problem statement has been formulated, research hypotheses have been stated, data collection planning has been conducted, and data have been collected from various sources (see Volume I for information and details on these phases of research). This chapter discusses how to combine and manage data streams, and how to use data management tools to produce analytical results that are error free and reproducible, once useful data have been obtained to accomplish the overall research goals and objectives. Purpose of Data Management Proper data handling and management is crucial to the success and reproducibility of a statistical analysis. Selection of the appropriate tools and efficient use of these tools can save the researcher numerous hours, and allow other researchers to leverage the products of their work. In addition, as the size of databases in transportation continue to grow, it is becoming increasingly important to invest resources into the management of these data. There are a number of ancillary steps that need to be performed both before and after statistical analysis of data. For example, a database composed of different data streams needs to be matched and integrated into a single database for analysis. In addition, in some cases data must be transformed into the preferred electronic format for a variety of statistical packages. Sometimes, data obtained from “the field” must be cleaned and debugged for input and measurement errors, and reformatted. The following sections discuss considerations for developing an overall data collection, handling, and management plan, and tools necessary for successful implementation of that plan.
    [Show full text]
  • Computer Files & Data Storage
    STORAGE & FILE CONCEPTS, UTILITIES (Pages 6, 150-158 - Discovering Computers & Microsoft Office 2010) I. Computer files – data, information or instructions residing on secondary storage are stored in the form of a file. A. Software files are also called program files. Program files (instructions) are created by a computer programmer and generally cannot be modified by a user. It’s important that we not move or delete program files because your computer requires them to perform operations. Program files are also referred to as “executables”. 1. You can identify a program file by its extension:“.EXE”, “.COM”, “.BAT”, “.DLL”, “.SYS”, or “.INI” (there are others) or a distinct program icon. B. Data files - when you select a “save” option while using an application program, you are in essence creating a data file. Users create data files. 1. File naming conventions refer to the guidelines followed while assigning file names and will vary with the operating system and application in use (see figure 4-1). File names in Windows 7 may be up to 255 characters, you're not allowed to use reserved characters or certain reserved words. File extensions are used to identify the application that was used to create the file and format data in a manner recognized by the source application used to create it. FALL 2012 1 II. Selecting secondary storage media A. There are three type of technologies for storage devices: magnetic, optical, & solid state, there are advantages & disadvantages between them. When selecting a secondary storage device, certain factors should be considered: 1. Capacity - the capacity of computer storage is expressed in bytes.
    [Show full text]
  • BEST PRACTICES for DATA MANAGEMENT TECHNICAL GUIDE Nov 2018 EPA ID # 542-F-18-003
    BEST PRACTICES FOR DATA MANAGEMENT TECHNICAL GUIDE EPA ID # 542-F-18-003 Section 1 - Introduction This technical guide provides best practices Why is EPA Issuing this technical guide? for efficiently managing the large amount of data generated throughout the data life The U.S. Environmental Protection Agency (EPA) cycle. Thorough, up-front remedial developed this guide to support achievement of the investigation and feasibility study (RI/FS) July 2017 Superfund Task Force goals. Two additional planning and scoping combined with companion technical guides should be used in decision support tools and visualization can conjunction with this data management technical help reduce RI/FS cost and provide a more guide: complete conceptual site model (CSM) • Smart Scoping for Environmental earlier in the process. In addition, data Investigations management plays an important role in • Strategic Sampling Approaches adaptive management application during the RI/FS and remedial design and action. This section defines the data life cycle approach and describes the benefits a comprehensive data life cycle management approach can accrue. What is the “Data Life Cycle” Management Approach? The Superfund program collects, reviews and works with large volumes of sampling, monitoring and environmental data that are used for decisions at different scales. For example, site-specific Superfund data developed by EPA, potentially responsible parties, states, tribes, federal agencies and others can include: • Geologic and hydrogeologic data; • Geospatial data (Geographic Information System [GIS] and location data); • Chemical characteristics; • Physical characteristics; and • Monitoring and remediation system performance data. In addition, EPA recognizes that regulatory information and other non-technical data are used to develop a CSM and support Superfund decisions.
    [Show full text]
  • File Format Guidelines for Management and Long-Term Retention of Electronic Records
    FILE FORMAT GUIDELINES FOR MANAGEMENT AND LONG-TERM RETENTION OF ELECTRONIC RECORDS 9/10/2012 State Archives of North Carolina File Format Guidelines for Management and Long-Term Retention of Electronic records Table of Contents 1. GUIDELINES AND RECOMMENDATIONS .................................................................................. 3 2. DESCRIPTION OF FORMATS RECOMMENDED FOR LONG-TERM RETENTION ......................... 7 2.1 Word Processing Documents ...................................................................................................................... 7 2.1.1 PDF/A-1a (.pdf) (ISO 19005-1 compliant PDF/A) ........................................................................ 7 2.1.2 OpenDocument Text (.odt) ................................................................................................................... 3 2.1.3 Special Note on Google Docs™ .......................................................................................................... 4 2.2 Plain Text Documents ................................................................................................................................... 5 2.2.1 Plain Text (.txt) US-ASCII or UTF-8 encoding ................................................................................... 6 2.2.2 Comma-separated file (.csv) US-ASCII or UTF-8 encoding ........................................................... 7 2.2.3 Tab-delimited file (.txt) US-ASCII or UTF-8 encoding .................................................................... 8 2.3
    [Show full text]
  • Data Management Best Practices
    Data ManagementBest Practices Data management best practices Properly managing data is of the upmost importance to ensure data quality and integrity, while also improving data access and discovery. To be successful, proper data management must be incorporated in all stages of data processing including collection, analysis, and long-term storage. The Washington University in St. Louis Libraries, has outlined step by step procedures for researchers to undertake to improve their data management techniques. Recommendations: 1. Conduct a data asset inventory. a. Assess and inventory data collected b. Create a standard data intake form to document details on data collection entities and methods for current and future data acquisitions 2. Adopt best practices for managing data a. Create an intuitive folder structure and hierarchy that reflects the project goals b. Adopt and apply consistent file naming conventions for all files and folders c. Actively manage digital objects d. Establish a robust backup strategy (preferably 3 copies – 2 locally – 1 offsite) 1. Inventory data An incremental, stepped approach to undertaking a data inventory is described below that can be applied flexibly according to context and needs. 1. Plan and define the purpose and scope of the inventory (determine which projects, whether or not to include off-site collaborators, what variables to include in the inventory, etc.). 2. Identify: a. what data assets exist, b. the values of the variables, c. where management could be improved, d. and any classifications of the data. Free, open source software can help automate the file inventory and identification process. FITS: http://projects.iq.harvard.edu/fits BitCurator: http://wiki.bitcurator.net/?title=Software Record the outputs of your inventory on a spreadsheet.
    [Show full text]
  • Overview of Data Format, Data Model and Procedural Metadata to Support Iot
    Overview of data format, data model and procedural metadata to support IoT Nakyoung Kim Scattered IoT ecosystem • Service and application–dedicated data formats & models • Data formats and models have been developed to suit the specific requirements of each industry, service, and application. • The current scattered-data ecosystem has been established, and it requires high costs to process and manage data for service and application convergence. • Needs of data integration and aggregation • The boundaries between industries are gradually crumbling down due to domain convergence. • For the future convergence markets of IoT and smart city, data integration and aggregation are important. Application-dedicated data management 2 Interoperability through the Web • Bridging the data • Data formats and models have been settled in the current forms over a period to resolve issues at each moment and fit case-by-case requirements. • Therefore, it is hardly possible to reformulate or replace the existing data formats and models. • Bridging the data formats and models is feasible. • Connecting via Web • A semantic information space for heterogeneous data, platforms, and application domains, Web can provide technologies that support the interoperability of IoT. • Enhanced interoperability in IoT ecosystems can be fulfilled with the concepts of Web of Things. Web Data-driven data management 3 Annotation and Microdata format • Structured data for annotation • To exploit what the Web offers, IoT data first needs to be structured and annotated just like how the other data is handled on the Web. • Microdata format • Microdata formats refer structured data markups describing and embedding the meanings of resources on the Web along with their properties and relationships.
    [Show full text]
  • Modular Data Storage with Anvil
    Modular Data Storage with Anvil Mike Mammarella Shant Hovsepian Eddie Kohler UCLA UCLA UCLA/Meraki [email protected] [email protected] [email protected] http://www.read.cs.ucla.edu/anvil/ ABSTRACT age strategies and behaviors. We intend Anvil configura- Databases have achieved orders-of-magnitude performance tions to serve as single-machine back-end storage layers for improvements by changing the layout of stored data – for databases and other structured data management systems. instance, by arranging data in columns or compressing it be- The basic Anvil abstraction is the dTable, an abstract key- fore storage. These improvements have been implemented value store. Some dTables communicate directly with sta- in monolithic new engines, however, making it difficult to ble storage, while others layer above storage dTables, trans- experiment with feature combinations or extensions. We forming their contents. dTables can represent row stores present Anvil, a modular and extensible toolkit for build- and column stores, but their fine-grained modularity of- ing database back ends. Anvil’s storage modules, called dTa- fers database designers more possibilities. For example, a bles, have much finer granularity than prior work. For ex- typical Anvil configuration splits a single “table” into sev- ample, some dTables specialize in writing data, while oth- eral distinct dTables, including a log to absorb writes and ers provide optimized read-only formats. This specialization read-optimized structures to satisfy uncached queries. This makes both kinds of dTable simple to write and understand. split introduces opportunities for clean extensibility – for Unifying dTables implement more comprehensive function- example, we present a Bloom filter dTable that can slot ality by layering over other dTables – for instance, building a above read-optimized stores and improve the performance read/write store from read-only tables and a writable journal, of nonexistent key lookup.
    [Show full text]