DOCSLIB.ORG

Downloaded from Google

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Downloaded from Google HathiTrust Digital Library Update On January Activities In This Newsletter Top News February 12, 2010 Top News • New Cost Model New Cost Model – The HathiTrust ex- workflow documentation. Guided by • Disaster Recovery Plans ecutive committee approved a new cost industry standards such as TRAC and • Digital Library Profile model for partnership in December that best practices in the digital preservation Working Groups will be adopted by all partners begin- community, the committee will ensure • Quality ning in 2013. In the new model, partners a high level of preparedness for known • Discovery Interface will share in the cost of public domain and unknown risks to the long-term in- • Development Environment and open access volumes preserved in tegrity and use of materials in the repos- • Storage HathiTrust, and in the cost of in copy- itory. A preliminary meeting of key staff Ingest right volumes that they hold, or have will occur in February, and membership • Internet Archive Ingest held, in their physical collections. The in the Disaster Recovery Planning Com- • Non-Google Ingest model will distribute the costs of curat- mittee will be finalized soon thereafter. Development Updates ing and managing the digital collections Digital Library Profile – As part • Shibboleth in a way that more accurately reflects of its participation in an NSF EAGER • Data API the benefits each partner receives from grant awarded in September 2009, Ha- • Large-scale Search deposited volumes. It will also allow in- thiTrust completed a technological pro- • PageTurner stitutions to join HathiTrust who do not file of its repository based on two frame- necessarily have content to deposit, but New Growth works developed by Johns Hopkins who wish to support and benefit from Number of volumes added: University. The profile can be found at the long-term curation and access ser- http://www.hathitrust.org/technology. January Total vices that HathiTrust provides. Such Indiana institutions are eligible for partnership Working Groups 38,344 151,816 Univ. effective immediately, and do not need Quality – In July 2009, the Strate- Penn to wait for the 2013 general adoption. 0 5016 gic Advisory Board (SAB) assembled a State Details of the new cost model are avail- working group to investigate issues sur- Univ. of able at http://www.hathitrust.org/doc- 972 1,156,339 rounding the quality of partner institu- California uments/hathitrust-cost-rationale-2013. Univ. of tion volumes downloaded from Google. 71,094 3,730,968 pdf. Please contact hathitrust-info@ Michigan The working group was asked to re- umich.edu for additional information Univ. of search and provide recommendations 691 268.044 and inquiries about partnership. Wisconsin on a quality threshold HathiTrust uses Total 104,342 5,312,183 Disaster Recovery Planning – Fol- to limit ingest of poor quality volumes. lowing an evaluation of disaster pre- The working group presented its recom- 5,384 public domain volumes were added in December, bring- paredness performed last summer by an mendations to the SAB in January and ing the total number to 764,331 IMLS-funded intern, and the hiring of a the SAB decided to continue the work- (about 14% of total content). preservation librarian in November, the ing group with a revised and expanded University of Michigan is taking steps to charge. The new charge is to a) develop a formalize and expand HathiTrust’s poli- set of quality principles for HathiTrust, cies and practices relating to disaster b) monitor quality control as related to recovery. The UM preservation librar- user experience, c) track developments ian is leading a process to form a Disas- in a separate quality working group es- ter Recovery Planning Committee and, tablished by Google and Google library with support of a winter intern from the partners following the Google partner UM School of Information, has begun summit in October, and d) evaluate Ha- to gather key inventory, personnel, and thiTrust practices with regard to thresh- HathiTrust Digital Library Update On January Activities February Forecast • Complete and deploy Shib- olding or limiting ingested content. with making recommendations on a boleth authentication sup- Membership in the new group, called third instance of storage for HathiTrust port the HathiTrust Quality Ingest and Error presented its final report to the Execu- • Complete quality assurance Rate Working Group, is currently being tive Committee in January. The group processes for pilot of UC’s determined. concluded that although there were Internet Archive-digitized significant benefits to implementing a Discovery Interface – With the ver- materials and begin ingest third instance of storage, given the high sion 1 catalog beta release only a few into the repository level of preservation confidence in Ha- months away, the Discovery Interface • Continue large-scale search thiTrust and the absence of economic Working Group is turning its focus to performance monitoring conditions favorable for acquiring and the usability of the catalog and its inte- • Make progress toward the operating new storage, there was no integration of Collection gration with existing HathiTrust Digi- urgency in establishing a new instance. Builder functionality in full- tal Library services (Collection Builder, The group noted, however, that Ha- text search results Page Turner, and Full-Text Search). thiTrust should be prepared to estab- The Working Group formed a usability lish a third instance of storage if such subgroup, which will collaborate with Presentations a course becomes more economically staff at OCLC to begin usability testing feasible. NISO Webinar Feb 10 of the catalog before it is released. Test- Please see http://www.hathitrust. ing will also be performed in post-re- The Executive Committee would like to org/papers for links to all Ha- lease phases. Aspects of the pre-release solicit broader feedback from partner thiTrust presentations, papers, and analysis will include verifying accurate institutions regarding these recommen- reports. functionality and fulfillment of agreed- dations (especially from a collection de- upon requirements. velopment perspective), and requests that thoughts on the report and a third In preparation for loading HathiTrust instance of storage be sent by email to volumes into Worldcat for the version [email protected]. Those who 1 release, staff at UM provided an API wish to remain anonymous should indi- that will allow OCLC to display Ha- cate this in their email. The full report of thiTrust volume information in World- the working group is available at http:// cat records. www.hathitrust.org/projects#wg_stor- Collaborative Development Envi- age. ronment – UM staff have been gather- ing specific topics for the working group Ingest to discuss when it reconvenes (now General – Ingest rates were low in planned for late February), and have January, due in part to challenges UC developed a draft timeline for the steps experienced in retrieving bibliographic ahead. A message to reassemble the records from one of its systems. UM group was sent in early February, and loaded the first set of bibliographic re- scheduling is underway. The area the cords for Minnesota, but could not be- group will address first is the design of gin ingest because of problems with a version control system. UM staff have Google’s delivery of the content files. also begun to research the GlusterFS Ingest numbers from other institu- cluster file system as a storage back-end tions were also low because HathiTrust for the environment. caught up with the rate that partner vol- Storage – The working group tasked umes were made available from Google. HathiTrust Digital Library Update On January Activities Internet Archive Ingest –UM began Data API – In January, staff at the time devoted to the ingest of materials testing validation routines on a batch University of Michigan began work on from the Internet Archive decreases. of 200 volumes of Internet Archive- a web application that will use the Data Outages – There were no outages in digitized volumes from the University API to facilitate the location and down- January. of California in January. The teams are load of complete book packages for revising validation strategies based on public domain volumes not digitized by the findings of these tests and the re- Google. The application is being creat- sults of quality assurance performed ed entirely with data and services avail- by UC staff on transformed, but not yet able to the general public and is meant ingested objects. UM and UC will pro- to demonstrate uses that can be made ceed with the ingest pilot in February, of the API. The first step of crawling testing all aspects of bibliographic and the repository for eligible volumes is in content loading, validation, and access. progress, and release of a beta version Completion of the pilot is projected for of the application is expected in Febru- late February. ary. New Programmer For Non-Google Large-scale Search – UM improved Ingest – UM extended the bidding pe- logging and log analysis in January, riod for the new programmer position enabling staff to monitor search per- through mid-January, and several new formance in a way that more closely re- qualified candidates have been inter- sembles the user’s experience. UM staff viewed. UM staff are in the final stages documented changes to large-scale of selecting candidates, and expect to search hardware in a new blog post en- have a new full-time staff member and titled “Scaling up Large Scale Search a new part-time staff member on board from 500,000 volumes to 5 Million vol- by the end of February. umes and beyond”. Development Updates New index servers were ordered for the Indiana site and are scheduled to be in Shibboleth – Shibboleth implemen- service before the end of March. The tation in HathiTrust is nearly complete. current index release process already Major portions of the code are in place synchronizes an updated version of the and UM staff have begun to contact index to be stored in Indiana on a daily partner institutions to exchange infor- basis.
Load more

Recommended publications
  • How to Find Free, Reusable Content Online Rhode Island Library
    Open Everything: How to find free, reusable content online Rhode Island Library Association Conference 2016, “Color Outside the Lines” Andrée Rathemacher • Julia Lovett • Angel Ferria University of Rhode Island Open Culture General Resources: Sites, Portals & Guides Digital Public Library of America — http://dp.la/ Aims to be a national digital library for the​ USA. Harvests metadata and content in all formats from other digital libraries and databases (HathiTrust, Internet Archive, state/consortium repositories, govt repositories etc. ­ full partner list here http://dp.la/partners) Does not yet allow searching/filtering by rights information. ​ ​ Europeana — http://www.europeana.eu/portal/ Europe’s portal t​o cultural collections: “Explore 52,219,831 artworks, artefacts, books, videos and sounds from across Europe.” Can filter search results by reuse rights. Internet Archive — http://archive.org Founded in 1996. A “no​n­profit library of millions of free books, movies, software, music, and more.” Searchable by Creative Commons license or Public Domain: See https://archive.org/about/faqs.php#1069 ​ Open Culture — http://www.openculture.com/ Founded in 2006. B​rings together free/open resources from around the web. Geared for a popular audience, with frequent blog posts and active social media presence. OpenGLAM Open Collections — http://openglam.org/open­collections/ A searchable index of open cultural her​ itage collections with freely reusable content. Shared Shelf Commons — http://www.sscommons.org Freely available images and oth​ er digital content from libraries, archives, and museums participating in Shared Shelf by Artstor. Copyright restrictions vary. Creative Commons Search — https://search.creativecommons.org/ Search CC­licensed content from m​ultiple sites such as Flickr, Google, and YouTube.
    [Show full text]
  • Hathitrust Preferred Internet Archive Book Package Overview
    HathiTrust Preferred Internet Archive Book Package Overview & Background As a by-product of the Internet Archive scanning process, a variety of different files and formats are available to everyone, everywhere. This differs from the Google output, which offers no file-level variations or options. However, this also means that files chosen for ingest into the HathiTrust repository must be carefully selected, with an eye towards both near-term and long-term utility. The process of selecting files that is described below attempted to balance the following important criteria: a baseline, cross-partner standard; functional consistency with the Google work products; a desire to keep the highest quality master images; a disinclination to discard useful information; and an attempt to minimize overall package size to reduce storage costs. Ingest into the HathiTrust repository will require pre-processing of the original file set described below in order to normalize files to an expected format. This normalization will allow HathiTrust processes to accommodate content from all partners. This process is currently in development and a link to the documentation of the process will be included here, once it is finalized. File Selection Criteria In the following section, the files selected for ingest into the HathiTrust repository are identified, along with a justification for why they were selected. Also listed are files that are available from the Internet Archive, but have not been selected. A description of each file can be found in the All Available Files & Characteristics section below. All files below are ​ ​ ​ named using the Internet Archive identifier, preceding the underscore (ex.
    [Show full text]
  • Using Digital Libraries: Search Strategies for Family Historians
    Using Digital Libraries: Search Strategies for Family Historians Elizabeth M. O’Neal PO Box 1259, Lompoc, CA 93436 [email protected] https://www.swangenealogy.net https://mydescendantsancestors.com Looking for books about your family history? You may be able to find them without leaving the comfort of your home! Millions of books have already been digitized and are free to use, as well as download to your personal library. Learn where to find the best digital book collections, how to strategically search them, and how to save your finds to your computer or cloud storage for later reference. Digital Libraries – Family History FamilySearch Digital Library - https://www.familysearch.org/library/books The newly-updated FamilySearch Digital Library contains more than 440,000 digitized genealogy and family history books and publications from the archives of family history libraries such as the Allen County Public Library, the Family History Library in Salt Lake City, and others. Included in the collection are family histories, county and local histories, genealogy magazines and how-to books, gazetteers, medieval histories and pedigrees. While some books are only viewable in a Family History Center, many can be viewed from – as well as downloaded to – your home computer. To access the FamilySearch Digital Library, visit FamilySearch.org, and click “Search” in the top menu. In the drop-down menu, select “Books.” On the home page of the digital library, you will see a simple search bar. Here, you can type in a surname, historical events, groups of people, or names of places. A search will cover every word of text.
    [Show full text]
  • Membership and Diversification of the Collection, to ROI and IIIF, Repository Infrastructure and Hathitrust Relevance to Open Scholarship
    What You Want to Know Answers from HathiTrust Staff This report includes HathiTrust staff answers to questions submitted during the registration process for the 2019 Member Meeting. HathiTrust asked future attendees, “Based on the theme, what question would you like answered at the meeting about HathiTrust's work towards "Driving and Supporting Change”? These questions covered everything from international membership and diversification of the collection, to ROI and IIIF, repository infrastructure and HathiTrust relevance to open scholarship. While we addressed many attendees’ questions during the meeting, the range of interests was too broad to allow adequate attention to answers during the meeting. Please contact us with any questions about the report. ([email protected]) ​ ​ Membership Open Infrastructure and Scholarship Collections: Diversity and Inclusion Technical Infrastructure and Interoperability Access and Formats Membership ● What developments are underway (governmental, outreach, other) to develop an increasingly international partnership? In the last several years we have seen a surge of interest in membership from non-US institutions. In 2018 and 2019 we added 28 new members, 3 of which are located in Canada and 3 in Australia. In 2019 we adopted new, formal membership criteria that clarified that membership is open to non-US institutions. December 5, 2019 Prepared by HathiTrust Staff 1 What You Want to Know Answers from HathiTrust Staff When evaluating how we might continue to expand the membership, HathiTrust
    [Show full text]
  • Law and Literacy in Non-Consumptive Text Mining: Guiding Researchers Through the Landscape of Computational Text Analysis Rachael G
    CHAPTER 17 Law and Literacy in Non-Consumptive Text Mining: Guiding Researchers Through the Landscape of Computational Text Analysis Rachael G. Samberg Cody Hennesy1 Imagine you are working with two digital humanities scholars studying post-WWII poetry, both of whom are utilizing a single group of copy- right-protected works. The first scholar has collected dozens of these poems to closely analyze artistic approach within a literary framework. The second has built a personal database of the poems to apply automat- ed techniques and statistical methods to identify patterns in the poems’ syntax. This latter methodology—in which previously unknown pat- terns, trends, or relationships are extracted from a collection of textual documents—is an example of “computational text analysis” (CTA),2 also commonly referred to as “text mining” or “text data mining.”3 289 290 Chapter 17 In accessing, building, and then working with these collections of texts (or “corpora” to use the jargon of the digital humanities), both scholars are exercising rights and making elections that carry legal impact. Indeed, they may not even be aware of the choices they can or must make: • From a copyright fair use perspective, does it matter whether a scholar compiles poems to read (or “consume”) or, like the CTA scholar above, uses algorithms to mine information within them (often referred to as “non-consumptive” analysis)? • How does an added layer of university database licensing, a pub- lisher-provided API (application programming interface), a uni- versity archives
    [Show full text]
  • 1 Comments of the Library Copyright Alliance On
    COMMENTS OF THE LIBRARY COPYRIGHT ALLIANCE ON INTELLECTUAL PROPERTY PROTECTION FOR ARTIFICIAL INTELLEGENCE INNOVATION The Library Copyright Alliance (“LCA”) consists of the American Library Association, the Association of College and Research Libraries, and the Association of Research Libraries. Collectively, these three library associations represent over 100,000 libraries in the United States employing over 350,000 librarians and other personnel. LCA appreciates the opportunity to respond to the USTPTO’s request for comments regarding intellectual property protection for artificial intelligence (“AI”) innovation published in the Federal Register at 84 Fed. Reg. 58141 on October 30, 2019. LCA’s response will focus on question 3: whether the statutory language of the fair use doctrine and related case law adequately address the lawfulness of the ingesting of large volumes of copyrighted material necessary for an AI algorithm or function. LCA believes that the fair use right is adequate to this task. However, license terms employed by website operators and database providers could interfere with the ingestion of materials. A statutory “contract override” provision similar to that found in various EU directives may be necessary to resolve this problem. I. Fair Use AI is opening new fields of scholarly research such as the digital humanities. See Matthew Sag, The New Legal Landscape for Text Mining and Machine Learning, 66 J. Copyr. Soc. USA __ (forthcoming 2019); Michael Carroll, Copyright and the Progress of Science: Why Text and Datamining is Lawful, 53 U.C. Davis L. Rev. __ (forthcoming 2019). This research requires the creation of searchable databases where AI is employed to enable users to detect patterns across a large number of works.
    [Show full text]
  • Corinna Baksik Hathitrust Discovery and Access @ Harvard Library
    HathiTrust discovery and access @ Harvard Library Corinna Baksik October 26, 2020 Code of Conduct HathiTrust events provide an inclusive environment that welcomes inquiry, constructive criticism and debate, and candor. HathiTrust does not tolerate personal attacks, harassment of any kind, verbal or physical violence, or disruptive behavior. All attendees are expected to be respectful of our community’s diversity and generous of others’ views. A full Code of Conduct and a complete process for handling reports of violations is in development. Until it is available, please bring concerns to us by contacting a member of the HathiTrust staff or by emailing [email protected]. 2 ● Zoom Meeting Features ● Mute and Unmute Technology ● Chat ● Speaker View and Gallery View overview ● Automated Transcript/Closed Captions ● Support Mute / Unmute When not speaking, please keep your Microphone muted To turn on your microphone, click the “Unmute” button in the bottom left corner Click the “Mute” button to then turn your microphone off Picture of the Zoom layout, pointing out where the mute and start video buttons are located Chat Use Chat for discussion or to ask a question To open the Chat window, click the Chat button at the bottom of the Zoom application By clicking the “Everyone” button in the Chat pop out window, you can Picture of the Zoom Chat pop out window select who you want to chat with Overview of HOLLIS, the online catalog of the Harvard Library What's in HOLLIS Harvard local Central Discovery CDI offers 2 separate collections Index HathiTrust collections ● ● for activation, both Print collection Scholarly articles ● ● updated monthly: Images Academic repositories ● Geospatial data ● Open access HathiTrust Digital ● Finding aids ● HathiTrust Library Full View U.S.
    [Show full text]
  • The Gutenberg-Hathitrust Parallel Corpus: a Real-World Dataset for Noise Investigation in Uncorrected OCR Texts
    The Gutenberg-HathiTrust Parallel Corpus: A Real-World Dataset for Noise Investigation in Uncorrected OCR Texts Ming Jiang1, Yuerong Hu2, Glen Worthey2[0000−0003−2785−0040], Ryan C. Dubnicek2[0000−0001−7153−7030], Boris Capitanu1, Deren Kudeki2, and J. Stephen Downie2[0000−0001−9784−5090] 1 Illinois Informatics Institute, University of Illinois at Urbana-Champaign 2 School of Information Sciences, University of Illinois at Urbana-Champaign fmjiang17|yuerong2|gworthey|rdubnic2|capitanu|dkudeki|[email protected] Abstract. This paper proposes large-scale parallel corpora of English- language publications for exploring the effects of optical character recog- nition (OCR) errors in the scanned text of digitized library collections on various corpus-based research. We collected data from: (1) Project Gutenberg (Gutenberg) for a human-proofread clean corpus; and, (2) HathiTrust Digital Library (HathiTrust) for an uncorrected OCR-impacted corpus. Our data is parallel regarding the content. So far as we know, this is the first large-scale benchmark dataset intended to evaluate the effects of text noise in digital libraries. In total, we collected and aligned 19,049 pairs of uncorrected OCR-impacted and human-proofread books in six domains published from 1780 to 1993. Keywords: Parallel Text Dataset · Optical Character Recognition · Digital Library · Digital Humanities · Data Curation 1 Introduction The rapid growth of large-scale curated datasets in digital libraries (DL) has made them an essential source for computational research among various schol- arly communities, especially in digital humanities (DH) and cultural analytics (CA). Particularly, recent studies in DH and CA have popularly employed state- of-the-art natural language processing (NLP) techniques for corpus analysis.
    [Show full text]
  • Digital Access to Materials
    Digital Access to Research Materials Patrons are able to access digital materials via the following services, while the Library is closed: Jacob Edwards Library: https://www.jacobedwardslibrary.org/ Local History & Genealogy: https://www.jacobedwardslibrary.org/genealogy.htm LEA-Library E-books and Audio-books: Massachusetts Library Networks are collaborating to bring you LEA, a new and innovative way to gain access to more eBooks and audiobooks. Powered by OverDrive, LEA makes it possible for you to borrow eContent in all Networks regardless of your home library: https://libraries.state.ma.us/pages/LEA OverDrive Digital Catalog: read e-books, audio-books, magazines, and watch video: https://cwmars.overdrive.com/ Project Gutenberg: a library of over 60,000 free eBooks. You will find the world's great literature here, with focus on older works for which U.S. copyright has expired: https://www.gutenberg.org/ Internet Archive: a non-profit library of millions of free books, movies, software, music, websites, and more: https://archive.org/ HathiTrust Digital Library: HathiTrust is a partnership of academic & research institutions, offering a collection of millions of titles digitized from libraries around the world: https://www.hathitrust.org/ Google Scholar: Google Scholar provides a simple way to broadly search for scholarly literature: https://scholar.google.com/ Boston Public Library e-card: https://www.bpl.org/ecard/ Digital Commonwealth: Explore historical collections from libraries, museums, and archives across Massachusetts: https://www.digitalcommonwealth.org/ DPLA: Digital Public Library of America: Discover 36,807,992 images, texts, videos, and sounds from across the United States: https://dp.la/ Europeana: Explore inspiring cultural heritage from European museums, galleries, libraries and archives: https://www.europeana.eu/portal/en .
    [Show full text]
  • Downloading E-Texts (PDF)
    "le LIBRARIES University at Buffalo The State Universityof New York E-Readers, Computers, and More: Downloading E-Texts The University at Buffalo Libraries subscribe to or own outright many downloadable e-resources. All can be read on a computer, using browsers such as Internet Explorer or Firefox, and most can be downloaded to a diversity of electronic devices, including dedicated e-readers such as the Nook, Sony Reader, or Kindle as well as Apple and Android devices. To find e-books, search the Libraries’ Catalog and select e-books on the far right of the basic search screen and then search. To find electronic journals, use our e-journals search engine. The Buffalo and Erie County Pubic (BECPL) provides access to the digital version of many popular and general interest books through its Overdrive subscription. Here you’ll find titles in several formats that will load onto Kindles, Nooks, Sonys, and other devices. Associated instructional materials are excellent and applicable, in many respects, to what follows. You’ll need a BECPL borrower’s card to take advantage of this service. Learn about it and using e-reading devices with these materials. The most flexible reading devices are tablets, since they generally allow the reading of material by rival companies with the appropriate app installed. See Reading E-Texts on Your Personal Computer and More toward the end of this guide. Putting E-Texts on a Device The basics: For digitized material not protected by DRM, create an archival copy of the text on your computer before moving the file to a device.
    [Show full text]
  • Students' Perceptions of Their Self-Efficacy in The
    OPEN PEDAGOGY APPROACHES: Faculty, Library, and Student Collaborations Edited by KIMBERLY DAVIES HOFFMAN and ALEXIS CLIFTON https://tinyurl.com/OpenPedApproaches Open Pedagogy Approaches, edited by Kimberly Davies Hoffman and Alexis Clifton, is licensed under a Creative Commons Attribution 4.0 International License, except where otherwise noted. Published July, 2020. SHARING THE END OF THE WORLD: STUDENTS’ PERCEPTIONS OF THEIR SELF-EFFICACY IN THE CREATION OF OPEN ACCESS DIGITAL LEARNING OBJECTS Sarah Hutton, Lisa Di Valentino, and Paul Musgrave Authors • Sarah Hutton, M.L.I.S., Ph.D. Candidate, University of Massachusetts Amherst • Lisa Di Valentino, University of Massachusetts Amherst • Paul Musgrave, University of Massachusetts Amherst Project Overview Institution: University of Massachusetts Amherst Institution Type: public, research, land-grant, undergraduate, post-graduate Project Discipline: Political Science Project Outcome: student-created podcasts; research study Tools Used: Audacity, LibGuides, Institutional Repository 8 | SHARING THE END OF THE WORLD: STUDENTS’ PERCEPTIONS OF THEIR SELF-EFFICACY IN THE CREATION OF OPEN ACCESS DIGITAL LEARNING OBJECTS Resources Included in Chapter: • Suggested resource list • Survey Instrument 2020 Preface The course that this chapter is based on took place in 2018, well before the COVID-19 pandemic that occurred at the time of publication. The University of Massachusetts, along with other schools, closed its campus beginning in the Spring semester of 2020 and moved to online instruction and library services. While many of the library’s materials are available online, and services such as HathiTrust and the Internet Archive National Emergency Library provided digital versions of a wide range of print materials, there remained a huge gap in what students and faculty could access in the “new normal” of virtual education.
    [Show full text]
  • Gathering Textual Data Bulk Retrieval in This Module We’Ll…
    Digging Deeper, Reaching Further Module 2.2: Gathering Textual Data Bulk retrieval In this module we’ll… §Introduce options for automated data retrieval à Facilitate researcher needs for data access §Practice web scraping with wget on the command line à Gain confidence automating data collection §See how Sam gathered data for his Creativity Corpus à Learn how one scholar used automated data collection M2.2 - 2 Where we’ll end up Run command from command line interface… …to scrape the text from a webpage. M2.2 - 3 Bulk retrieval §Most researchers will need more than 1 or 10 texts • Hundreds, thousands, or millions of texts §Getting lots of data could take lots of time! • Point-and-click is inefficient • Automate when possible M2.2 - 4 Automating retrieval Transferring files § FTP or SFTP: (Secure) File Transfer Protocol • moves files from one place to another on the internet § rsync • Efficient: sends only the differences • Run from command line • Used by HathiTrust, can be used to download Extracted Features data M2.2 - 5 Automating retrieval Web scraping (grabbing text on the web) § Avoids tedious copying-and-pasting § Some ways to scrape text from the web: • Run commands such as wget or curl in the command line • Write and run a script (a file of programming statements) • Use software such as webscraper.io or Kimono We’ll do an activity with this one later! M2.2 - 6 Web scraping for the wise §Web scraping puts a large workload on targeted server • This can upset the data holder §Some data providers are more than willing to share • Ask
    [Show full text]