JUSTICEPage 1 RESEARCHWeb AND STATISTICS ASSOCIATION WEB SCRAPING DECEMBER FACT SH2017EET Web Scraping An Emerging Data Collection Method for Criminal Justice Researchers Erin J. Farley, Ph.D. & Lisa Pierotte, B.S. Introduction With the continual advancement and economical than techniques of computer technology and the traditionally used in the past, and proliferation of the Internet, the it arguably holds great promise amount of criminal justice-related for researchers working in the information being placed on-line criminal justice community (Levy, has dramatically increased over 2017). the last decade. As a result, public access to certain types of This brief is intended to: introduce criminal justice data and criminal justice researchers to statistical information on the web scraping and explain what Internet has rapidly expanded, web scraping is and how it works; presenting new and provide examples of how web fundamentally different data scraping has been used in access opportunities for criminal criminal justice research; and justice researchers. One method describe several issues one should researchers are using to harness be aware of if thinking about these new data access using this type of data collection opportunities is web scraping. method for criminal justice research purposes. Web scraping is essentially an automated tool for searching and extracting data from websites and other on-line What is Web Scraping? sources. Pioneered in the fields of data science and e-commerce, Web scraping is an automated web scraping provides a user with tool for finding and extracting an automated way to find and data from on-line sources. It collect data of interest from on- utilizes computer programming line sources that is more efficient Page 2 Web WEB SCRAPING FACT SHEET software and customized software code to mine data or other information from on-line How Does Web Scraping sources in order to remove a Work? copy of the data and store it in an external database for analysis. Web scraping involves the Typically, the data harvested development and use of two through web scraping is analyzed customized software programs – to answer questions that could a crawler and a scraper. The not be answered, or answered crawler systematically downloads efficiently, using the data as it data from the Internet; then the was originally presented on-line. scraper systematically pulls the Essentially, web scraping is a way relevant information to pull information from particular (unstructured, semi-structured, or web pages and re-purpose it for structured) from the downloaded customized analysis (Marres & data, codes it, and relocates it in Weltevrede, 2013). a database or file based on a pre-determined structure and Web scraping is also referred to format defined by the user. This as automated data collection, new external database or file – web extracting, web crawling, or populated with data originally web content mining. Web presented on-line – is scraping has arguably been subsequently analyzed in ways around since the inception of the the original on-line presentation World Wide Web, but it has of data did not support. primarily been utilized in the field of data science and is commonly Common software programming associated with e-commerce languages like R and Python are (Marres & Weltevrede, 2013). typically used to write the Indeed, a form of web scraping is software code for both the often used by travel-related crawler and the scraper. Hence, websites readers may be familiar software programming skills are with, specifically those that allow essential for building and consumers to compare prices for deploying a web scraper. The airline tickets or hotel rooms software code, however, is offered by different companies. constructed based on specific In the past decade, however, the search and data extraction use of web scraping has criteria established by the emerged in several other fields researcher based on his/her including journalism, marketing, understanding of the on-line data policy analysis, and psychology source(s) of interest and the research (Baker & Yacef, 2009; research questions the analysis Marres & Weltevrede, 2013; will attempt to answer. In Youyou, Kosinski, & Stillwell, 2015) Page 3 Web WEB SCRAPING FACT SHEET practice, a data source theory, however, does not necessarily developed by the researcher, mean an individual has been guides the programmer’s convicted of a crime. While a development of the crawler and criminal history record is scraper. This theory describes the generated when someone is researcher’s and programmer’s arrested, an arrest does not assumptions about the always result in a criminal charge; information source and its and a charge does not always content, as well as their result in a criminal conviction. understanding of how the Hence, it is possible for someone available data is maintained and who has not been adjudicated to how key measures are have engaged in criminal operationalized. behavior to still have a criminal record, and this information can be, and sometimes is, used by employers to screen out job Web Scraping as a Criminal applicants, arguably unfairly Justice Research Tool limiting employment opportunities for D.C. residents with such records. The use of web scraping by criminal justice researchers is a One of the key information needs relatively new phenomenon. In a in understanding the extent of this search of the literature for problem in DC requires criminal justice-related research determining what percentage of employing web scraping as a individuals with criminal records data collection tool, only a were and were not charged or handful of studies were found in convicted of a criminal offense. which web scraping was utilized. Researchers have attempted to answer this question in the past; One of these studies was but due to data fragmentation conducted by the Urban Institute across law enforcement agencies (2017) as part of a larger and the courts, the ability to exploration of how criminal accurately answer this question background checks by for D.C. has been a challenge employers may create barriers to (Council for Court Excellence, employment among residents of 2011; Duane, Reimel, and Lynch, the District of Columbia (D.C.). 2017). Background checks are utilized by potential employers, in D.C. According to the Urban Institute and around the nation, to screen researchers, web scraping job applicants and to identify provided a viable way to those with a criminal record. overcome some of the existing Having a criminal history record, data access and analysis issues Page 4 Web WEB SCRAPING FACT SHEET that resulted from this data obstacles encountered, Eads fragmentation. Specifically, Urban worked with computer Institute researchers used a web programmers proficient in writing scraper to collect publicly software code to create and available criminal history record deploy a web scraper for data for Washington, D.C. extracting publicly available data residents over a 10-year period. from the Cook County jail website These data were then used to (maintained by the Sheriff’s estimate how many D.C. residents department), including inmate had a criminal record yet had not names, their date of birth, and been convicted of a crime. The the location of the jail in which an researchers determined that of inmate was held. The information the 68,000 D.C. residents who extracted from the website using were flagged as having an arrest web scraping will be utilized as during the 10-year period one part of a larger project examined, about half had not aimed at tracking the flow of been convicted of a crime during inmates through the entire that time span. This use of web criminal justice system in Illinois. scraping allowed Urban researchers to pull information off A third example comes from a the web to produce more National Institute of Justice- accurate estimates of the funded study currently in progress number of residents with criminal at JRSA. The study is exploring records who had not been how the characteristics of various convicted of a crime. This, in turn, on-line advertisements for escorts, better informed policy discussions such as those posted on CraigsList regarding employment barriers and other on-line sources, can for D.C. residents. potentially be used to identify human trafficking cases. The Another recent example of how objective of this project is to utilize web scraping has been used for the information pulled from criminal justice-related research websites (as well as from other involves the work being done by sources like interviews) to create journalists from ProPublica Illinois, a profile of escort ads highly a non-profit news agency. In an correlated with human trafficking, article published in July 2017, thereby providing law David Eads describes ProPublica’s enforcement officers and efforts and ultimate failure to prosecutors with practical obtain certain information on the guidance to more efficiently and Cook County jail population from effectively target escort ads, the Cook County Sheriff’s thereby leading to the successful Department through a Freedom prosecution of human traffickers. of Information Act (FOIA) request. To overcome the data access Page 5 Web WEB SCRAPING FACT SHEET As part of this project, researchers are relying upon a pre-existing, large-scale web scraping tool Web Scraping Issues to known as Memex. Launched by the U.S. Department of Defense in Consider 2015, Memex searches on-line escort ads and extracts While the use of web scraping for information of interest on a daily criminal justice research is indeed basis. Since its inception, the in its infancy, the technology Memex Program1 has pulled arguably has the potential to billions of ads off the internet to provide criminal justice keep law enforcement informed researchers with an important about trends in online sex new data collection tool. Given exploitation as well as to assist the proliferation in the amount of with anti-trafficking investigations data being placed on-line, web (Sneed, 2015).