R for Libraries Session 2: Data Exploration Clarke Iakovakis Scholarly Communications Librarian University of Houston-Clear Lake This work is licensed under a Creative Commons Attribution 4.0 Hosted by ALCTS, Association for Library Collections and International License. Technical Services About me Scholarly Communications Librarian at the University of Houston-Clear Lake Hosted by ALCTS, Association for Library Collections and Technical Services Outline for Today • R Packages • Getting data into R • Exploring your data Hosted by ALCTS, Association for Library Collections and Technical Services This will (still) be overwhelming! Refer to the handouts and help resources provided in Session 1 and stick with it! Hosted by ALCTS, Association for Library Collections and Technical Services Packages • Users write their own function libraries and release them as R Packages. https://twitter.com/benmiller314/status/877223260265615360 Hosted by ALCTS, Association for Library Collections and Technical Services Packages • To install • Click on Packages > Install • Or type directly in the console: > install.packages("dplyr") Hosted by ALCTS, Association for Library Collections and Technical Services Packages Install this helpful package of packages: > install.packages("tidyverse") The tidyverse is "a coherent system of packages for data manipulation, exploration and visualization that share a common design philosophy" Are you getting error message Warning: unable to move temporary installation ? It’s possibly due to antivirus protection. Install directly from R (as opposed to R Studio) or see • https://stackoverflow.com/a/44256520 • https://stackoverflow.com/a/23167680 Image by Joseph Rickert, "What is the tidyverse?" https://rviews.rstudio.com/2017/06/08/what-is-the-tidyverse/ Hosted by ALCTS, Association for Library Collections and Technical Services Packages Installing tidyverse giving you trouble? Just install these for now: > install.packages("dplyr") > install.packages("readxl") > install.packages("stringr") > install.packages("ggplot2") Hosted by ALCTS, Association for Library Collections and Technical Services Packages After installing a package, you must load it into your R session > select(books, SUBJECT) Error: could not find function "select" > library("tidyverse") Hosted by ALCTS, Association for Library Collections and Technical Services Reading data into R • When you read a file into R, you must save it as an object. In other words, create a copy of it in R’s working memory. • When you manipulate the data, you are manipulating that object in the R environment, not the data file itself. • You have to write the data frame to disk if you want to create an actual file; e.g. use functions write.table() or write.csv() Hosted by ALCTS, Association for Library Collections and Technical Services Getting data into R • Manually create it • Import it from a file • Text file: TXT, CSV, TSV • Excel: XLSX • Statistics program: Workflow image from Garrett Grolemund & Hadley Wickham, R for Data Science. SPSS, MATLAB, Stata http://r4ds.had.co.nz/explore-intro.html. Licensed under CC BY-NC-ND. • Databases: MySQL • JSON, HTML, XML • Gather it from the web • Connect to webpages, servers, or APIs • Webscraping Hosted by ALCTS, Association for Library Collections and Technical Services Set your working directory The working directory is the folder on your computer R will use for reading and writing files > setwd("C:/Users/iakovakis/Documents/ALCTS R Webinar") • Put in quotation marks " " • Use forward slash: / Hosted by ALCTS, Association for Library Collections and Technical Services Set your working directory You can copy/paste from directory window and do a Find \ Replace / in R Studio Hosted by ALCTS, Association for Library Collections and Technical Services Set your working directory Print the current working directory to the console: > getwd() [1] "C:/Users/iakovakis/Documents/ALCTS R Webinar" Hosted by ALCTS, Association for Library Collections and Technical Services Set your working directory You can then use ./ as the root directory > read.csv(file = "./data/books.csv") will be the same as > read.csv(file = "C:/Users/iakovakis/Documents/ALCTS R Webinar/data/books.csv") Hosted by ALCTS, Association for Library Collections and Technical Services Reading tabular data Main functions for reading tabular data: • read.table(): reads all text files in tabular format • read.csv(): reads comma-separated values files • read.delim(): reads tab-separated values files Hosted by ALCTS, Association for Library Collections and Technical Services Structure of text data files • Text files use delimiters to separate data values • comma separated values (CSV) use commas • Tab separated values use tabs • Data values are typically in quotation marks "TITLE", "AUTHOR", "TOT.CHKOUT" "Macbeth", "Shakespeare", "25" "Dracula", "Stoker", "15" "1984", "Orwell", "18" Hosted by ALCTS, Association for Library Collections and Technical Services R functions take arguments • A function can take a specified number of arguments • Arguments can be found on the help page for that function > ?read.table Hosted by ALCTS, Association for Library Collections and Technical Services read.table arguments file = "./data/books.csv" The path and file which data are to be read from. header = TRUE The file contain the names of the variables as its first line sep = "," Specify the character separating fields (columns) Hosted by ALCTS, Association for Library Collections and Technical Services read.table arguments colClasses = c("ISBN" = "character") Specify the class of columns in the dataset na.strings = "" In the original dataset, how are missing values represented? stringsAsFactors = FALSE Should character vectors be converted to factors? I always set this to FALSE Hosted by ALCTS, Association for Library Collections and Technical Services read.table • The only required argument is read.table(file = ) • But specifying more arguments helps bring in the dataset faster and in the shape you want it in Hosted by ALCTS, Association for Library Collections and Technical Services read.table sample > books <- read.table("./books.csv" , stringsAsFactors = F , header = T , sep = "," , colClasses = c("TOT.CHKOUT" = "integer")) Hosted by ALCTS, Association for Library Collections and Technical Services Reading tabular data R Studio has an Import Dataset tool in the Environment Pane (upper right) It runs the read.table() or read.csv() function Hosted by ALCTS, Association for Library Collections and Technical Services Reading Excel files To read Microsoft Excel files, either: a) Save your file in Excel as a CSV and load it in R with read.csv() or read.table() b) Or, use the readxl package in R > install.packages("readxl") > library("readxl") > read_excel("./data/books.xlsx) Hosted by ALCTS, Association for Library Collections and Technical Services Tidying & transforming data Hosted by ALCTS, Association for Library Collections and Technical Services Exploring data frames > View(books) Hosted by ALCTS, Association for Library Collections and Technical Services Exploring data frames > dim(books) Dimensions: [1] 4000 12 4,000 rows, 12 columns > nrow(books) Number of rows [1] 4000 > ncol(books) Number of columns [1] 12 Hosted by ALCTS, Association for Library Collections and Technical Services Exploring data frames > str(books) 'data.frame': 4000 obs. of 12 variables: $ callnumber : chr "SI 1.2:Af 8/3" "HC110.C6 L852 1999" "HX91.O5 B57 1999" "I 20.47:P 93" ... $ title : chr "A human ideal in African art :~Bamana figurative sculpture /" "Untold millions :~secret truths about marketing to gay and lesbian consumers /" "Agrarian socialism in America :~Marx, Jefferson, and Jesus in the Oklahoma countryside, 1904-1920 /" "1:100 000- scale metric topographic map of Price, Utah, 1980 :~30 x 60 minute series (topographic) /" ... $ author : chr "Kate Ezra." "Grant Lukenbill." "Jim Bissett." "United States Department of the Interior, Bureau of Indian Affairs." ... $ LOCATION : chr "clusd" "clstk" "clstk" "clusd" ... $ TOT.CHKOUT : int 0 0 1 0 0 0 0 0 1 0 ... $ LOUTDATE : chr " - - " " - - " "03-04-2015 14:53" " - - " ... Hosted by ALCTS, Association for Library Collections and Technical Services Exploring vectors Use the dollar sign $ to specify a variable in a data frame > str(books$LOCATION) chr [1:4000] "clstk" "clstk" "clstk" "clstk" "clstk"... Hosted by ALCTS, Association for Library Collections and Technical Services unique() unique() returns all the distinct values in a variable > unique(books$LOCATION) [1] clcdr clchi clcir clcre clfhd clids cljre cljuv clref clrsd clstk clthe cltxd clua clusd In this case, it displays all unique book location codes. Hosted by ALCTS, Association for Library Collections and Technical Services table() table() returns frequency counts for a specified variable > table(books$LOCATION) clcdr clchi clcir clcre clfhd clids cljre cljuv clref clrsd clstk clthe cltxd clua clusd 11 35 1 3 350 8 136 217 186 1 2342 27 127 17 522 You can use it in combination with relational operators > table(books$TOT.CHKOUT > 50) FALSE TRUE 3993 7 Hosted by ALCTS, Association for Library Collections and Technical Services duplicated() duplicated() will give you a logical vector of duplicated values > z <- c("111", "222", "111", "333", "444") > duplicated(z) [1] FALSE FALSE TRUE FALSE FALSE > !duplicated(z) [1] TRUE TRUE FALSE TRUE TRUE Hosted by ALCTS, Association for Library Collections and Technical Services duplicated() Use duplicated() nested in table() to find out how many values are dupes