Ohiolink Librarians and Google Scholar Over Time: a Longitudinal Analysis of Attitudes and Uses

University of Dayton eCommons

Roesch Library Faculty Publications Roesch Library

8-20-2020

OhioLINK Librarians and Google Scholar Over Time: A Longitudinal Analysis of Attitudes and Uses

David Luftig Washington State University

Joan Plungis University of Dayton, [email protected]

Follow this and additional works at: https://ecommons.udayton.edu/roesch_fac

Part of the Scholarly Communication Commons eCommons Citation David Luftig and Joan Plungis (2020). OhioLINK Librarians and Google Scholar Over Time: A Longitudinal Analysis of Attitudes and Uses. Reference Services Review. , https://ecommons.udayton.edu/roesch_fac/67

This Article is brought to you for free and open access by the Roesch Library at eCommons. It has been accepted for inclusion in Roesch Library Faculty Publications by an authorized administrator of eCommons. For more information, please contact [email protected], [email protected]. Page 1 of 34 Reference Services Review

1 2 3 4 5 6 OhioLINK Librarians and Google Scholar Over Time: A Longitudinal 7 8 9 Analysis of Attitudes and Uses 10 11 This study provides a longitudinal analysis of the opinions and uses of Google Scholar 12 Reference Services Review 13 by members of the OhioLINK consortium of libraries. Using survey data collected in 2007, 14 15 2014, and 2019 via the OhioLINK Listserv, this study explores how librarians use and 16 17 18 promote Google Scholar within their library instruction sessions and how Google Scholar 19 20 and its Library Links feature are promoted on library websites. This information is then 21 22 analyzed across the three date ranges. The results of this research suggested that there 23 24 were significant changes in the use and opinions of Google Scholar between 2007 and 25 26 2014 with more normalization of uses and opinions occurring between 2014 and 2019. 27 28 Google Scholar Background 29 30 The beta launch of Google Scholar™ a little over fifteen years ago introduced a new 31 32 resource into the world of academic search tools. If academic librarians would embrace this 33 34 upstart as a worthy and welcome complement to existing discovery methods -- or not, and 35 36 37 whether they would ultimately help their users understand its strengths and weaknesses, were 38 39 questions posed early and often in the literature. 40 41 A Google engineer, Anurag Acharya, initially conceptualized GS as a one-stop place to 42 43 find scholarly information across disciplines and languages (Giles, 2005). Since its inception in 44 45 2004, Google Scholar (GS) has claimed to provide “a simple way to broadly search for scholarly 46 47 literature” across disciplines so as to locate “articles, theses, books, abstracts and court 48 49 opinions, from academic publishers, professional societies, online repositories, universities and 50 51 other web sites” (Google, 2020a). Since 2004, both the popularity and the number of indexed 52 53 materials within GS have continued to grow (Cothran, 2011; De Winter, Zadpoor and Dodou, 54 55 56 2014; Harzing & Alakangas, 2015; Fagan, 2017; Cole, Davis, Eyer and Meier, 2018; 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 2 of 34

1 2 3 Gusenbauer, 2018; Oh & Colón-Aguirre, 2019). The amount and quality of documents within GS 4 5 is often equal to or better than that of the well-known scholarly subscription databases that 6 7 many academic libraries subscribe to such as Scopus, Web of Science, and Compendex (De 8 9 10 Winter, Zapdoor, & Dodou 2014; Wu & Chen, 2014; Harzing & Alakangas, 2015). 11 12 GS also allows libraries to make their electronic resources discoverable and accessible 13 Reference Services Review 14 through the GS interface. This means that a library’s subscription resources can be searched, 15 16 located, and linked to via the GS interface. Such library search software is typically referred to 17 18 as a discovery layer. Researchers have observed that GS provides more or equally salient 19 20 search results as other proprietary systems such as Ex Libris Primo and Summon (Zhang, 21 22 2013; Namei & Yang, 2015). 23 24 Since 2005, OhioLINK has shared its link resolver information with GS. OhioLINK is a 25 26 consortium of over one hundred university and college libraries located among 88 member 27 28 institutions in the State of Ohio. The State of Ohio Library is the only non-academic library within 29 30 31 OhioLINK. OhioLINK membership includes 16 public university libraries, 51 independent college 32 33 libraries, 23 two-year college libraries, 16 regional campus libraries, 8 law school libraries, and 5 34 35 medical school libraries. (OhioLINK, 2019b.) This study presents longitudinal research that 36 37 examines the opinions and uses of GS by OhioLINK librarians1. It seeks to understand how 38 39 librarians and their institutions teach, promote, and view GS. Furthermore, it looks to understand 40 41 how these factors have changed over time. This data was collected through a mixed 42 43 quantitative and qualitative survey of OhioLINK librarians that was sent out on the consortium 44 45 listserv in 2007, 2014, and 2019. Although there have been numerous studies that examine the 46 47 efficacy of GS, there is a lack of longitudinal research regarding librarian opinions and uses of 48 49 50 GS. 51 52 53 54 1 We are using the term librarians to refer to members of the OhioLINK listserv who responded to the 55 invitation to complete the survey. The survey did not ask for job title, function, or rank, so responses likely 56 came from both librarians and staff members at OhioLINK libraries. 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 3 of 34 Reference Services Review

1 2 3 Literature Review 4 5 Google Scholar (GS) was introduced in 2004 as an open research database that 6 7 indexes scholarly abstracts, research articles, law reviews, gray literature, books, materials 8 9 10 within institutional repositories, and more through Google’s search interface (Giles, 2005; 11 12 Zhang, 2013, Google, 2020b). The number of documents indexed by GS is unclear, but 13 Reference Services Review 14 researchers have estimated the number to be above 160 million records across all languages 15 16 and disciplines (Orduna-Malea et al., 2015; Gusenbauer, 2018). In 2020, the quantity and scope 17 18 of indexed materials within GS compares favorably to subscription databases such as 19 20 Compendex (Cusker, 2013; Cole, Davis, Eyer and Meier, 2018), Web of Science, (De Winter, 21 22 Zadpoor, Dodou, 2014; Harzing and Alakangas 2015; Martin-Martin, Orduna-Malea, Thelwall, 23 24 Lopez-Cozar; 2018), and Scopus (Harzing and Alakangas 2015; Martin-Martin, Orduna-Malea, 25 26 Thelwall, Lopez-Cozar; 2018). 27 28 Librarian Reception of Google Scholar 29 30 31 In the immediate years following the release of GS, librarians’ opinions were often mixed 32 33 (Cathcart and Roberts, 2005; York, 2005; Kesselman and Watstein, 2006; Pomeranz, 2006; 34 35 Taylor, 2007; Giglierano, 2008; Antell, Strothmann, Chen, & O’Kelly, 2013). In Cathcart and 36 37 Roberts’ (2005) evaluation of GS, the authors voiced enthusiasm that GS could be an easy to 38 39 use and monetarily free research tool but they also feared that GS could lure students away 40 41 from library resources.. According to Cathcart and Roberts’ (2005), this was especially troubling 42 43 because GS did not provide their indexing methodology. This sentiment was echoed by 44 45 Kesselman and Watstein (2006) who valued GS’s easy to use interface and lack of subscription 46 47 cost but were also concerned that GS would be the only place students would go to find 48 49 50 research. Donlan and Cooke (2005) also noted that GS lacked transparency regarding data 51 52 retrieval and display, but they also stated that GS has the potential to be a popular and useful 53 54 tool. In Giglierano’s (2008) survey of fifty OhioLINK librarians, respondents typically stated that 55 56 they were concerned that users would abandon the library’s subscription resources for a 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 4 of 34

1 2 3 platform that was inferior. Ashmore and Grogg (2006) opined that the commercial nature of 4 5 Google and the subsequent growth of GS may change the way libraries provide information in 6 7 unforeseen ways. 8 9 10 Many librarians did show enthusiasm for GS during the early days. Much of that 11 12 enthusiasm was generated by GS’s quest to index an unprecedented scope of materials to go 13 Reference Services Review 14 along with their easy-to-use search interface (Tenopir, 2005; Friend, 2006; Norris, 2006; 15 16 Pomeranz, 2006; Arendt, 2008). Tenopir (2005) noted that that GS “has real potential to provide 17 18 easy, one-stop access to articles in both subscription journals and items in institutional 19 20 repositories, open access journals, and e-print servers (p. 32).” Friend (2006) posited that GS 21 22 would encourage more open-access content while simultaneously indexing unprecedented 23 24 amounts of materials. Giles (2005) noted that GS is a multilingual and multidisciplinary 25 26 repository that may assist researchers from around the world. Cathcart and Roberts (2005) 27 28 noted that although GS does have its flaws, the articles returned within their searches compared 29 30 31 favorably to PubMed. 32 33 Today, GS is often viewed by librarians as one useful research resource among many 34 35 (Antell, Strothmann, Chen, & O’Kelly, 2013; Karlsson, 2014; Namei & Young; 2015). Librarians' 36 37 acceptance of GS corresponds to the tremendous growth in research regarding the platform. 38 39 Such research has demonstrated that GS has continued to grow their indexed materials and 40 41 work on metadata problems while maintaining their familiar search interface (Cothran, 2011; De 42 43 Winter, Zadpoor & Dodou, 2014; Harzing, 2014; Harzing & Alakangas, 2015; Waltman, 2016; 44 45 Fagan, 2017; Cole, Davis, Eyer, & Meier, 2018; Gusenbauer, 2018; Oh & Colón-Aguirre, 2019). 46 47 Furthermore, GS has introduced popular features such as research impact indicators and 48 49 50 citation alerts, which continue to make GS one of the top resources for researchers across all 51 52 levels and disciplines (Cothran, 2011, Wu & Chen, 2014, Waltman, 2016, Thoma & Chan, 2019; 53 54 Google, 2020c). 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 5 of 34 Reference Services Review

1 2 3 Although GS indexes many open-access resources, many other resources require a 4 5 subscription. Martín-Martín, Orduna-Malea, and Ayllon (2016) observed that about 40 percent of 6 7 the resources they collected were freely available via GS. The open-access content that is 8 9 10 available within GS has been observed to fall across a broad spectrum of scholarliness. This 11 12 spectrum includes peer-reviewed journals and conference proceedings as well as non-scholarly 13 Reference Services Review 14 materials found in academic and disciplinary repositories and elsewhere (Haddaway, Collins, 15 16 Coughlin, & Kirk, 2015; Fagan, 2017). 17 18 Google Scholar Library Links Integration 19 20 Beginning in February 2005, Google introduced GS Library Links. Library Links utilizes a 21 22 library’s OpenURL link resolver (such as Ex Libris’ SFX or EBSCO’s Link Source) to allow users 23 24 to discover and access their library’s subscription resources (Oder, 2005; Ashmore & Grogg, 25 26 2006; Friend, 2006; Google, 2020d). If the user is on-campus such access through GS is 27 28 typically automatic. If the user is off-campus, they must either access GS through their library’s 29 30 31 custom portal or they must set their GS settings to recognize their participating library. 32 33 Librarian opinions of GS Library Links within the first few years of its inception were 34 35 mixed. Some assessed that Library Links returned unsatisfactory search results when 36 37 compared to other discovery layers (Donlan & Cooke, 2005; Friend, 2006; Jacsó, 2008; Asher, 38 39 Duke, & Wilson, 2013). Other librarians saw the potential and benefits of GS Library Links. For 40 41 example, some librarians observed that users using GS Library Links had higher satisfaction 42 43 than when using other library resources to link to journal articles (Dixon, Duncan, Fagan, 44 45 Mandernach, and Warlick, 2010; Wang and Howard, 2011). Librarians also noticed an increase 46 47 in the library’s web traffic when implementing Library Links (Ashmore & Grogg, 2006; Dewan, 48 49 50 2012). More recent research has suggested that GS’s link resolver provides equally salient 51 52 search results when compared to traditional discovery layers, such as Ex Libris Primo (Wang 53 54 and Howard, 2011; Zhang, 2013; Namei & Young, 2015; Oh & Colón-Aguirre, 2019). 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 6 of 34

1 2 3 One way to observe the changes in opinions of GS by libraries over time is through 4 5 changes regarding how libraries promote GS on their website (Mullen & Hartman, 2006; 6 7 Neuhaus, Neuhaus, & Asher, 2008). This is because a library providing links to GS is, in a 8 9 10 sense, endorsing its use (as libraries typically do not intentionally provide links to low-quality 11 12 databases on their websites). Researchers during the first five years of GS observed that the 13 Reference Services Review 14 majority of academic libraries did not have links to GS anywhere on their websites (including on 15 16 their subject guides and alphabetical database list) (Mullen & Hartman, 2006; Giglierano, 2008; 17 18 Neuhaus, Neuhaus, & Asher, 2008). Hartman and Bowering Mullen (2008) analyzed 113 19 20 academic library homepages and observed that between 2005 and 2008, the number of links to 21 22 GS from library web pages, as well as the number of libraries utilizing Library Links, more than 23 24 doubled. Hartman and Bowering Mullen (2008) noticed a significant rise in GS links on the 25 26 alphabetical databases lists during this period (from 24 percent to 68 percent), subject database 27 28 lists (from 14 percent to 32 percent), subject guides (from 12.5 percent to 38 percent), and 29 30 31 directly on their library homepage (from 5 percent to 11 percent). There have not been more 32 33 recent studies examining where, and how often, libraries provide GS links on their websites. 34 35 Google Scholar in Library Instruction 36 37 Early opinions on providing assistance with GS within library instruction sessions were 38 39 also mixed (Ettinger, 2008; Giglierano, 2008; Jacsó, 2008). Giglierano (2008) observed that 40 41 among libraries in the OhioLINK consortium surveyed, 58 percent of respondents stated that 42 43 they did not mention GS in undergraduate instruction sessions, and 62 percent said they did not 44 45 mention GS in graduate instruction sessions. Often the reasons for not providing resources 46 47 regarding GS within library instruction classes were that GS was flawed or that there were too 48 49 50 many other, stronger resources to cover (Giglierano, 2008; Ettinger, 2008). Currently there is a 51 52 lack of research regarding how GS is utilized and promoted within library instruction sessions. 53 54 Early on, Cathcart & Roberts (2005) posited that GS that it was a resource worth 55 56 teaching within information literacy classes due to its ease of use, large index, and lack of a paid 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 7 of 34 Reference Services Review

1 2 3 subscription. Since then, studies have continued to show the desire of researchers of all levels 4 5 to use and learn about GS (Cothran, 2011, Wu & Chen, 2014, Waltman, 2016, Thoma & Chan, 6 7 2019). Such popularity is due to the ease of use and how it compares favorably to many 8 9 10 subscription databases (see De Winter, Zapdoor, & Dodou 2014; Wu & Chen, 2014; Harzing & 11 12 Alakangas, 2015; Fagan, 2017) as various discovery layers (Wang and Howard, 2011; Namei & 13 Reference Services Review 14 Young, 2015; Oh & Colón-Aguirre, 2019). Because GS has continued to grow its index while 15 16 addressing metadata and other problems, GS may continue to be a popular platform among 17 18 researchers. Such popularity may increase as libraries face budgetary shortfalls, due in large 19 20 part to the COVID-19 pandemic, which may force them to cancel subscription databases and 21 22 pay more attention to GS as a resource. 23 24 Although GS has been the subject of much research, no recent studies have examined 25 26 longitudinal opinions and uses of GS from within one group. Therefore, this research project 27 28 examines the opinions and uses of GS by OhioLINK consortium library respondents over three 29 30 31 time periods: 2007, 2014, and 2019. This project builds upon Giglierano’s (2008) original study. 32 33 Methods 34 35 This research project was a longitudinal study in which OhioLINK librarians were invited 36 37 to participate in an identical survey in 2007, 2014, and 2019. OhioLINK is a consortium of 116 38 39 private and public academic libraries and the State Library of Ohio (OhioLink 2019a; OhioLink 40 41 2019b). The institutional sizes of OhioLINK schools vary considerably. Schools range from a 42 43 total enrollment of only a few hundred students. to universities such as The Ohio State 44 45 University with tens of thousands of students. The scope of colleges and universities in 46 47 OhioLINK also vary ranging from theological schools, art, and design institutions, two-year 48 49 50 community colleges, technical schools, and agricultural institutes, small liberal arts colleges, 51 52 mid-sized universities, and Research I institutions. The OhioLINK consortium began in 1992 and 53 54 began sharing link resolver information with GS in 2005. The invitation to participate in the 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 8 of 34

1 2 3 survey was posted on the OhioLINK listserv, [email protected], which is open to all 4 5 staff at OhioLINK member libraries (OhioLINK, 2019c). 6 7 The survey contained seven questions. Six of those questions were multiple choice (with 8 9 10 a chance to write in more elaborate answers) and one question was a short answer prompting 11 12 respondents to share their comments regarding the promotion of GS. None of the questions 13 Reference Services Review 14 were required, and not all respondents answered every question. The survey was administered 15 16 via SurveyMonkey in July 2007 (n=50) and November 2014 (n=65) and in July 2019 via Google 17 18 Forms (n=58). It was the intention that the 2014 data be analyzed at the time but it was not due 19 20 to other work demands. 21 22 In the 2019 data set, a survey with an error in the second question was distributed for 23 24 two days. In that survey, respondents could not select more than one answer from a multiple- 25 26 choice list. Because of this error, twelve surveys were discarded, and a new, corrected, survey 27 28 was distributed to the list. The data sets were then compared to analyze trends in how, why, 29 30 31 and where OhioLINK librarians provided GS resources within their institutions. Qualitative 32 33 responses were evaluated using thematic analysis. 34 35 Results 36 37 Increase and standardization in libraries linking to Google Scholar 38 39 Over the fifteen-year period, the survey results show the percentages of libraries linking 40 41 to GS from their websites had flipped (Table 1). Although the percentage changes from 2007 to 42 43 2014 were dramatic, they were relatively minimal between 2014 and 2019. In 2007, 36.4 44 45 percent of those responding said there were links to GS on their library’s website; which rose to 46 47 58.5 percent in 2014; and remained a relatively stable 60.3 percent by 2019. 48 49 50 Table I. 51 52 In the 2007 results, GS links were most often provided on the library home page, in an 53 54 internet resources or other search engines list, in an alphabetical database list or a subject 55 56 database list, or in other unspecified locations (Table 2). Between 2007 and 2014, the 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 9 of 34 Reference Services Review

1 2 3 percentage of reporting libraries providing links to GS via their alphabetical database list (often 4 5 called A-Z Databases) more than doubled (from 18.8 percent to 38.9 percent). These 6 7 percentages are nearly identical between 2014 and 2019 (from 38.9 percent to 39.6 percent). 8 9 10 Table II. 11 12 Springshare LibGuides were new to the library landscape in 2007; when survey 13 Reference Services Review 14 respondents stated that their libraries listed GS links in “other resources,” none of these links 15 16 were reported to be on research or library guides. Since that time, LibGuides have become a 17 18 prevalent platform for libraries wishing to create online guides that promote their research 19 20 resources (Bushhousen, 2009; Springshare, 2020a; Springshare 2020b). Responses to the 21 22 2014 and 2019 surveys reflect such popularity, indicating that GS links were shared almost 23 24 entirely within research/library guides after 2007 (Table 3). 25 26 Table III. 27 28 Reasons for not Promoting Google Scholar via Online Library Resources 29 30 31 Overall, there were several reasons given by librarians who stated that their library did 32 33 not link to GS. Such responses were written in by the respondents and could be easily grouped 34 35 into one of four overarching themes (Table 4). By far, the main reason for not linking to GS, over 36 37 all time periods, was due to a belief that such promotion would interfere with providing support 38 39 for better, typically subscription, research resources (88.4 percent in 2007; 73.9 percent in 2014, 40 41 81.2 percent in 2019). 42 43 Table IV. 44 45 Some of the written responses that typified this belief include a 2007 respondent who 46 47 stated, “I'm not sure it's been seriously tabled for discussion to include Google Scholar on our 48 49 50 web site, but it seems to be the general consensus amongst our staff that Google Scholar does 51 52 not really compare to the versatility and access provided by our subscription databases.” In 53 54 2019, a respondent stated that they “assume the idea is to teach students to use other 55 56 resources with more tools for finding peer-reviewed and evidence-based material.” 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 10 of 34

1 2 3 Although the belief that providing links to GS takes away from promoting better 4 5 resources (or GS is simply too flawed to promote) was the most common written response for 6 7 respondents whose libraries did not promote GS on their website, other themes did emerge. 8 9 10 Between 2007 and 2014, some respondents stated that their libraries had not considered, or 11 12 had just begun considering, promoting GS (3.8 percent in 2009 and 17.3 percent in 2014). By 13 Reference Services Review 14 2019, no respondent stated that their libraries had either not considered promoting, or had just 15 16 begun considering, promoting GS on their website. 17 18 Other respondents stated that they did not know why their library did not promote GS on 19 20 their website. Such responses are an indication of the many different roles and responsibilities 21 22 of the respondents on the OhioLINK listserv. For example, a respondent in 2019 stated, “I am 23 24 not on the web team and couldn't answer that.” 25 26 In 2019, ten out of twenty-four written responses pointed out flaws with GS or stated that 27 28 it took away from providing better resources. By 2014, the idea emerged that GS can be one 29 30 31 useful tool among many. For example, one respondent stated: 32 33 We don't promote it so much as mention it as one of several "most reliable" tools - EDS, 34 35 the library's catalog, Journal Finder (A-Z), databases, Google Scholar. To our librarians 36 37 and most of our students, it is a tool for finding full text when other resources have not 38 39 provided it rather than as a preferred tool for search/research. 40 41 A similar theme was echoed in 2019: 42 43 I encourage triangulation of research strategies including Google Scholar when 44 45 appropriate. However, I give preferences to subscription databases. In my experience, 46 47 once students overcome the initial hurdle of using a new interface, they got much better 48 49 50 results than they would from Google Scholar, and they prefer the subscription 51 52 databases. 53 54 Still, others see the value of GS as a standalone tool or even a resource that could 55 56 potentially replace some of the costlier subscription resources. In 2014, a respondent stated that 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 11 of 34 Reference Services Review

1 2 3 “Google Scholar is one of the better engineering content databases. With the cancellation of 4 5 Compendex, I am mentioning it more and more.” Another respondent in 2019 stated, “I love 6 7 google scholar and use it in my own research. It's far better than any library catalog or database 8 9 10 I've ever used.” 11 12 A few responses fell into the “other” category. In 2014, a respondent stated, "[there is an] 13 Reference Services Review 14 assumption that people know about it, I guess.” In 2019, a respondent stated that it was most 15 16 likely the website design that was preventing the library from providing links to GS on its 17 18 website. 19 20 Most Libraries Do Not Provide Resources for Setting up Google Scholar Library Links 21 22 One way that libraries promote and utilize GS online, and a powerful way libraries can 23 24 integrate their content into GS, is by making it available via GS Library Links (Google, 2020d). 25 26 By setting up GS Library Links, users can locate and access their library’s subscription 27 28 resources within the GS interface (while simultaneously utilizing all of the other aspects of GS). 29 30 31 Although such library integration within GS may be automatic when on campus, it must 32 33 be set up by the user when off campus. GS Library Links usually only need to be configured 34 35 once, but it does require several steps that may not be intuitive. GS does not provide a guide to 36 37 doing this so libraries often provide personalized guides for integrating their content (see The 38 39 Ohio State University, 2020; Tiffin University, 2020). 40 41 The majority of the responses indicate that respondents’ libraries did not include 42 43 instructions or assistance regarding the setting up of GS Library Links (Table 5). In all surveys, 44 45 over half of libraries did not provide assistance to setting up GS Library Links (68.2 percent in 46 47 2007; 60.0 percent in 2014; and 56.4 percent in 2019). 48 49 50 Table V. 51 52 Google Scholar within the Undergraduate Classroom 53 54 The survey also asked if GS was routinely mentioned in undergraduate instruction 55 56 classes (Table 6). It was at the respondent's discretion as to how they interpreted "routinely." 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 12 of 34

1 2 3 Unfortunately, the researchers do not have a sense regarding how the term routinely was 4 5 interpreted by respondents. Between 2007 and 2014, there was an increase in respondents 6 7 who discussed GS with their undergraduate classes (from 23.3 percent to 38.1 percent), but 8 9 10 that percentage decreased between 2014 to 2019 (from 38.1 percent to 29.3 percent). 11 12 Additionally, there was a steady decrease of respondents who did not regularly mention GS to 13 Reference Services Review 14 their undergraduate classes (58.1 percent in 2007; 34.9 percent in 2014; and then to 25.9 15 16 percent in 2019). 17 18 Table VI. 19 20 The written responses regarding the reasons that respondents did not routinely mention 21 22 GS in undergraduate classes were grouped into one of five mutually exclusive themes (Table 23 24 7). For those respondents who did not routinely mention GS in undergraduate-level instruction 25 26 classes their reasons for not doing so were overwhelmingly because they viewed GS negatively 27 28 and/or it was not worth the limited time to mention in a class (71.4 percent in 2007; 45.0 percent 29 30 31 in 2014; and 53.8 percent in 2019). 32 33 Table VII. 34 35 In 2007, twenty-one of twenty-nine written responses mentioned that either they do not 36 37 promote GS because they, or other faculty, had a negative view of GS. One respondent stated, 38 39 “We use GS sometimes in helping students do research, but we do not do any promoting of it. 40 41 We still find that GS is still weak in providing current articles and information.” Another 42 43 respondent in 2007 seemed to sense the usefulness of GS, even if it was not widely accepted 44 45 by faculty or other librarians, stating: 46 47 I have used Google Scholar with students during a reference transaction. I don't promote 48 49 50 it regularly because many of the faculty have advised their students "not to use Google." 51 52 ...I think for Google Scholar to be successful locally, our faculty have to buy in that it is a 53 54 valuable resource. 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 13 of 34 Reference Services Review

1 2 3 By 2014, only nine out of twenty-nine written responses mentioned that GS was either 4 5 flawed or took away from time that could be better allocated teaching other resources. One 6 7 respondent stated: 8 9 10 I have a limited time to explain to students on how to effectively research. I want to 11 12 spend most of my time in the classroom teaching the resources we have through the 13 Reference Services Review 14 library. In reality, Google Scholar is a great tool but it is much more effective to go 15 16 straight to the source. Additionally, I spend a lot of time teaching subject searching and 17 18 this is not possible with Google Scholar. 19 20 The newness of GS in 2007 and the subsequent time that has passed since seems to be 21 22 reflected within the data. For example, a 2007 respondent stated that their reason for not 23 24 mentioning GS within their undergraduate classrooms was that “(GS) has yet to prove itself as a 25 26 viable pedagogical tool.” 2007 was also the only year that any respondents stated that their 27 28 reason for not routinely mentioning GS within undergraduate classes was that they did not know 29 30 31 enough about GS to teach it (9.5 percent of responses). 32 33 Another theme that emerged from the written responses was that whether to mention 34 35 GS or not varied from class to class with 2007 being the only year that there were no responses 36 37 (0 percent in 2007; 15.0 percent in 2014; and 15.4 percent in 2019). For example, a 2014 38 39 respondent wrote that “some mention it...others do not. It depends on the needs of the class 40 41 being taught.” 42 43 Finally, some respondents stated that they either did not know if GS was being taught 44 45 within undergraduate classes or that the question was not applicable to the respondent's 46 47 institution. For example, one respondent in 2014 stated, “I'm a cataloger. I virtually have no idea 48 49 50 if any of the reference staff mentions Google Scholar.” This seems reflective of the OhioLINK 51 52 consortium listserv, where membership is open to different types of academic institutions as well 53 54 as different job classifications. 55 56 Google Scholar Within the Graduate Classroom 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 14 of 34

1 2 3 In answering if GS is routinely mentioned within graduate-level classes, it was at the 4 5 respondent's discretion as to how they interpreted "routinely" on the survey. Unfortunately, the 6 7 response rates to this question were very low, which may have had significant effects on the 8 9 10 results (Table 8). One of the main reasons for such a low response rate was that many 11 12 respondents were from institutions that did not have graduate programs. 13 Reference Services Review 14 Table VIII. 15 16 With regard to if GS was routinely mentioned in graduate-level classes, there was a 17 18 substantial increase in those stating that they did mention GS from 2007 to 2014, which 19 20 stabilized by 2019 (from 18.6 percent in 2007; to 36.5 percent in 2014; to 33.8 percent in 2019). 21 22 Interestingly, there was a dramatic decline among those who said that they did not routinely 23 24 mention GS to graduate-level classes (from 62.8 percent in 2007; to 41.3 percent in 2014; to 25 26 10.7 percent in 2019). There was also an increase in respondents who did not know if GS was 27 28 routinely mentioned to graduate-level classes (from 18.6 percent in 2007; to 22.2 percent in 29 30 31 2014; to 30.4 percent in 2019). Such uncertainty could, once again, be reflective of the many 32 33 potential roles and positions of those on the OhioLINK listserv. 34 35 Respondents once again had the opportunity to write a reason that librarians did not 36 37 routinely mention GS within graduate-level classes (Table 9). These responses were easily 38 39 grouped into one of four themes. For many, the question was simply not applicable to the 40 41 respondent’s university or instruction sessions (58.8 percent in 2007; 59.0 percent in 2014; and 42 43 16.8 percent in 2019). As with previous responses, a popular theme given for not routinely 44 45 mentioning GS within graduate-level classes was that the time would either be better spent 46 47 examining other research resources or that GS was too flawed to discuss. The percentages of 48 49 50 such responses remained relatively consistent over the surveys with the caveat that only six 51 52 responded to the question in 2019 (from 35.2 percent in 2007; to 31.8 percent in 2014, to 16.6 53 54 percent in 2019). 55 56 Table IX. 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 15 of 34 Reference Services Review

1 2 3 By 2014, more responses demonstrate a positive, or at least a not so negative, view of 4 5 GS. They also mention faculty and graduate assistants promoting GS to students. For example, 6 7 one respondent stated that “we think Google Scholar represents an interesting half-way house 8 9 10 between Google search and scholarly resources. It also represents another opportunity to teach 11 12 evaluation.” Another respondent from 2014 asserted that: 13 Reference Services Review 14 I typically do not promote Google Scholar, but students come in for research 15 16 help and say they started with Google Scholar because that is what their 17 18 professor told them. I think some faculty (particularly graduate assistants that 19 20 teach introductory courses) use Google Scholar a lot so they show their 21 22 students how to use it outside of the library instruction session. 23 24 Some respondents did not know if others were using GS within their graduate-level 25 26 classes (0 percent in 2007; 4.5 percent in 2014; and 50.0 percent in 2019). Although there is a 27 28 dramatic jump in the percentages, this might be explained due to the low response rate for this 29 30 31 question as well as the varied roles of librarians on the OhioLINK listserv. Within each of the 32 33 three surveys there was one response that fell into the “other” category. In 2007, a respondent 34 35 stated that “the instructions for linking from GS to OhioLINK were recently posted, so we will 36 37 include this information in future graduate instruction classes.” In 2014, a respondent stated that 38 39 the reason “depends entirely on the librarian and discipline.” In 2019, a respondent stated, 40 41 without any context, that the reason for not routinely mentioning GS within graduate-level 42 43 classes was simply “institutional inertia.” 44 45 Discussion 46 47 The purpose of this research was to ascertain the level of acceptance, over time, of the 48 49 50 new discovery tool, Google Scholar, among members of an academic library consortium: To 51 52 determine if, and how, attitudes of librarians and other library staff changed toward promoting it 53 54 to their users on library websites and by teaching it in undergraduate and graduate classes. 55 56 Though we do not know how many, if any, of the same individuals responded to each of the 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 16 of 34

1 2 3 different survey invitations, the percentages of positive and negative responses to the same 4 5 questions, along with the qualitative comments, provided the insight we sought. 6 7 In his foreword to a recent book on resource discovery in twenty-first-century libraries, 8 9 10 Lorcan Dempsey outlines three broad trends in the current information ecosystem: "discovery 11 12 often happens elsewhere"; the library collection is not necessarily the central, nor the first used, 13 Reference Services Review 14 part of the library’s information universe; and, researchers expect library resources to fit into 15 16 their workflows, which now include many additional resources within the information network. 17 18 He includes a graphic showing the evolution of library services, from "owned" and "licensed" 19 20 collections -- materials purchased and stored -- to "facilitated" collections, including external 21 22 resources such as Google Scholar, free e-books, open access, and web resources, which meet 23 24 research and learning needs in the best way (Dempsey, 2020). 25 26 The responses in the three surveys provide an interesting juxtaposition to these trends, 27 28 conveying the viewpoints of individuals currently working in the rapidly evolving information 29 30 31 universe within a consortium made up of a diversity of types of academic libraries. This research 32 33 shows a consistent mix of attitudes toward GS over the years of its existence. Many people 34 35 responded in each survey that they did not promote or tell users about GS because they wanted 36 37 them to use the resources their libraries paid for (Dempsey's "licensed" collection). There were 38 39 also those with a profound distrust of anything Google. Others embraced GS, seeing it as a tool 40 41 that could be better for many types of research, and that could be integrated with their libraries' 42 43 link resolvers to meet their users where they were (Dempsey's "facilitated" collection). GS is far 44 45 from universally accepted by OhioLINK librarians, even if the faculty and students they work 46 47 with often start with it and find it perfectly fits their needs. This mix of attitudes was not entirely a 48 49 50 surprise, given the independent thinking characteristic of librarians when it comes to the quality 51 52 of resources. However, it is somewhat surprising how many respondents cling fiercely to the 53 54 idea that the sources their institutions pay for are most worthy of teaching and promoting, and 55 56 have not moved to more of the facilitated collection mindset. 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 17 of 34 Reference Services Review

1 2 3 Opinions regarding GS from librarians have changed since 2007, with more librarians 4 5 seeing the benefits of using GS (either as one of several research resources or as a standalone 6 7 resource). Despite these changes in opinions, many continue to believe that GS takes away 8 9 10 from time that would be better spent promoting other, often subscription, research resources. 11 12 Limitations of the Study 13 Reference Services Review 14 Respondents were not asked for the type of library where they work or to identify their 15 16 position within their libraries, which necessarily limits the conclusions that can be drawn from 17 18 the survey responses. In retrospect, limiting the sample to instruction librarians or faculty 19 20 librarians might have yielded more meaningful results. However, the trends noted in the survey 21 22 results concur with the gradual incorporation of GS into the research routines of today's 23 24 students and faculty, and the willingness of many libraries to leverage its capabilities. 25 26 Additionally, this project did not evaluate the uses and opinions of librarians using GS at 27 28 the reference desk. By providing this information it would be easier to understand the uses and 29 30 31 opinions of librarians about GS. 32 33 Conclusion 34 35 This research project examined how librarians at OhioLINK institutions viewed and used 36 37 GS over three times, 2007, 2014, and 2019. Overall, the results indicate that between 2007 and 38 39 2014 there was still much experimentation, if not apprehension, regarding GS from librarians. 40 41 By 2014, GS seems to be more accepted as a research resource by librarians. There were no 42 43 significant changes between 2014 and 2019 regarding the uses and opinions of GS by 44 45 librarians. 46 47 For example, between 2007 and 2014 there was more experimentation regarding if, 48 49 50 how, and where to provide links to GS within library websites when compared to 2014 to 2019. 51 52 In 2007, the majority of respondents stated that their libraries did not provide links to GS from 53 54 their websites. By 2014, the majority of libraries were providing links to GS from their website. 55 56 Those percentages remained relatively consistent from 2014 to 2019. 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 18 of 34

1 2 3 Although there was a significant increase in the number of library websites that provided 4 5 links to GS after 2007, the number of libraries that provided online assistance to their users 6 7 regarding setting up GS Library Links did not change significantly. Over the three surveys, the 8 9 10 percentage of libraries that provided assistance to users regarding setting up GS Library Links 11 12 was between 20.5 percent and 26.2 percent. It is unclear as to why there was never an 13 Reference Services Review 14 increase, but it may reflect a belief that paid subscription resources are better curated and less 15 16 confusing to novice researchers. 17 18 The results of the three surveys also suggest that as time went on, more librarians were 19 20 mentioning GS within both undergraduate and graduate-level classes. Between 2007 and 2019, 21 22 the percentage of respondents who did not mention GS within either their undergraduate or 23 24 graduate-level instruction classes declined precipitously. In 2014, written responses indicated 25 26 librarians see GS as a useful research resource when utilized alongside other resources, 27 28 echoing findings of other researchers (Antell, Strothmann, Chen, & O’Kelly, 2013; Karlsson, 29 30 31 2014; Namei & Young, 2015). 32 33 Starting in 2014, there appears to be a less overt cynicism regarding GS. Such changes 34 35 in opinion are reflected within the written comments of the survey and may have been caused 36 37 by the respondents’ increasing familiarity with GS as time went by, the large number of studies 38 39 that have evaluated GS, the indexing and metadata improvements made to GS since 2007, and 40 41 the increase of materials indexed within GS (See Harzing, 2014; De Winter, Zadpoor & Dodou, 42 43 2014; Orduna-Malea et al., 2015; Fagan, 2017). Another factor in play could be the perspectives 44 45 of newer librarians who grew up with GS as a fixture in the research environment. 46 47 Although librarians may view GS with less suspicion than when the product was new, 48 49 50 there remains a dominant opinion among librarians that teaching GS does take away valuable 51 52 time and resources that would be better spent promoting other, typically subscription, research 53 54 resources. Looking toward the future, it will be interesting to see if opinions and uses of GS 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 19 of 34 Reference Services Review

1 2 3 change as more libraries face budget setbacks which may force them to cancel some of their 4 5 subscription research resources. 6 7 References 8 9 10 Antell, K., Strothmann, M., Chen, X. and O’Kelly, K. (2013), “Cross-Examining Google Scholar”, 11 12 Reference & User Services Quarterly, Vol. 52 No. 4, pp. 279–282. 13 Reference Services Review 14 Arendt, J. (2013), “Imperfect Tools: Google Scholar vs. Traditional Commercial Library Databases”, 15 16 17 Against the Grain, Vol. 20 No. 2, available at: https://doi.org/10.7771/2380-176X.2737. 18 19 Asher, A.D., Duke, L.M. and Wilson, S. (2013), “Paths of Discovery: Comparing the Search 20 21 Effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and Conventional 22 23 Library Resources”, College & Research Libraries, Vol. 74 No. 5, pp. 464–488. 24 25 26 Ashmore, B. and Grogg, J.E. (2006), “Google and OCLC open libraries on the open web”, Searcher, 27 28 Vol. 14 No. 10, pp. 44–52. 29 30 Bushhousen, E. (2009), “LibGuides”, Journal of the Medical Library Association: JMLA, Vol. 97 31 32 33 No. 1, pp. 68–69. 34 35 Cathcart, R. and Roberts, A. (2005), “Evaluating Google Scholar as a Tool for Information Literacy”, 36 37 Internet Reference Services Quarterly, Vol. 10 No. 3–4, pp. 167–176. 38 39 40 Cole, C., Davis, A.R., Eyer, V. and Meier, J.J. (2018), “Google Scholar’s Coverage of the 41 42 Engineering Literature 10 years Later”, The Journal of Academic Librarianship, Vol. 44 No. 3, 43 44 pp. 419–425. 45 46 Cothran, T. (2011), “Google Scholar acceptance and use among graduate students: A quantitative 47 48 49 study”, Library & Information Science Research, Vol. 33 No. 4, pp. 293–301. 50 51 Cusker, J. (2013), “Elsevier Compendex and Google Scholar: A Quantitative Comparison of Two 52 53 Resources for Engineering Research and an Update to Prior Comparisons”, The Journal of 54 55 56 Academic Librarianship, Vol. 39 No. 3, pp. 241–243. 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 20 of 34

1 2 3 De Winter, J.C.F., Zadpoor, A.A. and Dodou, D. (2014), “The expansion of Google Scholar versus 4 5 6 Web of Science: a longitudinal study”, Scientometrics, Vol. 98 No. 2, pp. 1547–1565. 7 8 Dempsey, L. (2020). “Library discovery directions.” In S. McLeish (Ed.), Resource discovery for the 9 10 twenty-first century library: Case studies and perspectives on the role of IT in user engagement 11 12 and empowerment (pp. 4-13). Facet Publishing. 13 Reference Services Review 14 15 https://www.oclc.org/content/dam/research/publications/2020/oclcresearch-library-discovery- 16 17 directions-preprint.pdf 18 19 Dewan, P. (2012), “Making the Most of Google Scholar in Academic Libraries”, Feliciter, Vol. 58 20 21 22 No. 6, pp. 41–42. 23 24 Dixon, L., Duncan, C., Fagan, J.C., Mandernach, M. and Warlick, S.E. (2010), “Finding Articles and 25 26 Journals via Google Scholar, Journal Portals, and Link Resolvers”, Reference & User Services 27 28 29 Quarterly, Vol. 50 No. 2, pp. 170–181. 30 31 Donlan, R. and Cooke, R. (2005), “Running with the Devil: Accessing Library-Licensed Full Text 32 33 Holdings Through Google Scholar”, Internet Reference Services Quarterly, Vol. 10 No. 3–4, pp. 34 35 149–157. 36 37 38 Ettinger, D. (2008), “The Triumph of Expediency: The Impact of Google Scholar on Library 39 40 Instruction”, Journal of Library Administration, Vol. 46 No. 3–4, pp. 65–72. 41 42 Fagan, J.C. (2017), “An Evidence-Based Review of Academic Web Search Engines, 2014-2016: 43 44 45 Implications for Librarians’ Practice and Research Agenda”, Information Technology and 46 47 Libraries, Vol. 36 No. 2, pp. 7–47. 48 49 Friend, F.J. (2006), “Google Scholar: Potentially Good for Users of Academic Information”, The 50 51 52 Journal of Electronic Publishing, Vol. 9 No. 1, available at: 53 54 https://doi.org/10.3998/3336451.0009.105. 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 21 of 34 Reference Services Review

1 2 3 Giglierano, J. (2008), “Attitudes of OhioLINK Librarians Toward Google Scholar”, Journal of 4 5 6 Library Administration, Vol. 47 No. 1–2, pp. 101–113. 7 8 Giles, J. (2005), “Start your engines”, Nature, Vol. 438 No. 7068, pp. 554–555. 9 10 Google. (2020a), “About Google Scholar”, About Google Scholar, available at: 11 12 https://scholar.google.com/intl/en/scholar/about.html (accessed 27 March 2020). 13 Reference Services Review 14 15 Google. (2020b), “Inclusion Guidelines for Webmasters”, Inclusion Guidelines for Webmasters, 16 17 available at: https://scholar.google.com/intl/en/scholar/inclusion.html#content (accessed 27 18 19 March 2020). 20 21 22 Google. (2020c), “Google Scholar Citations”, Google Scholar Citations, available at: 23 24 https://scholar.google.com/intl/en/scholar/citations.html#citations (accessed 27 March 2020). 25 26 Google. (2020d), “Library Support”, Library Support, available at: 27 28 29 https://scholar.google.com/scholar/libraries.html (accessed 27 March 2020). 30 31 Gusenbauer, M. (2019), “Google Scholar to overshadow them all? Comparing the sizes of 12 32 33 academic search engines and bibliographic databases”, Scientometrics, Vol. 118 No. 1, pp. 177– 34 35 214. 36 37 38 Haddaway, N.R., Collins, A.M., Coughlin, D. and Kirk, S. (2015). “The Role of Google Scholar in 39 40 evidence reviews and its applicability to grey literature searching”, PLoS One, Vol. 10 No. 9. 41 42 Hartman, K.A. and Bowering Mullen, L. (2008), “Google Scholar and academic libraries: an update”, 43 44 45 New Library World, Vol. 109 No. 5/6, pp. 211–222. 46 47 Harzing, A.W. (2014), “A longitudinal study of Google Scholar coverage between 2012 and 2013”, 48 49 Scientometrics, Vol. 98 No. 1, pp. 565–575. 50 51 52 Harzing, A.W. and Alakangas, S. (2016), “Google Scholar, Scopus and the Web of Science: a 53 54 longitudinal and cross-disciplinary comparison”, Scientometrics, Vol. 106 No. 2, pp. 787–804. 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 22 of 34

1 2 3 Jacsó, P. (2008), “Google Scholar revisited”, Online Information Review, Vol. 32 No. 1, pp. 102–114. 4 5 6 Karlsson, N. (2014), “The crossroads of academic electronic availability: How well does Google 7 8 Scholar measure up against a university-based metadata system in 2014?”, Current Science, Vol. 9 10 107 No. 10, pp. 1661–1665. 11 12 Kesselman, M. and Watstein, S. (2005), “Google ScholarTM and libraries: point/counterpoint”, 13 Reference Services Review 14 15 Reference Services Review, Vol. 33 No. 4, pp. 380–387. 16 17 Martín-Martín, A., Orduna-Malea, E. and Ayllon, J. (2016), “A two-sided academic 18 19 landscape: Portrait of highly-cited documents in Google Scholar (1950-2013)”, Revista 20 21 22 Española de Documentación Científica (Preprint), Vol. 39 No. 4. 23 24 Martín-Martín, A., Orduna-Malea, E., Thelwall, M. and Delgado López-Cózar, E. (2018), “Google 25 26 Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject 27 28 29 categories”, Journal of Informetrics, Vol. 12 No. 4, pp. 1160–1177. 30 31 Mullen, L.B. and Hartman, K.A. (2006), “Google Scholar and the Library Web Site: The Early 32 33 Response by ARL Libraries”, available at: https://doi.org/10.7282/T3KH0KQ0. 34 35 Namei, E. and Young, C.A. (2015), “Measuring Our Relevancy: Comparing Results in a Web-Scale 36 37 38 Discovery Tool, Google & Google Scholar”, Proceedings of ACRL, Portland, OR, pp. 522–535. 39 40 Neuhaus, C., Neuhaus, E. and Asher, A. (2008), “Google Scholar Goes to School: The Presence of 41 42 Google Scholar on College and University Web Sites”, The Journal of Academic Librarianship, 43 44 45 Vol. 34 No. 1, pp. 39–51. 46 47 Norris, B.P. (2006), “Google: Its Impact on the Library”, Library Hi Tech News, Vol. 23 No. 9, pp. 9– 48 49 11. 50 51 52 Oder, N. (2005), “Google Scholar Links with Libs.”, Library Journal, Vol. 130 No. 7, pp. 17–18. 53 54 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 23 of 34 Reference Services Review

1 2 3 Oh, K. and Colón-Aguirre, M. (2019), “A Comparative Study of Perceptions and Use of Google 4 5 6 Scholar and Academic Library Discovery Systems”, College & Research Libraries, Vol. 80 No. 7 8 6, pp. 876–891. 9 10 OhioLINK. (2020a), “OhioLINK history”, OhioLINK History, available at: 11 12 https://www.ohiolink.edu/content/history (accessed 30 June 2020). 13 Reference Services Review 14 15 OhioLINK. (2020b). “About OhioLINK”, About OhioLINK, available at: 16 17 https://www.ohiolink.edu/content/about_ohiolink (accessed 30 June 2020). 18 19 OhioLINK. (2020c), “OhioLINK email listserv assistance”, OhioLINK Email Listserv Assistance, 20 21 22 available at: https://www.ohiolink.edu/forms/ohiolink_email_listserv_assistance (accessed 30 23 24 June 2020). 25 26 Orduna-Malea, E., Ayllón, J.M., Martín-Martín, A. and Delgado López-Cózar, E. (2015), “Methods 27 28 29 for estimating the size of Google Scholar”, Scientometrics, Vol. 104 No. 3, pp. 931–949. 30 31 Pomeranz, J. (2006), “Google Scholar and 100 percent availability of information”, Information 32 33 Technology and Libraries, Vol. 25 No. 2, pp. 52–56. 34 35 Springshare. (2020a), “About Springshare”, About Springshare, available at: 36 37 38 https://springshare.com/about.html (accessed 8 July 2020). 39 40 Springshare. (2020b), “LibGuides”, LibGuides, available at: https://springshare.com/libguides/ 41 42 (accessed 27 March 2020). 43 44 45 Taylor, S. (2007), “Google Scholar – friend or foe?”, Interlending & Document Supply, Vol. 35 No. 1, 46 47 pp. 4–6. 48 49 Tenopir, C. (2005), “Google in the academic library”, Library Journal, Vol. 130 No. 2, p. 32. 50 51 52 The Ohio State University. (2020), “Make Google Scholar work for you”, 8 July, available at: 53 54 http://guides.osu.edu/c.php?g=833138. 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 24 of 34

1 2 3 Thoma, B. and Chan, T.M. (2019), “Using Google Scholar to track the scholarly output of research 4 5 6 groups”, Perspectives on Medical Education, Vol. 8 No. 3, pp. 201–205. 7 8 Tiffin University. (2020), “Using Google Scholar”, Using Google Scholar, available at: 9 10 https://library.tiffin.edu/googles/setup (accessed 8 July 2020). 11 12 Waltman, L. (2016), “A review of the literature on citation impact indicators”, Journal of 13 Reference Services Review 14 15 Informetrics, Vol. 10 No. 2, pp. 365–391. 16 17 Wang, Y. and Howard, P. (2012), “Google Scholar Usage: An Academic Library’s Experience”, 18 19 Journal of Web Librarianship, Vol. 6 No. 2, pp. 94–108. 20 21 22 Wu, M. and Chen, S. (2014), “Graduate students appreciate Google Scholar, but still find use for 23 24 libraries”, The Electronic Library, Vol. 32 No. 3, pp. 375–389. 25 26 York, M.C. (2005), “Calling the Scholars Home: Google Scholar as a Tool for Rediscovering the 27 28 29 Academic Library”, Internet Reference Services Quarterly, Vol. 10 No. 3–4, pp. 117–133. 30 31 Zhang, T. (2013), “User-Centered Evaluation of a Discovery Layer System with Google Scholar”, in 32 33 Marcus, A. (Ed.), Design, User Experience, and Usability. Web, Mobile, and Product Design, 34 35 Vol. 8015, Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 313–322. 36 37 38 Appendix 39 40 Survey Instrument: 41 42 1. Does your library’s web site link to Google Scholar? 43 44 2. If yes, please select as many as applicable from the following: 45 46 [Link from library home page] 47 48 [Link from alphabetical databases list] 49 50 [Link from subject databases list(s)] 51 52 [Link from Internet resources or search engines list] 53 54 55 [Widget on page] 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 25 of 34 Reference Services Review

1 2 3 [Other] If other, please specify 4 5 3. If you do not link to Google Scholar from your library’s web site, why not? 6 7 4. Does your library's web site include instructions about setting up preferences so Library Links 8 9 10 will show in Google Scholar results? (See https://bit.ly/2xYEanT for information on Library 11 12 Links.) 13 Reference Services Review 14 5. Do instruction librarians at your library routinely mention Google Scholar in undergraduate 15 16 instruction classes? If no, please explain 17 18 6. Do instruction librarians at your library routinely mention Google Scholar in graduate 19 20 instruction classes? If no, please explain 21 22 7. Please share any other comments you have about your library’s decision to promote Google 23 24 Scholar to your users, or not to do so. 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 26 of 34

1 2 3 4 Table I. Does your library’s website link to Google Scholar? 5 6 7 8 Year Yes No Don’t know n= 9 10 11 12 13 2007 Reference36.4% Services63.6% 0%Review44 14 15 16 17 18 2014 58.5% 40.0% 1.5% 65 19 20 21 22 23 2019 60.3% 32.8% 4% 58 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 27 of 34 Reference Services Review

1 2 3 4 5 6 7 8 9 Table II. If your library does link to GS from its web site, where is that link located? 10 11 Users selected all answers that are applicable 12 13 Reference Services Review 14 15 Year Library Alphabetical Subject Internet Other n= 16 homepage database list database resources resources 17 list or other (See Table 3) 18 search 19 engine list 20 21 22 23 2007 31.3% 18.8% 18.8% 31.3% 56.3 16 24 25 26 27 28 2014 25.0% 38.9% 25.0% 25.0% 47.2% 36 29 30 31 32 33 2019 10.3% 39.6% 15.5% 17.2% 15.5% 58 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 28 of 34

1 2 3 4 5 6 Table III. Other resources mentioned by librarians as linking to Google Scholar 7 8 9 10 Year LibGuide/Research Discovery Other n= 11 Guide Layer/Link 12 13 Reference ServicesResolver Review 14 15 16 17 2007 0% 0% 100% 9 18 19 20 21 2014 66.6% 9.5% 23.8% 21 22 23 24 25 26 2019 70.0% 0% 20.0% 12 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 29 of 34 Reference Services Review

1 2 3 4 5 6 Table IV. Self-reported reasons that libraries did not link to Google Scholar from their 7 8 websites 9 10 11 Year It takes away I do not know Our institution Other reason n= 12 13 Referencefrom why GS is Servicesnot had not Review 14 promoting promoted considered, or 15 other only 16 resources beginning to 17 and/or GS is consider 18 19 too flawed promoting GS 20 21 22 23 2007 88.4% 3.8% 3.8% 0 26 24 25 26 27 28 2014 73.9% 4.3% 17.3% 4.3% 23 29 30 31 32 2019 81.2% 12.5% 0 6.2% 16 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 30 of 34

1 2 3 4 Table V. Does your library’s website include instructions about setting up preferences so 5 Library Links will show up in Google Scholar results? 6 7 8 9 Year Yes No Don’t know n= 10 11 12 13 Reference Services Review 14 2007 20.5% 68.2% 11.4% 44 15 16 17 18 19 2014 26.2% 60.0% 13.8% 65 20 21 22 23 24 2019 25.5% 56.4% 18.2% 55 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 31 of 34 Reference Services Review

1 2 3 4 Table VI. Do instruction librarians at your library routinely mention Google Scholar in 5 undergraduate instruction classes? 6 7 8 9 Year Yes No Don’t know n= 10 11 12 13 Reference Services Review 14 2007 23.3% 58.1% 18.6% 43 15 16 17 18 19 2014 38.1% 34.9% 27.0% 63 20 21 22 23 24 2019 29.3% 25.9% 37.9% 54 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 32 of 34

1 2 3 4 Table VII. Reasons stated that instruction librarians do not routinely mention Google 5 Scholar in undergraduate classes 6 7 8 9 Year GS is viewed The The Question not Depends on n= 10 negatively respondent respondent applicable to particular 11 12 and/or it or institution did not know the class 13 takesReference away did not know Services if others respondent’s Review 14 time that enough were utilizing university or 15 would be about GS to GS instruction 16 better teach it sessions 17 18 devoted to 19 other 20 resources 21 22 23 24 2007 71.4% 9.5% 4.7% 14.2% 0 21 25 26 27 28 29 2014 45.0% 0 13.0% 25.0% 15.0% 20 30 31 32 33 34 2019 53.8% 0 23.0% 7.6% 15.4% 13 35 36 37 38

39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Page 33 of 34 Reference Services Review

1 2 3 4 Table VIII. Do instruction librarians at your library routinely mention Google Scholar in 5 graduate instruction classes? 6 7 8 9 Year Yes No Don’t know n= 10 11 12 13 Reference Services Review 14 2007 18.6% 62.8% 18.6% 43 15 16 17 18 19 2014 36.5% 41.3% 22.2% 63 20 21 22 23 24 2019 33.9% 10.7% 30.4% 32 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr Reference Services Review Page 34 of 34

1 2 3 4 5 6 7 8 9 Table IX. Reasons stated that instruction librarians do not routinely mention Google 10 Scholar in graduate-level classes 11 12 13 Reference Services Review 14 Year Focus on other The respondent The question was Other n= 15 information did not know if not applicable to the Reason 16 resources others are utilizing respondent’s 17 and/or GS is GS university or 18 flawed and not instruction sessions 19 20 worth the effort 21 22 23 24 25 26 2007 35.2% 0 58.8% 5.8% 17 27 28 29 30 2014 31.8% 4.5% 59% 4.5% 22 31 32 33 34 35 2019 16.8% 50% 16.8% 16.8% 6 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 http://mc.manuscriptcentral.com/rsr