SPECIAL DOUBLE ISSUE October/December 2003 ON NEW MEASURES 230/231 24 ...... 26 Y PACE IBRAR S L ACITY OF AP ERVICES C ...... S Duane E. Webster Kaylyn Hipps Members—$25 per year for additional subscription; TIONAL ACILITIES AND F Kevin Osborn, Research & Design, Ltd., Arlington, VA Kevin Osborn, Research & Design, Ltd., Arlington, Jaia Barrett, Deputy Executive Director Jaia Barrett, Deputy Executive FFECTIVENESS . E G by Susan J. Beck by Consuella Askew by Mary E. Jackson by Dilys Morris by Clifford H. Haka and Jim Hensley by Carla Stoffle and Shelley Phipps by Jim Self Measurement Program ...... 31 ARL Service Quality Evaluation Academy . . .31 Creating a Culture of Assessment— An ARL/OLMS Workshop ...... 31 Measuring Library Service Quality—An ARL/OLMS Online Lyceum Course ...... 31 Staying Current with the ARL Statistics & Assessing ILL/DD Services Study: Assessing ILL/DD Services Study: Initial Observations ...... 21 Time and Cost Analysis: Staff Allocations Project ...... 23 Evaluating the Utilization of Facilities in ARL Libraries Creating a Culture of Assessment: The University of Arizona Experience Using Data to Make Choices: The Balanced Scorecard at the University of Virginia Library . . .28 ...... Data-Informed Decision Making ...... 30 PERATIONS AND RGANIZA OST IBRARY reprint request should be sent to ARL Publications ditor: policy may be noted for certain articles. For commercial use, policy may be noted for certain articles. For commercial a
e AINSTREAMING EW EASURES
D M N M / r e
b by Julia C. Blixrud, Assistant Executive Director, External Relations, ARL o t c
O ithin the past four years, “new measures” Development of the Initiative has entered the lexicon of library ARL has been collecting descriptive data from its E performance and assessment literature. members for dozens of years and a robust data file U
S In the 1990s, escalating economic pressures and covers 1907–08 to the present. However valuable S S
E demands for accountability from governing boards, these data are for describing long-established I
R W university administrators, and government agencies characteristics of research libraries and for E
U strongly encouraged ARL member libraries and monitoring trends, they are not adequate to L S
B many other libraries to find new ways to illustrate address emerging uses of technology for access to A
U the contribution their collections and services information nor are they helpful for describing the E
O provided to their constituencies. Traditional, input changing nature of service in research libraries. Over
M measures (i.e., size of collections, budget, and staff) the years ARL experimented with gathering data to D were not meeting institutional accountability needs. assess library performance. As early as 1982, ARL L W Although efforts were in place to measure some began a Supplementary Statistics form to test the E A I forms of output—e.g., activities such as circulation, gathering of different statistics, and if the variables N C interlibrary borrowing, and group presentations— proved useful, they were moved into the main E Statistics. For example, collecting data on N
P these, too, did not always provide a good or complete government documents and reference transactions O S indication of library performance and the cost was tested in the Supplementary Statistics and those effectiveness of their services. In the 1990s, the ARL items were added to the main Statistics in 1990–91 Statistics and Measurement Committee began
1 and 1994–95 respectively. discussing the need to move beyond the traditional It was through the Supplementary Statistics test measures, building on earlier work in organizational bed that ARL began collecting information on the 3 development conducted by ARL’s Office of character of research library investment in electronic Management Services in the late 1980s and 1 resources. The definitional problems immediately throughout the 1990s. In early 1999, the ARL
2 became apparent, as did the difficulties for libraries New Measures Initiative officially began to develop in readily collecting some of the data. Between 1996 measures that would better demonstrate libraries’ and 1998, Timothy Jewell (University of Washington) / institutional value and describe their current served as an ARL Visiting Program Officer to assess operations. early efforts to collect measures of electronic 0 Now after four years of intensive work, the New resources. The goal of his inquiry was to develop Measures Initiative is no longer new. ARL member standard definitions for more systematic collection
3 libraries have integrated some of the measures into of information about the transformation of research the mainstream of library operations and the libraries. Through a series of meetings, discussions, Statistics and Measurement program has an active reports, and analysis of existing data, the Jewell 2 agenda for research, experimentation, and study documented a community seeking to measure demonstration projects. facets of a complex environment subject to radical PECIAL SSUE ON EW EASURES ContinuedS I N M
shifts due to rapidly changing technology and methodologies available to the full ARL membership applications. Not surprisingly, his final report as well as the broader library community. Retreat highlights the wide range of efforts underway to participants agreed that not all ARL members would be capture this information but also two concerns: that required to participate in all projects, which would give any information gathered have clear utility, and that each project the flexibility it needed to test and refine time and effort required to supply the information measures without placing undue burdens on the entire be minimized. ARL membership. In addition, it was proposed that The tenor of library measurement discussions in these new projects would not be used to modify the the late 1990s was one of frustration peppered with existing data-collection efforts without considerable examples of false starts in an environment that changed testing and that the traditional measures would be faster than it could be measured. There was a sense of “frozen” in their current form in order to retain their urgency to develop better measures for the networked integrity. As new measures became available, the ARL environment but there was Statistics and Measurement no sense of commonality What emerged from the experience of the late Committee would review regarding which areas were each one for addition to the most important to work on. 1990s was a realization that any definition of regular ARL data- What emerged from the new measures would need to be accepted by the collection activities. experience of the late 1990s community as part of the larger context of After the retreat, brief was a realization that any institutional assessment. working papers were definition of new measures developed by directors for would need to be accepted some of the eight areas to by the community as part of the larger context of expand on the ideas expressed and to identify possible 4 institutional assessment. There was also a realization approaches for investigation. Authors were that developing new measures carried costs and that encouraged to build upon previous research to better this work had to be shared in order to minimize the ensure early success for the New Measures Initiative. 2 expenses for any single institution. The areas of interest were tested with the broader ARL community at the May 1999 Membership Meeting ARL Director Retreat and plans were set in motion to begin developing Seeing the need for a common direction, Carla Stoffle the measures. (University of Arizona) and Paul Kobulnicky (University of Connecticut), chairs of the ARL Statistics Progress in Areas of Interest and Measurement Committee and the Research Library The Statistics and Measurement Committee, under the Leadership and Management Committee, called a leadership of Carla Stoffle, directed the New Measures retreat in early 1999 in Tucson, Arizona. The goal of the Initiative. An ambitious agenda was set forth directing retreat was to discuss how to develop strategies that attention to a number of areas. A series of projects and 3 would address the development of new measures. special events were initiated in 2000 and considerable Retreat participants identified eight areas of interest progress has been made in nearly all of the areas of for which measures would be particularly helpful: interest identified at the retreat. • User Satisfaction User Satisfaction and Market Penetration • Market Penetration Combining two of the retreat’s areas of investigation— • Ease and Breadth of Access reaching the library community and identifying the success libraries have serving users—resulted in the 5 • Library Impact on Teaching and Learning largest new measures activity: LibQUAL+™. This • Library Impact on Research research and development project undertaken by ARL • Cost Effectiveness of Library Operations in collaboration with Texas A&M University is defining and Services and measuring library service quality across institutions and creating useful quality assessment tools. The Web- • Library Facilities and Space based, total-market survey measures library user • Organizational Capacity perceptions of service quality and identifies gaps In order for any specific new measures projects to between minimum, desired, and perceived expectations succeed, it was determined that there must be of service. It provides project participants with reliable, collaboration among member leaders with strong tested, comparable data for internal and external interest in them; individual projects could and should benchmarking. In addition, a number of training be developed with different models for exploration; and activities have been developed to assist libraries in there must be an intent to make resulting tools and deploying the survey, interpreting results, and 2 ARL 230/231 • OCTOBER/DECEMBER 2003 implementing changes in their operations in response to Forum, DLESE (Digital Library for Earth System the survey data. Colleen Cook and Fred Heath have led Education), and MERLOT (Multimedia Educational 8 this effort from Texas A&M and Martha Kyrillidou and Research for Learning and Online Teaching) to date. Duane Webster have led this effort from ARL. Two Developing an understanding of the dimensions of leading researchers from Texas A&M, Yvonna Lincoln digital library service quality is surfacing questions and Bruce Thompson, have served as qualitative and related to electronic services, quality, and educational quantitative experts respectively and the project’s impact, an area where relatively little theoretical and development has been well documented in the practical understanding exists to date. A report on the 6 literature. evaluation work of the NSDL appears on page 17. Two events jumpstarted the LibQUAL+™ project. Ease and Breadth of Access First, ARL held a symposium on measuring service 7 The increasing use of electronic resources, documented quality in October 2000. Experts in the field of library by expenditures reported in ARL Supplementary service quality and ARL members participating in a Statistics and confirmed by LibQUAL+™ findings each pilot project to test a library user survey based on year, was another area identified by retreat participants SERVQUAL—a service quality evaluation tool used in as a high priority for new measures. In February 2004, the private sector—spoke about the need for an another set of ARL members began the E-Metrics instrument that looked at library service from the user’s project, an effort to explore the feasibility of defining point of view. The symposium raised interest in the and collecting data on the use and value of electronic LibQUAL+™ project specifically and the issue of resources. ARL had some experience in tracking measuring library service quality generally. Also in the expenditures for electronic resources through the ARL fall of 2000, ARL and Texas A&M received a three-year Supplementary Statistics but there was recognition that grant from the U.S. Department of Education Fund for much more work was needed. The E-Metrics project the Improvement of Postsecondary Education (FIPSE) was coordinated by Project Co-Chairs Sherrie Schmidt to test the LibQUAL+™ instrument in a broad range (Arizona State) and Rush Miller (Pittsburgh). ARL of institutions. contracted with Florida State University’s Information Begun with only 12 ARL libraries in 1999, the Use Management and Policy Institute to direct a study, LibQUAL+™ project has now reached the end of its gather data, and do the necessary analysis. The project FIPSE grant and is international in scope with just over was conducted in three phases: (1) an inventory of 300 libraries participating in the spring 2003 survey. what libraries were already doing in this area and Since the project began, over 250,000 individuals have identification of any libraries that could provide best responded to the survey, creating the richest dataset of practices; (2) identification and testing of data elements information on user perception of library service quality that could be collected and used as measures of that exists. Although most of the 2003 participants were electronic resources for trend analysis and academic libraries, specialized libraries such as health benchmarking; and (3) analysis of the connection sciences and law also participated. Consortia such as between the use of electronic resources and institutional OhioLINK, the NY3Rs, and the Network of Alabama outcomes. Using surveys, consultations, field tests, Academic Libraries (NAAL) have provided financial and site visits, the project investigators identified support for their members and received reports that 16 measures in four categories (resources, use, analyze the survey data for consortia. For the first time expenditures, and digitization) along with three in 2003, the protocol was applied to a set of public performance indicators that ARL libraries might use libraries in New York; a British English version was to track electronic resource use. E-Metrics project 9 tested with United Kingdom postsecondary libraries, documents can be found on the project’s Web site and a Canadian French translation of the instrument and were also published in print. was prepared to support the participation of French A related ARL Task Force on Statistics from Canadians. Results from this year’s project are Vendor-Based Database Products focused on working summarized in an article on page 8. After 2003, with vendors to arrive at agreed-upon data element LibQUAL+™ will evolve into an ongoing service definitions and terms, specific data that can be collected, quality assessment program at ARL. and methods for reporting data to libraries. This task Additional funding from the National Science force built on existing work in the field. ARL also Foundation has been secured to adapt the LibQUAL+™ participated in international efforts culminating in early instrument for assessing digital library service quality sponsorship of Project COUNTER (Counting Online within the National Science Digital Library (NSDL) Usage of NeTworked Electronic Resources), an framework. Qualitative re-grounding is ongoing with international initiative designed to serve librarians, site visits, interviews, and focus groups held at the Math publishers, and intermediaries by facilitating the ARL 230/231 • OCTOBER/DECEMBER 2003 3 PECIAL SSUE ON EW EASURES ContinuedS I N M
10 recording and exchange of online usage statistics. associations to identify those that are working in this Release 1 of the COUNTER Code of Practice, which area, identify specific offerings that libraries could focuses on journals and databases, was agreed to and develop to meet outcomes thus far identified, establish issued in December 2002. a plan to take the offerings to faculty in a select group of After completion of the E-Metrics study, ARL institutions, explore with ACRL the offering of training participating libraries decided to continue further on information literacy skills, define and create investigation and tested the proposed data elements generalized tools for assessment, and, perhaps most again in 2002. Gordon Fretwell (University of importantly, define what is meant by learning outcomes Massachusetts) served as a Visiting Program Officer and at what level ARL should focus its attention during the first half of the year and Martha Kyrillidou (student, course, program, department, etc.). completed the data collection in the latter half. This The working group began by scanning the higher testing experience led to the decision to open the education environment for consensus on or best 2003–04 pilot process to additional participants. The practices in identifying learning outcomes, without data elements collected as part of this project then will much success. The results confirmed the importance, move into the regular ARL Supplementary Statistics but also the difficulties, of advancing their agenda: collection cycle for the year starting in July 2004. Most • National organizations had not yet agreed on a of these items relate to the number and use of electronic standard definition of learning outcomes. resources and also seek to describe emerging digital library operations. • Accrediting agencies were leaving the discussion of learning outcomes up to the Library Impact on Teaching and Learning individual campuses. For many institutions, the accreditation process has • Since definition of learning outcomes is vague, provided the strongest impetus for a shifting campus most campuses have not yet grappled with the emphasis on institutional outcomes. The retreat issue—those that have are doing it because of the participants recognized that some of the more difficult pressure of accreditation. measures to develop would identify how a library contributes to the teaching and learning process. In • Identification of the people within an order to address this issue, several ARL members organization concerned with learning provided funds to contract with Eller Distinguished outcomes is difficult. Service Professor Kenneth Smith (University of • Compilation of a common list of learning Arizona) to prepare a white paper, “New Roles and outcomes from ARL members is not yet possible. Responsibilities for the University Library: Advancing 11 • Best practices have been limited to smaller Student Learning through Outcomes Assessment.” institutions—none are of the size and complexity Smith’s paper encourages libraries to develop strategies of large research organizations. for becoming involved in campus assessment activities. • Faculty feel threatened by campus discussions He encouraged libraries to: of learning outcomes. • Move from a content view (books, subject Fortunately, within the past couple of years, some knowledge) to a competency view (what students are able to do) of these challenges are being addressed. Recent literature on higher education assessment illustrates • Understand learning outcomes of academic evidence that general education outcomes and, in some degree programs cases, outcomes for some professional programs are • Develop curriculum segments or “offerings” being developed and used. Common themes and through which the library achieves learning learning objectives have arisen that can be applied by 12 outcomes. institutions beginning to develop their own outcomes ARL subsequently established a Learning Outcomes assessment efforts. Accrediting agencies are a key Working Group under the leadership of Carla Stoffle to source in encouraging these developments. The expand on Smith’s recommendations and identify working group is preparing a white paper that measures that would describe the library’s contribution summarizes the literature on higher education to student learning outcomes. Several tasks were put assessment, examines the definitions and sources of forward to the group: survey accreditation agencies to learning outcomes, identifies domains of learning, identify best practice institutions, compile learning describes the levels of assessment, and suggests outcomes used by ARL institutions for specific frameworks for putting these pieces together within academic departments or general education an institution. See the related article on page 14 of requirements (if they exist), contact professional this issue for a summary of the white paper. 4 ARL 230/231 • OCTOBER/DECEMBER 2003 One specific recommendation to discover libraries’ the library’s contribution to research quality or contributions to the teaching and learning process is to productivity. There may be correlation between begin with a process over which libraries have control, research outcome and library support, but causation is such as instruction, information literacy, or reference. more difficult to prove. He suggested that libraries ARL has taken that approach and is now partnering identify their institution’s most productive researchers with Kent State University on Project SAILS and ensure that their information needs are met. (Standardized Assessment of Information Literacy If researcher access to information in a networked Skills), which has received partial funding from the environment contributes to their success, finding a way Institute of Museum and Library Services (IMLS) to to capture data on use of resources is one way that a develop an instrument for programmatic-level library can document its contributions. Brinley Franklin assessment of information literacy skills. SAILS is based (University of Connecticut) and Terry Plum (Simmons on outcomes defined by the ACRL Information Literacy Graduate School of Library and Information Sciences) 13 Competency Standards for Higher Education. Seven conducted patron use surveys of networked electronic libraries participated in Phase I of SAILS and met at the services at five geographically disparate academic ALA Annual Conference in June 2003 to review their health science libraries and two main libraries between test results, suggest how data could be best formatted 1999 and 2003. They developed a pop-up Web-survey for use by libraries, and share their experiences in methodology that can be exploited to identify not only administering the survey to different student groups. how library patrons use electronic services, but also to Over forty libraries are participating in Phase II of the what purpose that use was being made. This study, project, which will continue to test both the items in the called Measuring the Impact of Networked Electronic survey as well as how best to administer the instrument Services (MINES), is becoming an ARL new measures within different library and academic settings. A fuller project. See the article on page 20. description of the SAILS Project appears on page 18. Cost Effectiveness of Library Operations Library Impact on Research and Services In 2001, Douglas Jones (University of Arizona) was Libraries continually seek ways to improve the cost- appointed ARL Visiting Program Officer to identify effectiveness of their operations. Retreat participants issues and explore possible measures that demonstrate identified this area as one that needed to be addressed, the role of the library in support of the university but did not identify any specific operations to be research mission. He investigated a set of dimensions explored. There was general agreement that the 1997 15 that might lead to library success in support of the ILL/DD Performance Measures study resulted in research mission of the university. His focus was on significant changes in the operations of interlibrary loan correlating research outcomes with different types of departments across the membership. Members of the library support, identification of best practices at those Statistics and Measurement Committee were interested institutions for which comparable levels of library in a follow-up study to see whether departments had support resulted in greater productivity, and interviews become more efficient and whether costs had been with key faculty and administrators to gather insights reduced or contained. A new study, Assessing ILL/DD into how libraries might improve support for research. Services, was begun in fall 2002. This self-funded study Some of the sources for institutional rankings and updated, replicated, and expanded the 1997 ARL performance measures that might be used as indicators ILL/DD Performance Measures Study and obtained for research outcomes included U.S. News and World data on the performance of mediated and user-initiated Report rankings, The Center (University of Florida) (unmediated) interlibrary loan/document delivery Top American Research Universities, the Association operations in research, academic, and special libraries. of American Universities Data Exchange (AAUDE) A total of 75 ARL and non-ARL libraries participated. Data Warehousing Initiative, and the Integrated Tom Delaney (Colorado State University) and Bruce 14 Postsecondary Education Data System (IPEDS). In a Kingma (Syracuse University) collaborated with Mary presentation to the ARL survey coordinators in January Jackson (ARL) in the study. See page 21 for an article on 2002, Jones identified some possible research-related this project. ARL will publish a final report and sponsor outcome measures: (1) scale of research (e.g., total R&D a series of workshops to disseminate the findings. expenditures and federally sponsored R&D); (2) faculty The Statistics and Measurement program also (e.g., membership in national academies or other investigated the Technical Services Cost Study significant faculty awards indicating faculty conducted by five universities across the country. distinction); (3) students (e.g., Ph.D.s awarded, post- Using a time- and cost-recording process developed by doctoral appointments); (4) scholarly output (e.g., Dilys Morris (Chronos Group, formerly of Iowa State articles, books). Jones noted that it is difficult to isolate University), staff time on tasks performed within ARL 230/231 • OCTOBER/DECEMBER 2003 5 PECIAL SSUE ON EW EASURES ContinuedS I N M
different cost centers is tracked over several sampled thinking requires library staff to not only learn new tools, time periods. A variety of work processes can be traced but also to recognize the importance of data in making and the resulting data can be analyzed by employee decisions that will improve library services and operations. type, time, non-staff costs (such as vendors, systems, ARL and the Office of Leadership and Management utilities), etc. Software developed by this project will Services have developed a number of activities to support enable libraries to input their own data and use it for the learning needed by library staff. The LibQUAL+™ benchmarking against the collective results. program has a range of training, including introductory Along the same lines, Eileen Hitchingham (Virginia workshops, project-planning sessions, data-review Tech), with other members of the Statistics and sessions, and post-survey seminars. The program also Measurement Committee, worked on a Staff Allocation offers the Service Quality Evaluation Academy, an Project to determine if a set of spreadsheets could be intensive five-day workshop focusing on qualitative and developed into a tool for library managers to determine quantitative methods for collecting and analyzing library how staff time was being distributed across a range of service quality data. In collaboration with the ARL/OLMS functions. Two institutions, Online Lyceum, a Measuring Notre Dame and Vanderbilt, In order to change the culture, Library Service Quality will be combining the Course has been offered to all methodologies from the change what you measure. LibQUAL+™ participants and Technical Services Cost a range of libraries interested Study and the Staff Allocation Project to look at staff in the background and theory of measuring service quality allocation costs across the library. See the article by and methods to assess and improve service. The Dilys Morris on page 23. ARL/OLMS offers a two-day workshop on Creating a Culture of Assessment, which identifies the main forces Library Facilities and Space influencing current library goals and work environments While no specific projects have begun yet regarding and how these forces demand the adoption of a new measurements of library space, this area has not paradigm of thinking, new attitudes, and the support for languished. ARL and OCLC cosponsored a Strategic structures that make assessment and evaluation an integral Issues Forum, “Future Library Architecture: part of library work. See page 31 for more information. Conception, Design, and Use of Library Space” in “Toward Rethinking Academic Library Performance February 2002 in Las Vegas, Nevada. The goals of the in the Digital Age,” an ARL/OCLC Strategic Issues Forum forum were (1) to engage library leaders in an held in 2001, introduced senior library administrators to exploration of important issues affecting the future issues regarding performance measurement, and provided development of library space and (2) to help prepare an overview of the Balanced Scorecard methodology them to envision and make the case for innovative as one means for libraries to establish performance library spaces that are directly responsive to user needs management systems. The University of Virginia is one and interests on campus. In addition, ARL’s May 2003 of the first libraries to use this methodology; an article by Membership Meeting examined the connections Jim Self describing their experience is on page 28. between the concepts of research libraries as place and In 1998, the University of Arizona Library as space and how they contribute to a sense of 16 implemented a performance measurement system to community. assess itself. An article on page 26 describes Arizona’s One question retreat participants asked about experience with the Performance Effectiveness space was, “How can libraries tell if they are effectively Management System that aligns individual, unit, and utilizing their space?” Cliff Haka and Jim Hensley whole organization efforts with library strategic goals and (Michigan State University), in an article on page 24, provides measures that indicate success, progress, and the describe Michigan State University’s effort to conduct a need for improvement. The article also gives examples of space-utilization study using CAD 2000 software. They benchmarks that Arizona identified in order to provide postulate that this might be a useful tool for other other libraries with a point of departure for measuring libraries and could perhaps provide an opportunity for their own operations. benchmarking the use of space in ARL libraries. Susan Beck (Rutgers University) served as an ARL Organizational Capacity Visiting Program Officer to investigate the impact of Retreat participants acknowledged that one of the assessment on library management decision making and challenges facing research libraries is the importance of the degree to which assessment data has influenced developing an institutional culture that understands the organizational change. She studied nine public ARL need to focus on new measures (i.e., outcomes) rather institutions, selected primarily because they are involved than solely on traditional measures. One way to change in assessment activities. See the article on page 30 for a culture is to change what is measured. This shift in summary of her research. 6 ARL 230/231 • OCTOBER/DECEMBER 2003 Moving Measures into the Mainstream for specific projects. As highlighted in this article, What does it take for an institution to move into a culture individual institutions and library directors have taken of assessment? Based on experience and research over leadership for new measures projects. Their personal the last few years, the following have emerged as interest and dedication to the initiative itself or to a essential elements: particular area for investigation cannot be overstated. • Resources (i.e., time and money) They have ensured not only that projects are well defined and launched, but also have kept the • Individual and institutional buy-in enthusiasm for the work in the forefront of the ARL • Access to individuals to evaluate community. Another characteristic has been the project • Expertise to conduct evaluation nature of the initiative. As agreed to by retreat • Project management experience participants, it was evident that no single measure or set of measures would work to address all eight areas of • Appropriate benchmarks concern. By allowing a variety of projects and • Conceptual clarity approaches, the initiative was able to quickly address all • Measurement and design requirements of the areas relative to the degree of interest expressed • Instrument validity and reliability by individual project participants. Also, member libraries were not obligated to join all projects. They The ARL New Measures Initiative has worked to could choose only those that were of most interest based assure that all of these components are part of the on local situations and those choices guaranteed their projects undertaken. After several years of hard work interest and active participation. Joining a project could testing and refining measures and developing a broader mean project leadership, active participation by the base of experience and expertise among staff in libraries, director and staff, or support through funding. As it is now time to begin moving many of the “new” results of each of the project are made available, the full measures into the mainstream of library operations. ARL community as well as the larger library Moving new measures into the mainstream of the community benefits. Another aspect of success for these ARL Statistics and Measurement program does not mean projects is the opening up of some of them to the wider ARL is throwing the baby out with the bathwater. The library community. The cross-fertilization especially data ARL traditionally gathers are still important for with other academic institutions has allowed the documenting the transformation of research library projects to expand their support base and strengthened collections, services, and staff, and for identifying trends the acceptance of the tools and methodologies in the costs of operations. However, the changing nature throughout the library community. of research libraries also calls for new and different The data that libraries collected and reported in the measures that not only describe the library, but also past were primarily input data (how much do you indicate how well a library is serving its community. have). In the 1990s there was encouragement to provide There is no doubt that demands for library more output data (how much do you do). Libraries are accountability will continue. The users of library services now in an environment that asks for outcomes (what is have an insatiable appetite for quality information the benefit of your service) or impact (what difference anytime, anywhere. Parent institutions are pressured to do you make). Institutions are identifying their demonstrate their value to legislators, funding outcomes, defining criteria for success, and determining authorities, students, etc., and are asking for data from all methods to look for evidence of success. ARL has organizational units, including libraries, in order to demonstrated that it is possible to move from input to provide it. An exploding growth in the use and impact measures through its New Measures Initiative. applications of technology along with an increased As the initiative becomes a regular component of the competition of resources makes it more important than Statistics and Measurement program, its success factors ever for libraries to use hard data to make the case for can serve as elements in a model for other resources. This means libraries will have to continually organizational units to use as they look to improve their measure and monitor their effectiveness and, as own institutional effectiveness. technology and user behavior changes, strive to develop The year 2003 is transitional for the New Measures measures that have the greatest potential in helping Initiative—leadership of the Statistics and Measurement library managers lead change. Committee passed from Carla Stoffle to Brinley Franklin. The Success of the New Measures Initiative The initiative has matured and been integrated into the There are several characteristics of success of the ARL Statistics and Measurement program. For example, ARL New Measures Initiative. The first has been the has assumed full responsibility for LibQUAL+™, plans commitment by individual institutions to take leadership are in place for E-Metrics to move into the ARL ARL 230/231 • OCTOBER/DECEMBER 2003 7 PECIAL SSUE ON EW EASURES ContinuedS I N M
Supplementary Statistics, Project COUNTER is being implemented, and results of the new ILL/DD study will EASURING SER be used by libraries to improve their operations. We M U have reached a point where some measures that have SATISFACTION AND been successfully tested will be moved to the main data- ARKET ENETRATION collection program while others are dropped from ARL’s M P surveys. Other measures will continue to be tested among a larger community in a continuous cycle of LIBRARY USERS ASSESS SERVICE evaluate and deploy or drop. Ideas for new measures QUALITY WITH LIBQUAL+™ continue to be welcomed by the program. Several years AND E-QUAL ago our challenge was to develop our own new by Kaylyn Hipps, Editorial and Research Associate, ARL, measures or they would be determined by others and Martha Kyrillidou, Senior Program Officer for Statistics for us. We have met that challenge successfully and Measurement, ARL through the New Measures Initiative. ibQUAL+™ is a suite of services aimed at 1 Sarah Pritchard, “New Directions for ARL Statistics,” ARL measuring library user perceptions and Bimonthly Report, no. 161; Martha Kyrillidou and William Crowe, expectations of service quality. At the core of the “In Search of New Measures,” ARL Bimonthly Report, no. 197 (April project is a Web-based survey that seeks feedback from 1998),
The e-QUAL project will develop and test an initial set of survey questions in late 2003. A second round of A LIBQUAL+™ PRIMER Excerpted from Colleen Cook, Fred Heath, and Bruce testing is planned for spring 2004. Thompson, “LibQUAL+™ from the U.K. Perspective,” ~~~ presented at the Fifth Northumbria International Conference The LibQUAL+™ and e-QUAL team members on Performance Measurement in Libraries, Durham, England, include: July 28, 2003,
ETRICS HRONOLOGY E-Metrics Pilot Projects E-M C Acceptance of the E-Metrics project report from the 1993 82 libraries report e-resources data for ARL FSU Institute was one step in the series of ARL activities Supplementary Statistics addressing the issue of measuring electronic resources. 1996–98 Assessment of ARL Supplementary Statistics Next came a call for participation in the summer of 2002 survey data on e-resources undertaken by for ARL member libraries to continue the work of the Tim Jewell (University of Washington) as project and test the recommended measures in an ARL Visiting Program Officer E-Metrics Pilot Project during 2002–03. Thirty-five institutions responded, half of them new to the project 1999 ARL New Measures Retreat, e-resources For the 2002–03 E-Metrics Pilot Project, the identified as one of the key areas of concern participating libraries were asked to report back 105 libraries report e-resources for ARL through a structured questionnaire to Gordon Fretwell Supplementary Statistics (University of Massachusetts), who served as an ARL Visiting Program Officer. The questionnaire gathered 2000 Two-day meeting of library directors and information on what data the library was currently senior staff focuses on gathering data on collecting, how much of it could be reported for FY e-resources 2001–02, any additional statistics being collected on in- ARL E-Metrics Project launched with house electronic resources, and what types of internal 24 libraries participating; conducted in three analysis were being done with the data collected. phases over two years by Florida State Participants were also asked to report data for the first University’s (FSU) Information Use quarter of the fiscal year 2002–03 (July 2002 through Management and Policy Institute September 2003). The goal was to collect some in-depth annual data from some vendors at some libraries, and a Working Group on Vendor Statistics established wider collection of data for a briefer window of activity 2001 FSU Project team recommends 16 measures of (first quarter of current fiscal year) from a larger number e-resources and 3 measures of the use or of libraries. Variations in vendor reporting may mean performance of these resources. that data initially examined would not be strictly comparable across vendors. ARL joins Project COUNTER An in-person project meeting at the January 2003 2002 Call for participation to ARL libraries to test ALA Midwinter Meeting provided an opportunity for the measures recommended by FSU team; 35 some libraries to present and discuss the analysis techniques and specific output or products that they libraries respond and commit to collect the 8 data for 2001–02 or part of the 2002–03 have developed and utilized. As the year progressed, a academic year and compare results with number of the 2002–03 E-Metrics participants spent time vendor data organizing the collection of “database use” statistics (sessions, searches, items). The redesigned ARL Gordon Fretwell (University of Massachusetts), E-Metrics Web page includes a section where Visiting Program Officer, and Martha participating libraries’ spreadsheets can be seen as 9 Kyrillidou (ARL) collect and analyze data examples of the new E-Metrics data-collection process. Project COUNTER issues draft guidelines for Current Status of Pilot Project and Next Steps measuring usage of online information products In light of their proven utility and reliability, several data 2003 Decision reached that data being collected elements for electronic resources previously collected related to expenditures for e-resources is through the ARL Supplementary Statistics will be moving stable enough to move from Supplementary into the regular ARL Statistics data-collection activity Statistics to the main ARL Statistics as of the beginning in July 2004 with collection of data for 2003–04. 2003–04 collection cycle (July 2004) They are items related to expenditures on electronic resources: Computer Files (one-time/monographic Decision reached to move all other E-Metrics purchase); Electronic Serials; Bibliographic Utilities, data elements still being tested into the Networks, and Consortia (Library); and Bibliographic Supplementary Statistics beginning with the Utilities, Networks, and Consortia (External). Items 2003–04 collection cycle (July 2004) remaining in the ARL Supplementary Statistics that are Call for participation in the 2003–04 data continuing to be tested as part of the E-Metrics Project collection of E-metrics issued include Expenditures on Electronic indexes and reference tools and Expenditures on Electronic full-text periodicals. 12 ARL 230/231 • OCTOBER/DECEMBER 2003 the increased availability of electronic resources. In the E-METRICS BEING TESTED new measures environment, we are faced with the Patron-Accessible Electronic Resources challenge of identifying (a) new “species” of metrics and • Number of electronic full-text journals (b) new methods for measuring the effectiveness of these • Number of electronic reference sources new species. The work described above primarily focuses • Number of electronic books on the identification of new species, i.e., new data categories that will give us some general sense of the Use of Networked Resources & Related Infrastructure evolution of electronic resources in terms of input and • Number of electronic reference transactions output measures. There are also efforts underway, such as • Number of logins (sessions) to electronic databases the MINES Project described by Franklin and Plum in this • Number of queries (searches) in electronic issue and the effort to develop e-QUAL for the digital databases library environment, that are attempting to develop new • Items requested in electronic databases user-based methods for understanding the value derived • Virtual visits to library’s Web site and catalog from the use of electronic resources. ARL continues to Expenditures for Networked Resources seek and develop the best measures to demonstrate how & Related Infrastructure the provision of electronic resources contributes to library • Cost of electronic full-text journals users’ success. • Cost of electronic reference sources 1 ARL Supplementary Statistics 2001–02 (Washington, D.C.: ARL, 2003). • Cost of electronic books 2 A printed compilation of the investigator reports and the instruction • Library expenditures for bibliographic utilities, manuals are available for purchase and all the documents are on the networks, & consortia E-Metrics Web site
Definitions of “Learning Outcomes” IBRARY MPACT ON Numerous authors express the concept of student L I learning outcomes as the changes that occur within the 2 TEACHING AND student, not what the instructor does. That is, learning outcomes are “a level of knowledge, skills, abilities that EARNING 3 L a student has attained.” This emphasis on the change that takes place within the student is also reflected in a RESEARCH LIBRARY INVOLVEMENT definition developed by Battersby and embraced by IN LEARNING OUTCOMES members of the working group. ASSESSMENT PROGRAMS “Learning outcomes are the essential and by Sheila Young, Science Reference Librarian/Engineering, enduring knowledge, abilities (skills), and Arizona State University, and Julia C. Blixrud, Assistant attitudes (values, dispositions) that constitute Executive Director, External Relations, ARL the integrated learning needed by a graduate of 4 ow could research libraries participate in the a course or program.” assessment of learning outcomes at the institutional level and determine how to measure Sources of Learning Outcomes: National their contribution to the learning process? This is the Regional accrediting agencies are one source of the Hquestion now being pursued by the ARL Learning criteria upon which an institution will be evaluated. Outcomes Working Group. In 2000, Eller Distinguished Accreditation criteria have been recently revised Service Professor Kenneth Smith, University of Arizona, reflecting an increased emphasis on assessment and prepared a white paper for ARL that discussed how a institutions are beginning to gain experience with library might begin developing shared learning outcomes these new expectations. Lindauer has examined and with the larger campus community and then create analyzed documents of the accreditation agencies and “offerings” based on those shared outcomes, to integrate evaluates trends and specific text related to libraries. 1 into the curriculum. Dr. Smith recommended that Her work presents examples from these documents libraries “understand the processes that are used to and makes five observations and recommendations for define learning outcomes, to select measures, to library action. These concern the role of the library in collaborate with other academic departments, and to use information literacy, the role of the library as an the results to improve their programs.” “academic support unit,” librarian involvement in The Learning Outcomes Working Group, initially accreditation standards, promotion on campus by chaired by Carla Stoffle, Dean of Libraries at the librarians of the ACRL information literacy documents, University of Arizona, was established to follow up on and the need for librarians to stay informed about the Dr. Smith’s recommendations. The working group is current assessment projects of the ACRL, ARL, the currently led by Karen Williams, Team Leader for Association of American Higher Education, and other 5 Undergraduate Services, University of Arizona Library. organizations involved with assessment. As with other New Measures Initiative projects, the Discipline-specific documents are an additional working group was charged to identify existing measures source of learning outcomes that are valued by that could be built upon and applied in libraries. A scan departments across campuses. On a national level, of the higher education environment found that, while the American Chemical Society, the American the emphasis on establishing learning outcomes and Psychological Association, and the Accreditation Board assessing those outcomes has gained momentum in for Engineering and Technology Inc. have all developed recent years, campuses are still in various stages of such statements. The Council for Higher Education grappling with this issue. Given this, the working group Accreditation (CHEA) lists a directory of “Specialized decided to develop their own white paper that would and Professional Accrediting Organizations” that are 6 describe this environment and, using it as a foundation either recognized or will undergo a review by them. for understanding the current state of research and The accreditation documents of these professional implementation in this field, decide how to take Dr. organizations periodically undergo revision and this Smith’s recommendations a step further. may provide librarians an opportunity to contribute The working group’s white paper, now in draft, to the new standards. examines the definitions and sources of learning Another source of learning outcomes from a outcomes, domains of learning, levels of assessment, national body is a 1995 report from the U.S. Department and frameworks suggested for putting these pieces of Education, Office of Educational Research and together within an institution. What follows is a Improvement: “National Assessment of College Student summary of the working group white paper. Learning: Identifying College Graduates’ Essential Skills 14 ARL 230/231 • OCTOBER/DECEMBER 2003 in Writing, Speech and Listening, and Critical Thinking: assessment illustrates that some common themes and 7 Final Project Report.” The report identifies “higher learning objectives are arising among the general order communication and thinking skills that college educational outcomes being tested and used. A key graduates should achieve” and describes the skills of resource for monitoring developments in this rapidly analyzing, locating, and evaluating information as evolving area is Assessment Update, a journal edited by dimensions of critical thinking. Trudy W. Banta, Vice Chancellor for Planning and Within the library community, ACRL has produced Institutional Improvement at Indiana University- 12 information literacy and competency standards for Purdue University at Indianapolis. 8 9 higher education and a standards tool kit that lists In order to assess how library programs and the outcomes for each standard. In addition, an ACRL services contribute to student learning outcomes at the “Best Practices” initiative has student, department, or developed “Characteristics of Recent literature on higher education institutional level, librarians Programs of Information need to inform themselves on Literacy that Illustrate Best assessment illustrates that some common how learning outcomes Practices: A Guideline,” which themes and learning objectives are assessment is practiced on their was approved by the ACRL own campus. In addition, the 10 arising among the general educational Board in 2003. All of this work outcomes being tested and used. contributions of the library to is useful as librarians examine learning outcomes could be the needs and challenges that described in a broader context are particular to their own institutions. of the students’ experiences with the library; not just Sources of Learning Outcomes: Local those limited to the instruction activity. Since the library has multiple roles such as service, Statements by institutional governing boards indicate support for research, and instruction, these roles may valued outcomes, including learning outcomes, if need to be addressed separately and by different means present. Examining the documents produced by the of assessment. For example, the LibQUAL+™ project governing board, institution (e.g., the mission and the addresses assessment of the quality of service provided strategic plan), and the individual colleges, by a library while the Assessing ILL/DD project, as well departments, and programs will indicate those learning as the MINES methodology proposed to document outcomes that are of greatest importance. usage patterns of networked electronic services, Integration of Learning Outcomes both contribute to an assessment of library support through All Levels for research. Writing learning outcomes that are internally consistent In the movement toward learner-centered with the department or discipline, the institution, state, education, the emphasis on the student and on learning and accrediting agency requires an examination of outcomes is also growing in importance as part of the documents from all these levels. Huba describes this evaluation of academic programs. Evidence of the 11 process of integration as a triangular process. She contribution of the library to student learning outcomes describes the relationship of a “design backward” is therefore an important aspect of demonstrating the (from the broadest learning outcome of the institution value of the library to academic programs and the or discipline to the most specific learning outcomes of institution. an individual lesson) and the “deliver forward” Measures experience (from the students’ perspective). Since Developing measures to document how the library many institutions are in various stages of developing contributes to the learning outcomes at the department learning outcomes at each level, librarians have many or institutional level has been challenging. Focusing on opportunities to participate in the process of integrating the course level, a project by ACRL resulted in a report learning outcomes through all levels of the institution. on how librarians in 23 institutions are working with Assessment faculty in assessing information literacy learning 13 Assessing learning outcomes is an exceptionally outcomes. In a study to evaluate an information complex process and, to guide the process, certain literacy program, librarians who worked with faculty in fundamental questions need to be answered: the the English department noted that the collaboration purpose, the content, and the level of assessment all with the English faculty increased awareness that give direction to the development of assessment “information literacy standards are remarkably similar measures. A number of models for assessing learning to the standards of critical thinking they try to impart in 14 outcomes are being developed and tested on different freshman English.” This recognition of the campuses. Recent literature on higher education relationship of critical thinking and information literacy ARL 230/231 • OCTOBER/DECEMBER 2003 15 PECIAL SSUE ON EW EASURES ContinuedS I N M
provides a foundation upon which to build mutual scores. The study also found that “the greater the usage learning outcomes. of the library resources, the greater the Breadth and Depth 17 Measures can be categorized as direct or indirect of the student’s responses to the scenarios.” and within each of these groups, the measures can be The SAILS Project, at Kent State University and ARL, quantitative or qualitative. Indirect measures may not seeks to develop a standardized test “to measure be adequate alone, but may enrich or illuminate aspects information literacy skills, gather national data, provide of what direct measures say norms, and compare information about levels of academic literacy measures with other achievement. In order to assess how library programs indicators of student One indirect measure that and services contribute to student learning achievement.” The instrument, has been used by institutions is outcomes at the student, department, or based on the ACRL standards, is the College Student Experiences institutional level, librarians need to “for programmatic level Questionnaire (CSEQ). This assessment of information literacy survey asks students to indicate inform themselves on how learning skills that is valid and thus the amount of time they spend outcomes assessment is practiced credible to university engaged in various activities administrators and other on their own campus. 18 that affect learning. It includes academic personnel.” It is eight questions regarding possible that the SAILS instrument library use. Analysis of the responses shows that could be used as a direct measure of information skills “focused library activities” have a “significant” impact development. It would then be necessary to link this to the 15 on self-reported development of critical thinking. learning outcomes at the department, program, or 19 Others found that “students who perceive that their institution level. campus emphasizes information literacy gain more in 16 Performance Indicators this area, net of other influences.” The same Lindauer identifies several performance indicators that researchers also found that the “library experiences of might be used as measures. For example, undergraduates are related to select educational purposeful activities” and that the library “appears to • “Perceptions of recent graduates about their be a positive learning environment.” However, the information literacy skills training/experience from results also indicated “the library use does not appear to undergraduate study contributes to their success in directly contribute to gains in information literacy and graduate/professional programs.” other desirable outcomes.” The authors discuss the • “Success in applying information literacy skills on following possible explanations: CSEQ may not be the the job as perceived by alumni and employers.” best measure for information literacy, “lack of baseline The same author suggests that “libraries could measures” and “a variety of experiences during strengthen their evidence by triangulating data and college…contribute to gains and satisfaction, not just supporting information. They could group such measures one type of experience.” as selective qualitative descriptions of resources and At the system level, the California State University services with usage data for specific services and with has launched a three-phase assessment of information user satisfaction and perceptions of benefit from survey literacy skills of the students at all 23 campuses. This findings. All this could be grouped to show the impact of 20 project uses both qualitative and quantitative measures library and information resources and services…” and employs multiple methods of assessment. One part of the inquiry includes questions related to information Conclusion and Next Steps scenarios as well as sets of questions scaled to areas The Learning Outcomes Working Group white paper related to information competence “such as library use, brings together more detail about these and other research process skills, achievement, presence of resources on the state of assessment of learning outcomes reference materials in the home, and computer and especially as they relate to libraries. The goal is that this media literacy.” The “information scenarios were environmental scan of available research and experience constructed to elicit evidence of information will enable the working group to focus on identifying the competence when applied to general knowledge, rather most promising mechanisms or instruments that would be than discipline-based knowledge.” Responses are useful in demonstrating the research library’s direct or measured in terms of “Breadth” (number of different indirect impact on learning outcomes at the department or ideas) and “Depth” (number of discrete ideas). In a institutional level. A draft of the white paper will be preliminary report, the author finds that the students available on the ARL Web site by the end of the calendar that self-reported difficulty finding and using year. For more information, contact Sheila Young information tended to have lower Breadth and Depth
1 Lee Zia, “The NSF National Science, Technology, Engineering, and Mathematics Education Digital Library (NSDL) Program: New Projects and a Progress Report” D-Lib Magazine 7, no. 11 (November 2001),
ROJECT TANDARDIZED able to document information literacy skill levels, P SAILS: S establish internal and peer benchmarks of ASSESSMENT OF INFORMATION performance, pinpoint areas for improvement, LITERACY SKILLS identify and justify resource needs, and assess and by Julia C. Blixrud, Assistant Executive Director, External demonstrate effect of changes in their instructional Relations, ARL programs. nformation literacy is the ability to “recognize Using the Wisconsin Ohio Reference Evaluation when information is needed and…locate, evaluate, Project (WOREP) as inspiration, Lisa O’Connor, 1 and use effectively the needed information.” This Instructional Services Coordinator; Carolyn Radcliff, skill set is developed through information literacy Head of Reference Services; and Julie Gedeon, Iprograms operated by libraries primarily, but not Manager of Academic Technology Services Evaluation exclusively, for undergraduates at Kent State University in academic institutions. Are Libraries & Media Services these programs successful? Do … if university administrators were to began using the process of they advance the skills of be persuaded to allocate resources systematic instructional students? How can we measure necessary to grow the information design to develop such a tool in 1998. They chose to use these programs’ success? Kent literacy program, librarians must be State University and ARL have item response theory (IRT) partnered on Project SAILS prepared to substantiate the claim that as the measurement model (Standardized Assessment of these skills indeed make a difference on which to create a new 2 Information Literacy Skills), institutionally. instrument. which is developing an A set of questions of instrument for programmatic varying difficulty level is level assessment of information literacy skills that is being produced to measure information literacy. valid and thus credible to university administrators Each question addresses one of the model learning and other academic personnel. outcomes as identified by the ACRL Information 3 Several years ago, librarians at Kent State Literacy Competency Standards for Higher Education. University Libraries & Media Services realized that, Use of these outcomes will maximize the applicability if university administrators were to be persuaded to of the instrument to a wide variety of academic allocate resources necessary to grow the information institutions for internal and external benchmarking. literacy program, librarians must be prepared to The questions are tested with individual students, in substantiate the claim that these skills indeed make a small groups, and in field trials (which closely emulate difference institutionally. They must be able to answer actual survey administrations). such questions as: What are students’ entry skills While the SAILS survey is intended to be upon admission to the university, and is there a delivered in electronic form over the Web, it can be significant change in skill levels from the freshman delivered using paper and pencil. Using the Web, a year to graduation? If there are significant changes random set of about 45 questions drawn from the in students’ information literacy skills, do those skills instrument’s data bank of now approximately 150 then have any correlation to student’s academic items are delivered at the time the test is administered success and retention? The librarians conducted a to a student. Project participants can choose the most thorough search of the library literature and appropriate testing population for their institution determined that the profession was not yet in a as long as the students are undergraduates. position to agree upon the best method for assessing Institutions can test students in one of three information literacy skills, let alone assert that those situations: skills make a difference. • Highly monitored: The test-taking is proctored Librarians need a tool to measure information and the procedures are consistent across all literacy that could be standardized, is proven to be students. Students check in and out of a valid and reliable, contain items not specific to a specified testing site and the proctor is aware of particular institution or library but assesses at the the testing. A variation of this is when students institutional level, can be administered easily, and take the test as a group during class time. provides for both external and internal benchmarking. With such a tool, a library could measure information • Loosely monitored: Students come to a specified literacy skills, gather national data, provide norms, testing site, but do not check in or out. and compare information literacy measures with other • Unmonitored: Students take the test anytime indicators of student achievement. Libraries would be and from anywhere. 18 ARL 230/231 • OCTOBER/DECEMBER 2003 The answers are collected and analyzed using Phase II covers the fall 2003 and spring 2004 WINSTEPS, a modeling program created by researchers semesters and over 40 institutions have agreed to at the Mesa Institute at the University of Chicago. participate. The Kent State University staff received a grant in Phase III is scheduled for fall 2004 and spring 2005, fall 2002 from the Institute of Museum and Library with a projected participation of 100 libraries. A call for Services (IMLS) for Project SAILS. The three-year grant participation for Phase III will be issued in January 2004. is supporting continued development of the instrument The rapidity by which institutions signed up for and testing at other institutions. It will also enable Project SAILS and the enthusiasm for participation project staff to create the foundation of a results signals that libraries are looking for tools that can help database that will address internal and external them determine the effectiveness of their information benchmarking. The grant also includes funding for the literacy programs. It is another example of how the commissioning of three discipline-specific assessment leadership of one library working collaboratively with modules, which will serve as a model for future other interested libraries can create a useful, development of other discipline-specific modules. standardized tool to benefit the larger library Since there is significant overlap between student community. learning outcomes and information literacy, ARL’s More information on Project SAILS can be found at Learning Outcomes Working Group recommended that
LIBRARY IMPACT FIRST PAGE OF SURVEY ON RESEARCH Networked Electronic Services Library User Survey
OCUMENTING SAGE ATTERNS OF This survey is being conducted by the University in order D U P to assess usage of the Library’s electronic services.
NETWORKED ELECTRONIC SERVICES All responses are confidential. by Brinley Franklin, Director of Libraries, University of Connecticut, and Terry Plum, Assistant Professor, Simmons After completing the survey, you will be connected to the service you selected. Graduate School of Library and Information Sciences new initiative now being considered by the ARL Thank you for your help. Statistics & Measurement Committee is a Web- based survey that a library could administer to Patron Status: Select Patron Status collect reliable data on the usage of networked Affiliation: Select Affiliation Aelectronic services from both within the library and remotely through the Web. The idea is to build on the Location: Select Location experience of studies conducted at five academic health science libraries and two main libraries between 1999 and 2003. In those studies, more than 15,000 library Purpose A. Sponsored (Funded) users were surveyed as they accessed their libraries’ for Using Research —Definition networked electronic services. The survey documented Online why patrons were using electronic resources in those Resources: B. Instruction/Education/ Departmental Research libraries and tracked the differences between in-house —Definition and Web usage, comparing the location of the user and C. Patient Care their status (graduate student, faculty, undergraduate, —Definition etc.) with the purpose of the use. D. All Other Activities The methodology measured both in-house and —Definition remote usage in a way that allowed the costs of providing networked electronic services to be distributed among the university’s primary functions. Please fill out Status, Affiliation, Location, and Purpose Fields. In this study, those functions were defined in four categories: sponsored (funded) research, instruction/ education/departmental research, patient care, and Submit all other activities. The survey is a pop-up screen Response administered randomly to users of networked electronic services throughout the year using a statistically valid sampling plan. The study, now called “Measuring the Impact of Networked Electronic Services” (or MINES), was designed and undertaken by the authors. The and passed on the request. Such an arrangement researchers concluded that although the computing prevents lost data due to bookmarks, non-library Web environments and survey implementations in each pages, and other non-library routes to access library library were different, the methodology and overall networked electronic services. approach was workable at each library. As a result, Initial use of the methodology produced a number they are confident that the data collected are somewhat of findings useful for guiding decisions on how to comparable, recognizing that there will always be manage networked electronic resources. For example, distinguishing characteristics driven by the local user results from the first four academic health science population and institutional characteristics. libraries showed that: Experience also led the researchers to conclude that the most effective means to meet the Web survey • There were approximately four remote criteria was to run all access for networked electronic networked electronic users for every in-house resources through a gateway that authenticated access user. 20 ARL 230/231 • OCTOBER/DECEMBER 2003 • Remote users were demographically different from in-library users; the 4-1 ratio was even OST FFECTIVENESS OF higher for the category of users who identified C E themselves as faculty/staff/research fellows. LIBRARY OPERATIONS • The purpose of use reported for remote usage of AND SERVICES networked electronic resources was significantly different from in-house usage. For example, those using networked electronic resources to ASSESSING ILL/DD SERVICES STUDY: conduct sponsored research do so most INITIAL OBSERVATIONS intensively from on-campus but not from in the by Mary E. Jackson, Director of Collections and Access library. Those using these services for Programs, ARL instruction/education/non-sponsored research hat are the characteristics of a cost-effective were more likely to be in the library than people interlibrary loan (ILL) and document delivery using the resources for other purposes. (DD) operation? To get a clearer answer to this question, ARL has undertaken a third study of the • Patrons using electronic services for sponsored performance of ILL/DD operations in North American research represented 34% of the usage in the W libraries. The current study, Assessing ILL/DD health science libraries, as compared to 16% Services, collected data on the 2001–02 performance of in the two main libraries. mediated and user-initiated ILL/DD operations in The study also found that, despite proxy servers, 72 research, academic, and special libraries. modem pools, and similar remote enabling services, New to this study is a focus on user-initiated, the patrons being surveyed went to the university to use or unmediated, ILL services. User-initiated ILL is their library’s networked services. That is, when they defined, for this study, as requests that are initiated used these services they were either on campus or in without ILL staff involvement and received directly by the library many more times than off-campus. a potential supplier. This is the first comprehensive However, the users’ purposes for using electronic study of a range of user-initiated services: INNReach, resources ranged significantly among the four libraries, URSA, Loansome Doc, ILLINET Online, RAPID, and underscoring the need to consider local conditions user-initiated use of commercial document delivery as libraries have different characteristics and user suppliers. In addition, the project collected data on the populations. performance of local document delivery services in The authors concluded that MINES’s Web-based 25 libraries. survey methodology employed at the seven libraries The project measured unit cost, fill rate, and could serve as a model for similar user studies at other turnaround time for mediated and for user-initiated libraries. The discussions now underway within the interlibrary borrowing and lending. Cost data includes ARL Statistics and Measurement Committee focus on the costs for staff, network/communications, delivery, how to take a next step to apply this methodology to photocopying, supplies, equipment, and borrowing fees. the networked electronic services offered by a more Turnaround time for borrowing, as in the previous complete complement of research libraries. study, was measured in calendar days from the date the A full description of the research project and its user submitted the request to the date ILL staff notified findings was first delivered at the Northumbria Lite the user that the item was available for pickup (or conference, part of IFLA in Glasgow in 2002. See mailed the item to the user). Turnaround time for Brinley Franklin and Terry Plum, “Networked lending, a measure not included in the previous study, Electronic Services Usage Patterns at Four Academic captures the elapsed time between a library receiving Health Sciences Libraries,” Performance Measurement and and shipping a request. Metrics 3, no. 3 (2002): 123–133,
ASSESSING ILL/DD SERVICES STUDY RESULTS FOR ARL LIBRARY PARTICIPANTS, MEAN VALUES
Borrowing Lending 2002 1996 2002 1996
Unit cost, Mediated $17.50 $18.35 $9.28 $9.48
Unit cost, $2.04–$14.70 NA $3.47–$12.06 NA User-initiated
Fill rate, Mediated 86% 85% 58% 58%
Fill rate, 84–91% NA 76%–88% NA User-initiated
Turnaround time, 9.29 days (loans) 16.9 days (loans) 1.75 days (loans) NA Mediated & 6.23 days (copies) & 14.9 days (copies) & 1.37 days (copies)
Turnaround time, 2.30–8.23 days NA 0.57–1.45 days NA User-initiated (loans)
information on 1991 costs incurred by research libraries costs, higher fill rates, and faster turnaround times than 1 for ILL transactions. That study found that research mediated ILL. libraries spent an average borrowing unit cost of $18.62 Mediated borrowing has improved compared to the and lending unit cost of $10.93. 1996 study. The current study reports a unit cost for In 1995, ARL conducted a second study that built on mediated borrowing of $17.50 in 2002 dollars. When the 1992 study by measuring not only the cost but also adjusted for inflation, that unit cost would represent several performance attributes of ILL/DD operations in $14.88 in 1996 dollars, compared with $18.35 in the research libraries and, for comparison purposes, in previous study and represents a 19% reduction in 2 college libraries. This study found that research libraries constant dollars. Staff costs now represent less than spent an average of $18.35 on a borrowing transaction, 60% of the unit cost. Photocopy as a percentage of the $9.48 on a lending transaction, and took an average 15.6 overall lending unit cost has decreased and borrowing calendar days to receive a borrowing request. fees now represent a larger percentage of the borrowing unit cost. The Current Results Turnaround time for mediated borrowing has The accompanying chart presents the current findings decreased as has the volume of mediated borrowing and for the 59 participating ARL libraries and compares the lending compared with the 1996 study. results with those for the 97 ARL participants in the Bruce Kingma, Associate Dean, School of Information 1996 study. It is important to remember that different Studies, Syracuse University, and Tom Delaney, Head of ILL libraries participated in both studies, which may result at Colorado State University, are collaborating with Mary in discrepancies when comparing the two different Jackson, ARL, in the conduct and analysis of this study. For studies. Additional analysis will be undertaken to more information, contact Mary Jackson
files for the Main Library facility. This was done by IBRARY ACILITIES employing CAD 2000 software. L F CAD files containing the basic layout of the library AND SPACE building were obtained from the Physical Plant Engineering Department. Unfortunately, these files proved to be incomplete and in some cases inaccurate. Therefore it was necessary to verify dimensions by re-measuring many EVALUATING THE UTILIZATION OF areas before proceeding. Following the establishment of an FACILITIES IN ARL LIBRARIES accurate interior, additional elements such as stacks, service by Clifford H. Haka, Director of Libraries, and Jim Hensley, points, and furniture were added to the drawings. (As an Facilities Manager, Michigan State University Libraries aside, once completed these documents have proven to be ow can libraries tell if they are effectively utilizing extremely useful for space-planning projects, especially their space? Is there a way to measure how we when units move to new locations. Precise plans can be currently utilize the physical space in research developed, reviewed, and modified without the tedious libraries? collect data and compare it across libraries? and and time-consuming sketching and re-sketching, often Hthen assess the impact of various space allocation decisions inaccurately, that was the previous norm.) on library activity such as building use and circulation? The next step was to define different types of spaces For example, two libraries might pursue radically found within the typical library. Two primary resources 1 different approaches to shelving materials, with one using were used, starting with Metcalf. Using guidelines traditional shelving while the other utilizes compact contained within Metcalf, assignable and non-assignable shelving. This would result in the allocation of space for space was identified throughout the building. The shelving in one library being much higher than in the identification of non-assignable space—such as mechanical, other library, and enable us to assess if this has an impact custodial, and telecommunication areas—is an important on use, such as the total number of circulations. Similar first step as these spaces cannot typically be used, but they examples might include the extent to which libraries still make up a significant percentage of the overall square subscribe to electronic journals, thereby limiting the footage of all buildings. The next step was to discern what space required for storage of physical journal titles, functions had been assigned to the remaining space. To or the degree of outsourcing, which would decrease help with these definitions, a second resource, published by 2 the amount of space required for staff operations. the Buildings and Equipment Section of LAMA, was No single measure, such as the examples suggested utilized, resulting in the following seven categories: above, would allow libraries to respond to the overall question of the future role of physical library facilities. • Non-Assignable: Areas designated for other uses— Such measures should, however, provide insight regard- structural elements, HVAC chases, electrical vaults, ing those factors that encourage, or possibly discourage, telecommunication closets, custodial areas, etc. the utilization and subsequent effectiveness of library • Service: All areas that are points of contact between facilities. staff and patrons, or are areas for patron self-service While a review of space utilization along these lines such as copy machines and telephones. might not fit the classic definition of a “performance • Staff Area: Office space, work areas, and facilities- measure,” this topic was, in fact, identified as an area of support areas such as loading dock and mailroom. interest during the initial ARL-sponsored New Measures • Stacks: All areas designated for storage of books Retreat held in Tucson in 1999. Could procedures be and materials regardless of format. For instance, developed that might assist in the assessment of current microfilm cabinets fall within this category. space utilization within ARL libraries? No “ready-made tool” exists that would enable a library to review this • Seating: All seating without network capability. matter objectively and expeditiously. However, many at • Networked Seating: All other seating that has the retreat felt that it would be interesting, and potentially network capability. useful, to conduct a detailed review of space utilization • Traffic: These areas are used primarily for within several ARL libraries to determine if trends, or movement around the building and for no other possibly of more interest, abnormalities, exist in regard to purpose. Main corridors and hallways to elevators, the manner in which ARL libraries use their space. service areas, and restrooms were included. Library Space Evaluation at It would certainly be possible to divide many if not all Michigan State University of these categories into countless subdivisions. However, To get this ball rolling, the Michigan State University every division requires further definition and increases the (MSU) Libraries undertook to establish detailed CAD complexity of compiling the data for a library. Therefore, 24 ARL 230/231 • OCTOBER/DECEMBER 2003 in the hope that other ARL libraries might be willing to replicate these efforts to provide comparative data, we MSU SPACE UTILIZATION, strove for a limited number of straightforward and SEATING BY TYPE unambiguous categories. Seating/special format (microfilm) 0.44% Seating/computer Seating serves to depict this dilemma. Differentiating 10% between carrel seating, table seating, and “casual” seating Seating/carrel/ netlink is not that difficult. When one adds the element of 9% “laptop-ready seating” however, which cuts across all Seating/group Seating/fac/ 42% grad/carrel/ these seating types, it was decided, at least for the netlink purposes of coding the MSU floor plans, to collapse 3% seating types into networked and all other seating. It was Seating/ a/v subsequently decided to depict additional detail via other 2% techniques. For example, a pie chart was developed to display a more detailed range of seating options, as well as their relative availability (see accompanying chart). To use such detailed distinctions when coding floor plans Seating/carrel would, however, yield an indecipherable maze that 25% would be exceptionally daunting and time consuming to maintain with any degree of accuracy. Hence, we Seating/casual 9% pursued fewer categories for the overall coding of the floor plans with detail represented via alternative charts. Chart created by Jin-hee Kim, MSU Graduate Student It would seem that delineating and marking the plans into these seven categories would be relatively straightforward at this point. This proved not to be the of a space?” Calculations for book-stack capacity would be similarly instructive and potentially useful in case. This part of the project was more time-consuming argumentation for additional space. than anticipated. For example, defining spaces for The creation of the CAD drawings was the most seating that are adjacent to walkways is difficult to complex and time-consuming portion of this project. measure in that the chair push-back space from the table However, the hours spent on this task have paid huge or carrel frequently encroaches into the area defined as a dividends for space-planning discussions. The process walkway. Specifically the push-back from a 30-inch work is, of course, ongoing as staff and users move and surface carrel was defined in the LAMA document as 4 ft. relocate chairs, seating, and other items. 6 in., a significant amount of floor space that frequently The MSU Libraries plan to update and review these spilled into walkway space. In a very real sense this was documents on a regular basis. We hope that we can the confirmation that original space allowances had been engage one or more ARL libraries to become partners discarded, at least in this instance at the MSU Library. with us to further define this process. Our goal is to This space was then defined as dual purpose and it create a tool that will prove helpful for comparative became an issue to address. In this particular case it was purposes and to assess the impact of various space decided that the hallway space took precedence and was allocation decisions on library activity. If a group of measured as such. However, there were many such ARL libraries can be identified to compile comparable measurement decisions that made “fine tuning” of our data, this could provide “benchmark measures” against CAD document somewhat challenging. Listing all of the which other libraries could compare their situations. various definitions and/or conventions we considered We look forward to this possibility. too exhaustive for the purposes of this paper. However, most are found in the two sources cited. 3 —Copyright 2003 Clifford H. Haka and Jim Hensley Upon completion of floor plan coding, it was relatively straightforward to produce charts and graphs 1 Keyes D. Metcalf’s Planning Academic and Research Library Buildings to depict specific aspects of building use. In turn, this (New York, McGraw-Hill: [1965]) has been employed in countless library construction projects over the past 35 years. enabled the calculation of average space allocations for 2 American Library Association, Library Administration & items such as seating. For example, one calculation made Management Association, Buildings and Equipment Section, across all types of seating found that an average of 31 Functional Space Requirements Committee, Building Blocks for Planning Functional Library Space (Lanham, Md.: Scarecrow Press, square feet per seat has been provided, which falls well 2001). within the standard established by Metcalf. Information 3 MSU’s coded floor plan and samples charts and graphs produced such as this can prove useful in considering issues such with the CAD software are available on the ARL Web site as, “Have too many seats been introduced into too small
Creating this culture can ensure that there is a RGANIZATIONAL commitment to continuing assessment, development of O new tools, and use of data for continuous improvement. CAPACITY Without an emphasis on culture change, barriers will remain to the full acceptance of the use of performance measures. PEMS: How Does It Work? CREATING A CULTURE OF The PEMS system requires units to continuously assess ASSESSMENT: THE UNIVERSITY OF needs of their assigned faculty and student groups. ARIZONA EXPERIENCE Using varying techniques, teams engage in discovering by Carla Stoffle, Director of Libraries, and Shelley Phipps, what is most important to their customer groups about Assistant Dean for Team and Organizational Development, the services they provide. They use this information to University of Arizona Library formulate standards or performance targets for the particular service activity. They gather data on progress Especially given her leadership role in launching the toward these standards and report periodically to the ARL New Measures Initiative, it is not surprising that Library Cabinet and Strategic Planning Team. Carla Stoffle has led the University of Arizona Library to be After analyzing the needs and assessing where an early adapter of performance measures. The article below improvement can make a difference, the teams either describes how one research library is assessing itself by engage in specific projects to increase their effectiveness implementing a performance measurement system and or assign individuals the responsibility for increasing integrating the results of the New Measures Initiative the amount or quality of their individual work that projects as they become available. The benchmarks identified results in outcomes for students and faculty. Thus the and described by Arizona are made public to provide other system supports individual goal setting that furthers the libraries with a point of departure for measuring their own unit’s service effectiveness related to what customers operations. This article is reprinted with permission from have identified as important. Strategic, future focus is the authors and the Editor of Library Issues, where it structured into the system as the teams start the year’s appeared in the March 2003 issue, vol. 23, no. 4. planning by understanding their role in achieving the n addition to the national efforts that have set Library’s three to five-year strategic goals. a new direction for performance measures in As teams create their strategic framework for the academic research libraries, individual institutions year, they set outcome, output and quality measures to are experimenting with using assessment techniques assess success. In some cases outcomes are measured to quickly learn about changes in local users’ directly with customers. In other cases, needs I assessments lead to the creation of output or quality expectations. The importance of implementing an organizational performance measurement system has measures that can be deduced from the identified need. been one conclusion of such experiments. One For example, if a team assesses that only a limited institution, the University of Arizona Library, created portion of its potential user group is using its services, a Performance Effectiveness Management System they may set a standard to increase the number of (PEMS) in 1998. instructional sessions or increase the number of PEMS is designed to align individual, unit, and students reached. If they recognize that limited budgets whole organization efforts with strategic goals set by require selection of materials that have a high potential the Library and to provide measures that indicate for usage, they may set a standard that measures actual success, progress, and the need for improvement. usage of material purchased. If they identify that Developing a system approach to measurement helps timeliness is an important aspect of the service they develop an internal culture of assessment where give, they may set a standard for turnaround time, from decisions are guided by facts, research and analysis, and request to delivery. If they identify that accuracy of where services are planned and delivered to maximize information is critical to the users of their service they positive outcomes for customers and stakeholders. may set a related standard. Staff are encouraged: Examples. Examples of outcome measures may include: 95 percent of the serials acquired in • to care about what results they produce; science/engineering areas are used at least once each • to value the actual impact they have on the year. educational or research process; and Some standards use increased output as surrogate • to know how these results relate to user measures for an intended outcome. For example, 95 expectations. percent of UA faculty will receive information about 26 ARL 230/231 • OCTOBER/DECEMBER 2003 intellectual property rights and copyright (the intended Access to electronic information is indicated as highly outcome is that faculty are able to use this information desired by LibQUAL+™ respondents and confirmed by in their teaching and publication efforts). team interviews and surveys. This information also Some standards relate to the user’s need for access contributed to the setting of the standards in the Strategic or service in a timely fashion or a need for accuracy that Long Range Plan and influenced the UA Library’s will facilitate access: 90 percent of traditional reserve decision to invest a larger portion of its information requests at all sites will be available to customers within access budget in electronic information resources. 48 hours after receiving the request. Evaluation of the University’s budgetary situation Assessment with users has led to process has led to plans to increase consortial purchases as a way improvement efforts that have decreased cycle time, of reducing overall costs to the University and a standard increased amount and quality of service, and saved was set to save at least $100,000 per year on such hundreds of thousands of dollars that have been purchases. The UA Library will be joining the SAILS reallocated to the purchase and project to research more efficient refresh of technology and the ways of measuring learning implementation of a competitive …teams engage in discovering what is outcomes. As the Library builds salary structure. most important to their customer groups its new technological Learning the tools and infrastructure, we will be using methods for assessing about the services they provide. the research from the E-Metrics performance has been a project to assess the challenge. The ARL New effectiveness of our electronic Measures Initiative has afforded the opportunity to use access systems—from the customer point of view and in more reliable measurement methods that will also yield comparison with peers involved in this national effort. peer benchmarking data. The Library’s Strategic Plan Fulfilling our Educational Mission now incorporates measures that can be derived from the The PEMS system and its ability to use the results of LibQUAL+™ instrument as well as measures based on national initiatives such as the “New Measures team data gathering. For example: Initiatives,” enables the Library to demonstrate to the • Degree to which access to electronic information campus that all resources and staff efforts will be focused exceeds customers’ minimum expectations (from on the changing expectations of our users by providing LibQUAL+™): 15 percent increase within 5 value-added services. Incorporating efficiency measures years. and developing process improvement projects also • Scholarly electronic resources newly developed demonstrates the Library’s commitment to maximizing by the Library and available remotely to return on investment. Accountability is demonstrated at customers (120 new resources within the next 5 the institution, unit and individual level. years). Use of performance measurement is a way of ensuring that academic libraries retain their ability to Using Data for Strategic Decision Making perform their special mission within the educational Information from the first pilot years of LibQUAL+™ process. These include: indicated that users desired delivery of electronic • Valuing freedom of access to information, information to the desktop and that the Library was not • Eliminating censorship, reaching even minimum expectations. A strategic decision was made to create a Document Delivery Team • Providing equitable access to all levels of users, and set standards for electronic delivery of interlibrary- and loaned material and reserve articles. Cost information • Increasing the information literacy of the students from the ILL/DD project was used to research best and faculty in this increasingly complex scholarly practices and to join with other libraries using more communication system. efficient technology. This enabled customers to order Developing systems and using evolving tools created interlibrary loans directly from other libraries, which by ARL-sponsored projects and others, increases the reduced turnaround time. A systems analysis project ability to measure and communicate performance team was formed to research the capability of present outcomes and the effect of libraries on the quality of systems to provide the infrastructure necessary to teaching and learning processes in our institutions. improve access in the future. As a result, the UA Library is now participating in the development of the —Copyright 2003 Mountainside Publishing Co., Inc. “Scholar’s Portal” software with a number of ARL Library partners. ARL 230/231 • OCTOBER/DECEMBER 2003 27 PECIAL SSUE ON EW EASURES ContinuedS I N M
SING ATA TO AKE HOICES To provide resources and services that have a high U D M C : ratio of value to cost, the library has established a metric THE BALANCED SCORECARD AT THE for Expenditures for Circulating Monographs as a Proportion UNIVERSITY OF VIRGINIA LIBRARY of All Monographic Expenditures: by Jim Self, Director of Management Information Services, Target1: Monographs that circulate within the first University of Virginia Library two years should account for at least 60% of the n early 2001 the University of Virginia (U.Va.) Library monographic budget. began implementation of the Balanced Scorecard. Target2: The circulated items should account for at This initiative continues a long history of collecting least 50% of the monographic budget. and using data to improve services, collections, and processes at the U.Va. Library. Method: A yearly study similar to the 1993–95 I baseline study will be conducted. The study will The Balanced Scorecard (BSC) allows an organization to focus on a relatively small number of track purchased monographs for two years to carefully chosen measurements. These measures are determine if a customer uses them. aligned with the library mission and strategies, and The U.Va. Library is using this technique for several they provide a quick but comprehensive picture of reasons. In essence the Balanced Scorecard enables us organizational performance. to gain better control of our statistical operations. By The concept dates from a 1992 article in the Harvard limiting the number of scorecard metrics, it forces us to 1 Business Review by Kaplan and Norton. The scorecard decide what is important, and to identify those numbers was designed for private business, but is increasingly that truly make a difference. It also introduces some being used by governmental and nonprofit organizations. balance into our statistical work. Like many libraries, In 2001 ARL and OCLC sponsored a three-day forum on we have collected many data regarding resources and performance measures, which featured the Balanced user services, but other areas have not received the same 2 Scorecard. Several German libraries, under the leadership attention. The BSC compels us to take a focused look at of the German Research Council, have implemented the finance, internal processes, and the future. Another 3 Balanced Scorecard. However, implementation in North important aspect of the BSC is the assigning of targets or American libraries has been slow to develop. goals. We not only decide what measures are The Balanced Scorecard examines the organization important; we also state what constitutes success for from four perspectives: User, Finance, Internal each measurement. A final rationale for the scorecard is Processes, and Learning and the Future. Four to eight its intelligibility; each year we can literally see how well measurements, or metrics, are devised for each category we have done. We get a clear picture of organizational or perspective. Each metric has a specific and performance in the form of a set of pie charts. We have unambiguous target or set of targets. For example, two one pie chart for the overall results, and individual of the metrics adopted by U.Va. for 2002–03 are charts for each of the four perspectives. We can described below. Each of these metrics contains two evaluate, at a glance, the performance of the library. targets. Target1 indicates complete success in achieving It is equally easy to compare the four perspectives the metric; Target2 indicates partial success. and to note where we are doing well, and where we To recruit, develop, and retain productive, highly need to improve. qualified staff, the library established a metric for the Choosing the metrics may be the most crucial part Retention Rate of Commended Employees: of the process. Once a metric is established, it has organizational approval and recognition—as the saying Target1: 95% of commended employees remain on goes: what gets measured, gets managed. The choice the staff. of metrics depends upon the values of our organization. Target2: 90% of commended employees remain on There must be some agreement as to what matters, what the staff. counts within the library. Adopting the Balanced Method: Staff members who have received at least Scorecard requires the library to confront issues in a one extraordinary contributor write-up and faculty more rigorous fashion, to make some hard decisions, who have received at least one “extra-meritorious” and to make those decisions explicit. or “highest” salary increase over the past three The subject matter of the metric (e.g., reference performance cycles are identified at the beginning of service, cataloging, fundraising, interlibrary loan) is each fiscal year. The library may also take other obviously important, but not the only issue. The nature forms of recognition into consideration. At the end of the measurement is equally important. Do we count of the year, we will determine what percentage of the number of times we perform a task? Do we measure these employees still work at the library. the cost of performing the task? Do we calculate the 28 ARL 230/231 • OCTOBER/DECEMBER 2003 THE SCORECARD MEASURES ARE “BALANCED” INTO FOUR AREAS
User Perspective Finance Perspective How can we fulfill user How can we secure and use resources expectations? in an effective way? • Satisfaction rating of our customers • Unit cost for various transactions in our periodic surveys • Cost per use of purchased materials • E-metrics • Revenue from state and private sources • Turnaround time for patron requests
Internal Process Perspective Future Perspective How should we organize internal processes How can we secure the future capability to meet user needs and expectations? of our services? • Throughput times for making • Staff training outcomes available new materials • Employing the best people • Turnaround times for internal • Replacement/updating of equipment service requests and technical resources
time it typically takes to do the task? Or do we survey management tool. It has improved our statistical focus, our customers and ask them how well we performed clarified and balanced our organizational values, and the task? Any of these techniques might be appropriate, helped us develop a culture of assessment. We think its but our choice of techniques is reflective of the library’s utility will only increase as we become more priorities. experienced in its implementation. The metrics also reflect practical considerations. Collecting data generates costs, and we attempt to —Copyright 2003 Jim Self minimize those costs. We do not want the Balanced Scorecard to be a burden; it should not appreciably Additional information can be found at
DATA-INFORMED DECISION MAKING server log analyses can help determine how those by Susan J. Beck, Head of Public Services, Paul Robeson collections can best be used. Library, Rutgers, the State University of New Jersey Human resources can conduct climate surveys to As part of her sabbatical, Susan Beck served as an determine work satisfaction and identify problem areas. ARL Visiting Program Officer in 2002 to investigate the They often use data from focus groups, user (staff) impact of assessment on library management decision making surveys, and exit interviews. Analyses of peak service and the degree to which assessment data has influenced use can highlight staffing patterns to provide informa- change. She presented papers on the results of her research tion for better scheduling. Process improvement studies at the ACRL 11th National Conference and the 5th can be used to discover areas for change in allocating Northumbria International Conference on Performance staff resources. Measurement in Libraries.1 Data are critical for making decisions about library space and equipment. Increased demand for n this age of increased accountability, academic instructional space points to the need for additional research libraries must effectively present classrooms and sophisticated equipment. Performance information on their accomplishments to of equipment can be tracked through logs and repair university administrators. Often, these administrators records. Student surveys often provide the needed data are looking for hard data as evidence of organizational I to support proposals for increased technology. improvement. In order to learn more about how data Decisions about services are influenced by user can influence decisions, nine ARL public universities surveys, use data, focus groups, budget cuts, and actively involved with assessment activities were process improvement activities. As an example, library examined for patterns in their use of data to make service hours in all libraries visited changed as a result decisions (Arizona, Arizona State, Connecticut, Texas, of data on use of services. Other services in which Toronto, Virginia, Virginia Tech, Washington, York). changes were made included interlibrary loan, desktop Fifty-nine senior library managers and directors delivery and virtual reference services, shelving, and were interviewed about issues such as accountability, electronic reserves. governance, service evaluation, planning, decision Results of the interviews indicate that assessment processes, assessment tools, and new data measures. data provide necessary evidence to implement change Interview responses were analyzed using Atlas.ti: The in research libraries. Each visited library could point to Knowledge Workbench, a qualitative data analysis tool. several examples of the successful use of assessment As expected, library administrators use data for data. Libraries that create, nurture, and integrate planning and setting organizational priorities. Data are assessment into their everyday activities are models for necessary for developing policies, allocating staff, other institutions. The willingness of leadership to building collections, and planning service delivery. The incorporate assessment data into decision-making kinds of data used include formal assessments, survey activities can be reflected in their own interest and the results, and anecdotal information. Informed decision organizational culture, as demonstrated in staff making lends credibility to both the library and its expertise and concern for service quality. leadership. At some institutions the libraries have taken Accountability pressures from the parent institution are the lead in providing data, while at others the parent also a significant factor for these institutions and the use institution requires that all funding requests be of assessment data provides the library with an accompanied by appropriate data. One library has a opportunity to highlight its accomplishments with formal Management Information System by which it credibility and integrity. can generate the data necessary for units to support their planning processes. Some library administrators —Copyright 2003 Susan J. Beck also feel that it is important to encourage data-driven decisions at all levels in the organization. 1 Susan J. Beck, “Making Informed Decisions: The Implications of Nearly every unit in the library can use data for Assessment,” paper presented at ACRL 11th National Conference, Charlotte, N.C., April 10–13, 2003; and “The Extent and Variety of decision making. Collection development data for both Data-Informed Decisions at Academic Libraries: An Analysis of print and electronic resources include usage data, Nine Libraries of the Association of Research Libraries in the U.S.A. survey data, economic information, departmental needs, & Canada,” paper presented at 5th Northumbria International Conference on Performance Measurement in Libraries, shelving and interlibrary loan statistics, server logs, and Collingwood College, Durham, England, July 28–31, 2003. information from faculty consultations. Some librarians have developed specialized local tools such as databases or Web sites. Special and digital collections data include information about users, research topics, and materials use. Usability testing of digital resources and 30 ARL 230/231 • OCTOBER/DECEMBER 2003 ARL SERVICE QUALITY CREATING A CULTURE OF VALUATION CADEMY SSESSMENT N E A ™ A —A ARL/OLMS by Consuella Askew, LibQUAL+ Program Specialist WORKSHOP n May 16, 2003, ARL’s Service Quality ustomer- or user-centered decision making Evaluation Academy graduated its second class and programming requires relevant data and of participants at the close of an intensive five-day information but also an organizational bias toward workshop in San Antonio, Texas. The Academy continuous assessment. The ARL Office of Leadership Oemphasizes basic concepts and skills in measurement and and Management Services (OLMS) offers a workshop on data analysis for library service quality evaluation. The C Creating a Culture of Assessment by investigating the curriculum focuses on qualitative and quantitative main forces influencing the library and how these forces methods with some time spent on relevant data-analysis beg for the adoption of assessment and evaluation as an software, such as ATLAS.ti and SPSS. The workshop is integral part of library work. directed towards conducting service quality evaluations Workshop participants practice: integrating in general, with the expectation that participants will assessment work into organizational systems and experiment with methods and eventually create their own structures and everyday work activities; determining tools to enhance the ones being developed by libraries. current and preferred future paradigms and skills to The Curriculum move groups toward the new paradigm; and customer- The first half of the workshop was devoted to focused data gathering. For more information, including workshop schedule quantitative methods and was led by Bruce Thompson, and registration form, see