The Slow Assessment Movement: Using Homegrown Rubrics and Capstone Projects for DIY Information Literacy Assessment

Darcy Gervasio, Kimberly Detterbeck, and Rebecca Oling

This paper will explore how librarians at Purchase College leveraged ideas from the slow movement and authentic assessment to design a rubric to assess information literacy skills in senior projects. We will discuss how we normed and tested the rubric and share the next steps in this process.

Introduction: find a means to measure student success, standard- Purchase College was established in the late 1960s ized tests initially seemed like the fastest, most con- by then Governor Nelson Rockefeller as a landmark venient, most “measurable” option. During the campus within the SUNY system that could marry the 2010–2011 academic year, the Library’s Assessment liberal and creative arts. Born of that perspective, the Working Group compared several commercial tools college has long required students to create “senior for programmatic level assessment of students’ infor- projects.” A college-wide requirement for both bach- mation literacy skills, including Information Literacy elors of arts and science degrees, the senior project is Test (ILT), Project SAILS, and iSkills by ETS. Our the capstone and hallmark of the Purchase education- analysis determined that the one-size-fits-all nature al experience; many projects cross disciplines with of these third-party assessments did not reflect the re- unique results. Students devote two semesters to an search experience of students using Purchase College in-depth, original, creative study. These research arti- Library resources.2 facts provide evidence of how graduating seniors use Standardized tests for information literacy assess- information and demonstrate research skills. Since ment are also challenging to implement longitudi- 1973, the Purchase College Library has collected, ar- nally. Working with the College Writing program, the chived, and cataloged senior projects from the School Library could easily recruit freshmen to take a stan- of Liberal Arts and Sciences.1 Since they are archived dardized test, but it would be significantly more diffi- in the Library (or, more recently, online), senior proj- cult for the Library to recruit seniors to participate in ects are easy for librarians to access. such an assessment without a broader campus-wide Despite easy access to senior projects, when the curricular mandate.3 Without a mandate, assessment Library’s Assessment Working Group was asked to of seniors would rely on self-selected volunteers and

Darcy Gervasio is Reference and Instruction Librarian, Purchase College, e-mail: [email protected]; Kimberly Detterbeck is Art Librarian, Purchase College, e-mail: [email protected]; Rebecca Oling, Coordinator of Instruction and Literature Librarian, Purchase College, e-mail: [email protected]

721 722 Darcy Gervasio, Kimberly Detterbeck, and Rebecca Oling

therefore diminish the authenticity of our results. We namism of the information age. The secret is needed meaningful data based on actual student re- balance: instead of doing everything faster, do search products, not test scores, in order to fully and everything at the right speed. Sometimes fast. practically assess the level of information literacy Sometimes slow. Sometimes somewhere in be- skills demonstrated by Purchase College students. We tween. Being Slow means never rushing, never needed something local and more authentic. And we striving to save time just for the sake of it. It had that in front of us in our senior projects. As an means remaining calm and unflustered even alternative to testing, we developed a rubric-based as- when circumstances force us to speed up.7 sessment of senior projects. The philosophy of Slow has been applied to food, Authentic Assessment & Slow travel, work, sex, architecture, parenting, fashion, web Assessment design, technology, reading, and education. The Slow Focusing on senior projects was a natural fit for us. Movement is making small inroads into Library and Not only were they easy for librarians to access, but Information Science as well. A 2014 Delphi study of they represent the culmination of a student’s under- seventeen librarians concluded that Slow principles graduate education. Each project is created within the are useful as a user-centric lens for understanding student’s own discipline, and students often use these information-seeking behavior: “Slow may be better to market themselves to employers after graduation. seen as a focus on speed and choice, in information Authentic assessment, as defined by Gulikers, is as- terms” and on “the power that an individual has, par- sessment that requires students to “use the same com- ticularly in the context of everyday life, to choose [re- petencies, or combinations of knowledge, skills, and search] options according to reflectively determined attitudes, that they need to apply in the criterion situ- need, rather than being swayed by conventional soci- ation in professional life.”4 Sometimes called “perfor- etal forces.”8 The focus on choice and socio-economic mance or direct assessment,” authentic assessment is forces recalls the values in the “threshold concepts” distinguished from standardized testing because stu- that form the basis for ACRL’s revised Framework for dents “actually demonstrate skills rather than answer Information Literacy in Higher Education, suggesting questions about them.”5 that Slow principles have much relevance to librarian- Authentic assessment dovetails perfectly with the ship today. principles of the Slow Movement which focus on re- The common threads in Slow Reading, Slow Edu- flection, choice, leveraging local assets, and resisting cation, and Slow Technology are reflection, choice, cultural pressures to hurry. The Slow Movement has and meeting local or individual needs. Slow Technol- its origins in the Slow Food Movement. Begun in Italy ogy seeks to counter technological determinism and in 1989, Slow Food resists fast food and “fast life,” the the better, faster, stronger mentality which have driven over-industrialization and obsession with speed that information technology (IT) development for years afflicts many Western nations.6 The Slow Movement and have lead to planned obsolescence and a culture rejects this “cult of efficiency” and “time sickness” in that is always on. Instead, Slow Tech proponents ad- favor of doing things at a more natural pace. Yet it is vocate clean, sustainable IT driven by careful reflec- not necessarily about doing everything slowly. As Carl tion.9 Similarly, Slow Education emphasizes reflective Honoré explains in his seminal book In Praise of Slow- learning. Arguing for reforms to accelerated nursing ness: education programs, Sellman points out that “ideas of Slow are consistent with much that is known about [T]he Slow movement offers [a] middle path, learning. It is known that students learn in different a recipe for marrying la dolce vita with the dy- ways and at different speeds.”10 Sellman recommends

ACRL 2015 The Slow Assessment Movement 723 building time for reflection into nursing curricula Defining Slow Assessment and valuing quality over quantity. “Slow encourages Drawing on the ideas of the Slow Movement, par- a step back from haste in all things, a re-appraisal of ticularly Slow Technology, Slow Reading, and Slow the obsession with quantity towards the appreciation Education, we offer Slow Assessment as a model of quality.”11 Ashworth explains that it’s important for that emphasizes reflection, choice, and meeting lo- K-12 teacher education programs to set aside time for cal needs to provide more meaningful evaluations contemplation and deliberation as well; she laments of library services and students’ research skills and “standardized degrees” and how the rigid nature of to combat the corporatization of higher education. teacher training programs in North America “mimic Slow Assessment embraces reflection throughout the the structure and speed of the business world.”12 Berg process. Common pitfalls such as poorly-construct- and Seeber address similar concerns from the pered surveys and a failure to “close the loop” can result spective of university faculty in their empowering ar- from rushed planning and opting for the quickest, ticle about Slow principles in higher education: “By most convenient path, often compounded by outside taking time for deliberation, reflection, and dialogue, pressures and deadlines set by administrators or ac- the Slow Professor takes back the intellectual life of creditation bodies. Program-level assessment in uni- the university.”13 versities is often treated as a series of hoops to jump Slow Education and Slow Reading proponents op- through so administrators can check boxes to ensure pose standardized testing and “teaching to the test” and a program is meeting bare-bones requirements. Slow seek to combat the McDonaldization of education.14 In Education advocates object to this corporate mental- a widely cited critique of No Child Left Behind, Mau- ity which prizes fast metrics over reflective, meaning- rice Holt compared the standardization of education ful evaluation. These pressures lead libraries to resort with the “deterministic thinking that governs the pro- to cursory, easily quantifiable metrics like gate counts, duction of fast food.”15 In Holt’s view, “between the pre- circulation stats, website hits, or raw numbers of in- cision of tests and the raw variety of classroom life lies a struction sessions taught. Such data can be useful, but vast gulf” that is not as easy to quantify as administra- data alone does not reflect the true mission and value tors might like.16 In his book on Slow Reading, Thomas of a library in the life of its users. Newkirk warns against timed tests and assessing for When librarians try to conduct more meaningful its own sake and addresses the hidden values students assessments or research studies, they are often ham- learn through standardized test preparation: “We all pered by the same problems Berg and Seeber iden- want our students to do well on tests, but how many of tify: time poverty, competing responsibilities, work- us know what values underlie these tests? … Compre- life balance, workplace stress, outside deadlines, and hension becomes identical to doing well on this test; lack of space for reflection and professional develop- the test, rather than being built on a value system, be- ment.18 As a result, many libraries turn to one-size- comes the value system. We become our numbers.”17 fits-all commercial tools or tests for assessing public Slow Education advocates are not against all as- services and information literacy, which may not be sessment. After all, educators need to know whether the best fit for their specific institution’s culture or -as their teaching methods are working and students are sessment goals. Even when an assessment is home- learning. Reflective papers, experiential learning, and grown, the pressure to be efficient can lead librarians capstone projects are the preferred assignments of the to deploy hastily-crafted online surveys rather than Slow Education movement; however, little has been take the time necessary to construct a proper research written on how Slow principles can help educators be- study. Berg and Seeber remind us that “efficiency” come better assessors or what Slow Assessment might was originally a corporate value and argue that “[s] look like. tandardization loses sight of the open-endedness of

March 25–28, 2015, Portland, Oregon 724 Darcy Gervasio, Kimberly Detterbeck, and Rebecca Oling

intellectual inquiry. The consumerism that has taken laying the foundation deliberately and with care, we hold in higher education propels the belief that time is are creating sustainable assessment practices at Pur- money, resulting in superficial learning.”19 chase College that can be adapted and enriched for Slow principles give librarians and instructors years to come. the choice to resist the cultural temptation to speed through assessment and assess for its own sake. Slow Developing the Rubric: Mapping to Assessment does not mean librarians have to be less Standards efficient; it means you can choose to assess at a speed We began developing the Senior Project Information that is appropriate for your library, students, sched- Literacy Skills Assessment rubric (SPILSA) in 2011. ules, resources, and staff. It promotes reflection and Although SUNY has an “information management” quality over quantity. The first step is giving yourself requirement for all campuses, this is generically de- permission to be slow and deliberate. You will likely fined, and Purchase College does not currently have still have outside deadlines, but Slow Assessment lets any campus-wide assessment of students’ information you focus on the choices you do have. Within imposed literacy competencies.22 Because of the lack of local limitations, librarians can still choose which skills to standards and our desire to assess Purchase College assess, which methods to employ, which standards or graduates against current national standards, we used learning outcomes will underpin an assessment. A ACRL’s Information Literacy Competency Standards Slow mindset can help you work through setbacks by for Higher Education as the basis for our rubric.23 We allowing you to strive for balance and focus on what’s identified the most measurable criteria in the ACRL most important to your library. Standards, matching our rubric categories and bench- In addition to reflection and choice, Slow Assess- marks to specific standards, indicators, and outcomes. ment is about thinking locally. The Slow Food move- Our final criteria include: presence of a thesis statement focuses on sustainability and local agriculture. ment, authority of references, variety of references, Exemplified by the farmer’s market, “eating local” re- consistency of attribution, quality of citations (in text duces pollution, prizes freshness, and connects farm- & works cited), ability to paraphrase/summarize/ ers and eaters.20 Likewise, Slow Assessment involves quote effectively, integration of resources to support tapping into a library’s local assets. Adapt your assess- a thesis, overall organization of content, and limita- ment strategy to the needs, weaknesses, and strengths tions of research. For each criterion, the SPILSA ru- of your library and your campus—exploit the unique bric defines the attributes a senior project must have quirks of your institution rather than forcing yourself to meet a minimum level of competency in informa- to conform to a cookie-cutter, standardized tool. Lo- tion literacy.24 In our benchmarks, we are looking for cal factors will dictate the speed with which you plan a basic level of competency in information literacy, and conduct assessments, but there is no right or not full mastery or evidence of exhaustive graduate- wrong speed. The tempo guisto is one that allows you, level research. The rubric reflects this by using the your colleagues, your library, and your institution benchmarks “exceeds expectations,” “meets expecta- ample time to reflect, plan, and carry out a thought- tions,” “meets some expectations,” and “meets few or ful assessment that will yield usable data and meet no expectations.” your most important goals.21 In the end, it may take In 2012, the Assessment Working Group won a the Purchase College Library assessment team several campus wide assessment award for our efforts. The months or several years to complete our rubric assess- Associate Provost for Academic Affairs and Direc- ment, but we believe the process of Slow Assessment tor of Assessment took an interest in our project and will provide richer results that are usable and sustain- asked us to revise the rubric to adhere to SUNY’s able while avoiding staff burnout. We hope that by system-wide student learning outcomes, particularly

ACRL 2015 The Slow Assessment Movement 725 those related to critical thinking. We strove to balance tionally, the ACRL Information Literacy Standards the needs of the library with the desires of campus ad- are not obviously exemplified in creative works, mak- ministration. The result was a compromise: we added ing assessment via the SPILSA rubric less applicable. a column to our rubric tying each of our existing crite- To compile an anonymous random sample of ria to SUNY’s Student Learning Outcomes. This is an projects for review, we drew from our 2011-2012 example of a “slow assessment” mindset in practice: Moodle repository because it was readily accessible to rather than completely rework our rubric, we mapped us in a digital format. We attempted to represent a mix our rubric to fit administrators’ needs in keeping with of disciplines but were not concerned about having an our values and goals. equal number of projects from each discipline.27 To Although the revised Framework for Information avoid biasing the raters, we trained student workers Literacy for Higher Education is still under review at to choose every third project in the Moodle archive the time of publication, academic librarians are clear- and redact all identifying information. This process ly moving towards a “threshold concepts” approach produced a random sample of 130 anonymous senior to information literacy instruction which focuses on projects, which were uploaded to Dropbox. “core concepts with flexible options for implementa- Recognizing the importance of norming the ru- tion” rather than “any prescriptive enumeration of bric, we used a random number generator (www.ran- skills.”25 While we support this approach, instruction dom.org) to choose a test sample of ten senior projects librarians still need to be able to measure skill acquisi- from our larger sample. Each project was scored by all tion through authentic assessments. We feel the SPIL- three of us; we then calculated basic correlations for SA rubric allows us to look at student work as a whole each of the nine rubric criteria. As a result of this first and as a practical artifact of learning. As Megan Oak- test, which took place in 2013, we revised the phrasing leaf explains, “by articulating exactly what librarians of certain rubric items. We also converted the rubric are looking for in student achievement of outcomes into a Google form to facilitate scoring and added the at each stage in the student journey, rubrics ensure a exact definitions of each benchmark to the form. more valid approach to assessment. When rubrics are In 2014, we conducted a second round of testing, ‘normed’ or calibrated for use by multiple raters, they using the revised rubric to score a random sample also lead to reliable assessment results.”26 We believe of twenty senior projects. Each of the three librar- there is room for both flexibility and norming within ians scored each project. To measure interrater reli- the new Framework for Information Literacy because ability, i.e. consistency and agreement among scorers, it aims to make information literacy instruction— we calculated the Intraclass Correlation Coefficient and, by extension, assessment—more localized and (ICC) for each of the nine rubric criteria and for the customizable. average score for each senior project.28 The ICC test corrects for chance and measures agreement among Testing the Rubric three or more raters who use the same set of crite- We chose to assess only research-based senior proj- ria to assess the same group of subjects.29 In SPSS, we ects from the humanities, social sciences, and natu- ran a two-way random effects ICC test on the total ral sciences. We did not include senior projects from score, average score, and each of the nine rubric cri- art and design (painting, drawing, graphic design, teria.30 The single measures ICC for the average score printmaking, sculpture, and new media) or conser- was 0.725, suggesting an acceptable level of agreement vatory programs (music, dance, theater arts, creative among the three raters. According to Landers, “ICC writing) because these majors do not always have a tends to be low if you are assessing any…constructs written, research component or require students to with only single items each.”31 In other words, because submit senior projects to the Library archive. Addi- of our small sample size and relatively low number of

March 25–28, 2015, Portland, Oregon 726 Darcy Gervasio, Kimberly Detterbeck, and Rebecca Oling

variables (only nine rubric criteria, each assessing a the curriculum. Based on the data we gather from different skill), calculating ICC for the overall average SPILSA, which can be broken down by board of study scores, rather than each individual rubric criterion, is (department) and by academic year, the Assessment a more accurate measure of our interrater agreement. Working Group can identify where Purchase seniors For this reason, the ICC for individual criteria such as excel and where there is room for improvement. This “Attribution” or “Addresses Thesis” is lower than for information can be used by the Library to adjust our the overall average score. Landers explains that “for research instruction session curricula. Individual ‘practical decisions’ like grades, the reliability level boards of study could also use the results of this as- recommended by Nunnally is 0.8 to 0.9. In practice, sessment to build information literacy education into that’s obviously quite rare.”32 Generally speaking, an the curriculum for their majors and senior project ICC of 0.7 is considered a minimum acceptable level students. We expect the SPILSA data will help us in of interrater agreement for research purposes.33 our outreach efforts so that we will better understand While the testing process has taken many months, which courses and boards of study need more guid- especially since all three librarians are working around ance in supporting juniors and seniors’ research skills. our primary reference and instruction duties, we feel The Library Assessment Working Group will that taking the time now to assure interrater reliability assess a sample of senior projects from various dis- will help us reach our long-term goals of creating a ciplines, but individual boards of study can also use rubric assessment that can be implemented for years the SPILSA rubric to assess a larger number of senior to come. Adopting a Slow Assessment mindset has projects within their major(s). We envision that sub- allowed us to avoid cutting corners and develop this ject liaison librarians will partner with boards of study rubric at a pace that is both deliberate and feasible. to implement SPILSA and produce results tailored to a particular major or discipline. Thanks to Moodle Implementing the Rubric: Next Steps Rubrics, SPILSA could easily be integrated into an in- Given the promising results of the second round of dividual instructor’s grading process for senior proj- testing, our next step is to conduct a frame-of-ref- ects—or research projects in other courses—on an erencing session to unify the mental models of each ongoing, sustainable basis.34 of the three raters. We will compare papers with the In the future, we hope SPILSA can go beyond se- highest degree of disagreement and come to a consen- nior projects. Our rubric could be adapted by indi- sus on the “right” score. We will also choose “anchor vidual boards of study to determine students’ infor- papers” as examples of senior projects that exceed, mation literacy competency levels at different points meet, meet some, and meet no expectations. This pro- during students’ progression through their major. cess of norming will allow each librarian to score a Because the information literacy standards in the ru- separate set of senior projects with high confidence in bric are general enough to apply to most undergrad- the consistency of our grading. Once we have finished uate-level research projects, the SPILSA rubric could norming the rubric, we will assess a full set of 100 ran- also be used to assess information literacy skills using domized, anonymous senior projects. Ultimately, we research papers of students in introductory courses, believe our data will show to what degree Purchase junior seminars, or senior seminars. While boards College seniors meet minimum national information of study have their own student learning outcomes, literacy competency standards as defined by ACRL. SPILSA specifically focuses on the information litera- The Library is hopeful that data gathered from cy component and can therefore complement broader SPILSA will help inform and hone our instructional or discipline-specific assessment plans. The data gath- approach in information literacy sessions and further ered from SPILSA could help a board of study de- our mission to more fully embed our support across velop a curriculum that embeds information literacy

ACRL 2015 The Slow Assessment Movement 727 into key classes throughout the major, ensuring that presentations, or other research artifacts for authentic seniors are better prepared for their senior project re- assessment. For example, Purchase College librarians search as well as life after college. recently shared our rubric with colleagues at Manhat- Thinking long-term, we hope to use the SPILSA tan College. Manhattan College does not have an insti- rubric to compare senior work with research papers tutional repository of capstone projects, but librarians submitted by freshmen. This comparison will provide are heavily embedded in the Communications depart- a longitudinal picture and further our understand- ment and are able to obtain a set of final papers from ing of Purchase students’ retention of, improvement those courses to assess. in, and acquisition of information literacy skills over Identifying potential allies inside and outside time. This information could lead to a campus-wide the library can help make authentic, slow assessment set of tiered, minimum competencies for information seem less daunting. Many campuses have an assess- literacy at each level (freshman, sophomore, junior, ment office or office of institutional research that can senior). The Library hopes that by sharing the results assist with data analysis. You can also reach out to of this assessment, we will bring attention to the im- individual faculty members and statisticians who are portance of information literacy instruction at Pur- willing to partner with you. At Purchase, we adapted chase College, both inside the Library and in various our rubric to campus-wide student learning outcomes boards of study. through our collaboration with the Associate Provost. Eventually, we plan to use the Library’s existing part- Sharing the Rubric: Making it Local nership with the College Writing program to apply Any library can embrace the principles of Slow As- the SPILSA rubric to freshman research papers, com- sessment and create meaningful, authentic assess- paring the results to our senior project assessment for ments that fit local needs. We encourage other librar- a longitudinal view. ies to adapt our rubric and embrace Slow Assessment Finally, Slow Assessment gives you permission to by keeping it local, collaborating with key allies on take time to reflect on what is right for your staff, your campus, and giving yourself permission to be slow students, and your library. Our assessment team made and reflective. concessions to the Provost’s agenda, but we tried to Local needs, assets, staffing, and resources will dif- remain true to our core goal of gathering usable, au- fer from library to library, but you can leverage your thentic data that can be used long-term to improve in- strengths to create sustainable, local assessment. Some formation literacy instruction. When you find your- libraries may not have access to senior theses, but they self under pressure from external forces, try to remain may offer credit-bearing information literacy courses focused on your overarching goal. If that means scal- where standardized information literacy tests could be ing down an assessment, assessing a smaller sample of deployed thoughtfully and effectively. One library may research artifacts, or assessing fewer or more targeted have a designated assessment librarian, while another learning outcomes, so be it. Slowing down and look- may need to assemble a team of librarians whose pri- ing beyond inauthentic, standardized assessments in mary duties lie elsewhere. Some campuses may have favor of a reflective, continuous, ever-growing Slow librarians embedded in junior-level research methods Assessment model can foster a “culture of assessment” courses who could ask instructors for student papers, at your library in a more sustainable, deliberate way.

March 25–28, 2015, Portland, Oregon 728 Darcy Gervasio, Kimberly Detterbeck, and Rebecca Oling Meets Few or No Meets Few Expectations (1) Unclear or absent or thesis statement question research Most of the references that sources from are not peer-reviewed are uncertain and have reliability Uses only one type of or obtains reference one from all references all articles (i.e. source the same journal from or same author) no attribution. to Little is confused Reader of about the source ideas and information. Meets Some Expectations (2) Defines a thesis based on statement need (it is information but may not present, be clear and concise) Although most of the references professionally are are a few legitimate, questionable within that discipline (e.g., books, internet trade popular sources, magazines, …). on relies heavily Too or source one source to type (as appropriate discipline) Occasional attribution, statements but many seem unsubstantiated. of ideas Source is and information sometimes identified. Meets Expectations (3) Clear and concise based thesis statement need on information are References primarily peer- professional reviewed journals or other discipline appropriate sources Uses a mix of sources, too Does not rely on one source heavily type or source is clear in Attribution of most cases. Source ideas and information is mostly identified.

APPENDIX. http://tinyurl.com/SPILSARubric Exceeds Exceeds Expectations (4) easily presents "Writer focused, identifiable, and original thesis based on statement need" information peer- are All references professional reviewed journals or other discipline appropriate sources Uses a mix of sources, the to appropriate and discipline, balances use of these and source sources types throughout is clear in Attribution of all cases. Source ideas and information identified. is always Performance Thesis Statement of Authority References of Variety References Attribution Senior Projects Information Literacy Skills Assessment Rubric. Purchase College Library College Rubric. Purchase Skills Assessment Literacy Information Senior Projects SUNY Learning SUNY Learning Outcome 1: Students will SLO coherent produce within common texts forms. college level an presents ("Writer easily identifiable and controlling focused purpose or thesis.") 9: Students will SLO and evaluate, locate, information synthesize a variety of from ("all sources sources of good quality, are and well-selected the to appropriate topic") 9: Students will SLO and evaluate locate information synthesize variety of forma ( "student sources adequate has located a from information variety of print and sources.") electronic 3: Students SLO a will research an develop topic, and argument, supporting organize details. ("carefully sources documented style") in an accepted 1.1.b 1.2.b 1.2.b 2.5.c ACRL ACRL Standard

ACRL 2015 The Slow Assessment Movement 729 Most in-text and Most in-text citations bibliographic incomplete/ are inconsistent Does not summarize, or quote paraphrase, integrate to in order into of others the work their own The goals (if any) outlined in the not are introduction unclear met. Reader what conclusions are drawn any Does not address limitations of research for areas or outline any further research. Does not organize in a manner content that supports the thesis. Lacks clear or meaningful transitions or sections. Some in-text and Some in-text citations bibliographic incomplete/ are inconsistent Does not consistently summarize, or quote, paraphrase, and does not always select appropriate integrating method for into of others the work their own Some goals of thesis not met. Some are parts of paper don't thesis. back to relate May mention, but does not substantively limitations address or outline of research further for areas inquiry. in a content Organizes manner that somewhat supports the thesis. some Provides or sections transitions but not consistently. Most in-text and Most in-text citations bibliographic and complete are consistent Adequately summarizes, or quotes paraphrases, integrate to in order of others the work are (ex: their own into tables, and graphs, other images well integrated) Most parts of paper thesis to relate in a statement meaningful way. limitations of Discusses and outlines research of further inquiry. areas content Organizes in a manner that supports adequately the thesis. Generally clear provides or sections. transitions All in-text and All in-text citations bibliographic and complete are consistent Expertly (in both quality and quantity) summarizes, or quotes paraphrases, integrate to in order of others the work are (ex: their own into tables, and graphs, other images well integrated) All parts of paper thesis to relate in a statement meaningful way. Substantively limitations of discusses and outlines research of further inquiry. areas in a content Organizes manner that expertly supports the thesis. provides Always clear and meaningful or sections. transitions

"Quality of Citations (in text & works cited)" Paraphrasing, summarizing, quoting Addresses & relates thesis to statement and research question Limitations of Research Organization of Content

SLO 3: Students SLO a will research an develop topic, and argument, supporting organize details. ("carefully sources documented style") in an accepted 3: Connects SLO and paraphrases the quotations to thesis. SLO larger are 9: "[Sources] well- consistently the into integrated paper" 3: "conclusions SLO writer's demonstrate conscious attempts his or her integrate to thinking with an own of outside analysis sources" 6: Students will SLO and analyze, identify, arguments evaluate as they occur in their and others' own 7: Students SLO work.. well- will develop arguments reasoned 3: Students "SLO a topic, will research an argument, develop and organize supporting details. 7: Students will SLO well-reasoned develop arguments" 2.5.c 3.1.b 3.7.a 4.7.a 4.1.a

March 25–28, 2015, Portland, Oregon 730 Darcy Gervasio, Kimberly Detterbeck, and Rebecca Oling

Notes 17. Newkirk, The Art of Slow Reading, 12. 1. The Library has a small selection of senior projects from the 18. Berg and Seeber, “The Slow Professor,” 1-7. School of the Arts, but these are not collected with consis- 19. Berg and Seeber, “The Slow Professor,” 5. tency. 20. Honoré, In Praise of Slowness. Portinari, “The Slow Food 2. For example, on one standardized test, the correct answer Manifesto.” for where one goes to look for a journal title was the li- 21. Honoré uses the Italian expression “tempo guisto” meaning brary’s catalog. In our case, only current print subscriptions “the right speed” to define the Slow Movement. are listed in our OPAC. We must access updated journal 22. According to SUNY’s Information Management Com- information through our “eJournals by Title” link off the petency, “students will…perform the basic operations of library’s homepage. personal computer use; understand and use basic research 3. College Writing is a required freshman course that includes techniques; and locate, evaluate and synthesize informa- a research paper and a library workshop. Librarians at tion from a variety of sources.” Office of the Provost, State Purchase College work closely with administrators and University of New York, “Guidelines for the Approval of instructors in this program. State University General Education Requirement Courses,” 4. Judith Gulikers, “A Five-Dimensional Framework for SUNY, last modified September 9, 2010, http://system.suny. Authentic Assessment: EBSCOhost,” Educational Technol- edu/media/suny/content-assets/documents/academic-af- ogy Research & Development 52, no. 3 (2004): 69. For a fairs/general-education/GenedCourseGuidelines_20120530. complete literature review on authentic assessment, please pdf see Gulikers, “A Five-Dimensional Framework” and Carol 23. Association of College & Research Libraries (ACRL), Perruso Brown and Barbara Kingsley-Wilson, “Assessing “Information Literacy Competency Standards for Higher Organically: Turning an Assignment into an Assessment,” Education,” ACRL, February 2004, http://www.ala.org/acrl/ Reference Services Review 38, no. 4 (2010): 536–56. standards/informationliteracycompetency. 5. Brown and Kingsley-Wilson, “Assessing Organically,” 538. 24. To view the SPILSA rubric, please see Appendix A or visit 6. Folco Portinari, “The Slow Food Manifesto,” Slow Food, http://tinyurl.com/SPILSARubric. December 10, 1989, http://www.slowfood.com/about_us/ 25. Association of College & Research Libraries (ACRL), eng/manifesto.lasso. “Framework for Information Literacy for Higher Educa- 7. Carl Honoré, In Praise of Slowness : How a Worldwide tion,” ACRL, February 2, 2015, http://www.ala.org/acrl/ Movement Is Challenging the Cult of Speed (San Francisco: standards/ilframework. HarperOne, 2004), 275. 26. Megan Oakleaf, “A Roadmap for Assessing Student Learn- 8. Elizabeth Poirier and Lyn Robinson, “Slow Delphi: An ing Using the New Framework for Information Literacy for Investigation into Information Behaviour and the Slow Higher Education,” Journal of Academic Librarianship 40, Movement,” Journal of Information Science 40, no. 1 ( 2014): no. 5 (2014): 513. For a detailed explanation of assessment 95. rubrics, please see Megan Oakleaf, “Using Rubrics to Assess 9. Norberto Patrignani and Diane Whitehouse, “Slow Tech: A Information Literacy: An Examination of Methodology and Quest for Good, Clean and Fair ICT,” Journal of Informa- Interrater Reliability, Journal of the American Society for In- tion, Communication & Ethics in Society 12, no. 2 (2014): formation Science and Technology 60, no. 5 (2009): 969-983. 78–92. 27. In 2011-2012, students submitted senior projects electroni- 10. Derek Sellman, “Does the Slow Movement Have Anything cally as PDFs to a locally-developed repository in Moodle, to Offer Nursing Education?”Nurse Education Today 34, no. our learning management system. Disciplines included in 12 (2014): 1415. our assessment are: art history, arts management, anthro- 11. Ibid. pology, cinema studies, economics, environmental studies, 12. Elizabeth Ashworth, “Slow and Local: A Re-Vision of history, literature, psychology, and sociology. Teacher Education in Ontario,” Journal of Unschooling & 28. The authors thank Dr. Amy H. Gervasio, Dr. Craig Wen- Alternative Learning 8, no. 15 (2014): 71. dorf, and Dr. Richard N. Landers for their invaluable as- 13. Maggie Berg and Barbara Seeber. “The Slow Professor: sistance and advice in helping us understand and calculate Challenging the Culture of Speed in the Academy,” Trans- interrater reliability and intraclass correlation statistics. formative Dialogues: Teaching & Learning Journal 6, no. 3 29. Domenic V. Cicchetti, “Guidelines, Criteria, and Rules of (2013): 6. Thumb for Evaluating Normed and Standardized Assess- 14. Thomas Newkirk,The Art of Slow Reading : Six Time- ment Instruments in Psychology,” Psychological Assessment, Honored Practices for Engagement (Portsmouth, NH: Special Section: Normative Assessment, 6, no. 4 (December Heinemann, 2012). Maurice Holt, “It’s Time to Start the 1994): 286. Patrick E. Shrout and Joseph L. Fleiss, “Intra- Slow School Movement,” Phi Delta Kappan 84, no. 4 (2002): class Correlations: Uses in Assessing Rater Reliability,” 265–71. Maggie Berg and Barbara Seeber, “The Slow Profes- Psychological Bulletin 86, no. 2 (March 1979): 420–28. sor: Challenging the Culture of Speed in the Academy,” Richard Landers, “Computing Intraclass Correlations (ICC) Transformative Dialogues: Teaching & Learning Journal 6, as Estimates of Interrater Reliability in SPSS,” NeoAcademic, no. 3 (2013): 1-7. November 16, 2011. http://neoacademic.com/2011/11/16/ 15. Holt, “It’s Time,” 268. computing-intraclass-correlations-icc-as-estimates-of-inter- 16. Holt, “It’s Time,” 270. rater-reliability-in-spss/. 30. A Two-Way Random ICC(2) calculation is appropriate here

ACRL 2015 The Slow Assessment Movement 731

because all raters scored all ratees, and as Landers explains in “Computing Intraclass Correlations”: “1) it models both an effect of rater and of ratee (i.e. two effects) and 2) -as sumes both are drawn randomly from larger populations.” Also, as Cicchetti explains in “Guidelines, Criteria, and Rules”: “(a) ICC can distinguish those paired assessments made by the same set of examiners from those made by different sets of examiners; (b) it distinguishes those sets of scores that are merely ranked in the same order from test to retest from those that are not only ranked in the same order but are in low, moderate, or complete agreement with each other; and (c) it corrects for the extent of test-retest (or interexaminer) agreement expected on the basis of chance a l on e .” 31. Landers, “Computing Intraclass Correlations.” 32. Ibid. 33. Matthew Graham, Anthony Milanowski, and Jackson Miller, Measuring and Promoting Inter-Rater Agreement of Teacher and Principal Performance Ratings, Center for Edu- cator Compensation Reform, February 2012, http://cecr. ed.gov/pdfs/Inter_Rater.pdf, 9. 34. To learn more about Moodle Rubrics please see https:// docs.moodle.org/22/en/Rubrics

March 25–28, 2015, Portland, Oregon