A n n C. SchafSCOMMUNIATIOCCHOLNARLY fner, editor

On scholarly evaluation and scholarly communication

Increasing the availability of quality work

by David E. Shulenburger

s the scholarly communication crisis In 1997, the Pew Higher Education I largely a creature o f the faculty evalua­ Roundtable published a treatise entitled “To tion system? Do academic department Publish and Perish,” which urged univer­ heads, deans, and members o f promotion sities to “place greater emphasis on qual­ and tenure committees simply count the ity rather than quantity in the promotion faculty members’ publications and award and tenure process.” salary increases, promotion, and tenure by In March 2000, a gathering o f academ­ the numbers? If w e reformed the faculty ics, administrators, and librarians drew up evaluation system, would the scholarly the “Tempe Principles for Emerging Sys­ communication crisis disappear? tems o f Scholarly Publishing,” which have One commonly encounters anecdotes since been endorsed by both the AAU and that appear to support affirmative answers NASULGC membership. One o f the prin­ to these questions. Faculty sometimes boast ciples states: “To assure quality and reduce o f publishing the “least publishable unit,” proliferation o f publications, the evalua­ a reference to dividing significant work into tion o f faculty should place a greater em­ several smaller pieces to derive the maxi­ phasis on quality o f publications and a mum number o f articles from it. Others reduced emphasis on quantity.” describe mechanical systems they have Thus both anecdote and study point to established that, upon rejection o f a manu­ the faculty evaluation system’s role in gen­ script by one journal, will automatically erating published scholarship that adds submit that manuscript to the journal next little to the fund o f knowledge. H ow im­ in the status pecking order, continuing portant is this problem? through as many journals as needed until I have served on and chaired faculty one finally agrees to publish the manu­ evaluation committees at the school and script. university level for more than 20 years. Dur­ At least two significant efforts aimed at ing those years, I have review ed many gaining control o f the scholarly communi­ resumes that list publications that are at cation crisis have identified the faculty best marginal when evaluated against the evaluation system as part o f the problem. criterion of generation of new knowledge.

About the author

David E. Shulenburger is provost at the University of Kansas in Lawrence, e-mail: [email protected]

808 / C&RL News ■ September 2001 Editor's note

A common theme in the debate about librarians, publishers, faculty, and admin­ scholarly communication has been the istrators for many years. need for faculty to publish in a large num­ For this month’s column, we have in­ ber of publications to receive tenure. vited David Schulenburger, provost of the “Publish or perish” is an accepted con­ University of Kansas, to share his views cept in higher education everywhere. on this subject. Schulenburger is well Are tenure committees truly blind to known for his efforts to help us under­ the issues of quality? How do they deter­ stand the economics of scholarly commu­ mine quality? Do we have too many low- nication and to reform the scholarly com­ quality publications, or do they serve a munication process. We hope that this purpose in the scholarly communication column will help to spur discussion of process? The answers to these questions these issues on your campus.—A nn C. are complex and have been debated by Schaffner, [email protected]

Why did the faculty member write them? almost always weeds out bad science. Why were they published? However, I do not believe that the pro­ Mark Twain said that one should not cess admits only research that makes a sig­ criticize others on the grounds that one nificant addition to knowledge. Peer re­ cannot stand perpendicular to himself. It viewers are simply too close to the process is very difficult for an author to determine to be expected to know what will be the ultimate worth of his or her research. judged by future generations to represent No one sets out to do inconsequential significant additions to the discipline. Thus work, and having invested weeks, months, the refereeing process tends to weed out or years in a project, it is expecting too the bad but does not eliminate the insig­ much of human beings to judge their work nificant. to be inconsequential. Thus the norm is to But back to those résumés. Based upon write up the work and submit it for peer my many discussions with provosts across review so that others make the judgment. the nation about the evaluation process, I But peer reviewers have similar diffi­ believe that evaluation committees at the culties. Referees are themselves research­ University of Kansas are similar to those ers. As researchers they are entangled in at most research-intensive institutions. In the web of knowledge and become easily our process, volume of publication alone fascinated by a new detail or by the carries no weight. Evaluation committees resubstantiation of an old one. They look examine the perceived significance of the to see whether the data used should be faculty member’s work and if, and only if, relied upon, whether the work followed it is perceived to be of significance do they the methods required to produce valid sci­ begin to measure the quantity of the work. ence, whether it appropriately built upon Quantity takes on importance once qual­ the literature, etc., and then make a judg­ ity is established. Doing very small ment from the middle of the same thicket amounts of quality work simply is not suf­ as to whether it should be published. ficient justification for the standard expec­ tation that 40 percent of a faculty member’s Refereeing weeds out the bad time should be devoted to research. I have great respect for the refereeing pro­ The committee’s judgment of the ulti­ cess. While I am aware of the growing criti­ mate significance of a faculty member’s cism of this process, I have faith that it work is suspect for the same reason that

About the editor

A n n C .Schaffner is an M BA Candidate at Simmons Graduate School of Managem ent e-mail: [email protected]

C&RL News ■ September 2001 / 809 nals and ultimately—through the weight of The real damage done by the published findings in low-level, peer-reviewed faculty evaluation process ... is not journals—finds its way over time into the top journals in the field. If faculty evaluation com­ by rewarding faculty for quantity of mittees or peer reviewers were true judges of publication; it is ... by basing ultimate significance, such articles would command great respect at first reading rather quality judgments on the rigor of than suffer automatic dismissal because of the the peer review process in journals low esteem for the publications in which they where their work appears, a process originally appeared. The real damage done by the faculty which is perceived to be strongest evaluation process then is not by reward­ in the top-ranked journals. ing faculty for quantity of publication; it is by rewarding faculty for quality of publi­ cation and by basing quality judgments on peer reviewers’ evaluations are suspect: the rigor of the peer review process in jour­ committee members simply don’t have the nals where their work appears, a process right perspective to make an infallible judg­ which is perceived to be strongest in the ment. The evidence used by evaluation top-ranked journals. Evidence that this is committees comes from their own reading true is the lack of uproar when a library of the work, their judgment of the rigor of cancels a subscription to a journal per­ review given the work by the journal of ceived to be of low quality. The lack of publication, and, especially in promotion turmoil over such decisions confirms that and tenure cases, the opinion of outside the problem is the reinforcement of de­ reviewers who evaluate the entire body of mand for top-quality journals, not the pro­ the faculty member’s work. liferation of journals of low quality. The latter group is particularly impor­ tant as outside reviewers are chosen be­ What can be done? cause they are experts in the faculty What we must do is restore the public member’s field. Given the narrowness of goods nature of journals by reducing the some fields, only by including external ability of journals to use the market power reviewers can real expertise be brought to they possess to raise prices. There are many the evaluation process. By reviewing the efforts now underway to accomplish this entire body of work from the viewpoint of aim, and SPARC (Scholarly Publishing and the discipline, outside reviewers are in a Academic Resources Coalition) represents position to judge the cumulative impact of one such strategy. By sponsoring modestly the faculty member’s work. priced new journals edited and refereed This evaluation process places essen­ by top scholars, SPARC endeavors to ac­ tially zero weight on publication in so- celerate the supply of prestigious journals called “backwater” journals. Evaluation and thereby reduce the possibility of fur­ committees generally take for granted that ther price increases by existing top tier jour­ work appearing in such outlets got there nals. By creating products like BIOONE, either because the author judged it to be SPARC keeps in the public domain a large of little worth and sent it directly to the group of journals in the biological sciences journal or because it failed to gain accep­ for which prices will not be raised. tance in one of the top journals in the field Three years ago I proposed the creation and by default landed in a lesser one. of NEAR, the National Electronic Article Re­ Sometimes such automatic dismissal is pository. By making scholarly journal ar­ a mistake. Sometimes manuscripts that dis­ ticles available for free three months after play extraordinarily significant new knowl­ publication, I surmised demand for the edge are rejected by top journals because journals would become more price elastic. its ideas challenge the orthodox views. That is, the ability to raise prices would be Thus a revolutionary idea like plate tec­ limited severely by the fact that many tonics reaches the field through lesser jour­ purchasers would choose to wait a short time

810 / C&RL News ■ September 2001 until articles were freely available rather than nals make articles available to the public af­ pay the higher subscription prices. ter a lapse of six months. While manuscript authors need no direct Public Library of Science is the conscious- return in order to generate articles, publish­ ness-raising mechanism to encourage jour­ ers do. By having journals retain the exclu­ nals to move from a profit motive to a public sive right to an article for three months, the goods orientation. Thus far, about 25,000 sci­ journals would maintain the ability to charge entists have signed the pledge. I am optimis­ a smaller subscription price, but a subscrip­ tic that many more scientists will join them tion price that would cover necessary costs. and this effort will be effective. Thus the proposal aimed to keep alive the These initiatives may soon have an im­ current refereed journal system. However, my pact on the ability of journals to raise prices. proposal suffered from the lack of a mecha­ In fact, I am optimistic that these initiatives nism to make it happen. Two subsequent will lower prices and reverse the decades of developments have created such mechanisms. untrammeled inflation. Exploitation of the First, the National Institutes of Health, economics of electronic publication, while under the leadership of Harold Varmus, cre­ returning journals to their deserved public ated PubMed Central, a virtual location in goods status, will permit an increased vol­ which bio-medical journals could be securely ume of quality work to be published and archived. acquired within the reach of existing library Second, a group of scholars initiated the budgets. PublicLibraryofScience.org petition, which Universities should not encourage quan­ constitutes a pledge that its signers will avoid tity of publication over quality in faculty evalu­ journals that do not agree to make their con­ ations. But the imperative is that quality schol­ tents publicly available six months after pub­ arly work has the opportunity to be published lication. By signing the petition, scientists in rigorously refereed journals and that it be agree not to subscribe, submit papers, edit or readily and affordably available to all schol­ referee papers for journals unless those jour­ ars. ■

C&RL News ■ September 2001 / 811 LITA Publications L ib ra r y & In fo rm a tion Te chnology Association

UsabEWNility Assessm ent of Library-R elated Web Sites: Methods and Case Studies Nicole Campbell, editor 2001 $25 ($23 LITA members)

Practical approaches to analyzing the usability of your W eb sites.

Getting the Most out of Web-based Surveys David Ward 2000 $20 ($18.00 LITA members) Create Web-based surveys and use data with common database and spread­ sheet programs.

Getting Mileage Out of Metadata: Applications for the Library Jean Hudgins, Grace Agnew, and Elizabeth Brown 1999 $22 ($19.20 LITA members) State-of-the-art metadata cataloging and standards overview, includes practical steps from project planning to training to merging metadata cataloging into nor­ mal workflow.

Digital Imaging of Photographs: A Practical Approach to Workflow Design and Project Management Lisa Macklin and Sarah Lockmiller 1999 $20 ($18 LITA members) A comprehensive approach to management of digital imaging in libraries.

Other LITA publications and a printable order form can be found at www.lita.org/litapubs Fax orders to (312) 280-3257 or call 1-800-545-2433, press 1 and ext. 4269.

812 / C&RL N ew s ■ September 2001