Publon #1
The Future of Academic Research A. R. H. Preston and D. R. Johnston
The manner in which academic research is published has remained remarkably static relative to the upheaval currently underway in most other publishing industries. In this paper we examine the incentive structure that caused this outcome, outline a strategic approach to creating a system that encourages collaboration and faster scientific development, and introduce publons.com, our implementation of these ideas.
Introduction In 1905 Albert Einstein published four papers in Annalen der Physik and changed the world.1 The first, published in June, introduced the concept of the photon and explained the photoelectric effect. The second, published in July, explained Brownian motion in a way that provided strong proof for the existence of the atom. The final two papers, published in September and November, introduced special relativity. It was truly the annus mirabilis, heralding modern physics and triggering a century of spectacular discoveries. Physics would never be the same. Yet the manner in which academic research is published has remained remarkably static.
The concept of the scientific journal has its roots in the 17th and 18th centuries.2 It was a great technological advance. Up to that point new results were communicated either by word of mouth, letter, or monograph. The journal standardized the way novel results were collated, validated (via editors and peer-review), and disseminated. In short, communication became more efficient, and research moved forward at a faster rate.
By any measure this technology has developed into an essential facet of academic research. This year, tens of thousands of journals will publish millions of articles which will be read by at least as many researchers.3 Perhaps most importantly, billions of dollars4 in research funding and academic positions will be allocated based largely on academic publication records.5 To support this an ecosystem has evolved comprised of four key players: academics, journals, libraries, and funding agencies, each of whom has somewhat conflicting goals.
1 Einstein’s Miraculous Year: Five Papers that Changed the Face of Physics; Princeton University Press 2005. 2 David Kronick - A History of Scientific & Technical Periodicals: The Origins and Development of the Scientific and Technical Press, 1665-1970, 2nd ed; Scarecrow Press, 1976. 3 Information Research 14, 391 (2009). 4 http://www.quora.com/What-are-the-largest-sources-of-funding-for-scientific-research 5 http://en.wikipedia.org/wiki/Publish_or_perish This system is by no means perfect. Most journals charge significant fees to access their content. For example, a subscription to most of the hard science journals will cost a library thousands of dollars per year.6 Even more astonishingly, access to a single article will cost an individual more than most books.7 This, for a publication written and reviewed by authors who receive absolutely no direct income from the journal. Even if you ignore the money, the current system has costs. Journals and journal articles may encourage the sharing of completed work, but their hegemony means that more open collaboration, or even experiments with different forms of collaboration, is in some sense discouraged.
Likewise, peer-review as it stands is flawed. Peer-review serves an essential role as the gatekeeper to research, allowing readers to approach published literature in a more credulous manner. However, there is no inherent reason that only one or two experts in a field should determine the validity of a paper, that the review process should take months, or that their critiques of the paper should be hidden from other interested parties. This is an accident of history, a result of the logistical issues involved in two-way communication between large groups of people before the advent of the internet.
Innovation In the last few years the internet has significantly disrupted virtually every publishing industry (e.g., music, tv, movies, books, and newspapers). Historically, economic models tend to change whenever a new communication medium is developed,8 so it is surprising to see such little experiment and innovation in the way academic research is published.
To be fair, each of the entrenched players (researchers, journals, libraries, and funding agencies) has made tentative steps online. In physics, for example, major publishers have recently launched online only open access journals.9 The Cornell University library hosts the enormously successful arXiv.10 A number of prominent researchers blog their results and ideas.11 And, although not yet common in physics, some major funding agencies (the NIH in particular), have begun to mandate that any research they fund must be published in open access journals.12
However, institutions tend to try to preserve the problem to which they are the solution.13 No entrenched player is going to produce innovations that threaten their livelihood, and this is
6 http://www.library.illinois.edu/scholcomm/journalcosts.html 7 32 USD when attempting to access e.g., doi:10.1038/nature10602 on November 23, 2011. 8 Tim Wu - The Rise and Fall of Information Empires; Knopf 2010. 9 See e.g., Phys. Rev. X by The American Physical Society, and AIP Advances by The American Institute of Physics. 10 http://arxiv.org 11 Some of our favorites: Cosmic Variance, Ross McKenzie’s Condensed Concepts, Doug Natelson’s Nanoscale Views, Tommaso Dorigo’s A Quantum Diaries Survivor, and Shaun Hendy’s A Measure of Science. 12 http://publicaccess.nih.gov/ 13 Clay Shirky - Cognitive Surplus: Creativity and Generosity in a Connected Age; Penguin 2011. reflected in the innovations they attempt. For example, most open access journals require authors to pay a publication fee. Likewise, authors have proven unwilling to switch from high impact factor journals to less well-regarded open access systems.14
In any case, while open access might be important, it is not revolutionary. It may decrease the cost of access to information but it will not (alone) speed up the rate of discovery or distribution of new ideas; it is a necessary, but not sufficient, part of the future of the academic publishing industry. A truly innovative approach will result in new tools that increase the rate at which human knowledge is developed and disseminated, just as the development of the journal drastically improved the efficiency with which we created new knowledge in the 20th century.
It is hard to say exactly what these new tools will be. Their development will require a great deal of experiment and iteration. Therefore, instead of attempting to a priori identify and build the right tool, we need to develop a framework (and culture) in which experimentation with different forms of collaboration is encouraged.
Incentives In order to build a culture of innovation, we need to understand why there is so little of it today. It is sometimes said that most of economics can be summarized in four words: “people respond to incentives”, and this can often lead to paradoxical outcomes.15 We believe that while the incentive structures created by the journal system originally encouraged a culture of open collaboration, they now stifle innovation.
These incentives are best illustrated by introducing the publon, the facetious particle that is to academic research what an electron is to charge. It is brilliantly described by Peter Koveski: “Originally discovered in Oxford, the publon is the elementary particle of scientific publication. It has long been known that publons are mutually repulsive. The chances of finding more than one publon in a paper are negligible. Even more intriguing is the apparent ability of the same publon to manifest itself at widely separated instants in time. One reason why this has not emerged until now seems to be that a publon can manifest itself with different words and terminology … defeating observations with even the most powerful database scanners.”16
This humorous notion underscores an uneasy truth in academia: your publication record determines your success in obtaining research funding, jobs, and academic prestige. The more you publish the more you get paid.
This is fundamentally a result of the fact that it is hard to spot a good researcher. As the business of research grew during the 20th century, so did the number of researchers. It became important for funding agencies and employers to have an efficient and somewhat objective
14 Hence the perceived need for mandates by funding agencies like the NIH. 15 S. Levitt and S. Dubner - Freakonomics: A Rogue Economist Explores the Hidden Side of Everything; William Morrow 2009. 16 http://www.csse.uwa.edu.au/~pk/publons.html measure of research performance. Publication and citation records fit that bill. They are easily measured, and derivative metrics, like the h-index, are easily calculated.17 In this way professional academic research and the journal developed symbiotically, to the point where it is now almost impossible to imagine a different system for communicating results or allocating funding.
This is our theory: in order to build innovative systems for collaboration we need to modify this incentive structure. A corollary of this is that any new approach must augment the current system, at least initially. Too many players have too much invested in the status quo for a sudden change to occur. Any change in incentive structure will be gradual, but we believe that once those changes reach critical mass everything else will follow.
Our Strategy We intend to create a platform for experimenting with these ideas. Our research indicates that academics are interested in three general features: the ability to discuss published research; better indicators (than citations) of the value of their contributions; and recommendation engines to help them discover relevant research. This is roughly the order in which we intend to develop our platform.
In the first phase, we have built a framework for discussing published journal articles. Specifically, we have created a central repository where academics are able to comment on, or ask questions about, the papers they already rely on to do their research. Technically, this is enabled by algorithms that import journal article metadata from around the web as soon as a user requests it.18 In many ways our approach is to journal articles as stackoverflow is to computer programming questions.19 In a twist that is specific to academia, comments can be extended to create a form of peer-review, augmenting the process behind the original publication of the article.
We firmly believe that academics will only engage if there is some way to measure their contributions and thus signal their worth. Therefore, we are in the process of building a system for rating contributions. It should be possible to rate any comment, review, or publication. With the input of the community we hope that these ratings can be distilled into a set of metrics that provide a detailed, but easily measured, account of contributions to academia.20 Intuitively, an academic that writes an insightful review of an article has contributed more to academic research than an academic who has not, ceteris paribus. These are the contributions we hope to encourage.
In essence we are building a reputation management system. To help highlight contributions
17 http://images.webofknowledge.com/WOK45/help/WOS/h_citationrpt.html 18 As of November 2011 we support 10 major physics journals (including the arXiv), with many more to come soon. 19 http://stackoverflow.com/about 20 http://altmetrics.org/manifesto/ we generate a profile for all authors that have published a paper that is added to our site.21 Upon registering, an academic takes control of their profile and can use it to signal their value in much the same way that a resume, github profile,22 or portfolio is used today. It is important to note that the profile includes standard metrics like publication count, impact factor, and h-index, in alignment with our hypothesis that any change in incentives must begin by augmenting the current system.
New metrics will only become truly relevant after a gradual cultural shift in which a significant number of people begin to value them. This means building a community, and building a community is hard.23 This will be our biggest challenge, and thus our primary focus.24 We are hopeful though; in our experience most academics are genuinely interested in questions about their work and will gain great utility from stimulating discussions,25 particularly if they can signal their contributions.
It is only once a new set of metrics gain traction that journals become superfluous and true innovation in collaboration can occur. We envisage a future where researchers can submit their results, data, and publications to our site, or anywhere else for that matter; where the data are published is not important.26 What matters is what the community thinks of the work. This community is what we intend to build, and once it exists there is no limit to what we can create.
Incidentally, this is the reason we are launching with a focus on the hard sciences. We do have a great deal of domain knowledge in this area, but more importantly the culture is particularly well suited to this sort of experiment. The success of the arXiv is proof of that.
The Future Once a community exists, and the infrastructure for metric driven science is in place, we can experiment with more exciting features. For example, it would be interesting to try to incentivize the sharing of source code and experimental data, and to encourage dozens of readers to review the specific areas of an article in which they are true experts.
These features will become more important as research becomes increasingly multidisciplinary. As Michael Nielsen points out in his must-read book, Reinventing Discovery,27 human knowledge has advanced to the point that any researcher is, by definition, hyper-specialized; there is no way for a single individual to grasp the intricacies and solve the problems of any field, no matter how narrow. Nielsen makes a good case that new forms of collaboration and recommendation engines - systems that encourage serendipitous meetings between those with
21 As of November 2011 we have more than 10,000 profiles on the site. 22 An interesting example of github profiles being used in this manner can be found on linkedin: http:// blog.linkedin.com/2011/03/08/github-linkedin/. 23 http://hanselminutes.com/211/jeff-atwood-on-the-future-of-stack-overflow 24 http://blog.stackoverflow.com/category/community/ 25 http://www.codinghorror.com/blog/2010/06/the-vast-and-endless-sea.html 26 At this point, open access is inevitable. 27 Michael Nielsen - Reinventing Discovery: The New Era of Networked Science; Princeton University Press 2011. a very specific problem and the one or two researchers who may have an idea for how to solve that problem - are essential for a more efficient research economy.
In this paper we have raised a hypothesis for how to change the incentive structures faced by academics but have largely ignored the three other major players in the ecosystem we hope to disrupt: journals, libraries, and funding agencies. While it may be possible to marginalize journals and libraries, our ideas for the future of research will go nowhere if funding agencies and universities continue to rely entirely on conventional publication records to allocate jobs and funding. We anticipate that considerable work will be necessary to convince these institutions to change, and are open to suggestions for ways in which to affect this change.28
These ideas could be implemented by us, or by others. The key point is that we need to initiate a revolution in academic research. Once it begins, there is really no limit to the things we could try.
Join Us In summary, we have begun with a simple goal: to generate a dialog around published research by building a set of metrics that incentivize high-quality discussion. This is a necessary step in the right direction, but it is by no means the end goal.
This publication, Publon #1, not only describes our ideas, but is also the first active article on publons.com, the site that implements these ideas. This means that you are able to discuss, critique, and rate our idea. Help us to answer the question of how to create the future of academic publishing by sharing your thoughts.
But don’t stop there. In order to build a community we need your involvement, so next time you, or one of your colleagues, have a question about a paper you’re reading, ask that question at publons.com. We hope to see you there.
28 The way research funding is allocated is an interesting topic in itself. As it stands today, most research proposals attempt to convince the funding agency of two things: that the research in question is worthwhile, and that the academics proposing the research are the best people for the job. These two functions do not necessarily need to be so closely coupled. There are plenty of other ways to determine which specific ideas are funded. For example, why not create a futures market for voting on the ideas that should be funded? See e.g., http://www.overcomingbias.com/2010/12/let-gambling-save-science.html.