sign in sign up
<<
Home , ArXiv

The future of arXiv and knowledge discovery in open

Steinn Sigurdsson arXiv, [email protected] & Pennsylvania State University University Park, PA 16803

Abstract the 2 pm switch show up first, both in to subscribers and in web page listings. This to arχiv, the server for the physical and a well known effect that the e–prints first in this mathematical , is in its third decade of rank order are disproportionately cited compared to operation. As the flow of new, increases inexorably, the challenges later ranked e–prints (Haque and Ginsparg, 2009, to keep up with and discover research content 2010). While this provides a form of knowledge also become greater. I will discuss the sta- discovery, and one that undoubtedly correlates with tus and future of arχiv, and possibilities and some attributes of the researchers who wrote and plans to make more effective use of the re- submitted the e–print, it is not an optimal technique search database to enhance ongoing research for efficient discovery. efforts. Additional discovery is enabled by the sorting of 1 Introduction e–prints into categories within their subject areas. While this is primarily done by authors at the time arχiv as of the time of writing contains 1, 777, 731 of submission, the ultimate choice of categorization e–prints across 158 categories in 8 different subject is done by arχiv moderators, with the assistance of areas. The e–prints are distributed under license, automated tools. arχiv reserves the right to set the and archived, free to the author and free to the primary category in which the submissions appear, reader. Distribution is fast, by daily e–mail blasts and to assign or remove secondary categories or by category, RSS feeds, and direct web access. later cross listings. arχiv gets approximately a quarter of a million hits per hour, and currently receives about 15,000 new Moderation of Categories: The moderation pro- submissions per month. Each submission, upon ac- cess within arχiv currently occurs in three primary ceptance, is assigned a unique arXiv:###.#####v# stages. On submission the e–print is processed and ID which is stable and maintains version control checked for a number of technical issues. Approx- for revisions. imately 10% of e–prints have some error and are The average category has about five new primary referred back to the author (not counting minor er- submissions per day, but the larger categories have rors which are generally fixed during submission). many dozens of new e–prints per day, and keeping The median time from a submitting author logging up with the literature is accurately likened to try- into the system and a submission being complete ing to drink from a firehose. Finding the research is 34 minutes. you want and need seems to become harder as the During submission an automatic classification tools to access the literature improve. Curating the system scans the paper and recommends a choice flow of and enabling discovery are crit- of categories. For most submissions the choice ical tasks and expediting knowledge discovery will of primary category matches that of the author, in likely enable more rapid progress across a wide other cases the system may recommend a different range of fields. category. The author may choose to accept the new A peculiarity of the arχiv category system is that recommended category, or add it as a secondary the submissions, each day, are ordered by time of category in which to list the submission, or they submission, with the breakpoint being 2 pm eastern may proceed with their original category and ignore time each day. The e–prints submitted first after the system recommendation.

7 Proceedings of the First Workshop on Scholarly Document Processing, pages 7–9 Online, November 19, 2020. c 2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 Each category has an assigned moderator, with 2010). Currently three classifiers are operational: 194 voluntary moderators covering the range of the original full text classifier from Ginsparg; a beta subject categories. Some categories have two or version of a more general broad classification pack- more moderators, generally the high volume cate- age, including full text, run asynchronously; and a gories. At any given time there are gaps in mod- new fast metadata classifier developed by Papers eration, sometimes covered on an ad hoc by with Code. Note that arχiv submissions come with volunteer super–moderators, who may have respon- very sparse metadata. Demanding large amounts sibility for multiple categories. The moderators are of metadata provided by the author puts a burden generally senior PhD researchers who are active in on the author during submission and discourages the field they are moderating. The ideal moderator use of arχiv for rapid distribution of research. is someone who was checking the distribution in Moderators see the classifier score, and a pa- “their” category every morning anyway, someone per with a high score in a particular category not who wants to see first what is new. selected by the author may be queued up for consid- The moderators have primary responsibility for eration by the moderators of that category, for selec- the choice of categories, and whether a submission tion either as a primary category or as a secondary is released, held for further checks, or rejected. A choice of category. A few percent of submissions small fraction of submissions get held for an ex- typically get some category changes, often the addi- tended period (currently about 0.5% are on hold tion of one or more secondary categories for listing. for two or more weeks). Extended holds mostly Category changes also to significant fraction occur due to coordination issues between modera- of holds of submissions, both when a moderator tors, including moderators who are out of action, deems a submission unsuitable for the choice of disagreements on choice of categories, or policy category, or when moderators disagree amongst questions. Submissions may be rejected for being themselves about the choice of category. Orphaned out of scope, not being the type of content arχiv submissions, those rejected from all choices of pri- accepts, or not meeting the threshold for standard mary categories, may be rejected as out of scope. of acceptance in that subject and category. arχiv Moderators will often recommend alternative pos- makes substantial effort to maintain consistent stan- sible choices of primary, or recommend secondary dards across categories and subject areas, but there categories for submission. are some differences in approach and style across The classifiers are imperfect, in particular when academic fields, and where these cross is often trying to determine a fit for the smaller categories, where papers are put on hold or rejected. arχiv is even after balancing, and some broad categories are a curated collection, it is not a general repository. treated as exception cases. Training the classifier arχiv is not the . In order to remain use- is an iterative process, and more work is needed. ful to its community of users arχiv curates content for relevance and interest, while trying to avoid 3 Knowledge Discovery gatekeeping and active refereeing of content. This means there are always borderline cases. The bor- Ultimately researchers and other readers want to derline cases are generally not important in the discover the latest research that is relevant to their aggregate, but they are important to the individ- interests, and to find other relevant results, novel ual authors, and there is a strong motivation not to methods, complementary insights or other useful exclude original or innovative approaches through or interesting knowledge. A lot of discovery comes overly strong filtering. However, there is always from finding your lane and staying in it, the cate- some border between accepting and rejecting, and gories provide useful silos for a significant fraction wherever that border is, there are always edge cases of researchers and push most of the directly rele- which end up being judgement calls. Moving the vant science to that community. Beyond the silo, border does not resolve the issue, it merely moves searches of the literature are useful for discovery, which submissions are borderline. but are often constrained by what can be indexed for searching and how the search algorithm keys 2 Classification in on search terms. Improved search algorithms and federated cross platform searches generally Papers submitted to arχiv are run through a natu- improve prospects for discovery. ral language processing classifier (Ginsparg et al., arχiv currently has formal relations with several

8 entities to expedite cross platform discovery, in- provide bulk access to arχiv contents, providing cluding INSPIRE, ADS, DBLP, both aggregate metadata, and access to processing and . We are working to improve full text. The kaggle dataset is updated periodically discovery including author disambiguation. Au- and is free to use. Text retrieval of any particular thor ID services such as ORCID and Institutional paper still goes through .org. Identifiers like ROR also expedite searches. An ambition of arχiv is to provide custom delivery 4 arχiv Labs of new submissions beyond the current categories, arχiv has set up a framework for us to work on to include among other options, author selection, a range of issues with external partners through types of content, and inclusion and exclusion by arχiv Labs, https://labs.arxiv.org. Currently arχiv keywords and relevance. Labs includes the arχiv Bibliographic Explorer, Balancing this impetus is the danger of loss of a new collaboration with Papers with Code to discovery by browsing, the serendipitous discovery link papers and code (https://paperswithcode.com), that came when browsing a physical journal and and the CORE Recommender (Knoth et al., 2017) finding a surprise article adjacent to the one you (https://core.ac.uk). were seeking, or a topical book you were not aware arχiv Labs is committed to , and of shelved next to those you browsed. Refined and partners working through the framework are ex- narrow searches limit surprises. Sometimes what pected to abide by the general arχiv principles. We you are looking for is not the ”known unknowns” are interested in working with individuals or groups but rather the ”unknown unknowns”. It is tempting on third party services, as well as more structured to consider providing a small fraction of random or services that could be brought in–house and run semi-random search results in searches in the hope from the arχiv side as services to our users, or even of triggering the rare discovery of an unknown. part of our core operations. We are in discussion A more formal process may be more efficient about several other projects. and likelier to succeed, and arχiv is interested in pursuing knowledge discovery techniques, includ- References ing knowledge graphs and novel techniques for finding relevant results that are not adjacent to the Colin B. Clement, Matthew Bierbaum, Kevin P. research area being searched. There are very large O’Keeffe, and Alexander A. Alemi. 2019. On the use of arxiv as a dataset. Research benefits to finding an existing solution to an exper- Repository, arXiv:1905.00075. imental problem, a new computational technique making your modeling tractable, a statistical or , Paul Houle, Thorsten Joachims, and Jae- Hoon Sul. 2010. Last but not least: Additional po- mathematical method making your problem solv- sitional effects on and readership in arxiv. able, or the novel theoretical insight from a differ- Computing Research Repository, arXiv:1010.2757. ent subfield. Asif-ul Haque and Paul Ginsparg. 2009. Positional ef- More broadly we want to find emerging new di- fects on citation and readership in arxiv. Computing rections of research, even before those doing the Research Repository, arXiv:0907.4740. research realize there is an emergent effort which is Asif-ul Haque and Paul Ginsparg. 2010. Last but not headed in a new direction, to see disparate subfields least: Additional positional effects on citation and converge into new synergistic research opportuni- readership in arxiv. Computing Research Reposi- ties, and adjacent subfields diverge to nucleate new tory, arXiv:1010.2757. areas of research. These are hard problems, but Petr Knoth, Lucas Anastasiou, Aristotelis Char- exciting and with very high potential for discovery alampous, Mattero Cancellieri, Samuel Pearce, and speeding up research. Nancy Pontika, and Vaclav Bayer. 2017. To- arχiv core functionality is to get the paper to wards effective research recommender systems for repositories. Computing Research Repository, the reader, but quantity has a quality all of its own arXiv:1705.00578. (Clement et al., 2019). Bulk downloads of content for natural language processing, and other aggregate exploration has been enabled for some time through Amazon’s S3, with the user paying. arχiv has now partnered with kaggle to

9