JOSS Documentation
Open Journals
Sep 28, 2021
Author and Reviewer Guides
1 About this site 3
2 Submitting a paper to JOSS 5
3 Sponsors and affiliates 7 3.1 Submitting a paper to JOSS...... 7 3.1.1 Submission requirements...... 7 3.1.1.1 What we mean by research software...... 8 3.1.1.2 Substantial scholarly effort...... 8 3.1.1.3 Co-publication of science, methods, and software...... 8 3.1.1.3.1 Other venues for reviewing and publishing software packages...... 9 3.1.1.4 Should I write my own software or contribute to an existing package?...... 9 3.1.1.5 Questions? Open an issue to ask...... 9 3.1.2 Typical paper submission flow...... 9 3.1.3 What should my paper contain?...... 9 3.1.4 Example paper and bibliography...... 10 3.1.5 Checking that your paper compiles...... 13 3.1.5.1 JOSS paper preview service...... 13 3.1.5.2 GitHub Action...... 14 3.1.5.3 Docker...... 14 3.1.6 Submitting your paper...... 14 3.1.7 No submission fees...... 14 3.1.8 Preprint Policy...... 14 3.1.9 Authorship...... 14 3.1.10 Submissions using proprietary languages/development environments...... 15 3.1.11 The review process...... 15 3.1.12 Confidential requests...... 15 3.2 Reviewing for JOSS...... 15 3.2.1 Guiding principles...... 15 3.2.2 JOSS Conflict of Interest Policy...... 16 3.3 Review criteria...... 16 3.3.1 The JOSS paper...... 16 3.3.2 Review items...... 17 3.3.2.1 Software license...... 17 3.3.2.2 Substantial scholarly effort...... 17 3.3.2.3 Documentation...... 18 3.3.2.3.1 A statement of need...... 18
i 3.3.2.3.2 Installation instructions...... 18 3.3.2.3.3 Example usage...... 18 3.3.2.3.4 API documentation...... 18 3.3.2.3.5 Community guidelines...... 18 3.3.2.4 Functionality...... 18 3.3.2.5 Tests...... 19 3.3.3 Other considerations...... 19 3.3.3.1 Authorship...... 19 3.3.3.2 An important note about ‘novel’ software and citations of relevant work...... 19 3.3.3.3 What happens if the software I’m reviewing doesn’t meet the JOSS criteria?.... 19 3.3.3.4 What about submissions that rely upon proprietary languages/development environ- ments?...... 19 3.4 Review checklist...... 19 3.4.1 Conflict of interest...... 20 3.4.2 Code of Conduct...... 20 3.4.3 General checks...... 20 3.4.4 Functionality...... 20 3.4.5 Documentation...... 20 3.4.6 Software paper...... 21 3.5 Editorial Guide...... 21 3.5.1 Pre-review...... 21 3.5.1.1 How papers are assigned to editors...... 21 3.5.1.2 Finding reviewers...... 22 3.5.1.3 Starting the review...... 23 3.5.2 Review...... 23 3.5.2.1 Adding a new reviewer once the review has started...... 23 3.5.3 After reviewers recommend acceptance...... 24 3.5.4 Handling of papers published together with AAS publishing...... 24 3.5.5 Processing of rOpenSci-reviewed and accepted submissions...... 25 3.5.6 Rejecting a paper...... 25 3.5.6.1 Voting on papers flagged as potentially out of scope...... 25 3.5.7 Sample messages for authors and reviewers...... 26 3.5.7.1 Sample email to potential reviewers...... 26 3.5.7.2 Query scope of submission...... 26 3.5.7.3 GitHub invite to potential reviewers...... 27 3.5.7.4 Message to reviewers at the start of a review...... 27 3.5.7.5 Message to authors at the end of a review...... 27 3.5.7.6...... 27 3.5.8 Overview of editorial process...... 27 3.5.9 Visualization of editorial flow...... 29 3.5.10 Expectations on JOSS editors...... 30 3.5.10.1 Responding to editorial assignments...... 30 3.5.10.2 Continued attention to assigned submissions...... 30 3.5.11 Out of office...... 30 3.5.12 Editorial buddy...... 30 3.5.13 Managing notifications...... 31 3.5.13.1 Things you should do when joining the editorial team...... 31 3.6 Interacting with Whedon...... 32 3.6.1 Author commands...... 33 3.6.1.1 Compiling papers...... 33 3.6.1.1.1 Compiling papers from a non-default branch...... 34 3.6.1.2 Finding reviewers...... 34 3.6.2 Editorial commands...... 34 3.6.2.1 Assigning an editor...... 34 ii 3.6.2.2 Inviting an editor...... 34 3.6.2.3 Adding and removing reviewers...... 34 3.6.2.4 Starting the review...... 35 3.6.2.5 Reminding reviewers and authors...... 35 3.6.2.6 Setting the software archive...... 35 3.6.2.7 Changing the software version...... 36 3.6.3 Accepting a paper (dry run)...... 36 3.6.3.1 Check references...... 36 3.6.4 Accepting a paper (for real)...... 36 3.7 Installing the JOSS application...... 36 3.7.1 Setting up a local development environment...... 37 3.7.2 Deploying your JOSS application...... 37 3.7.3 Making modifications to launch your own site...... 38 3.7.3.1 Modifying your site configuration...... 38 3.7.3.1.1 Updating your application database...... 39 3.7.3.2 Modifying your site contents...... 39 3.7.4 Deploying your Whedon-API Application...... 39 3.7.5 Modifying your Whedon-API deployment...... 40 3.7.5.1 Modifying your Whedon-API configuration...... 40
iii iv JOSS Documentation
The Journal of Open Source Software (JOSS) is a developer friendly journal for research software packages. JOSS is an academic journal (ISSN 2475-9066) with a formal peer-review process that is designed to improve the quality of the software submitted. Upon acceptance into JOSS, we mint a CrossRef DOI for your paper and we list it on the JOSS website.
Author and Reviewer Guides 1 JOSS Documentation
2 Author and Reviewer Guides CHAPTER 1
About this site
This site contains documentation for authors interested in submitting to JOSS, reviewers who have generously volun- teered their time to review submissions, and editors who manage the JOSS editorial process. If you’re interested in learning more about JOSS, you might want to read: • Our announcement blog post describing some of the motivations for starting a new journal • The paper in Computing in Science and Engineering introducing JOSS • The paper in PeerJ CS describing the first year of JOSS • The about page on the main JOSS site
3 JOSS Documentation
4 Chapter 1. About this site CHAPTER 2
Submitting a paper to JOSS
If you’d like to submit a paper to JOSS, please read the author submission guidelines in the Submitting a paper to JOSS section.
5 JOSS Documentation
6 Chapter 2. Submitting a paper to JOSS CHAPTER 3
Sponsors and affiliates
JOSS is a proud affiliate of the Open Source Initiative. As such, we are committed to public support for open source software and the role OSI plays therein. In addition, Open Journals (the parent entity behind JOSS) is a NumFOCUS- sponsored project.
3.1 Submitting a paper to JOSS
If you’ve already developed a fully featured research code, released it under an OSI-approved license, and written good documentation and tests, then we expect that it should take perhaps an hour or two to prepare and submit your paper to JOSS. But please read these instructions carefully for a streamlined submission.
3.1.1 Submission requirements
• The software must be open source as per the OSI definition. • The software must have an obvious research application. • You must be a major contributor to the software you are submitting, and have a GitHub account to participate in the review process. • Your paper must not focus on new research results accomplished with the software.
7 JOSS Documentation
• Your paper (paper.md and BibTeX files, plus any figures) must be hosted in a Git-based repository together with your software (although they may be in a short-lived branch which is never merged with the default). In addition, the software associated with your submission must: • Be stored in a repository that can be cloned without registration. • Be stored in a repository that is browsable online without registration. • Have an issue tracker that is readable without registration. • Permit individuals to create issues/file tickets against your repository.
3.1.1.1 What we mean by research software
JOSS publishes articles about research software. This definition includes software that: solves complex modeling problems in a scientific context (physics, mathematics, biology, medicine, social science, neuroscience, engineering); supports the functioning of research instruments or the execution of research experiments; extracts knowledge from large data sets; offers a mathematical library, or similar. While useful for many areas of research, pre-trained machine learning models are not in-scope for JOSS.
3.1.1.2 Substantial scholarly effort
JOSS publishes articles about software that represent substantial scholarly effort on the part of the authors. Your software should be a significant contribution to the available open source software that either enables some new research challenges to be addressed or makes addressing research challenges significantly better (e.g., faster, easier, simpler). As a rule of thumb, JOSS’ minimum allowable contribution should represent not less than three months of work for an individual. Some factors that may be considered by editors and reviewers when judging effort include: • Age of software (is this a well-established software project) / length of commit history. • Number of commits. • Number of authors. • Total lines of code (LOC). Submissions under 1000 LOC will usually be flagged, those under 300 LOC will be desk rejected. • Whether the software has already been cited in academic papers. • Whether the software is sufficiently useful that it is likely to be cited by your peer group. In addition, JOSS requires that software should be feature-complete (i.e. no half-baked solutions) and designed for maintainable extension (not one-off modifications of existing tools). “Minor utility” packages, including “thin” API clients, and single-function packages are not acceptable.
3.1.1.3 Co-publication of science, methods, and software
Sometimes authors prepare a JOSS publication alongside a contribution describing a science application, details of algorithm development, and/or methods assessment. In this circumstance, JOSS considers submissions for which the implementation of the software itself reflects a substantial scientific effort. This may be represented by the design of the software, the implementation of the algorithms, creation of tutorials, or any other aspect of the software. We ask that authors indicate whether related publications (published, in review, or nearing submission) exist as part of submitting to JOSS.
8 Chapter 3. Sponsors and affiliates JOSS Documentation
3.1.1.3.1 Other venues for reviewing and publishing software packages
Authors wishing to publish software deemed out of scope for JOSS have a few options available to them: • Follow GitHub’s guide on how to create a permanent archive and DOI for your software. This DOI can then be used by others to cite your work. • Enquire whether your software might be considered by communities such as rOpenSci and pyOpenSci.
3.1.1.4 Should I write my own software or contribute to an existing package?
While we are happy to review submissions in standalone repositories, we also review submissions that are significant contributions made to existing packages. It is often better to have an integrated library or package of methods than a large number of single-method packages.
3.1.1.5 Questions? Open an issue to ask
Authors wishing to make a pre-submission enquiry should open an issue on the JOSS repository.
3.1.2 Typical paper submission flow
Before you submit, you should: • Make your software available in an open repository (GitHub, Bitbucket, etc.) and include an OSI approved open source license. • Make sure that the software complies with the JOSS review criteria. In particular, your software should be full-featured, well-documented, and contain procedures (such as automated tests) for checking correctness. • Write a short paper in Markdown format using paper.md as file name, including a title, summary, author names, affiliations, and key references. See our example paper to follow the correct format. • (Optional) create a metadata file describing your software and include it in your repository. We provide a script that automates the generation of this metadata.
3.1.3 What should my paper contain?
Important: Begin your paper with a summary of the high-level functionality of your software for a non-specialist reader. Avoid jargon in this section.
JOSS welcomes submissions from broadly diverse research areas. For this reason, we require that authors include in the paper some sentences that explain the software functionality and domain of use to a non-specialist reader. We also require that authors explain the research applications of the software. The paper should be between 250-1000 words. Authors submitting papers significantly longer than 1000 words may be asked to reduce the length of their paper. Your paper should include: • A list of the authors of the software and their affiliations, using the correct format (see the example below). • A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience. •A Statement of Need section that clearly illustrates the research purpose of the software.
3.1. Submitting a paper to JOSS 9 JOSS Documentation
• A list of key references, including to other software addressing related needs. Note that the references should include full names of venues, e.g., journals and conferences, not abbreviations only understood in the context of a specific discipline. • Mention (if applicable) a representative set of past or ongoing research projects using the software and recent scholarly publications enabled by it. • Acknowledgement of any financial support. As this short list shows, JOSS papers are only expected to contain a limited set of metadata (see example below), a Statement of Need, Summary, Acknowledgements, and References sections. You can look at an example accepted paper. Given this format, a “full length” paper is not permitted, and software documentation such as API (Applica- tion Programming Interface) functionality should not be in the paper and instead should be outlined in the software documentation.
Important: Your paper will be reviewed by two or more reviewers in a public GitHub issue. Take a look at the review checklist and review criteria to better understand how your submission will be reviewed.
3.1.4 Example paper and bibliography
This example paper.md is adapted from Gala: A Python package for galactic dynamics by Adrian M. Price-Whelan http://doi.org/10.21105/joss.00388:
--- title: 'Gala: A Python package for galactic dynamics' tags: - Python - astronomy - dynamics - galactic dynamics - milky way authors: - name: Adrian M. Price-Whelan^[co-first author] # note this makes a footnote
˓→saying 'co-first author' orcid: 0000-0003-0872-7098 affiliation: "1, 2" # (Multiple affiliations must be quoted) - name: Author Without ORCID^[co-first author] # note this makes a footnote saying
˓→'co-first author' affiliation: 2 - name: Author with no affiliation^[corresponding author] affiliation: 3 affiliations: - name: Lyman Spitzer, Jr. Fellow, Princeton University index: 1 - name: Institution Name index: 2 - name: Independent Researcher index: 3 date: 13 August 2017 bibliography: paper.bib
# Optional fields if submitting to a AAS journal too, see this blog post: # https://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing aas-doi: 10.3847/xxxxx <- update this with the DOI from AAS once you know it. aas-journal: Astrophysical Journal <- The name of the AAS journal. (continues on next page)
10 Chapter 3. Sponsors and affiliates JOSS Documentation
(continued from previous page) ---
# Summary
The forces on stars, galaxies, and dark matter under external gravitational fields lead to the dynamical evolution of structures in the universe. The orbits of these bodies are therefore key to understanding the formation, history, and future state of galaxies. The field of "galactic dynamics," which aims to model the gravitating components of galaxies to study their structure and evolution, is now well-established, commonly taught, and frequently used in astronomy. Aside from toy problems and demonstrations, the majority of problems require efficient numerical tools, many of which require the same base code (e.g., for performing numerical orbit integration).
# Statement of need
`Gala` is an Astropy-affiliated Python package for galactic dynamics. Python enables wrapping low-level languages (e.g., C) for speed without losing flexibility or ease-of-use in the user-interface. The API for `Gala` was designed to provide a class-based and user-friendly interface to fast (C or Cython-optimized) implementations of common operations such as gravitational potential and force evaluation, orbit integration, dynamical transformations, and chaos indicators for nonlinear dynamics. `Gala` also relies heavily on and interfaces well with the implementations of physical units and astronomical coordinate systems in the `Astropy` package [@astropy] (`astropy.units` and `astropy.coordinates`).
`Gala` was designed to be used by both astronomical researchers and by students in courses on gravitational dynamics or astronomy. It has already been used in a number of scientific publications [@Pearson:2017] and has also been used in graduate courses on Galactic dynamics to, e.g., provide interactive visualizations of textbook material [@Binney:2008]. The combination of speed, design, and support for Astropy functionality in `Gala` will enable exciting scientific explorations of forthcoming data releases from the *Gaia* mission [@gaia] by students and experts alike.
# Mathematics
Single dollars ($) are required for inline mathematics e.g. $f(x) = e^{\pi/x}$
Double dollars make self-standing equations:
$$\Theta(x) = \left\{\begin{array}{l} 0\textrm{ if } x < 0\cr 1\textrm{ else} \end{array}\right.$$
You can also use plain \LaTeX for equations \begin{equation}\label{eq:fourier} \hat f(\omega) = \int_{-\infty}^{\infty} f(x) e^{i\omega x} dx \end{equation} and refer to \autoref{eq:fourier} from text.
# Citations
Citations to entries in paper.bib should be in [rMarkdown](http://rmarkdown.rstudio.com/authoring_bibliographies_and_citations.html) (continues on next page)
3.1. Submitting a paper to JOSS 11 JOSS Documentation
(continued from previous page) format.
If you want to cite a software repository URL (e.g. something on GitHub without a
˓→preferred citation) then you can do it with the example BibTeX entry below for @fidgit.
For a quick reference, the following citation commands can be used: - `@author:2001` -> "Author et al. (2001)" - `[@author:2001]` -> "(Author et al., 2001)" - `[@author1:2001; @author2:2001]` -> "(Author1 et al., 2001; Author2 et al., 2002)"
# Figures
Figures can be included like this:  and referenced from text using \autoref{fig:example}.
Figure sizes can be customized by adding an optional second parameter: { width=20% }
# Acknowledgements
We acknowledge contributions from Brigitta Sipocz, Syrtis Major, and Semyeong Oh, and support from Kathryn Johnston during the genesis of this project.
# References
Example paper.bib file:
@article{Pearson:2017, url= {http://adsabs.harvard.edu/abs/2017arXiv170304627P}, Archiveprefix= {arXiv}, Author= {{Pearson}, S. and {Price-Whelan}, A.~M. and {Johnston}, K.~V.}, Eprint={1703.04627}, Journal= {ArXiv e-prints}, Keywords= {Astrophysics- Astrophysics of Galaxies}, Month= mar, Title= {{Gaps in Globular Cluster Streams: Pal5 and the Galactic Bar}}, Year= 2017 }
@book{Binney:2008, url= {http://adsabs.harvard.edu/abs/2008gady.book.....B}, Author= {{Binney}, J. and {Tremaine}, S.}, Booktitle= {Galactic Dynamics: Second Edition, by James Binney and Scott
˓→Tremaine.~ISBN 978-0-691-13026-2 (HB).~Published by Princeton University Press,
˓→Princeton, NJ USA, 2008.}, Publisher= {Princeton University Press}, Title= {{Galactic Dynamics: Second Edition}}, Year= 2008 }
@article{gaia, author= {{Gaia Collaboration}}, title="{The Gaia mission}", journal= {Astronomy and Astrophysics}, archivePrefix="arXiv", (continues on next page)
12 Chapter 3. Sponsors and affiliates JOSS Documentation
(continued from previous page) eprint={1609.04153}, primaryClass="astro-ph.IM", keywords= {space vehicles: instruments, Galaxy: structure, astrometry,
˓→parallaxes, proper motions, telescopes}, year= 2016, month= nov, volume= 595, doi={10.1051/0004-6361/201629272}, url= {http://adsabs.harvard.edu/abs/2016A%26A...595A...1G}, }
@article{astropy, author= {{Astropy Collaboration}}, title="{Astropy: A community Python package for astronomy}", journal= {Astronomy and Astrophysics}, archivePrefix="arXiv", eprint={1307.6212}, primaryClass="astro-ph.IM", keywords= {methods: data analysis, methods: miscellaneous, virtual observatory
˓→tools}, year= 2013, month= oct, volume= 558, doi={10.1051/0004-6361/201322068}, url= {http://adsabs.harvard.edu/abs/2013A%26A...558A..33A} }
@misc{fidgit, author={A.M. Smith and K. Thaney and M. Hahnel}, title= {Fidgit: An ungodly union of GitHub and Figshare}, year={2020}, publisher= {GitHub}, journal= {GitHub repository}, url= {https://github.com/arfon/fidgit} }
Note that the paper ends with a References heading, and the references are built automatically from the content in the .bib file. You should enter in-text citations in the paper body following correct Markdown citation syntax. Also note that the references include full names of venues, e.g., journals and conferences, not abbreviations only understood in the context of a specific discipline.
3.1.5 Checking that your paper compiles
JOSS uses Pandoc to compile papers from their Markdown form into a PDF. There are a few different ways you can test that your paper is going to compile properly for JOSS:
3.1.5.1 JOSS paper preview service
Visit https://whedon.theoj.org and enter your repository address (and custom branch if you’re using one). Note that your repository must be world-readable (i.e., it cannot require a login to access).
3.1. Submitting a paper to JOSS 13 JOSS Documentation
3.1.5.2 GitHub Action
If you’re using GitHub for your repository, you can use the Open Journals GitHub Action to automatically compile your paper each time you update your repository. The PDF is available via the Actions tab in your project and click on the latest workflow run. The zip archive file (including the paper.pdf) is listed in the run’s Artifacts section.
3.1.5.3 Docker
If you have Docker installed on your local machine, you can use the same Docker Image to compile a draft of your paper locally. In the example below, the paper.md file is in the paper directory. Upon successful execution of the command, the paper.pdf file will be created in the same location as the paper.md file:
docker run --rm \ --volume $PWD/paper:/data \ --user $(id -u):$(id -g) \ --env JOURNAL=joss \ openjournals/paperdraft
3.1.6 Submitting your paper
Submission is as simple as: • Filling in the short submission form • Waiting for the managing editor to start a pre-review issue over in the JOSS reviews repository: https://github.com/openjournals/joss-reviews
3.1.7 No submission fees
There are no fees for submitting or publishing in JOSS. You can read more about our cost and sustainability model.
3.1.8 Preprint Policy
Authors are welcome to submit their papers to a preprint server (arXiv, bioRxiv, SocArXiv, PsyArXiv etc.) at any point before, during, or after the submission and review process. Submission to a preprint server is not considered a previous publication.
3.1.9 Authorship
Purely financial (such as being named on an award) and organizational (such as general supervision of a research group) contributions are not considered sufficient for co-authorship of JOSS submissions, but active project direction and other forms of non-code contributions are. The authors themselves assume responsibility for deciding who should be credited with co-authorship, and co-authors must always agree to be listed. In addition, co-authors agree to be accountable for all aspects of the work, and to notify JOSS if any retraction or correction of mistakes are needed after publication.
14 Chapter 3. Sponsors and affiliates JOSS Documentation
3.1.10 Submissions using proprietary languages/development environments
We strongly prefer software that doesn’t rely upon proprietary (paid for) development environments/programming languages. However, provided your submission meets our requirements (including having a valid open source license) then we will consider your submission for review. Should your submission be accepted for review, we may ask you, the submitting author, to help us find reviewers who already have the required development environment installed.
3.1.11 The review process
After submission: • An Associate Editor-in-Chief will carry out an initial check of your submission, and proceed to assign a handling editor. • The handling editor will assign two or more JOSS reviewers, and the review will be carried out in the JOSS reviews repository. • Authors will respond to reviewer-raised issues (if any are raised) on the submission repository’s issue tracker. Reviewer and editor contributions, like any other contributions, should be acknowledged in the repository. • Upon successful completion of the review, authors will make a tagged release of the software, and deposit a copy of the repository with a data-archiving service such as Zenodo or figshare, get a DOI for the archive, and update the review issue thread with the version number and DOI. • After we assign a DOI for your accepted JOSS paper, its metadata is deposited with CrossRef and listed on the JOSS website. • The review issue will be closed, and an automatic tweet from @JOSS_TheOJ will announce it! If you want to learn more details about the review process, take a look at the reviewer guidelines.
3.1.12 Confidential requests
Please write [email protected] with confidential matters such as retraction requests, report of misconduct, and retroac- tive author name changes. In case of a name change, the DOI will be unchanged and the paper will be updated without publishing a correction notice or notifying co-authors. JOSS will also update CrossRef metadata.
3.2 Reviewing for JOSS
Firstly, thank you so much for agreeing to review for the Journal of Open Source Software (JOSS), we’re delighted to have your help. This document is designed to outline our editorial guidelines and help you understand our requirements for accepting a submission into the JOSS. Our review process is based on a tried-and-tested approach of the rOpenSci collaboration.
3.2.1 Guiding principles
We like to think of JOSS as a ‘developer friendly’ journal. That is, if the submitting authors have followed best practices (have documentation, tests, continuous integration, and a license) then their review should be rapid.
3.2. Reviewing for JOSS 15 JOSS Documentation
For those submissions that don’t quite meet the bar, please try to give clear feedback on how authors could improve their submission. A key goal of JOSS is to raise the quality of research software generally and you (the experienced reviewer) are well placed to give this feedback. A JOSS review involves checking submissions against a checklist of essential software features and details in the submitted paper. This should be objective, not subjective; it should be based on the materials in the submission as perceived without distortion by personal feelings, prejudices, or interpretations. We encourage reviewers to file issues against the submitted repository’s issue tracker. When you have completed your review, please leave a comment in the review issue saying so. You can include in your review links to any new issues that you the reviewer believe to be impeding the acceptance of the repository. (Similarly, if the submitted repository is a GitHub repository, mentioning the review issue URL in the submitted repository’s issue tracker will create a mention in the review issue’s history.)
3.2.2 JOSS Conflict of Interest Policy
The definition of a conflict of Interest in peer review is a circumstance that makes you “unable to make an impartial scientific judgment or evaluation.” (PNAS Conflict of Interest Policy). JOSS is concerned with avoiding any actual conflicts of interest, and being sufficiently transparent that we avoid the appearance of conflicts of interest as well. As a reviewer (or editor), COIs are your present or previous association with any authors of a submission: recent (past four years) collaborators in funded research or work that is published; and lifetime for the family members, business partners, and thesis student/advisor or mentor. In addition, your recent (past year) association with the same organization of a submitter is a COI, for example, being employed at the same institution. If you have a conflict of interest with a submission, you should disclose the specific reason to the submissions’ editor (or to an EiC if you are an editor, or to the other EiCs if you are an EiC). This may lead to you not being able to review or edit the submission, but some conflicts may be recorded and then waived, and if you think you are able to make an impartial assessment of the work, you should request that the conflict be waived. For example, if you and a submitter were two of 2000 authors of a high energy physics paper but did not actually collaborate. Or if you and a submitter worked together 6 years ago, but due to delays in the publishing industry, a paper from that collaboration with both of you as authors was published 2 year ago. Or if you and a submitter are both employed by the same very large organization but in different units without any knowledge of each other. Declaring actual, perceived, and potential conflicts of interest is required under professional ethics. If in doubt: ask the editors.
3.3 Review criteria
3.3.1 The JOSS paper
As outlined in the submission guidelines provided to authors, the JOSS paper (the compiled PDF associated with this submission) should only include: • A list of the authors of the software and their affiliations. • A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience. • A clear statement of need that illustrates the purpose of the software. • A description of how this software compares to other commonly-used packages in this research area. • Mentions (if applicable) of any ongoing research projects using the software or recent scholarly publications enabled by it.
16 Chapter 3. Sponsors and affiliates JOSS Documentation
• A list of key references including a link to the software archive.
Important: Note the paper should not include software documentation such as API (Application Programming Interface) functionality, as this should be outlined in the software documentation.
3.3.2 Review items
Important: Note, if you’ve not yet been involved in a JOSS review, you can see an example JOSS review checklist here.
3.3.2.1 Software license
There should be an OSI approved license included in the repository. Common licenses such as those listed on chooseal- icense.com are preferred. Note there should be an actual license file present in the repository not just a reference to the license. Acceptable: A plain-text LICENSE or COPYING file with the contents of an OSI approved licenseNot acceptable: A phrase such as ‘MIT license’ in a README file
3.3.2.2 Substantial scholarly effort
Reviewers should verify that the software represents substantial scholarly effort. As a rule of thumb, JOSS’ minimum allowable contribution should represent not less than three months of work for an individual. Signals of effort may include: • Age of software (is this a well-established software project) / length of commit history. • Number of commits. • Number of authors. • Lines of code (LOC): These statistics are usually reported by Whedon in the pre-review issue thread. • Whether the software has already been cited in academic papers. • Whether the software is sufficiently useful that it is likely to be cited by other researchers working in this domain. These guidelines are not meant to be strictly prescriptive. Recently released software may not have been around long enough to gather citations in academic literature. While some authors contribute openly and accrue a long and rich commit history before submitting, others may upload their software to GitHub shortly before submitting their JOSS paper. Reviewers should rely on their expert understanding of their domain to judge whether the software is of broad interest (likely to be cited by other researchers) or more narrowly focused around the needs of an individual researcher or lab.
Note: The decision on scholarly effort is ultimately one made by JOSS editors. Reviewers are asked to flag sub- missions of questionable scope during the review process so that the editor can bring this to the attention of the JOSS editorial team.
3.3. Review criteria 17 JOSS Documentation
3.3.2.3 Documentation
There should be sufficient documentation for you, the reviewer to understand the core functionality of the software under review. A high-level overview of this documentation should be included in a README file (or equivalent). There should be:
3.3.2.3.1 A statement of need
The authors should clearly state what problems the software is designed to solve and who the target audience is.
3.3.2.3.2 Installation instructions
There should be a clearly-stated list of dependencies. Ideally these should be handled with an automated package management solution. Good: A package management file such as a Gemfile or package.json or equivalent OK: A list of depen- dencies to install Bad (not acceptable): Reliance on other software not listed by the authors
3.3.2.3.3 Example usage
The authors should include examples of how to use the software (ideally to solve real-world analysis problems).
3.3.2.3.4 API documentation
Reviewers should check that the software API is documented to a suitable level. Good: All functions/methods are documented including example inputs and outputs OK: Core API func- tionality is documented Bad (not acceptable): API is undocumented
Note: The decision on API documentation is left largely to the discretion of the reviewer and their experience of evaluating the software.
3.3.2.3.5 Community guidelines
There should be clear guidelines for third-parties wishing to: • Contribute to the software • Report issues or problems with the software • Seek support
3.3.2.4 Functionality
Reviewers are expected to install the software they are reviewing and to verify the core functionality of the software.
18 Chapter 3. Sponsors and affiliates JOSS Documentation
3.3.2.5 Tests
Authors are strongly encouraged to include an automated test suite covering the core functionality of their software. Good: An automated test suite hooked up to continuous integration (GitHub Actions, Circle CI, or sim- ilar) OK: Documented manual steps that can be followed to objectively check the expected functionality of the software (e.g., a sample input file to assert behavior) Bad (not acceptable): No way for you, the reviewer, to objectively assess whether the software works
3.3.3 Other considerations
3.3.3.1 Authorship
As part of the review process, you are asked to check whether the submitting author has made a ‘substantial contribu- tion’ to the submitted software (as determined by the commit history) and to check that ‘the full list of paper authors seems appropriate and complete?’ As discussed in the submission guidelines for authors, authorship is a complex topic with different practices in different communities. Ultimately, the authors themselves are responsible for deciding which contributions are sufficient for co-authorship, although JOSS policy is that purely financial contributions are not considered sufficient. Your job as a reviewer is to check that the list of authors appears reasonable, and if it’s not obviously complete/correct, to raise this as a question during the review.
3.3.3.2 An important note about ‘novel’ software and citations of relevant work
Submissions that implement solutions already solved in other software packages are accepted into JOSS provided that they meet the criteria listed above and cite prior similar work. Reviewers should point out relevant published work which is not yet cited.
3.3.3.3 What happens if the software I’m reviewing doesn’t meet the JOSS criteria?
We ask that reviewers grade submissions in one of three categories: 1) Accept 2) Minor Revisions 3) Major Revisions. Unlike some journals we do not reject outright submissions requiring major revisions - we’re more than happy to give the author as long as they need to make these modifications/improvements.
3.3.3.4 What about submissions that rely upon proprietary languages/development environments?
As outlined in our author guidelines, submissions that rely upon a proprietary/closed source language or development environment are acceptable provided that they meet the other submission requirements and that you, the reviewer, are able to install the software & verify the functionality of the submission as required by our reviewer guidelines. If an open source or free variant of the programming language exists, feel free to encourage the submitting author to consider making their software compatible with the open source/free variant.
3.4 Review checklist
JOSS reviews are checklist-driven. That is, there is a checklist for each JOSS reviewer to work through when com- pleting their review. A JOSS review is generally considered incomplete until the reviewer has checked off all of their checkboxes. Below is an example of the review checklist for the Yellowbrick JOSS submission.
3.4. Review checklist 19 JOSS Documentation
Important: Note this section of our documentation only describes the JOSS review checklist. Authors and reviewers should consult the review criteria to better understand how these checklist items should be interpreted.
3.4.1 Conflict of interest
• I confirm that I have read the JOSS conflict of interest policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.
3.4.2 Code of Conduct
• I confirm that I read and will adhere to the JOSS code of conduct.
3.4.3 General checks
• Repository: Is the source code for this software available at the repository url? • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license? • Contribution and authorship: Has the submitting author made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
3.4.4 Functionality
• Installation: Does installation proceed as outlined in the documentation? • Functionality: Have the functional claims of the software been confirmed? • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)
3.4.5 Documentation
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is? • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution. • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems). • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)? • Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified? • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support
20 Chapter 3. Sponsors and affiliates JOSS Documentation
3.4.6 Software paper
• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided? • A statement of need: Does the paper have a section titled ‘Statement of Need’ that clearly states what problems the software is designed to solve and who the target audience is? • State of the field: Do the authors describe how this software compares to other commonly-used packages? • Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)? • References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?
3.5 Editorial Guide
The Journal of Open Source Software (JOSS) conducts all peer review and editorial processes in the open, on the GitHub issue tracker. JOSS editors manage the review workflow with the help of our bot, @whedon. The bot is summoned with commands typed directly on the GitHub review issues. For a list of commands, type: @whedon commands.
Note: To learn more about @whedon’s functionalities, take a look at our dedicated guide.
3.5.1 Pre-review
Once a submission comes in, it will be in the queue for a quick check by the Editor-in-chief (EiC). From there, it moves to a PRE-REVIEW issue, where the EiC will assign a handling editor, and the author can suggest reviewers. Initial direction to the authors for improving the paper can already happen here, especially if the paper lacks some requested sections.
Important: If the paper is out-of-scope for JOSS, editors assess this and notify the author in the PRE-REVIEW issue.
Editors can flag submissions of questionable scope using the command @whedon query scope. The EiC assigns an editor (or a volunteering editor self-assigns) with the command @whedon assign @username as editor in a comment.
Note: If a paper is submitted without a recommended editor, it will show up in the weekly digest email under the category ‘Papers currently without an editor.’ Please review this weekly email and volunteer to edit papers that look to be in your domain. If you choose to be an editor in the issue thread type the command @whedon assign @yourhandle as editor or simply @whedon assign me as editor
3.5.1.1 How papers are assigned to editors
By default, unless an editor volunteers, the Associated Editor-in-chief (AEiC) on duty will attempt to assign an incom- ing paper to the most suitable handling editor. While AEiCs will make every effort to match a submission with the
3.5. Editorial Guide 21 JOSS Documentation most appropriate editor, there are a number of situations where an AEiC may assign a paper to an editor that doesn’t fit entirely within the editor’s research domains: • If there’s no obvious fit to any of the JOSS editors • If the most suitable editor is already handling a large number of papers • If the chosen editor has a lighter editorial load than other editors In most cases, an AEiC will ask one or more editors to edit a submission (e.g. @editor1, @editor 2 - would one of you be willing to edit this submission for JOSS). If the editor doesn’t re- spond within ~3 working days, the AEiC may assign the paper to the editor regardless. Editors may also be invited to edit over email when an AEiC runs the command @whedon invite @editor1 as editor.
3.5.1.2 Finding reviewers
At this point, the handling editor’s job is to identify reviewers who have sufficient expertise in the field of software and in the field of the submission. JOSS papers have to have a minimum of two reviewers per submission, except for papers that have previously been peer-reviewed via rOpenSci. In some cases, the editor also might want to formally add themself as one of the reviewers. If the editor feels particularly unsure of the submission, a third (or fourth) reviewer can be recruited. To recruit reviewers, the handling editor can mention them in the PRE-REVIEW issue with their GitHub handle, ping them on Twitter, or email them. After expressing initial interest, candidate reviewers may need a longer explanation via email. See sample reviewer invitation email, below. Reviewer Considerations • It is rare that all reviewers have the expertise to cover all aspects of a submission (e.g., knows the language really well and knows the scientific discipline well). As such, a good practice is to try and make sure that between the two or three reviewers, all aspects of the submission are covered. • Selection and assignment of reviewers should adhere to the JOSS COI policy. Potential ways to find reviewers Finding reviewers can be challenging, especially if a submission is outside of your immediate area of expertise. Some strategies you can use to identify potential candidates: • Search the reviewer spreadsheet of volunteer reviewers. – When using this spreadsheet, pay attention to the number of reviews this individual is already doing to avoid overloading them. – It can be helpful to use the “Data > Filter Views” capability to temporarily filter the table view to include only people with language or domain expertise matching the paper. • Ask the author(s): You are free to ask the submitting author to suggest possible reviewers by using the reviewer spreadsheet and also people from their professional network. In this situation, the editor still needs to verify that their suggestions are appropriate. • Use your professional network: You’re welcome to invite people you know of who might be able to give a good review. • Search Google and GitHub for related work, and write to the authors of that related work. – You might like to try this tool from @dfm. • Ask on social networks: Sometimes asking on Twitter for reviewers can identify good candidates. • Check the work being referenced in the submission:
22 Chapter 3. Sponsors and affiliates JOSS Documentation
– Authors of software that is being built on might be interested in reviewing the submission. – Users of the the software that is being submission be interested in reviewing the submission • Avoid asking JOSS editors to review: If at all possible, avoid asking JOSS editors to review as they are generally very busy editing their own papers. Once a reviewer accepts, the handling editor runs the command @whedon assign @username as reviewer in the PRE-REVIEW issue. Add more reviewers with the command @whedon add @username as reviewer. Under the uncommon circumstance that a review must be started before all reviewers have been identified (e.g., if finding a second reviewer is taking a long time and the first reviewer wants to get started), an editor may elect to start the review and add the remaining reviewers later. To accomplish this, the editor will need to hand-edit the review checklist to create space for the reviewers added after the review issues is created.
Note: The assign command clobbers all reviewer assignments. If you want to add an additional reviewer use the add command.
3.5.1.3 Starting the review
Next, run the command @whedon start review. If you haven’t assigned an editor and reviewer, this command will fail and @whedon will tell you this. This will open the REVIEW issue, with prepared review checklists for each reviewer, and instructions. The editor should close the PRE-REVIEW issue, at this point, and move the conversation to the separate REVIEW issue.
3.5.2 Review
The REVIEW issue contains some instructions, and reviewer checklists. The reviewer(s) should check off items of the checklist one-by-one, until done. In the meantime, reviewers can engage the authors freely in a conversation aimed at improving the paper.
Note: When a new review is started, Whedon invites the reviewers to the GitHub repository. The reviewers must accept the invitation to the GitHub repository in order for Whedon to be able to assign the review issue to them, and reviewers are unable to check off checklist items until they have accepted the repository invitation.
If a reviewer recants their commitment or is unresponsive, editors can remove them with the command @whedon remove @username as reviewer. You can also add new reviewers in the REVIEW issue, but in this case, you need to manually add a review checklist for them by editing the issue body. Comments in the REVIEW issue should be kept brief, as much as possible, with more lengthy suggestions or requests posted as separate issues, directly in the submission repository. A link-back to those issues in the REVIEW is helpful. When the reviewers are satisfied with the improvements, we ask that they confirm their recommendation to accept the submission.
3.5.2.1 Adding a new reviewer once the review has started
Sometimes you’ll need to add a new reviewer once the main review (i.e. post pre-review) is already underway. In this situation you should do the following: • In the review thread, do @whedon add @newreviewer as reviewer. • Manually edit the first message in the thread to add a checklist for @newreviewer.
3.5. Editorial Guide 23 JOSS Documentation
In the future, this will be more automated but for now, there’s some manual work required.
3.5.3 After reviewers recommend acceptance
When a submission is ready to be accepted, we ask that the authors issue a new tagged release of the software (if changed), and archive it (on Zenodo, figshare, or other). The authors then post the version number and archive DOI in the REVIEW issue. The handling editor executes the pre-publication steps, and pings the EiCs for final processing. Pre-publication steps: • Get a new proof with the @whedon generate pdf command. • Download the proof, check all references have DOIs, follow the links and check the references. – Whedon can help check references with the command @whedon check references • Proof-read the paper and ask authors to fix any remaining typos, badly formed citations, awkward wording, etc.. • Ask the author to make a tagged release and archive, and report the version number and archive DOI in the review thread. • Check the archive deposit has the correct metadata (title and author list), and request the author edit it if it doesn’t match the paper. • Run @whedon set
3.5.4 Handling of papers published together with AAS publishing
JOSS is collaborating with AAS publishing to offer software review for some of the papers submitted to their journals. A detailed overview of the motivations/background is available in the announcement blog post, here we document the additional editorial steps that are necessary for JOSS to follow: Before/during review • If the paper is a joint publication, make sure you apply the AAS label to both the pre-review and the review issues. • Before moving the JOSS paper from pre-review to review, ensure that you (the JOSS editor) make the reviewers aware that JOSS will be receiving a small financial donation from AAS publishing for this review (e.g. like this). After the paper has been accepted by JOSS • Once the JOSS review is complete, ask the author for the status of their AAS publication, specifically if they have the AAS paper DOI yet. • Once this is available, ask the author to add this information to their paper.md YAML header as documented in the submission guidelines.
24 Chapter 3. Sponsors and affiliates JOSS Documentation
# Optional fields if submitting to a AAS journal too, see this blog post: # https://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing aas-doi: 10.3847/xxxxx<- update this with the DOI from AAS once you know it. aas-journal: Astrophysical Journal<- The name of the AAS journal.
• Pause the review (by applying the paused label) to await notification that the AAS paper is published. Once the AAS paper is published • Ask the EiC on rotation to publish the paper as normal (by tagging @openjournals/joss-eics).
3.5.5 Processing of rOpenSci-reviewed and accepted submissions
If a paper has already been reviewed and accepted by rOpenSci, the streamlined JOSS review process is: • Assign yourself as editor and reviewer • Add a comment in the pre-review issue pointing to the rOpenSci review • Add the rOpenSci label to the pre-review issue • Start the review issue • Add a comment in the review issue pointing to the rOpenSci review • Add the rOpenSci label to the review issue • Compile the paper and check it looks ok • Tick off all the review checkboxes • Go to to the source code repo and grab the Zenodo DOI • Accept and publish the paper
3.5.6 Rejecting a paper
If you believe a submission should be rejected, for example, because it is out of scope for JOSS, then you should: • Ask Whedon to flag the submission as potentially out of scope with the command @whedon query scope. This command adds the query-scope label to the issue. • Mention to the author your reasons for flagging the submission as possibly out of scope, and give them an opportunity to defend their submission. • The EiC on rotation will make a final determination of whether a submission is in scope, taking into account the feedback of other editors.
3.5.6.1 Voting on papers flagged as potentially out of scope
Once per week, an email is sent to all JOSS editors with a summary of the papers that are currently flagged as potentially out of scope. Editors are asked to review these submissions and vote on the JOSS website if they have an opinion about a submission.
3.5. Editorial Guide 25 JOSS Documentation
3.5.7 Sample messages for authors and reviewers
3.5.7.1 Sample email to potential reviewers
Dear Dr. Jekyll,
I found you following links from the page of The Super Project and/or on Twitter. This message is to ask if you can help us out with a submission to JOSS (The Journal of
˓→Open Source Software), where I’m an editor.
JOSS publishes articles about open source research software. The submission I'd like
˓→you to review is titled: "great software name here" and the submission repository is at: https://github.com/< ... >
JOSS is a free, open-source, community driven and developer-friendly online journal (no publisher is seeking to raise revenue from the volunteer labor of researchers!).
The review process at JOSS is unique: it takes place in a GitHub issue, is open, and author-reviewer-editor conversations are encouraged.
JOSS reviews involve downloading and installing the software, and inspecting the
˓→repository and submitted paper for key elements. See https://joss.readthedocs.io/en/latest/
˓→review_criteria.html
Editors and reviewers post comments on the Review issue, and authors respond to the
˓→comments and improve their submission until acceptance (or withdrawal, if they feel unable to satisfy the review).
Would you be able to review this submission for JOSS? If not, can you recommend someone from your team to help out?
Kind regards,
JOSS Editor.
3.5.7.2 Query scope of submission
:wave: thanks for your submission to JOSS. From a quick inspection of this submission
˓→it's not entirely obvious that it meets our [submission criteria](https://joss.
˓→readthedocs.io/en/latest/submitting.html#submission-requirements). In particular,
˓→this item:
> - Your software should have an obvious research application
Could you confirm here that there _is_ a research application for this software (and
˓→explain what that application is)? The section [_'what should my paper contain'_
˓→](https://joss.readthedocs.io/en/latest/submitting.html#what-should-my-paper-
˓→contain) has some guidance for the sort of content we're looking to be present in
˓→the `paper.md`.
Many thanks!
26 Chapter 3. Sponsors and affiliates JOSS Documentation
3.5.7.3 GitHub invite to potential reviewers
:wave: @reviewer1 & @reviewer2, would any of you be willing to review this submission
˓→for JOSS? We carry out our checklist-driven reviews here in GitHub issues and
˓→follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html
3.5.7.4 Message to reviewers at the start of a review
@authorname @reviewer1 @reviewer2 this is the review thread for the paper. All of
˓→our communications will happen here from now on.
Both reviewers have checklists at the top of this thread with the JOSS requirements.
˓→As you go over the submission, please check any items that you feel have been
˓→satisfied. There are also links to the JOSS reviewer guidelines.
The JOSS review is different from most other journals. Our goal is to work with the
˓→authors to help them meet our criteria instead of merely passing judgment on the
˓→submission. As such, the reviewers are encouraged to submit issues and pull
˓→requests on the software repository. When doing so, please mention `openjournals/
˓→joss-reviews#REVIEW_NUMBER` so that a link is created to this thread (and I can
˓→keep an eye on what is happening). Please also feel free to comment and ask
˓→questions on this thread. In my experience, it is better to post comments/questions/
˓→suggestions as you come across them instead of waiting until you've reviewed the
˓→entire package.
We aim for reviews to be completed within about 2-4 weeks. Please let me know if any
˓→of you require some more time. We can also use Whedon (our bot) to set automatic
˓→reminders if you know you'll be away for a known period of time.
Please feel free to ping me (@editorname) if you have any questions/concerns.
3.5.7.5 Message to authors at the end of a review
At this point could you: - [ ] Make a tagged release of your software, and list the version tag of the
˓→archived version here. - [ ] Archive the reviewed software in Zenodo or a similar service (e.g., figshare,
˓→an institutional repository) - [ ] Check the archival deposit (e.g., in Zenodo) has the correct metadata. This
˓→includes the title (should match the paper title) and author list (make sure the
˓→list is correct and people who only made a small fix are not on it). You may also
˓→add the authors' ORCID. - [ ] Please list the DOI of the archived version here.
I can then move forward with accepting the submission.
3.5.7.6
3.5.8 Overview of editorial process
Step 1: An author submits a paper.
3.5. Editorial Guide 27 JOSS Documentation
The author can choose to select an preferred editor based on the information available in our biographies. This can be changed later. Step 2: If you are selected as an editor you get @-mentioned in the pre-review issue. This doesn’t mean that you’re the editor, just that you’ve been suggested by the author. Step 3: Once you are the editor, find the link to the code repository in the pre-review issue Step 4: The editor looks at the software submitted and checks to see if: • There’s a general description of the software • The software is within scope as research software • It has an OSI-approved license Step 5: The editor responds to the author saying that things look in line (or not) and will search for reviewer Step 6: The editor finds >= 2 reviewers • Use the list of reviewers: type the command @whedon list reviewers or look at list of reviewers in a Google spreadsheet • If people are in the review list, the editor can @-mention them on the issue to see if they will review: e.g. @person1 @person2 can you review this submission for JOSS? • Or solicit reviewers outside the list. Send an email to people describing what JOSS is and asking if they would be interested in reviewing. • If you ask the author to suggest potential reviewers, please be sure to tell the author not to @-tag their sugges- tions. Step 7: Editor tells Whedon to assign the reviewer to the paper • Use @whedon assign @reviewer as reviewer • To add a second reviewer use @whedon add @reviwer2 as reviewer
Note: The assign command clobbers all reviewer assignments. If you want to add an additional reviewer use the add command.
Step 8: Create the actual review issue • Use @whedon start review • An issue is created with the review checklist, one per reviewer, e.g. https://github.com/openjournals/joss- reviews/issues/717 Step 9: Close the pre-review issue Step 10: The actual JOSS review • The reviewer reviews the paper and has a conversation with the author. The editor lurks on this conversation and comes in if needed for questions (or CoC issues). • The reviewer potentially asks for changes and the author makes changes. Everyone agrees it’s ready. Step 11: The editor pings the EiC team to get the paper published • Make a final check of the paper with @whedon generate pdf and that all references have DOIs (where appropriate) with @whedon check references.
28 Chapter 3. Sponsors and affiliates JOSS Documentation
• If everything looks good, ask the author to make a new release (if possible) of the software being reviewed and deposit a new archive the software with Zenodo/figshare. Update the review thread with this archive DOI: @whedon set 10.5281/zenodo.xxxxxx as archive. • Finally, use @whedon recommend-accept on the review thread to ping the @openjournals/ joss-eics team letting them know the paper is ready to be accepted. Step 12: Celebrate publication! Tweet! Thank reviewers! Say thank you on issue.
3.5.9 Visualization of editorial flow
Editorial flow
3.5. Editorial Guide 29 JOSS Documentation
3.5.10 Expectations on JOSS editors
3.5.10.1 Responding to editorial assignments
As documented above, usually, papers will be assigned to you by one of the AEiCs. We ask that editors do their best to respond in a timely fashion (~ 3 working days) to invites to edit a new submission.
3.5.10.2 Continued attention to assigned submissions
As an editor, part of your role is to ensure that submissions you’re responsible for are progressing smoothly through the editorial process. This means that once or twice per week we ask that you check your GitHub notifications and/or your editorial dashboard (e.g. http://joss.theoj.org/dashboard/youreditorname) for updates to the papers you are handling. If reviews go stale Sometimes reviews go quiet, either because a reviewer has failed to complete their review or an author has been slow to respond to a reviewer’s feedback. As the editor, we need you to prompt the author/or reviewer(s) to revisit the submission if there has been no response within 7-10 days unless there’s a clear statement in the review thread that says an action is coming at a slightly later time, perhaps because a reviewer committed to a review by a certain date, or an author is making changes and says they will be done by a certain date. Whedon has functionality to remind an author or review to return to a review at a certain point in the future. For example:
@whedon remind @reviewer in five days
3.5.11 Out of office
Sometimes we need time away from our editing duties at JOSS. The joss-reviews repository has the OoO bot installed which means you can mark yourself as out of the office (and unable to respond to reviews) for a period of time e.g.: Mark yourself as OoO in one of the reviews you’re editing in the joss-reviews repository like this:
/ooo January 18 until February2
Ooo bot will then respond to any mentions in the joss-reviews repository to let people know you’re away. Note, if you’re planning on being out of the office for more than two weeks, please let the JOSS editorial team know.
3.5.12 Editorial buddy
New editors are assigned an editorial ‘buddy’ from the existing editorial team. The buddy is there to help the new editor onboard successfully and to provide a dedicated resource for any questions they might have but don’t feel comfortable posting to the editor mailing list. Buddy assignments don’t have a fixed term but generally require a committment for 1-2 months. Some things you might need to do as a buddy for a new editor: • Respond to questions via email or on GitHub review issues. • Check in with the new editor every couple of weeks if there hasn’t been any other communication. • (Optionally) keep an eye on the new editor’s submissions.
30 Chapter 3. Sponsors and affiliates JOSS Documentation
3.5.13 Managing notifications
Being on the JOSS editorial team means that there can be a lot of notifications from GitHub if you don’t take some proactive steps to minimize noise from the reviews repository.
3.5.13.1 Things you should do when joining the editorial team
Unsubscribe from the reviews repository on GitHub When you’re added to the editorial team on GitHub, you will almost certainly find yourself subscribed (watching) to the joss-reviews repository. The first thing you should do is set yourself to ‘not watching’:
Repository notifications settings Please note, that by not watching the reviews repository, you will still receive notifications for issues (reviews) where you are @mentioned. Sometimes another editor might mention you in a review issue (for example to ask you a question). If you’ve responded and no-longer want to receive messages for that review, you can manually unsubscribe by clicking the button in the right-hand column on the review issue page. Curate your GitHub notifications experience GitHub has extensive documentation on managing notifications which explains when and why different notifications are sent from a repository. Set up email filters Email filters can be very useful for managing incoming email notifications, here are some recommended resources: • A GitHub blog post describing how to set up email filters.
3.5. Editorial Guide 31 JOSS Documentation
If you use Gmail: • https://gist.github.com/ldez/bd6e6401ad0855e6c0de6da19a8c50b5 • https://github.com/jessfraz/gmailfilters • https://hackernoon.com/how-to-never-miss-a-github-mention-fdd5a0f9ab6d Use a dedicated tool For papers that you are already assigned to edit, the dedicated JOSS dashboard aggregates notifications as- sociated with each paper. The dashboard is available at: https://joss.theoj.org/dashboard/
3.6 Interacting with Whedon
Whedon or @whedon on GitHub, is our editorial bot that interacts with authors, reviewers, and editors on JOSS reviews. @whedon can do a bunch of different things. If you want to ask @whedon what it can do, simply type the following in a JOSS review or pre-review issue:
# List all of Whedon's capabilities @whedon commands
# Assign a GitHub user as the sole reviewer of this submission @whedon assign @username as reviewer
# Add a GitHub user to the reviewers of this submission @whedon add @username as reviewer
# Re-invite a reviewer (if they can't update checklists) @whedon re-invite @username as reviewer
# Remove a GitHub user from the reviewers of this submission @whedon remove @username as reviewer
# List of editor GitHub usernames @whedon list editors
# List of reviewers together with programming language preferences and domain
˓→expertise @whedon list reviewers
# Change editorial assignment @whedon assign @username as editor
# Set the software archive DOI at the top of the issue e.g. @whedon set 10.0000/zenodo.00000 as archive
# Set the software version at the top of the issue e.g. @whedon set v1.0.1 as version
# Open the review issue @whedon start review
(continues on next page)
32 Chapter 3. Sponsors and affiliates JOSS Documentation
(continued from previous page) EDITORIAL TASKS
# All commands can be run on a non-default branch, to do this pass a custom # branch name by following the command with `from branch custom-branch-name`. # For example:
# Compile the paper @whedon generate pdf
# Compile the paper from alternative branch @whedon generate pdf from branch custom-branch-name
# Remind an author or reviewer to return to a review after a # certain period of time (supported units days and weeks) @whedon remind @reviewer in 2 weeks
# Ask Whedon to do a dry run of accepting the paper and depositing with Crossref @whedon recommend-accept
# Ask Whedon to check the references for missing DOIs @whedon check references
# Ask Whedon to check repository statistics for the submitted software, for license,
˓→and # for Statement of Need section in paper @whedon check repository
EiC TASKS
# Flag submission for editoral review, due to size or question about being research
˓→software @whedon query scope
# Invite an editor to edit a submission (sending them an email) @whedon invite @editor as editor
# Reject a paper @whedon reject
# Withdraw a paper @whedon withdraw
# Ask Whedon to actually accept the paper and deposit with Crossref # (supports custom branches too) @whedon accept deposit=true
3.6.1 Author commands
A subset of the Whedon commands are available to authors (and reviewers):
3.6.1.1 Compiling papers
When a pre-review or review issue is opened, @whedon will try to compile the JOSS paper by looking for a paper.md file in the repository specified when the paper was submitted.
3.6. Interacting with Whedon 33 JOSS Documentation
If it can’t find the paper.md file it will say as much in the review issue. If it can’t compile the paper (i.e. there’s some kind of Pandoc error), it will try and report that error back in the thread too.
Note: If you want to see what command @whedon is running when compiling the JOSS paper, take a look at the code here.
Anyone can ask @whedon to compile the paper again (e.g. after a change has been made). To do this simply comment on the review thread as follows:
@whedon generate pdf
3.6.1.1.1 Compiling papers from a non-default branch
By default, Whedon will look for papers in the default git branch. If you want to compile a paper from a non-default branch, this can be done as follows:
@whedon generate pdf from branch custom-branch-name
3.6.1.2 Finding reviewers
Sometimes submitting authors suggest people the think might be appropriate to review their submission. If you want the link to the current list of JOSS reviewers, type the following in the review thread:
@whedon list reviewers
3.6.2 Editorial commands
Most of @whedon’s functionality can only be used by the journal editors.
3.6.2.1 Assigning an editor
Editors can either assign themselves or other editors as the editor of a submission as follows:
@whedon assign @editorname as editor
3.6.2.2 Inviting an editor
Whedon can be used by EiCs to send email invites to registered editors as follows:
@whedon invite @editorname as editor
This will send an automated email to the editor with a link to the GitHub pre-review issue.
3.6.2.3 Adding and removing reviewers
Reviewers should be assigned by using the following commands:
34 Chapter 3. Sponsors and affiliates JOSS Documentation
# Assign a GitHub user as the sole reviewer of this submission @whedon assign @username as reviewer
# Add a GitHub user to the reviewers of this submission @whedon add @username as reviewer
# Remove a GitHub user from the reviewers of this submission @whedon remove @username as reviewer
Note: The assign command clobbers all reviewer assignments. If you want to add an additional reviewer use the add command.
3.6.2.4 Starting the review
Once the reviewer(s) and editor have been assigned in the pre-review issue, the editor starts the review with:
@whedon start review
Important: If a reviewer recants their commitment or is unresponsive, editors can remove them with the command @whedon remove @username as reviewer. You can also add new reviewers in the REVIEW issue, but in this case, you need to manually add a review checklist for them by editing the issue body.
3.6.2.5 Reminding reviewers and authors
Whedon can reminders authors and reviewers after a specified amount of time to return to the review issue. Reminders can only be set by editors, and only for REVIEW issues. For example:
# Remind the reviewer in two weeks to return to the review @whedon remind @reviewer in two weeks
# Remind the reviewer in five days to return to the review @whedon remind @reviewer in five days
# Remind the author in two weeks to return to the review @whedon remind @author in two weeks
Note: Most units of times are understood by Whedon e.g. hour/hours/day/days/week/weeks.
Important: For reviewers, the reminder will only be triggered if the reviewer’s review is outstanding (i.e. outstanding checkboxes).
3.6.2.6 Setting the software archive
When a submission is accepted, we ask that the authors create an archive (on Zenodo, figshare, or other) and post the archive DOI in the REVIEW issue. The editor should add the accepted label on the issue and ask @whedon to add
3.6. Interacting with Whedon 35 JOSS Documentation the archive to the issue as follows:
@whedon set 10.0000/zenodo.00000 as archive
3.6.2.7 Changing the software version
Sometimes the version of the software changes as a consequence of the review process. To update the version of the software do the following:
@whedon set v1.0.1 as version
3.6.3 Accepting a paper (dry run)
Whedon can accept a paper from the review issue. This includes generating the final paper PDF, Crossref metedata, and depositing this metadata with the Crossref API. JOSS topic editors can ask for the final proofs to be created by Whedon with the following command:
@whedon recommend-accept
On issuing this command, Whedon will also check the references of the paper for any missing DOIs. This command can be triggered separately:
3.6.3.1 Check references
@whedon check references
Note: Whedon can verify that DOIs resolve, but cannot verify that the DOI associated with a paper is actually correct. In addition, DOI suggestions from Whedon are just that - i.e. they may not be correct.
3.6.4 Accepting a paper (for real)
If everything looks good with the draft proofs from the @whedon accept command, JOSS editors-in-chief can take the additional step of actually accepting the JOSS paper with the following command:
@whedon accept deposit=true
Note: This command is only available to the JOSS editor-in-chief, or associate editors-in-chief.
3.7 Installing the JOSS application
Any Open Journal (JOSS, JOSE, etc.) can be considered in three parts: 1. The website 2. The Whedon gem
36 Chapter 3. Sponsors and affiliates JOSS Documentation
3. A thin API wrapper around the Whedon gem to interface with GitHub For JOSS, these correspond to the: 1. JOSS, 2. Whedon, and 3. Whedon-API code bases.
3.7.1 Setting up a local development environment
All Open Journals are coded in Ruby, with the website and Whedon-API developed as Ruby on Rails applications. If you’d like to develop these locally, you’ll need a working Ruby on Rails development environment. For more information, please see this official guide.
3.7.2 Deploying your JOSS application
To deploy JOSS, you’ll need a Heroku account. We also recommend you gather the following information before deploying your application to Heroku: 1.A public ORCID API key 2. A GitHub Personal Access Token for the automated account that users will interact with (e.g., @Whedon, @RoboNeuro). In order to be able to send invitations to reviewers and collaborators, the automated GitHub account must be an admin of the organization the reviews take place at. And the Personal Access Token should include the admin:org scope. 3. An email address registered on a domain you control (i.e., not gmail or a related service)
Warning: Do not put these secrets directly into your code base! It is important that these keys are not under version control. There are different ways to make sure your application has access to these keys, depending on whether your code is being developed locally or on Heroku. Locally, you can store these locally in a .env file. The .gitignore in JOSS is already set to ignore this file type. On Heroku, they will be config variables that you can set either with the Heroku CLI or directly on your applica- tion’s dashboard. See this guide from Heroku for more information.
Assuming you have forked the JOSS GitHub repository to your account, you can configure Heroku as a git remote for your code. This makes it easy to keep your Heroku deployment in sync with your local development copy. On the JOSS Heroku deployment, you’ll need to provision several add-ons. Specifically, you’ll need: 1. Elasticsearch add-on 2. PostgreSQL add-on 3. Scheduler add-on For the scheduler add-on, you’ll need to designate which tasks it should run and when. These can be found in the lib/tasks folder, and involve things such as sending out weekly reminder emails to editors. Each task should be scheduled as a separate job; for example, rake send_weekly_emails. You can also optionally configure the following add-ons (or simply set their secret keys in your config variables):
3.7. Installing the JOSS application 37 JOSS Documentation
1. SendGrid add-on for sending emails 2. Honeybadger add-on for error reporting Once you’ve pushed your application to Heroku and provisioned the appropriate add-ons, you’re ready to update your config with the appropriate secrets. For a list of the expected secret key names, see the app.json file.
Warning: One “gotcha” when provisioning the Bonsai add-on is that it may only set the BONSAI_URL variable. Make sure that there is also an ELASTICSEARCH_URL which is set to the same address.
We will not cover Portico, as this requires that your application is a part of the openjournals organization. If you do not already have access to these keys, you can simply ignore them for now.
Note: One secret key we have not covered thus far is WHEDON_SECRET. This is because it is not one that you obtain from a provide, but a secret key that you set yourself. We recommend using something like a random SHA1 string. It is important to remember this key, as you will need it when deploying your Whedon-API application.
After pushing your application to Heroku, provisioning the appropriate add-ons, and confirming that your config variables are set correctly, you should make sure that your username is registered as an admin on the application. You can do this on a local Rails console, by logging in and setting the boolean field ‘admin’ on your user ID to True. If you’d prefer to do this on the Heroku deployment, make sure you’ve logged into the application. Then you can directly modify this attribute in the deployments Postgres database using SQL. For more information on accessing your application’s Postgres database, see the official docs.
3.7.3 Making modifications to launch your own site
Some times you may not want to launch an exact copy of JOSS, but a modified version. This can be especially useful if you’re planning to spin up your own platform based on the Open Journals framework. NeuroLibre is one such example use-case.
3.7.3.1 Modifying your site configuration
In this case, there are several important variables to be aware of and modify. Most of these are accessible in the config folder. First, there are three files which provide settings for your Rails application in different development contexts: 1. settings-test.yml 2. settings-development.yml 3. settings-production.yml These each contain site-specific variables that should be modified if you are building off of the Open Journals frame- work. Next, you’ll need to modify the repository.yml file. This file lists the GitHub repository where you expect papers to be published, as well as the editor team ID. For your GitHub organization, make sure you have created and populated a team called editors. Then, you can check its ID number as detailed in this guide. In config you should also modify the orcid.yml file to list your site as the production site.
38 Chapter 3. Sponsors and affiliates JOSS Documentation
Finally, you’ll need to set up a GitHub webhook for reviews repository. This should be a repository that you have write access to, where you expect most of the reviewer-author interaction to occur. For JOSS, this corresponds to the openjournals/joss-reviews GitHub repository. In this GitHub repository’s settings, you can add a new webhook with the following configuration: • Set the Payload URL to the /dispatch hook for your Heroku application URL. For example, https://neurolibre.herokuapp.com/dispatch • Set the Content type to application/json • Set the secret to a high-entropy, random string as detailed in the GitHub docs • Set the webhook to deliver all events
3.7.3.1.1 Updating your application database
Optionally, you can edit seeds.rb, a file in the db folder. “DB” is short for “database,” and this file seeds the database with information about your editorial team. You can edit the file seeds.rb to remove any individuals who are not editors in your organization. This can be especially useful if you will be creating the database multiple times during development; for example, if you add in testing information that you’d later like to remove. You can reinitialize the database from your Heroku CLI using the following commands: heroku pg:reset DATABASE_URL heroku run rails db:schema:load heroku run rails db:seed heroku run rails searchkick:reindex:all
3.7.3.2 Modifying your site contents
You can modify site content by updating files in the app and docs folders. For example, in app/views/ notifications you can change the text for any emails that will be sent by your application. Note that files which end in .html.erb are treated as HTML files, and typical HTML formatting applies. You can set the HTML styling by modifying the Sass files for your application, located in app/assets/stylesheets. There are currently a few hard-coded variables in the application which you will also need to update. Note that these are mostly under lib/tasks. For example, in stats.rake, the reviewer sheet ID is hard-coded on line 37. You should update this to point to your own spreadsheet where you maintain a list of eligible reviewers. In the same folder, utils.rake is currently hard-coded to alternate assignments of editor-in-chief based on weeks. You should modify this to either set a single editor-in-chief, or design your own scheme of alternating between mem- bers of your editorial board.
3.7.4 Deploying your Whedon-API Application
Whedon-API can also be deployed on Heroku. Note that — for full functionality — Whedon-API must be deployed on Hobby dynos, rather than free dynos. Hobby dynos allow the Whedon-API application to run continuously, without falling asleep after 30 minutes of inactivity; this means that Whedon-API can respond to activity on GitHub at any time. Whedon-API specifically requires two Hobby dynos: one for the web service and one for the worker service. On the Whedon-API Heroku deployment, you’ll need to provision several add-ons. Specifically, you’ll need: 1. Cloudinary add-on 2. Scheduler add-on
3.7. Installing the JOSS application 39 JOSS Documentation
3. Redis To Go add-on For the scheduler add-on, you’ll need to designate which tasks it should run and when. The only task that needs to be scheduled is the restart.sh script, which should be set to execute every hour. As before, once you’ve pushed your application to Heroku and provisioned the appropriate add-ons, you’re ready to update your config with the appropriate secrets. For a list of the expected secret key names, see the app.json file. Many of these will be re-used from deploying your JOSS application. Specifically, the GH_TOKEN should be the same personal access token as before. The JOSS_API_KEY should match the WHEDON_SECRET key that you created in your JOSS deployment. You’ll also need to provide a HEROKU_APP_NAME, HEROKU_CLI_TOKEN, and HEROKU_CLI_USER that the restart.sh script can use when executing. You can find these directly from the heroku-cli as detailed in their documentation.
3.7.5 Modifying your Whedon-API deployment
Some times you may not want to launch an exact copy of the Whedon-API, but a modified version. This can be especially useful if you’re planning to spin up your own platform based on the Open Journals framework. RoboNeuro is one such example use-case.
3.7.5.1 Modifying your Whedon-API configuration
Similar to the JOSS deployment described above, the Whedon-API configuration is controlled through a series of YAML files included in the config/ folder. Each of these files provide relevant configuration for a different devel- opment context. Specifically, two files are defined: 1. settings-test.yml 2. settings-production.yml which can be used to define testing and production environment variables, respectively. Much of the information previously defined for your JOSS deployment will carry over, including the editor team ID. Finally, you’ll need to set up a GitHub webhook for your reviews repository. As a reminder, this should be a repository that you have write access to. For JOSS, this corresponds to the openjournals/joss-reviews GitHub repos- itory. This is in addition to the webhook you previously created for the JOSS deployment, although it points to the same repository. In this GitHub repository’s settings, you can add a new webhook with the following configuration: • Set the Payload URL to the /dispatch hook for your Heroku application URL. For example, https://roboneuro.herokuapp.com/dispatch • Set the Content type to application/json • Set the secret to a high-entropy, random string as detailed in the GitHub docs • Set the webhook to deliver all events
40 Chapter 3. Sponsors and affiliates