The MICCAI Hackathon on reproducibility, diversity, and selection of papers at the MICCAI conference Fabian Balsigera,b,∗, Alain Jungoa,∗, Naren Akash R Jc, Jianan Chend, Ivan Ezhove, Shengnan Liuf, Jun Mag, Johannes C. Paetzolde, Vishva Saravanan Rh, Anjany Sekuboyinae, Suprosanna Shite, Yannick Sutera, Moshood Yekinii, Guodong Zengj, Markus Rempflerk,∗ aARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland bSupport Center for Advanced Neuroimaging (SCAN), Institute for Diagnostic and Interventional Neuroradiology, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland cCenter for Visual Information Technology, IIIT Hyderabad, Hyderabad, India dDepartment of Medical Biophysics, University of Toronto, Toronto, Canada eTechnical University of Munich, Munich, Germany fDepartment of Cardiology, Erasmus MC University Medical Center, Rotterdam, The Netherlands gDepartment of Mathematics, Nanjing University of Science and Technology, Nanjing, China hIIIT Hyderabad, Hyderabad, India iAfrican Masters of Machine Intelligence, Accra, Ghana jsitem Center for Translational Medicine and Biomedical Entrepreneurship, Bern, Switzerland kFriedrich Miescher Institute for Biomedical Research (FMI), Basel, Switzerland Abstract The MICCAI conference has encountered tremendous growth over the last years in terms of the size of the community, as well as the number of contributions and their technical success. With this growth, however, come new challenges for the community. Methods are more difficult to reproduce and the ever-increasing number of paper submissions to the MICCAI conference poses new questions regarding the selection process and the diversity of topics. To exchange, discuss, and find novel and creative solutions to these challenges, a new format of a hackathon was initiated as a satellite event at the MICCAI 2020 conference: The MICCAI Hackathon. The first edition of the MICCAI Hackathon covered the topics reproducibility, diversity, and selection of MICCAI papers. In the manner of a small think-tank, participants collaborated to find solutions to these challenges. In this report, we summarize the insights from the MICCAI Hackathon into immediate and long-term measures to address these challenges. The proposed measures can be seen as starting points and guidelines for discussions and actions to possibly improve the MICCAI conference with regards to reproducibility, diversity, and selection of papers. Keywords: MICCAI, reproducibility, diversity, review, hackathon 1. Introduction to present and get to know the latest research re- lated to MICCAI. With an acceptance rate of ap- arXiv:2103.05437v2 [cs.CV] 28 Apr 2021 The MICCAI conference is an annual scientific proximately 30 % (Martel et al., 2020), the MICCAI meeting for research in medical image computing conference is highly competitive and can be consid- (MIC) and computer assisted interventions (CAI). ered among the top scientific conferences research Approximately 2500 (MICCAI Society, 2020) re- in MIC and CAI. Besides the main conference that searchers attend the MICCAI conference every year stretches over three consecutive days, there exist two satellite event days, one before and one after the three days, which allow the community to orga- ∗Equal contribution and corresponding authors. The nize workshops, challenges, and tutorials dedicated other authors are listed alphabetically. to specific topics. At MICCAI 2020, for the first Email addresses: [email protected] (Fabian Balsiger), [email protected] (Alain Jungo), time, a hackathon was organized as such a satellite [email protected] (Markus Rempfler) event. the procedure is detailed in the supplementary ma- terial. The outcome of the first MICCAI Hackathon is summarized in this article. For the three topics reproducibility, diversity, and selection, immediate and long-term measures are presented in Section 2 to 4. Immediate measures might be realizable for Figure 1: The logo of the MICCAI Hackathon. upcoming MICCAI conferences and long-term mea- sures would most likely require more time and ef- The MICCAI Hackathon was initiated to ben- fort to implement. All presented measures base efit from the exchange among researchers during on the keynotes, mentoring sessions, contributions, the MICCAI conference. A hackathon is espe- and opinions of the MICCAI Hackathon organiz- cially well-suited as a satellite event at a conference, ers. Finally, a short discussion concludes the arti- where junior and senior researchers go to exchange, cle. We hope that the distilled measures could help discuss, and learn. Therefore, as a bottom-up ap- to potentially improve MICCAI, as reproducibility, proach, the MICCAI Hackathon benefits from this diversity, and selection of MICCAI papers concern fruitful environment in order to foster collabora- the whole MICCAI community. tive work. The MICCAI Hackathon can be consid- ered as a small think-tank rather than traditional 2. Reproducibility workshops with talks and poster session, podium discussions, lecture-based tutorials, hands-on ses- Reproducibility refers to that results of an exper- sions, and challenges known from satellite event iment should be achieved with similar or equal re- days. Accordingly, in this new format, participants sults when the experiment is performed by another gather and receive input from keynote speakers and researcher. In MICCAI, this often involves execut- mentors providing impulses about the hackathon's ing computational methods on medical image data. topic. The participants then work individually or However, this is not trivial to achieve as most meth- together in teams to find solutions. Finally, they ods are highly specialized and often not publicly present their outcome at the end of the hackathon. available. For instance, the collection of MICCAI The first edition of the MICCAI Hackathon1 cov- 2020 papers with code (Ma, 2020) shows that there ered the topics reproducibility, diversity, and selec- is still only a fraction of the papers that share their tion of MICCAI papers. The advance of machine code and thereby facilitate reproducibility. Further, learning has had a considerable impact on MICCAI, as MICCAI often involves protected medical data, pushing the limits of algorithms, opening up com- sharing that data might not be straightforward or pletely new applications and ultimately, leading to even impossible. Therefore, in the category repro- an increased overall interest in MICCAI. With this ducibility, two questions were specifically investi- success, however, came new challenges for the com- gated: munity. Complex, data-driven algorithms are more difficult to reproduce and the ever-increasing num- • What does it need for a MICCAI paper to be ber of paper submissions to the MICCAI conference reproducible? poses new questions regarding the selection process • What could MICCAI do to encourage repro- and the diversity of topics. To provide inputs to the ducibility? participants regarding these topics, two keynotes were given and six mentoring sessions were held. Three out of five contributions addressed the cat- Overall, five contributions were received providing egory of reproducibility. Overall, reproducibility ideas to tackle the challenges of reproducibility, di- was the aspect with the most frequent feedback that versity, and selection of MICCAI papers. For the an improvement is desired. It is likely the category interested reader, the contributions and keynotes of where measures are the most straightforward to im- the MICCAI Hackathon are available online2, and plement and where we consider the potential for immediate improvements to be the highest. This is mostly due to the fact that MICCAI can bene- 1https://2020.miccai-hackathon.com/ 2https://www.youtube.com/playlist?list= fit from the experiences of measures taken at other PLflMBx361ODCPE3TK-ROFSblPMK5cNOto. conferences (e.g., NeurIPS (Pineau et al., 2020)). 2 2.1. Immediate Measures garding reproducibility should be enhanced. One Incorporate the reproducibility checklist way to achieve this could be an official list of open in the paper submission form (at intent-to- source MICCAI papers which could be either on submit & final submission). Incorporating a the MICCAI society website and/or even be incor- reproducibility checklist to the paper submission porated into the proceedings. might raise awareness to this important topic and Communicate best practices on repro- nudge authors towards adding important details to ducibility and code submission. Similar to the their manuscript. This measure has been effec- guidelines for authors and reviewers, there could be tive at NeurIPS 2019 (Pineau et al., 2020). The guidelines for reproducibility (or a dedicated sec- initial checklist can adhere to the list by Pineau tion in the guidelines for authors) pointing to the (2020) without adaptions. We consider it impor- reproducibility checklist (Pineau, 2020) as well as tant that the list is already present at the inten- resources like a code completeness checklist (Papers tion to submit, but with the possibility to update with Code, 2020). Similar efforts at other confer- the choices at the time of the final submission. By ences have already been made (e.g. at the EMNLP having such a list in the submission form, papers conference (EMNLP Conference, 2020b,a)) and do might be improved upon submission regarding re- not need to be recreated from scratch. producibility as people tend to forget, not intention- ally, certain aspects when writing their papers (e.g.,