Received: 16 July 2019 | Accepted: 19 July 2019 DOI: 10.1002/humu.23876

OVERVIEW

Reports from the fifth edition of CAGI: The Critical Assessment of Interpretation

Gaia Andreoletti1 | Lipika R. Pal2 | John Moult2,3 | Steven E. Brenner1,4

1Department of Plant and Microbial Biology, University of California, Berkeley, California Abstract 2Institute for Bioscience and Biotechnology Interpretation of genomic variation plays an essential role in the analysis of Research, University of Maryland, Rockville, cancer and monogenic disease, and increasingly also in complex trait disease, with Maryland 3Department of Biology and Molecular applications ranging from basic research to clinical decisions. Many computational Genetics, University of Maryland, College impact prediction methods have been developed, yet the field lacks a clear consensus Park, Maryland on their appropriate use and interpretation. The Critical Assessment of Genome 4Center for , University of California, Berkeley, California Interpretation (CAGI, /'kā‐jē/) is a community experiment to objectively assess computational methods for predicting the phenotypic impacts of genomic variation. Correspondence John Moult, Institute for Bioscience and CAGI participants are provided genetic variants and make blind predictions of Biotechnology Research, University of resulting phenotype. Independent assessors evaluate the predictions by comparing Maryland, 9600 Gudelsky Drive, Rockville, MD 20850. with experimental and clinical data. Email: [email protected] CAGI has completed five editions with the goals of establishing the state of art in

Steven E. Brenner, Department of Plant and genome interpretation and of encouraging new methodological developments. This Microbial Biology, University of California, special issue (https://onlinelibrary.wiley.com/toc/10981004/2019/40/9) comprises Berkeley, CA 94720. Email: [email protected] reports from CAGI, focusing on the fifth edition that culminated in a conference that took place 5 to 7 July 2018. CAGI5 was comprised of 14 challenges and engaged Funding information National Research Institute hundreds of participants from a dozen countries. This edition had a notable increase and National Cancer Institute, Grant/Award in splicing and expression regulatory variant challenges, while also continuing Number: U41 HG007346 and R13 HG006650 challenges on clinical , as well as complex disease datasets and missense variants in diseases ranging from cancer to Pompe disease to schizophrenia. Full information about CAGI is at https://genomeinterpretation.org.

KEYWORDS CAGI, Critical Assessment of Genome Interpretation, genomics, genetic variation, cancer genetics, variant impact predictors, SNP

1 | INTRODUCTION The Critical Assessment of Genome Interpretation (CAGI, pronounced /'kā‐jē/) addresses this need by providing an objective Interpretation of genome sequence variation plays a major and evaluation of the state‐of‐the‐art in relating human genetic variation increasing role in both basic research and in clinical medicine. to phenotype, particularly health. Each edition of CAGI provides These applications necessitate robust and reliable computational about a dozen challenges to understand performance of prediction approaches to aid in determining the phenotypic impact of methods in a given scenario. Participants in a challenge are provided variants. Many such methods have been developed (see genetic variants and make predictions of resulting molecular, cellular, Hu et al., 2019 in this issue). However, the appropriate use or organismal phenotypes. Data sets for the challenges include and accuracy of most methods have not been objectively germline and somatic cancer variation, rare disease, common disease, determined. and pharmacogenomics. The scale of challenges ranges from single

Human Mutation. 2019;40:1197–1201. wileyonlinelibrary.com/journal/humu © 2019 Wiley Periodicals, Inc. | 1197 1198 | ANDREOLETTI ET AL. nucleotides to whole , as well as complementary multiomics CAGI4 SickKids challenge, also described in the assessment paper and environmental information. Variant types include those affecting here. expression, splicing, and amino acid sequence, and may be single base Over all CAGI editions, the plurality of challenges have been on changes, insertions or deletions, and structural variation. the interpretation of isolated missense variants, and CAGI5 The CAGI cycle commences with the organizers soliciting data continues that trend. There are assessment, data provider, and sets from researchers and clinicians, whose studies demonstrate participant papers for the prediction of the destabilizing effect of relationships between genotype and phenotype. Such data sets must missense mutations in a cancer‐relevant protein (Frataxin, with be fully developed (“ripe”) but not publicly available (“spoiled”) until biophysical measurements of protein stability; Petrosino et al., 2019; after the CAGI prediction season. Data providers and organizers Savojardo, Petrosino et al., 2019; Strokach, Corbi‐Verge, & Kim, work together to develop these data sets into prediction challenges 2019); on the effect of missense changes in a human calmodulin, with well‐defined data sets and goals. The CAGI Ethics Forum assayed using a high‐throughput yeast complementation assay evaluates and adjusts these challenges for suitability and sensitivity. (Zhang et al., 2019); the effect of missense mutations related to In parallel, predictors are enrolled, bound by the CAGI Data Use schizophrenia in human Pericentriolar Material 1 (PCM1), using a Agreement [https://genomeinterpretation.org/data‐use‐agreement], zebrafish development model (Miller, Wang, & Bromberg, 2019; and vetted for access to the CAGI experiment with tiers reflecting Monzon et al., 2019); the effect of missense mutations in two the sensitivity of the data. The prediction season launches with cancer‐related proteins, PTEN and TPMT, on intracellular protein release of the challenges and concludes with the submission of levels, measured in a high‐throughput assay (Pejaver et al., 2019); predictions from predictors. For the fifth edition of CAGI (CAGI5), and the effect of missense changes in a monogenic disease related the prediction season extended until May 2018. The submitted protein, acid alpha‐glucosidase (GAA), with measurements of total predictions are evaluated by independent assessors. Each CAGI intracellular enzyme activity (Adhikari, 2019). Three participant edition culminates in a conference to discuss the outcome; the papers describe results on all the missense challenges (Garg & Pal, CAGI5 conference was held from 5 to 7 July 2018, for which videos 2019; Katsonis & Lichtarge, 2019; Savojardo, Babbi et al., 2019). The and slide sets are publicly available to registered CAGI members. issue also contains assessment articles from two earlier missense [https://genomeinterpretation.org/content/5‐conference] challenges on monogenic disease related proteins: N‐acetyl‐glucosa- Results from the previous CAGI edition (CAGI4) were published minidase (NAGLU; Clark et al., 2019), with total intracellular enzyme in a special issue of this journal (Hoskins et al., 2017). The present activity measured; and cystathionine beta‐synthase (CBS), using the issue contains assessment and participant papers primarily from the metric of yeast growth in a complication assay (Kasak, Bakolitsa most recent experiment, CAGI5, which included 14 challenges and et al., 2019). attracted participants from 12 countries. In addition to the other cancer‐related challenges outlined above, Notable in this CAGI edition is the increasing representation of there are two that required prediction of the pathogenicity of regulatory and noncoding variant challenges. Previously, CAGI has germline variants in cancer‐related proteins: one for breast cancer had just one small splicing challenge [https://genomeinterpretation. risk from variants in BRCA1 and BRCA2 as characterized by the org/content/Splicing‐2012]. CAGI5 included two full‐scale splicing ENIGMA consortium (Cao et al., 2019; Cline et al., 2019; Padilla et al., challenges (Mount et al., 2019) and these have resulted in five papers 2019; Parsons et al., 2019), and the other for cancer risk of variants from participants (Chen, Lu, Zhao, & Yang, 2019; Cheng, Çelik, in CHEK2 in Latina breast cancer cases and ancestry matched Nguyen, Avsec, & Gagneur, 2019; Gotea, Margolin, & Elnitski, 2019; controls (Voskanian et al., 2019). Naito, 2019; Wang, Wang, & Hu, 2019). The issue also contains an CAGI5 will continue to bear fruit. This edition introduced a time‐ overview paper from one of the splicing data providers (Rhine et al., based challenge in association with dbNSFP, whereby CAGI accepted 2019). There had also been only one previous expression regulatory predictions for all possible missense variants in the genome. The variant challenge, in CAGI4 (Kreimer et al., 2017). CAGI5 has a new results are to be vetted periodically in the future as the impact of expression regulatory challenge (Shigaki et al., 2019), and there are some of these variants are experimentally or clinically established. also two participant papers (Dong & Boyle, 2019; Kreimer, Yan, Additional papers on the challenges and other aspects of CAGI are Ahituv, & Yosef, 2019). presently in development and will be added to the CAGI5 papers CAGI5 continued the emphasis on the interpretation of clinically collection at Human Mutation (https://onlinelibrary.wiley.com/toc/ relevant large‐scale sequence data, with a challenge on the risk of 10981004/2019/40/9). thrombosis in African‐American cohort given whole exome sequence Full information about CAGI5 and earlier editions is at (McInnes et al., 2019; Wang & Bromberg, 2019); the identification of https://genomeinterpretation.org. variants contributing to intellectual disability phenotypes given gene panel sequence (Aspromonte et al., 2019; Carraro et al., 2019; Chen, ACKNOWLEDGMENTS 2019); and a challenge of matching whole genome sequences to clinical profiles for patients at Toronto’s Hospital for Sick Children The primary contributors to CAGI are predictors, and without their (SickKids) and identifying causal variants (Kasak, Hunter et al., 2019; willingness to test their methods in this way, the experiments would Pal, Kundu, Yin, & Moult, 2019). The latter challenge is related to the not be possible. Participants over the five CAGI editions are: Allison ANDREOLETTI ET AL. | 1199

Abad, Ogun Adebali, Aashish Adhikari, Ivan Adzhubey, Talal Amin, Shmulevich, Bradford R. Silver, Nasa Sinnott‐Armstrong, Vasily V. Žiga Avsec, Johnathan R. Azaria, Giulia Babbi, Eraan Bachar, Sitnik, Naveen Sivadasan, Ben Smithers, Yeşim A. Son, Rajgopal Benjamin Bachman, Minkyung Baek, Greet De Baets, Michael A. Srinivasan, Mario Stanke, Nathan Stitziel, Alexey Strokach, Andrew Beer, Violeta Beleva‐Guthrie, Riccardo Bellazzi, , Brady Su, Yuanfei Sun, Laksshman Sundaram, Uma Sunderam, Paul Tang, Bernard, Rajendra R. Bhat, Rohit Bhattacharya, Samuele Bovo, Alan Sean V. Tavtigian, Nuttinee Teerakulkittipong, Natalie Thurlby, Janita P. Boyle, Marcus Breese, Aharon S. Brodie, Yana Bromberg, Thusberg, Kevin Tian, Collin Tokheim, Scott Topper, Silvio C. E. Binghuang Cai, Colin Campbell, Chen Cao, Yue Cao, Emidio Capriotti, Tosatto, Yemliha Tuncel, Tychele Turner, Ron Unger, Aneeta Uppal, Liran Carmel, Marco Carraro, Hannah Carter, , Gurkan Ustunkar, Jouni Valiaho, Mauno Vihinen, Ilya E. Vorontsov, Muhammed Hasan Çelik, Billy H. W. Chang, Chien‐Yuan Chen, Mary Wahl, Michael Wainberg, Li‐San Wang, Maggie Wang, Meng Haoran Chen, Jingqi Chen, Ken Chen, Shann‐Ching Chen, Yun‐Ching Wang, Robert Wang, Xinyuan Wang, Yanran Wang, Liping Wei, Qiong Chen, Jun Cheng, Melissa Cline, Noa Cohen, Carles Corbi‐Verge, Wei, Rene Welch, Stephen Wilson, Chunlei Wu, Lijing Xu, Qifang Xu, Andrea Corredor, Chen Cui, Dvir Dahary, Carla Davis, Xavier de la Zhongxia Yan, Yang Yang, Yuedong Yang, Zhaomin Yao, Christopher Cruz, Mark Diekhans, Rezarta I. Dogan, Shengcheng Dong, Christo- Yates, Yizhou Yin, Nir Yosef, Erin Young, Chen‐Hsin Yu, Yao Yu, pher Douville, Ian Driver, Roland Dunbrack, Joost van Durme, Orland Dejian Yuan, Jan Zaucha, Haoyang Zeng, Fengfeng Zhou, Yaoqi Zhou, Díez, Andrea Eakin, Matthew Edwards, Laura Elnitski, Gokcen and Maya Zuhl. Eraslan, Hai Fang, Alexander V. Favorov, Tzila Fenesh, Xin Feng, CAGI also depends critically on the generosity of the groups who Carlo Ferrari, Simon Fishilevich, Anna Flynn, Lukas Folkman, Colby T. make their data available and help develop the challenges, usually Ford, Adam Frankish, Zaneta Franklin, Yao Fu, Julien Gagneur, Aditi before their own publications: Nadav Ahituv, , Adam P. Garg, Alessandra Gasparini, Tom Gaunt, David Gifford, Manuel Arkin, Madeleine P. Ball, Jason Bobe, Paolo Bonvini, Bethany Buckley, Giollo, Nina Gonzaludo, Valer Gotea, Julian Gough, Solomon Grant, Roberta Chiaraluce, George Church, Wyatt T. Clark, Valerio Consalvi, Rafael Guerrero, Yuchun Guo, Sara Gutiérrez‐Enríquez, James Jisu The ENIGMA Consortium, Garry R. Cutting, Emma D’Andrea, Roxana Han, Jennifer Harrow, Marcia Hasenahuer, Lim Heo, Tamar Holzer, Daneshjou, Lisa Elefanti, Will Fairbrother, Aron W. Fenton, Douglas M. Ramin Homayouni, Alex Hawkins‐Hooker, Raghavendra Hosur, Fowler, Andre Franke, David Goldgar, Nina Gonzaludo, Brenton R. Cheng L. V. Huang, Chad D. Huff, Peter Huwe, Sohyun Hwang, Graveley, Joe W. Gray, Linnea Jannson, John P. Kane, Nicholas Katsanis, Tadashi Imanishi, Jules Jacobsen, Chan‐Seok Jeong, Yuxiang Jiang, Martin Kircher, Pui‐Yan Kwok, Rick Lathrop, Jonathan H. LeBowitz, David T. Jones, Daniel Jordan, Thomas Joseph, Beomchang Kang, Emanuela Leonardi, Xiaoming Liu, Federica Lovisa, Angel C. Y. Mak, Rachel Karchin, Mostafa Karimi, Panagiotis Katsonis, Sunduz Keles, Mary J. Malloy, Kenneth Matreyek, Christian Marshall, Richard , Henry Kenlay, Nikki Kiga, Dongsup Kim, Eiru Kim, McCombie,ChiaraMenin,M.StephenMeyn,JohnMoult,Alessandra Philip M. Kim, Jack F. Kirsch, Michael Kleyman, Sujatha Kotte, Murgia, Thomas Nalpathamkalam, Robert L. Nussbaum, Lipika R. Pal, Andreas Kraemer, Anat Kreimer, Ivan Kulakovskiy, Anshul Kundaje, Michael Parsons, Britt‐Sabina Petersen, Mehdi Pirooznia, James B. Kunal Kundu, Pui‐Yan Kwok, Carmen Lai, Ernest Lam, Doron Lancet, Potash, Clive R. Pullinger, Jasper Rine, Frederick P. Roth, Kevin Ru, Dae Lee, Gyu Rie Lee, Insuk Lee, Pietro Di Lena, Emanuela Leonardi, Pardis Sabeti, Jeremy Sanford, Maria C. Scaini, Nicole Schmitt, Jay Andy Li, Biao Li, Mulin Jun Li, Yue Li, Olivier Lichtarge, Ivan Shendure, Molly Sheridan, Michael Snyder,AmandaSpurdle,LeaStarita, Limongelli, Chiao‐Feng Lin, Quewang Liu, Rhonald C. Lua, Angel Mak, Wilson Sung, Bhooma Thiruvahindrapuram, Tim Sterne‐Weiler, Joe Vsevolod J. Makeev, Gennady Margolin, Pier Luigi Martelli, David Whitney, Paul L. F. Tang, Sean Tavtigian, Ryan Tewhey, Silvio C. E. Masica, Zev Medoff, Aziz M. Mezlini, Maximilian Miller, Gilad Mishne, Tosatto, Jochen Weile, G. Karen Yu, Peter Zandi, and Elad Ziv. Rahul Mohan, Alejandro Moles‐Fernández, Gemma Montalban, We offer great thanks to the CAGI assessors who have devoted Alexander M. Monzon, Sean D. Mooney, Oluwaseyi Adebayo extensive time to analysis and evaluation of the results: Aashish Moronfoye, Matthew Mort, John Moult, Steve Mount, Eliseos Adhikari, Michael A. Beer, Yana Bromberg, Emidio Capriotti, Marco Mucaki, Jonathan Mudge, Nikola Mueller, Chris Mungall, Katsuhiko Carraro, John‐Marc Chandonia, Rui Chen, Luigi Chiricosta, Wyatt T. Murakami, Yoko Nagai, Tatsuhiko Naito, Thi Yen Duong Nguyen, Clark, Melissa Cline, Roxana Daneshjou, Roland Dunbrack, Iddo Giovanna Nicora, Noushin Niknafs, Abhishek Niroula, Conor M. L. Friedberg, Mabel Furutsuki, Gad Getz, Manuel Giollo, Nick Grishin, Nodzak, Robert O’Connor, Yanay Ofran, Ayodeji Olatubosun, Lars Maricel Kann, Rachel Karchin, Anat Kreimer, Greg McInnes, M. Ootes, Selen Özkan, Kivilcim Ozturk, Natàlia Padilla, Kymberleigh Stephen Meyn, Sean D. Mooney, Alexander A. Morgan, John Moult, Pagel, Debnath Pal, Lipika R. Pal, Taeyong Park, Nathaniel Pearson, Steve Mount, Robert L. Nussbaum, Ayoti Patra, Francesco Reggiani, Vikas Pejaver, Jian Peng, Dmitry D. Penzar, Alexandra Piryatinska, Jeremy Sanford, David B. Searls, Dustin Shigaki, Artem Sokolov, Josh Catherine Plotts, Jennifer Poitras, Predrag Radivojac, Sadhna Rana, Stuart, Shamil Sunyaev, Sean Tavtigian, Silvio C. E. Tosatto, Alin Aditya Rao, Aliz R. Rao, Sandra Bonache Real, John Reid, Casandra Voskanian, Qifang Xu, and Nir Yosef. Riera, Graham Ritchie, Ettore Rizzo, Peter Rogan, Frederic Rousseau, Our advisory board and scientific council members have been the Vangala Govindakrishnan Saipradeep, Castrense Savojardo, Jana M. source of much wisdom and assistance. The board has comprised: Russ Schwarz, Joost Schymkowitz, Chaok Seok, George Shackelford, Altman, , George Church, Tim Hubbard, Scott Kahn, Sohela Shah, Maxim Shatsky, Yang Shen, Yuan Shi, Ron Shigeta, Rachel Karchin, Sean D. Mooney, Pauline Ng, Robert L. Nussbaum, Hashem A. Shihab, Jung E. Shim, Junha Shin, Sunyoung Shin, Ilya Susanna Repo, John Shon, and Michael Snyder; our Scientific Council 1200 | ANDREOLETTI ET AL. additionally has included: , Yana Bromberg, Atul Butte, John Moult http://orcid.org/0000-0002-3012-2282 Garry R. Cutting, Laura Elnitski, Reece Hart, Ryan Hernandez, Michael Steven E. Brenner http://orcid.org/0000-0001-7559-6185 Snyder, Shamil Sunyaev, Joris Veltman, Liping Wei, and Justin Zook. The ethicists and patient advocates of the CAGI Ethics Forum have REFERENCES worked with organizers, undertaking a critical role in ensuring CAGI operates on a firm ethical foundation and informing its activities: Adhikari, A. N. (2019). Gene‐specific features enhance interpretation of ‐ Human Wylie Burke, Larry Carr, Flavia Chen, Steve Grossman, Julie Harris‐ mutational impact on acid alpha glucosidase enzyme activity. Mutation, https://doi.org/10.1002/humu.23846. Wai, Kirsten Isgro, Barbara A. Koenig, Selena Martinez, Robert L. Aspromonte, M. C., Bellini, M., Gasparini, A., Carraro, M., Bettella, E., Polli, Nussbaum, Jodi Paik, and Mark Yarborough. R., … Leonardi, E. (2019). Characterization of Intellectual disability and We also thank those who have helped organize the CAGI Autism comorbidity through gene panel sequencing. Human Mutation, experiments and who assisted with the associated technology: Daniel https://doi.org/10.1002/humu.23822 Cao, Y., Sun, Y., Karimi, M., Chen, H., Moronfoye, O., & Shen, Y. (2019). Barsky, John‐Marc Chandonia, Ajithavalli Chellappan, Flavia Chen, Predicting pathogenicity of missense variants with weakly supervised “ ” Navya Dabbiru, Reece Hart (who coined the term CAGI ), Roger regression. Human Mutation, 23826. https://doi.org/10.1002/humu.23826 Hoskins, Naveen Kollipara, Melissa K. Ly, Andrew J. Neumann, Carraro, M., Monzon, A. M., Chiricosta, L., Reggiani, F., Aspromonte, M. C., Gaurav Pandey, Sadhna Rana, Susanna Repo, Rajgopal Srinivasan, Bellini, M., … Leonardi, E. (2019). Assessment of patient clinical descriptions and pathogenic variants from gene panel sequences in Stephen Yee, Sri Jyothsna Yeleswarapu, and Maya Zuhl. We specially the CAGI‐5 intellectual disability challenge. Human Mutation, https:// thank those who helped in writing several papers for CAGI5 special doi.org/10.1002/humu.23823 issue: Constantina Bakolitsa, Zhiqiang Hu, and Laura Kasak. Chen, J. (2019). A fully‐automated event‐based variant prioritizing We are truly grateful to NHGRI and NCI for the support of the CAGI solution to the CAGI5 intellectual disability gene panel challenge. Human Mutation experiment and conference. We especially recognize Tata Consultancy , https://doi.org/10.1002/humu.23781 Chen, K., Lu, Y., Zhao, H., & Yang, Y. (2019). Predicting the change of exon Services (TCS), which has been a collaborator in organizing all the CAGI splicing caused by genetic variant using support vector regression. experiments. We also thank the members of the Brenner and Moult Human Mutation, https://doi.org/10.1002/humu.23785 laboratories who have contributed to CAGI. Cheng, J., Çelik, M. H., Nguyen, Y. D., Avsec, Ž., & Gagneur, J. (2019). For this special issue we greatly appreciate the efforts of the CAGI5 splicing challenge: Improved exon skipping and intron retention predictions with MMSplice. Human Mutation, https://doi. anonymous peer reviewers, the Lead Guest Editor Rachel Karchin, and org/10.1002/humu.23788 the Consulting Guest Editors who oversaw the peer‐review process: Clark, W. T., Andreoletti, G., Kasak, L., Bakolitsa, C., Hu, Z., Babbi, G., … Tim Hubbard, Sean D. Mooney, Robert L. Nussbaum, Predrag LeBowitz, J. H. (2019). Assessment of predicted enzymatic activity of ‐ ‐ Radivojac, Shamil Sunyaev, and Joris Veltman. John Moult and Steven alpha N acetylglucosaminidase (NAGLU) variants of unknown significance for CAGI 2016. Human Mutation, https://doi.org/10.1002/humu.23875 E. Brenner were the Issue Editors, and Lipika R. Pal was the Organizing Cline, M. S., Babbi, G., Bonache, S., Cao, Y., Casadio, R., … de la Cruz, X. Human Mutation Editor for this special issue of . We thank Mohammad ENIGMA Consortium (2019). Assessment of blind predictions of the Israr, Christine Murray, Mark Paalman, and Allain Selga for their work clinical significance of BRCA1 and BRCA2 variants. Human Mutation, in coordinating the editorial and production operations at Wiley. https://doi.org/humu.23861 Dong, S., & Boyle, A. P. (2019). Predicting functional variants in enhancer Finally, we also wish to acknowledge our great debt to the many and promoter elements using RegulomeDB. Human Mutation, https:// individuals who shared their private genetic and phenotypic or doi.org/10.1002/humu.23791 clinical information as participants in the research studies and clinical Garg, A., & Pal, D. (2019). Exploring the use of molecular dynamics in data sets included in the CAGI challenges. assessing protein variants for phenotypic alterations. Human Mutation, https://doi.org/10.1002/humu.23800 Gotea, V., Margolin, G., & Elnitski, L. (2019). CAGI experiments: Modeling sequence variant impact on gene splicing using predictions from CONFLICT OF INTERESTS computational tools. Human Mutation, https://doi.org/10.1002/humu. 23782 There is no conflict of interest involved in this study. Hoskins, R. A., Repo, S., Barsky, D., Andreoletti, G., Moult, J., & Brenner, S. E. (2017). Reports from CAGI: The Critical Assessment of Genome Interpretation. Human Mutation, 38, 1039–1041. https://doi.org/10. FUNDING 1002/humu.23290 Hu, Z., Yu, C., Furutsuki, M., Andreoletti, G., Ly, M., Hoskins, R., … Brenner, The CAGI experiment coordination is supported by NIH / NHGRI and S. E. (2019). VIPdb, a genetic variant impact predictor database. NCI U41 HG007346 and the CAGI conference by R13 HG006650. Human Mutation, https://doi.org/10.1002/humu.23858 Support has also been provided via a research agreement with Tata Kasak, L., Bakolitsa, C., Hu, Z., Rine, J., Dimster‐Denk, D. F., Pandey, G., … Consultancy Services (TCS). Friedberg, I. (2019). Assessing computational predictions of the phenotypic effect of cystathionine‐beta‐synthase variants. Human Mutation, https://doi.org/10.1002/humu.23868 Kasak, L., Hunter, J. M., Udani, R., Bakolitsa, C., Hu, Z., Adhikari, A. N., … ORCID Meyn, M. S. (2019). CAGI SickKids challenges: Assessment of phenotype and variant predictions derived from clinical and genomic Gaia Andreoletti http://orcid.org/0000-0002-0452-0009 data of children with undiagnosed diseases. Human Mutation, https:// Lipika R. Pal http://orcid.org/0000-0002-3390-110X doi.org/10.1002/humu.23874 ANDREOLETTI ET AL. | 1201

Katsonis, P., & Lichtarge, O. (2019). CAGI5: Objective performance variants in cancer tissues. Human Mutation, https://doi.org/10.1002/ assessments of predictions based on the evolutionary action equation. humu.23789 Human Mutation, https://doi.org/10.1002/humu.23873 Rhine, C. L., Neil, C., Glidden, D. T., Cygan, K. J., Fredericks, A. M., Wang, J., Kreimer, A., Yan, Z., Ahituv, N., & Yosef, N. (2019). Meta‐analysis of … Fairbrother, W. G. (2019). Future directions for high‐throughput massively parallel reporter assays enables prediction of regulatory splicing assays in precision medicine. Human Mutation, https://doi.org/ activity. Human Mutation, https://doi.org/10.1002/humu.23820 10.1002/humu.23866 Kreimer, A., Zeng, H., Edwards, M. D., Guo, Y., Tian, K., Shin, S., … Yosef, N. Savojardo,C.,Petrosino,M.,Babbi,G.,Bovo,S.,Corbi‐verge, C., (2017). Predicting gene expression in massively parallel reporter Casadio, R., … Capriotti, E. (2019). Evaluating the predictions of the assays: A comparative study. Human Mutation, 38, 1240–1250. protein stability change upon single amino acid substitutions for https://doi.org/10.1002/humu.23197 the FXN CAGI5 challenge. Human Mutation, https://doi.org/10. McInnes,G.,Daneshjou,R.,Katsonis,P.,Lichtarge,O.,Srinivasan,R.G.,Rana,S., 1002/humu.23843 … Altman, R. (2019). Predicting venous thromboembolism risk from Savojardo, C., Babbi, G., Bovo, S., Capriotti, E., Martelli, P. L., & Casadio, R. exomes in the Critical Assessment of Genome Interpretation (CAGI) (2019). Are machine learning based methods suited to address challenges. Human Mutation, https://doi.org/10.1002/humu.23825 complex biological problems? Lessons from CAGI‐5 challenges. Human Miller, M., Wang, Y., & Bromberg, Y. (2019). What went wrong with Mutation, https://doi.org/10.1002/humu.23784 variant effect predictor performance for the PCM1 challenge. Human Shigaki,D.,Adato,O.,Adhikar,A.N.,Dong,S.,Hawkins‐Hooker, A., Inoue, F., Mutation, https://doi.org/10.1002/humu.23832 … Beer, M. A. (2019). Integration of multiple epigenomic marks improves Monzon, A. M., Carraro, M., Chiricosta, L., Reggiani, F., Han, J., Ozturk, K., prediction of variant impact in saturation mutagenesis reporter assay. … Tosatto, S. C. E. (2019). Performance of computational methods for Human Mutation, https://doi.org/10.1002/humu.23797 the evaluation of Pericentriolar Material 1 missense variants in CAGI‐ Strokach, A., Corbi‐Verge, C., & Kim, P. M. (2019). Predicting changes in 5. Human Mutation, https://doi.org/10.1002/humu.23856 protein stability caused by mutation using sequence‐ and structure‐ Mount,S.M.,Avsec,Ž., Carmel, L., Casadio, R., Celik, M. H., Chen, K., … Zhao, based methods in a CAGI5 blind challenge. Human Mutation, https:// H. (2019). Assessing predictions of the impact of variants on splicing in doi.org/10.1002/humu.23852 CAGI5. Human Mutation, https://doi.org/10.1002/humu.23869 Voskanian, A., Katsonis, P., Lichtarge, O., Pejaver, V., Radivojac, P., Naito, T. (2019). Predicting the impact of single nucleotide variants on Mooney, S. D., … Kann, M. G. (2019). Assessing the performance of in‐ splicing via sequence‐based deep neural networks and genomic silico methods for predicting the pathogenicity of variants in the gene features. Human Mutation, https://doi.org/10.1002/humu.23794 CHEK2, among Hispanic females with breast cancer. Human Mutation, Padilla, N., Moles‐Fernández, A., Riera, C., Montalban, G., Özkan, S., Ootes, https://doi.org/10.1002/humu.23849 L., … de la Cruz, X. (2019). BRCA1‐ and BRCA2‐specific in silico tools Wang, R., Wang, Y., & Hu, Z. (2019). Using secondary structure to predict for variant interpretation in the CAGI 5 ENIGMA challenge. Human the effects of genetic variants on alternative splicing. Human Mutation, Mutation, https://doi.org/10.1002/humu.23802 https://doi.org/10.1002/humu.23790 Pal, L. R., Kundu, K., Yin, Y., & Moult, J. (2019). Matching whole Wang, Y., & Bromberg, Y. (2019). Identifying mutation‐driven changes in genomes to rare genetic disorders: Identification of potential gene functionality that lead to venous thromboembolism. Human causative variants using phenotype‐weighted knowledge in the Mutation, https://doi.org/10.1002/humu.23824 CAGI SickKids5 clinical genomes challenge. BioRxiv, 707687. Zhang, J., Kinch, L. N., Cong, Q., Katsonis, P., Lichtarge, O., Savojardo, C., … https://doi.org/10.1101/707687 Grishin, N. V. (2019). Assessing predictions on fitness effects of Parsons, M. T., Tudini, E., Li, H., Hahnen, E., Wappenschmidt, B., missense variants in calmodulin. Human Mutation, https://doi.org/10. Feliubadaló, L., … Spurdle, A. B. (2019). Large scale multifactorial 1002/humu.23857 likelihood quantitative analysis of BRCA1 and BRCA2 variants: An ENIGMA resource to support clinical variant classification. Human Mutation, https://doi.org/10.1002/humu.23818 Pejaver, V., Babbi, G., Casadio, R., Folkman, L., Katsonis, P., Kundu, K., … How to cite this article: Andreoletti G, Pal LR, Moult Bromberg, Y. (2019). Assessment of methods for predicting the J, Brenner SE. Reports from the fifth edition of CAGI: The effects of PTEN and TPMT protein variants. Human Mutation, https:// Critical Assessment of Genome Interpretation. Human Mutation. doi.org/10.1002/humu.23838 – Petrosino, M., Pasquo, A., Novak, L., Toto, A., Gianni, S., Mantuano, E., … 2019;40:1197 1201. https://doi.org/10.1002/humu.23876 Consalvi, V. (2019). Characterization of human frataxin missense