OPINION

Deep Learning in Radiology: Now the Real Work Begins

Carolina Lugo-Fagundo, BS, Bert Vogelstein, MD, Alan Yuille, PhD, Elliot K. Fishman, MD

INTRODUCTION emphasize the importance of the building of complex data [11]. This Machine learning systems build data collection required to build the method might function in simpler statistically based mathematical algorithms. tasks like recognizing a dog from models that program computers to In general, the scientificcom- a cat, but not for compound optimize a performance criterion munity agrees that large volumes of data like pancreatic anatomy and from sample data [1].Deeplearning data are required for building pathology. is a form of machine learning that robust deep learning algorithms attempts to mimic the interactivity [9,10]. Most recently, Esteva et al of layers of neurons in the developed an algorithm to separate THE FELIX PROJECT neocortex, the part of the cerebral gross images of skin lesions into The Felix Project, funded by the cortex where most higher thinking nonproliferative benign or malignant Lunstgarten Foundation, is a multi- occurs [2]. Deep learning methods lesions [9]. The investigators utilized disciplinary research study designed can now be used to recognize a set of 129,450 images of skin to determine if deep learning algo- patterns in digital representations lesions for computer training; rithms can be developed to aid in the of sound and images as well as in 2,032 of those images were of interpretation of CT and MR images other data types. skin diseases diagnosed using of the pancreas. The ultimate goal is Recognizing its power, medical noninvasive visual analysis or biopsy to develop a tool to reduce radiolo- researchers and scientists have testing by dermatologists [9]. gist errors and to diagnose changes been exploring the role of deep Esteva et al specified that the in the pancreas that can signal early learning methods to create tools to machine’s performance can be stage, curable pancreatic cancers. alert clinicians to focus on outliers enhanced if it is trained with a The Felix team includes experts identified in data sets and as aids larger data set. Similarly, for deep in medical imaging, pathology, for earlier and more accurate learning to be successful in radiology, oncology, and computer science, diagnosis. Recent studies have it will be necessary to collect and including experts with experience in highlighted the potential power of annotate potentially thousands of artificial intelligence and deep deep learning in a variety of cases representing a range of learning. medical applications such as pathologies. The first step of the project is to imaging-guided survival time pre- These detailed data are used to carefully annotate a large set of diction of patients with brain create deep networks that are trained normal images for machine learning tumors [3], bladder cancer treatment in a supervised way, with a method training. Once the computer is response assessment [4],lung that uses more knowledge and trained on the normal CT data sets, nodule [5] and mammographic information. We have previously the computer should be able to classifications [6], diagnosis of proven that weak supervision, which identify and recognize a normal breast lesion pathologies [7],and specifies a bounding box around an pancreas. The next step will be to body part recognition [8]. object instead of per pixel labeling, train the computer to identify an However, many studies fail to does not work for deep network abnormal pancreas by providing it

ª 2017 American College of Radiology 364 1546-1440/17/$36.00 n http://dx.doi.org/10.1016/j.jacr.2017.08.007 with images of patients with a wide range of pancreatic pathologies, including adenocarcinomas, neuro- endocrine tumors, and pancreatic cysts of various types.

DEEP LEARNING, DEEP WORK If we have learned anything from the initial phase of our project, it is this: deep learning requires deep work. The algorithms developed using deep learning are only as good as the training sets used to train the system. In the case of CTs, this requires the careful annotation of hundreds or even thousands of images in both the arterial and venous phases. Each CT image we provide shows 21 distinct (Fig. 1) segmentations, including segmentation of the pancreas as well as other organs and vessels adjacent to the pancreas. The creation of well-annotated training sets for this study is extremely labor intensive. Our initial goal is to segment at least 1,000 venous and arterial CT images of normal pancreas, and to date we have annotated over 800 cases and are using them as the ground truth to create our complex algorithms. The veracity of the ground truth is critical because the deep learning algorithms can only be as truthful as this reference [9]. Segmentation of the CT images requires accuracy and precision. Using state-of-the-art segmentation software Velocity (Varian Medical System, Atlanta, Georgia, USA) (Fig. 2) and a four-member seg- mentation team, each data set takes about 3 hours to segment. Each organ is manually contoured using 0.75-mm CT images, making sure Fig 1. List of the 21 separate organs and vessels that are contoured. that the margins are clearly defined and differentiated from edge-detection tools and a range of To streamline the process and the other structures. Even though texture-sensitive tools, contouring minimize variations in contouring, the segmentation software offers still requires tedious manual input. we developed best practice

Journal of the American College of Radiology 365 Lugo-Fagundo et al n Opinion Fig 2. Schematic representation of the segmentation work screen. It contains 3-D axial and coronal views of an abdominal 0.75-mm CT in the arterial phase.

guidelines and optimization tech- increasing the number of normal 2. Hof RD. Deep learning: with massive niques that can be updated if and abnormal data sets available. amounts of computational power, machines can now recognize objects needed. This segmentation pro- Finally, it is likely that these algo- and translate speech in real time. cess is also expensive, considering rithms will undergo continuous Artificial intelligence is finally getting smart. MIT Technology Review the cost of salaries, segmentation learning and optimization as they are 2013;116:32. software, and training. Thus a applied in clinical practice in much 3. Nie D, Zhang H, Adeli E, Liu L, Shen D. considerable amount of time, the same way voice recognition 3D deep learning for multi-modal imaging- guided survival time prediction of brain resources, expertise, and labor algorithms improve from their real- tumor patients. Medical image computing are necessary for a project like world deployment. With the exten- and computer-assisted intervention. this, even before the creation sive amount of work that lies Medical Image Computing and Computer- ’ Assisted Intervention 2016 Lecture Notes of the deep learning computer ahead, Winston Churchill s state- in Computer Science, Athens, Greece algorithms. ment seems appropriate: This “is not 2016:212-20. even the beginning of the end. But 4. Cheng JZ, Ni D, Chou YH, et al. Bladder cancer segmentation in CT for it is, perhaps, the end of the treatment response assessment: applica- CONCLUSION beginning” [12]. tion of deep-learning convolution neural The quantity and quality of the network—a pilot study. Tomography training set are critically important 2016;2:421-9. ACKNOWLEDGMENTS 5. Wang C, Elazab A, Wu J, Hu Q. Lung in the development of state-of-the- fi The authors acknowledge the nodule classi cation using deep feature art deep learning in radiology. One fusion in chest radiography. following people who contributed to way to expedite the process would be Comput Med Imaging Graph 2016;57: this article: Ralph Hruban, MD, 10-8. to have a number of institutions 6. Karen Horton, MD, Linda Chu, Kooi T, Litjens G, Ginneken BV, develop standard processes and best et al. Large scale deep learning for MD, and Kenneth Kinzler, MD. computer aided detection of practice guidelines for data segmen- mammographic lesions. Med Image tations and share these data sets. Anal 2017;35:303-12. These collaborations would not limit REFERENCES 7. Cheng J-Z, Ni D, Chou Y-H, et al. 1. Alpaydın E. Introduction to machine Computer-aided diagnosis with deep algorithm development, which is the learning. Cambridge, MA: Massachusetts learning architecture: applications to ultimate goal, but speed it along by Institute of Technology Press; 2014. breast lesions in US images and

366 Journal of the American College of Radiology Volume 15 n Number 2 n February 2018 pulmonary nodules in CT scans. Sci Rep 9. Esteva A, Kuprel B, Novoa RA, et al. learning of a deep convolutional network 2016;6:24454. Dermatologist-level classification of skin for semantic image segmentation. The 8. Yan Z, Zhan Y, Peng Z, et al. Bodypart cancer with deep neural networks. Nature IEEE International Conference on Com- recognition using multi-stage deep 2017;542(7639):115-8. puter Vision (ICCV), IEEE Computer learning. Lecture notes in computer science 10. Jha S, Topol EJ. Adapting to artificial in- Society, Washington DC 2015:1742-50. information processing in medical imaging. telligence. JAMA 2016;316:2353. 12. Knowles, E. (2014). Oxford dictionary of Inf Process Med Imaging, Springer, New 11. Papandreou G, Chen LC, Murphy K, quotations (5th ed.). Oxford: Oxford York. 2015:449-61. Yuille AL. Weakly- and semi-supervised University Press.

Elliot K. Fishman, MD and Carolina Lugo-Fagundo, BS, are from The Russell H. Morgan Department of Radiology and Radiologic Science, Johns Hopkins School of Medicine, Baltimore, Maryland. Bert Vogelstein, MD, is from the Department of Molecular Biology and Genetics. Alan Yuille, PhD is from the Department of Cognitive Science, Johns Hopkins School of Medicine, Bal- timore, Maryland.

The Lustgarten Foundation, Myriad Genetics, miDiagnostics, PapGene, Personal Genome Diagnostics, Morphotek, Syxmex Inostics, Exelixis GP. The Felix project is funded in part by a grant from The Lustgarten Foundation. The authors have no conflicts of interest related to the material discussed in this article. Elliot K. Fishman, MD: The Russell H. Morgan Department of Radiology and Radiologic Science, Johns Hopkins School of Medicine, 601 N Caroline Street, Room 3254, Baltimore, MD 21287; e-mail: efi[email protected].

Journal of the American College of Radiology 367 Lugo-Fagundo et al n Opinion