Multimodal Data Acquisition for Dermatology using Google Glass Joao˜ Guilherme Antunes Martins Thesis to obtain the Master of Science Degree in Biomedical Engineering Supervisors Prof. Dr. Joao˜ Paulo Salgado Arriscado Costeira Prof. Dr. Jorge dos Santos Salvador Marques Examination Committee Chairperson: Prof. Dr. Joao˜ Pedro Estrela Rodrigues Conde Supervisor: Prof. Dr. Joao˜ Paulo Salgado Arriscado Costeira Member of the Committee: Prof. Dra. Ana Lu´ısa Nobre Fred November 2014 They don’t think it be like it is, but it do. - Cpt. Oscar Gamble Acknowledgments I would like to thank my grandfather, Henrique, for his warmth and sense of humor. To my grand- mother, Lourdes, I thank her for showing me that love is always the way. To professor Jose´ Tribolet, I thank you for being a steady compass in my time of intelectual turbulence. To professor Joao˜ Paulo Costeira, I thank you for changing my life. It won’t be forgotten. I would also like to thank professor Jorge Salvador. Dra. Martinha, I would also like to thank you for never saying no to what we, as engineering-minded people, asked of you, over the course of this thesis. Finally, I thank my mother for shaping me and my life. And to Joana, I thank you for everything. iii Abstract Melanoma is now the deadliest form of cancer. The best survivability rate is still attained with early detection. To try to ensure that, several diagnosis methodologies have been developed specifically to optimize fast melanoma screening, like the ABCD rule and Menzies’ method. To complement this, several computer aided diagnosis systems have also surfaced. These systems are plagued by the low availability of public labeled datasets, due to the resource intensity the task of creating a dataset entails, with which to learn and train. Also, state-of-the-art systems feature no easy way to locate magnified lesion images in space (where in the patients’ body). This thesis proposes a hands-free non-invasive, wireless system using Google Glass that enables labeled dataset creation on-the-fly during a doctor’s appointment, without the need to have an expert clinician spending hundreds on man-hours labelling image sets. The proposed system also includes a module to, without any extra input from the user, locate the lesion in the patients body, resorting to identifiable tags. Finally, a tele-dermatology framework using the proposed system is described. The proposed system worked as expected, but some limitations were encountered, mainly due to Google Glass’s immaturity as a product, and due to the fact that the detail camera used to obtain the lesions was a smartphone camera, as opposed to a dermoscope camera. Nevertheless, the proof-of-concept entailed was successful. Keywords Labelled dataset, Google Glass, Super-Resolution, Melanoma, Dermoscopy v Resumo A forma de cancro com maior mortalidade e,´ neste momento, melanoma. A estrategia´ de trata- mento com maior taxa de sobrevivenciaˆ e´ a detecc¸ao˜ precoce. Para auxiliar a detecc¸ao˜ precoce varias´ estrategias´ foram criadas, como a regra ABCD e o metodo´ de Menzies. Para complementar estas estrategias,´ foram desenvolvidos sistemas informaticos´ de aux´ılio ao diagnostico.´ Estes sis- temas sao˜ limitados pela baixa disponibilidade de bases de dados com anotac¸oes,˜ devido a` quanti- dade de recursos que estas requerem para ser criadas, com as quais os sistemas sao˜ desenvolvidos e treinados. Alem´ disso, os sistemas estado-da-arte nao˜ possuem maneira de localizar imagens magnificadas de lesoes˜ no espac¸o (onde, no corpo do doente, se situa a lesao).˜ Este trabalho propoe˜ um sistema maos˜ livres, nao˜ invasivo e sem fios, com recurso ao Google Glass, que permite a criac¸ao˜ de bases de dados anotadas durante uma consulta, sem recorrer a ter um especialista a anotar manualmente centenas (milhares) de imagens. O sistema proposto inclui um modulo´ que, sem requerer informac¸ao˜ extra do utilizador, localiza no espac¸o as imagens de lesoes˜ adquiridas, usando marcadores. Finalmente, uma estrutura para tele-dermatologia usando o sistema proposto e´ descrita. O sistema proposto funciona conforme o esperado, embora possua algumas limitac¸oes,˜ derivando principalmente da imaturidade do Google Glass como produto, e devido ao facto de a cameraˆ de detalhe utilizada para obter imagens das lesoes˜ provir de um smartphone por oposic¸ao˜ a um der- matoscopio.´ Ainda assim, a prova-de-conceito foi bem sucedida. Palavras Chave Base-de-dados Anotada, Google Glass, Super-Resolution, Melanoma, Dermoscopia vii Contents 1 Introduction 1 1.1 Motivation . .2 1.2 Thesis Scenario . .3 1.3 Original Contributions . .5 1.4 Thesis Outline . .5 2 Dermoscopy 7 2.1 Dermoscopy - Introduction . .8 2.2 Differential Structures relevant for Melanoma Diagnosis . .8 2.3 Melanocytic Lesions . 13 2.4 ABCD rule . 15 2.5 7-point checklist . 17 2.6 Menzies’ Method . 17 2.7 Relevance to this thesis . 18 3 Computer-Aided Diagnosis Systems 19 3.1 Definition of Computer-Aided Diagnosis (CAD) systems . 20 3.2 Image Preprocessing . 20 3.3 Border Detection . 21 3.4 Feature Extraction . 21 3.5 Lesion Classification . 22 3.6 CAD performance . 23 3.7 Challenges found when developing a CAD system . 24 3.8 Extra Parameter - Change detection . 24 3.9 Labeled Datasets freely available . 25 4 Multimodal Data Acquisition System - Glass Aided Diagnosis (GAD) system 27 4.1 System Overview . 28 4.2 Interaction with the system . 30 4.3 System Details . 33 4.3.1 Google Glass . 33 4.3.2 Detail Camera . 34 ix 4.3.3 Server . 34 4.4 Live examples of the system running . 35 4.4.1 First Example - Two lesions on an arm . 36 4.4.2 Second Example - One lesion on the back . 38 4.5 Tele-Dermatology . 39 4.6 Image Registration . 40 4.6.1 Camera Model and Pose Estimation . 41 4.6.2 Determining the location of the lesions . 42 5 Super Resolution 45 5.1 Super-Resolution in the scope of this thesis . 46 5.2 Observation Model . 46 5.3 Super-Resolution techniques . 48 5.4 Example of a Super-Resolved Lesion . 51 6 Conclusions and Future Work 55 6.1 Conclusions . 56 6.2 Future Work . 57 Bibliography 59 x List of Figures 1.1 Proposed system . .3 1.2 Thesis Scenario . .4 2.1 Dermoscope . .9 2.2 Camera - Pigment network in a Clark Nevus . .9 2.3 Dermoscope - Pigment network in a Clark Nevus . 10 2.4 Camera - Streaks found in a melanoma . 10 2.5 Dermoscope - Streaks found in a melanoma . 10 2.6 Camera - Dots and globules found in a Clark Nevus . 11 2.7 Dermoscope - Dots and globules found in a Clark Nevus . 11 2.8 Camera - Blue-whitish veil found in a Melanoma . 11 2.9 Dermoscope - Blue-whitish veil found in a Melanoma . 12 2.10 Camera - Scar like regression structures found in a Melanoma . 12 2.11 Dermoscope - Scar like regression structures found in a Melanoma . 13 4.1 Diagram of the whole system . 29 4.2 Diagram of the communication pipeline . 30 4.3 View of all the components of the proposed system . 31 4.4 Google Glass main prompt . 32 4.5 Diagram of a complete acquisition session . 32 4.6 Google Glass R by Google . 33 4.7 Matlab Graphical User Interface (GUI) . 35 4.8 Matlab GUI Start-Up . 36 4.9 Example 1: Global Image . 36 4.10 Example 1: First Lesion . 37 4.11 Example 1: Second Lesion . 37 4.12 Example 1: Lesion Map . 38 4.13 Example 2: Global Image . 38 4.14 Example 2: Detail Image of the Back Lesion . 39 4.15 Example 2: Lesion Map . 39 4.16 Displayed image . 40 4.17 Auxiliary image . 41 xi 4.18 GUI with the lesion map . 42 4.19 Diagram demonstrating the relationship between the referentials. 42 5.1 Super-Resolution: Original Camera Image . 51 5.2 Super-Resolution: Super-Resolved Camera Image . 52 5.3 Super-Resolution: Dermatoscope Image . 52 xii List of Tables 2.1 ABCD rule . ..