Bachelor's Degree in Telecommunication Technologies Engineering Academic Year 2018/2019 Bachelor Thesis “A review on Generative Adversarial Networks for unsupervised Machine Learning” Pablo Javier de Paz Carbajo Supervised by: Pablo Martínez Olmos Madrid, June 2019 This work is licensed under Creative Commons Attribution – Non Commercial – Non Derivatives ABSTRACT This thesis is conceived to provide an in-depth review of Generative Adversarial Networks (GANs) for unsupervised Machine Learning by introducing the basic concepts of Deep Learning as one of the main current techniques for unsupervised learning with deep neural networks as well as carefully explaining what and how a GAN works. The approach followed during this year-long work consists in learning and internalizing the technical concepts of Deep Learning and, particularly, GANs. After that, an analysis of the most used techniques takes place. For the experimentation, TensorFlow over Python is used to obtain experimental results for different types of public datasets of images (MNIST, Fashion-MNIST and CIFAR–10). GANs are outstandingly interesting in Deep Learning because of its endless capabilities and huge potential in many fields such as the health industry, the agriculture, banking and insurance, transportation, security, education… Its capabilities to generate new data that, mimics the input learning data given to it, are impressive and never seen before. This method is very nice approach for Data Science when dealing with a huge amount of (unlabeled) data, like most of the companies, institutions and governments do today. This thesis is restricted to image applications, but the outcomes can be applied to other datasets or fields such as text or voice applications. The results obtained and presented in this document provide a clear evidence that the adversarial networks used in my experiments are pretty powerful as they are creating coherent images from random noise. The evaluation datasets correspond to the ones typically used in ML and the results obtained meet the technical specifications of these networks to greater or lesser extent depending on the computational power available and the required one. Overall, you will find out the incredible importance of Generative Adversarial Networks in today’s world and the almost infinite number of processes and activities that they can be applied to. Key words: Unsupervised Learning, Generative Model, Data Science, Deep Learning, Artificial Neural Networks, Generative Adversarial Networks, Python, TensorFlow. I II ACKNOWLEDGMENTS This Bachelor Thesis would not have been possible without the help and support of many friends and acquaintances that I have made during my life. I am very grateful to them because they always believe in me. First of all, I would really like to thank Pablo Martínez Olmos, tutor of this thesis. Around one year ago, I asked him to be my thesis supervisor. Without hesitation, he agreed, so I could start researching some topic to work on it related to the field of Machine Learning. Later on, we agreed on a topic to do the thesis on and, in addition, he taught me a very practical course about the most common Machine Learning techniques. He has been very supportive, flexible and patient in everything I do. Pablo is an outstanding professor and for that reason, I have been able to learn a lot from him and from all the references he has suggested me to use. I am very glad for having worked with Pablo in my bachelor thesis. Thanks a lot! Secondly, I really like to thank Sinno Jialin Pan too. He is a professor in Nanyang Technological University (Singapore), and he taught me a course about Machine Learning while I was studying there in third year. This course aroused my curiosity and taste for this topic. Next, I am very grateful to my family and, more specifically, my parents and my sister. They have always believed in me, even knowing that the degree was not easy and required a lot of hard work. They have done whatever possible to make me feel comfortable and in a good mood while studying. In moments when I felt down, they have undoubtedly supported and encouraged me to keep on studying and working. Thank you! Finally, and not least, there are many people, friends, that have helped me a lot during the last four years of university. I have no words to say how much I appreciate all the help and loyalty I have received in moments of need. They have been up to the challenge and I have been able to learn a lot from them, both in a personal and academic way. III IV TABLE OF CONTENTS 1. INTRODUCTION ............................................................................................... 1 1.1. Motivation ...................................................................................................... 1 1.2. Objectives ....................................................................................................... 2 1.3. Learning Outcomes ......................................................................................... 2 1.4. Thesis Structure .............................................................................................. 3 2. SOCIAL AND ECONOMIC ENVIRONMENT................................................. 5 2.1. Impact and applications of GANs.................................................................... 5 2.2. Project planning .............................................................................................. 6 2.3. Project budget ................................................................................................. 8 3. LEGISLATIVE AND REGULATORY FRAMEWORK .................................. 9 4. STATE OF THE ART ....................................................................................... 11 4.1. Artificial Intelligence and Machine Learning ................................................ 11 4.2. Deep Learning and Artificial Neural Networks (ANN).................................. 13 4.3. Artificial Neural Networks (ANN) ................................................................ 14 4.3.1. Multiple Linear Regression .......................................................................................... 16 4.3.2. Activation Function ..................................................................................................... 17 4.3.3. Gradient Descent ......................................................................................................... 17 4.3.4. Logistic Regression ..................................................................................................... 19 4.3.5. Loss Function .............................................................................................................. 20 4.3.6. Types of ANN ............................................................................................................. 21 Feedforward Networks ............................................................................................................. 21 Convolutional Neural Networks ............................................................................................... 21 5. ANALYSIS OF A GENERATIVE ADVERSARIAL NETWORK ................. 23 5.1. Introduction and motivation .......................................................................... 23 5.2. Generator and Discriminator ......................................................................... 24 5.3. Implementation and technology in use .......................................................... 27 5.4. Evaluation and results ................................................................................... 28 5.4.1. MNIST Dataset ........................................................................................................... 28 5.4.2. Fashion MNIST Dataset .............................................................................................. 31 5.4.3. CIFAR-10 Dataset ....................................................................................................... 34 V 1. “Automobile” class .............................................................................................................. 36 2. “Airplane” class ................................................................................................................... 38 3. “Dog” class .......................................................................................................................... 40 Comparison between tested classes .......................................................................................... 42 6. CONCLUSIONS ................................................................................................ 43 6.1. Thesis conclusions ........................................................................................ 43 6.2. Future work .................................................................................................. 44 GLOSSARY .............................................................................................................. 45 REFERENCES ......................................................................................................... 46 VI LIST OF FIGURES Figure 1. Cumulative number of papers related to GANs (up to October 2018) [8] ........ 5 Figure 2. List of activities for the proposed project ........................................................ 7 Figure 3. GANTT Diagram for the proposed project ..................................................... 7 Figure 4. Different ways AI can be achieved today [18]. ............................................