Ways to use Machine Learning approaches for software development Nicklas Jonsson Nicklas Jonsson VT 2018 Examensarbete, 30 hp Supervisor: Eddie wadbro Extern Supervisor: C4 Contexture Examiner: Henrik Bjorklund¨ Master of Science Programme in Computing Science and Engineering, 300 hp Abstract With the rise of machine learning and in particular deep learning enter- ing all different types of fields, including software development. It could be a bit hard to know where to begin to search for the tools when some- one wants to use machine learning for a one’s problems. This thesis has looked at some available technologies of today for applying machine learning to one’s applications. This thesis has looked at some of the available cloud services, frame- works, and libraries for machine learning and it presents three different implementation structures that can be used with these technologies for the problem of image classification. Acknowledgements I want to thank C4 Contexture for giving me the thesis idea, support, and supplying me with a working station. I also want to thank Lantmannen¨ for supplying me with the image data that was used for this thesis. Finally, I want to thank Eddie Wadbro for his guidance during this thesis and of course a big thanks to my family and friends for their support during this period of my life. 1(45) Content 1 Introduction 3 1.1 The Client 3 1.1.1 C4 Contexture PIM software 4 1.2 The data 4 1.3 Goal 5 1.4 Limitation 5 2 Background 7 2.1 Artificial intelligence, machine learning and deep learning. 7 2.1.1 Artificial intelligence 7 2.1.2 Machine learning 8 2.1.3 Learning representations from data 8 2.1.4 Deep learning 9 2.2 Cloud services for Machine learning 10 2.2.1 IBM Watson 10 2.3 Frameworks and libraries for machine learning 11 2.4 Convolutional neural networks 11 2.4.1 The problem space 11 2.4.1 What is an image to a CNN? 12 2.4.2 Structure 12 2.4.2 The convolution operation 13 2.4.2 The max-pooling operation 15 2.5 Using a pretrained CNN 16 3 Methods 17 3.1 Cleaning up the data 17 3.2 IBM Watson Visual recognition 18 3.2.1 Create custom classifiers on Watson 19 3.3 Using frameworks and libraries for machine learning 20 3.3.1 Tensorflow 20 3.3.2 Keras 20 3.3.3 Tensorflow.js 21 3.3.4 Models 21 3.3.5 Retrain the models 21 3.4 Prepare the imagedata for the training phase 23 3.5 As applications 25 4 Results 27 4.1 Watson Visual Recognition 27 4.2 TensorFlow with Inception V3 28 4.3. Keras with MobileNet 29 2(45) 4.4 Converting the MobileNet Keras model into TensorFlow.js format 30 4.5 As applications 31 4.5.1 Swing application using Watson 31 4.5.2 Web application using Inception V3 32 4.5.3 MobileNet and TensorFlow.js 33 5 Discussion 35 5.1 Services 35 5.2 Frameworks and libraries 35 5.3 The applications 36 5.3.1 Watson as service 36 5.3.2 Own service for hosting models 36 5.3.3 Loading models into the browser. 36 6 Conclusion 39 6.1 Future work 39 References 41 3(45) 1 Introduction “ The consequences of this sudden progress extend to almost every industry. But in order to begin deploying deep-learning technology to every problem that it could solve, we need to make it accessible to as many people as possible, including non-experts, people who aren’t researchers or graduate students. For deep learning to reach its full potential, we need to radically democratize it [8].” - Francois Chollet, the creator of the Keras deep-learning library, about deep learning. Artificial intelligence and in particular its subfields machine learning and deep learning are changing all types of business functions, and software development is not left unaffected by this [1]. Traditional programming is mostly combined with logic operations that will tell the program what to do. We used to be tough that computers are dumb [3], meaning they does only do exactly what they are told to do, nothing more and nothing less. But in the last couple of years, this concept has been challenged [4]. AI for businesses is today mostly made up of different type of machine learning algorithms. These algorithms are compiled in order to teach the systems to learn something from some type of data that it has been handed. This simplifies, scales and also introduces solutions for problems that previously was considered impossible to solve when just using the traditional programming approach [2]. This thesis explores how machine learning approaches can be used in software development. This thesis is gonna focus and present solutions in the machine learning field of vision recognition and image tagging, but also briefly explain where software developers can find machine learning solutions for other functionalities. The focus of this work has been to find some of the “easier” ways to implement machine learning solutions for developers that are non-experts within the field of machine learning. This report will present solutions that takes almost zero knowledge of machine learning, but also some solutions that will require some more learning within the field for a developer to be able to use them in a sufficient way. 1.1 The Client This master thesis was made with the help of C4 Contexture. C4 Contexture is a company that expertise themselves in information logistics [5]. Their main product, also called C4 Contexture, Is a Product Information Management (PIM) software. 4(45) 1.1.1 C4 Contexture PIM software C4 Contexture PIM makes it possible for users to store all product related information. PIM is a system that stores, structure, and distributes the user's information. It makes the information searchable, easy to manage and accessible from any location. The user controls the data flow and decides who sees what [6]. PIM makes it possible to assemble: ● images, videos, and text ● templates, drawings, and documents ● product details (name, price, and supply status) ● files like PDF, InDesign, Excel, and many others. Figure 1.1: C4 Contexture PIM system overview. PIM supports publication of information on multiple platforms like the web, mobile devices, printed media, and more. Figure supplied by C4 Contexture [7]. 1.2 The data The image data used for this thesis is supplied by Lantmännen. Lantmännen is an agricultural cooperative and they are northern Europe's leading player in agriculture, machinery, bioenergy, and food [22]. The data are images of different types of food products. The images mostly depict different kinds of bread and pastries, with both singular products on 5(45) them, but also products that are mixed together, and that are in environments (environment example: second image to the left in figure 1.2 below). ​ ​ Figure 1.2: examples of some of the image data being used. 1.3 Goal The goal of this thesis is to present available techniques and methods that can be used to simplify machine learning approaches for the average software developer. The thesis will also show that it is possible to make different types of implementations with machine learning. In the case of C4 Contexture, they want to implement a solution in their PIM system that automatically tags the image data that they handle for their business customers so that the tags of the images will become more continuously and structured. 1.4 Limitation Even though the machine learning services and frameworks used in this thesis supports many different kinds of fields, this thesis is mostly gonna focus and present solutions for the field of image tagging. The reason for this is that this is the functionality that C4 Contexture wanted to have implemented for their PIM system. 6(45) 7(45) 2 Background This chapter will explain the definitions of artificial intelligence, machine learning, deep learning. And it will also talk about services and frameworks that can be used for machine learning. Also, some extra explanation on the neural network called: Convolutional Neural Network (CNN) will be made since this is the main machine learning model that was used in this thesis. 2.1 Artificial intelligence, machine learning and deep learning. So what is the difference between these “buzzwords” that there is so much talk about? How do they relate to one another? This subchapter will explain more about that. Figure 2.1: the relation between artificial intelligence, machine learning, and deep learning. 2.1.1 Artificial intelligence The concept of artificial intelligence (AI) is old. It originates all the way back to the 1950s. Back then, a handful of pioneers from the early field of computer science asked themselves: “can computers think?”. This is a question that we still exploring today. In short, the field can be explained as follows: the effort to automate intellectual tasks normally performed by humans. As such, AI is a very general field that also includes machine learning and deep learning, but it also includes many more approaches than just learning approaches. The early approaches from the 1950s to the late 1980s were known as symbolic AI. The main example of these are the early chess programs that followed a set of hardcoded rules created by programmers and didn't qualify as machine learning since they never learned to play, they were just programed to “know” how to play chess. For fairly long this approach was believed to be the answer of reaching human-level AI.