Domain Adaptation from 3D Synthetic Images to Real Images
Total Page：16
File Type：pdf, Size：1020Kb
Master of Science in Computer Science September 2020 Domain Adaptation from 3D synthetic images to real images Krishna Himaja Manamasa Faculty of Computing, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfilment of the requirements for the degree of Master of Science in Computer Science. The thesis is equivalent to 20 weeks of full time studies. The authors declare that they are the sole authors of this thesis and that they have not used any sources other than those listed in the bibliography and identified as references. They further declare that they have not submitted this thesis at any other institution to obtain a degree. Contact Information: Author: Krishna Himaja Manamasa E-mail: [email protected] University supervisor: Prof. Hakan Grahn Department of Computer science External supervisor: Xiaomeng Zhu E-mail: [email protected] Faculty of Computing Internet : www.bth.se Blekinge Institute of Technology Phone : +46 455 38 50 00 SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57 Abstract Background: Domain adaptation is described as, a model learning from a source data distribution and performing well on the target data. This concept, Domain adaptation is applied to assembly-line production tasks to perform automatic qual- ity inspection. Objectives: The aim of this master thesis is to apply this concept of 3D domain adaptation from synthetic images to real images. It is an attempt to bridge the gap between different domains (synthetic and real point cloud images), by implementing deep learning models that learn from synthetic 3D point cloud (CAD model images) and perform well on actual 3D point cloud (3D Camera images). Methods: Through this course of thesis project, various methods for understand- ing the data and analysing it for bridging the gap between CAD and CAM to make them similar is looked into. Literature review and controlled experiment are research methodologies followed during implementation. In this project we experiment with four different deep learning models with data generated and compare their perfor- mance to know which deep learning model performs best for the data. Results: The results are explained through metrics i.e, accuracy and train time, which were the outcomes of each of the deep learning models after experiment. These metrics are illustrated in the form of graphs for comparative analysis between the models on which the data is trained and tested on. PointDAN showed better results with higher accuracy compared to the other 3 models. Conclusions: The results attained show that domain adaptation for synthetic im- ages to real images is possible with the data generated. PointDAN deep learning model which focus on local feature alignment and global feature alignment with single-view point data show better results with our data. Keywords: Domain adaptation, Transfer learning, Deep learning, 3D point clouds Acknowledgments This thesis is an industrial research, carried out in Smart Factory Lab, TEED, Sca- nia CV AB, Södertälje, Sweden. I take immense pleasure in thanking my supervisor Xiaomeng Zhu for guiding me throughout the whole research project. Many thanks for her patience and support. I would like to thank Prof. Håkan Grahn, my university supervisor and advisor for all the valuable insights and support for making it an incredible learning experi- ence. My special thanks to my manager Franz A Waker for providing such an amazing environment to develop my thesis and to my collegue Juan Luis for helping me gen- erate CAD data. My colleagues at TEED, Smart Factory Lab equally helped me throughout the whole period and made it enjoyable. Lastly, I thank my parents Rammohan Manamasa and Radha Rani Manamasa, fam- ily and friends for all the love and support who stood by me through these tough times during covid-19 pandemic situation. ii Contents Abstract i Acknowledgments ii 1 Introduction 3 1.1 Problem statement ............................ 3 1.2Aim.................................... 4 1.2.1 Objectives ............................. 4 1.3 Research questions ............................ 4 1.4 Contribution ................................ 5 1.5 Ethical, Societal and Sustainable aspects ................ 6 1.6 Outline ................................... 6 2 Background 7 2.1 Automatic quality inspection ....................... 7 2.1.1 Quality inspection . ..................... 7 2.1.2 Automatic quality inspection of assembly ............ 7 2.2 Machine learning and Deep learning ................... 8 2.2.1 Transfer learning ......................... 9 2.3 Artificial Neural Network ......................... 9 2.3.1 Activation function ........................ 10 2.3.2 Loss function ........................... 11 2.3.3 Back propagation . ...................... 11 2.4 Convolutional Neural Network ...................... 11 2.4.1 Convolutional layer ........................ 12 2.4.2 Pooling layer ........................... 12 2.4.3 Fully-connected layer ....................... 12 2.5 Domain adaptation ............................ 12 2.5.1 Unsupervised domain adaptation ................ 14 2.6 Point cloud data ............................. 15 3 Related Work 16 3.1 Literature review on previous studies .................. 16 3.2Drawbacks................................. 17 iii 4 Deep learning models 18 4.1PointDAN................................. 18 4.1.1 Architecture ............................ 19 4.2 Self-supervised domain adaptation network ............... 20 4.2.1 Architecture ............................ 21 4.3 Pointnet++ ................................ 21 4.3.1 Architecture ............................ 22 4.4Pointnet.................................. 22 4.4.1 Architecture ............................ 23 5 Methods 24 5.1 Data preparation ............................. 24 5.1.1 Source data ............................ 25 5.1.2 Target data ............................ 25 5.2 Data .................................... 26 5.3 Dataset labelling ............................. 26 5.4 Data cleaning ............................... 26 5.5 Model selection through literature review ................ 27 5.5.1 Search terminology and Search strings ............. 27 5.6 Methodology and Experimental design ................. 29 5.6.1 Experimentation on the deep learning models ......... 29 5.6.2 Evaluation ............................. 31 6 Understanding the data 32 6.1 WRL to point cloud conversion ..................... 32 6.2 Single view point cloud .......................... 33 6.3 Generation of data from camera ..................... 37 6.4 Other methods .............................. 39 7 Results 40 7.1 Results ................................... 40 7.1.1 Model 1-PointDAN ....................... 40 7.1.2 Model 2-Self-supervised network ................. 42 7.1.3 Model 3-Pointnet++ ....................... 43 7.1.4 Model 4-Pointnet ......................... 45 8 Analysis and Discussions 48 8.1PointDAN................................. 48 8.2 Self-supervised network .......................... 49 8.3 Pointnet++ ................................ 49 8.4Pointnet.................................. 50 8.5 Limitations and Challenges ........................ 51 8.5.1 Validity threats .......................... 52 iv 9 Conclusions and Future work 54 9.1 Conclusion ................................. 54 9.2 Answering the research questions .................... 55 9.3 Recommended Future work ....................... 56 References 57 A UI for automatic quality inspection 62 v List of Figures 1.1 Domain Adaptation [38] ......................... 3 2.1 Simple neural network[8] ......................... 10 2.2 Different activation functions[44] ..................... 10 2.3 Convolutional Neural Network[29] ................... 12 2.4 Categories of domain adaptation[19] .................. 13 2.5 Deep domain adaptation[46] . ..................... 14 2.6 Unsupervised domain adaptation[9] ................... 15 2.7 Example point cloud image [14] ..................... 15 4.1 Feature alignment[37] ........................... 18 4.2 PointDAN Network [37] ......................... 19 4.3 Self-supervised domain adaptation network [1] ............. 21 4.4 pointnet++ network [36] ......................... 22 4.5 pointnet network [34] ........................... 23 6.1 MeshLab software ............................ 33 6.2 Example of perspective projection[49] ................. 34 6.3PedalCar................................. 35 6.4 Cropping the pedal car to project only the wheel with bounding box using functions and libraries in python. ................. 35 6.5 Cropped wheel .............................. 36 6.6 Single view point cloud wheel ...................... 36 6.7 TrispectorP1000 along with the robot .................. 37 6.8 Operator of the robot to scan the wheel ................ 38 6.9 CloudCompare Software ......................... 39 7.1 Overall accuracy vs epoch graph when the data is test with pointDAN network. .................................. 41 7.2 Loss vs epoch graph when the data is trained and tested on PointDAN 41 7.3 Class-wise accuracy for PointDAN model ................ 42 7.4 Overall accuracy vs epoch graph when the data is test with self- supervised network. ............................ 42 7.5 Loss vs epoch graph when the data is trained and tested on self- supervised network. ............................ 43 7.6 Class-wise accuracy on Self-supervised model. ............. 43 7.7 Overall accuracy vs epoch graph