Technical Report OBU-ECM-AFM-2018-02 Datamorphic Testing: A Methodology for Testing AI Applications Hong Zhu(*), Dongmei Liu(+), Ian Bayley(*), Rachel Harrison(*), and Fabio Cuzzolin(*) (*) School of Engineering, Computing and Mathematics, Oxford Brookes University, Oxford, UK Emails: {hzhu, ibayley, rachel.harrison, fabio.cuzzolin}@brookes.ac.uk (+) School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China E-mail: [email protected] 27 December 2018 Version 4.3 1 H. Zhu et al., Datamorphic Testing Technical Report OBU-ECM-AFM-2018-02 Abstract With the rapid growth of the applications of machine learning (ML) and other artificial intelligence (AI) techniques, adequate testing has become a necessity to ensure their quality. This paper identifies the characteristics of AI applications that distinguish them from traditional software, and analyses the main difficulties in applying existing testing methods. Based on this analysis, we propose a new method called datamorphic testing and illustrate the method with an example of testing face recognition applications. We also report an experiment with four real industrial application systems of face recognition to validate the proposed approach. Version 4.3 i H. Zhu et al., Datamorphic Testing Technical Report OBU-ECM-AFM-2018-02 Table of Contents I. Motivation ...................................................................................................................................................... 1 II. The Challenges ............................................................................................................................................. 1 A. Distinctive Features of AI Applications .................................................................................................................... 1 B. Applicability of Existing Software Testing Methods ................................................................................................ 3 III. Basic Concepts of Datamorphic Testing ..................................................................................................... 4 A. Datamorphism ............................................................................................................................................................ 4 B. Metamorphism ........................................................................................................................................................... 4 C. Seed Test Cases .......................................................................................................................................................... 6 IV. Testing Process and Strategies .................................................................................................................... 6 A. Process of Datamorphic Testing ................................................................................................................................ 6 B. Datamorphic Testing Strategies ................................................................................................................................. 7 V. Experiment .................................................................................................................................................... 8 A. Goal of the experiment .............................................................................................................................................. 8 B. Design of the experiment ........................................................................................................................................... 8 C. Execution of The Experiment .................................................................................................................................... 9 D. Analysis of The Results ............................................................................................................................................. 9 VI. Conclusion ................................................................................................................................................. 10 A. Related work ............................................................................................................................................................ 10 B. Further work ............................................................................................................................................................. 12 Acknowledgement .............................................................................................................................................. 12 References ........................................................................................................................................................... 12 Appendix A. Results of Testing on Mutant Test Cases with Datamorphisms ................................................... 13 1. Results of Testing Tencent Face Recognition with Datamorphisms ....................................................................... 13 2. Results of Testing Baidu Face Recognition with Datamorphisms ........................................................................... 16 3. Results of Testing Face++ with Datamorphisms ..................................................................................................... 19 4. Results of Testing SeetaFace with Datamorphisms ................................................................................................. 22 Appendix B. Results of Testing on Real Images ............................................................................................... 25 1. Results of Testing Tencent Face Recognition on Real Images ................................................................................ 25 2. Results of Testing Baidu Face Recognition on Real Images ................................................................................... 28 3. Results of Testing Face++ on Real Images .............................................................................................................. 30 4. Results of Testing SeetaFace on Real Images .......................................................................................................... 33 Appendix C. Summary and Comparisons of Test Results ................................................................................. 36 1. Summary of The Results of Testing on Mutant Test Cases ..................................................................................... 36 2. Comparison of Test Results between Using and Without Datamorphisms ............................................................. 36 Version 4.3 ii H. Zhu et al., Datamorphic Testing Technical Report OBU-ECM-AFM-2018-02 Datamorphic Testing: A Methodology for Testing AI Applications Hong Zhu(*), Dongmei Liu(+), Ian Bayley(*), Rachel Harrison(*), and Fabio Cuzzolin(*) (*) School of Engineering, Computing and Mathematics, Oxford Brookes University, Oxford, UK Emails: {hzhu, ibayley, rachel.harrison, fabio.cuzzolin}@brookes.ac.uk (+) School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China E-mail: [email protected]. Abstract. With the rapid growth of the applications of development, which is widely used in industry [5]. machine learning (ML) and other artificial intelligence However, the current practice of testing AI (AI) techniques, adequate testing has become a necessity applications lags far behind the maturity of testing to ensure their quality. This paper identifies the traditional software applications. Testing has become a characteristics of AI applications that distinguish them grave challenge because the distinctive features of AI from traditional software, and analyses the main applications disqualify existing software testing methods, difficulties in applying existing testing methods. Based on techniques and tools. this analysis, we propose a new method called This paper examines the difficulties when testing AI datamorphic testing and illustrate the method with an applications using existing software testing methods and example of testing face recognition applications. We also techniques, and proposes a novel approach called report an experiment with four real industrial application datamorphic testing. systems of face recognition to validate the proposed The paper is organized as follows. Section II discusses approach. the challenges of AI applications currently confronting software test engineers. Section III introduces the basic I. MOTIVATION concepts of datamorphic testing method. Section IV We have seen a rapid growth in the application of machine discusses the process of datamorphic testing and various learning (ML), data mining and other artificial intelligence testing strategies of datamorphic testing. Section V reports (AI) techniques to software systems in recent years [1]. an experiment with four real industry ML applications of Typical examples of such applications include driverless face recognition. Section VI concludes the paper with a vehicles, face recognition and fingerprint recognition in discussion of related work and future work. security control, workload pattern learning in computer cluster operation, personalization of social media II. THE CHALLENGES networking for business intelligence, situation recognition In this section, we first identify the distinctive features of and action rule learning in healthcare, smart homes and AI and ML applications.