ADVANCED ALGORITHMS FOR COMBINATORIAL AND SEQUENTIAL TEST GENERATION WITH CONSTRAINTS by FENG DUAN DISSERTATION Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at The University of Texas at Arlington May, 2021 Arlington, Texas Supervising Committee: Yu Lei, Supervising Professor Christoph Csallner Hao Che Jiang Ming ABSTRACT Advanced Algorithms for Combinatorial and Sequential Test Generation with Constraints Feng Duan, Ph.D. The University of Texas at Arlington, 2021 Supervising Professor: Jeff (Yu) Lei Combinatorial and sequential testing are software testing strategies that have attracted significant interests from both academic and industrial communities. This dissertation addresses the problem of how to efficiently generate tests for both combinatorial and sequential testing. The dissertation makes two major contributions. For combinatorial testing, we present several optimizations on an existing t-way test generation algorithm called IPOG. These optimizations are designed to reduce the number of tests generated by IPOG. For sequential testing, we develop a notion for expressing commonly used sequencing constraints and present a t-way test sequence generation algorithm that support constraints expressed using notation. We demonstrate the effectiveness of our notation and test generation algorithm using a real-life protocol that exhibits sequencing behavior. This dissertation is presented in an article-based format, including three published research papers and one manuscript that is currently under review. The first two papers are about ii combinatorial test generation. The other two papers are about sequential test generation. The first paper reports our work on reducing the number of tests generated during the vertical growth phase of the IPOG algorithm, where the vertical growth problem is modeled as the classical “Minimum Vertex Coloring” problem. The second paper reports our work on extending the approach in the first paper to support constraints, where constraints are represented as edges and hyperedges in a graph structure. The third paper reports our work on addressing the problem of constraint handling in sequential testing, where we design a notation for sequencing constraint specification and develop a new algorithm to handle constraints expressed using our notation. The fourth paper reports our latest work on sequential testing, where we translate constraints expressed using our notation to Deterministic Finite Automaton (DFA) and use DFA to check sequence validity and extensibility. iii Copyright by Feng Duan 2021 ACKNOWLEDGEMENTS I would like to express the deepest appreciation to my supervising professor Dr. Jeff (Yu) Lei, whose expertise was invaluable in formulating the research questions and methodology. Without his supervision and constant help this dissertation would not have been possible. I would like to thank my committee members, Dr. Christoph Csallner, Dr. Hao Che and Dr. Jiang Ming for generously sharing their time and ideas. In addition, I would like to appreciate my parents for their constant love and patience. I am grateful to my younger brother for his support. Finally, I wish to give my heartfelt thanks to my wife whose unconditional love and continual support enabled me to complete this dissertation. v TABLE OF CONTENTS ABSTRACT .................................................................................................................................... ii ACKNOWLEDGEMENTS ............................................................................................................ v TABLE OF CONTENTS ............................................................................................................... vi LIST OF FIGURES ....................................................................................................................... xi LIST OF TABLES ....................................................................................................................... xiv CHAPTER 1. INTRODUCTION ................................................................................................... 1 1.1 Research overview ..................................................................................................... 1 1.2 Summary of publications ........................................................................................... 7 CHAPTER 2. Improving IPOG's Vertical Growth Based on a Graph Coloring Scheme ............. 12 2.1 Introduction .............................................................................................................. 12 2.2 The IPOG Algorithm................................................................................................ 14 2.3 Motivating Examples ............................................................................................... 17 2.3.1 Example for 3-way test generation .................................................................. 18 2.3.2 Example for 2-way test generation .................................................................. 20 2.4 Approach for IPOG Improvement ........................................................................... 21 2.4.1 Conflict graph with existing tests .................................................................... 21 2.4.1.1 Graph model with type 2 existing tests .................................................... 21 2.4.1.2 Graph model with type 3 existing tests .................................................... 23 2.4.1.3 Graph model with type 2 and 3 existing tests .......................................... 25 vi 2.4.2 Implementation ................................................................................................ 27 2.5 Experiment ............................................................................................................... 29 2.5.1 Experiment design ........................................................................................... 30 2.5.2 Results and analysis ......................................................................................... 31 2.6 Related Work ............................................................................................................ 33 2.7 Conclusion and Future Work ................................................................................... 34 2.8 Acknowledgment ..................................................................................................... 35 2.9 References ................................................................................................................ 35 CHAPTER 3. Optimizing IPOG's Vertical Growth with Constraints Based on Hypergraph Coloring ........................................................................................................................................ 39 3.1 Introduction .............................................................................................................. 39 3.2 Vertical Growth vs Greedy Coloring ....................................................................... 43 3.3 Motivating Examples ............................................................................................... 45 3.3.1 Example for 3-way test generation .................................................................. 45 3.3.2 Example for 2-way test generation .................................................................. 49 3.4 Hypergraph Coloring-based optimization ................................................................ 50 3.4.1 Reduction to Hypergraph Coloring ................................................................. 50 3.4.2 Compute order for greedy coloring ................................................................. 56 3.5 Experiment ............................................................................................................... 57 3.5.1 Experiment design ........................................................................................... 57 vii 3.5.2 Results and analysis ......................................................................................... 59 3.6 Related Work ............................................................................................................ 61 3.7 Conclusion and Future Work ................................................................................... 63 3.8 Acknowledgment ..................................................................................................... 64 3.9 References ................................................................................................................ 65 CHAPTER 4. An Approach to T-way Test Sequence Generation With Constraints .................... 68 4.1 Introduction .............................................................................................................. 68 4.2 Motivating Example................................................................................................. 72 4.2.1 Model Sequencing Behavior ........................................................................... 72 4.2.2 Test Sequence Generation ............................................................................... 74 4.3 Notation for Constraint Specification ...................................................................... 76 4.3.1 Syntax of Sequencing Constraints ................................................................... 76 4.3.2 Semantics of Sequencing Operators ................................................................ 78 4.3.2.1 Always sequencing operator