PONTIFICAL CATHOLIC UNIVERSITY OF RIO GRANDE DO SUL FACULTY OF INFORMATICS GRADUATE PROGRAM IN COMPUTER SCIENCE PLETS - A PRODUCT LINE OF MODEL-BASED TESTING TOOLS ELDER DE MACEDO RODRIGUES Thesis presented as partial requirement for obtaining the degree of Ph. D. in Computer Science at Pontifical Catholic University of Rio Grande do Sul. Advisor: Prof. Avelino Francisco Zorzo Porto Alegre 2013 Dados Internacionais de Catalogação na Publicação (CIP) R696P Rodrigues, Elder de Macedo PLETS – A product line of model-based testing tools / Elder de Macedo Rodrigues. - Porto Alegre, 2013. 130 p. Tese (Doutorado) – Fac. de Informática, PUCRS. Orientador: Prof. Dr. Avelino Francisco Zorzo. 1. Informática. 2. Engenharia de Software. 3. Software – Análise de Desempenho. 4. Dados de Teste (Informática). I. Zorzo, Avelino Francisco. II. Título. CDD 005.1 Ficha Catalográfica elaborada pelo Setor de Tratamento da Informação da BC-PUCRS To my family, colleges and friends, who offered me unconditional support during the course of this thesis. “Passam às mãos da minha geração Heranças feitas de fortunas rotas Campos desertos que não geram pão Onde a ganância anda de rédeas soltas” (Engenheiros do Hawaii) ACKNOWLEDGMENTS I would like to thank my parents for their endless love and support. I would like a special thank to my wife Ildevana and to my daughter Ana Carolina for their love, kindness and support during the past four years - actually in the last 10 and 8 years. I love you girls!! A special thank to Prof. Avelino Zorzo for believe in my work and for being my supervisor and mentor in the last 6 years. Thank you AZ! A thank to Prof. Itana Gimenez, Prof. Adenilso da Silva Simão and Prof. Rafael Prikladnicki for being part of my committee and for being available to discuss and contribute with this thesis. I would like to thank all my colleagues and friends at Cepes: Prof. Flávio Oliveira, Leandro Costa, Maicon Bernardino, Artur Freitas, Priscila Guarnieri, Anderson Domingues, Tiago Chagas, Ana Luiza Cunha e Murilo Arantes. Thank you guys for the collaboration, discussion, support and barbecues. I would like to thank Prof. Krzysztof Czarnecki for give me the opportunity to conducted my research at the Generative Software Development lab at University of Waterloo. Moreover, a special thank to all my "canadian" friends: Stephen Liu, Leonardo Passos, Aline Ferreira, Leopoldo Teixeira, Zubair Aktar, Arrah Leonardo, Pedro and Aline Drimel. I also acknowledge, again, Prof. Itana Gimenez for her valuable feedback and for give me the opportunity to spend a whole month in her research lab at UEM. Furthermore, I would like to thank Prof. Edson Junior for his support during my stay in Maringá and his valuable contributions to this work. Last but not least, I would like to thank Capes, Dell Inc., INCT-Sec and Procad. PLETS - A PRODUCT LINE OF MODEL-BASED TESTING TOOLS RESUMO O teste de software é uma atividade fundamental para garantir a qualidade de software. Além disso, teste de software é uma das atividades mais caras e demoradas no processo de desenvolvimento de software. Por esta razão, diversas ferramentas de teste foram desenvolvidas para apoiar esta atividade, incluindo ferramentas para Teste Baseado em Modelos (TBM). TBM é uma técnica de teste para automatizar a geração de artefatos de teste a partir de modelos do sistema. Esta técnica apresenta diversas vantagens, tais como, menor custo e esforço para gerar casos de teste. Por este motivo, nos últimos anos, diversas ferramentas para TBM foram desenvolvidas para melhor explorar essas vantagens. Embora existam diversas ferramentas TBM, a maioria delas tem o seu desenvolvimento baseado em um esforço individual, sem a adoção de técnicas de reuso sistemático e com base em uma única arquitetura, dificultando a integração, evolução, manutenção e reutilização dessas ferramentas. Uma alternativa para mitigar estes problemas é adotar os conceitos de Linhas de Produto de Software (LPS) para desenvolver ferramentas de TBM. LPS possibilitam gerar sistematicamente produtos a custos mais baixos, em menor tempo e com maior qualidade. A principal contribuição desta tese de doutorado é apresentar uma LPS de ferramentas de teste que suportam TBM (PLeTs) e um ambiente automatizado para apoiar a geração dessas ferramentas (PlugSPL). Além disso, esta tese apresenta uma abordagem para gerar ferramentas para TBM, que foram aplicadas em dois exemplos de uso. Com base nos resultados obtidos nos exemplos de uso, podemos inferir que LPS pode ser considerada uma abordagem relevante para melhorar a produtividade e o reuso durante a geração de ferramentas de TBM. Além disso, também foi realizado um estudo experimental com o objetivo de avaliar o esforço para se utilizar uma ferramenta derivada da PLeTs para geração de scripts de teste. Os resultados apontaram que o esforço para gerar scripts de teste foi reduzido consideravelmente, quando comparado com a uma ferramenta de Capture and Replay. Palavras Chave: Linha de Produto de Software, Teste de Software, Teste Baseado em Modelos. PLETS - A PRODUCT LINE OF MODEL-BASED TESTING TOOLS ABSTRACT Software testing is recognized as a fundamental activity for assuring software quality. Furthermore, testing is also recognized as one of the most time consuming and expensive activities of software development process. A diversity of testing tools has been developed to support this activity, including tools for Model-based Testing (MBT). MBT is a testing technique to automate the generation of testing artifacts from the system model. This technique presents several advantages, such as, lower cost and less effort to generate test cases. Therefore, in the last years a diversity of commercial, academic, and open source tools to support MBT has been developed to better explore these advantages. In spite of the diversity of tools to support MBT, most of them have been individually and independently developed from scratch based on a single architecture. Thus, they face difficulties of integration, evolution, maintenance, and reuse. In another perspective, Software Product Lines (SPL) offers possibility of systematically generating software products at lower costs, in shorter time, and with higher quality. The main contribution of this Ph.D thesis is to present a SPL for testing tools that support MBT (PLeTs) and an automated environment to support the generation of these tools (PlugSPL). Furthermore, our strategy was initially applied to generate some MBT testing tools which were applied in two examples of use performed in collaboration of an IT company. Based on the feedback from the examples of use we can infer that SPL can be considered a relevant approach to improve productivity and reuse during generation of MBT testing tools. Moreover, we also performed an experimental study carried out to evaluate the effort to use an MBT tool derived from our SPL to generate test scripts and scenarios. Thus, the results point out that the effort to generate test scripts, when compared with a Capture and Replay based tool, was reduced considerably. Keywords: Software Product line, Software Testing, Model-based Testing. LIST OF FIGURES 2.1 Error Propagation (adapted from [ALRL04]) ............... ............... 22 2.2 LevelsofTesting[UL06]............................. ................ 26 2.3 MBTMainActivities[EFW01] ........................... ............. 29 2.4 Software Product Line Engineering Life-cycles [LSR07] . .................. 33 2.5 Example of a Car Feature Model - Slightly Adapted from [KCH+90] ............ 37 2.6 Realising Variability at Architecture [LSR07] . ................... 38 3.1 Systematic Mapping Studies Process (adapted from [PFMM08]) . .............. 42 3.2 Bubble Plot of the Domain Studies Distribution by Publication Year and Testing Level 51 3.3 Venn Diagram of the Test Models Categorization . ................. 52 4.1 MBTToolsProcess .................................. .............. 61 4.2 PLeTsFeatureModel................................ ............... 62 4.3 PLeTs UML Component Diagram with SMarty ................ ............ 65 4.4 PlugSPLModules .................................... ............. 70 4.5 PlugSPLFMEditor ................................... ............. 71 4.6 Component Management Module ........................ .............. 72 4.7 CodeBeforetheReplacement......................... ................ 73 4.8 CodeAftertheReplacement........................... ............... 73 5.1 An Approach for Generating Performance Scripts and Scenarios ................ 76 5.2 PLeTsPerfLR Tool Architecture ....................... ................ 79 5.3 PLeTsPerfVS Tool Architecture ........................ ............... 80 5.4 TPC-WUseCaseDiagram.............................. ............. 80 5.5 TPC-W Activity Diagram - Shop Use Case .................. ............. 81 5.6 Abstract Test Scenario of the Actor Customer ............................ 82 5.7 Example of Abstract Test Case Generated from the Shop UseCase............. 83 5.8 Example of Data File Containing Some Parameters . ................ 83 5.9 XML of Test Scenario Generated for the Visual Studio (*.LoadTest) ............ 84 5.10 Test Script Generated for the Visual Studio . ................. 84 5.11 Test Script Generated for the LoadRunner . .................. 84 5.12 An Approach for Generating Structural Test Cases . ................ 86 5.13 PLeTsStructJabuti Tool Architecture . .................. 89 5.14 PLeTsStructEmma Tool Architecture ................... ................ 89 5.15 SequenceDiagram .................................