Test Process Improvement & Test/Build Tool Evaluation
Total Page：16
File Type：pdf, Size：1020Kb
Test Process Improvement & Test/Build Tool Evaluation Master thesis 30hp, Advanced level Students: Jesper Söderlund [email protected] Thomas Sörensen [email protected] Supervisor: Markus Lindgren [email protected] Examiner: Daniel Sundmark [email protected] School of Innovation, Design and Engineering P.O. Box 883, SE-721 23 Västerås. Tel: +46 21 10 31 60. E-mail: [email protected] Web: www.mdh.se/idt 2 Sammanfattning De produkter som företaget tillverkar används i huvudsak inom ett område av branschen där fel som leder till stopp i produktionen kan vara ganska dyrt. Detta gör testning av produkterna viktiga och tester kan också ge indikationer om kvaliteten på produkterna. Företaget är i en fas där man utvecklar en ny produktlinje som ska stödja alla befintliga och framtida produkter. I denna fas har man beslutat att alla produkterna ska använda ett gemensant ramverk för enhetstestning och även använda ett gemensamt byggsystem för samtliga produkter. En del av examensarbetet var att undersöka och utvärdera olika ramverk för enhets testning och verktyg för byggsystem. De ramverk som utvärderades var CppUnit, cfix, NUnit, Boost test library, unitTest++ och CxxTest. Utvärderingen ledde fram till att CppUnit rekommenderades till företaget. Verktyg som utvärderades för byggsystem var MSBuild, NAnt, Automated Build Studio och Cruise Control .Net. För byggsystem rekommenderas MSBuild i kombination med Cruise Control .Net ifall företaget är intresserade av den extra funktionalitet som Cruise Control .Net har att erbjuda. Företaget har även ett intresse av att utvärdera den nuvarande testprocessen och identifiera förbättringar som ett led i att befintliga produkter skall följa en gemensam testprocess. För att kunna identifiera dessa förbättringar utfördes en litteraturstudie över fyra stycken test process förbättrings ramverk (Test Process Improvent, Test Maturity Model integrated, Minimal Test Practice Framework och Test Improvement Model). Utav dessa fyra ramverk så valdes Test Process Improvement (TPI) ut som en hjälp för att identifiera förbättringar. Med hjälp av TPI utfördes en begränsad bedömning av företagets mogenhetsgrad på tre produkter, där två av produkterna har en låg mogenhetsgrad. Resultat av förbättringsåtgärderna kan sammanfattas med att man bör harmonisera dokument, standardisera och dokumentera olika processer. Som en sista del i examensarbetet så utvärderades möjligheten att automatisera testning på två av produkternas grafiska användargränssnitt med programmet TestComplete. För en av produkterna blev resultatet att det fungerade tillfredställande och för den andra produkten så fungerade det inte alls. Resultatet blev rekommendationer för hur företaget borde gå vidare med automatisering av testnig på det grafiska användargränssnittet. 3 Abstract The products The Company manufactures are used in an area of the industry where errors leading to a stop in production can be quite expensive. Therefore are testing of the products important and the tests can also give indications about the quality of the products. The Company is in a phase where they are developing a new product line to support all existing and future products. In this phase, it was decided that all products will use a common framework for unit testing and a common build system for all products. One part of the thesis was to investigate and evaluate different frameworks for unit testing and tools for a build system. The unit test framework that were evaluated are Cppunit, cfix, NUnit, Boost test library, Unit Testing++ and CxxTest. The result of the evaluation was that CppUnit were recommended. For the build system MSBuild, NAnt, Automated Build Studio and Cruise Control .NET were evaluated. The recommended tools for a build system is MSBuild in combination with Cruise Control .Net if The Company is interested in the functionality Cruise Control .Net has to offer. The Company also has an interest in evaluating the current test processes and identify improvements as a part of The Company’ s objective were all products should follow a common test process. In order to identify these improvements a literature study of four test process improvement frameworks (Test Process Improvement, Test Maturity Model Integrated, Minimal Test Practice Framework and Test Improvement Model) were carried out. Out of these four frameworks Test Process Improvement (TPI) were chosen to assist when identifying improvements. With the help of TPI a limited assessment took place to give indications about the test maturity for three of The Company’s products where two of the products had low maturity. Results of the improvement measures can be summed up with the need to harmonize the documents, standardize and document the various processes. As a last part of the thesis the possibility to automate testing of two of the products graphical user interfaces with the program Test Complete were investigated. For one of the products the result was that it worked satisfactorily and for the other product it did not work at all. This resulted in recommendations for how The Company should proceed with automated testing of the graphical user interface. 4 1 Definitions and acronyms ....................................................................................... 6 2 Introduction .............................................................................................................. 7 3 Testing ...................................................................................................................... 8 3.1 Testing Methods ........................................................................................... 8 3.2 Testing levels ............................................................................................... 8 3.3 Code Coverage ............................................................................................ 8 4 Unit testing ............................................................................................................. 10 4.1 Introduction ................................................................................................ 10 4.2 Literature survey ........................................................................................ 10 4.3 Interviews ................................................................................................... 11 4.4 Evaluation of unit test Frameworks ............................................................. 11 4.4.1 CxxTest ........................................................................................ 13 4.4.2 Boost test library ........................................................................... 13 4.4.3 Cfix ............................................................................................... 13 4.4.4 CppUnit ........................................................................................ 13 4.4.5 Unittest++ ..................................................................................... 13 4.4.6 NUnit ............................................................................................ 14 4.5 Summary of Evaluation of Unit Test Frameworks ....................................... 14 4.6 Results ....................................................................................................... 14 5 Build systems ........................................................................................................ 16 5.1 Introduction ................................................................................................ 16 5.2 Literature survey ........................................................................................ 16 5.3 Evaluation of Build Tools ............................................................................ 17 5.3.1 MSBuild ........................................................................................ 18 5.3.2 NAnt ............................................................................................. 18 5.3.3 Automated Build Studio ................................................................ 19 5.3.4 Cruise Control .Net ....................................................................... 19 5.4 Result ......................................................................................................... 19 6 Test improvement .................................................................................................. 20 6.1 Introduction ................................................................................................ 20 6.2 Literature survey ........................................................................................ 20 6.2.1 TPI (Test Process Improvement) .................................................. 20 6.2.2 TMMI (Test Maturity Model Integrated) ......................................... 21 6.2.3 MTPF (Minimal Test Practice Framework) .................................... 22 6.2.4 TIM (Test Improvement Model) ..................................................... 22 6.3 Choice of Framework ................................................................................. 22 6.4 TPI Assessment ......................................................................................... 22 6.5 TPI Assessment summary .......................................................................... 30 6.6 Improvement suggestions .......................................................................... 30 6.7 Results ......................................................................................................