• Abstract and Figures
• Public Full-text

MASTER’S THESIS | LUND UNIVERSITY 2013 Evaluating Open-Source and Free Software Tools for Test Management Albin Rosberg Department of Computer Science Faculty of Engineering LTH ISSN 1650-2884 LU-CS-EX 2013-37 Evaluating Open-Source and Free Software Tools for Test Management Albin Rosberg [email protected] Sunday 29th September, 2013 Master's thesis work carried out at System Verication Sweden AB for Lund University, Computer Science. Supervisor: Magnus C. Ohlsson, [email protected] Examiner: Per Runeson, [email protected] Abstract There are currently a very large amount of Test Management tools available, many of which are free to use. The aim of these tools is to support decision making and management for test management. To nd which tools is the best, they need to be evaluated and compared to each other. There is also a question whether or not free software or open source tools are as good, or better, than proprietary alternatives. The aim of the master thesis project is to select tools and compare free tools to a baseline tool using evaluation criteria and documentation of scenarios dened by the author. The combination of these was the basis for a case study conducted on the matter and then followed up by hosting a demonstration event for Test Managers, showcasing a selection ot Test Management tools. The participants in the demo event clearly put Microsoft Test Man- ager (MTM) as the best option for a Test Management tool - an idea which was supported in the case study. The tool does not integrate well with non-Microsoft software which can be a problem for some organisa- tions which does not evolve around Microsoft products. This means such organisations might have a harder time integrating the MTM and should therefore look for an open source tool such as TestLink, which holds a lot more options for the additional software required. Keywords: Test Management, Tool Evaluation 2 Acknowledgements I would like to thank the consultants at System Verication for lending me their time and knowledge and giving me a taste of how testing is run in a modern organisation, special thanks to Magnus for creating this masters thesis proposal and making it happen. I would also like to thank Per and Magnus for the feedback of my work - and to my family for all the support. 3 4 Contents 1 Introduction 7 1.1 Purpose . .7 1.2 Outline . .8 2 Background 9 2.1 Test Management . .9 2.2 Software Licence Models . 10 2.3 Related work . 10 3 Method 13 3.1 Questionnaire . 13 3.2 Case Study Protocol . 14 3.2.1 Context . 15 3.2.2 Hypothesis . 15 3.2.3 Scenarios . 15 3.3 Demo event . 17 4 Tool Selection 19 4.1 Candidate Tool Selection . 19 4.2 Microsoft Test Manager . 20 4.3 XStudio . 20 4.4 TestLink . 21 4.5 Testopia . 21 5 Results 23 5.1 Scenario Results . 23 5.2 Case Study . 24 5.3 Demo event results . 25 5.4 Validity . 26 5 CONTENTS 6 Discussion 29 6.1 Participation . 29 6.2 Maintainability . 29 6.3 Installability . 30 6.4 Functionality versus interoperability . 30 6.5 Usability . 31 7 Conclusions 35 Appendix A Questionnaire 43 Appendix B Test management tool setup 45 Appendix C Demo scenarios 47 6 Chapter 1 Introduction Test management is the process of managing and organising test within an organ- isation, used by members of the test organisation and supervised by one or more test managers. The purpose of a test management organisation is, among other things, to plan testing eorts, provide product and process quality information and maintain a repository of test-related information [5]. The larger an organisation, the more coordination and management is required which means that at one point the workload becomes too great to handle manually. This is where test management tools come in, to help organise testing and issues related to this. 1.1 Purpose The purpose of the thesis project is to compare dierent Test Management tools in order to get an overview and hopefully give an answer to which is superior. Test Management tools are multi-function tools with the purpose to give an overview over the test situation of a project. The functionalities of a test management tool can vary from simple bug handling to include conguration management, scheduling and requirements tracking. In order to determine which tool is superior there is need for a set of evaluation criteria on which the dierent tools are tested. There are a lot of dierent tools for Test Management and even though a majority of them are commercial tools, plenty are open source and free software alternatives. This report will focus on the free-to-use tools and compare these to a baseline com- mercial tool. In order to get an idea of which tools are used and what functionality is sought-after in a Test Management tool a case study is performed. The participants in the case study use tools in their day-to-day work and have experience in one or more Test Management tools. The objective is to study the Test Management tools, using test managers to collect the data needed. Furthermore the participants were invited to a demonstration of the chosen tools to give feedback on the functionalities 7 1. Introduction and usability of the tools. Not only will functional aspects of a tool be regarded, but also quality aspects like usability, maintainability and installability. Examples of usability dierences in tools are how the data extracted from a tool is presented to a user and which information is highlighted in dierent ways. To be able to display the proper information, as a user, is a valuable tool in itself [9] and can aid in the decision making for e.g. release of software in a project. The non-functional aspects of Test Management tools are regarded to give a broader view and not only focus on functionality, but also on other aspects which may be as important when determining which tool to use e.g. how easy it is to install and maintain the tool. 1.2 Outline In chapter 2 a general background of Test Management in general as well as the dierent types of Software Licence Models and their signicance. Chapter 3 de- scribes the research methods used for the thesis project; including the outline of the interviews, case study, scenarios and demonstration event. Chapter 4 includes how the tools were selected as well as further descriptions of the selected tools. In chapter 5 the results from the dierent elements in the thesis project are presented alongside threats and limitations of the case study. Chapter 6 draws conclusions from observations as well as results from the case study. Finally in chapter 7 a short summary of the results is presented. 8 Chapter 2 Background The master thesis project is divided into two parts; 1) establishing and executing an evaluation framework, and 2) presenting the results analysing pros and cons of the evaluated tools. The main focus of the evaluation framework are the functional aspects, though some aspects of ISO 9126 [28] are also considered. The aspects usability, maintainability and portability (mainly installability within portability), will be considered in order to make sure a tool does not require an excessive amount of work for e.g. installation and maintenance. 2.1 Test Management The term Test Management is vague and has no clear denition. International Software Testing Qualications Boards (ISTQB) denes the test manager's role as performing risk management . reviews, assessments, quality gate evaluation, quality assurance and audits [16]. The work description of a test manager may vary depending on which test methodology is followed, however the hierarchical status of a test manager in an organisation is very much the same. In order to structure the work of test managers within an organisation dierent methodologies and models may be applied in order to structure the management process [27]. The model chosen will have an impact on the testing resources since the test manager manages the testing resources. To assist in the tasks linked to Test Management, several tools are available to assist the test manager in planning, analysing etc [26]. Dierent tool developers have dierent ideas of what is needed for Test Management and thus the functionality available in dierent tools vary greatly. The tools compared in the study are tools most mentioned by the interviewees or similar tools which have similar functionality as the ones emerged from the interviews. A test manager is generally the person within a test organisation responsible 9 2. Background for stang, connections with the management of other departments and being the central person around all testing and quality issues [5]. Typical work tasks as stated by Burnstein may include test policy making, . test planning, test documentation, controlling and monitoring of tests, . reviewing test work. 2.2 Software Licence Models There are at the moment of writing a vast amount of tools designed to be used for test management purposes and they can generally be divided into two categories: • Tools with licenses which cost money to use. • Tools with licenses which do not. The categorisation is very general with the rst category including licenses for which any user must pay a fee to use the software, periodically or a one time fee. The proprietary license has no xed rules set and is rather mentioned as a single group due to the fact that it is not free to use. This Master's Thesis' focus lie primarily on the second category which can be divided into two sub-categories; open source and freeware.

Load more

  54

Copy Link