Software Quality Characteristics Tested for Mobile Application Development

Home , Apex (tournament), C More Entertainment, Driver (series), Groove Coaster, Halfbrick Studios, Hottrix

Thesis no: MGSE-2015-02

Software Quality Characteristics Tested For Mobile Application Development

Literature Review and Empirical Survey

WALEED ANWAR

Faculty of Computing Blekinge Institute of Technology SE-371 79 Karlskrona Sweden This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Software Engineering. The thesis is equivalent to 10 weeks of full time studies.

Contact Information: Author(s): WALEED ANWAR E-mail: [email protected]

University advisor: Dr. Simon Poulding Department of Software Engineering

Faculty of Computing Internet : www.bth.se Blekinge Institute of Technology Phone : +46 455 38 50 00 SE-371 79 Karlskrona, Sweden Fax : +46 455 38 50 57

ABSTRACT

Context. Smart phones use is increasing day by day as there is large number of app users. Due to more use of apps, the testing of mobile application should be done correctly and flawlessly to ensure the effectiveness of mobile applications. Objectives. The objective of this research is to find out the important mobile application quality characteristics from developer’s perspective and how developers actually test for them. Apart from that how the developers test their mobile applications are also addressed. Methods. Two methodologies were used: the literature survey and the empirical survey. The literature survey was used to get familiar with the most commonly known mobile application quality characteristics for which mobile applications are tested for. The empirical survey was used to get data from developers by sending an online questionnaire link to the Google Play store developers and their response was recorded and further evaluated to present results. I categorized apps as top rated apps and low rated apps. Results. I got 43 responses as a whole from 1000 developers. I got 20 responses out of 500 from the top rated apps developers and 23 responses out of 500 from the low rated apps developers. The results were used to compare the quality characteristics importance to testing. According to the responses from both top rated and low rated apps developer, functional testing and usability testing is considered to be the most important and are mostly tested for “Android” mobile applications. Load testing and energy testing is considered to be not that important from developer’s point of view but energy testing and memory testing is not so often tested for “Android” mobile applications by developers. When it comes to testing process, top rated and low rated apps developer perform testing by creating test cases that are predefined in scripts or documents. They prefer to execute their tests on emulator rather than on mobile and run their tests automatically via tool or service as a cloud.

Conclusion. My study reports the quality characteristics importance for which the mobile applications are tested for from developer’s perspective, actual testing done by developers and the testing process used by them. Functional testing and usability testing are considered to be most important and mostly tested quality characteristics according to developer’s perspective. Testing is done by running their tests on emulator automatically through some tool. They prefer to create their test cases that are predefined in scripts.

Keywords: Mobile Application Testing, Quality Characteristics

ACKNOWLEDGEMENT

I would like to extend my gratitude to my supervisor Simon Poulding for his vital support and feedback till end. This thesis would not have been possible without Simon Poulding guidance throughout the research. I really feel honored working with him.

I would like to thank my mom, siblings and friends who helped me and supported me. Finally, I sincerely appreciate all the participants of the web questionnaire from all over the world because without their response my thesis would not be accomplished.

TABLE OF CONTENTS

ABSTRACT ...... 3 ACKNOWLEDGEMENT ...... 4 TABLE OF CONTENTS ...... 5 LIST OF FIGURES ...... 7 LIST OF TABLES ...... 8 1 INTRODUCTION ...... 9 1.1 AIMS AND OBJECTIVES ...... 10 1.2 RELATED WORK ...... 10 2 METHODOLOGY: ...... 12 2.1 RESEARCH QUESTIONS ...... 12 2.2 LITERATURE REVIEW ...... 12 2.2.1 EXTENDED DESIGN AND DERIVING THE START SET OF PAPERS: ...... 13 2.3 EMPIRICAL SURVEY ...... 14 2.4 PREPARATION OF QUESTIONNAIRE ...... 15 2.5 RESEARCH DESIGN AND DATA COLLECTION: ...... 15 3 RESULTS ...... 17 3.1 TOP RATED APPS DEVELOPER’S FEEDBACK ...... 17 3.1.1 IMPORTANCE OF QUALITY CHARACTERISTICS ...... 17 3.1.2 TESTING OF QUALITY CHARACTERISTICS ...... 18 3.1.3 TESTING PROCESS OF TOP RATED APPLICATIONS ...... 20 3.2 LOW RATED APPS DEVELOPER’S FEEDBACK ...... 20 3.2.1 IMPORTANCE OF QUALITY CHARACTERISTICS ...... 21 3.2.2 TESTING OF QUALITY CHARACTERISTICS ...... 22 3.2.3 TESTING PROCESS OF LOW RATED APPLICATIONS ...... 23 4 ANALYSIS ...... 25 4.1 DATA ANALYSIS ...... 25 4.2 TOP RATED APPS ...... 25 4.2.1 IMPORTANCE BY TOP RATED DEVELOPERS ...... 25 4.2.2 TESTING BY TOP RATED DEVELOPERS ...... 27 4.2.3 TESTING PROCESS OF TOP RATED APPS ...... 28 4.3 LOW RATED APPS ...... 29 5

4.3.1 IMPORTANCE BY LOW RATED DEVELOPERS ...... 29 4.3.2 TESTING LEVEL BY LOW RATED DEVELOPERS ...... 30 4.3.3 TESTING PROCESS BY LOW RATED DEVELOPERS ...... 32 4.4 COMPARISON OF QUALITY CHARACTERISTICS IMPORTANCE, TESTING AND THEIR TESTING PROCESS ...... 33 5 DISCUSSION ...... 34 6 CONCLUSION ...... 36 6.1 VALIDITY ...... 36 6.2 FUTURE WORK...... 37 7 REFERENCES ...... 38 8 APPENDIX A: QUESTIONNAIRE...... 40 9 APPENDIX B: TOP RATED DEVELOPER’S INFORMATION: ...... 44 10 APPENDIX C: LOW RATED DEVELOPER’S INFORMATION: ...... 60

LIST OF FIGURES

Figure 1: Comparison of important quality characteristics from top rated developer’s perspective………………………………………………………………………………………..26 Figure 2: Comparison of testing level of quality characteristics from top rated developer’s perspective...... 28 Figure 3: Comparison of important quality characteristics from low rated developer’s perspective……………………………………………………………………………………… 30 Figure 4: Comparison of testing level of quality characteristics from low rated developer’s perspective...... 32

LIST OF TABLES

Table 1: Research Questions ...... 12 Table 2: Quality characteristics importance by top rated apps developers...... 17 Table 3: Quality characteristics testing by top rated apps developers...... 18 Table 4: Testing performed by top rated developers...... 20 Table 5: Quality characteristics importance by low rated apps developers...... 21 Table 6: Quality characteristics testing by low rated apps developers...... 22 Table 7: Testing performed by low rated apps developers...... 23 Table 8: Analysis for the importance of quality characteristics from top Rated apps developer’s perspective...... 25 Table 9: Analysis for the testing level of quality characteristics from top rated apps developer’s perspective...... 27 Table 10: Analysis for the importance of quality characteristics from low Rated apps developer’s perspective...... 29 Table 11: Analysis for the testing level of quality characteristics from low Rated apps developer’s perspective...... 31

1 INTRODUCTION

A mobile application (or mobile app) is a software application that is designed to run on smartphones, tablet computers and on other mobile devices [1]. Mobile applications are becoming popular and also very complex. As mobile application users expect the applications to be reliable and fault free, so it makes the duty for application testers to ensure the quality of mobile applications before it is released in market [2]. Testing is a technique used to identify the faults in applications and to satisfy their customers by providing them highly reliable and error free applications [27]. Testing of mobile application is extremely difficult because there is a lot of diversity in mobile devices, their runtime environment and the resources needed to test a mobile application differs from mobile phone to tablet [28]. According to Mobile Application Testing Tutorial published by the IEEE Computer Society [7], mobile application testing is different from the conventional software testing because of the following several unique requirements. 1. Mobile applications must be able to work correctly anytime and anywhere. 2. As mobile applications are often developed for a set of targeted devices, these applications must work properly on different platforms having different operating systems, screen sizes, processing power, network bandwidth and battery life. 3. Mobile applications must also include multiple input channels (e.g. keyboard, voice, and gestures), multimedia support, and other enhanced usability features in order to provide the rich experience as expected by the mobile device users. 4. As mobile devices support a range of wireless networks (e.g. 2G, 3G, 4G, Wi-Fi, WiMAX), mobile applications must also able to work in different network connectivity environment. Considering these unique requirements for mobile applications and to be fully aware of important mobile application quality characteristics, I was interested in using the research method literature survey and empirical survey. The literature survey was carried out to know about the well-known quality characteristics tested for mobile applications. The quality characteristics is defined as what characteristics should an application have for example application should function properly, application should be easy to use for users, application should be compatible with different devices etc. The empirical survey was performed to find out which quality characteristics are important from developer’s perspective and how often those quality characteristics are tested for “Android” mobile applications by developers. Apart from that I was also interested in performing empirical survey on the testing process used by developers for testing of “Android” mobile applications. In empirical surveys, information is gathered by asking people questions [8]. Afterwards, those quality characteristics were compared according to respondent’s answers which helped me in finding out which quality characteristics were important from their point of view and how they actually use them. Moreover, how the testing is performed was also figured out.

The empirical survey is the method used to collect data from developers. An online questionnaire was made through Google Form and sent to 1000 developers and their response was recorded and further evaluated to present results. The apps whose rating was 4 or above was categorized as top rated apps and the apps whose rating was below 4 came into the category of low rated apps. The developer’s information is given in Appendix B and Appendix C. I contacted 500 developers of top rated apps and 500 developers of low rated apps to balance my findings and it was also time 9

consuming to find developer’s information manually. The apps were chosen randomly from different categories available on play store i.e. top grossing apps, top selling apps, new and updated games, new and updated apps, apps, recommended for you apps, movies apps, books apps, tools apps, communication apps, photography apps, offline apps etc. The developers contact information is given in Appendix which can be reused when someone wants to survey again on “Android” mobile applications. This information will help them to just copy the email addresses and sent them their questionnaire as it will save time and effort. It took me two weeks to get this information from random apps by checking its rating and storing it. This thesis contributes to find out which quality characteristics are important from developer’s perspective and how often these quality characteristics are tested for Android mobile applications. Apart from that it also contributes to know the current testing state of “Android” mobile applications by addressing how the developers test their “Android” mobile applications. The results can be used by developers to know which quality characteristics should be given high priority in testing of “Android” mobile applications and how to actually test mobile applications.

1.1 Aims and Objectives This research will be organized in two surveys (i.e. a literature survey and an empirical survey). Main aim of the literature survey is to know about well-known quality characteristics tested for mobile application development. Main objective of conducting the empirical survey is to find out which well-known quality characteristics for which mobile applications are tested for is important from developer’s point of view and how they actually test them. Moreover, the testing process used by developers to test mobile applications is also surveyed.

1.2 Related Work Mobile application testing is an activity which is aimed at evaluating quality of a program and also for improving it, by identifying defects and problems [2]. In 2013, Android’s Google Play crossed one million available applications and fifty billion downloads [3]. Therefore due to large number of app users and their expectations from the app to be error free, it is very important to meet the quality criteria of an application. Mobile applications growth is increasing intensively today. According to Gartner Inc. [4], in 2013, 879.8 million mobile devices with “Android” operating system were shipped, and in 2014 the expectation of this growth might increase to over one billion. "Every month, around 20,000 new applications are released, and the current number of apps in the “Android” market is over 1,200,000 [5]". These numbers reflect that this platform is intensively increasing day by day and that the focus of developers is to develop more and more mobile applications. Furthermore, apps are easily available through Google Play Store so it has attracted a large number of developers and companies to develop their apps and put it on Google Play Store [9]. Due to this ease there are millions of apps on this platform but it doesn’t ensure that apps are error free. The apps containing errors can significantly affect the developer’s image and may harm the users. So it is the duty of developers to test the apps before releasing to the market to ensure the quality of apps [2]. The apps with almost no errors are most downloaded and used by users and trust is built between user and developers [2].

Mobile application testing is an activity which is aimed at evaluating quality of a program and also for improving it, by identifying defects and problems [2]. It is much more difficult than the conventional software testing because the impact of mobile applications are smaller than the desktop applications on a virtual machine. Mobile application testing has become a great challenge that needs immediate action to be tackled and there is no doubt that mobile applications need specific quality characteristics [6]. The quality characteristics are used by mobile application developers to ensure the quality of mobile applications [2]. Google Play framework makes it very easy for users to search and install the app. Therefore, some developers think it is not necessary to test the app before releasing as it requires a lot of time, so they wait for the users feedback if they report any problem they try to fix it and release the new version of app.[9] In my research methodology course, I contacted 200 developers to perform the empirical survey on quality characteristics tested for “Android” mobile applications by developers. The developers were contacted to know about the most quality characteristic that the mobile applications are tested for. Those 200 developers are the part of these 1000 developers that are contacted. The sample size was small and few quality characteristics were asked. But the importance, use and testing process of “Android” apps still needs to be figured out. There have been many empirical studies conducted on “Android” mobile application testing. Takala applied model based user interface testing on “Android” applications and he has reported his experience [17]. Bhattacharya et al. performed an investigation on bug reports and bug fixing method of “Android” applications [18]. Kropp and Morales examined strengths and weaknesses of “instrumentation framework and positron framework” for testing mobile graphical user interface applications [19]. Syer et al. studied the 15 most common “Android” applications and then compared it with 3 desktop applications [20]. McDonnell et al. studied the constancy and acceptance of APIs in “Android” network [21]. Maji et al. studies and characterized the failures in both “Android” and Symbian mobile OSes [22]. Recently there was a research done that used survey methodology, the tools were identified that are used in testing of “Android” mobile applications by calculating their code coverage, the presence of test case and the challenges faced by developers in testing of “Android” mobile applications [9]. Another research was done to understand the challenges faced by “Android” mobile application developer’s [23]. The quality characteristics tested for windows phone has been researched recently [9]. Different testing techniques are being introduced to overcome the challenges faced by developers but still quality characteristics that “Android” developers think are important and are tested for “Android” mobile applications is unknown and are being ignored. So, to overcome this gap I was interested in performing the empirical survey on the quality characteristics importance and how often these quality characteristics are tested for “Android” mobile applications. Apart from that I was also interested in performing empirical survey on the testing process used by developers for testing of “Android” mobile applications. I surveyed “Android” mobile application developers to know about current testing state of “Android” mobile applications. The rest of the sections are organized as follows: Section 2 presents the research questions and the methodologies used. Section 3 presents the results of an empirical survey. Section 4 presents the analysis of the results. Section 5 presents discussion section in which answers to the research questions are explained briefly and connected with RQS. Section 6 describes validity threats, conclusion and the future work needed.

2 METHODOLOGY:

A mixed approach: literature review and empirical survey was used in this research to carry out the answers of research questions. The literature review was conducted to know about the well- known quality characteristics for which mobile applications are tested for from the published work and the empirical survey was used to know the current testing state of “Android” mobile applications.

2.1 Research questions The research questions considered in this work are as follows: Table 1: Research Questions

Research Questions Description RQ1: What are the quality characteristics that This research question aims at finding the set mobile applications are tested for? of quality characteristics used to test mobile applications from the published work.

RQ2: Which are the most and least important This research question aims to find out the quality characteristics for which mobile most important and least important quality applications is tested for according to characteristics for which the mobile developer’s perspective? applications is tested for according to developer’s perspective.

RQ3: Which are the frequently and barely This research question aims to find out the quality characteristics for which the mobile frequently and barely quality characteristics applications is actually tested for by for which the mobile applications is actually developers? tested for from the developers perspective.

RQ4: Does the importance and testing of This research question aims to find out quality characteristics for which the mobile whether the importance and testing of quality application is tested for differs between top characteristics for which the mobile rated and low rated apps developer? applications are tested differs between top rated and low rated apps developer.

RQ5: How are Android mobile applications This research question aims to find out the tested? current testing process used by developers in testing of Android mobile applications.

2.2 Literature review A mixed approach: database search and snowballing approach was used to extract the papers. Some papers were extracted from IEEE and ACM databases by using the keywords that were derived from research questions. The snowballing approach is then used which involves two steps: (1) deriving the start set of the papers and (2) performing forward and backward snowballing. The research papers extracted from IEEE and ACM databases were used as a start set to perform snowballing.

2.2.1 Extended Design and deriving the Start set of papers: According to Wohlen, a good start set should have the following characteristics [29].

 Diversity of the papers should be targeted which should cover different publishers, authors, years. To obtain this, it is recommended to cover this in start set.  Size of start set depends on the size of area being studied for example if you are focusing on a specific area it requires lesser papers than focusing on a broader area.  If search string is not formulated well too many papers can be found, To mitigate it papers which are highly cited and have more relevant references can be used by to obtain perfect start set.  There can be a risk to miss relevant papers by using different terminology. To mitigate it, synonyms can be used with keywords that are derived from research questions.

The following search string was used to identify the start set was the following: (Testing types OR practices OR quality characteristics) And (Mobile applications) I used backward snowballing which is looking at the reference of the papers by applying inclusion and exclusion criteria. The main advantage of using snowballing is that it just focuses on the referenced and cited papers [26]. The snowballing approach is as good as database search and easy to perform as compared to database. I did the literature review not a systematic literature review due to time constraint. Systematic review is basically in depth study of the relevant topic that identifies, selects all high quality research evidence which is relevant to that topic and to do it successfully it requires a lot of time. Literature review is basically qualitatively reviews on a topic using general methods to collect data. The literature review was conducted to find different quality characteristics for which the mobile applications is tested. The literature review covers the answer of RQ1. Following are the different testing characteristics that the mobile applications are tested for is found from the published work.

I. Usability Testing: Usability testing is used for activities that assess UI content and alerts, user operation flows and scenarios, media richness, and gesture interaction [7] [9]. II. Performance Testing: Performance testing is used for the activities to determine the speed or effectiveness of computer, network, software program or device [6] [7] [9] [24]. III. Reliability Testing: Reliability Testing relates to testing software’s ability to function in given environmental conditions for a particular amount of time [6] [7] [24].

IV. Security Testing: Security testing includes encryption/decryption techniques to ensure the data communication, checks for multi user support, and checks within app for the access of files [6] [7] [23] [24]. V. Memory Testing: 13

Memory testing is used to check how much memory the application is taking. It is also used to check for the memory leakage, if the applications are already killed and their active processes are still taking memory [6] [24].

VI. Energy Testing: Energy testing is used to ensure that the applications don’t use more energy than required one. If more energy is taken by single application then it will reduce the battery life of device [6] [24].

VII. Functional Testing: Functional testing is a quality assurance (QA) process and a type of black box testing that is used to check each and every function works perfectly or not. Functional testing usually describes what the system actually does. Functional testing tests the portion of the functionality of the whole system [6] [7] [9].

VIII. Interoperability Testing: Interoperability testing involves testing whether a given software program or technology is compatible with others and promotes cross-use functionality [7].

IX. Compatibility Testing: Compatibility testing is part of software non-functional tests, is testing conducted on the application to evaluate the application's compatibility with the computing environment [23] [6] [9]. X. Connectivity Testing: Connectivity testing is a testing activity designed to validate the continuity of network communications [6] [7] [23] [24]. XI. Load Testing: Load testing is the process of putting demand on a system or device and measuring its response to check its efficiency [9].

2.3 Empirical Survey There are many different ways to perform a survey like street surveys, telephone survey and electronic surveys. I used the empirical survey (online survey) to gather required data because it is a suitable strategy to collect data from more number of people (methods, tools, developers and companies) to gain the understanding of larger population [10]. With an increasing use of internet users it is easy to perform surveys because you can contact the developers or companies online (i.e. online surveys) and with the availability of this technology the interest to perform surveys is increasing day by day this. Kitchenham and Pfleeger’s [16] show that the interest in doing a survey is increasing in the field of software Engineering. There is another study which is done within the German ViSEK project depicts the growing interest of empirical survey. My survey was based on an online questionnaire with three close ended questions and one open ended question (optional) to collect the required information. All the three close ended questions [11] have a fixed number of options that the respondent has to choose while in open ended question, the respondent is required to write something as answer. The online questionnaire was 14

created on “Google Forms” [12] and the questionnaire link was provided to respondents via email. There are two challenges arise in collecting the online survey: (1) keeping track of response sources and (2) preventing multiple responses from the same person [25]. I filtered responses received based on submission time [15].Since there can be multiple submissions from one system due to browser or internet connectivity problem so screening for time of submission (within a 2- minute interval) eliminated such duplicate responses. Ambiguous responses were not received as our survey was limited to the “Android” developers of mobile applications in play store. Survey responses have to be saved in the database such as Oracle or MySQL [25]. My survey was done through “Google Forms” so responses were saved online in Google database.

2.4 Preparation of Questionnaire Based on the results of the literature review, the questionnaire was developed .The questionnaire was then discussed with the supervisor and some changes were made according to the research questions. The questionnaire was then finalized and sent it to the developers for their feedback. The questionnaire was divided into three parts: The first part was related to the quality characteristics importance from their perspective. The second part was related to the actual testing of quality characteristics by “Android” mobile application developers. The third part was related to what testing process is used by developers in testing of “Android” mobile application. The questions were close ended and every question has checkboxes provided for the predefined answer because nowadays mostly people prefer to answer close ended questions [10].There was also one open ended question but that was optional in case if a developer wants to add any other information . The questionnaire is provided in the Appendix (A). There was a sequence in the questions so that respondents get interest in filling the survey. The questionnaire did not have a lot of questions as it is said that the more questions you ask, the more time it will take to answer it and respondent will lose their interest in filling the questionnaire [10].

2.5 Research Design and Data Collection: I divided the quality characteristic importance into three levels (most important, least important and not important) and testing of quality characteristic also in three levels (i.e. mostly used, rarely used and never used). By doing the literature review in section 2.2, I found eleven unique quality characteristics for which mobile applications are tested for. I included them in the online questionnaire as close ended questions. In addition, I added an open ended question in order to get a response about any other quality characteristic that is also uniquely used by mobile application developers. I decided to limit my research to only one online mobile application store. So I performed the survey on Google Play App. store developers, because according to the Smartphone OS Market Share analysis by IDC [13], “Android” OS has the largest OS market share as compared to other mobile operating systems. Google Play App. Store website [14] enabled us to easily find contact information of developers.

The questionnaire was sent on 20th May to 400 developers but I didn’t get enough responses. The developers’ number was then increased from 400 to 1000. Again the email was sent to remaining 600 developer’s (300 top rated and 300 low rated) on June15. Three reminders were given to them. First one was in the month of May, second was in the month of July and last reminder was given on the month of August. The questionnaire was open from 20th May to 5th September. 15

Total 43 responses were obtained out of 1000 developers contacted of the “Google Play App. Store”.20 responses out of 500 were obtained from the top rated apps developers and 23 responses out of 500 were obtained from the low rated apps developers. All the developers contact information was gathered from Google Play store.

3 RESULTS

The results are categorized according to the top rated developer’s response and low rated developer’s response separately. The summary of each quality characteristics importance, testing and testing process is given in the tables with the help of a pie chart. There is one box on which it is mentioned any other quality characteristics importance and its testing level. Blue color indicates the “Important” and “Mostly used” quality characteristic from developer’s perspective, Red color indicates the “Less important” and “Rarely used” quality characteristic and yellow color indicates “Not important” and “Never used” quality characteristics. This color scheme applies to all the graphs given in the results section. The analysis on this raw results are performed in section 4.

3.1 Top rated apps developer’s feedback The results shown below are the feedback I got from the top rated Apps developers in terms of importance, testing of quality characteristics and the testing process used to test “Android” mobile applications

3.1.1 Importance of quality characteristics The summary of each quality characteristics importance according to the top rated developer’s perspective are given below in Table 2 with the help of pie chart. Table 2: Quality characteristics importance by top rated apps developers.

1.1 Usability Testing 1.2 Performance Testing 1.3 Reliability Testing

Important 16 80% Important 16 80% Important 12 60% Less Important 4 20% Less Important 3 15% Less Important 7 35% Not Important 0 0% Not Important 1 5% Not Important 1 5%

1.4 Security Testing 1.5 Memory Testing 1.6 Energy Testing

Important 14 70% Important 13 65% Important 12 60% Less Important 4 20% Less Important 4 20% Less Important 3 15% Not Important 2 10% Not Important 3 15% Not Important 5 25%

1.7 Functional Testing 1.8 Interoperability Test 1.9 Compatibility Testing

Important 19 95% Important 11 55% Important 12 60% Less Important 1 5% Less Important 7 35% Less Important 6 30% Not Important 0 0% Not Important 2 10% Not Important 2 10%

1.10 Connectivity Testing 1.11 Load Testing

Any other quality characteristic and its importance?

None

Important 12 60% Important 10 50% Less Important 8 40% Less Important 6 30% Not Important 0 0% Not Important 4 20%

3.1.2 Testing of quality characteristics The Summary of each quality characteristics testing according to the top rated developer’s perspective are given below in Table 3 with the help of pie chart.

Table 3: Quality characteristics testing by top rated apps developers.

2.1 Usability Testing 2.2 Performance Testing 2.3 Reliability Testing

Mostly Used 14 70% Mostly Used 12 60% Mostly Used 15 75% Rarely used 6 30% Rarely used 7 35% Rarely used 4 20% Never Used 0 0% Never Used 1 5% Never Used 1 5%

2.4 Security Testing 2.5 Memory Testing 2.6 Energy Testing

Mostly Used 12 60% Mostly Used 9 45% Mostly Used 8 40% Rarely used 5 25% Rarely used 7 35% Rarely used 4 20% Never Used 3 15% Never Used 4 20% Never Used 8 40%

2.7 Functional Testing 2.8 Interoperability Test 2.9 Compatibility Testing

Mostly Used 17 85% Mostly Used 10 50% Mostly Used 13 65% Rarely used 3 15% Rarely used 7 35% Rarely used 5 25% Never Used 0 0% Never Used 3 15% Never Used 2 10%

2.10 Connectivity Testing 2.10 Load Testing

Any other quality characteristic and its testing level?

None

Mostly Used 9 45% Mostly Used 9 45% Rarely used 7 35% Rarely used 5 25% Never Used 4 20% Never Used 6 30%

3.1.3 Testing process of top rated applications Below in Table 4 it’s been summarized how do the top rated apps developer perform testing on “Android” mobile applications. Table 4: Testing performed by top rated developers.

3.2 Low rated apps developer’s feedback The results shown below are the feedback I got from the low rated Apps developers in terms of importance, testing of quality characteristics and the testing process used to test “Android” mobile applications.

3.2.1 Importance of quality characteristics The Summary of each quality characteristics importance according to the low rated developer’s perspective are given below in Table 5 with the help of pie chart. Table 5: Quality characteristics importance by low rated apps developers.

1.1 Usability Testing 1.2 Performance Testing 1.3 Reliability Testing

Important 21 91.3% Important 20 87% Important 11 47.8% Less Important 2 8.7% Less Important 3 13% Less Important 12 52.2% Not Important 0 0% Not Important 0 5% Not Important 0 0 %

1.4 Security Testing 1.5 Memory Testing 1.6 Energy Testing

Important 19 82.6% Important 12 52.2% Important 11 47.8% Less Important 3 13% Less Important 6 26.1% Less Important 7 30.4% Not Important 1 4.3% Not Important 5 21.7% Not Important 5 21.7%

1.7 Functional Testing 1.8 Interoperability Test 1.9 Compatibility Testing

Important 22 95.7% Important 13 56.5% Important 14 60.9% Less Important 1 4.3% Less Important 9 39.1% Less Important 8 34.8% Not Important 0 0% Not Important 1 4.3% Not Important 1 4.3%

1.10 Connectivity 1.11 Load Testing Testing Any other quality characteristic and its importance?

None

Important 10 43.5% Important 12 52.2% Less Important 11 47.8% Less Important 10 43.5% Not Important 2 8.7% Not Important 1 4.3%

3.2.2 Testing of quality characteristics The summary of each quality characteristics testing according to the low rated developer’s perspective are given below in Table 6 with the help of pie chart. Table 6: Quality characteristics testing by low rated apps developers.

2.1 Usability Testing 2.2 Performance Testing 2.3 Reliability Testing

Mostly Used 14 70% Mostly Used 12 60% Mostly Used 15 75% Rarely used 6 30% Rarely used 7 35% Rarely used 4 20% Never Used 0 0% Never Used 1 5% Never Used 1 5%

2.4 Security Testing 2.5 Memory Testing 2.6 Energy Testing

Mostly Used 12 60% Mostly Used 9 45% Mostly Used 8 40% Rarely used 5 25% Rarely used 7 35% Rarely used 4 20% Never Used 3 15% Never Used 4 20% Never Used 8 40%

2.7 Functional Testing 2.8 Interoperability Test 2.9 Compatibility Testing

Mostly Used 17 85% Mostly Used 10 50% Mostly Used 13 65% Rarely used 3 15% Rarely used 7 35% Rarely used 5 25% Never Used 0 0% Never Used 3 15% Never Used 2 10%

2.10 Connectivity Testing 2.10 Load Testing

Any other testing characteristic and its testing level?

None

Mostly Used 9 45% Mostly Used 9 45% Rarely used 7 35% Rarely used 5 25% Never Used 4 20% Never Used 6 30%

3.2.3 Testing process of low rated applications Below in Table 7 it’s been summarized how do the low rated apps developer perform testing on “Android” mobile applications. Table 7: Testing performed by low rated apps developers.

4 ANALYSIS

4.1 Data Analysis The purpose of data analysis was to compare the quality characteristics with respect to their importance and how they are actually used by mobile application developers. To obtain this, I assigned the points to every type of answer. I.e. I assigned 1 point to “Most Important” and “Mostly used” characteristic, 0.5 point to “Less Important” and “Rarely used” and (0) point to “Not important” and “Never used”. I added these points for each quality characteristic to obtain the total points for that characteristic. Moreover, to represent how the testing is performed and the comparison of quality characteristics I decided to use Bar Chart to represent my analysis result.

4.2 Top Rated Apps

4.2.1 Importance by top rated developers Here the analysis is being done to understand which quality characteristics are important, less important and not important from top rated developer’s perspective. Table 8 represents the analysis done for the importance of quality characteristics from top rated apps developer’s perspective. Table 8: Analysis for the importance of quality characteristics from top Rated apps developer’s perspective.

S. Quality Important Less Not No Characteristic (1p) Important important Total Points (0.5p) (0p)

1- Usability Testing 16*(1) = 16 4*(0.5) = 2 0*(0) = 0 16+2+0 = 18

2- Performance Testing 16*(1) = 16 3*(0.5) = 1.5 1*(0) = 0 16+1.5+0 = 17.5

3- Reliability Testing 12*(1) = 12 7*(0.5) = 3.5 1*(0) = 0 12+3.5+0 = 15.5

4- Security Testing 14*(1) = 14 4*(0.5) = 2 2*(0) = 0 14+2+0 = 16

5- Memory Testing 13*(1) = 13 4*(0.5) = 2 3*(0) = 0 13+2+0 = 15

6- Energy Testing 12*(1) = 12 3*(0.5) = 1.5 5*(0) = 0 12+1.5+0 = 13.5

7- Functional Testing 19*(1) = 19 1*(0.5) = 0.5 0*(0) = 0 19+0.5+0 = 19.5

8- Interoperability 11*(1) = 11 7*(0.5) = 3.5 2*(0) = 0 11+3.5+0 = 14.5 Testing 9- Compatibility Testing 12*(1) = 12 6*(0.5) = 3 2*(0) = 0 12+3+0 = 15

10- Connectivity Testing 12*(1) = 12 8*(0.5) = 4 0*(0) = 0 12+4+0 = 16

11- Load Testing 10*(1) = 10 6*(0.5) = 3 4*(0) = 0 10+3+0 = 13

The calculation results for each quality characteristic importance according to top rated developer’s perspective can also be illustrated by using bar chart as shown in Figure 1 below:

Figure 1: Comparison of important quality characteristics from top rated developer’s perspective.

According to the survey responses received, the most important quality characteristic for which mobile applications are tested for among mobile application developers is functional testing and the least important quality characteristic is load testing as illustrated in the Figure 1. We can order these quality characteristics with respect to their importance as following: 1. Functional Testing 2. Usability Testing 3. Performance Testing 4. Security Testing 4. Connectivity Testing 6. Reliability Testing 7. Memory Testing 7. Compatibility Testing 9. Interoperability Testing 10. Energy Testing 11. Load Testing

4.2.2 Testing by top rated developers Here the analysis is being done to understand which quality characteristics are mostly used, rarely used and never used from top rated developer’s perspective. Table 9 represents the analysis done for the testing level of quality characteristics from top rated apps developer’s perspective.

Table 9: Analysis for the testing level of quality characteristics from top rated apps developer’s perspective.

S. Quality Mostly Rarely Used Never Used No Characteristic Used (0.5p) (0p) Total Points (1p)

1- Usability Testing 14*(1) = 14 6*(0.5) = 3 0*(0) = 0 14+3+0 = 17

2- Performance Testing 12*(1 ) = 12 7*(0.5) = 3 1*(0) = 0 12+3+0 = 15

3- Reliability Testing 15*(1) = 15 4*(0.5) = 2 1*(0) = 0 15+2+0 = 17

4- Security Testing 12*(1 ) = 12 5*(0.5) = 2.5 3*(0) = 0 12+2.5+0 = 14.5

5- Memory Testing 9*(1) = 9 7*(0.5) = 3.5 4*(0) = 0 9+3.5+0 = 12.5

6- Energy Testing 8*(1) = 8 4*(0.5) = 2 8*(0) = 0 8+2+0 = 10

7- Functional Testing 17*(1) = 17 3*(0.5) = 1.5 0*(0) = 0 17+1.5+0 = 18.5

8- Interoperability 10*(1) = 10 7*(0.5) = 3.5 3*(0) = 0 10+3.5+0 =13.5 Testing

9- Compatibility Testing 13*(1) = 13 5*(0.5) = 2.5 2*(0) = 0 13+2.5+0 = 15.5

10- Connectivity Testing 9*(1) = 9 7*(0.5) = 3.5 4*(0) = 0 9+3.5+0 = 12.5

11- Load Testing 9*(1) = 9 5*(0.5) = 2.5 6*(0) = 0 9+2.5+0 = 11.5

The calculation results for each quality characteristics testing level according to top rated developers can also be illustrated by using Bar Chart as shown in Figure 2 below:

Figure 2: Comparison of testing level of quality characteristics from top rated developer’s perspective.

According to the survey responses received, the most used quality characteristic among mobile application developers is functional testing and the least used quality characteristic is Interoperability Testing as illustrated in the Figure2. We can order these quality characteristics with respect to their testing as following: 1. Functional Testing 2. Usability Testing 2. Reliability Testing 4. Compatibility Testing 5. Performance Testing 6. Security Testing 7. Interoperability Testing 8. Connectivity Testing

9. Memory Testing 10. Load Testing 11. Energy Testing

4.2.3 Testing process of top rated apps In this section, it is explained how the top rated developers actually test their applications. According to the survey responses received most of the top rated apps developer while

creating test cases they prefer to use the test cases that are predefined in scripts or documents instead of doing testing by themselves (for example random tests, exploratory test etc.). To run their tests, top rated app developers prefer to run their tests on emulator rather than running on phone and 55% of the developers prefer to run their tests automatically via a tool or service as a cloud while remaining 45% run their tests manually.

4.3 Low Rated Apps

4.3.1 Importance by low rated developers Here the analysis is being done to understand which quality characteristics are important, less important and not important from low rated developer’s perspective. Table 10 represents the analysis done for the importance of quality characteristics from low rated apps developer’s perspective.

Table 10: Analysis for the importance of quality characteristics from low Rated apps developer’s perspective.

S. Quality Important Less Not No Characteristic (1p) Important important Total Points (0.5p) (0p)

1- Usability Testing 21*(1) = 21 2*(0.5) = 1 0*(0) = 0 21+1+0 = 22

2- Performance Testing 20*(1 ) = 20 3*(0.5) = 1.5 0*(0) = 0 20+1.5+0 = 21.5

3- Reliability Testing 11*(1) = 11 12*(0.5) = 6 0*(0) = 0 11+6+0 = 17

4- Security Testing 19*(1 ) = 19 3*(0.5) = 1.5 1*(0) = 0 19+1.5+0 = 19.5

5- Memory Testing 12*(1) = 12 6*(0.5) = 3 5*(0) = 0 12+3+0 = 10

6- Energy Testing 11*(1) = 11 7*(0.5) = 3.5 5*(0) = 0 11+3.5+0 = 14.5

7- Functional Testing 22*(1) = 22 1*(0.5) = 0.5 0*(0) = 0 22+0.5+0 = 22.5

8- Interoperability 13*(1) = 13 9*(0.5) = 4.5 1*(0) = 0 13+4.5+0 = 17.5 Testing

9- Compatibility 14*(1) = 14 8*(0.5) = 4 1*(0) = 0 14+4+0 = 18 Testing

10- Connectivity Testing 12*(1) = 12 10*(0.5) = 5 1*(0) = 0 12+5+0 = 17

11- Load Testing 10*(1) = 10 11*(0.5) = 5.5 2*(0) = 0 10+5.5+0 = 15.5

The calculation results for each quality characteristic importance can also be illustrated by using Bar Chart as shown in figure (3) below:

Figure 3: Comparison of important quality characteristics from low rated developer’s perspective.

According to the survey responses received, the most important quality characteristic among mobile application developers is functional testing and the least important quality characteristic is memory testing as illustrated in the Figure 3. We can order these quality characteristics with respect to their importance as following: 1. Functional Testing 2. Usability Testing 3. Performance Testing 4. Security Testing 5. Compatibility Testing 6. Interoperability Testing 7. Reliability Testing 7. Connectivity Testing 9. Load Testing 10. Energy Testing 11. Memory Testing

4.3.2 Testing level by low rated developers Here the analysis is being done to understand which quality characteristics are mostly used, rarely used and never used from low rated developer’s perspective. Table 11 represents the

analysis done for the testing level of quality characteristics from low rated apps developer’s perspective. Table 11: Analysis for the testing level of quality characteristics from low Rated apps developer’s perspective.

S. Quality Mostly Rarely Used Never Used Total Points No Characteristic Used (0.5p) (0p) (1p)

1- Usability Testing 18*(1) = 18 5*(0.5) = 2.5 0*(0) = 0 18+2.5+0 = 20.5

2- Performance Testing 13*(1 ) = 13 10*(0.5) = 5 0*(0) = 0 13+5+0 = 18

3- Reliability Testing 10*(1) = 10 10*(0.5) = 5 3*(0) = 0 10+5+0 = 15

4- Security Testing 11*(1 ) = 11 8*(0.5) = 4 4*(0) = 0 11+4+0 = 15

5- Memory Testing 3*(1) = 3 14*(0.5) = 7 6*(0) = 0 3+7+0 = 10

6- Energy Testing 5*(1) = 5 8*(0.5) = 4 10*(0) = 0 5+4+0 = 9

7- Functional Testing 19*(1) = 19 4*(0.5) = 2 0*(0) = 0 19+2+0 = 21

8- Interoperability 6*(1) = 6 15*(0.5) = 2*(0) = 0 6+7.5+0 = 13.5 Testing 7.5

9- Compatibility 8*(1) = 8 14*(0.5) = 7 1*(0) = 0 8+7+0 = 15 Testing

10- Connectivity Testing 7*(1) = 7 15*(0.5) = 1*(0) = 0 7+7.5+0 = 14.5 7.5

11- Load Testing 4*(1) = 4 15*(0.5) = 4*(0) = 0 4+7.5+0 = 11.5 7.5

The calculation results for each quality characteristic can also be illustrated by using Bar Chart as shown in figure (4) below:

Figure 4: Comparison of testing level of quality characteristics from low rated developer’s perspective.

According to the survey responses received, the most used quality characteristic among mobile application developers is functional testing and the least used quality characteristic is energy testing as illustrated in the Figure 4. We can order these quality characteristics with respect to their testing as following: 1. Functional Testing 2. Usability Testing 3. Performance Testing 4. Security Testing 4. Compatibility Testing 4. Reliability Testing 7. Connectivity Testing 8. Interoperability Testing 9. Load Testing 10. Memory Testing 11. Energy Testing

4.3.3 Testing process by low rated developers In this section, it is explained how the low rated developers actually test their applications. According to the survey responses received most of the low rated apps developer while creating test cases they prefer to use the test cases that are predefined in scripts or documents instead of doing testing by themselves (for example random tests, exploratory test etc.). To 32

run their tests, low rated app developers also prefer to run their tests on emulator rather than running on phone and 70% of the developers prefer to run their tests automatically via a tool or service as a cloud while remaining 30% run their tests manually.

4.4 Comparison of quality characteristics importance, testing and their testing process

In this section the analysis which was done earlier for the top rated and low rated is compared. The most important quality characteristics among top rated and low rated apps developer for which mobile applications are tested is functional testing and usability testing and these quality characteristics are mostly used by both of them when testing mobile applications. Quality characteristics which are not that important from top rated developer’s perspective for which mobile applications are tested is load testing and energy testing and they are not used often by top rated apps developer. Quality characteristics which are not that important from low rated developer’s perspective is energy testing and memory testing and these quality characteristics are not used often by low rated apps developers. When it comes to testing, top rated and low rated developer’s both create test cases that are predefined in scripts or documents. Both developer’s prefer to execute their tests on emulator rather than on mobile and they prefers to run their tests automatically via a tool or service as a cloud. The importance, testing and testing process of top rated apps and low rated apps developer doesn’t differ much but when it comes to unimportance and not testing of quality characteristics for which mobile applications are tested for it differs between top rated and low rated apps.

5 DISCUSSION

In this research study I explored various quality characteristics that are used in testing of “Android” mobile applications. Then an empirical study was performed to verify the literature findings by conducting an empirical survey. The importance and testing of quality characteristics of both the top rated and low rated were figured out. Moreover, the testing process used by top rated apps developer and low rated apps developer was also figured out. Answer to Research questions: RQ1: What are the quality characteristics that mobile applications are tested for? Answer: 11 different quality characteristics that the mobile applications are tested for is found from the literature I-e Usability Testing, Performance Testing, Reliability Testing, Security Testing, Memory Testing, Energy Testing, Functional Testing, Compatibility Testing, Interoperability Testing, Connectivity Testing and Load Testing. RQ2: Which are the most and least important quality characteristics for which mobile applications is tested for according to developer’s perspective? Answer: According to the survey responses top rated apps developers and low rated apps developer both responded functional testing and usability testing are the most important quality characteristics for which the mobile application is tested for. Top rated apps developer responded energy testing, load testing are not important while low rated apps developer responded memory testing and energy testing are not important when testing “Android” mobile applications. So the results differ between top rated and low rated apps developer when asked about the not important testing characteristics for which the mobile applications are tested for. RQ3: Which are the frequently and barely quality characteristics for which the mobile applications is actually tested for by developers? Answer: According to the survey responses top rated apps developers and low rated apps developers both responded functional testing, usability testing are the most tested quality characteristics for which the mobile applications is tested for. Top rated apps developer responded energy testing, load testing are not tested often while low rated apps developer responded memory testing and energy testing are not tested often when testing “Android” mobile applications. So the results differ between top rated and low rated apps developer when asked about the not tested quality characteristics for which the mobile applications are tested for. RQ4: Does the importance and testing of quality characteristics for which the mobile application is tested for differs between top rated and low rated apps developer? Answer: According to the results, the importance and testing of quality characteristics that the mobile applications are tested for doesn’t differ between top rated and low rated apps developer doesn’t differ. They both responded functional testing and usability testing is most important and mostly tested quality characteristics for which mobile applications are tested for. There is a difference between top rated apps developer and low rated apps developer regarding unimportance and not tested quality characteristic for which the mobile applications are tested for. Top rated apps developer responded energy testing and load testing are not that important from their perspective and is not tested often. Low rated apps developer responded memory testing and energy testing is not important so they don’t test it often. 34

RQ5: How are the Android mobile applications tested? Answer: Most of the developers create test cases that are predefined in scripts or documents. They prefer to execute their tests on emulator rather than on mobile. They run their tests automatically via tool or service as a cloud.

6 CONCLUSION

My research objective was to find out which quality characteristics are important and not important for which the mobile applications are tested for from developer’s perspective and how they are actually tested by “Android” mobile application developers and the testing process used by them for testing of “Android” mobile applications. My research result shows that functional testing and usability quality characteristics for which the mobile applications are tested for are very important from top rated and low rated app developer’s perspective and are mostly tested by mobile application developers in testing of “Android” mobile applications. Top rated apps developer think energy testing and load testing are not that important from their perspective and they don’t test it often. Low rated apps developer thinks energy testing and load testing is not that important and they don’t test it often. When it comes to testing process, both the developers create test cases that are predefined in scripts or documents. Both of them prefer to execute their tests on emulator rather than on mobile and prefers to run their tests automatically via a tool or service as a cloud. The importance, testing and testing process of “Android” mobile application doesn’t differ between top rated and low rated apps developer. The quality characteristics unimportance and not tested differs between top rated and low rated apps developer.

6.1 Validity I filtered responses received based on submission time [15]. Since there can be multiple submissions from one system due to browser or internet connectivity problem so screening for time of submission (within a 2-minute interval) eliminated such duplicate responses. Ambiguous responses were not received as my survey was limited to the “Android” developers of mobile applications in play store. In the start I contacted 400 developers and I expected to get a response from at least 50 people but unfortunately I couldn’t get that response. So I had to increase the number of developers to 1000 and I got 43 responses and received response ratio was 4.3%. Reminder email was sent continuously to increase the number of responses. Following are the validity threats to my research work: 1. I targeted the developers of only one mobile platform and didn’t consider other platform developers. The results claimed are only for “Android” mobile application developers. 2. I didn’t ask about the experience of developers during the survey, so every respondent is considered as equally experienced. 3. The quality of apps developed by respondents was not examined. Every app was given equal importance and the apps were chosen on their rating. Number of downloads or year of the app was not measured. 4. I didn’t consider the domain of mobile application developers as the importance and testing of quality characteristics also depends on the mobile application type. 5. The questionnaire might not be designed properly which can result in confusion and misinterpretation for developers. 6. I only used backward snowballing approach and did not use forward snowballing approach which can be added to threats to validity. 36

6.2 Future Work For future research, the survey can be extended to more number of developers and different platforms so that more number of responses can be received that allows statistical analysis and therefore brings significant to our results. Comparison can be done with different platforms and see whether all the platforms use the same quality characteristics for which mobile applications are tested for or not. Moreover, in future it can be figured out why the unimportance and not testing of quality characteristics for which mobile applications are tested for differ between top rated and low rated apps developer.

7 REFERENCES

[1] G. Chen and D. Kotz, “A Survey of Context-Aware Mobile Computing Research,” Hanover, NH, USA, Tech. Rep., 2000. [2] Zhifang Liu,Xiaopeng Gao, Xiang Long. “Adaptive Random Testing of Mobile Application”,Computer Engineering and Technology (ICCET), 2010 2nd International Conference on (Volume:2) April 2010,pp V2-297 - V2-301. [3] Coimbra Morgado, I. Paiva, A.C.R.; Faria, J.P.”Automated Pattern-Based Testing of Mobile Applications”.Pattern Recognition, Informatics and Mobile Engineering (PRIME), 2014 International Conference on Sept. 2014.pp 294-299. [4] Gartner, "Gartner Says Worldwide Traditional PC, Tablet, Ultramobile and Mobile Phone Shipments Are On Pace to Grow 6.9 Percent in 2014",March, 2014, http://www.gartner.com/newsroom/id/269231, last acessed April 2014. [5] AppBrain, "AppBrain Stats", https://www.appbrain.com/stats/number-ofandroid-apps, last accessed May, 2015. [6] Kirubakaran, B. Karthikeyani,V.”Mobile Application Testing – Challenges and Solution Approach through Automation”.Pattern Recognition, Informatics and Mobile Engineering (PRIME), 2013 International Conference on Feb. 2013. pp 79-84. [7] Jerry Gao, Xiaoying Bai, Wei-Tek Tsai, Tadahiro Uehara. "Mobile Application Testing: A Tutorial". Computer (Volume: 47, Issue: 2), 02 January 2014. pp. 46-55. [8] Robert M. Groves, Floyd J. Fowler, Jr., Mick P. Couper, James M. Lepkowski, Eleanor Singer, Roger Tourangeau. "Survey Methodology, 2nd Edition". ISBN: 978-0-470- 46546-2. July 2009. [9] Kochhar, P.S ; Thung, F. ; Nagappan, N. ; Zimmermann, T. “Understanding the Test Automation Culture of App Developers” Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference on 13-17 April 2015,pp 1-10. [10] Teade Punter, Marcus Ciolkowski, Bernd Freimut, Isabel John. “Conducting Online Surveys in Software Engineering”. Empirical Software Engineering, 2003. ISESE 2003. Proceedings. 2003 International Symposium. Oct 2003. pp 80 – 88. [11] Dillman D., Smyth J., & Christioan LM. (2009) Internet and Mixed-Mode Surveys. The Tailored Design Method. John Wiley & Sons. New Jersey.Q. [12] Google Form”https://docs.google.com/forms”. Last accessed August 30, 2015. [13] Smartphone OS Market Share, Q3 2014. "http://www.idc.com/prodserv/smartphone- os-market-share.jsp". [14] Google Play App. Store. https://play.google.com/store. Last accessed August 2015. [15] M. V. Selm and N. W. Jankowski, “Conducting online surveys,” Qual. Quantity, vol. 40, no. 3, pp. 435–456, 2006. [16] B. Kitchenham, S. Pfleeger, Principles of Survey Research – series of 6 articles, in: ACM Sigsoft Software Engineering Notes, Vol 26 (No.6), Vol 27 (No. 1,3,5), Vol 28 (No.2), 2001-2003.

[17] T. Takala, M. Katara, and J. Harty, “Experiences of system-level modelbased GUI testing of an Android application,” in ICST, pp. 377–386, 2011. [18] P. Bhattacharya, L. Ulanova, I. Neamtiu, and S. C. Koduru, “An empirical analysis of bug reports and bug fixing in open source Android apps,” in CSMR, pp. 133–143, 2013. [19] M. Kropp and P. Morales, “Automated GUI testing on the Android platform,” Testing Software and Systems, p. 67, 2010. [20] M. D. Syer, M. Nagappan, B. Adams, and A. Hassan, “Revisiting prior empirical findings for mobile apps: An empirical case study on the 15 most popular open source Android apps,” in CASCON, pp. 283–297, 2013. [21] T. McDonnell, B. Ray, and M. Kim, “An empirical study of API stability and adoption in the Android ecosystem,” in ICSM, pp. 70–79, 2013. [22] Kumar Maji, K. Hao, S. Sultana, and S. Bagchi, “Characterizing failures in mobile cases: A case study with Android and Symbian,” in ISSRE, pp. 249–258, 2010. [23] Harleen K. Flora, Xiaofeng Wang, Swati V.Chande,” An Investigation into Mobile Application Development Processes: Challenges and Best Practices”in IJMECS Vol.6, No. 6, June 2014. [24] Muccini, H. Di Francesco, A, Esposito, P, “Software testing of mobile applications: Challenges and future research directions” Automation of Software Test (AST), 2012 7th International Workshop on 2-3 June 2012.pp 29-35. [25] Singh A., Taneja A., Mangalaraj G. "Creating online surveys: some wisdom from the trenches tutorial". Professional Communication, IEEE Transactions. Volume: 52, Issue: 2. 2009.pp.197-212. [26] Wohlin, Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering, in: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE'14), ACM, London, UK, 2014. [27] S M K Quadri & Farooq, S.U. 2010, "Software Testing - Goals, Principles, and Limitations", International Journal of Computer Applications, vol. 6, no. 9. [28] Logeshwaran Murugesan, Prakash Balasubramanian.”Cloud Based Mobile Application Testing” Computer and Information Science (ICIS), 2014 IEEE/ACIS 13th International Conference on 4-6 June 2014, PP 287-289.

[29] C. Wohlin, Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering, in: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE'14), ACM, London, UK, 2014.

8 APPENDIX A: QUESTIONNAIRE Thank you for taking the time to complete the survey. The survey consists of multiple choice questions and will take about 5 minutes. The responses from this survey will be used to analyze which testing characteristic is important and is mostly used in mobile application, and how actually the applications are tested.

Q1: Which Testing Characteristic is important in testing of mobile applications?

1-Usability Testing (activities that assess UI content and alerts, user operation flows and scenarios, media richness, and gesture interaction)

o Important o Less important o Not important

2-Performance Testing (activities for determining the speed or effectiveness of computer, network, software program or device)

o Important o Less important o Not important

3-Reliability Testing (relates to testing software’s ability to function in given environmental conditions for a particular amount of time)

o Important o Less important o Not important

4-Security Testing (includes elements of confidentiality, integrity, authentication, authorization)

o Important o Less important o Not important

5-Memory Testing (memory leaks may occupy the limited memory resource)

o Important o Less important o Not important

6-Energy Testing (active processes of killed applications may reduce the device battery freedom)

o Important o Less important o Not important

7-Functional Testing (To test each and every function works properly or not)

o Important o Less important o Not important 40

8-Interoperability Testing (involves testing whether a given software program or technology is compatible with others and promotes cross-use functionality)

o Important o Less important o Not important

9- Compatibility Testing (part of software non-functional tests, is testing conducted on the application to evaluate the application's compatibility with the computing environment)

o Important o Less important o Not important

10-Connectivity Testing (A testing activity designed to validate the continuity of network communications.)

o Important o Less important o Not important

11-Load testing (the process of putting demand on a system or device and measuring its response)

o Important o Less important o Not important

Any Other Testing Characteristic? (Please write in the box with its importance)

Q2: Which Testing Characteristic is mostly used in testing of mobile applications?

1-Usability Testing