Usability Evaluation Framework for Mobile Apps Using Source Code Analysis

Thesis submitted in partial fulﬁllment of the requirements for the degree of

Master of Science in Computer Science & Engineering by Research

Neeraj Mathur 201350902 [email protected]

International Institute of Information Technology Hyderabad - 500032, INDIA Feb, 2019 Copyright c Neeraj Mathur, 2019 All Rights Reserved International Institute of Information Technology Hyderabad, India

CERTIFICATE

It is certiﬁed that the work contained in this thesis, titled “Usability Evaluation Framework for Mobile Apps Using Source Code Analysis” by Neeraj Mathur, has been carried out under my supervision and is not submitted elsewhere for a degree.

Date Adviser: Prof. Y. Raghu Reddy To my Parents Acknowledgments

My journey in IIIT-Hyderabad has been a wonderful experience. It could not have been so without the support of many people. As I submit my MS thesis, I would like to extend my gratitude to all those people who helped me on successfully completing this journey. First of all, I want to thank my guide Prof.Y. Raghu Reddy, for accepting me as a student and constantly guiding me during MS Research program. His guidance has helped me improve not only as a researcher but also as a person. His help and support during difﬁcult times strengthened and motivated me to move further. I thank my fellow research mates at Software Engineering Research Center: Lalit Mohan, Sai Anirudh, Shivam Khandelwal, Sai Gollapudi and Sridhar C for stimulating discussions. In particular, I am grateful to Lalit Mohan and Sai Anirudh for constantly guiding me and giving directions to my research work. Special thanks to Sai Anirudh for closely helping me in every aspect of my research work. My sincere thanks to Mr. Rohit Udasi - Sr. Development Manager, Sustenance Engineering at Oracle India Pvt. Ltd., for creating enthusiasm and motivation to pursue research at IIIT Hyderabad. My sincere thanks to Mr. Hemendra Pal (Sr. Director) and Mr. Amar Rao (Director) at Oracle India Pvt. Ltd for providing an opportunity to pursue research at IIIT Hyderabad. I could not have accomplished it without the support and understanding of my family. I wish to thank my parents Smt. Sneh Lata Mathur, Shri Yogendra Mathur, my wife Vidhi Saxena and my daughter Navdha Mathur for being my constant support and motivation. Last, but not the least, thanks to IIIT community for giving me an inspiring environment and loads of opportunities to grow.

v Abstract

Apple Inc. unveiled the first iPhone in 2007. Cutting edge technology was introduced in the smartphone with an interface that allowed users to interact with the device through various modes. Also, the ecosystem facilitated developers to create applications on mobile platforms, popularly referred to as “Mobile Apps”. Mobile apps are software applications designed to run on smart-phones, tablets, and other mobile devices. They are typically available through app stores operated by entities that own the corresponding mobile operating system. By 2020, its been forecasted that mobile apps shall generate around 189 billion USD [49] in revenues via app stores and in-app advertising. The increasing use of smart-phones has led to mobile applications replacing or supplementing traditional web-based applications. Given the limitations of the form factor in smart-phones, Usability is one of the essential attributes that determine the success of a mobile apps. The measures available for assessing the usability of mobile apps tend to focus more on the human aspects and less on the functional aspects of Usability. Also, quantitative metrics that measure usability of mobile apps are very limited. As part of our initial research, we studied mobile apps in banking and finance domain. An analysis of the usability issues reported by the mobile users helped capture the challenges faced by these users as part of their day-to-day transactions. Various market giants have proposed varied usability guidelines to enrich user experience (like ISO, Android, Neilson Norman group etc). These guidelines are the repository for software practitioners to build usable mobile apps. However, there are no definitive standards or effective tools to detect and address usability issues. This made us propose a few banking specific usability guidelines based on a usability index called Mobile App Usability Index (MAUI). This study helped us establish an understanding of the functional and non-functional aspects of usability features while building a mobile app. Traditionally, usability evaluation is considered as a post-development activity which is usually taken care of by UX/UI Experts. Usability evaluation was done manually using various methods like cognitive walk-through, field studies, group studies etc, which delays detection of usability issues and requires re-development to address violations. To address this challenge, we formulated a usability framework called UMETRIX to conduct an automated usability evaluation for mobile app developers. The framework is flexible and developer friendly. The framework helps software practitioners validate and verify the presence of a usability guideline in a mobile app using code analysis. We built an automated tool using our framework that has the flexibility to create developer friendly use-cases to evaluate the pres-

vi vii ence of a guideline along with a code-snippet recommendation engine to ease developers to choose a code template for implementing the desired usability guideline. We consider this framework as a productivity tool for developers to pre-validate and evaluate usability issues during the development phases of a mobile app. As part of our research, we collaborated with open source software mobile app vendors for a ﬁeld study. Additionally, we conducted industrial empirical study on a few FinTech app vendors and captured their observations and results. We collaborated with industry practitioners in their mobile app development to address mobile usability issues on their pre-released version of the mobile app. The collaboration helped us document the guidelines which are possible for evaluation using code analysis. Contents

Chapter Page

1 Introduction ...... 1 1.1 What is Usability ...... 2 1.2 Need for Automated Mobile App Usability Evaluation ...... 4 1.3 Usability Testing in Mobile Apps ...... 4 1.4 Code Analysis For Usability Evaluation ...... 5 1.5 Research Contributions ...... 6 1.6 Scope ...... 6 1.7 Thesis Organization ...... 7

2 Related Work ...... 8 2.1 Usability Models ...... 8 2.2 Reviews and Field Studies ...... 9 2.3 Experimental Analysis ...... 9 2.4 Pattern Detection ...... 10

3 Usability Guidelines and Code Templates ...... 11 3.1 Usability Guidelines ...... 11 3.2 Usability Code Template ...... 13 3.2.1 Usability Testing in Mobile Apps ...... 14 3.2.2 Observations ...... 15

4 Identifying Usability Challenges in Mobile Banking Apps in India ...... 17 4.1 Mobile Banking App Usability Challenges ...... 17 4.2 Usability Measurement ...... 18 4.3 Recommendations For Improving Usability ...... 19 4.4 Time Taken to Complete Task ...... 19 4.5 User Interface Display Parameters ...... 20 4.6 Error Handling ...... 21 4.7 Conclusions ...... 23 4.8 Observation ...... 23

5 UMETRIX - Architecting Usability Evaluation Framework for Mobile Applications ..... 24 5.1 Design Goal ...... 24 5.2 Architecture ...... 24 5.2.1 Incorporating guideline into validation test case ...... 26

viii CONTENTS ix

5.3 Outcomes ...... 29

6 Proof of Concept for Android Mobile Apps ...... 30 6.1 Validation Test Case Generator ...... 30 6.2 Architecture ...... 31 6.2.1 Process ...... 32 6.2.2 User Interface ...... 34 6.3 Guideline Evaluator ...... 34 6.3.1 Architecture ...... 34 6.3.2 Process ...... 36 6.3.3 User Interface ...... 36 6.4 Summary ...... 38

7 Evaluation of Framework ...... 41 7.1 Experiment 1: Accuracy and Robustness ...... 41 7.2 Experiment 2: Evaluation with open source mobile apps ...... 41 7.3 Experiment 3: Industry Evaluation and Results ...... 43 7.4 Threats to Validity ...... 47 7.5 Observations ...... 47

8 Conclusion and Future Work ...... 49 8.1 Contributions ...... 49 8.2 Limitations ...... 50 8.3 Future Work ...... 50

Bibliography ...... 52

Appendix A: Usability Guideline Classiﬁcation along with Code Snippets ...... 59 List of Figures

Figure Page

1.1 Comparison of usability models (courtesy Harrison et al [29]) ...... 3 1.2 Mobile Usability Testing Tools ...... 5

4.1 MAUI Parameters ...... 22

5.1 Usability Evaluation Framework ...... 25 5.2 Validation Test Case ...... 28

6.1 Architecture of Validation Test Case Generator ...... 32 6.2 Flow Diagram of Validation Test Case Generator ...... 33 6.3 Process to upload validation case to validation case DB ...... 34 6.4 Validation Test Case Generator Screen Shot ...... 35 6.5 Architecture of Web Guideline Evaluator ...... 36 6.6 Guideline Evaluator Process Diagram ...... 37 6.7 Upload APK ﬁle ...... 38 6.8 Basic Information of APP ...... 38 6.9 Validation Plan ...... 39 6.10 Usability Results ...... 40

7.1 Validation test case results of open source apps - Part 1 ...... 44 7.2 Validation test case results of open source apps - Part 2 ...... 45

x List of Tables

Table Page

3.1 Usability Guidelines ...... 13 3.2 Automated - Usability Guideline ...... 16 3.3 Semi-Automated - Usability Guideline ...... 16 3.4 Non-Automated - Usability Guideline ...... 16

7.1 Validation test results of Froogal and Vlab App ...... 42 7.2 Mobile Apps used for UMETRIX Case Study ...... 46

xi Chapter 1

Introduction

The computing potential of mobile devices is increasing day-by-day. Internet access via mobile broadband is also increasing at a signiﬁcant rate. Almost all the websites are now offering their products and services via mobile apps. These mobile apps are quickly made available to all the users through distribution platforms called ‘app stores’. A recent article1 published in SmartInsights in April 2016, titled - “Mobile Marketing Statistics compilation” describes the details of an extensive survey conducted on mobile device usage over the past decade by capturing the user activity automatically from their mobile devices. Their survey revealed that the mobile subscriptions outnumbered the global human population in 2015. They have also projected that this number might double in the coming decade. It was also identiﬁed that users spend most of their time on mobile gadgets in a typical working day. Their observations revealed that the mobile devices had become the most prevalent and the most preferred personal computing devices in this century. With this increase in mobile device usage, there has also been an increase in mobile app development across all mobile platforms. Mobile app developers are now developing a variety of mobile apps with an aim to replace existing web applications at a skyrocketing rate. An article2 published by a leading survey agency Nielsen titled “An Era of Growth: The Cross-Platform Report” in March 2014, found that the end users spend 90% of their time on mobile apps when compared to web-based applications on their mobile devices. This shows that there exists a dominance of mobile apps over web applications on mobile devices. This is a complete shift from the previous decade wherein mobile apps are preferred rather than using web-based applications on mobile devices. Today’s E-Commerce has taken advantage of this rise. There is a parallel rise in the release of various mobile apps offering almost similar kind of products and services into the consumer market. To withstand the market competition and to reach the desired consumers, it has become mandatory for mobile app developers to build apps with a really good feature set as well as quality. To enhance the quality, usability becomes an important attribute that assesses how easy user interfaces is to use [27]. Additionally, an ease-of-access of mobile apps has become a golden rule for the mobile software

1https://goo.gl/oo8CdZ 2https://goo.gl/1J5gEv

1 practitioners. Fiona Fui-Hoon Nah et al. [27] performed a study on behalf of their utility company to understand the current value of mobile apps in various markets and have listed significant observations. It was observed that to attain a strong resistance from the competitors, it is now obligatory for mobile app firms to follow the usability guidelines and adapt them during the development phase. In reality, mobile apps have minimal generalized usability standards when compared to the usability standards of web-based applications. Usability guidelines for mobile apps detail the ease of use, efficiency, etc. of mobile apps. It is mostly a human-centric activity and perception based. However, there are several guidelines that are functional in nature and can be directly correlated with specific parts of implementation at the code level. For example, an app developer can implement the following guidelines at the code level: ”User can select to reveal or hide password as type, during sign up or sign-in”. This guideline can impact the assessment of Usability from a user perspective. Several such guidelines that can be automated at the code level but are not specified as requirements by the majority of the stakeholders. Such functional aspects of usability tend to be downplayed in focus groups and other human-centric perception based studies. Mechanisms that automatically evaluate existing mobile apps for such guidelines and/or recommend code-snippets that lead to enhanced usability levels of mobile apps can certainly add value to organizations. In this thesis, we present a ”Usability Evaluation Framework” for mobile apps based on prevailing usability guidelines. The framework can be used to assess various mobile applications with respect to their specified guidelines. It is flexible and extensible in the following ways: (1) New guidelines can be added/deleted/modified (2) Validation cases to existing guidelines can be added/deleted/modified (3) Code-snippets for existing, or new guidelines can be added/deleted/modified. As a result, an organization can create their instance of the framework and perform a usability evaluation against their mobile app based on their business need. The work presented as part of this thesis is primarily motivated by following facts: • There is a significant rise in mobile device usage. This brings market competition on offering services via mobile apps. • There is a need to study available usability guidelines and evaluate the possibility of automation. • There are several guidelines which are functional in nature and can be detected at code level • There is a great need for evaluating usability in mobile apps so as to make them more usable and useful. • There is a need to build a Usability Evaluation Framework to address usability issues based on usability code template analysis using usability guidelines

1.1 What is Usability

An application or product is considered “usable” if it has a pleasant user interface, easy to use and behaves as expected. ISO 9241-11 [52] deﬁnes usability as “The extent to which a product can be used

2 by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” Usability is defined based on the following five main quality attributes:

• Learnability: The system should be easy to learn so that the user can rapidly start getting work done with the system

• Efﬁciency: Resources expended in relation to the accuracy and completeness with which users achieve goals

• Memorability: The system should be easy to remember so that the casual user is able to return to the system after some period of not having used it without having to learn everything all over again

• Satisfaction: Freedom from discomfort, and positive attitudes towards the use of the product.

• Errors: The system should have a low error rate, so that users make few errors during the use of the system and that if they do make errors, they can easily recover from them. Further, catastrophic errors must not occur

Figure 1.1 describes the most popular usability evaluation attributes based on Nielsen, ISO and found in the recent integration of usability attributes in PACMAD model obtained from the literature. The study [29] introduced a new usability model named PACMAD (People at the Centre of Mobile Applica- tion Development) to overcome the limitations that exist in the present usability models used to measure the usability of mobile.

Figure 1.1 Comparison of usability models (courtesy Harrison et al [29])

3 1.2 Need for Automated Mobile App Usability Evaluation

Usability Evaluation (UE) is the process of measuring usability and identifying speciﬁc usability issues. The aim is to recognize issues in the user interface that may lead to human error, discontinue the user interaction with the system and cause user frustration. Automation of this process can contribute to better mobile app usability. The merits of Automated Usability Evaluation are:-

• Reduction in cost: Usability evaluation process is costly and takes time if done manually [59] [60]. Automating usability evaluation can lead to faster evaluation and reduce costs signiﬁcantly.

• Usability problems recognized in a consistent fashion: Human evaluation is subjective and may be inconsistent in identifying usability issues. Automation ensures that usability guidelines are applied consistently and homogeneously eliminating any subjectivity and misinterpretations as reported by some researchers [61] [62] [63] [64].

• More accurate and predictable cost/effort prediction: Automated usability evaluation tools are systematic and methodical. This can lead to a more accurate prediction of usability issues.

• Dependence on Usability Experts is reduced: Usability experts are rare to ﬁnd. Automation reduces dependence and enables designers lacking expertise in usability skills [61] [65] [66].

• Enhanced Coverage: Automation can help focus on speciﬁc usability aspects increasing the depth of evaluation.

• Comparative evaluation of different designs: Automated evaluation analysis of evaluation data enables faster and easier comparative evaluation of different designs [67].

1.3 Usability Testing in Mobile Apps

Most of the UX/UI quality engineers who perform mobile app testing rely on various tools to validate the quality of their mobile app. Currently usability testing tools perform evaluation based on User Feedback, Session recording and Event watching. Most of these tools are the best ﬁt for either black/grey box testing or manual testing over a post-release version of a mobile app. Finding usability problems in a developed version will essentially require redesigning of UI and develop a new version, thus increase in the overall development time and cost. However, the existing tools may not be able to study the usability while the mobile app is under development. Usability Code-template (see section 3.2) analysis is practically not possible using these existing tools. The available static code analysis tools do not analyze different type of programming language artefacts (JAVA, XML, SWIFT etc.). Therefore it is difﬁcult to extend a single static analysis tool for mobile app usability evaluation purpose. Thus there is a need to build a framework which suits for usability evaluation purpose. Fig 1.2 gives us the scope of the widely used tools in mobile usability.

4 Most of the available tools do not analyze usability code template so as to verify the usefulness of the mobile app. userlytics 3, userzoom 4 are the most prominent tools which perform usability evaluation on released apps based on user feedback. They automatically collect user experiences and observations to evaluate the usefulness of a mobile app. Applause 5 tool helps software testers to perform evaluation based on the sessions available in the mobile app. Session-based validation is performed to understand the user effectiveness and the performance of an app with respect to the speciﬁc tasks and actions. Appsee 6 is most widely used event-based tool for usability evaluation. It studies the events of actions in sequence and validates the user experience. Group of events can be studied to realize the user challenges post-release of mobile apps. However, to the best of our knowledge, none of these tools can express the usefulness of the mobile apps based on usability code templates. Code template analysis can be applied during the development phase enabling usability problem insights to developer/practitioners compared to other automation technique. Thus there is a great need for a usability code template analyzer to help mobile practitioners address functional usability issues during mobile app development rather waiting for usability testing post mobile app release.

Figure 1.2 Mobile Usability Testing Tools

1.4 Code Analysis For Usability Evaluation

In a product development life cycle there are various artefacts available like software requirement specification, functional requirement specification, test cases, build/release documents, source code etc. of a product. Given the focus of faster release cycles, it is often found that all these artefacts will nec- essarily not be present or may be present with insufficient details expect the source code. Functionality

3https://www.userlytics.com/ 4https://www.userzoom.com/ 5https://www.applause.com/ 6https://www.appsee.com/

5 is considered Sacrosanct. So, invariably, the one artefact that’s always going to be present is the source code. Source code can provide better insights compared to other artefacts. Thus source code analysis is vital in analyzing all aspects of software quality. In our work, we have restricted our study to source code template analysis and focuse our effort to analyze usability related quality issues. Louridas [69] et al. has explained the importance of static code analysis and how static code checker tools detect problems which are usually missed during the development process.

1.5 Research Contributions

The research work presented as part of this thesis makes the following contributions: • We have studied usability issues related to the mobile banking apps and proposed a Mobile App Usability Index (MAUI) for measuring its usability. • We have studied the available usability guidelines (from Nielson et al. [54] and Google [55]) and came up with an approach to detect the presence of these guidelines using usability code template analysis. • We have proposed a Usability Evaluation framework to detect the presence of code template, which is extensible to incorporate new guideline(s), code template(s) and validation test case(s). • We have implemented an instance of our proposed framework using DotNet technology stack. • We have implemented 18 guidelines and its related code templates in the proof of concept application • We have built a recommendation engine to suggest code snippets to implement a usability guideline. • We have validated tool against 45 open source apps available in open source play store (F-DROID) and veriﬁed its robustness. As an outcome, the proposed tool can be used to detect usability issues during the development phase of a mobile app and provides code snippet recommendations on how to implement a particular usability guideline. This framework will be helpful for the developer(s), tester(s) to understand the usability issues in a given mobile app during multiple iterations of the development process. Due to the automated nature of the framework, it will also reduce the testing effort and the cost.

1.6 Scope

The research work presented as part of this thesis is limited to the below scope: • The usability evaluation framework proposed as part of this thesis is formulated for mobile apps due to below challenges: – Available tools and methods are mostly used in post development of a mobile app. – Usability evaluation by human experts is expensive and time-consuming.

6 – Framework/tool that can be used by developers during the development phase will help to validate the conformance of usability guidelines. • The development of an extensive list of validation test cases for available usability guideline is not in the scope of our current research work. It can be crowdsourced as part of future work. • User can implement and extend our work to build their set of validation test cases as per the need. In all the above cases, usability evaluation framework will help developers, testers, and product managers understand the conformance of usability for the mobile app.

1.7 Thesis Organization

The rest of the thesis is organized as follows: Chapter 2 provides detailed related work on usability on mobile apps. Describes existing work proposed by the research community about usability models, tools and methods to detect usability issues. Chapter 3 provides background for the proposed framework and its underline implementation. Chapter 4 introduces to our study on usability issue in mobile Apps for the Indian banking sector. Provide details about proposed guidelines and mobile app usability index (MAUI) for mobile banking. Validation through banking experts. This study helped us understand the need for usability evaluation framework. Chapter 5 introduces UMETRIX - usability evaluation framework and explains technical details and architecture of the proposed framework. Chapter 6 proof of concept, introduces to the implementation of an instance of UMETRIX framework using “.Net” technology stack. Chapter 7 introduces to the veriﬁcation and validation of proposed framework on open source 45 mobile apps available from free play store (F-Droid) and an industrial survey conducted for accessing the usefulness and effectiveness of framework. Chapter 8 concludes the thesis with a summary, future work recommendation and highlights of the thesis.

7 Chapter 2

Related Work

This chapter introduces the state of the art usability evaluation approaches for mobile application. Sections 2.1, 2.2, 2.3 and 2.4 give us an overview of these topics respectively. The usability evaluation research in mobile apps studied can be broadly classiﬁed into the following categories. • Usability Models - includes work on models suggested by different usability practitioners/standard bodies ISO, Nielsen Norman group, PACMAD model. • Reviews and Field Studies - includes usability issues, human behaviour evaluation, usability evaluation, by performing interview and ﬁeld studies to understand end-user-observations in real time mobile app use. • Experimental Analysis - includes work done by the research community by performing experimental analysis using human experts, behaviour analysis in understanding usability challenges for mobile apps. • Pattern Detection - includes various ways to detect patterns used by the research community. Our approach is inspired by these techniques to detect usability code templates by code analysis. Below section covers a brief discussion on the above categories together and provides their short- comings respectively.

2.1 Usability Models

Harrison et al. [29] conducted an extensive literature review on the usability of mobile applications. They listed out the research trends in usability evaluation in mobile apps and introduce a usability model called PACMAD (People At the Center of Mobile Application Development) considers user observations as standard to evaluate the usability of the mobile app. However, the veracity of their usability model required a vast user sample set to validate their observations. Sanjeev et al. [9] performed a systematic literature review on usability challenges in mobile web and identiﬁed the list of practices and usability models. Yadav et al. [21] proposed an AHP-Entropy model as a usability evaluation of mobile phones which rank alternatives and help in selecting mobile phones based on usability factors of

8 software quality model. Kutar et al. [24] proposed a Goal Question Metric (GQM) based on Usability Guidelines for Mobile applications. Cuadrat Seix et al. [28] introduced validation methods for quantitative mobile testing based on desktop eye-tracking. This was a state-of-art review on a human experimental setup to study the human interaction speciﬁc to interfaces. Chen et al. [26] investigated the differences between the usability tests on mobile phones in a laboratory and real-life situations. The end user behaviour and subjective responses towards the applications in mobile devices were captured. Nielsen [25] UI research group have formulated an exhaustive list of heuristic evaluation methods. Kascak et al. [14] came up with a guidelines based validation study to improve usability in healthcare domain speciﬁc apps for old- age mobile users. Rahmat et al. [12] proposed a framework with an integrated approach to usability evaluation for a smartphone app in view of abstraction levels of usability criteria and user interface elements of an app.

2.2 Reviews and Field Studies

Kim et al. [23] performed a meta-analytical review on existing empirical mobile usability studies and listed out contextual factors, the core and peripheral usability dimensions and key findings for future of mobile usability research. Singh et al. [22] came up with twenty-five dimensions based on a review model and proposed these dimensions as key attributes for designing a usable mobile app. Gulla et al. [20] performed a field study test on participants find the ability to actively shape its news recommendation strategy a useful and desirable feature. This was a real-time usability study based on the user-behaviour evaluation. Sari et al. [19] studied the role of visual usability and visual aesthetics in an experimental research setup. This study examined user experiences and preferences in relation to the visual elements of color and perceived dimensionality of two different mobile apps. Ian et al. [18] presented their findings of a comparative usability evaluation of a tag-based interface and the present conventional interface in the Australian banking context. Kortam et al. [17] used System Usability Scale (SUS) to collect data on usability applications on mobile platforms based on popularity and captured SUS rating to benchmark the design decisions. Ayob et al. [13] came up with UI guidelines identified based on a human evaluation of mobile apps using various interface design rules. These field studies tell us the dependency of end-user observations to evaluate the mobile usability issues in real-time mobile apps.

2.3 Experimental Analysis

Abran et al. [16] conducted an expert based usability study to understand the usability challenges in iOS mobile apps. Stringer et al. [15] proposed a mobile framework for two types of cognitive tests - Trail Making and Reaction Time and investigated its performance. Lopez et al. [11] study have allowed identifying different interaction patterns and contextual factors not likely to be simulated in a laboratory

9 environment. Part et al. [10] discussed five types of bad symptoms and illustrated examples for each one based on human behaviour analysis. Lalit et al. [4] performed a survey-based analysis to understand the usability challenges in banking mobile apps and proposed a usability index to usability evaluation. Hussain et al. [8] considered software quality attributes as the basis to evaluate the usability of banking apps and few usability dimensions were captured. Kramer et al. [7] proposed a design-oriented usability evaluation questionnaire to validate the human interaction with business mobile apps and published their significant observations. Zbick et al. [5] proposed a web-based framework to deploy learning mobile apps and studied learnability, usability, and acceptance using human observations. Luna et al. [3] determine the capability of design patterns to find usability issues in mobile groupware interfaces. A particular collection of patterns was used for this purpose to estimate the reliability of usability levels of mobile user interfaces. Jailani et al. [2] have proposed usability guidelines for mobile apps for Construction Industry based on an experimental study. Az-zahra et al. [1] conducted an evaluation of the usability performance of the mobile device apps, which is part of a culinary recommendation system, through usability testing. The usability performance between two different interaction methods for each input was measured in terms of their overall effectiveness, learnability, and satisfactory performance.

2.4 Pattern Detection

A pattern describes a commonly-recurring structure of communicating components that solve a general design problem in a particular context [70]. Hummel et al. used natural language pattern guidelines from the literature and quantified them for static code analysis with data mined from a significant collection of open source systems for suggesting opportunities for applying strategy design pattern [68]. Louridas et al. explained the importance of static code analysis and how static code checker tools detect problems which are usually missed during the development process [69]. Similar to a design pattern, programmatic Idioms are low-level patterns specific to a programming language. An idiom describes how to implement particular aspects of components or the relationships between them using the features of the given language [70]. Allamanis et al. [71] used statistical natural language processing to mine idioms from source code, with their tool they were able to detect presence of code snippets. Overall, there are many mobile usability studies published in the literature to address usability challenges. However, to the best of our knowledge there is no practical framework/tools in literature for addressing usability challenges from a developer perspective. All the above-proposed methods are not robust enough from a developer perspective and do not provide support to software practitioners in iden- tification of usability challenges during the application development phase. In our research work, we formulated a framework which uses usability guidelines as a reference to detect guideline violations using source code analysis.

10 Chapter 3

Usability Guidelines and Code Templates

In this chapter, we introduce Usability Guidelines and Usability code template concept, needed to understand the proposed approach for a framework and implementation in subsequent sections.

3.1 Usability Guidelines

The usefulness of mobile devices has increased greatly in recent years allowing users to perform more tasks in a mobile context. This rise in usefulness has come at the expense of the usability of these devices. ISO defines usability [52] as “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use”. The word ‘usability’ also refers to methods for improving ease-of-use during the design process. As per ISO - Learnability, Efficiency, Memorability, Errors and Satisfaction are considered to be the traditional usability attributes. However, as part of a recent study - PACMAD Usability model [29] suggested Effectiveness and Cognitive Load as significant additional usability attributes specific to mobile devices. Usability Guidelines are statements that determine an action of usefulness. Usability guidelines aim to streamline particular processes according to a set routine or sound practice. By definition, a guideline is not mandatory. Guidelines are not binding and are not enforced. Below are few examples of usability guidelines which are practiced by mobile app developers. • Menu drawer is accessible from all app levels • Menu list terminology is obvious to the target user • If app sign-up is mandatory, the user can choose to demo it first • If app requests sign-up, the user can choose to continue as a guest • In a form, a cursor placed in the first field, and the appropriate input displayed by default • Text in text boxes should be in single line and not spread across multiple lines Various mobile operating system producers like Apple-iOS, Google-Andriod, Windows-Mobile etc. have also come up with their own usability guidelines. These guidelines are specific to their mobile device and are more comprehensive when compared to existing ISO Guidelines (ISO -ISO/IEC FCD 18021 - User interface for mobile tools [53]). Most of the mobile usability guidelines are non-generic

11 and are domain specific in implementation. With the increase in a number of unique usability guidelines, mobile software practitioners have become unclear on implementing these guidelines through code. It is also difficult to validate and map these guidelines with usability attributes. Classifying the available usability guidelines in-line with existing usability attributes is difficult. However, this classification helps mobile software testers to validate the usefulness of their mobile apps based on usability attributes and will help them improve the quality of their product. We have thoroughly reviewed the available usability guidelines [54] [55] and found that these guidelines are broadly classified into two categories Functional and Non-Functional. Table - 3.1 contains more details in this regard. Functional: A Functional guideline is defined based on behaviour or feature which is associated with Usability aspect of an app. Let us consider a Functional guideline [54] “Password field should have a feature to reveal password in clear text”. This guideline has a generic or a non-generic way of code-level implementation i.e. this feature can be implemented in multiple ways. However, the validation case can be build based on the code-template which is distinct to a given app. Non Functional: A Non Functional guideline is defined as a formal statement which is not related to a functionality of a given app but it is indirectly associated with usability aspect of an app. It is not possible to interpret these guidelines using code templates as there are no standard code templates associated with them. Let us consider a Non Functional guideline [54] “Label button in a descriptive way’’. Here defining a validation case is difficult as this cannot be realized through code. Such guidelines require manual validation and they are difficult to automate using a tool unless the “descriptive way” is clearly defined. We have further classified these two categories based on their level of automation which can be achieved through code analysis.

• Automated: Usability Guidelines can be totally validated using Code

• Semi-Automated: Usability Guidelines can be partially validated using Code. Additional human or manual veriﬁcation is required

• Non-Automated: Usability Guidelines requires complete manual or human intervention for validation

As part of thesis work, we have reviewed guidelines from Nielson et al. [54] (336 guidelines) and Google [55] (107 guidelines) over the period of three months. As part of the review process, a team of 4 people from academia and industry led by the author of this thesis was involved in reviewing and classifying. Participants were able to identify code templates for approx 75 guidelines and were able to classify into identified categories i.e. Functional and Non-Functional along with the level of automation can be achieved to detect guideline violation. They have also documented the identified code snippets which are used to implement the functional guidelines. Details regarding the classification is available in Appendix - A.

12 Table 3.1 Usability Guidelines Category Type Guideline 1. (G#24, NN#183) User can select to reveal or hide password as they type, duringsignup or sign-in 2. (G#23) First-time user has multiple sign-up options (e.g.,username / Automated password and social sign-up facebook.com, google, twitter) 3. (NN#88) Minimize scrolling in a drop-down box by using the whole screen to display long lists of options 1. (G#40) Product images can be easily zoomed into (and are still high quality) Functional 2. (G#100) In a form, telephone number entry is restricted to a dial-pad Semi-Automated (i.e. numbers only) 3. (G#104) In a form, user can select times or dates from a picker control (e.g. for entering travel dates, date of birth or time of arrival) 1. (NN#112) Label buttons in a descriptive way. 2. (NN#121) An error message should tell users what the problem is, Non Functional Not Automated and what they can do to ﬁx it. 3. (NN#232) If no search box is visible, use a standard magniﬁer-glass icon for search

3.2 Usability Code Template

We define Usability Code Template as an unfinished design that can be transformed directly into a code to address a usability problem. They are simpler versions of design patterns which are limited to User-Interface design. Usability Code templates are theme/template based solutions for mobile apps. These code templates help software practitioners to address most of their design challenges and avoid development constraints during mobile app development. Let us take a Mobile Usability Guideline [51] by Nielsen Norman Group - “User can select to reveal or hide password as they type, during signup or sign-in”. Listing 5.1 and 5.2 shows the Android and IOS based code template to display show/hide icon in the password field. This code template is available as part of Android Software Development ToolKit. Most of the code-templates are repeatable during implementation and can be generalized for standard mobile SDK like Andorid, Windows Mobile, and iOS Mobile.

Listing 3.1 Reveal or Hide Password

13 Listing 3.2 Reveal or Hide Password (for iOS) if (! self .password.secureTextEntry) { self .password.secureTextEntry = TRUE; [hideShow setTitle :@”SHOW” forState :UIControlStateNormal ]; } e l s e { self .password.secureTextEntry = FALSE; [hideShow setTitle :@”HIDE” forState :UIControlStateNormal]; }

In practice, it is difficult to quantify the effectiveness of the usability code template unless they are developed based on usability guideline. Code template can become the bridge to solve the usability issues in the mobile apps and help mobile app practitioners to build better apps. Usability Code template and Usability Guidelines are closely associated with each other. A Usability Guideline can be achieved by instantiating a corresponding usability code template, similarly, a usability code template can be used to validate and verify the compliance of usability guideline in a given mobile app. However, not all usability guidelines can be realized using usability code template. Above classification has become difficult for us to construct as almost all the usability guidelines were informally defined. As part of our research, we depend on reviewing usability code template in mobile software code to evaluate usability guideline. This helps us to understand the level of automation required to realize these guidelines and map our results with the usability attributes to confirm the degree of usefulness of a mobile app. Table 3.2 describes an example of an Automated use case of a Usability Guideline for a functional requirement in a mobile app. Table 3.3 describes an example of a Semi-automated use case of usability guidelines for a functional requirement in a mobile app. Table 3.4 describes an example of a Non-automated use case of usability guidelines for Non-functional requirements in a mobile app. Most of the functional requirement can be implemented using Code. However, it is practically difficult to implement and verify the Non-functional requirement. Various testing methods like Load/Per- formance testing, Reliability testing, Stress Testing and Usability Testing etc. are practiced in Industry with some level of human intervention in most of the software testing groups. Similarly, mobile software practitioners adopt task-based studies with human subjects to validate and verify the non-functional requirements. Most of the Usability Guidelines which cannot be validated via automated methods require human observations and field studies. Thus finding a right code-template becomes difficult for Non-functional requirements.

3.2.1 Usability Testing in Mobile Apps

According to Chi et al. [74] there can be two ways to set-up an automatic usability evaluation. They are:

• Make use of conformance to standards or guidelines

14 • Try to predict the usage pattern of a system

We also carried out a review of available automation tools used for evaluating usability issues on Mobile apps. Majority of these tools are the best ﬁt for either black/gray box testing or manual testing on a post-release version of a Mobile app. These tools offer User Feedback, Session recording and Event watching and analyze usage behaviour. These tools may not help developers solve usability challenges during design and development phase. Justine et al. [75] developed a framework that makes use of standards for usability evaluation of websites. However, their framework is limited in scope to websites, which uses HTML code analysis. The HTML code analysis developed as part of their research is not possible to apply on mobile apps since the source code of mobile native apps is not HTML.

3.2.2 Observations

With the background of usability guidelines, classiﬁcation and code template analysis. We would now like to understand the actual challenges faced by users by testing various mobile apps with poor usability. In upcoming chapter we will conduct a case study on Indian mobile banking apps to understand the usability challenge’s and requirement of guideline based usability evaluation method.

15 Table 3.2 Automated - Usability Guideline NN#233.The length of the search box should be the largest pos- Guideline sible size that will ﬁt on the screen Source NN/g (Nielson & Norman Group) Type Functional Requirement When we use app:showAsAction=“always”, the search box will Code Template ﬁt automatically available space (AndroidSDK) Validation Search for above code in Source Code

Table 3.3 Semi-Automated - Usability Guideline

G#40.Product images can be easily zoomed into (and are still high Guideline quality) Source Android-UX Type Functional Requirement Code Template Zoom In/Out Code Check the presence of above source code. Also, need a human Validation validator to experience the behavior

Table 3.4 Non-Automated - Usability Guideline

NN#232.If no search box is visible, use a standard magniﬁer- Guideline glass icon for search Source NN/g (Nielson & Norman Group) Type Non-Functional Requirement Code Template Not applicable Validation Need a Human observer to experience the feature

16 Chapter 4

Identifying Usability Challenges in Mobile Banking Apps in India

In this chapter, we explain our study performed on usability challenges in mobile apps in the context of India banking sector. We have chosen banking domain for our study due to the recent push towards digital India, demonetization and cashless economy to curb black money. This made an interesting domain to study usability challenges and the number of banks offering services through mobile apps. India has 868+ million active subscribers with 160+ million smartphone users. However, the number of mobile banking transactions on smartphones is less than 1/6th of the smartphone users and 1/5th of the other digital transactions excluding ATM channel. One of the major reasons for such low numbers is the usability of the mobile banking app. We focus on usability issues related to mobile banking apps and proposed guidelines and a Mobile App Usability Index (MAUI) metric for improving usability based on various usability parameters.

4.1 Mobile Banking App Usability Challenges

A survey with a sample of 1434 participants with diverse backgrounds (Chief Information Security Ofﬁcers of the banks, Mobile Banking Channel Managers and smartphone users - working women and men from Information Technology (IT) and Non-IT companies, homemakers, retired staff of public sector ﬁrms, etc.) was conducted using WhatsApp, Facebook, LinkedIn, emails and face-to-face interactions to understand the usability related challenges of mobile banking. The participants of the survey were aware that the authors were involved in banking technology research and hence gave feedback with an expectation for improvement of mobile banking adoption. Additionally, a total of 26,131 comments posted by mobile banking app users on Google play store since Jan 2014 were analysed. We used Google playstores public API to fetch comments of major mobile banking apps (public/private). We performed sentiments analysis of the user comments to understand their satisfaction level with mobile banking apps. We evaluated usage of RapidMiner tool for sentimental analysis and later resorted to likert scale (1 to 4) and assessed the positive and negative sentiments of the users. The typical usability challenges in mobile banking apps can be summarized by some of the comments received during the survey:

17 “Simplify apps with inbuilt tamper free security rather answering questions start use of digital sig- natures assigned to individuals” “User friendly app which would not be difficult to navigate even when network speed is not great” “Taking least and only required inputs.. For any operation on mobile banking app..Building trust in users to adopt mobile banking/marketing providing security pin generator token/device..even to farm- ers..and rest of the banking should be carried out with dtmf/sms based inputs..as these are the easiest to use.. any person can easily adopt it..Separate/dedicated communication channels via service providers should be opened with highest security measures” Some common comments from Google play store were: “Screen flickering and UI goes blank sometime” “Application tends to take much more time then compared to other competitive bank, on 2G it tends to hang does not show proper error messages.” “Screen with poor UI and color combinations. Less user friendly.” From the survey and user comments analysis, there was a clear correlation perceived by the users between usability issues and reliability. For example, if the app had difficult activation process, customers lost interest in the app and stated that the mobile banking app was not reliable. On the contrary, some other mobile banking apps had a strong positive sentiment and customers were satisfied with the app. The survey results concur with our supposition that better usability leads to better adoption and in turn better revenues for the banks.

4.2 Usability Measurement

Usability is captured as a set of non-functional requirements in software engineering practices. The user interface designer develops wireframes and mock-ups based on the requirements and available organization standards. In our interaction with the Indian banks mobile banking teams, we realized that most of them do not have a user interface designer role. Business Analyst or technology teams develop mock-ups or screen designs. In some cases, the interfaces were developed directly without business team/user involvement. To understand the usability requirements and factors to measure usability, some of the widely adopted mobile banking apps of major banks (Wells Fargo, Bank of America, Barclays Bank, Citi Bank, and JP Morgan Chase Bank) and Mobile Wallets (Square, Starbucks, PayPal, mPay, etc) were installed and the usability factors were studied. As there are no BIS guidelines or assessment factors on usability of mobile applications, the Human Computer Interface and the User experience guidelines for mobile devices available from Apple for iOS (iOS Human Interface Guidelines) [33], Google for Android (Mobile App Design from Android) [34] and Microsoft for Windows Mobile (Usability Guidelines) were studied. Majority of banking apps run on these platforms, hence it covered the entire gamut of mobile banking apps. Also, the usability models suggested by Nielsen and Norman group (Mobile website and application usability) and, People at the

18 Centre of Mobile Application Development (PACMAD) model [38] [31] were studied. In addition, the ﬁve human computer interface laws were analysed: (1) Miller’s law of STM (short term memory) [39], (2) Fitts’ Law [37], (3) Hick Hyman Law [46], (4) Power law of practice [41], and (5) Zipf law [30](Apitz, G., Guimbretire, F. and Zhai, S.,2008). The main reason to study these laws was to get a grasp of the human aspects of remembrance, time taken to make a decision based on the available choices and user expectations on keeping most frequently used as the ﬁrst option as they have an impact on user perception on usability. ISO 9241 [36] on Ergonomics of Human System Interaction Guidance on World Wide User Inter- faces (Ergonomics of Human System Interaction) manual was studied as well. However, the guidelines proposed were generic to web applications rather mobile apps interfaces. Like most other non-functional requirements, measuring usability is challenging. In our work, usability is measured based on the following factors: 1. Time taken to complete the task. Intuitively this can be a measure as the number of clicks. 2. User interface display parameters. This refers to the font, colours, etc. 3. Error handling. This focuses on the error messages and the techniques to handle them.

4.3 Recommendations For Improving Usability

Several researchers and organizations have provided usability recommendations for web applications. Some of the banks we studied seemed to use these recommendations for developing their mobile banking applications. It is imperative that banks not develop the user interface of mobile banking apps similar to internet banking sites as the display screens, network connection, and user attention span are all different from a desktop/laptop usage. As the saying goes, “what gets measured gets managed”. After the adoption of the proposed parameters, a usability lab can be set-up for testing or mobile apps like Userzoom, Loop, Magitest, etc. can be used for testing. The usability parameters suggested in this paper are for the following factors: • Time taken to complete the task • User Interface Display • Error Handling Considering the above factors, we recommend a heuristic based evaluation method that computes an index score called MAUI (Mobile App Usability Index). MAUI can be used for measuring the effectiveness of implementation of Usability standards for banks mobile app.

4.4 Time Taken to Complete Task

Below are the usability parameters for the time taken to complete a speciﬁc task. For example, adding a beneﬁciary account, performing money transfer, making chequebook request, navigating through the

19 screens, etc. are tasks that need to be completed using the speciﬁed number of clicks on the mobile phone.

• A1 - The account summary (using SIM, IMEI and other device information without disclosing any Personally identifiable information), nearest ATM/Branch, and contact information of call centre (with option of click to call) should be available without login • A2 - Maximum five fields should be sought from thecustomers while completing a form in the screen • A3 - The screen navigation should start with morefamiliar fields (amount to transfer/deposit, depositperiod, beneficiary name, account number, IFSC code,etc.) • A4 - Based on users previous actions, there should bean option to set user/default favorites • A5 - Breadcrumbs should be available to keep users informed on the navigation • A6 - Labels of the fields should be simple to understand and unambiguous • A7 - The option for Select All or Delete All should be removed • A8 - The banking operation that started on a desktop/branch/ATM should continue over the mobile app without keying in data again • A9 - The mobile app registration should not require going to bank branch and can be loaded from authorized app stores. The registration should be free of any SMS charges • A10 - Sensitive information as Date of Birth, Customer Account number that is already known should not be asked • A11 - Related fields should be grouped together (for example, beneficiary, user account details, etc.). Also known fields should be pre-populated.

4.5 User Interface Display Parameters

The font size/type, display colours, controls size and labels can change the user perception of the app. Though each individual has their own liking for a colour, font and other display parameters, the implementation of suggested parameters can enrich user experience with respect to the interface of the mobile banking app. Following are the parameters for user interface display.

• B1 - Colour combination in foreground and background should be consistent across screens and contrasting without any gradient/progressing colours • B2 - The text information should be in mixed/sentence case instead of upper case • B3 - Avoid pagination, vertical scrolling and horizontal scrolling • B4 - Text in text boxes should be in single line and not spread across multiple lines • B5 - The measures for size of button, textbox and other controls should be relative to screen size instead of pixels • B6 - There should be bank logo, title page and frame on every screen • B7 - The white spacing between ﬁelds should be sufﬁcient to view labels without overlapping

20 • B8 - The language used should be simple and consistent with no long sentences and paragraphs in the screens.Having local languages based on user preference would be highly beneficial for users that are not comfortable with English • B9 - There should be clear character spacing avoiding any overlaps • B10 - Bold text should be used sparingly • B11 - The alignment of fields should be consistent (left for text fields and right for numbers) • B12 - There should be left navigation available for moving between menu options • B13 - There should not be any drag and drop based features • B14 - The image icons should be tested for varying resolutions (ldpi, mdpi, etc) and different operating systems • B15 - Apps should be built using HTML5 for consistent look

4.6 Error Handling

As the mobile phone screens are smaller in size than desktops and the user attention to detail on smaller screens is difﬁcult. It is important that apps are more thoroughly tested for various screens and device types and performance for various connectivity options. It is important that appropriate error messages are informed early rather later and thus reducing the call or other modes of escalations to banks operations team.

• C1 - System messages should be classified as Information (with text in Green/Blue color), Warn- ing(with text in Yellow color) and Error (with text in Red color) • C2 - If a particular mobile device is not supported, an error message should be displayed instead of allowing the user to install and then showing a error message • C3 - Error messages while filling a form should be displayed next to the fields and button • C4 - The message should provide the reason for error and suggests the next possible action • C5 - Application should maintain user action persistence and recovery from abrupt exits (network connection lost, battery down, memory shortage, etc) • C6 - System messages should be as configurable values rather hard coded for change at a later point of time • C7 - Language of the error message should be simple and easy to understand • C8 - Error messages while loading a page should be atthe top of the screen • C9 - The help icon should always be available and contextual to the screen • C10 - The app should be tested for varying network bandwidth, device models (make and screen size),flip/bump, back button and other buttons on the device, stylus, trackball/pad, swipe operations, screen rotation, Mobile Keys, Battery consumption and memory usage • C10 - The app should have an option for user to report the error • C11 - The version updates should be done on regular basis and ensured to keep past favorites intact

21 Based on the response to the recommendations, the banks should scale the implementation of usability standards using the spider diagram as shown in Figure 3. In the ﬁgure, the darker line represents a reference benchmark and grey line is the MAUI values of a banking app. MAUI value can be computed as:

MAUI = (Pc/Pn) ∗ scale

where Pc is the number of conformed parameters, Pn is the total number of parameters used for assessing usability. In our paper, for sake of simplicity, MAUI is measured on a scale of 1-10, 10 being the highest and 1 being the least. Since the goal is to provide a basis for standardizing usability rather than measure the relative importance of one parameter over another, each recommendation parameter is treated equally by assigning one point each (at times this may be context driven and relative weights may be assigned to each parameter) and measured against a scaling factor.

For example, if there are 15 parameters and about 11 parameters are met by the bank, the index can be calculated as (11/15) * 10 giving a value of 7.3. The score on time taken to complete the task, user interface display parameters and error handling are plotted on a spider diagram shown in Figure 4.1.

Figure 4.1 MAUI Parameters

The ﬁgure shows a benchmark/desirable score for each of the axis as 8.0. Authors believe that a threshold value of 8.0 provides a reasonable assurance that the mobile banking app provides good to very good user experience. Precise benchmark values and assessment agencies/organizations to assess the conformance can be established once the rate of adoption of mobile banking apps shows a steady increase. In addition, conformance of parameters can also be further broken down into multiple levels rather than the binary value of Yes or No shown in this paper.

22 4.7 Conclusions

MAUI guides Fintech apps to improve usability and thereby increase adoption rates. Financial in- stitutions could use MAUI for baselining the currently deployed app and increase the adoption with an improved MAUI and perform the cost beneﬁt analysis. By further reﬁning MAUI, we plan to build a semi-automated tool to measure MAUI. The accessibility requirements for different age groups and differently abled people would also be part of the future scope for further improving mobile banking apps adoption. Improving usability is a constant journey with changing customer experiences and technology innovation, hence, it is recommended for banks to review the usability requirements on a yearly basis monitoring the feedback on app stores and the customer queries being handled by bank operations team. The MAUI could be extended to apps that banks are planning to deploy for internal stakeholders and also for any organization building mobile apps for enterprise needs. Better usability of mobile apps improves customer loyalty and hence customer stickiness to the bank.

4.8 Observation

Our study on Fintech apps helped us understand the signiﬁcance of guideline-based usability evaluation methods is highly desired. This made us investigate the automated and non-automated means of usability evaluation for mobile apps. As part of the upcoming chapter, we explain our work on automating functional usability guidelines evaluation.

23 Chapter 5

UMETRIX - Architecting Usability Evaluation Framework for Mobile Applications

In this chapter, we brieﬂy describe the approach and architecture we employed for UMETRIX - Usability Evaluation Framework to identify and address usability issues using source code analysis. The remaining of this section reports the design goal, different components of the framework its inner- working and how natural text guidelines are converted to conformance rule.

5.1 Design Goal

The framework is designed with the main goal that it should fully automate the usability evaluation process by automatically analyzing the source code of the mobile app, automatically referencing usability guidelines, automatically check if the mobile app being analyzed violates any of the guidelines and should provide recommendations to the user on how to ﬁx the detected violations. The evaluation logic (i.e. the validation test cases) and the guidelines that are referenced should be separate so that guidelines can be modiﬁed without altering the evaluation logic. It should facilitate easy development of evaluation logic (i.e. conformance rules) for a guideline through a designer.

5.2 Architecture

We introduce UMETRIX 1 - a Usability Evaluation Framework to identify and address usability issues by analyzing usability code templates based on usability guidelines. The overall architecture of framework is depicted in Figure-5.1. A developer who builds mobile apps or a usability tester who validates a mobile app can make use of our instantiated framework. The key components of the framework are Mobile App Source Code, Validation Test Case Generator, Recommendation Engine and Validation Execution Engine.

1https://github.com/neerajmathur/UMETRIX

24 1.Source Code 2. Validation Test Case Generator 3. Recommendation Engine Code Validation Snippets DB Test Case APK Tool Android iOS Search Engine Library Windows dex2jar Recommendation jd-core Service If Condition For Loop APK Decompiler Switch Do While

Android Controls

iOS (iPhone)

Windows Validation Case DB

Validation Test Case Execution service Case(s) Guideline Evaluation Results Metric DB

Test Case Executor Reporting Service

4. Validation Execution Engine

Figure 5.1 Usability Evaluation Framework

1. Mobile App Source Code: An executable Mobile app file or the source code of the Mobile App are must for any type of code analysis. In case of executable mobile app file (eg. APK), the framework will execute the decompilation process over the file to get the source code. For decompiling Android (APK) file we can use open source tools such as, apk tool 2 it can decode resources to nearly original form, dex2jar 3 it is used to generate .jar file from a Dalvik executable (.dex) file, jd-core 4 is a library that reconstructs Java source code from one or more .class files. iOS app files are encrypted and not easy to decompile, therefore we need the actual source code for iOS platform.

2. Validation Test Case Generator: A validation test case is a test case built to check the conformance of a usability guideline. Validation test case consists of steps in XML format to detect the presence of a particular usability guideline by identifying necessary code snippets, as shown in listing-5.3. Usability guidelines can be considered as conformance rules here. Validation test case generator facilitates the creation of test cases through a tool as shown in ﬁgure-6.4, where a user can deﬁne what needs to be searched in a source code for analyzing the presence of a guideline. It consists of three key components:

2https://ibotpeaches.github.io/Apktool/ 3https://sourceforge.net/projects/dex2jar/ 4http://jd.benow.ca/

25 (a) Controls which provide ease for programmatic controls such as “IfConditions”, “ForLoop”, “Stwitch” and “DoWhile” etc. and custom controls developed speciﬁcally for mobile app source code analysis purpose. (b) Test Case Builder provide a drag and drop feature to build the sequential validation test steps by using controls. Using validation test case generator a non-programmer can build validation test case(s). (c) Arguments and Properties used to supply the variable input required at the evaluation execution time.

These test cases are stored in a database called “Validation Case DB”. The database contains a variety of usability guidelines that are indexed based on a domain and category of mobile software features. The developer should be able to choose and evaluate the required guidelines by executing these validation test cases.

3. Recommendation Engine: It provides code snippet suggestions for violated guidelines. The code snippets identiﬁed for a usability guideline while developing the validation test case will be added to a code snippet central library (as a live templates5 that can be imported into development IDE for ready to use code) by developer/tester along with guideline descriptions. Recommenda- tion engine uses the guideline descriptions to match the related code snippets for code suggestions during the evaluation process. During guideline evaluation process recommendation service is in- voked with guideline description and code snippets suggestions are sent back. These are stored in “Code Snippet DB”.

4. Validation Execution Engine: The validation case is loaded by “Test Case Executor” and invokes “Test Case Execution Service” where mobile app source code is passed along with the attributes and constraints in sequence. Every validation test case execution returns an individual result. These results are collated and are published in a database called “MetricDB” through a reporting service. These results are delivered to the external reporting system for visualization.

5.2.1 Incorporating guideline into validation test case

The validation test cases designed to evaluate usability guidelines can be stored and re-used for multiple mobile apps. At any given time, a guideline can be added, removed or modiﬁed for the overall evaluation. It is not required to rebuild the validation test case for a given guideline every time we perform the evaluation. This essentially implies that a non-programmer can add or remove the guidelines to perform the overall validation by just choosing the guideline. This also implies there can be multiple instances of the guidelines and corresponding validation test cases based on the various types of mobile apps. Below steps gives us an insight into how the guidelines are converted to a validation test case:

5https://www.jetbrains.com/help/idea/live-templates.html

26 The major challenges experienced when developing the framework was how to convert the guidelines into a structure that can be automatically referenced by the framework. The way this was achieved can be summarized in the following steps.

• Step 1: Select a usability guideline speciﬁed in plain text form. For example “User can select to reveal or hide password as they type, during signup or sign-in (e.g. by toggling a ‘reveal’ or ‘hide’ control)” is one of the important usability guidelines from Google’s Mobile App UX Principles [55].

• Step 2: Identify code snippet required to implement the guideline for example, if an EditText box of type password is wrapped inside the TextTemplate widget - it will automatically enable Show/Hide password button (an Eye icon) as shown in listing-5.1. This code snippet can be found in the Android SDK.

Listing 5.1 Reveal or Hide Password (Android)

Similarly, for iOS mobile app this can be achieved by setting “secureTextEntry” property to true/- false of a text input ﬁeld as shown in listing-5.2

Listing 5.2 Reveal or Hide Password (for iOS) if (! self .password.secureTextEntry) { self .password.secureTextEntry = TRUE; [hideShow setTitle :@”SHOW” forState :UIControlStateNormal ]; } e l s e { self .password.secureTextEntry = FALSE; [hideShow setTitle :@”HIDE” forState :UIControlStateNormal]; }

• Step 3: Identify steps which are required to be checked to validated presence of a guideline. In this case we have considered following steps to validate code snippet listing-5.1

1. Count total number of password ﬁeld which are present in source code 2. Count total password ﬁelds which are wrapped inside the TextInputLayout widget. 3. Check if there is a mismatch in counts

27 4. Report any ﬁeld which is not wrapped inside the TextInputLayout widget tag as a potential violation

Figure 5.2 Validation Test Case

• Step 4: Develop UI code snippet validation test case using validation test case generator Fig.6.4. Example of developed validation test case XML is shown in listing-5.3, which validates the steps identiﬁed in steps 3.

Listing 5.3 Validation Test Case XML

PwdBoxListWithShowHide . Count]”> True Validation Case Passed False Validation Case Failed

The validation test case XML files are used to perform usability evaluation. The validation test cases, once built, can be re-used based on the app chosen for evaluation. For example, in the above case, the validation test case XML file generated can be used for every Android mobile app which has a password text field in any given form.

5.3 Outcomes

We presented a usability evaluation framework. Though we have designed the framework for mobile apps, it can be easily extended for various testing requirement by developing custom control in validation test case generator. As an example, we have elaborated how natural text guideline can be converted to a formal format to check the conformance. In next chapter, we will develop a proof of concept based on the proposed framework.

29 Chapter 6

Proof of Concept for Android Mobile Apps

As a proof of concept, we have developed an instance of proposed framework to evaluate the usability guideline through code analysis for Android mobile apps [57]. The implementation consists of two key components - 1) Validation Test Case Generator and 2) Guideline Evaluator. Tool demo video is available at 1.

6.1 Validation Test Case Generator

We surveyed available open source workflow designer which should facilitate defining custom Tasks/Ac- tivities which is required for programming android specific code structure and should provide inbuilt feature for various control flow/conditional statement (example FOR LOOP, WHILE LOOP, IF ELSE, SWITCH) and exception handling, so that we do not have to write the entire engine from scratch. It should generate the output in a formal structure like XML/JSON, which can be easily integrated from our guideline evaluation logic. Following is the few lists of workflow engines we have studied and could be used to generate validation test case(s).

1. workflowengine.io 2 : Workflow Engine is a lightweight all-in-one component that enables you to add custom executable .NET and Java workflows of any complexity to any software, be it your own creation or a third-party solution, with minimal changes to existing code. But its free usage has very limited features.

2. jBPM 3 : jBpm is a flexible, extensible workflow management system. It is used to automate the process management system using its own developed language. However, we could not find a mechanism which can be used to create custom task/activites required for source code searching.

1https://www.youtube.com/watch?v=eeo0e1ajNnM 2https://workﬂowengine.io/ 3https://www.jbpm.org

30 3. Windows Workflow Foundation (WWF) 4 : The Windows Workflow Foundation (WWF) is a Microsoft technology that provides an API, an In-process workflow engine and a re-hostable designer to implement long-running processes as work-flows within .NET applications.

We discovered that Window Workflow Foundation (WWF) satisfies most of our requirements. We have chosen WWF over other technologies because it provides the rich functionality of existing controls and ability to add our own custom code activity/task, specific to mobile source code detection. Each work-flow consist of activities which are small actions to detect specific code template. Each workflow corresponds to a validation case. A validation case plan is created consisting of WWF validation cases. It provides a mechanism to extend and implement custom Activities and also provides User Interface to design workflows, just by drag and drop and providing arguments/constraints specific to a step/activity. This makes it extremely easy and useful to generate validation case for a usability code template.

6.2 Architecture

Figure 6.1 provides a high-level overview of validation test case generator architecture. In control toolbox we have inbuilt controls provided by WWF, we have added custom Android controls for search through different artifacts of Android source code. Details about custom android control developed for search through android source code is as follows:

• Read Layout Attributes: Provides a mechanism to search through layout XML using XPATH and XQUERY, It takes two arguments as Attribute ID to read from XML file and XPath which needs to be queried. For example, if we want to find all the EditText box in all the layout file and want to get their IDs as a result, then arguments will be XPATH=//EditText and Attribute- ToRead=android:id

• Read Manifest Attributes: Functionality is similar to Read Layout Attributes, but searching will be limited to on Manifest ﬁles

• Read Menu Attributes: Functionality is similar to Read Layout Attributes, but searching will be limited to on Menu ﬁles

• Search Java SRC Code: This control provides a mechanism to match the JAVA code tempalte passed to closely match to source ﬁles. After decompiling APK we will index all the java source ﬁles of an Android APK using Lucene.Net search engine. And use this index to match the required code template.

• Write Response: Provides a mechanism to write pass/fail response to the report.

4https://msdn.microsoft.com/en-us/library/ms734631(v=vs.90).aspx

31 Validation Test Builder

Custom Android Control Validation Test Case Read Layout Attributes Start Read Manifest Attributes

Read Menu Attributes

Search Java Src Code x

o Write Response b l o o T

l o r t n o Control Flow (In Built) C

If Condition For Loop

Switch Do While End

Error Handling (In Built)

Throw Try Catch Arguments Inputs

Figure 6.1 Architecture of Validation Test Case Generator

6.2.1 Process

Figure 6.2 shows flow diagram of Validation Test Case Generator and Figure 6.3 shows process to upload validation case to validation case DB, provides an overview of process flow diagram for generating validation test case. We start with a guideline in natural language form. Lets take a guideline User can select to reveal or hide password as they type, during signup or sign-in, as mentioned in section 5.2.1 Step 2 the code template to implement this guideline is to wrap EditText of type Password inside a android.support.design.widget.TextInputLayout element which is provided by Android SDK, this will enable an Eye Icon to reveal the password in clear text. The validation test case which is required to test this logic is to count how many password type field is present in the app and how many of them are wrapped inside the android.support.design.widget.TextInputLayout, if there is any discrepancy between the counts, then there exists a password type field which does not have Show/Hide password button. Below is the pseudo code for checking password fields which do not match the criteria.

Listing 6.1 Show Hide Password Field var PwdEditTextWithShowHideButton=//android.support.design.widget. TextInputLayout/ EditText[@android:inputType=textPassword]

32 UMETRIX – Validation Test Case Generation Code Snippet Validation Test Analysis Library Case

Take Usability Guideline in Natural Language

Convert UI Code Identify Pattern to UI Code Pattern Validation Case Workflow

Create Android Live Template (Code Snippet) for identified UI code pattern Code Snippet Format 1. Template Name ”uem_show_pwd” Generate 2. Description Validation Case (Should be the (XAML) Guideline Text) Comit to Github 3. Code Snippet code snippet repository

Figure 6.2 Flow Diagram of Validation Test Case Generator

var PwdEditText=// EditText[@android:inputType=textPassword] if (PwdEditText.Count>PwdEditTextWithShowHideButton . Count) return Passed; e l s e return Failed;

The identified code snippet will be added to code snippet repository as a Live Template. Which are then referenced by the recommendation engine to suggest code snippets for a violated guideline. The generated output file of a validation test case in XML form. Validation case can be saved in a .xaml file format so as to be re-used on future apps without re-building the entire validation case work-flow using the validation case designer. These .xaml validation case files can be stored in the Validation Case DB along with other usability guidelines to load/unload the rules on a new app for usability evaluation.

33 Load Update/Add Upload Assign Validation Validation Guideline Validation Case Case Plan to Validation Plan

Generate Validation Case .XAML file

Validation Test Case DB

Figure 6.3 Process to upload validation case to validation case DB

6.2.2 User Interface

Figure 6.4 Validation Test Case Generator displays a screen-shot of the validation test case generator, it is desktop utility which user case download and extends controls for custom needs. It consists of three main components.

• Toolbox Usability Controls: It list the controllers to create the Validation Case for an usability guideline

• Validation Case Workﬂow: This is a build area where the controllers are added by using drag and drop options. The Workﬂow of the validation case can be created here.

• Arguments and Properties: Input Variables and Constrains are deﬁned here as arguments to evaluate the validation case against the source code of mobile app.

Although we developed this tool for mobile app usability evaluation, but the architecture and approach we have used is ﬂexible enough to achieve numerous automatic testing requirements.

6.3 Guideline Evaluator

6.3.1 Architecture

Although we developed this tool for mobile app usability evaluation. However, the architecture and approach we have employed are ﬂexible enough to achieve numerous automatic testing requirements.

• Decompiler: Our approach is a code analysis which requires the source code. Therefore, the purpose of a decompiler is to extract the source code artifacts (Layout, JAVA, Manifest etc. ﬁles) from the software/App. The POC implementation is for Android Apps, therefore we used open source tools to decompile APK ﬁle. Following are the details of tools used.

34 Figure 6.4 Validation Test Case Generator Screen Shot

– APK Tool: A tool for reverse engineering, closed, binary Android apps. It can decode resources to nearly original form (including resources.arsc, classes.dex, and XMLs). – dex2jar : Converts Android Dalvik (.dex) Executable classes to JAR ﬁles. This step is required to extract JAVA source code. – Jd-core: Is a library that reconstructs Java source code from one or more .class ﬁles.

• Databases: We maintain three kinds of database

– Validation Test Cases: The validation case(s) developed by developer/testers are main- tained in this repository. – Validation Plan: Testing plan consist of usability guidelines and associated test cases. – Metric: Evaluation results and reports are stored in this DB for future use and analysis

• Inspection Engine: The purpose of inspection engine is to execute validation plan/test cases, generate report along with code suggestions. It consists of following components:

– Validation Plan Execution Service: Invokes .Net WWF execution service, load validation plan and test cases and executes them against the decompiled source code.

35 APK Decompiler Databases Code Snippet Engine

APK Tool dex2jar Code Snippet Library

Validation Validation jd-core Metric Lucene.Net Search Engine Plan Test Cases e

n Validation Plan Execution Service i

g Code Snippet n

E Recommendation Service

n o i t c e

p Reporting Service s n I

Figure 6.5 Architecture of Web Guideline Evaluator

– Code Snippet Recommendation Service: For each guideline, it connects to code snippet engine to get recommended code snippet for a guideline implementation. – Reporting Service: Generate a report for usability evaluation plan in a readable format and persists it to Metric DB

• Code Snippet Engine: Uses Lucene.Net search engine to index code snippets from a central GitHub repository and provides matching suggestions corresponding to a guideline.

6.3.2 Process

Figure 6.6 explains the high-level process ﬂow of guideline evaluator. User uploads the Android APK ﬁle in debug mode (release version are obfuscated on which decompiler will not work). Input basic information about App example category/domain it belongs to. Load usability validation plan according to the provided inputs, the system will load only relevant guideline; for example, implementing social media login functionality like Facebook, Twitter, Google will not be applicable to banking domain. Inspection service invokes validation plan execution service and code snippet recommendation service executes each validation test case and get code recommendation for failed guidelines.

6.3.3 User Interface

The user interface of guideline evaluator consists of following four steps to evaluate a Mobile App. Step 1 - Upload APK file: Figure 6.7 shows screen to upload APK file in debug mode. After successful upload, it will decompile the APK file to extract source code. Step 2 - Tell us about your App: Basic information about Mobile App will be asked, which includes App Name, Category, and Domain. This information is used to load the validation plan consisting of guidelines related to a specified category and domain. Figure 6.8 shows a screen-shot.

36 UMETRIX – Evaluate Android APK File Recommendation Code Snippet Guideline Evaluator Engine Library

released version will Open not get decompiled UMETRIX properly due to obfuscated code

Read Data Upload Android From Code Snippet Github code APK file Invoke Execution Repository snippet repository (debug version) Service

Create Index Decompile APK For Each Guideline using Lucene Search Engine

1. Template Name 2. Description Input APK basic Execute 3. Code Snippet information Validation Case (Category, Domain) Query Lucene for Get code Pass relevant code recommendation Guideline Text snippets for a Match Between guideline text Load Relevant Guideline Text and Guidelines/ Code Snippet Validation Plan Next Guideline Description

Generate Report With Code Recommendation

Figure 6.6 Guideline Evaluator Process Diagram

Step 3 - Review validation plan: User will review the validation plan consisting of loaded guidelines along with listed validation test case(s). User can select/deselect the guideline/case(s) which they intended to evaluate against the Mobile App source code. Here one can also add new guideline into the system along with validation test case(s). Figure 6.9 shows a screen-shot for this step.

Step 4 - Results: It will display the detailed results of guideline evaluation. List of evaluated guidelines along with the validation test cases is displayed. Statistics section will tell the conformance results of validation test case, failed match is the non-conformance/violation of a particular validation case and success match tells conformance towards the validation test case/guideline. Along with result we also display the code snippet suggestion on how to implement a particular guideline. Figure 6.10 shows the screen-shot for this step.

37 Figure 6.7 Upload APK ﬁle

Figure 6.8 Basic Information of APP

6.4 Summary

We have elaborated implementation of proof of concept for the proposed usability evaluation framework. We have examined various technologies for implementation and selected DotNet Framework and Windows Workﬂow Foundation (WWF) which suits the proposed framework requirement for validation test case generation and execution. In future, we will be working on implementing WWF using open source technologies to remove DotNet framework dependency. In next chapter, we will detail the evaluations performed to validate our proof of concept.

38 Figure 6.9 Validation Plan

39 Figure 6.10 Usability Results

40 Chapter 7

Evaluation of Framework

A total of three experiments were conducted to assess the effectiveness of proposed framework, and proof of concept tool is built to detect mobile app usability issues, and evaluation is conducted considering its ability to detect guideline violations, ease of use and implementation of the framework. The conducted experiments reported in the following sections:

7.1 Experiment 1: Accuracy and Robustness

AIM. To assess the accuracy and robustness of the framework for detecting usability guideline violations DATASET. Two mobile apps selected for this experiment, Froogal 1 is a mobile wallet developed by students of IIIT Hyderabad for paying bills within IIIT-Hyd Campus. The other app is VLab- which is a testing app developed to test usability guidelines. We incorporated all possible usability guidelines into it to do our validation. RESULTS. Table 7.1 shows the validation results of robustness of our tool when evaluated against these two apps and then manually veriﬁed the results by a usability expert. Most of the guideline results are correct except 4,5,6. In guideline 4,5 our tool failed to detect two failing cases of a search box due to the incorrect validation logic. In guideline 5 the validation case was not valid for the Froogal app, as there is no search ﬁlter available. By looking at the table 7.1, it is immediately observable that UMETRIX is able to correctly detect the majority of guideline violation.

7.2 Experiment 2: Evaluation with open source mobile apps

AIM. To assess the overall robustness and behaviour of the framework with a wide range of open source apps.

1https://www.froogal.in/

41 Table 7.1 Validation test results of Froogal and Vlab App Umetrix Manual # Guidelines Froogal Vlab Froogal Vlab M V M V M V M V First-time user has multiple sign-up options 1 (e.g.,username / password and social sign-up 1 2 3 - 1 2 3 - facebook.com, google, twitter) 2 Password field should have show/hide option - 1 1 - - 1 1 - Do not ask for too many inputs from user 3 24 2 8 - 24 2 8 - (Max five fields per Activity) If the search tool is inside the navigation menu, 4 1 - 2 - 1 2 2 - place it at the very top of that menu The length of the search box should be the 5 1 - 2 - 1 2 2 - largest possible size that will fit on the screen 6 User can easily filter search results - 1 1 - - - 1 - Form labels are either above form fields or are 7 - 1 - 1 - 1 - 1 floating labels (in Android) When app crashes it re-starts and returns the user to the 8 - 1 - 1 - 1 - 1 ,last screen used(to continue where they left off) Minimize scrolling in a drop-down box by using the whole 9 1 - 1 - 1 - 1 - screen to display long lists of options. Breadcrumbs should be available to keep users informed, 10 - 1 - 1 - 1 - 1 on the navigation Mobile app registration should not require going to 11 bank branch and can be loaded from authorized app 1 1 2 - 1 1 2 - stores. The registration should be free of any SMS Text in text boxes should be in single line and not spread 12 1 - - 1 1 - - 1 across multiple lines 13 The option for Select All or Delete All should be removed 1 - 1 - 1 - 1 - 14 Bold text should be used sparingly 1 - 1 - 1 - 1 - 15 There should not be any drag, and drop based features 1 - 1 - 1 - 1 - There should be left,navigation available for moving 16 1 - 1 - 1 - 1 - between menu options 17 Avoid pagination, vertical scrolling and horizontal scrolling - 1 - 1 - 1 - 1 Legends : M:- Matched , V:-Violated

DATASET. We study the robustness of the framework; we conducted a case study on Android mobile apps and captured results. Table 7.2 contains the list of 45 android mobile apps based on their business category and the app version used to perform the case study. We reached out to these Mobile app developers for the debug mode version of their Android mobile app and performed our study against our

42 UMETRIX tool. The Contents of the table 7.2 are deﬁned as LOC (Lines of Code), SLOC-P (Physical Executable Lines of Code), ‘M’ - Matched validation-cases per guideline, ‘V’ - Violated validation- cases per guideline and ‘%’ - Conformance Rate.

[Case Study Setup] - We reviewed available enterprise apps in Android PlayStore and F-Droid. We have chosen 45 mobile apps which belong different domains like Writing Apps, Reading Apps, Money or Payment Apps and other General daily purpose Apps for our study. We reached out to the developer groups of these apps, obtained the debug mode version of their Mobile App and decompiled it for source code analysis. We conducted a 2 phase study as deﬁned below.

• Phase 1 - We used the debug mode mobile APK ﬁle and executed it against our UMETRIX Usability Evaluation Tool with 17 usability guidelines and captured Results. • Phase 2 - We performed a Cognitive walk-through experiment in the Usability Lab at [Name removed for blind review] with the help of Usability Analysts to document the presence of these guidelines manually.

We have developed 25 validation test cases for 18 randomly selected usability guidelines from the classiﬁed guidelines for our case study (further details of developed validation test cases is available as part of supplementary material “validation-test-cases.pdf”), we are in the process of expanding to more guidelines. Each guideline consisted of a few use cases based on the code snippets deﬁned in the mobile app. We conducted similar evaluations and obtained results for all the chosen mobile apps. RESULTS. As per the Table 7.2 , we have results of 18 usability guidelines with the counts against matched (M) and violated (V) cases along with the conformance percentage.

• Phase 1 - Table 7.2 provides us the evaluation results of 18 usability guidelines with the counts against (Matched) and (Violated) validation test cases. List of 18 usability guidelines and detailed results (validation-results.pdf) are made available as part of the supplementary material. We could see that the tool was able to identify the use-cases based on code snippet for each guideline. It was able to provide us with the compliance of the respective usability guideline. • Phase 2 - Based on the above observations, we also conducted a Heuristic Cognitive Walk-through experiment using human subjects to examine these 18 usability guidelines (each app has multiple individual use-cases per guideline) on above 45 mobile apps. Here human participants were requested to evaluate the presence of the usability guideline manually. The results of this experiment were in line with tool results and provided us conﬁdence to proceed with a wide-spread study in Industry.

7.3 Experiment 3: Industry Evaluation and Results

AIM. To access the usefulness and effectiveness of using the framework from end user’s perspective.

43 Figure 7.1 Validation test case results of open source apps - Part 1

DATASET. We have conducted a Pre and Post UMETRIX usage survey consisting of eight questions having multiple choice and free-form format across ten bank(s)/corporate(s) to understand the end-user perspective of using the framework. The questions were designed to understand the usefulness of the framework. Following is the list of questions included in the survey. RESULTS.To understand the effectiveness of our framework, we have reached out to 25 premier Mobile App Companies based out in India to deploy and evaluate our UMETRIX framework. These vendors build and manage mobile apps for Banking, Travel and Payment Gateway domain. 16 vendors agreed to deploy our framework and have shared below observations as part of a Survey2. The intent of this survey is to understand their current usability evaluation practices and their experiences post usage of the framework. The consolidated results of survey questions are listed below:

Q1. Do you perform Usability Evaluation on your Mobile Apps based on Usability Guidelines? - 12 of 16 participants were not conducting usability evaluation based on usability guidelines. 4 of them were conducting usability evaluation based on usability guidelines.

Q2. What challenges did you face while performing overall usability evaluation on your mobile app? - Release deadlines, automation challenges, time consuming task, manual effort causing human errors and frequent changes in user requirements were top challenges. Lack of usability testing expertise and no clear usability strategy are few minor challenges.

Q3. What tools do you use to perform usability evaluation on your mobile app? - One ven-

2https://goo.gl/forms/KSZuHvL3nvOODQf92

44 Figure 7.2 Validation test case results of open source apps - Part 2

dor was using “Appsee” - mobile analytic app for usability evaluation, Two vendors were using mobile wire-frame validation tool. The rest of them were conducting manual usability experiments.

Q4. How would you rate the ease of use for UMETRIX WebUI based on your experience? - Here observations were captured based on factors like readability, understandability, deployment, use- case implementation and overall setup experience on a likert scale of very easy, easy, somewhat difﬁcult and difﬁcult. Overall 81% of the participants found that the tool was easy to setup.

Q5. How would you rate the ease of use for UMETRIX desktop validation case generator tool based on your experience? - Here observations were captured based on factors like readability, understandability, deployment, use-case implementation and overall setup experience on a Likert scale of very easy, easy, somewhat difﬁcult and difﬁcult. Overall 93% of the participants found that the tool setup was easy to use.

Q6. Do you want to suggest us any improvements or provide any recommendations? - UMETRIX WebUI improvements, cloud-based support for storing metric results, usability guidelines and validation use-cases, analytics on historical use-case results and recommendation to provide a tool support in open-source were few suggestions and recommendations from the participants.

Q7. What are the additional automation strategies you would like to suggest to the authors? - Event watching, event capture, screen capture and user-key store analysis were few of the automation strategies suggested to us. Static source code analysis with a feature to encrypt and decrypt the source

45 Table 7.2 Mobile Apps used for UMETRIX Case Study Results Results App Name Version LOC SLOC-P App Name Version LOC SLOC-P M V % M V % Money Writing My Expense 2.7.9 261626 224685 24 9 72.73 Notes 1.0.1 10338 9751 15 9 62.50 Lunary 1.28 444253 391765 17 8 68.00 To-Do List 1.0.1 5711 5349 12 11 52.17 Budget Watch 0.13 99156 85204 11 10 52.38 Omni Notes 5.3.2 117868 101868 13 12 52.00 Loyalty Card 0.11 138497 120541 11 10 52.38 Memento 1.1.0 12788 11831 16 9 64.00 SmartcoinsWallet 1.0.6 369971 319496 15 12 55.56 Cloud Notes 0.11.0 59038 49326 10 10 50.00 TREZORManager 1.0.5 155902 118819 15 8 65.22 Snotepad 1.2.1 6004 5712 11 11 50.00 WALLETH 0.16 175603 154777 11 9 55.00 SyncOrg 2.0.5.3 130466 100538 14 12 53.85 Bitcoin 1.074 251874 202688 14 10 58.33 Kolab Notes 2.7.2 97644 68935 11 13 45.83 Rental Calc 0.4 137121 117961 11 12 47.83 JustNotes 1.9 3112 2944 10 10 50.00 Gift CardGuard 0.4 148842 130847 11 9 55.00 NoteBuddy 1.2.2 2950 2582 13 9 59.09 General Reading Frrogal 3.1.0 412799 325970 34 11 75.56 Mi Manga Nu 1.65 77839 68389 16 10 61.54 Vlab 0.4 262076 215687 25 4 86.21 spaRSS 1.11.8 21548 16733 13 9 59.09 Wikipedia 2.5.195 236951 202568 9 12 42.86 FBReader 2.1 70522 56746 14 9 60.87 MyExpense 2.7.9 236951 202568 24 9 72.73 Anecdote 1.1.5 90553 77130 10 11 47.62 BlitzMail 0.6.1 51463 36013 9 11 45.00 Prayer Times 3.6.3 262209 226981 12 15 44.44 aMetro 2.0.1.5 104815 93839 10 13 43.48 OpenManga 3.2 60480 51211 11 10 52.38 DeltaChat 0.9.2 154207 134179 10 9 52.63 SurvivalManual 3.9 81195 73087 9 10 47.37 YaXim 0.9.2 88771 75527 17 9 65.38 DocumentViewer 2.7.9 96845 81764 12 13 48.00 SlimTwitter 3.2 936 852 11 7 61.11 Quoter 1.0.1 3638 3170 12 7 63.16 SmartCoinWallet 1.0.6 369971 319496 15 12 55.56 Library 5.1 4538 4214 11 8 57.89 Beem 0.1.7 96544 77736 14 12 53.85 MuPDF 1.10 5725 5070 11 9 55.00 FaceSlim 3.1.0 21832 17325 11 7 61.11 ------Telegram 3.18.0 503106 422183 10 9 52.63 ------PrePay 0.17 44798 36211 11 9 55.00 ------code was also suggested. This was to avoid exposing the undesired source code developers who were not part of a particular module of the mobile app development. This requirement was intended to secure the source code logic across developers.

Q8. Does UMETRIX based usability evaluation approach improves your mobile app usability? - 13 out of 16 participants found that the UMETRIX framework helped them address the usability evaluation requirements. They continue to practice this approach over their future apps to addressed usability issues. 3 participants were skeptical about the implications of code safety as they build mobile apps which are secure and they don’t want to expose their source code. It was suggested to us to encrypt the entire source code during the usability evaluation and decrypt only the required usability patterns for analysis.

46 7.4 Threats to Validity

This section describes the threats to the validity of our work: In our present work, we studied Nielsen Norman and Google Android UX Principles usability guidelines for classification. We assume guidelines from other vendors can also be classified into Functional and Non-Functional categories along with the level of automation. To classify guidelines we relied on the knowledge of volunteers who had a good amount of experience in developing the mobile application. To mitigate this threat we have asked volunteered to implement the guideline in a model app, to make sure that code snippet used can be converted to a validation test case and possible to detect through code analysis. We relied on software forums like StackOverflow, GitHub, ExpertsExchange, Android Developer Central to get the Android code snippets related to usability implementation. We received assistance from the Android developer community to validate the identified code snippets to make them useful as part of recommendation engine. One of the critical components of our proposed framework is “Validation Test Case Generator”, this is required to convert guideline in the natural text to a conformance rule. Our proposed approach is through a designer which contains controls to validate the presence of a particular small part of code snippets which is essential to implement the guideline. However, one can argue that there might a more feasible approach to achieve the same. [External Validity] In our work to realize the architecture we are dependent on Microsoft.Net framework and Windows Work-flow Foundation which matches to our Validation Test Case Generator design goal. This dependency can be easily mitigated once we develop a similar framework using open source tools which can provide programming control. For preliminary evaluation purpose, we have developed 20 validation cases for 18 usability guidelines. We are planning to develop an exhaustive list of validation cases for guidelines. However, this requires a substantial effort. So, crowd sourcing the development of validation cases is a possible option. [Internal Validity] As part of our current work we used Android UX Principles and Nielsen Norman Usability Guidelines to construct the tool. The developers can create their desired test cases using validation case generator and can use the Web Guideline evaluator to evaluate the mobile app. The current scope of our tool is limited to Android mobile apps. We are relying on Software forums like StackOverflow, GitHub, ExpertsExchange, Android Developer Central to get the android usability-code patterns. We received assistance from the Android Developer Community for documenting usability code-patterns so as to make them useful as part of recommendation engine.

7.5 Observations

From the various experiments performed we can say that our framework provides a ﬂexible mechanism to build validation case(s) and ability to evaluate the usability of a mobile app based on the func-

47 tional usability guidelines using these validation case(s). However, at present tool lacks an extensive list of existing validation case(s), which can be readily used by testers/developer. Creating validation cases is planned for the future scope of the work.

48 Chapter 8

Conclusion and Future Work

The following section concludes and summarizes the results from the research work presented and the contributions made to the existing knowledge base related to the ﬁeld of usability evaluation. With the availability of high range computing devices in the market, people started using mobile apps at an aggressive rate. This created a competition among mobile app manufactures to deliver similar products at competitive rates. Usability is one of the critical factors of improving the usefulness of your mobile app. Thus usability evaluation has become inevitable for mobile software tester before releasing their mobile app into the market. Many researchers have proposed different approaches and tools to address usability issues in the mobile app. These tools perform usability evaluation based on session recording, event watching, user feedback. Most of these tools are the best ﬁt for either black/grey box testing or manual testing over a post-release version of a mobile app. However, the existing tools may not be able to study the usability while the mobile app is under development.

8.1 Contributions

First - We studied the low adoption rate of Fintech mobile apps in the context of the Indian banking sector. We focus on usability issues related to mobile banking apps, proposed usability guidelines and Mobile App Usability Index (MAUI) metric for improving usability based on various usability parameters. This study helped us understand the need for a robust functional usability evaluation tool. Second - As part of our research, we proposed a usability evaluation framework to analyse the usability code-templates using usability guidelines to identify the usability issues. This helps mobile software developers and the mobile software testers to automatically recognise usability conformance issues before releasing the ﬁnal product into the market. It helps them to increase the usability of their mobile app. Third - We created a tool based on the proposed usability evaluation framework. We validated the proposed work-ﬂow and executed our approach against various real-time Android apps to review the usable aspects of the tool.

49 Fourth - We have validated the usefulness of our tool against 45 open source mobile apps. We have performed an industrial case study to understand the ease of implementation of the tools.

8.2 Limitations

As recommended by various researchers the purpose of any tool such as the one being proposed is to assist human evaluators. Ivory and Chevalier advise that such tools should always be used with caution and one should never entirely rely on their results alone. This is because, with current technology, it is difﬁcult to develop a tool that can behave like a human and exhibit human attributes such as common sense Another limitation is the difﬁculty encountered with incorporating certain guidelines into the validation test case, mainly because of their abstract nature.

8.3 Future Work

As part of our future work, we would like to increase the scope of our framework and tool as listed below:

• To build an extensive list of validation test cases for various domains and guidelines

• To extend our current prototype tool to include .ipa ﬁles of iOS and .xap/.appx ﬁles of Windows and release it as a desktop-based tool for developers. This covers almost all the varieties of mobile apps available in existing different mobile devices

• To create a domain-based guideline validation system for domain speciﬁc apps like banking, e- commerce, construction etc.

• To create a robust reporting system for this tool so as visualize the usability results in a better way for both desktop version and for the Web version

• To create a provision to integrate this tool with a Mobile Integrated Development Environment (IDE) for dynamic usability code pattern analysis based on chosen usability guidelines

• To remove dependency of .Net and WWF, create a workﬂow execution engine using open source technologies.

• To incorporate MAUI index in results, to facilitate index we need to ﬁrst have a robust list of validation test cases.

With these conclusions, I end the thesis here. Thank you!

50 Related Publications

1. Neeraj Mathur; Sai Anirudh Karre, Y. Raghu Reddy, “Usability Evaluation Framework of Mobile Applications using Code Analysis”, The 22nd International Conference on Evaluation and Assessment in Software Engineering (EASE) , June 2018 [Published]

2. Neeraj Mathur; Sai Anirudh Karre, Lalit Mohan, Y. Raghu Reddy, “Analysis of FinTech Mo- bile App Usability for Geriatric Users in India”, CHIuXiD 2018 4th International HCI and UX Conference, Yogyakarta, Indonesia, March 2018 [Published]

3. Lalit Mohan S; Neeraj Mathur; Y. Raghu Reddy, “Mobile App Usability Index (MAUI) for Improving Mobile Banking Adoption”, 10th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE), Barcelona, April 2015, pp:313-320 [Published]

4. Lalit Mohan S; Neeraj Mathur; Y. Raghu Reddy, “Improving Mobile Banking Usability Based on Sentimentss”, In proceedings of Communications in Computer and Information Science Jour- nal, Springer-Verlag Press, Vol: 599, Feb 2016, pp:180-194 [Published]

51 Bibliography

[1] H. M. Az-zahra, A. Pinandito and H. Tolle, “Usability evaluation of mobile application in culinary recommendation system,” 2015 IEEE Asia Paciﬁc Conference on Wireless and Mobile (APWiMob), Bandung, 2015, pp. 89-94

[2] N. Jailani, Z. Abdullah, M. A. Bakar and H. R. Haron, “Usability guidelines for developing mobile application in the construction industry,” 2015 International Conference on Electrical Engineering and Informatics (ICEEI), Denpasar, 2015, pp. 411-416

[3] H. Luna, R. Mendoza, M. Vargas, J. Munoz, F. J. Alvarez and L. C. Rodriguez, “Using Design Patterns as Usability Heuristics for Mobile Groupware Systems,” in IEEE Latin America Transactions, vol. 13, no. 12, pp. 4004-4010, Dec. 2015

[4] L. Mohan, N. Mathur and Y. R. Reddy, “Mobile App Usability Index (MAUI) for improving mobile banking adoption,” 2015 International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE), Barcelona, Spain, 2015, pp. 313-320

[5] J. Zbick, I. Nake, M. Milrad and M. Jansen, “A Web-Based Framework to Design and Deploy Mobile Learning Activities: Evaluating Its usability, Learnability and Acceptance,” 2015 IEEE 15th International Conference on Advanced Learning Technologies, Hualien, 2015, pp. 88-92

[6] M. Borys and M. Milosz, “Mobile application usability testing in quasi-real conditions,” 2015 8th International Conference on Human System Interaction (HSI), Warsaw, 2015, pp. 381-387

[7] M. Wich and T. Kramer, “Enhanced Human-Computer Interaction for Business Applications on Mobile Devices: A Design-Oriented Development of an Usability Evaluation Questionnaire,” 2015 48th Hawaii International Conference on System Sciences, Kauai, HI, 2015, pp. 472-481

[8] A. Hussain, H. I. Abubakar and N. B. Hashim, “Evaluating mobile banking application: Usability dimensions and measurements,” Proceedings of the 6th International Conference on Information Technology and Multimedia, Putrajaya, 2014, pp. 136-140

[9] M. Al-Ismail and A. S. M. Sajeev, “Usability challenges in mobile web,” 2014 IEEE International Conference on Communication, Networks and Satellite (COMNETSAT), Jakarta, 2014, pp. 50-55

52 [10] S. Park, J. Byeon, M. Whang and Y. B. Park, “Bad Symptoms for Usability of Mobile Applications,” 2014 International Conference on IT Convergence and Security (ICITCS), Beijing, 2014, pp. 1-4

[11] J. M. Lopez Gil, M. Urretavizcaya Loinaz, B. Losada and I. Fernandez Castro, “Field vs. Laboratory Usability Evaluations: a Study on a Context Dependent Mobile Application Developed with an Agile Methodology,” in IEEE Latin America Transactions, vol. 14, no. 1, pp. 339-348, Jan. 2016

[12] H. Rahmat, H. Zulzalil, A. A. Abd Ghani and A. Kamaruddin, “An approach towards development of evaluation framework for usability of smartphone applications,” 2015 9th Malaysian Software Engineering Conference (MySEC), Kuala Lumpur, 2015, pp. 178-182

[13] N. Z. b. Ayob, A. R. C. Hussin and H. M. Dahlan, “Three Layers Design Guideline for Mobile Application,” 2009 International Conference on Information Management and Engineering, Kuala Lumpur, 2009, pp. 427-431

[14] L. R. Kascak, C. B. Rbola and J. A. Sanford, “Integrating Universal Design (UD) Principles and Mobile Design Guidelines to Improve Design of Mobile Health Applications for Older Adults,” 2014 IEEE International Conference on Healthcare Informatics, Verona, 2014, pp. 343-348

[15] B. Cassidy, G. Stringer and M. H. Yap, “Mobile Framework for Cognitive Assessment: Trail Making Test and Reaction Time Test,” 2014 IEEE International Conference on Computer and Information Technology, Xi’an, 2014, pp. 700-705

[16] F. Nayebi, J. M. Desharnais and A. Abran, “An Expert-Based Framework for Evaluating iOS Application Usability,” 2013 Joint Conference of the 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement, Ankara, 2013, pp. 147-155

[17] Kortum, P., Sorber, M.,“Measuring the usability of mobile applications for phones and tablets,” 2015, International Journal Human Computer Interactions, 31, 518-529

[18] Rajinesh R, Ian M, Michael D, “Usability Evaluation of a Tag-Based Interface”, Journal of Usability Studies, Volume 7, Issue 4, pp. 143-160

[19] Johanna S, Marlene V, Sari K, “Experiencing Visual Usability and Aesthetics in Two Mobile Application Contexts”, Journal of Usability Studies, Volume 10, Issue 1, pp. 46-62

[20] X. Su, O. Ozgobek, J. A. Gulla, J. E. Ingvaldsen and A. D. Fidjestol, “Interactive mobile news recommender system: A preliminary study of usability factors,” 2016 11th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP), Thessaloniki, 2016, pp. 71-76

53 [21] S. K. Divyaa, N. Yadav and S. K. Dubey, “Usability evaluation of mobile phones by using AHP-entropy approach,” 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, 2016, pp. 4063-4067

[22] R. Baharuddin, D. Singh, R. Razali, “Usability Dimensions for Mobile Applications: A Review”, Research Journal of Applied Sciences, vol. 5, no. 6, February 2013, pp. 2225-2231

[23] C. K. Coursaris, D. J. Kim, “A meta-analytical review of empirical mobile usability studies”, Journal of Usability Studies, vol. 6, no. 3, pp. 117-171, 2011

[24] A. Hussain, M Kutar, “Usability Metric Framework for Mobile Phone Application”, PGNet, 2009, ISBN 978-971

[25] J. Nielsen, “Heuristic evaluation”, Journal Usability inspection methods, vol. 24, pp. 413, 1994

[26] H. B.-L. Duh, G. C. Tan, V. H.-h. Chen, “Usability evaluation for mobile device: a comparison of laboratory and ﬁeld tests”, Proceedings of the 8th conference on Human-computer interaction with mobile devices and services. ACM, pp. 181-186, 2006

[27] F. F.-H. Nah, K. Siau, H. Sheng, “The value of mobile applications”, Communications of the ACM, vol. 48, no. 2, pp. 85-90, Feb. 2005

[28] C. Cuadrat Seix, “Towards the Validation of a Method for Quantitative Mobile Usability Testing Based on Desktop Eyetracking. Doctoral Thesis”, Valencia: Universitat de LLeida, 2012

[29] Harrison et al., “Usability of mobile applications: literature review and rationale for a new usability model”. Journal of Interaction Science, 2013, doi:10.1186/2194-0827-1-1

[30] Georg Apitz, Franois Guimbretire, and Shumin Zhai. 2008. “Foundations for designing and evaluating user interfaces based on the crossing paradigm.” ACM Trans. Comput.-Hum. Interact. 17, 2, Article 9 (May 2008), 42 pages

[31] Fredrik Bostrm, Petteri Nurmi, Patrik Floren, Tianyan Liu, Tiina-Kaisa Oikarinen, Akos Vetek, and Pter Boda. 2008. “Capricorn - an intelligent user interface for mobile widgets”. In Proceedings of the 10th international conference on Human computer interaction with mobile devices and services (MobileHCI ’08). ACM, New York, NY, USA, 327-330.

[32] Conn, B. (2014). “The evolution of Mobile Marketing in India: Current Trends and Best Practices”. Mobile Marketing Summit report.

[33] Developer.apple.com, (2017). iOS Human Interface Guidelines: Designing for iOS. [online] Available at: https://developer.apple.com/library/ios/documentation/userexperience/conceptual/mobilehig/ [Accessed: 21 July. 2017].

54 [34] Developer.android.com, (2017). Mobile App Design from Android. [online] Available at: https://developer.android.com/design/material/index.html [Accessed: 21 July. 2017].

[35] Docs.google.com, (2017). Google report on the Mobile App Usability survey. [online] Available at: https://goo.gl/L8u6DK [Accessed: 27 July. 2017].

[36] “Ergonomics of Human System Interaction Guidance on World Wide User Interfaces”. (2011). ISO 9241-151: ISO/TC 159/SC.

[37] P. M. Fitts, “The Information Capacity of the Human Motor System in Controlling the Amplitude of Movement”, Journal of Experimental Psychology, Vol. 47, No. 6, 1954, pp. 381-391

[38] Harrison, R., Flood, D. and Duce, D. (2013). “Usability of mobile applications: literature review and rationale for a new usability model”. J Interact Sci, 1(1), p.1

[39] Miller, G. (1965). The magical number seven, plus or minus two.Psychological Review 63 (2): 81-97, 1956.

[40] Msdn.microsoft.com, (2014).“ Usability Guidelines”. [online] Available at: http://msdn.microsoft.com/en-us/library/bb158578.aspx [Accessed: 27 July. 2017].

[41] Newell, A., and Rosenbloom, P. S. (1993). “Mechanisms of skill acquisition and the law of practice. In J. R. Anderson (Ed.)”, Cognitive skills and their acquisition (pp. 1-55). Hillsdale, NJ: Erlbaum.

[42] Nielsen Norman Group. Mobile Website and Application Usability — Nielsen Norman Group Report.

[43] R. Khan, H. (2012). “Customizing mobile banking in India: issues & challenges”. Conference on-Sustainable excellence through customer engagement, employee engagement and right use of technology.FICCI-IBA (FIBAC) 2012

[44] RBI, (October 2014) “List of Banks permitted to provide Mobile Banking Service in India - Report from Reserve Bank of India (2014)” [Online] available at: http://www.rbi.org.in/scripts/bs viewcontent.aspx?Id=2463 [Accessed: 27 July. 2017].

[45] Banks wise volumes in ECS/NEFT/RTGS/Mobile Transaction - Report from Reserve Bank of India (2014) [Online] Available at: http://www.rbi.org.in/scripts/NEFTView.aspx [Accessed: 27 July. 2017].

[46] Rosati, L (2013). How to design interfaces for choice: Hick- Hyman law and classiﬁcation for information architecture. In Slavic, A.; Salah, A.; Davies, C. Classiﬁcation and visualization: interfaces to knowledge: proceedings of the International UDC Seminar. The Hague, The Netherlands. pp. 125138,

55 [47] Seongil, L. (2009). “Mobile Internet Services from Consumers Perspectives”. International Journal of Human- Computer Interaction, 25(5).

[48] Statista, (2014). “India: mobile phone internet user penetration” 2012-2018 — Statistic. [online] Available at: http://www.statista.com/statistics/309019/india-mobile-phone-internet-user-penetration/ [Accessed: 27 July. 2017].

[49] Statista, (2017). “Worldwide mobile app revenues in 2015, 2016 and 2020 (in billion U.S. dollars)” — Statistic [online] Available at: https://www.statista.com/statistics/269025/worldwide-mobile-app-revenue-forecast/ [Accessed: 27 July. 2017]

[50] CAPITA, (2017). “Why do people uninstall apps? [infographic]” — [online] Available at: https://www.capitatranslationinterpreting.com/why-do-people-uninstall-apps/ [Accessed: 27 July. 2017]

[51] Jakob Nielsen, Raluca Budiu, “Mobile Usability”, New Riders Publication, 2013, ISBN-13:978-0-321-88448-0, ISBN-10: 0-321-88448-5

[52] ISO 9241-11, Ergonomic requirements for ofﬁce work with visual display terminals (VDTs) Part 11: Guidance on usability (1998), International Organization for Standardization, Geneva, Switzerland

[53] ISO/IEC 18021, Information technology – User interfaces for mobile tools for management of database communications in a client-server model (2002), International Organization for Standardization, Geneva, Switzerland

[54] Jakob Nielsen, Raluca Budiu, “User Experience for Mobile Applications and Websites - Design Gudielines for Improving the Usability of Mobile Sites and Apps”, Technical Report, 3rd Edition.

[55] Stephen Grifﬁchs, “Mobile App UX Principles - Improving User Experience and Optimizing Conversion”, Google Technical Report, April 2015.

[56] Beirekdar, Abdo and Vanderdonckt, Jean and Noirhomme-Fraiture, Monique , “A Framework and a Language for Usability Automatic Evaluation of Web Sites by Static Analysis of HTML Source Code”, Springer Netherlands, pages=337–348

[57] Neeraj Mathur, Sai Anirudh Karre, “UMETRIX - GitHub Repository”, — [online] Available at: https://github.com/neerajmathur/UMETRIX

[58] Statista 2017 (Mobile Internet and App), “Number of apps available in leading app stores as of March” 2017, — [online] Available at:

56 https://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/ [Accessed: 27 July. 2017]

[59] Beirekdar, A., Vanderdonckt, J. Noirhomme-Fraiture, M., 2002. “A framework and a language for usability automatic evaluation of web sites by static analysis of HTML source code”, In 4th International Conference on Computer-Aided Design of User Interfaces., 2002.

[60] Tobar, L.M., Latorre Andrs, P.M. Lafuente Lapena, E., 2008. “WebA: a tool for the assistance in design and evaluation of websites”, Journal of Universal Computer Science, pp.1496-512.

[61] Beirekdar, A., Vanderdonckt, J. Noirhomme-Fraiture, M., 2002. “KWARESMI knowledge-based web automated evaluation tool with reconfigurable guidelines optimization”, In Proc. 9th International Workshop on Design, Specification, and Verification of Interactive Systems DSV-IS., 2002.

[62] Beirekdar, A. et al., 2005. “Flexible reporting for automated usability and accessibility evaluation of web sites”, INTERACT, pp.281-94.

[63] Borges, J.A., Morales, I. Rodrguez, N.J., 1996. “Guidelines for designing usable world wide web pages”, In Proc. Conference on Human Factors in Computing Systems: common ground. Vancouver, British Columbia, Canada, 1996.

[64] Burmester, M. & Machate, J., 2003. “Creative design of interactive products and use of usability guidelines a contradiction? In Jacko, J., Stephanidis, C. & Harris, D. Human-computer interaction: theory and practice”, Mahwah, NJ, United States: Lawrence Erlbaum Associates Inc. pp.43-46.

[65] Montero, F., Gonzles, P., Lozano, M. Vanderdonckt, J., 2005. “Quality models for automated evaluation of web sites usability and accessibility”, In International COST294 Workshop on User Interface Quality Model. Rome, Italy, 2005.

[66] Vanderdonckt, J., Beirekdar, A. & Noirhomme-Fraiture, M., 2004. “Automated evaluation of web usability and accessibility by guideline review”, Lecture Notes in Computer Science, pp.17-30.

[67] Ivory, M.Y. Hearst, M.A., 2001. “The state of the art in automating usability evaluation of user interface”, ACM Computing Surveys (CSUR), December. pp.470-516.

[68] Oliver Hummel, Stefan Burger, 2017 “Analyzing source code for automated design pattern recommendation”, Proceedings of the 3rd ACM SIGSOFT International Workshop on Software Analytics, Paderborn, Germany, pp.8-14

[69] Louridas, Panagiotis, July 2006, Static Code Analysis, IEEE Computer Society Press,Los Alamitos, CA, USA, pp.58–61

57 [70] Frank Buschmann, Regine Meunier, Hans Rohnert, Peter Sommerlad and Michael Stal, “Pattern-Oriented Software Architecture: A System of Patterns”, Volume1, Wiley publications, 2007.

[71] Allamanis, Miltiadis and Sutton, Charles, “Mining Idioms from Source Code”, Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, New York, NY, USA, pp.472–

[72] Justin Mifsud, Alexiei Dingli, “USEFul: A Framework to Mainstream Web Site Usability through Automated Evaluation”, International Journal of Human Computer Interaction (IJHCI), Volume (2) : Issue (1) : 2011, pp. 10–30

[73] Pre UMETRIX Usage Survey Results, https://goo.gl/whS1et, 14-Nov-2017.

[74] Chi Ed H. and Adam Rosien and Gesara Supattanasiri and Amanda Williams and Christiaan Royer and Celia Chow and Erica Robles and Brinda Dalal and Julie Chen and Steve Cousins, The bloodhound project: Automating discovery of web usability issues using the InfoScent simulator, 2003, pp.502–512

[75] Justin Mifsud and Alexiei Dingli, USEFul: A Framework to Mainstream Web Site Usability through Automated Evaluation, International Journal of Recent Trends in Human Computer Interaction (IJHCI)

——————————————————–

58 Appendix A

Usability Guideline Classiﬁcation along with Code Snippets

This section contains lists of guidelines from NN/g Nielsen Norman Group and Google Mobile App UX Principles. We have classiﬁed following 76 guidelines into functional and automatable category along with code templates which can be used to detect the presence of a guideline. The guidelines which are functional and automatable are marked with ﬂag “Yes” otherwise “No”. Source of guideline documents

1. Nielsen Norman Group (NN): UX for Mobile Applications and Websites 3rd edition 2. Google Android (G): Mobile App UX Principles

1. G#5 If onboarding is used, the user can choose to skip it (put the user in control)

Description: Users should be able to skip the onboarding sequence if they wish not to be inter- rupted by it or do not perceive a beneﬁt in swiping through it. Put the user in control of their time and initial experience of the app. Onboarding should only ever appear on ﬁrst app use, or until user is sign in.‘ Automatable: Yes IsFunctional: Yes Code Template: Refer Guideline# 70

2. G#21 If app requests sign-up, the user can choose to continue as a guest (the user is in control) Description: Users should be able to choose to ignore registration and continue as a guest, where they can sample and assess its value to them, before committing to register. This is even more important for apps from unknown brands, or apps that provide engaging content and which are dependent on users experiencing them before they can make a registration decision. Always put the user in control of their time and initial experience of the app. Automatable: Yes IsFunctional: Yes

59 Code Template: //In signup page, check presence of below code: //where MyPREFERENCES may be any name, perhaps the input parameter in test case sharedpreferences = getSharedPreferences(MyPREFERENCES, Context.MODE PRIVATE); SharedPreferences.Editor editor = sharedpreferences.edit(); editor.putString(“”, “”) //here we can use many ﬁelds editor.commit(); //the above steps are used to store the data about user //By using SharedPreferences data we can tell user is login are not

3. G#22 If app sign-up is mandatory, the user can choose to demo it ﬁrst (the user is in control) Description: Users should be able to choose to ignore registration and continue as a guest, where they can sample and assess its value to them, before committing to register. This is even more important for apps from unknown brands, or apps that provide engaging content and which are dependent on users experiencing them before they can make a registration decision. Always put the user in control of their time and initial experience of the app. Automatable: Yes IsFunctional: Yes

Code Template: //Get all the java classes //Check for sharedpreferences = getSharedPreferences(MyPREFERENCES, Context.MODE PRIVATE); //Here if the user signups the sharedpreferrences will get the signup details and uses for future references. //If the sharedpreferres are empty then the user is using the demo version of the app. //Check for homepage.java and onclicklistener for guest id in the model app for reference code. //Note: We cannot show the directly that the user is signed or not seeing the code. By tracing and getting the details of sharedpreferences i.e whether it is empty or not speciﬁes the signed in response.

4. G#24 NN#183 User can select to reveal or hide password as they type, during signup or sign-in Automatable: Yes IsFunctional: Yes

60 Code Template: for Android //Check if EditText is wrapped with the TextInputLayout

5. G#23 First-time user has multiple sign-up options (e.g. username / password and social signup facebook.com, google, twitter) Automatable: Yes IsFunctional: Yes Code Template: for Android //Check if XML contains following tags or or

6. G#25 Return-user is persistently signed-in to their app Automatable: Yes IsFunctional: Yes Code Template: Refer guideline# 3

7. NN#227 Most mobile websites and apps with a signiﬁcant amount of content beneﬁt from having a search tool on the homepage. Automatable: No IsFunctional: No

8. NN#228 The search functionality should be easy to locate. Automatable: No IsFunctional: No

9. NN#229 If search is frequently used on your site, make it prominent by using a visible search box placed at the top of the screen. Automatable: No IsFunctional: No

10. NN#230 If search is only occasionally used on your site, use a magniﬁer tool for the search box and place that close to the top of the page.

61 Automatable: No IsFunctional: No

11. NN#231 If the search tool is inside the navigation menu, place it at the very top of that menu Description: Most people are used to search being in a prominent place such as the top of the screen; if they don’t ﬁnd it quickly, they may assume it does not exist on that site or in that app. Automatable: Yes IsFunctional: Yes Code Template //Search all XML in menu folder //Check If any xml ﬁle contain following tags app:actionViewClass=”android.support.v7.widget.SearchView” app:showAsAction=”always” //(Example in shown in the model app : menu/menu search.xml) or //Find all java classes //If any java class contains this listener searchView.setOnQueryTextListener(new SearchView.OnQueryTextListener() //Where searchView is id

12. NN#232 If no search box is visible, use a standard magniﬁer- glass icon for search Automatable: No IsFunctional: No

13. NN#233 The length of the search box should be the largest possible size that will fit on the screen. Description: This guideline replicates the more general one (see guideline 60) for resizing textboxes in forms. Automatable: Yes IsFunctional: Yes Code Template: for Android Refer guideline# 11 //Check xmls tag xmlns:app=”http://schemas.android.com/apk/res-auto” app:showAsAction=”always” //When we use it will automatically fit to available space, this code is developed by ShowAsAction class //Source: http://stackoverflow.com/questions/31382174/android-searchview-in-actionbar

14. NN#243 Use history-based suggestions for search. Present them to the users when they move input focus to the search ﬁeld. Description: This guideline is a search version of the

62 more general guideline for typing (67 and 72), which recommends giving users autosuggestions and history-based defaults to minimize typing.

Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

15. NN#235 Do not use several search boxes with different functionalities on the same page. Description: We often see mobile users become disoriented and lose sense of context. When there are multiple search boxes, each of them implementing a different type of search, users may confuse one with the other. First, they may not even see all the search boxes at a time, and second, they may not read carefully the descriptions of any of them. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

16. G#32 User can easily sort search results. Description: We often see mobile users become disoriented and lose sense of context. When there are multiple search boxes, each of them implementing a different type of search, users may confuse one with the other. First, they may not even see all the search boxes at a time, and second, they may not read carefully the descriptions of any of them. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

17. G#33 ser can easily filter search results Description: We often see mobile users become disoriented and lose sense of context. When there are multiple search boxes, each of them implementing a different type of search, users may confuse one with the other. First, they may not even see all the search boxes at a time, and second, they may not read carefully the descriptions of any of them. Automatable: Yes IsFunctional: Yes Code Template: for Android //If following text is present in any of the JAVA file this means searchView with filter is present in the App setOnQueryTextListener(new SearchView.OnQueryTextListener(){ “adapter.getFilter().filter(newText);”

63 18. G#40 Product images can be easily zoomed into (and are still high quality) Description: Users should be able to easily zoom into a product image to inspect it in more detail, by double clicking or selecting a zoom button. The zoomed-in images need to maintain high quality. And to avoid making the user think, state the action required, e.g. Double Tap to Zoom. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

19. G#45 Primary call-to-action button is the most prominent on-screen element, and secondary call-to-action button is in close proximity to it Description: The primary CTA should be the most prominent button on screen, and your eyes should naturally be drawn to it, e.g. Add To Basket, Book Now, Submit Form, etc. And, secondary CTAs should be in close proximity to it, to satisfy cross-device behaviours (micro-conversions), via features such as Email Product, Add to Wish List, etc. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

20. G#47 User can see clear and persistent basket or booking symbols Description: A persistent basket or bookings symbol in the main navigation bar is easily accessible (usually top of screen, but not always) and reminds users of its contents and cost (could be itemised and / or total). It provides reassurance and simple editing of its contents. Automatable: No IsFunctional: No

21. G#85 If app supports push notifications, the user is asked to opt-in (to get their permission) Description: Your notification user base should be made up of opt-in users, who gave explicit permission for you to push messages to them. Be clear about the different types of notifications you intend sending them, when you intend sending them, and the frequency. Always respect the user’s attention - the one scarcity all brands are competing for. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

22. G#88 User can tap your widget or notification to go directly to that content in-app Descrip- tion: A user can view and act on your widget content by single tapping to open your main app to view the listed products, events or bookings. And, a user can tap your notification action and go to that specific app screen (and not the home screen) - deep linking will ensure a seamless transition

64 from notiﬁcation to app, and improve conversion Automatable: Yes IsFunctional: Yes

Code Template: for Android //Check presence of remoteViews which is responsible of open content directly RemoteViews remoteViews = new RemoteViews(context.getPackageName(), R.layout.vlabs app widget); Intent conﬁgIntent = new Intent(context, Carousel.class);

//Check following manifest ﬁle

//Note : android:name android:resource are user deﬁned parameters

23. G#90 Content is accessible when user has no connection (i.e. no 3G or WiFi) Description: Make sure that key content within the customer journey is accessible - i.e. cached in-app - even when there is little or no data connection via either 3G or WiFi. This needs to be carefully identiﬁed and prioritised, to enable a seamless user experience and to avoid abandonment. Automatable: Yes IsFunctional: Yes

65 Code Template: for Android // Find in all java classes If any java class contains this implementation //Note: the implimentation done by N ways. This is one way. A) ConnectivityManager manager = (ConnectivityManager) getSystemSer- vice(CONNECTIVITY SERVICE); //for mobile check B) manager.getNetworkInfo(ConnectivityManager.TYPE MOBILE) .isConnectedOrConnecting(); //For WiFi Check C) manager.getNetworkInfo(ConnectivityManager.TYPE WIFI) .isConnectedOrConnecting();

//Note: some times user-permission required

24. G#93 Primary call-to-action buttons are sticky / persistently visible (even when screen is scrolled) Description: Make sure that key content within the customer journey is accessible - i.e. cached in-app - even when there is little or no data connection via either 3G or WiFi. This needs to be carefully identiﬁed and prioritised, to enable a seamless user experience and to avoid abandonment. Automatable: Yes IsFunctional: Yes

66 Code Template: for Android //Check all the xml ﬁles //Check for list view ids tags and search for if there is any android:layout alignParentStart=”true” android:layout alignParentBottom=”true”

//And check if there is any button having the tag android:layout alignParentBottom=”true”

//Or

//Get all the java files and check for FloatingActionButton fab = (FloatingActionButton) findViewById(R.id.fab); fab.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View view) { Snackbar.make(view, ”Replace with your own action”, Snackbar.LENGTH LONG) .setAction(”Action”, null).show(); } }); //The above code snippet is used for floating action button

25. G#94 In forms, call-to-actions that are below the fold automatically scroll-up after the user has populated the necessary ﬁelds (users should never have to look for call-to-actions) De- scription: If a form has a call-to-action below the fold, make it easy for users to see and act on it, by automatically scrolling the button into view once the user has populated the necessary ﬁelds Automatable: Yes IsFunctional: Yes

67 Code Template: for Android //There are two ways we can do this operation

//Find all id’s in java class (Example: Signup.java) //If any id contains this ﬁeld (Eaxample: buttonid is id for submit) buttonid.setVisibility(View.INVISIBLE); or buttonid.setVisibility(View.GONE); or buttonid.setVisibility(View.VISIBLE);)//visible is used to show the button Or

//Find all xml ﬁles (Example: activity user signup.xml) android:visibility=”gone” android:visibility=”invisible” .setVisibility() action can be applied to any ﬁeld.

26. G#95 Form labels are either above form fields or are floating labels (in Android) Descrip- tion: Form labels should be above form fields so that users can easily see what they are filling in and why. Form fields to the side greatly reduce the space available for the fields. Form field descriptions within the field disappear once text entry begins and if a user is then distracted they may not remember what they need to complete later. In Android, instead of putting form labels above fields, a floating inline label can be used, e.g. when a user engages with a text input field it causes the label to move and float above the field, so as to guide the user whilst entering data. Automatable: Yes IsFunctional: Yes

Code Template: for Android // In activity ( Example homepage.xml in model app) //check for value

27. G#97 Form input is validated in real-time (after ﬁeld entry you will see if it is correct or not) Description: Warning people that they are making mistakes even as they make them will save them the time and frustration of submitting and then discovering mistakes. Source : Android Automatable: Yes IsFunctional: Yes

68 Code Template: for Android // In activity sign up.xml get all the button ids // In activity Signup.java check for setOnClicklistner // (Example signup.java class -> submit.setOnclickListener() //Where submit is button id

//Check if any button id is having .SetError()” //method

28. G#98 NN#66 In a form, cursor is placed in first field and the appropriate input is displayed by default Description: Reduce the number of form clicks and help focus the user, by placing the cursor in the first form field by default, and by displaying the appropriate input type (e.g. keyboard or dial-pad) by default. Make this effort on behalf of the user so that they can complete forms faster. Automatable: Yes IsFunctional: Yes Code Template: for Android // Get all the java files // Get all the edit text ids // Get the 1st edittext ID from that particular xml file of the activity // Check if this edittext id has a method editText.requestFocus(); or editText.setFocus();

29. G#100 In a form, telephone number entry is restricted to a dial-pad (i.e. numbers only) De- scription: When users need to enter a telephone number, only display the dial-pad, to limit their input to numbers and not characters. This will reduce their choices and mistakes, and accelerate completion. Considerations include: display the number format that they need to input by default with an input mask - and not by gradual reveal - to assist them; don’t accept characters entered (mask them out). Automatable: Yes IsFunctional: Yes Code Template: for Android // In activity Signup i.e android:id/activity sign up // check for value for telephone number related ﬁelds android:inputType=”number”

30. G#104 In a form, user can select times or dates from a picker control (e.g. for entering travel dates, date of birth or time of arrival) Description: When users need to select times or dates provide a native time and date picker control, which make selection fast and familiar, and support multiple formats

69 Automatable: Yes IsFunctional: Yes

Code Template: for Android // In activity DateTimePicker i.e android:id/activity datetimepicker // Check for activity datetimepicker.xml // Check for tags

//And

31. G#106 When app crashes it re-starts and returns the user to the last screen used(to continue where they left off) Description: In the event of an app crash, ensure it restarts and returns its state to the last screen used, so that the user can continue where they left off - especially if their journey involved entering data and required effort. Don’t require users to re-enter data. Automatable: Yes IsFunctional: Yes

Code Template: for Android // Note: This can be done in many ways // In manifest ﬁle ﬁnd the value android:alwaysRetainTaskState=”true”

32. NN#42 You can use borders and color to make targets look tappable OR Use different visual styling for tappable and non tappable design elements. Source : NN/g Automatable: Yes IsFunctional: Yes

70 Code Template: for Android // In activity checkbox android:id/activity checkbox; checkboxes 1 and 2 are the ids // Check for activity checkbox.xml // Get all the Id’s of the buttons // Check whether the button size is either android:layout width=”match parent” or android:layout width=”wrap content” android:layout height=”wrap content” or android:layout height=”36dp”

//where 36dp is minimum height

33. NN#43 To signal tappability follow target-placement conventions on your platform. Automatable: Yes IsFunctional: Yes Code Template: for Android // In activity checkbox with buttons(android:id/defaultbutton,android:id/ﬂatbutton // Check for activity checkbox.xml // get all the Id’s of the buttons // check whether the button size is either // for default button: android:layout width=”match parent” android:layout height=”wrap content” or max height is android:layout height=”36dp”

// for ﬂat button : android:layout width=”match parent” android:layout height=”wrap content” or max height is android:layout height=”48dp”

34. NN#44 Be consistent in your treatment of targets within your app or website. Automatable: No IsFunctional: No

35. NN#131 Use a dynamic progress indicator for every action that takes longer than 1 second.

Description: whenever the the user taps a speciﬁc operation such as getting content from other source then application may take some time to load. So as to notify the user that the process is loading we have show him a progress bar instead of displaying the blank page. Source: android http://stackoverﬂow.com/questions/12559461/how-to-show-progress-barcircle- in-an-activity-having-a-listview-before-loading Automatable: Yes IsFunctional: Yes

71 Code Template: for Android // Find all xml ﬁles // Find Progressbar tag in xml ﬁle

//Note: this Progress Bar can be done by many ways such as circular ring or horizontal bar

36. NN#75 Use GPS information to detect location automatically and allow the use of current location

Description: Our application has to get the location of the user instead of user manually entering his location. Automatable: Yes IsFunctional: Yes Code Template: for Android // Search in all java classes // Check if any class extends Service implements LocationListener // Open manifest.xml ﬁle // check this below tags

// Note: ACCESS FINE LOCATION (Which includes both ACCESS FINE LOCATION and ACCESS COARSE LOCATION). Also if you are getting network-based location then you need to add INTERNET permission too. // Source: http://www.androidhive.info/2012/07/android-gps-location-manager-tutorial/

37. NN#76 Allow users to edit the location provided by the GPS or other info captured via phone features. Automatable: Yes IsFunctional: Yes Code Template: for Android //Refer guideline# 35

38. NN#82 Make sure that your menus look expandable.

Description: these are often encountered in forms, when users have to choose from a set of alternatives hidden under the drop-down. Such as the date of birth or country

72 Automatable: Yes IsFunctional: Yes Code Template: for Android // Find all xml files (Exmaple xml filename: activity user sign up.xml in model app) // if xml file contains this spinner tag i,e:

39. NN#60 Make textboxes long enough so that users don’t have to scroll within. % Source : NN/g Guideline# 60 Automatable: Yes IsFunctional: Yes Code Template: for Android //In homepage.xml get all the ids //Check for EditText attributes : android:layout width=”match parent” android:layout height=”wrap content”

40. NN#70 Autoformat ﬁelds for users. Don’t force them to use characters such as dash or space to achieve a speciﬁc format. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

41. NN#71 Do not make people memorize information from one page to another. Automatable: No IsFunctional: No

42. NN#72 Use personalization and history to provide good defaults and suggestions for text that needs to be input. Automatable: No IsFunctional: No

43. NN#73 Allow users to easily delete default ﬁeld values. Because defaults are bound to be wrong occasionally, there should be a way to easily delete them. Users shouldn’t have to manually erase the text ﬁeld character by character, by clicking the delete key. Instead, they should be able to

73 erase the defaults by pressing a single button that clears the entire ﬁeld. Automatable: No IsFunctional: No

44. G#12 In Android apps, the app bar and tab bar should be located at the top of the screen (not bottom) Automatable: Yes IsFunctional: Yes Code Template: for Android //Check these tags in xml ﬁle (Example:activity simple tabs)

//In the tab layout check these fields: app:tabMode=”fixed” app:tabGravity=”fill”

45. NN#77 Use the camera to ﬁll in information where possible. Description: Most phones come with other features that can be employed to speed up user input. The camera can easily be used to take a picture of an object or of a barcode. There is a plethora of apps that incorporate barcode scanners: some, such as RedLaser and Bar Code Scanner were dedicated to reading barcodes and searching prices for the corresponding products, but many other apps (e.g., Target, Amazon) included a search-by-barcode feature. And taking a picture of a credit card is a lot more convenient than typing in the 16 or more Digits. Automatable: Yes IsFunctional: Yes Code Template: for Android //Note: this can be done in many ways. We used the IntentIntegrator i.e the BarcodeScanner application needs to be installed before using the scanner. //Get IntentIntegrator.java ﬁle //Check for P ackageManager pm = activity.getPackageManager(); //(which checks for the barcode scanner application is present or not in the mobile or not. //There is no unique listener to show the Scanner class.

46. NN#78 Use voice recognition to help users input information. Description: Voice recognition is well integrated in most mobile platforms: it’s easy to say a word and have the phone recognize it. Standard keyboards include a microphone key. Many of our users rely on Siri or a similar voice-recognition system to initiate tasks in situations where touching the screen is cumbersome. And some apps even accept voice control. For example, in

74 Epicurious (a cooking app), instead of having to use dirty hands to interact with the app, users could use their voice to go back and forth between different parts of a Recipe. Automatable: Yes IsFunctional: Yes Code Template: for Android //Go to the VoiceRecognition.java ﬁle //And search for the value RecognizerIntent.ACTION RECOGNIZE SPEECH

47. NN#79 Do not use sliders for inputting precise values. Only use sliders when the exact value is not important. Automatable: No IsFunctional: No

48. NN#80 Do not use spin wheels for items that don’t have a natural order. Automatable: No IsFunctional: No

49. NN#81 Do not use color to communicate the active choice. Automatable: No IsFunctional: No

50. NN#88 Minimize scrolling in a drop-down box by using the whole screen to display long lists of options. Description: Unlike on Android and Windows Phone, on iOS drop-downs take only half the screen space. Scrolling through screens and screens of information in a tiny window is painful already; doing it in a window half the screen size is twice as painful. If you have a long list (for instance, US states, countries, years), instead of using a drop-down, consider displaying it in-situ or on a separate page. Automatable: Yes IsFunctional: Yes Code Template: for Android // find all xml files // if xml file contains this spinner tag i,e:

51. NN#103 Users should reach the last item in the carousel in 3-4 steps (e.g., taps or swipes). Automatable: Yes IsFunctional: Yes

75 Code Template: for Android //Get all the .xml ﬁles //Check for value

Or //In Carousel.java ﬁle check for FeatureCoverFlow.OnScrollPositionListener()

//Note: values above can be changed as per the requirement

52. NN#110 Use arrows or the illusion of continuity to show users that they can see more items in a carousel. Automatable: Yes IsFunctional: Yes Code Template: for Android //Get all the .xml ﬁles //Check for

//In Carousel.java ﬁle check for FeatureCoverFlow.OnScrollPositionListener()

53. NN#112 Label buttons in a descriptive way. Automatable: No IsFunctional: No

54. NN#114 On a page, do not place main calls-to-action at the top of the page, in the tool bar or navigation bar. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

55. NN#115 Don’t use alerts, fading popups, or separate pages to signal an error in a form. Automatable: No IsFunctional: No

76 56. 90. Don’t use alerts, popovers, or separate pages to display instructions for ﬁlling in the form, or other information that the user must refer back to. Source : NN/g Guideline# 116 Automatable: No IsFunctional: No

57. NN#117 Use toasts only for feedback that is not crucial; do not use them for error messages. Automatable: Yes IsFunctional: Yes Code Template: for Android //Get all the java ﬁles. //Get all the Edit Text tags //Check if any id is having .SetError(); //Note: We had used the Toast for acknowledgement purpose like Login and signup cases rest we had given real time errors. This can’t be shown uniquely. It is non-functional.

58. NN#119 In a form, mark ﬁelds that need to be corrected to signal an error. Automatable: Yes IsFunctional: Yes Code Template: for Android //Go to the signup.java ﬁle //Get all tag ids of EditText //Check if any id is having .SetError();

59. NN#120 In a form always show the error message. Do not hide the error message and expect the users to uncover it. Source : NN/g Guideline# 120 Automatable: Yes IsFunctional: Yes Code Template: for Android //Go to the signup.java ﬁle //Get all tag ids of EditText //Check if any id is having .SetError(); //Example: id.setError(“valid name”)

60. NN#121 An error message should tell users (1) what the problem is, and (2) what they can do to ﬁx it.

77 Automatable: No IsFunctional: No Code Template: for Android //Refer Guideline# 120 code pattern

61. NN#122 After reporting an error, return to the state before the error. Automatable: No IsFunctional: No Code Template: for Android //Refer Guideline# 120 code pattern //Note: sir the guideline is some what a contradictory as, if the user mistypes a single character then according to this guideline it gets empty ﬁeld and again he should type the whole thing. So it may cause some inconvenience to user at some times.

62. NN#123 Don’t report an error regarding a feature that is not in use. Automatable: No IsFunctional: No

63. NN#125 Do not turn on notiﬁcations by default. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

64. NN#126 Avoid turning on sound notiﬁcations unless people explicitly agree to them. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

65. NN#135 Support both phone orientations (landscape and portrait) whenever possible. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

66. NN#146 In most situations, users need cues to ﬁgure out that they’re supposed to use the horizontal-swipe gesture for horizontal navigation. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD

78 67. NN#157 In a form always specify whether a ﬁeld is required (preferably by using an aster- isk). Automatable: Yes IsFunctional: Yes

Code Template: for Android //Go to the signup.java ﬁle //Get all tag ids of EditText //Check if any id is having .SetError();

68. NN#161 Use collapsed accordions to represent different steps of a form. Automatable: No IsFunctional: No

69. NN#167 Do not automatically subscribe users to newsletters or other email.Allow users to opt in, instead of requiring them to opt out. Automatable: Yes IsFunctional: Yes

Code Template: for Android //TBD

70. NN#170 Don’t have a login wall when the app is ﬁrst launched. Deliver value before asking users to log in. Automatable: Yes IsFunctional: Yes

79 Code Template: for Android //Check for the presence of the form in the starting of the app. We had implemented the landing page which will redirect to the homepage after 5 seconds. //The Landing page should always be shown in full screen. //In Landingpage.java check for requestWindowFeature(Window.FEATURE NO TITLE); getWindow().setFlags(WindowManager.LayoutParams.FLAG FULLSCREEN, WindowManager.LayoutParams.FLAG FULLSCREEN);

//The above code is for showing the activity in fullscreen //Note: ﬁnd xml ﬁle related to this java class using below syntax setContentView(R.layout.activity main); (check this in java class)

//Go to xml file check form fields(no of fields should be less than 1 or none) //Note: how to find first page or launcher page //Go to manifest.xml file //Find all activities //If any activity contains this below tag

//Then we can say this activity is launcher activity 71. NN#171 Do not ask users to log in or register unless absolutely necessary for security reasons. Automatable: Yes IsFunctional: Yes Code Template: for Android //TBD 72. NN#182 When logging in and out must be done for each session, let users define a numerical pin. Automatable: Yes IsFunctional: Yes Code Template: for Android //This can be done in many ways and there several authentication methods such as pattern matching,pin and fingerPrint recognition. //We had implemented this using the pin authentication. //There is no unique method to show this. This the java code which redirect the user to certain page after entering his password which is given at signup. //The Authentication.java file has the verification logic.

80 73. NN#185 A login form should include a Forgot password? link to allow users to recover their password. Automatable: Yes IsFunctional: Yes

Code Template: for Android //Find in all xml ﬁles(Example :content homepage.xml) //Find TextView ﬁeld and check below attributes android:autoLink=”web” or “email” android:linksClickable=”true” //Note: this can be implemented by many ways

74. NN#188 If user made an error in the log in or registration form, do not clear the password ﬁeld. Do not make them reenter their passwords from scratch. Automatable: Yes IsFunctional: Yes

Code Template: for Android //Ignoring this guideline based on the internal discussion with usability expert

75. NN#196 Remove duplicate ﬁelds (email, password). Automatable: Yes IsFunctional: Yes

Code Template: for Android //Find all xml ﬁles(Example: sign up.xml in model app) android:inputType=”textEmailAddress”

//if textEmailAddress contains more than once then we can say that it has duplicate ﬁelds. android:inputType=”textPassword”

//Check if the above ﬁelds contains more than once then we can say that it has duplicate ﬁelds.

76. NN#198 Do not use more than 5 options in a tab bar or top navigation bar. Automatable: Yes IsFunctional: Yes

81 Code Template: for Android //Check these tags in xml ﬁle (Example:activity simple tabs)

//In the tab layout check these ﬁelds:

//Go to java class(Example: SimpleTabsActivity) //Check these values: ViewPagerAdapter adapter = new ViewPagerAdapter(getSupportFragmentManager()) //Check if adapter contains more than 5 objects or not (length of adapter)