Usability Evaluation Framework for Mobile Apps Using Source Code Analysis Thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science & Engineering by Research by Neeraj Mathur 201350902 [email protected] International Institute of Information Technology Hyderabad - 500032, INDIA Feb, 2019 Copyright c Neeraj Mathur, 2019 All Rights Reserved International Institute of Information Technology Hyderabad, India CERTIFICATE It is certified that the work contained in this thesis, titled “Usability Evaluation Framework for Mobile Apps Using Source Code Analysis” by Neeraj Mathur, has been carried out under my supervision and is not submitted elsewhere for a degree. Date Adviser: Prof. Y. Raghu Reddy To my Parents Acknowledgments My journey in IIIT-Hyderabad has been a wonderful experience. It could not have been so without the support of many people. As I submit my MS thesis, I would like to extend my gratitude to all those people who helped me on successfully completing this journey. First of all, I want to thank my guide Prof.Y. Raghu Reddy, for accepting me as a student and constantly guiding me during MS Research program. His guidance has helped me improve not only as a researcher but also as a person. His help and support during difficult times strengthened and motivated me to move further. I thank my fellow research mates at Software Engineering Research Center: Lalit Mohan, Sai Anirudh, Shivam Khandelwal, Sai Gollapudi and Sridhar C for stimulating discussions. In particular, I am grateful to Lalit Mohan and Sai Anirudh for constantly guiding me and giving directions to my research work. Special thanks to Sai Anirudh for closely helping me in every aspect of my research work. My sincere thanks to Mr. Rohit Udasi - Sr. Development Manager, Sustenance Engineering at Oracle India Pvt. Ltd., for creating enthusiasm and motivation to pursue research at IIIT Hyderabad. My sincere thanks to Mr. Hemendra Pal (Sr. Director) and Mr. Amar Rao (Director) at Oracle India Pvt. Ltd for providing an opportunity to pursue research at IIIT Hyderabad. I could not have accomplished it without the support and understanding of my family. I wish to thank my parents Smt. Sneh Lata Mathur, Shri Yogendra Mathur, my wife Vidhi Saxena and my daughter Navdha Mathur for being my constant support and motivation. Last, but not the least, thanks to IIIT community for giving me an inspiring environment and loads of opportunities to grow. v Abstract Apple Inc. unveiled the first iPhone in 2007. Cutting edge technology was introduced in the smart- phone with an interface that allowed users to interact with the device through various modes. Also, the ecosystem facilitated developers to create applications on mobile platforms, popularly referred to as “Mobile Apps”. Mobile apps are software applications designed to run on smart-phones, tablets, and other mobile devices. They are typically available through app stores operated by entities that own the corresponding mobile operating system. By 2020, its been forecasted that mobile apps shall generate around 189 billion USD [49] in revenues via app stores and in-app advertising. The increasing use of smart-phones has led to mobile applications replacing or supplementing tradi- tional web-based applications. Given the limitations of the form factor in smart-phones, Usability is one of the essential attributes that determine the success of a mobile apps. The measures available for as- sessing the usability of mobile apps tend to focus more on the human aspects and less on the functional aspects of Usability. Also, quantitative metrics that measure usability of mobile apps are very limited. As part of our initial research, we studied mobile apps in banking and finance domain. An analysis of the usability issues reported by the mobile users helped capture the challenges faced by these users as part of their day-to-day transactions. Various market giants have proposed varied usability guidelines to enrich user experience (like ISO, Android, Neilson Norman group etc). These guidelines are the repository for software practitioners to build usable mobile apps. However, there are no definitive standards or effective tools to detect and address usability issues. This made us propose a few banking specific usability guidelines based on a usability index called Mobile App Usability Index (MAUI). This study helped us establish an understanding of the functional and non-functional aspects of usability features while building a mobile app. Traditionally, usability evaluation is considered as a post-development activity which is usually taken care of by UX/UI Experts. Usability evaluation was done manually using various methods like cogni- tive walk-through, field studies, group studies etc, which delays detection of usability issues and requires re-development to address violations. To address this challenge, we formulated a usability framework called UMETRIX to conduct an automated usability evaluation for mobile app developers. The frame- work is flexible and developer friendly. The framework helps software practitioners validate and verify the presence of a usability guideline in a mobile app using code analysis. We built an automated tool using our framework that has the flexibility to create developer friendly use-cases to evaluate the pres- vi vii ence of a guideline along with a code-snippet recommendation engine to ease developers to choose a code template for implementing the desired usability guideline. We consider this framework as a productivity tool for developers to pre-validate and evaluate usabil- ity issues during the development phases of a mobile app. As part of our research, we collaborated with open source software mobile app vendors for a field study. Additionally, we conducted industrial em- pirical study on a few FinTech app vendors and captured their observations and results. We collaborated with industry practitioners in their mobile app development to address mobile usability issues on their pre-released version of the mobile app. The collaboration helped us document the guidelines which are possible for evaluation using code analysis. Contents Chapter Page 1 Introduction :::::::::::::::::::::::::::::::::::::::::: 1 1.1 What is Usability . .2 1.2 Need for Automated Mobile App Usability Evaluation . .4 1.3 Usability Testing in Mobile Apps . .4 1.4 Code Analysis For Usability Evaluation . .5 1.5 Research Contributions . .6 1.6 Scope . .6 1.7 Thesis Organization . .7 2 Related Work ::::::::::::::::::::::::::::::::::::::::: 8 2.1 Usability Models . .8 2.2 Reviews and Field Studies . .9 2.3 Experimental Analysis . .9 2.4 Pattern Detection . 10 3 Usability Guidelines and Code Templates ::::::::::::::::::::::::::: 11 3.1 Usability Guidelines . 11 3.2 Usability Code Template . 13 3.2.1 Usability Testing in Mobile Apps . 14 3.2.2 Observations . 15 4 Identifying Usability Challenges in Mobile Banking Apps in India :::::::::::::: 17 4.1 Mobile Banking App Usability Challenges . 17 4.2 Usability Measurement . 18 4.3 Recommendations For Improving Usability . 19 4.4 Time Taken to Complete Task . 19 4.5 User Interface Display Parameters . 20 4.6 Error Handling . 21 4.7 Conclusions . 23 4.8 Observation . 23 5 UMETRIX - Architecting Usability Evaluation Framework for Mobile Applications ::::: 24 5.1 Design Goal . 24 5.2 Architecture . 24 5.2.1 Incorporating guideline into validation test case . 26 viii CONTENTS ix 5.3 Outcomes . 29 6 Proof of Concept for Android Mobile Apps :::::::::::::::::::::::::: 30 6.1 Validation Test Case Generator . 30 6.2 Architecture . 31 6.2.1 Process . 32 6.2.2 User Interface . 34 6.3 Guideline Evaluator . 34 6.3.1 Architecture . 34 6.3.2 Process . 36 6.3.3 User Interface . 36 6.4 Summary . 38 7 Evaluation of Framework ::::::::::::::::::::::::::::::::::: 41 7.1 Experiment 1: Accuracy and Robustness . 41 7.2 Experiment 2: Evaluation with open source mobile apps . 41 7.3 Experiment 3: Industry Evaluation and Results . 43 7.4 Threats to Validity . 47 7.5 Observations . 47 8 Conclusion and Future Work :::::::::::::::::::::::::::::::::: 49 8.1 Contributions . 49 8.2 Limitations . 50 8.3 Future Work . 50 Bibliography :::::::::::::::::::::::::::::::::::::::::::: 52 Appendix A: Usability Guideline Classification along with Code Snippets ::::::::::: 59 List of Figures Figure Page 1.1 Comparison of usability models (courtesy Harrison et al [29]) . .3 1.2 Mobile Usability Testing Tools . .5 4.1 MAUI Parameters . 22 5.1 Usability Evaluation Framework . 25 5.2 Validation Test Case . 28 6.1 Architecture of Validation Test Case Generator . 32 6.2 Flow Diagram of Validation Test Case Generator . 33 6.3 Process to upload validation case to validation case DB . 34 6.4 Validation Test Case Generator Screen Shot . 35 6.5 Architecture of Web Guideline Evaluator . 36 6.6 Guideline Evaluator Process Diagram . 37 6.7 Upload APK file . 38 6.8 Basic Information of APP . 38 6.9 Validation Plan . 39 6.10 Usability Results . 40 7.1 Validation test case results of open source apps - Part 1 . 44 7.2 Validation test case results of open source apps - Part 2 . 45 x List of Tables Table Page 3.1 Usability Guidelines . 13 3.2 Automated - Usability Guideline . 16 3.3 Semi-Automated - Usability Guideline . 16 3.4 Non-Automated - Usability Guideline . 16 7.1 Validation test results of Froogal and Vlab App . 42 7.2 Mobile Apps used for UMETRIX Case Study . 46 xi Chapter 1 Introduction The computing potential of mobile devices is increasing day-by-day. Internet access via mobile broadband is also increasing at a significant rate. Almost all the websites are now offering their products and services via mobile apps. These mobile apps are quickly made available to all the users through distribution platforms called ‘app stores’. A recent article1 published in SmartInsights in April 2016, titled - “Mobile Marketing Statistics compilation” describes the details of an extensive survey conducted on mobile device usage over the past decade by capturing the user activity automatically from their mobile devices.