Masaryk University Faculty of Informatics Usability Analysis of TLS API Documentation Master’s Thesis Bc. Matěj Grabovský Brno, Spring 2020 Masaryk University Faculty of Informatics Usability Analysis of TLS API Documentation Master’s Thesis Bc. Matěj Grabovský Brno, Spring 2020 This is where a copy of the official signed thesis assignment and acopyofthe Statement of an Author is located in the printed version of the document. Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Bc. Matěj Grabovský Advisor: RNDr. Martin Ukrop i Acknowledgements I thank my dearest Kateřina for her fair gaiety, affectionate assiduities, firm support and constant sense of belonging. This work would not be presented here as is without her presence and input. I thank my advisor Martin for his boundless patience, invaluable advice, wise guidance and support in all phases of our work. My thanks also goes to Lydia for her priceless help and advice on writing and data analysis. Though many of my friends and family remain unnamed here, mygrat- itude goes to them undiluted for they supported me selflessly throughout the time and provided a much needed relief and distraction innumerable times. ii Abstract This thesis deals with the usability of selected TLS libraries, their documen- tation, code samples and related resources. We describe the design and re- sults of our initial exploratory study and a follow-up study, both performed with students of the Faculty of Informatics. The results show that the selected libraries’ documentation suffers from many usability problems. Although these issues are not specific to TLS libraries, their consequences might be more serious. Quantitative rating of the severity of selected problems suggests that the libraries are comparable. Further research is necessary in order to determine what kind of exam- ples are most useful to developers and, especially, how to compel library developers to heed the research community’s recommendations. Keywords developer experience, documentation, human factors, usability, usable se- curity, user study, TLS iii Contents 1 Introduction 1 2 Exploratory Study 4 2.1 Design and Methods 4 2.2 Data Analysis 5 2.3 Setting and Participants 6 2.4 Results 6 3 Methodology 10 3.1 Research Questions 10 3.2 Study Design 10 3.3 Task Design 11 3.4 Setting 15 3.5 Participants 15 3.6 Data Collection, Processing and Analysis 17 3.7 Limitations 19 4 Results 22 4.1 Identified Issues 22 4.2 Severity of Obstacles 33 4.3 Other Sources of Information 35 4.4 Supplementary Analyses 40 5 Discussion 42 5.1 Obstacles 42 5.2 Other Sources 44 6 Related Work 45 7 Conclusion 47 7.1 Future Work 47 Bibliography 49 iv A Pre-Task Questionnaire 54 B Post-Task Questionnaire 57 C Assignment Microsite Snapshot 60 v 1 Introduction TLS and Usability. Transport Layer Security (TLS) is a family of security protocols for secure, encrypted and authenticated communication in com- puter networks. It is most widely deployed in the HTTPS (HTTP over TLS) protocol, which serves hunders of millions of websites over the world with the numbers growing steadily. The Let’s Encrypt (2020) initiative claims to have provided TLS certifi- cates to 225 million websites. According to Google (2020), more than 90% of all pages loaded in the Google Chrome browser are served over HTTPS. NetMarketShare (2020) estimates 87% of all web traffic is encrypted. The importance of TLS for the modern Internet is undisputable. How- ever, several studies have shown that the protocol implementations are misused and misunderstood by developers, risking their users’ data and privacy (Egele et al. 2013; Georgiev et al. 2012; Krüger et al. 2018). Moreover, it has been shown that security and cryptographic inter- faces in general are hard to use and too easy to misuse (Acar, Backes, Fahl, Garfinkel, et al. 2017; Iacono and Gorski 2017; Nadi et al. 2016). Even pro- fessionals developers tend to struggle with writing code that is both secure and functionally correct (Acar, Stransky, et al. 2017; Georgiev et al. 2012; Krombholz et al. 2017). Exploratory Study. To this end, it is clear that the usability of security interfaces, their documentation and other resources is essential for their pratical security. In order to generate new reserarch questions at the in- terscetion of TLS and developer experience (DX) research, we performed a small qualitative exploratory study with in 2018. Nine IT students partic- ipated in the study, tasked to create a small application with three estab- lished C-language TLS libraries. The results revealed and confirmed a few usability factors considered by developers when working with the APIs. They have also hinted at the indispensability of the availability of quality documentation and code sam- ples, in line with previous research (Acar, Backes, Fahl, Garfinkel, et al. 2017; Nadi et al. 2016). Follow-up Study. We therefore decided to narrow down on this partic- ular aspect of library usability in a larger, more focused follow-up study. 1 1. Introduction We designed a mixed-methods user study, again with IT students at our university as participants. We aimed to identify common issues in docu- mentation, quantify the severity of some issues, and observe patterns in use of external sources of information. This time, we selected two popular libraries for two programming lan- guages each, diversifying the field of possibilities. We also opted for a more realistic task of modifying an existing application rather than creating a new one. We then analysed the results of our follow-up study, identify the causes and implications of various issues, and situate our findings within the ex- isting body of research. Thesis Outline. The thesis is laid out as follows. At the beginning, Chap- ter 2 lays out the design, execution and results of our initial exploratory study on the usability of TLS programming interfaces. In Chapter 3, we set the research questions for our follow-up study and detail the methods we chose to answer them with. Chapter 4 then lists the results from our analyses of the qualitative and quantitative data we collected. In Chapter 5, we situate the results within a wider context, interpret them and speculate on possible implica- tions. Chapter 6 surveys the the related research grounds and where our work fits in. And finally, Chapter 7 sums up our findings, their possible limitations and potential directions for further research. Acknowledgement of Contributions It bears mentioning that the work presented here is in part the result of a collaboration with my advisor, RNDr. Martin Ukrop, and a consultant, Dr.-Ing. Lydia Kraus. They were both advising me and providing feedback in all stages ofthe research process – designing the studies, analysing and understanding data, and presenting results. It would be too difficult to list their contributions precisely, so I present here merely a short summary of the key moments. Martin Ukrop helped design both studies and the tasks, administered the assignments in both experiments, anonymised the collected data and preprocessed them before the analysis phase. He also served as the second coder in coding qualitative data. 2 1. Introduction Lydia Kraus assisted with the analysis of qualitative data, creating the codebook and calculating interrater agreement in the exploratory study. She also advised me in various stages of analysing the data in the follow- up study. Moreover, we wrote the work-in-progress short paper presenting our exploratory study collaboratively. Unless otherwise noted, all the remaining work presented on the fol- lowing pages is my own with all due responsibility. 3 2 Exploratory Study While the usability of lower-level cryptographic libraries has been under active scrutiny by the usable security and DX research community (Acar, Backes, Fahl, Garfinkel, et al. 2017; Arzt et al. 2015; Egele et al. 2013; Nadi et al. 2016), the issues specific to TLS-related APIs and the usability ofcon- crete TLS libraries have not yet been thoroughly investigated to our knowl- edge. Therefore, following the agenda set forth by Acar, Fahl and Mazurek (2016), we designed and performed a small exploratory user study with the aim to to uncover potential TLS-specific usability issues. We defined our scope to TLS libraries, their APIs, their documentation and other related resources. 2.1 Design and Methods Since we were aiming to generate new research questions and hypothe- ses in this area, we opted for a purely qualitative study with open-ended data. Qualitative research methods had been used successfully in several studies before (Iacono and Gorski 2017; Krombholz et al. 2017; Naiakshina, Danilova, Tiefenau, Herzog, et al. 2017; Naiakshina, Danilova, Tiefenau and Smith 2018), which reinforced our choice of methodology. Due to the exploratory nature of our endeavour, we established no re- search questions a priori. Instead, we designed a programming task to be completed by the participants in three different libraries with a subsequent written report focusing on elements of the usability of theAPI. We chose three C libraries with TLS support for our experiment: Open- SSL, GnuTLS and mbed TLS (formerly PolarSSL). At the time of writing, OpenSSL is arguably the most popular cryptographic library (per Internet- wide scans; Nemec et al. 2017). GnuTLS aims to be a full-featured GPL- compatible alternative, although with a substantially different API. The API of mbed TLS is also distinct and we expected interesting data could be generated from comparisons with the two mainstream libraries. The programming task consisted of implementing a simple HTTPS client, starting from a basic code skeleton with library initialisation and clean up code using each of the libraries.