LIU-ITN-TEK-A--15/059--SE Cross-Platform Post-Mortem Analysis in a Distributed Continuous Integration System Karl Johan Krantz 2015-09-22 Department of Science and Technology Institutionen för teknik och naturvetenskap Linköping University Linköpings universitet nedewS ,gnipökrroN 47 106-ES 47 ,gnipökrroN nedewS 106 47 gnipökrroN LIU-ITN-TEK-A--15/059--SE Cross-Platform Post-Mortem Analysis in a Distributed Continuous Integration System Examensarbete utfört i Datateknik vid Tekniska högskolan vid Linköpings universitet Karl Johan Krantz Handledare Alexander Bock Examinator Karljohan Lundin Palmerius Norrköping 2015-09-22 Upphovsrätt Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under en längre tid från publiceringsdatum under förutsättning att inga extra- ordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/ Copyright The publishers will keep this document online on the Internet - or its possible replacement - for a considerable time from the date of publication barring exceptional circumstances. The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page: http://www.ep.liu.se/ © Karl Johan Krantz Abstract This thesis aims to improve the cross-platform reliability of software components tested on distributed CI systems. More specifically, it is centered on extracting crash information from cross-platform crashes. Crash information was generated and parsed on Mac, Linux and Windows. The crash information was transformed into stack traces, and further work has been put into visualizing these using graph representation. The crash information proved to be valuable for developers in their day-to-day job, especially the raw crash information. However, the visualizations proved to be less than satisfactory for developers as they increased the mental load. Preface First off all, thanks to Skype for having me during the five months of the thesis, the amount of resources and raw skill at the Skype office in Stockholm is incredible and I am extremely thankful for the help. However, none of it would have been possible without the help of my dear friend and staffing colleague Ken Liu, who processed all the formalities regarding the thesis on Skype’s end. A giant thanks to the CI-Frameworks team as well for having me be a part of their team during the thesis. Everyone on the team made me feel at home in the office and made sure I got all the help I needed. Thanks to Selcuk for being my manager and advisor during the thesis and helping me get in touch with stakehold- ers and help spear-head my future career inside of Skype. A special thanks to Johan Losvik, who helped a ton in my introduction to the team and my development setup. Thanks to Markus for being the other thesis student and a great friend to discuss things with and complain to when things did not go so well. Alexander, thanks for all the help with the thesis and this report, could not have done it without you! Contents 1 Introduction 1 1.1 Purpose . 1 1.2 Research Questions . 2 1.3 Limitations . 2 2 Background 4 2.1 Testing . 4 2.2 Crashes . 4 2.3 Continuous Integration . 7 2.4 Transient Errors . 9 3 Theory 10 3.1 Post-Mortem Analysis . 10 3.2 Visualization . 11 4 Implementation 13 4.1 Post-Mortem Data Creation . 13 4.2 Data Parsing . 16 4.3 CI Integration . 19 4.4 Visualization . 22 5 Evaluation 25 5.1 Structure . 25 5.2 Interviews . 25 5.3 Team Feedback & Current Usage . 28 6 Result 29 6.1 Data Acquisition . 29 6.2 UI Results . 30 6.3 Evaluation . 32 7 Discussion 33 7.1 Cross Platform Data . 33 7.2 Data Correlation . 33 7.3 Visualizations . 34 8 Future Work & Conclusions 35 8.1 Future Work . 35 8.2 Conclusions . 37 Appendices 41 A CDB Output 42 B GDB Output 43 C LLDB Python Output 45 D CrashReport Output 46 E JSON Structure 47 F Base Parser Code 48 G Regular Expressions 50 1 | Introduction Today at Skype, reliability is a large key metric that developers need to track. It is one of the larger issues developers face in their day-to-day work as the product becomes more and more complex with each added feature. This gets further problematic as the amount of devices per person increases each year [10]. This extends the need for more reliable software, since everything from TVs to fridges are getting internet connectivity, and there is a high likelihood that some of these devices will need to run Skype. Expanding this point, in emerging markets such as India, low-range phones are vitally important to the development of the country. If the designed software is primarily tested against higher-end phones, it could have severe defects on lower-end phones. This could leave a significant part of the worlds population disconnected from the communication we take for granted each day. 1.1 Purpose This brings on an important point, can developer use infrastructure and tools in order to help their day-to-day work in increasing the reliability of the product? A Continuous Integration (CI ) server provides automatic test executions on code check-in, this improves the reliability of the product by checking for regressions in functionality. CI servers often relies on test executions being a black box without any out- put information on how tests actually ran, except the logical value of pass or fail for test cases. This causes issue in cases where the tests are crashing, as the root-cause of the software fault can be hard to identify without further information. In this thesis, the role of the continuous integration server and the extra information it could provide to developers in order to increase the reliability of the product 1 is the primary focus. The thesis will include discussions and trials of methods to help developers with root cause fault detection in a distributed cross-platform CI system. This includes discussing and interviewing developers about non-functional requirements to improve the developer infrastructure in place at Skype. 1.2 Research Questions We create two concrete research questions in RQ 1 and 2, that bring together the larger topic of the thesis. RQ 1 Can analyzing the crash data from multiple platforms help in root cause fault detection? RQ 2 Does visualizations of these crashes help developers in building a greater understanding of the fault? These two questions are then summarized into two larger areas of research and work, Post-Mortem Analysis and Visualization. Post-Mortem Analysis will be presented in sections 3.1, 4.1 and 4.2 where the crash cause of a process will be investigated. Visualization will be presented in sections 3.2, 4.3 and 4.4, and will investigate methods that can be used to visualize and utilize crash information. 1.3 Limitations In order to develop a workable solution for Skype, there need to be certain limitations put on the thesis. Certain work and implementation details are simply out of scope for the thesis due to time constraints and to make a more interesting thesis. Platform Limitations In order to properly test the research questions, the supported platforms of this thesis does not need to be the same as all the ones Skype support. The goal is rather to show that the methods and theories work cross-platform. As such, this thesis focuses on the following platforms: • Windows • Linux • Mac 2 Adding to this, there will be no work done in regards to the build processes. This is due to the wildly varying processes between teams and platforms and that the method and process described should be generic enough for any team to use. The work within the thesis also assumes native programming languages, as these languages are the target platforms for the CI system. The process involved would not be different for a managed language such as Java — excluding the engineering effort of extracting the information from the crashes. 3 2 | Background In this chapter, an outline of the background premises for the thesis will be presented. Abbreviations that are necessary for the rest of the thesis will be explained here as well. 2.1 Testing Testing is the process of increasing the confidence in code working as intended.