ROS-Industrial Quality-Assured Robot Software Components - ROSIN Quality Assurance Process and Community Management in ROS D3.1 Grant Agreement No. 732287 Start date of Project 1 January 2017 Duration of the Project 48 months Deliverable Number D3.1 Deliverable Leader ITU Dissemination Level PU Status 1 Submission Date 01/09/2017 Author Yvonne Dittrich, ITU ([email protected]) Claus Brabrand, ITU ([email protected]) Gijs van der Hoorn, TUD ([email protected]) Jon Azpiazu, TEC ([email protected]) Iñigo Martinez, TEC ([email protected]) Nicolas Limpert, FHA ([email protected]) Jonathan Hechtbauer, IPA ([email protected]) Jon Tjerngren, ABB ([email protected]) Adam Alami, ITU ([email protected]) Andrzej Wąsowski, ITU ([email protected]) This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 732287. The opinions expressed in this document reflect only the authors’ views and in no way reflect the European Commission’s opinions. The European Commission is not responsible for any use that may be made of the information it contains. PU=Public, CO=Confidential, only for members of the consortium (including the Commission Services), CI=Classified, as referred to in Commission Decision 2001/844/EC. 732287 ROSIN D3.1 Quality Assurance and Community Management in ROS This page has been intentionally left blank Version 1.0 01-09-2017 2 732287 ROSIN D3.1 Quality Assurance and Community Management in ROS Modification Control Version # Date Author Organisation 1.0 01-09-17 Yvonne Dittrich, Claus ITU, TUD, Tecnalia, Brabrand, Gijs van der FHA, IPA, and ABB Hoorn, Jon Azpiazu, Iñigo Martinez, Nicolas Limpert, Jonathan Hechtbauer, Jon Tjerngren, Adam Alami, and Andrzej Wasowski Release Approval Name Role Date A. Wasowski WP3 Leader 31-08-2017 C.Hernandez Corbato WP2 Leader/ Coordinator 01-09-2017 A. Ferrein WP4 Leader 31-08-2017 M. Bornignon WP5 Leader 01-09-2017 M. Wisse WP6 Leader 30-08-2017 History of Changes Section, page number Change made Version 1.0 01-09-2017 3 732287 ROSIN D3.1 Quality Assurance and Community Management in ROS Executive Summary The Deliverable 3.1 “Quality Assurance Process and Community Management in ROS” explore the current state of Quality Assurance (QA) in the ROS and ROS-Industrial open source communities and understand the challenges with regards to the quality of robot applications based on ROS and containing ROS software. The task is addressed along three avenues: the analysis of the wiki and other material provided by the of the ROS and ROS-Industrial communities; interviews with different community members; and the analysis of bugs and their correction in ROS. The analytical results are complemented with initial interventions improving the continuous integration infrastructure and starting a quality hub, a web site that is meant to promote QA in the community. The chapter 2 reports the investigated ROS and ROS-Industrial current quality assurance (QA) practices and processes based on the ROS and ROS-Industrial wiki and infrastructure. Some industry-wide accepted QA practices and processes, like well-defined development process, defect management, code review, continuous integration and testing and knowledge sharing, have already been embraced and adopted especially in the evolution of core packages. However, our analysis suggest that QA practices are experiencing implementation and execution challenges. Some of the key issues are highlighted below: e.g. standards are not consistent across the community development roles; documentation is outdated; inconsistency in the QA practices across the various development streams (i.e. Core, Drivers and Reusable packages). The interview study reported in chapter 3 confirms and complements this analysis: It reports how different developers – core developers and maintainers, a developer of drivers and application developers perceive the QA challenges of developing ROS, respectively, developing robot applications based on ROS; QA is not consistently handled by all maintainers; the community is unclear about what are the currently practiced standards; there is no onboarding and knowledge transfer process in place for new members; the complexity of robotics development is not considered in the current implementation of QA processes and tools. Especially, there is a lack of ownership of QA in the ROS and ROS-Industrial communities. Chapter 4 on "ROS Quality Issues" describes the results of a quantitative analysis of 177 known bugs harvested from six different ROS systems and a few confidential systems. The results show that, currently, most ROS bugs eventually get fixed; i.e., very few bugs remain unfixed indefinitely. The results, however, also show that one in five bugs in ROS are detected at runtime, often by users, which is not good for the reputation of ROS. The results indicate that a CI (Continuous Integration) service simply compiling and building ROS projects automatically ought to catch a number of errors. Since one out of five errors occur as a result of software evolution, the CI service should take care in re-checking all code that interacts with recently modified code. Our results indicate that a wide range of bugs could be caught by extending the CI service to run a collection of different kinds of code-scanning tools. About 36% bugs would require difficult higher-level development techniques. A number of domain specific languages (DSL) are involved in bugs, which would require development of dedicated tools. Finally, a number of cross-language bugs that occur as a result of inconsistencies between the languages used. Again, these would require dedicated tools to be built. Chapter 5 describes two "early interventions" in the current ROS quality assurance processes and tools. The first contribution is the extension of the ROS wiki and buildfarm to more clearly display Version 1.0 01-09-2017 4 732287 ROSIN D3.1 Quality Assurance and Community Management in ROS the results of unit tests and continuous integration for all ROS packages that have been registered with the ROS buildfarm. Instead of only showing whether a package has enabled testing, status indicators on the wiki show the actual results of tests, which provides a much better insight into the status - and quality - of software components. The second contribution is the creation of the ROS Quality Hub: a ‘home’ for QA for the ROS and ROS-Industrial communities. As a start, it is populated with a database of patterns documenting QA practices and tools for ROS components. The ROS Quality Hub will be hosted on a publicly accessible website that will be disseminated among ROS and ROS-Industrial developers. The material published on the quality hub will be further developed throughout the runtime of ROSIN. Chapter 6 concludes the deliverable by outlining a number of possible directions of how to address the identified challenges: clarifying QA processes; making QA practices and tools easily available; making the quality of community contributed packages and drivers visible; supporting knowledge sharing and on-boarding among contributors; and developing and implementing code scanning tools. Version 1.0 01-09-2017 5 732287 ROSIN D3.1 Quality Assurance and Community Management in ROS Table of Contents Executive Summary 4 1. Introduction 10 1.1 Purpose of this document 10 1.2 ROS as a Software Product 10 1.3 History and Background of ROS 11 1.4 Outline of the Report 12 2. The ROS Community Quality Assurance 13 2.1 Introduction to chapter 13 2.2 Method 13 2.3 Results 14 2.3.1 Core development 14 REPs: ROS Enhancement Proposals 14 Issue tracking and maintenance routines 15 Continuous Integration 21 Unit test and ROStest 23 2.3.2 Reusable package, tool and driver 28 Initial Quality Assurance 28 Issue Tracking and Maintenance 34 Continuous Integration 34 Unit test and ROStest 34 Best practices, Tutorials, Standards and Q&A 35 2.3.3 Support for QA for Application Development 39 Continuous Integration and Industrial CI 39 Unit test and ROStest 41 Simulation 43 Best practices, Tutorials, Standards and Q&A 47 2.4 Sub-Conclusion: Issues with the QA practices 48 2.4.1 Core Development 49 3. ROS-Industrial community 53 3.1 Introduction to the chapter 53 Version 1.0 01-09-2017 6 732287 ROSIN D3.1 Quality Assurance and Community Management in ROS 3.2 Methods 53 3.2.1 The Interviewees 53 3.3 The status of ROS and ROS-I 54 3.3.1 What makes ROS and ROS-I attractive 54 3.3.2 ROS Quality 55 3.3.3 Activities of the ROS-Industrial community 56 3.3.4 ROS and ROS2 57 3.4 Quality in the community development process 58 3.4.1 Communication between maintainers and developers 59 Issue Tracking and Pull Requests 59 The Wiki 59 Community Fora 60 3.4.2 Pull request reviews and peer reviews 60 3.4.3 Importance of Continuous Integration 61 3.4.4 Package release and maintenance by community members 61 3.4.5 Summary: Quality in the community process 61 3.5 Quality Challenges 61 3.5.1 Challenges from the core developers’ point of view 62 Lack of overview for new contributors. 62 Heterogeneous quality criteria of maintainers 62 Maintenance effort 62 Lack of Maintainers 63 Unmaintained packages 63 Errors that only show after some time of deployment 63 3.5.2 Challenges from the driver developer’s point of view 63 Connection to Hardware 63 Quality of architectural design 64 Lack of established corporate processes to quality assure open-source contributions 64 3.5.3 Challenges from the application development point