Discover Knowledge on FLOSS Projects Through Repofinder
Total Page:16
File Type:pdf, Size:1020Kb
Discover Knowledge on FLOSS Projects Through RepoFinder Francesca Arcelli Fontana1, Riccardo Roveda2 and Marco Zanoni1 1University of Milano-Bicocca, Milano, Italy 2 KIE S.r.l, Milano, Italy Keywords: Knowledge Discovery, Project Discovery, FLOSS Projects, Full-text Search. Abstract: We can retrieve and integrate knowledge of different kinds. In this paper, we focus our attention on FLOSS (Free, Libre and Open Source Software) projects. With this aim, we introduce RepoFinder, a web application we have developed for the discovery, retrieval and analysis of open source software. RepoFinder supports a keyword-based discovery process for FLOSS projects through google-like queries. Moreover, it allows to analyze the projects according to well-known software metrics and other features of the code, and to com- pare some structural aspects of the different projects. In the paper, we focus on the discovery capabilities of RepoFinder, evaluating them on different project categories and comparing them with a well-known search engine as Google. 1 INTRODUCTION aged by different portals and with different versioning technologies, e.g., Git, SVN, Mercurial (HG), CVS, Through the Web we are able to retrieve a huge GNU Bazaar. Moreover, to choose a project for inte- amount of data and information of different kinds. grating a FLOSS component, it is often necessary to Among them FLOSS (Free, Libre and Open Source spend significant time inspecting the project activity, Software) projects and applications are receiving a releases, and web forums or communities explaining continuous increasing interest across different com- strength and weaknesses of these projects. In this tra- munities, both in the industry, public administra- ditional way of evaluating new components to inte- tion and research institutions. FLOSS projects are grate in a software, there is very little knowledge of managed on dedicated web platforms, called Code the quality of the project, design, code and develop- Forges (Squire and Williams, 2012), designed to col- ment team. lect, promote, and categorize FLOSS projects, and We started facing these issues, by developing a to support development teams. Some examples are web application, called RepoFinder. Its aim is to be a Launchpad, Google Code, SourceForge, Github, Bit- platform able to support the discovery, retrieval, anal- bucket and GNU Savannah. Code Forges provide ysis and comparison of FLOSS projects. In the cur- tools and services for distributed development teams, rent development stage, we started supporting FLOSS e.g., version control systems, bug tracking systems, project discovery, retrieval and analysis. RepoFinder web hosting space. Currently, Github hosts more than is currently able to perform the following tasks: ten million software components, and is probably the • Crawling of widely known Code Forges (Github1, largest. SourceForge is one of the first Code Forges, SourceForge2, Launchpad3 and Google Code4), created in 1999, with more than 425,000 projects, all through their APIs or by parsing their web pages. of them fitting a categorization, while Google Code During this activity, projects contained in the hosts about 250,000 projects. Code Forges are indexed, and tagged with labels To find a project satisfying specific requirements, and other available metadata (e.g., tags, forks, we can start from a simple query on a generic search stars), depending on the particular Code Forge. engine or on each Code Forge. In this case, the risk is receiving many useless answers, because FLOSS 1https://github.com project names are often ambiguous and difficult to re- 2http://sourceforge.net cover. Even retrieving the source code of a partic- 3https://launchpad.net ular project can be difficult, because it may be man- 4https://code.google.com Arcelli Fontana F., Roveda R. and Zanoni M.. 485 Discover Knowledge on FLOSS Projects Through RepoFinder. DOI: 10.5220/0005156704850491 In Proceedings of the International Conference on Knowledge Discovery and Information Retrieval (KDIR-2014), pages 485-491 ISBN: 978-989-758-048-2 Copyright c 2014 SCITEPRESS (Science and Technology Publications, Lda.) KDIR2014-InternationalConferenceonKnowledgeDiscoveryandInformationRetrieval Currently, we indexed more than 450,000 projects • SonarQube5 (Arapidis, 2012; Campbell and Pa- and found more than 20,200 labels. papetrou, 2013) • Browsing projects through a search engine based – Similarities: performs different types of analy- on simple google-like queries. It is possible to sis, integrating different external tools; exploits search into the projects’ descriptions, names and also Checkstyle and PMD. all the available metadata. The code of the found – Differences: it can be used like a plugin of projects can be retrieved through their version Eclipse; SonarQube does not support projects control system repository. discovery, but every project must be added to • Analysis of the projects’ code. We integrated ex- the system. In addition to Java, it allows ana- ternal software for the computation of metrics and lyzing other languages by integrating different the detection of code smells (Fowler, 1999; Ar- plugins. celli Fontana et al., 2012) in Java code. Cur- • SECONDA (Prez et al., 2012)(Software Ecosys- rently, the following tools are integrated: Check- tem Analysis Dashboard) style (Ivanov and Sopov, 2014), PMD (Dangel, – Similarities: performs analyses and computes 2014) for code smell detection, NICAD (Roy and metrics on locally stored Git repositories. Uses Cordy, 2008) for clone detection, JDepend (Clark- a web GUI to visualize the results. ware Consulting Inc., 2014), Understand (Sci- – Differences: analyzes C code and allows to entific Toolworks, Inc., 2014), JavaNCSS (Lee, download code only using Git. 2014) and another tool developed at our Labo- ratory (ESSeRE-Evolution of Software Systems • Ohloh.net (Black Duck Software, 2014) and Reverse Engineering Lab), called DFMC4J, – Similarities: provides a search engine that use for metrics computation. tags and labels to improve performance. • Integration of data gathered by the code analyz- – Differences: computes few metrics (LOC and ers in a dataset, to be used for statistical analyses. number of commits). In Ohloh, any user can In the resulting dataset, data extracted by the an- freely modify labels, tags and other information alyzers is merged in a uniform format, allowing, about the indexed site. RepoFinder, instead, di- e.g., to compare the analyses performed by differ- rectly reports the information published on the ent tools on the same code artifact. official Code Forge page. • GlueTheros (Robles et al., 2004) In this paper, we describe the functionalities and the architecture of RepoFinder, we summarize the – Similarities: calculates metrics code reposito- amount and characteristics of the projects we in- ries repository and shows results using a Web dexed during the crawling phase, and we evaluate GUI. the discovery capabilities of the tool, by simulating – Differences: supports CVS only. In addiction a FLOSS project discovery use case, repeated on dif- to Java, it analyzes other languages. It does not ferent project categories. support the discovery of new projects. The paper is organized through the following sec- • Complicity (Neu et al., 2011) tions: In Section 2 we describe some related work, in Section 3 we introduce the main functionalities of- – Similarities: computes metrics from a project’s fered by RepoFinder and its software architecture; in repository. Extracts all projects from a single Section 4 we show the data we collected in the crawl- repository. Uses a dashboard to visualize re- ing phase; in Section 5 we evaluate a project discov- sults. ery use case. Finally, in Section 6 we conclude and – Differences: computes only few repository’s outline some future developments. metrics (e.g., number of commits, number of files). It does not support project discovery, but is initialized with an input project ecosys- tem. The supported analyses focus on the con- 2 RELATED WORK tributions the developers give to the ecosystem, rather than the source code and its quality. Other applications exist with functionalities similar to There are also a number of datasets (Van Antwerp the ones of RepoFinder. We cite below some of them, and Madey, 2008; Howison et al., 2006; Gousios, outlining the main similarities and differences with 2013) that gather data coming from different Code our tool. 5http://nemo.sonarqube.org/ 486 DiscoverKnowledgeonFLOSSProjectsThroughRepoFinder Forges, which could be integrated in the crawling and analysis processes of RepoFinder. However, their availability is variable and, unfortunately, some of these data-collection projects are closed. 3 REPOFINDER In this section, we introduce the overall architecture of RepoFinder and we briefly describe its main func- tionalities. 3.1 RepoFinder Functionalities Figure 1: RepoFinder architecture RepoFinder provides five different functionalities: • Crawling: it allows to find FLOSS projects on the • Search engine: indexes the projects collection and different Code Forges, and to save, update and in- performs the query execution. It exploits Lucene6 dex their information in a data store local to the to create and query the projects’ index. application. • Analyzer and project builder: downloads source • Indexing: it allows to obtain better results during code from the projects repositories; builds the the projects search and has been developed using projects and, after running all tools, saves all the the Lucene external