João André Nunes Agnelo REST
Total Page:16
File Type:pdf, Size:1020Kb
João André Nunes JoãoAgnelo Nunes André ERVICES S EB W FUL João André Nunes Agnelo REST FOR APPROACH A ROBUSTNESS TESTING APPROACH FOR RESTFUL WEB SERVICES ROBUSTNESS TESTING TESTING ROBUSTNESS A Dissertation in the context of the Master in Informatics Engineering, Specialization in Software Engineering, advised by Prof. Nuno Laranjeiro and Prof. Jorge Bernardino and presented to Faculty of Sciences and Technology / Department of Informatics Engineering. September 2020 Faculty of Sciences and Technology Department of Informatics Engineering A robustness testing approach for RESTful Web Services João André Nunes Agnelo Dissertation in the context of the Master in Informatics Engineering, Specialization in Software Engineering advised by Prof. Nuno Laranjeiro and Prof. Jorge Bernardino and presented to the Faculty of Sciences and Technology / Department of Informatics Engineering. September 2020 This page is intentionally left blank. Abstract Robustness is the degree to which a certain system or component can operate correctly in the presence of invalid inputs or stressful environmental conditions. With the increasing complexity and wide-spread use of software systems, obtaining assurances regarding their robustness has become of vital importance. This is especially true in mission- or business- critical systems where a failure may have severe consequences on the business or in human lives. Web services, in particular, are often exposed to abnormal and malicious input that could lead to serious issues, such as loss of data and information disclosure. Most modern web companies, including Google, Instagram, Spotify and Slack, are sup- ported by REST services (also known as REST APIs), a type of software system that follows the relatively loose REST architectural style. In this type of scenarios, hetero- geneity is prevalent and software is sometimes exposed to unexpected conditions that may activate residual bugs, leading service operations to fail, which may result in financial or reputation losses (e.g., information disclosure). Robustness is, therefore, a key property in REST services. In this dissertation, we present a systematic literature review on software robustness test- ing, and an approach and tool (named bBOXRT) for performing robustness tests over REST services. We begin by performing a comprehensive analysis, in the form of a sys- tematic literature review, of the state of the art on software robustness testing, which we complement with an overview of the related work on REST API testing. This allows us to show that multiple techniques and tools for robustness assessment have been thoroughly studied and applied to a large diversity of domains, but REST services still lack practi- cal approaches that specialize in robustness evaluation. We fill-in this gap by proposing bBOXRT, a tool for testing the robustness of REST services solely based on minimal information present in their interface descriptions. We used bBOXRT to evaluate an heterogeneous set of 52 REST services that comprise 1351 operations and fit in distinct categories (e.g., public, private, in-house). We were able to disclose several different types of robustness problems, including issues in services with strong reliability requirements and also a few security vulnerabilities. The results show that REST services are being deployed online holding software defects that harm service integration, and also carrying security vulnerabilities that can be exploited by malicious users. Keywords Software testing, software robustness, robustness evaluation, web services, RESTful web services, REST API iii This page is intentionally left blank. Resumo Robustez é o grau com que um certo sistema ou componente pode operar corretamente na presença de entradas inválidas ou condições de stress. Com o aumento da complexidade e o uso alargado de sistemas de software, obter garantias quanto à sua robustez tornou-se uma tarefa essencial. Isto é especialmente verdade nos sistemas críticos (quanto à sua missão ou ao seu objetivo de negócio) onde uma falha pode ter consequências graves no negócio ou em vidas humanas. Os serviços web, em particular, são frequentemente expostos a entradas inesperadas ou maliciosas que podem levar a problemas graves, tais como perda de dados e divulgação de informação. A maioria das empresas web modernas, incluindo a Google, o Instagram, o Spotify e o Slack, são suportadas por serviços REST (também conhecidos como APIs REST), um tipo de sistema de software que segue o estilo arquitetural REST. Neste tipo de cenários, a heterogeneidade é dominante e o software é, por vezes, exposto a condições inesperadas que poderão ativar bugs residuais, levando à falha do serviço, o que poderá resultar em perdas financeiras ou de reputação (e.g., divulgação de informação). A robustez é, como tal, uma propriedade essencial em serviços REST. Nesta tese, apresentamos uma revisão sistemática da literatura sobre testes de robustez em software, e uma abordagem e uma ferramenta (bBOXRT) para a execução de testes de robustez em serviços REST. Começamos por realizar uma análise exaustiva, sob a forma de uma revisão sistemática da literatura, do estado da arte sobre testes de robustez em soft- ware, que complementamos com uma visão geral do trabalho relacionado sobre testes em APIs REST. Isto permite-nos mostrar que, embora várias técnicas e ferramentas para avali- ação de robustez já tenham sido estudadas em detalhe e aplicadas a uma grande variedade de domínios, os serviços REST ainda necessitam de abordagens práticas especializadas na avaliação de robustez. Propomos preencher esta lacuna com a bBOXRT, uma ferramenta para testar a robustez de serviços REST que necessita apenas de uma pequena fração da informação presente nas suas descrições de interface. A bBOXRT foi usada para avaliar um conjunto heterogéneo de 52 serviços REST que englobam 1351 operações e se inserem em diferentes categorias (e.g., públicos, privados, in-house). Com esta avaliação, conseguimos revelar vários tipos de problemas de robustez, incluindo problemas em serviços com elevada necessidade de confiabilidade e também al- gumas vulnerabilidades de segurança. Os resultados mostram que os serviços REST estão a ser postos em operação online contendo ainda defeitos de software que prejudicam a sua integração, e também apresentam vulnerabilidades de segurança que podem ser exploradas por utilizadores maliciosos. Palavras-Chave Testes de software, robustez de software, avaliação de robustez, serviços web, serviços web RESTful, API REST v This page is intentionally left blank. Acknowledgments This work has been partially supported by the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement No. 823788 (project ADVANCE); by the project METRICS, funded by the Portuguese Foundation for Science and Technology (FCT) – agreement No. POCI-01-0145-FEDER-032504; by the project MobiWise: From mobile sensing to mobility advising (reference: P2020 SAICT- PAC/0011/2015), co-financed by COMPETE 2020, Portugal 2020 - Operational Program for Competitiveness and Internationalization (POCI), European Union’s ERDF (European Regional Development Fund), and the Portuguese Foundation for Science and Technology (FCT); and by the TalkConnect project “Voice Architecture over Distributed Network” (reference: POCI-01-0247-FEDER-039676), co-financed by the European Regional Devel- opment Fund, through Portugal 2020 (PT2020), and by the Competitiveness and Interna- tionalization Operational Programme (COMPETE 2020). Regarding personal acknowledgments, I would like to thank my MSc thesis advisors Prof. Nuno Laranjeiro and Prof. Jorge Bernardino for their endless patience and wisdom in this long-lasting endeavor. This work has given me much knowledge regarding how to formally approach a research proposal, and it has made me grow as an engineer. One particular takeaway from this work, is that in the future I must apply greater effort to the planning stage of a large project such as this, as time is the most valuable resource. vii This page is intentionally left blank. Contents 1 Introduction 1 2 Background 5 2.1 REST architecture . .5 2.2 Software testing concepts . .8 3 Related work on software robustness evaluation 13 3.1 Systematic review plan . 13 3.1.1 Analysis of related surveys . 14 3.1.2 Systematic review research questions . 15 3.1.3 Identification of studies . 16 3.1.4 Study selection and quality assessment . 17 3.1.5 Data extraction and synthesis . 17 3.1.6 Outcome of the study identification and selection . 18 3.2 Analysis of studies on software robustness evaluation . 19 3.2.1 Operating systems . 19 3.2.2 Communication systems . 27 3.2.3 Embedded systems . 31 3.2.4 Middleware . 38 3.2.5 Software components . 42 3.2.6 Web services . 49 3.2.7 Autonomous and Adaptive systems . 55 3.3 Discussion . 58 3.4 Highlights and research challenges . 63 4 Related work on REST service testing 67 4.1 Studies on REST service testing . 67 4.2 Tools for REST service testing . 72 4.3 Discussion . 80 5 Approach and robustness testing tool architecture 85 5.1 Approach overview . 85 5.2 Tool architecture and operation . 86 6 Experimental evaluation 92 6.1 Experiments description . 92 6.2 Experimental results . 94 6.3 Main findings . 101 7 Threats to validity 104 8 Conclusions and future work 107 ix This page is intentionally left blank. Acronyms API Application Programming Interface. 1, 2, 6–8, 22–25, 27, 34, 37, 46, 59–61, 64, 65, 67–72, 74–83, 85, 86, 107, 130, 131 bBOXRT black-BOX tool for Robustness Testing of rest services. 2, 3, 85, 86, 89, 92–94, 107, 131 BPEL Business Process Execution Language. 52, 54, 74, 80 CORBA Common Object Request Broker Architecture. 40, 42 COTS Commercial Off-The-Shelf. 19, 20, 22, 24, 33, 42, 43, 46, 48, 59 DBMS Database Management System. 43 DDS Data Distribution Service. 41, 42 DoS Denial-of-Service. 26, 41 DPI Driver Programming Interface. 24, 25 DSL Domain-Specific Language. 75, 78 FSM Finite State Machines. 29, 31, 37 FTP File Transfer Protocol. 74 GUI Graphical User Interface. 20, 47, 77 HA High Availability.