DOCSLIB.ORG

Fuzzing in the Bigger Picture

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Fuzzing in the Bigger Picture SM2-TES: Functional Programming and Property-Based Testing, Day 11 Jan Midtgaard MMMI, SDU Outline Motivation Challenges in C Programming Fuzz Testing Fuzzing in the Bigger Picture 2/44 Motivation Positive vs. negative testing QuickCheck (as we’ve covered so far) focuses mainly on positive testing: Testing that systems/programs/libraries act as desired on valid input. These days it is increasingly important to also perform negative testing: Testing that systems/programs/libraries also act meaningfully on invalid input. Security issues typically fall in the second category: SQL or JS injections, buffer overflows, . Many of these are memory safety issues 4/44 A real problem in practice! 2014 - Heartbleed was a security bug in the OpenSSL cryptography library From https://en.wikipedia.org/wiki/Heartbleed 5/44 A real problem in practice! https://www.zdnet.com/article/microsoft-70-percent-of-all-security-bugs-are-memory-safety-issues/ 6/44 A real problem in practice! https://www.zdnet.com/article/chrome-70-of-all-security-bugs-are-memory-safety-issues/ 6/44 2020 Most Dangerous Software Weaknesses Here’s the 2020 CWE Top 25: # ID Name 1 79 Improper Neutralization of Input During Web Page Generation (XSS) 2 787 Out-of-bounds Write 3 20 Improper Input Validation 4 125 Out-of-bounds Read 5 119 Improper Restriction of Operations within the Bounds of a Memory Buffer 6 89 Improper Neutralization of Special Elements used in an SQL Command 7 200 Exposure of Sensitive Information to an Unauthorized Actor 8 416 Use After Free 9 352 Cross-Site Request Forgery (CSRF) 10 78 Improper Neutralization of Special Elements used in an OS Command 11 190 Integer Overflow or Wraparound 12 22 Improper Limitation of a Pathname to a Restricted Directory 13 476 NULL Pointer Dereference 14 287 Improper Authentication 15 434 Unrestricted Upload of File with Dangerous Type . http://cwe.mitre.org/top25/archive/2020/2020_cwe_top25.html 7/44 Memory safety is a security concern These are mostly caused by using C and C++ which (unlike OCaml and Java) are memory unsafe languages To address these challenges working software engineers have developed: Sanitizers – to help raise alarms Fuzz testing (or fuzzing) – another randomized testing approach(!) Today we will study both. First: Let’s better understand why this is an issue... 8/44 Challenges in C Programming The challenges of C programming. (1/2) Writing C code is challenging: C’s manual memory management requires you to: allocate chunks of memory with malloc (or similar) release the allocated chunks again with free Furthermore: Allocated memory is (generally) not initialized for you Indexing out of bounds is not caught... What could possibly go wrong? 10 / 44 The challenges of C programming. (2/2) Writing safe and proper C code is error prone: writing out-of-bounds in the heap or stack reading out-of-bounds in the heap or stack uninitialized reads use-after-free double freeing . Historically compilers did not help catch these. With a crash you would be lucky. More likely: program sometimes exhibits weird behavior 11 / 44 Enter ASan Google software engineers at some point designed AddressSanitizer (ASan) which was a game changer. https://github.com/google/sanitizers/wiki/AddressSanitizer http://clang.llvm.org/docs/AddressSanitizer.html ASan’s job is simply to detect memory addressing errors In a language like Java we are spoiled and take, e.g., IndexOutOfBounds exceptions, for granted. These days LLVM (and GCC) has ASan support built in. LLVM supports Linux, *BSD, Mac, Android, Windows Historically, a range of tools predated ASan: Valgrind, Electric Fence, ... 12 / 44 An example C program #include <stdio.h> #include <stdlib.h> int main() { char *p = malloc(5); p[0] = 'h'; p[1] = 'e'; p[2] = 'l'; p[3] = 'l'; p[4] = 'o'; p[5] = '\0'; /* this index is out of bounds */ printf("%s\n",p); return 0; } 13 / 44 An example C program #include <stdio.h> #include <stdlib.h> int main() { char *p = malloc(5); p[0] = 'h'; p[1] = 'e'; p[2] = 'l'; p[3] = 'l'; p[4] = 'o'; p[5] = '\0'; /* this index is out of bounds */ printf("%s\n",p); return 0; } $ clang -Wall -Wextra -pedantic -o example2 example2.c $ ./example2 hello $ Hm, it runs without any flags raised... 13 / 44 Same example compiled with ASan $ clang -Wall -Wextra -pedantic -g -o example2 -fsanitize=address example2.c $ 14 / 44 Same example compiled with ASan $ clang -Wall -Wextra -pedantic -g -o example2 -fsanitize=address example2.c $ ./example2 ================================================================= ==4922==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000000b5 at pc 0x000102604df5 bp 0x7fff5d5fb7d0 sp 0x7fff5d5fb7c8 WRITE of size 1 at 0x6020000000b5 thread T0 #0 0x102604df4 in main example2.c:12 #1 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234) 0x6020000000b5 is located 0 bytes to the right of 5-byte region [0x6020000000b0,0x6020000000b5) allocated by thread T0 here: #0 0x102666e9c in wrap_malloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x58e9c) #1 0x102604bfa in main example2.c:6 #2 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234) SUMMARY: AddressSanitizer: heap-buffer-overflow example2.c:12 in main Shadow bytes around the buggy address: 0x1c03ffffffc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03fffffff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c0400000000: fa fa fd fd fa fa fd fd fa fa 00 04 fa fa 00 00 =>0x1c0400000010: fa fa 00 06 fa fa[05]fa fa fa fa fa fa fa fa fa 0x1c0400000020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [...] ==4922==ABORTING Abort trap: 6 14 / 44 Same example compiled with ASan $ clang -Wall -Wextra -pedantic -g -o example2 -fsanitize=address example2.c $ ./example2 ================================================================= ==4922==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000000b5 at pc 0x000102604df5 bp 0x7fff5d5fb7d0 sp 0x7fff5d5fb7c8 WRITE of size 1 at 0x6020000000b5 thread T0 #0 0x102604df4 in main example2.c:12 #1 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234)Line causing overflow 0x6020000000b5 is located 0 bytes to the right of 5-byte region [0x6020000000b0,0x6020000000b5) allocated by thread T0 here: #0 0x102666e9c in wrap_malloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x58e9c) #1 0x102604bfa in main example2.c:6 #2 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234) SUMMARY: AddressSanitizer: heap-buffer-overflow example2.c:12 in main Shadow bytes around the buggy address: 0x1c03ffffffc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03fffffff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c0400000000: fa fa fd fd fa fa fd fd fa fa 00 04 fa fa 00 00 =>0x1c0400000010: fa fa 00 06 fa fa[05]fa fa fa fa fa fa fa fa fa 0x1c0400000020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [...] ==4922==ABORTING Abort trap: 6 14 / 44 Same example compiled with ASan $ clang -Wall -Wextra -pedantic -g -o example2 -fsanitize=address example2.c $ ./example2 ================================================================= ==4922==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000000b5 at pc 0x000102604df5 bp 0x7fff5d5fb7d0 sp 0x7fff5d5fb7c8 WRITE of size 1 at 0x6020000000b5 thread T0 #0 0x102604df4 in main example2.c:12 #1 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234)Line causing overflow 0x6020000000b5 is located 0 bytes to the right of 5-byte region [0x6020000000b0,0x6020000000b5) allocated by thread T0 here: #0 0x102666e9c in wrap_malloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x58e9c) #1 0x102604bfa in main example2.c:6 #2 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234)Line that allocated SUMMARY: AddressSanitizer: heap-buffer-overflow example2.c:12 in main Shadow bytes around the buggy address: 0x1c03ffffffc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03fffffff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c0400000000: fa fa fd fd fa fa fd fd fa fa 00 04 fa fa 00 00 =>0x1c0400000010: fa fa 00 06 fa fa[05]fa fa fa fa fa fa fa fa fa 0x1c0400000020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [...] ==4922==ABORTING Abort trap: 6 14 / 44 ASan doesn’t catch everything For example: #include <stdio.h> #include <stdlib.h> int main(int argc, char** argv) /* argc is an argument counter */ { char a[10]; /* stack allocation */ a[5] = 0; printf("running %s argc is %i\n", argv[0], argc); if (a[argc]) /* Reading uninitialized memory */ printf("what?
Load more

Recommended publications
  • Effectiveness of Software Testing Techniques in Enterprise: a Case Study
    MYKOLAS ROMERIS UNIVERSITY BUSINESS AND MEDIA SCHOOL BRIGITA JAZUKEVIČIŪTĖ (Business Informatics) EFFECTIVENESS OF SOFTWARE TESTING TECHNIQUES IN ENTERPRISE: A CASE STUDY Master Thesis Supervisor – Assoc. Prof. Andrej Vlasenko Vilnius, 2016 CONTENTS INTRODUCTION .................................................................................................................................. 7 1. THE RELATIONSHIP BETWEEN SOFTWARE TESTING AND SOFTWARE QUALITY ASSURANCE ........................................................................................................................................ 11 1.1. Introduction to Software Quality Assurance ......................................................................... 11 1.2. The overview of Software testing fundamentals: Concepts, History, Main principles ......... 20 2. AN OVERVIEW OF SOFTWARE TESTING TECHNIQUES AND THEIR USE IN ENTERPRISES ...................................................................................................................................... 26 2.1. Testing techniques as code analysis ....................................................................................... 26 2.1.1. Static testing ...................................................................................................................... 26 2.1.2. Dynamic testing ................................................................................................................. 28 2.2. Test design based Techniques ...............................................................................................
    [Show full text]
  • Studying the Feasibility and Importance of Software Testing: an Analysis
    Dr. S.S.Riaz Ahamed / Internatinal Journal of Engineering Science and Technology Vol.1(3), 2009, 119-128 STUDYING THE FEASIBILITY AND IMPORTANCE OF SOFTWARE TESTING: AN ANALYSIS Dr.S.S.Riaz Ahamed Principal, Sathak Institute of Technology, Ramanathapuram,India. Email:[email protected], [email protected] ABSTRACT Software testing is a critical element of software quality assurance and represents the ultimate review of specification, design and coding. Software testing is the process of testing the functionality and correctness of software by running it. Software testing is usually performed for one of two reasons: defect detection, and reliability estimation. The problem of applying software testing to defect detection is that software can only suggest the presence of flaws, not their absence (unless the testing is exhaustive). The problem of applying software testing to reliability estimation is that the input distribution used for selecting test cases may be flawed. The key to software testing is trying to find the modes of failure - something that requires exhaustively testing the code on all possible inputs. Software Testing, depending on the testing method employed, can be implemented at any time in the development process. Keywords: verification and validation (V & V) 1 INTRODUCTION Testing is a set of activities that could be planned ahead and conducted systematically. The main objective of testing is to find an error by executing a program. The objective of testing is to check whether the designed software meets the customer specification. The Testing should fulfill the following criteria: ¾ Test should begin at the module level and work “outward” toward the integration of the entire computer based system.
    [Show full text]
  • Manual on Quality Assurance for Computer Software Related to the Safety of Nuclear Power Plants
    SIMPLIFIED SOFTWARE LIFE-CYCLE DIAGRAM FEASIBILITY STUDY PROJECT TIME I SOFTWARE P FUNCTIONAL I SPECIFICATION! SOFTWARE SYSTEM DESIGN DETAILED MODULES CECIFICATION MODULES DESIGN SOFTWARE INTEGRATION AND TESTING SYSTEM TESTING ••COMMISSIONING I AND HANDOVER | DECOMMISSION DESIGN DESIGN SPECIFICATION VERIFICATION OPERATION AND MAINTENANCE SOFTWARE LIFE-CYCLE PHASES TECHNICAL REPORTS SERIES No. 282 Manual on Quality Assurance for Computer Software Related to the Safety of Nuclear Power Plants f INTERNATIONAL ATOMIC ENERGY AGENCY, VIENNA, 1988 MANUAL ON QUALITY ASSURANCE FOR COMPUTER SOFTWARE RELATED TO THE SAFETY OF NUCLEAR POWER PLANTS The following States are Members of the International Atomic Energy Agency: AFGHANISTAN GUATEMALA PARAGUAY ALBANIA HAITI PERU ALGERIA HOLY SEE PHILIPPINES ARGENTINA HUNGARY POLAND AUSTRALIA ICELAND PORTUGAL AUSTRIA INDIA QATAR BANGLADESH INDONESIA ROMANIA BELGIUM IRAN, ISLAMIC REPUBLIC OF SAUDI ARABIA BOLIVIA IRAQ SENEGAL BRAZIL IRELAND SIERRA LEONE BULGARIA ISRAEL SINGAPORE BURMA ITALY SOUTH AFRICA BYELORUSSIAN SOVIET JAMAICA SPAIN SOCIALIST REPUBLIC JAPAN SRI LANKA CAMEROON JORDAN SUDAN CANADA KENYA SWEDEN CHILE KOREA, REPUBLIC OF SWITZERLAND CHINA KUWAIT SYRIAN ARAB REPUBLIC COLOMBIA LEBANON THAILAND COSTA RICA LIBERIA TUNISIA COTE D'lVOIRE LIBYAN ARAB JAMAHIRIYA TURKEY CUBA LIECHTENSTEIN UGANDA CYPRUS LUXEMBOURG UKRAINIAN SOVIET SOCIALIST CZECHOSLOVAKIA MADAGASCAR REPUBLIC DEMOCRATIC KAMPUCHEA MALAYSIA UNION OF SOVIET SOCIALIST DEMOCRATIC PEOPLE'S MALI REPUBLICS REPUBLIC OF KOREA MAURITIUS UNITED ARAB
    [Show full text]
  • Test-Driven Development in Enterprise Integration Projects
    Test-Driven Development in Enterprise Integration Projects November 2002 Gregor Hohpe Wendy Istvanick Copyright ThoughtWorks, Inc. 2002 Table of Contents Summary............................................................................................................. 1 Testing Complex Business Applications......................................................... 2 Testing – The Stepchild of the Software Development Lifecycle?............................................... 2 Test-Driven Development............................................................................................................. 2 Effective Testing........................................................................................................................... 3 Testing Frameworks..................................................................................................................... 3 Layered Testing Approach ........................................................................................................... 4 Testing Integration Solutions............................................................................ 5 Anatomy of an Enterprise Integration Solution............................................................................. 5 EAI Testing Challenges................................................................................................................ 6 Functional Testing for Integration Solutions................................................................................. 7 EAI Testing Framework ..................................................................................
    [Show full text]
  • Continuous Quality and Testing to Accelerate Application Development
    Continuous Quality and Testing to Accelerate Application Development How to assess your current testing maturity level and practice continuous testing for DevOps Continuous Quality and Testing to Accelerate Application Development // 1 Table of Contents 03 Introduction 04 Why Is Continuous Quality and Testing Maturity Important to DevOps? 05 Continuous Testing Engineers Quality into DevOps 07 Best Practices for Well- Engineered Continuous Testing 08 Continuous Testing Maturity Levels Level 1: Chaos Level 2: Continuous Integration Level 3: Continuous Flow Level 4: Continuous Feedback Level 5: Continuous Improvement 12 Continuous Testing Maturity Assessment 13 How to Get Started with DevOps Testing? 14 Continuous Testing in the Cloud Choosing the right tools for Continuous Testing On-demand Development and Testing Environments with Infrastructure as Code The Right Tests at the Right Time 20 Get Started 20 Conclusion 21 About AWS Marketplace and DevOps Institute 21 Contributors Introduction A successful DevOps implementation reduces the bottlenecks related to testing. These bottlenecks include finding and setting up test environments, test configurations, and test results implementation. These issues are not industry specific. They can be experienced in manufacturing, service businesses, and governments alike. They can be reduced by having a thorough understanding and a disciplined, mature implementation of Continuous Testing and related recommended engineering practices. The best place to start addressing these challenges is having a good understanding of what Continuous Testing is. Marc Hornbeek, the author of Engineering DevOps, describes it as: “A quality assessment strategy in which most tests are automated and integrated as a core and essential part of DevOps. Continuous Testing is much more than simply ‘automating tests.’” In this whitepaper, we’ll address the best practices you can adopt for implementing Continuous Quality and Testing on the AWS Cloud environment in the context of the DevOps model.
    [Show full text]
  • Devops Point of View an Enterprise Architecture Perspective
    DevOps Point of View An Enterprise Architecture perspective Amsterdam, 2020 Management summary “It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change.”1 Setting the scene Goal of this Point of View In the current world of IT and the development of This point of view aims to create awareness around the IT-related products or services, companies from transformation towards the DevOps way of working, to enterprise level to smaller sizes are starting to help gain understanding what DevOps is, why you need it use the DevOps processes and methods as a part and what is needed to implement DevOps. of their day-to-day organization process. The goal is to reduce the time involved in all the An Enterprise Architecture perspective software development phases, to achieve greater Even though it is DevOps from an Enterprise Architecture application stability and faster development service line perspective, this material has been gathered cycles. from our experiences with customers, combined with However not only on the technical side of the knowledge from subject matter experts and theory from organization is DevOps changing the playing within and outside Deloitte. field, also an organizational change that involves merging development and operations teams is Targeted audience required with an hint of cultural changes. And last but not least the skillset of all people It is specifically for the people within Deloitte that want to involved is changing. use this as an accelerator for conversations and proposals & to get in contact with the people who have performed these type of projects.
    [Show full text]
  • API Integration Tutorial: Testing, Security and API Management
    API Integration Tutorial: Testing, security and API management Tutorial In this tutorial In this tutorial: A roundup of the leading API As application program interface integration increases, so do the management tools available challenges with maintaining management, testing, and security. today………………………………..…. p.2 This API integration tutorial compares leading API management What are some solid options tools currently available on the market, as well as strategies for for open source API RESTful API testing. management tools?.............. p.14 Get a better understanding of API integration and management The basics of establishing a strategies. RESTful API testing program ……………………………………………. p.17 Testing microservices and APIs in the cloud……………... p.24 How do I create a secure API for mobile?............................... p.29 About SearchMicroservices.com ……………………………………………. p.31 Page 1 of 31 Tutorial In this tutorial A roundup of the leading API management A roundup of the leading API tools available today management tools available today………………………………..…. p.2 Zachary Flower, Freelance writer, SearchMicroservices.com API management is a constantly growing market with more new products What are some solid options popping up every year. As a result, narrowing all of the potential choices for open source API down to one perfect platform can feel like an overwhelming task. The sheer management tools?.............. p.14 number of options out there makes choosing one an extremely tough decision. The basics of establishing a To help relieve the strain involved with this decision, we've put together this RESTful API testing program detailed product roundup covering 10 of the leading API management tools ……………………………………………. p.17 currently available on the market.
    [Show full text]
  • Accessibility Testing in Agile Software Development
    Accessibility Testing in Agile Software Development Empowering agile developers with accessibility methods Nikolai Sverdrup Thesis submitted for the degree of Master in Programming and Networks 60 credits Department of Informatics Faculty of mathematics and natural sciences UNIVERSITY OF OSLO Autumn 2018 Accessibility Testing in Agile Software Development Empowering agile developers with accessibility methods Nikolai Sverdrup c 2018 Nikolai Sverdrup Accessibility Testing in Agile Software Development http://www.duo.uio.no/ Printed: Reprosentralen, University of Oslo Abstract This thesis delves into some of the methods that can be used to do accessibility testing in software development. Challenging traditional conventions of working for accessibility in agile development, where slow and resource intensive methods dominate, I investigate several methods that could ease the task of creating accessible software. I will mainly be focusing on web accessibility in tandem with a wider world effort working on the same goal. I have deployed several methods to professional software developers working in many different specialties. There are five methods I deploy and evaluate: A type of glasses that emulate bad eyesight, a software tool that analyzes and reports accessibility faults on websites, Screen-readers used a debugging tool, a method where the testers act out the user experience of imagined disabled users and a dyslectic emulation tool. There is also a smaller evaluation of using traditional WCAG testing. A case study was used to try and get a closer look at real life accessibility testing and aiding in finding out what obstacles accessibility testing need to climb if the testing methods are to be deployed successfully.
    [Show full text]
  • Beginners Guide to Software Testing
    Beginners Guide To Software Testing Beginners Guide To Software Testing - Padmini C Page 1 Beginners Guide To Software Testing Table of Contents: 1. Overview ........................................................................................................ 5 The Big Picture ............................................................................................... 5 What is software? Why should it be tested? ................................................. 6 What is Quality? How important is it? ........................................................... 6 What exactly does a software tester do? ...................................................... 7 What makes a good tester? ........................................................................... 8 Guidelines for new testers ............................................................................. 9 2. Introduction .................................................................................................. 11 Software Life Cycle ....................................................................................... 11 Various Life Cycle Models ............................................................................ 12 Software Testing Life Cycle .......................................................................... 13 What is a bug? Why do bugs occur? ............................................................ 15 Bug Life Cycle ............................................................................................... 16 Cost of fixing bugs .......................................................................................
    [Show full text]
  • Testing in a Devops Era: Perceptions of Testers in Norwegian Organisations
    Testing in a DevOps Era: Perceptions of Testers in Norwegian Organisations Daniela Soares Cruzes1, Kristin Melsnes2, Sabrina Marczak3 1 SINTEF - Norway 2 Bouvet – Norway 3 PUCRS - Brazil [email protected], [email protected], [email protected] Abstract. To better understand the challenges encountered by testers in DevOps development, we have performed an empirical investigation of what are the trends and challenges for the testers in the DevOps environ- ment. We have discussed the quality assurance in the difference focus areas of DevOps: Social Aspects, automation, leanness, sharing, measurement. The results were then themed in five different topics of concern to testers: collaboration, roles and responsibilities, types of tests, automation and monitoring and infrastructure. In Testing, there has been a change on the roles and responsibilities of testers, where there is much more focus on the responsibilities for testing across the teams, instead of a sole responsibility of the tester. Testers are also forced to collaborate more with other stake- holders as operations and business. Testing is brought to another level of automation in DevOps but there is still need for manual tests, that have to be much more risk-based than before. And finally, testing transparency is a must in this process and should involve not only development team but also operations and customers. This paper contributes to the body of knowledge on what are the areas we need to focus for improvement in test- ing for the DevOps environment. This paper also contributes to practition- ers to improve their testing focusing on specific areas that needs attention.
    [Show full text]
  • Testing in Iterative Product Development Environment Shivageeta S
    Testing In Iterative Product Development Environment Shivageeta S. Choodi [email protected] Software Consultant Novell Bangalore 6th Annual International Software Testing Conference 2006 Novell Software Development India Pvt. Ltd. No.49/1 and 49/3, Garvebhavipalya, 7th mile, Hosur Road Bangalore – 560 068 Software Testing in Iterative Model Abstract Most of the software product companies are adopting iterative model for product development because of the following reasons, • At any given point of time there is a working model of the product. • Increased control over project progress. • Customers are more confident as companies can demonstrate proof of concept in the earlier stages of product development life cycle. Iterative model demands a new process for product development and testing. This study is very critical to Novell, as many of Novell products are adopting iterative model for the development. This paper deals with the testing processes followed in the industry and Novell for iterative model of product development and challenges faced by the test teams in adapting to this new model and changes required from the testing point of view. QAI-STC 2006 1 Software Testing in Iterative Model Contents Testing In Iterative Product Development Environment..........................................................0 Abstract...................................................................................................................................1 1 Background..........................................................................................................................3
    [Show full text]
  • White Paper Agile Software Testing Ten Tips for Launching and Testing High Quality Apps in an Agile Environment
    Agile Software Testing Ten Tips for Launching and Testing High Quality Apps in an Agile Environment White Paper June 2011 White Paper: Agile Software Testing White Paper Agile Software Testing Ten Tips for Launching and Testing High Quality Apps in an Agile Environment Table of Contents • Agile Overview……………………………..……….. 3 - The Agile Manifesto…………………………….…. 3 - The Problem With Waterfall………………...…… 3 - The Emergence of Agile………...……………….. 4 - It’s Okay To Use Waterfall………...…………….. 4 “Simplicity is the ultimate • Ten Tips of Agile Testing…………………………. 5 sophistication.” 1. Understand the True Role of Testing……... 5 2. Unify SRS and Test Plans…………………. 5 3. Define Your Testing Matrix………………… 6 - Leonardo da Vinci 4. Tell a Story – Not a Use Case…………….. 7 5. Capture Meaningful Data……….………….. 8 6. Fix Broken Windows………………………… 9 7. Make Testing Your Feedback Loop…..…… 9 8. Timing is Everything……………………….... 10 9. Run Frequent Load Tests……………….….. 10 10. Stick to Your Scrums………………………… 11 • About uTest………………………………………….. 12 2 White Paper: Agile Software Testing Introduction Agile Overview A Google search for “agile development” returned 2.7 million results at the time this was written. It’s safe to say the word is getting out. But although the basic concepts have been actively discussed in books, blogs and everything in between, we’re going to first review them anyway. We promise to be quick. If you’ve heard this story before, feel free to skip ahead to the next section: Tips for Agile Testing. The Agile Manifesto Though ‘Agile’ is a relatively new term, the shift towards more iterative development methodologies began years ago. Eventually, in 2001, a small group of CTOs, academics and thought leaders published the well-known Agile Manifesto.
    [Show full text]