DOCSLIB.ORG

Flying Black Swans Into Stress Testing

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Flying Black Swans Into Stress Testing Flying black swans into stress testing Geopolitical risks stress tested Dr. Andrea Burgtorf Head of Group Risk Operating Office Stress Test Europe Conference London, Oct 6th, 2016 Disclaimer • This document contains certain statements regarding methodology and approaches poten4ally applicable under future accoun4ng rules. • The informaon herein cannot be used to infer any impact of such future accoun4ng rules on Erste Group Bank AG. • The views expressed herein are the views of the author and may not reflect the official opinion of Erste Group Bank AG. • This document does not cons4tute an offer or invitaon to purchase or subscribe for any shares or other securi4es and neither it nor any part of it shall form the basis of or be relied upon in connec4on with any contract or commitment whatsoever. • The informaon contained herein and this presentaon shall not be further disseminated without wriCen consent of the author. 2 Agenda 1 Introduction 2 Black Swans impact and Stress testing environment 3 Modeling of Black swans 4 Integration in Risk Management Framework 3 Black Swans: an erup6on of financial risks ? 4 Fundamental shi>s in Risk Landscape are ongoing... “Known Companies iden4fy and plan for in an 1 Risks” effort to avoid or mi4gate them. uncertainty “Emerging Risks that have come onto radar, but 2 whose full extent and implicaons are Risks” not yet clear. “Black Events which are highly unlikely to 3 happen but would have severe Swans” consequences if they did. Thorough understanding and managing of risks puts companies in a better place to pursue their strategy with a confidence that they have the business resilience to manage known risks and respond to the unexpected 5 ….Black Swans - Management exposed to uncertainty ? Increased speed and Outdated risk Insufficient Protec4on impact of risk events mgmt. processes They feel the risk They are seeing rapid They are spending too frameworks and processes increases in both speed much me and money on that are currently in place with which risk events take running their current risk in their organizaons are place, and the extent to management processes, no longer giving them the which their impacts on the rather than moving quickly level of protec6on they business are “contagious”. and flexibly to tackle new need. risks. Source: PwC Large organizations may now have blind spots from which high-impact risks could emerge to damage or potentially destroy their business 6 Management awareness of external risks Awareness of Risk Total impact External risks originate from outside the Crime and terrorism 16,00% corporate organizaon—from disrup4ons in Internaonal trade and payments 20,70% Geopoli4cal 30,80% geopoli4cal, economic, regulatory, market, Energy and commodity costs / prices 39,30% Disrup4ve technologies 40,60% technological, or environmental condi4ons. Commercial market shi\s 54,20% Because these risks arise from outside of the Data privacy and security 56,30% Talent and labor 57,50% organizaon, management teams o\en have Financial market 59,90% Regulaons and government policies 62,10% not developed the exper4se to monitor and Compeon 62,80% manage them effec4vely Economic uncertainty 76,30% % of respondents who selected The worst-managed risks: "very poor" or "neither poor nor well" Geopoli4cal 59,80% Disrup4ve technologies 57,10% Talent and labor 55,40% Internaonal trade and payments 52,90% Energy and commodity costs / prices 49,30% Crime and terrorism 49,10% Commercial market shi\s 45,10% Data privacy and security 42,80% Regulaons and government policies 37,70% Compeon 35,90% Financial market 29,50% Source: PwC 7 What are Black Swan events ? A Black Swan event is characterized by the following: 1. Carries extreme impact: the dispropor4onate role of high-impact, hard-to-predict, and rare events that are beyond the realm of normal expectaons in history, science, finance and technology 2. An outlier: the non-computability of the probability of the consequen4al rare events using scien4fic methods (owing to the very nature of small probabili4es) 3. Produces explanaons only a>er the fact : the psychological biases that make people individually and collec4vely blind to Source: ??? uncertainty and unaware of the massive role of the rare event in historical affairs 8 Recent developments: Are black swans actually turning grey ? Recent experience suggests events that fit the defini4on of black swans are happening more and more frequently Chronology of black Swan events 9/11: Series of coordinated terrorist aacks by the Islamic terrorist group al-Qaeda on the United States on the morning of Tuesday, September 11, 2001 (damaged: at least $10 billion in property, $3 trillion in total costs) Fukushima Daiichi nuclear disaster was an energy accident at the Fukushima I Nuclear Power Plant in Fukushima, ini4ated primarily by the tsunami following the Tōhoku earthquake on 11 March 2011 Number of events The dot-com bubble was a historic speculave bubble covering roughly 1995–2001 during which stock markets in industrialized naons saw their equity value rise rapidly from growth in the Internet sector and related fields 1900 2000 2015 2025 Are they now just part of a faster-changing and more uncertain world ? 9 Conclusion: increasing number of Black Swans to impact financial sector § The structure of the world is fundamentally different that it was 20 years ago. § Contribu4ng factors: o increased interconnectedness & complexity o huge debt leverage o hyper-efficiency (just-in-4me supply chains) o lack of redundancy etc… We must build more resilient systems that are better able to withstand the shocks of Black Swan events because it appears they are going to keep coming at an increased pace 10 Was Brexit a Black Swan as well ? 11 … Brexit does indeed make the cut as a black swan 1. It was a surprise, or a “stascal outlier” “No country has ever le\ the European Union… and no opinion poll or beng market suggested it was possible. So the financial markets assumed a UK vote to leave to be impossible” 2. Must have an extreme impact “the impact was definitely outsized. No one knew at the 4me – or knows now – what the full impact of Brexit will be once all is said and done” 3. … and despite it being unpredictable, once it’s happened we spin a story that makes it seem completely predictable in hindsight “A\er the fact it became “obvious” that Brexit would happen because of the dissasfac4on of older, blue-collar Bri4sh voters, the rise of Donald Trump in America or any number of other reasons” 12 Agenda 1 Introduction 2 Black Swans impact and Stress testing environment 3 Modeling of Black swans 4 Integration in Risk Management Framework 13 Impact of the past events on Financial Markets (Equity) 1D 1W 1M 1Y 0% Dot.com bubble -5% Nasdaq -10% Dow Jones -15% FTSE -20% DAX -25% Nikkei -30% -35% 1D 1W 1M 1Y 09/11 0% Nasdaq -5% -10% Dow Jones -15% FTSE -20% DAX -25% Nikkei -30% -35% 1D 1W 1M 1Y Fukushima 10% Nasdaq 5% Dow Jones 0% FTSE -5% -10% DAX -15% Nikkei -20% 14 Impact of the past events on Financial Markets (Credit Spreads) 1D 1M 3M 20 DE5Y BREXIT 15 GB5Y 10 US5Y 5 AT5Y - FR5Y (5) (10) ES5Y (15) IT5Y (20) Banks EUR A (25) Banks USD A (30) 1D 1M 3M 60 DE5Y Fukushima GB5Y 40 US5Y 20 AT5Y FR5Y - ES5Y (20) IT5Y Banks EUR A (40) Banks USD A (60) 15 Type of stress test: selected examples Stress Tests Corresponding Material Risks Scenario Standard Encompasses all risk areas, such as adverse market events, Enterprise-Wide Stress Tests CP defaults, technology failure, operaonal events, ~150 bp - 250 bp on CET 1 Rao (EST) operaonal error/li4gaon Ideosyncrac Stress Tests Iden4fies en4ty- or LOB-specific vulnerabili4es ~100 bp - 150 bp on CET 1 Rao Fulfills Fed/DFAST requirements to withstand stressed Comprehensive Capital economic environment and obtain approval for capital ~300 bp on CET 1 Rao Analysis & Review (CCAR) ac4ons; leverages EST & RST Reverse Stress Tests (RST) Iden4fies scenarios that could "break the bank" >400 bp on CET 1 Rao Recovery Stress Tests Liquidity Stress Tesng Assesses liquidity posi4on aer deposit-run off % of LCR, SPA, etc. Technology Risk Table Top Explores emerging threats, such as cyber aacks wide range or outcomes Exercises Develops con4ngency plans for high-risk, high Playbook / Simulaons focus on mi4gang ac4on probability scenarios Whilst regulatory compliance is challenging, financial institutions should leverage the stress testing exercises to build long-term value, rather than treating it like a check-the-box exercise 16 ∆ CET 1 Ra6o 2018 (bps) - Adverse Scenario Or cover so Black Swans as well ? Do regulators overshoot in scenario seng? -1500 -1000 -500 0 EBA Stress Test 2016 – 3Y horizon Banca Monte dei Paschi di Siena SpA Allied Irish Banks Plc The Royal Bank of Scotland Group Public Limited NRW.BANK Düsseldorf N.V. Bank Nederlandse Gemeenten Landesbank Baden-WürCemberg Bayerische Landesbank Commerzbank AG Banco Popular Español SA ABN AMRO Groep N.V. Bank of Ireland Deutsche Bank AG Coöperaeve Centrale Raiffeisen- Nykredit Realkredit Deka Bank Deutsche Girozentrale OP-Pohjola Group Belfius Banque SA Raiffeisen-Landesbanken-Holding GmbH NORD/LB Norddeutsche Landesbank OTP Bank Nyrt. Erste Group Bank AG Barclays Plc Banco Popolare Società Cooperava Banco Santander SA ING Groep N.V. BFA Tenedora De Acciones S.A. KBC Group NV Banco Bilbao Vizcaya Argentaria SA LandesbankHessen-Thüringen Girozentrale Banco de Sabadell SA La Banque Postale UniCredit SpA Société Générale SA Groupe BPCE Unione di Banche Italiane SCpA HSBC Holdings Plc Crédit Agricole Group Lloyds Banking Group Plc Intesa Sanpaolo SpA Criteria Caixa Holding S.A. Svenska Handelsbanken - group BNP Paribas SA VW Financial Services AG Source: ECB/EBA Website Nordea Bank - group Skandinaviska Enskilda Banken - group Danske Bank Jyske Bank Crédit Mutuel Group Swedbank - group Powszechna Kasa Oszczednosci Bank Polski SA DNB ASA 17 Black Swans qualify for various types of stress tes6ng § In order to capture the impact of so-called unexpected events, stress tes4ng could be 1.
Load more

Recommended publications
  • Software Testing: Essential Phase of SDLC and a Comparative Study Of
    International Journal of System and Software Engineering Volume 5 Issue 2, December 2017 ISSN.: 2321-6107 Software Testing: Essential Phase of SDLC and a Comparative Study of Software Testing Techniques Sushma Malik Assistant Professor, Institute of Innovation in Technology and Management, Janak Puri, New Delhi, India. Email: [email protected] Abstract: Software Development Life-Cycle (SDLC) follows In the software development process, the problem (Software) the different activities that are used in the development of a can be dividing in the following activities [3]: software product. SDLC is also called the software process ∑ Understanding the problem and it is the lifeline of any Software Development Model. ∑ Decide a plan for the solution Software Processes decide the survival of a particular software development model in the market as well as in ∑ Coding for the designed solution software organization and Software testing is a process of ∑ Testing the definite program finding software bugs while executing a program so that we get the zero defect software. The main objective of software These activities may be very complex for large systems. So, testing is to evaluating the competence and usability of a each of the activity has to be broken into smaller sub-activities software. Software testing is an important part of the SDLC or steps. These steps are then handled effectively to produce a because through software testing getting the quality of the software project or system. The basic steps involved in software software. Lots of advancements have been done through project development are: various verification techniques, but still we need software to 1) Requirement Analysis and Specification: The goal of be fully tested before handed to the customer.
    [Show full text]
  • Stress Testing Embedded Software Applications
    Embedded Systems Conference / Boston, MA September 20, 2007 ESC-302 Stress Testing Embedded Software Applications T. Adrian Hill Johns Hopkins University Applied Physics Laboratory [email protected] ABSTRACT This paper describes techniques to design stress tests, classifies the types of problems found during these types of tests, and analyzes why these problems are not discovered with traditional unit testing or acceptance testing. The findings are supported by citing examples from three recent embedded software development programs performed by the Johns Hopkins University Applied Physics Laboratory where formal stress testing was employed. It also defines what is meant by the robustness and elasticity of a software system. These findings will encourage software professionals to incorporate stress testing into their formal software development process. OUTLINE 1. INTRODUCTON 1.1 What can be learned by “breaking” the software 2. DESIGNING A STRESS TEST 2.1 What is a reasonable target CPU load? 2.2 Other ways to stress the system 2.3 Characteristics of a Stress Test 3. REAL WORLD RESULTS 3.1 Case #1: Software Missed Receiving Some Commands When CPU Was Heavily Loaded 3.2 Case #2: Processor Reset when Available Memory Buffers Were Exhausted 3.3 Case #3: Unexpected Command Rejection When CPU Was Heavily Loaded 3.4 Case #4: Processor Reset When RAM Disk Was Nearly Full 3.5 Synopsis of All Problems Found During Stress Testing 4. SUMMARY 1. INTRODUCTON Traditional Software Acceptance Testing is a standard phase in nearly every software development methodology. Test engineers develop and execute tests that are defined to validate software requirements.
    [Show full text]
  • Testing Techniques Selection Based on SWOT Analysis
    International Journal of Knowledge, Innovation and Entrepreneurship Volume 2 No. 1, 2014, pp. 56—68 Testing Techniques Selection Based on SWOT Analysis MUNAZZA JANNISAR, RUQIA BIBI & MUHAMMAD FAHAD KHAN University of Engineering and Technology, Pakistan Received 02 March 2014; received in revised form 15 April 2014; approved 22 April 2014 ABSTRACT Quality is easy to claim but hard to achieve. Testing is a most essential phase in software development life cycle not only to ensure a project’s success but also customer satisfaction. This paper presents SWOT Analysis of three testing tech- niques—White-Box, Black-Box and Grey Box. The testing techniques are evaluated on the basis of their strengths, weaknesses, opportunities and threats. The analysis in this paper shows that selection of the techniques should be based on a set of defined attrib- utes, context and objective. The findings in this paper might be helpful in highlighting and addressing issues related to testing techniques selection based on SWOT analysis. Keywords: Black Box Testing, White Box Testing, Grey Box, SWOT Introduction Software plays a substantial role in every sphere of life. It is a key reason why software engineering continually introducing new methodologies and improvement to software development. The capacity of organisations to achieve customer satisfaction or to cor- rectly identify the needs of customers can be misplaced, leading to ambiguities and un- wanted results 1. Researchers have suggested many testing techniques to deal with fault identification and removal subjects, before the product [software] is shipped to cus- tomer. Despite many verification and validation approaches to assuring product quality, defect-free software goal is not very often achieved.
    [Show full text]
  • Putting the Right Stress Into WMS Volume Testing Eliminate Unwelcome Surprises
    v i e w p o i n t Putting the Right Stress into WMS Volume Testing Eliminate unwelcome surprises. When implementing or upgrading a Warehouse Management System (WMS) in a high volume environment, it should be a given that a significant amount of effort needs to be dedicated to various phases of testing. However, a common mistake is to ignore or improperly plan for volume testing. So what types of processes should be tested? There is unit testing that validates segments of functionality, RF Floor Transactions - This should involve the more interface testing that checks the flow of data between systems, commonly used RF transactions such as putaway and picking. user acceptance testing that involves the end user group It is not a good idea to include every type of RF activity. That working through scripted functional flows, and field testing that means you probably will not be paying too much attention to executes an augmented version of the user acceptance testing cycle counts as a part of your stress test. on the distribution center floor. Wave Processing - It is absolutely necessary to know how Volume testing, also known as stress testing, is an assessment long it will take to process each day’s orders. In addition, it is that may be ignored. This is a mistake, because this puts a strain important to validate if there are any important processes that on the processes within the WMS, as well as its interfaces in cannot be executed during wave processing. For example, the order to identify conditions that would result in a slow-down testing may reveal that RF unit picking is an activity that cannot or catastrophic crash of the system.
    [Show full text]
  • Scalability Performance of Software Testing by Using Review Technique Mr
    International Journal of Scientific & Engineering Research Volume 3, Issue 5, May-2012 1 ISSN 2229-5518 Scalability performance of Software Testing by Using Review Technique Mr. Ashish Kumar Tripathi, Mr. Saurabh Upadhyay, Mr.Sachin Kumar Dhar Dwivedi Abstract-Software testing is an investigation which aimed at evaluating capability of a program or system and determining that it meets its required results. Although crucial to software quality and widely deployed by programmers and testers, software testing still remains an art, due to limited understanding of the principles of software, we cannot completely test a program with moderate complexity. Testing is more than just debugging. The purpose of testing can be quality assurance, verification and validation, or reliability estimation. Testing can be used as a generic metric as well. Correctness testing and reliability testing are two major areas of testing. Software testing is a trade-off between budget, time and quality. Index Terms--Software testing modules, measurement process, performance and review technique. —————————— —————————— 1-INTRODUCTION throughput of a web application based on requests per esting is not just finding out the defects. Testing is second, concurrent users, or bytes of data transferred as not just seeing the requirements are Satisfied which well as measure the performance. T are necessary in software development. Testing is a process of verifying and validating all wanted requirements 3-PROCESS FOR MEASUREMENT is there in products and also verifying and validating any unwanted requirements are there in the products. It is also In the measurement of configured software there are seeing any latent effects are there in the product because of several technique can be applicable these requirements.
    [Show full text]
  • Performance Testing, Load Testing & Stress Testing Explained
    Performance Testing, Load Testing & Stress Testing Explained Performance testing is key for understanding how your system works. Without good performance testing, you don’t know how your system will deal with expected—or unexpected—demands. Load testing and stress testing are two kinds of performance testing. Knowing when to use load testing and when to use stressing testing depends on what you need: Do you need to understand how your system performs when everything is working normally? Do you want to find out what will happen under unexpected pressure like traffic spikes? To ensure that your systems remain accessible under peak demand, run your system through performance testing. Let’s take a look. Performance vs load vs stress testing Performance testing is an umbrella term for load testing and stress testing. When developing an application, software, or website, you likely set a benchmark (standard) for performance. This covers what happens under: Regular parameters: If everything goes as planned, how does it work? Irregular parameters: Can my website application survive a DDoS attack? Load testing is testing how an application, software, or website performs when in use under an expected load. We intentionally increase the load, searching for a threshold for good performance. This tests how a system functions when it faces normal traffic. Stress testing is testing how an application, software, or website performs when under extreme pressure—an unexpected load. We increase the load to its upper limit to find out how it recovers from possible failure. This tests how a system functions when it faces abnormal traffic. Now let’s look at each in more detail.
    [Show full text]
  • A Load Test Guide
    WHITE PAPER Preparing for peak traffic: A load test guide Preparing for peak traffic: A load test guide. 1 WHITE PAPER Table of Contents. What is a load test? ...................................................................................................3 Why load testing? ......................................................................................................4 Best practices ............................................................................................................5 Running a load test ...................................................................................................8 Test configurations .................................................................................................10 Final thoughts .........................................................................................................12 About WP Engine .....................................................................................................14 WHITE PAPER Preparing for peak traffic: A load test guide like scaling capabilities, lifecycle hooks, security risks, automatic Introduction. code deployment, health checks, and target tracking. With the holidays approaching, most marketing teams are In this informative white paper, we’ll break down the basics of prepping for impactful, revenue-driving campaigns. Some load testing, why you should load test, best practices, and how might even be preparing for the launch of a new product on to get started load testing your site before the holidays. Black Friday. If
    [Show full text]
  • A Multi-Hazard Risk Assessment Methodology, Stress Test Framework and Decision Support Tool for Transport Infrastructure Networks
    Available online at www.sciencedirect.com ScienceDirect Transportation Research Procedia 14 ( 2016 ) 1355 – 1363 6th Transport Research Arena April 18-21, 2016 A multi-hazard risk assessment methodology, stress test framework and decision support tool for transport infrastructure networks Julie Clarke a,*, Eugene Obrien a aRoughan and O’Donovan Innovative Solutions Limited, Dublin, Ireland Abstract Natural hazards can cause serious disruption to societies and their transport infrastructure networks. The impact of extreme hazard events is largely dependent on the resilience of societies and their networks. The INFRARISK project is developing a reliable stress test framework for critical European transport infrastructure to analyse the response of networks to extreme hazard events. The project considers the spatio-temporal processes associated with multi-hazard and cascading extreme events (e.g. earthquakes, floods, landslides) and their impacts on road and rail transport infrastructure networks. As part of the project, an operational framework is being developed using an online INFRARISK Decision Support Tool (IDST) to advance decision making approaches, leading to better protection of existing transport infrastructure. The framework will enable the next generation of European infrastructure managers to analyse the risk to critical road and rail infrastructure networks due to extreme natural hazard events. To demonstrate the overarching risk assessment methodology developed in the project, the methodology is demonstrated for two case studies, which comprise portions of the European TEN-T network; a road network in the region of Bologna, Italy and a rail network extending from Rijeka to Zagreb in Croatia. This paper provides an overview of the INFRARISK multi-hazard risk assessment methodology and a brief introduction to the case studies, as the project is currently ongoing.
    [Show full text]
  • Performance Testing: Methodologies and Tools
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by International Institute for Science, Technology and Education (IISTE): E-Journals Journal of Information Engineering and Applications www.iiste.org ISSN 2224-5758 (print) ISSN 2224-896X (online) Vol 1, No.5, 2011 Performance Testing: Methodologies and Tools H. Sarojadevi * Department of Computer Science and Engineering, Nitte Meenakshi Institute of Technology, PO box 6429, Yelahanka, Bengaluru -64 * E-mail of the corresponding author: [email protected] Abstract Performance testing is important for all types of applications and systems, especially for life critical applications in Healthcare, Medical, Biotech and Drug discovery systems, and also mission critical applications such as Automotives, Flight control, defense, etc. This paper presents performance testing concepts, methodologies and commonly used tools for a variety of existing and emerging applications. Scalable virtual distributed applications in the cloud pose more challenges for performance testing, for which solutions are rare, but available; one of the major providers is HP Loadrunner. Keywords: Performance testing, Application performance, Cloud computing 1. Introduction Building a successful product hinges on two fundamental ingredients — functionality and performance. ‘Functionality’ refers to what the application lets its users accomplish, including the transactions it enables and the information it renders accessible. ‘Performance’ refers to the system’s ability to complete transactions and to furnish information rapidly and accurately despite high multi-user interaction or constrained hardware resources. Application failure due to performance-related problems is preventable with pre-deployment performance testing. However, most teams struggle because of lack of professional performance testing methods, and guaranteeing problems with regard to availability, reliability and scalability, when deploying their application on to the “real world”.
    [Show full text]
  • Performance Testing in the Cloud. How Bad Is It Really?
    Performance Testing in the Cloud. How Bad is it Really? Christoph Laaber Joel Scheuner Philipp Leitner Department of Informatics Software Engineering Division Software Engineering Division University of Zurich Chalmers | University of Gothenburg Chalmers | University of Gothenburg Zurich, Switzerland Gothenburg, Sweden Gothenburg, Sweden [email protected] [email protected] [email protected] Abstract to evaluate the performance of applications under “realistic condi- Rigorous performance engineering traditionally assumes measur- tions”, which nowadays often means running it in the cloud. Finally, ing on bare-metal environments to control for as many confounding they may wish to make use of the myriad of industrial-strength 1 factors as possible. Unfortunately, some researchers and practition- infrastructure automation tools, such as Chef or AWS CloudFor- 2 ers might not have access, knowledge, or funds to operate dedicated mation , which ease the setup and identical repetition of complex performance testing hardware, making public clouds an attractive performance experiments. alternative. However, cloud environments are inherently unpre- In this paper, we ask the question whether using a standard dictable and variable with respect to their performance. In this study, public cloud for software performance experiments is always a we explore the effects of cloud environments on the variability of bad idea. To manage the scope of the study, we focus on a specific performance testing outcomes, and to what extent regressions can class of cloud service, namely Infrastructure as a Service (IaaS), still be reliably detected. We focus on software microbenchmarks and on a specific type of performance experiment (evaluating the as an example of performance tests, and execute extensive experi- performance of open source software products in Java or Go using 3 ments on three different cloud services (AWS, GCE, and Azure) and microbenchmarking frameworks, such as JMH ).
    [Show full text]
  • Software Testing
    Software Testing Carnegie Mellon University 18-849b Dependable Embedded Systems Spring 1999 Authors: Jiantao Pan [email protected] Abstract: Software testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results. [Hetzel88] Although crucial to software quality and widely deployed by programmers and testers, software testing still remains an art, due to limited understanding of the principles of software. The difficulty in software testing stems from the complexity of software: we can not completely test a program with moderate complexity. Testing is more than just debugging. The purpose of testing can be quality assurance, verification and validation, or reliability estimation. Testing can be used as a generic metric as well. Correctness testing and reliability testing are two major areas of testing. Software testing is a trade-off between budget, time and quality. Contents: Introduction Key Concepts Taxonomy Testing automation When to stop testing? Alternatives to testing Available tools, techniques, and metrics Relationship to other topics Conclusions Annotated Reference List & Further Reading Introduction Software Testing is the process of executing a program or system with the intent of finding errors. [Myers79] Or, it involves any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results. [Hetzel88] Software is not unlike other physical processes where inputs are received and outputs are produced. Where software differs is in the manner in which it fails. Most physical systems fail in a fixed (and reasonably small) set of ways. By contrast, software can fail in many bizarre ways.
    [Show full text]
  • Fuzzing Hardware Like Software
    Fuzzing Hardware Like Software Timothy Trippel, Kang G. Shin Alex Chernyakhovsky, Garret Kelly, Matthew Hicks Computer Science & Engineering Dominic Rizzo Computer Science University of Michigan OpenTitan Virginia Tech Ann Arbor, MI Google, LLC Blacksburg, VA ftrippel,[email protected] Cambridge, MA [email protected] fachernya,gdk,[email protected] Abstract—Hardware flaws are permanent and potent: hard- Software Hardware ware cannot be patched once fabricated, and any flaws may undermine even formally verified software executing on top. Custom Consequently, verification time dominates implementation time. Test Coverage The gold standard in hardware Design Verification (DV) is Generator concentrated at two extremes: random dynamic verification and Tracing TB DUT formal verification. Both techniques struggle to root out the PriorWork subtle flaws in complex hardware that often manifest as security vulnerabilities. The root problem with random verification is its undirected nature, making it inefficient, while formal verification Software SW is constrained by the state-space explosion problem, making it Fuzzer à infeasible to apply to complex designs. What is needed is a HW HW Generic TB solution that is directed, yet under-constrained. Fuzzing HW DUT Model DUT Instead of making incremental improvements to existing hard- ware verification approaches, we leverage the observation that =Inject Coverage Tracing Instrumentation existing software fuzzers already provide such a solution; we adapt it for hardware verification, thus leveraging existing—more Fig. 1. Fuzzing Hardware Like Software. Unlike prior Coverage Directed Test Generation (CDG) techniques [15]–[18], we advocate for fuzzing soft- advanced—software verification tools. Specifically, we translate ware models of hardware directly, with a generic harness (testbench) and RTL hardware to a software model and fuzz that model.
    [Show full text]