Whitepaper operational acceptance testing OBusiness continuity assurance Trusted partner for your Digital Journey Summary Operational Acceptance Testing (OAT) Acceptance Testing Irrespective of the delivery model em- is the penultimate phase of Acceptance Formal testing with respect to user needs, ployed for the project (waterfall, agile, Table of contents Table of figures Testing in a Software Testing Life cycle requirements, and business processes con- devops), 3 aspects need to be addressed: and is the last defence line between a ducted to determine whether or not a system software development project and de- satisfies the acceptance criteria and to enable 1. Conformance of the asset built with 03 01 ployment of software on production. the user, customers or other authorized entity to the requirements and specifications FFor decedes, Operational Acceptance has determine whether or not to accept the system. 2. Operate and support the asset Summary Main Activities and Operational been undermined and misunderstood. Where once deployed to production User Acceptance has been written about Operational Acceptance Testing 3. Ensure adherence with Business KPI, Acceptance Testing stakeholders and hailed as a “final phase in testing before Operational testing in the acceptance IT SLA and business continuity. production”. User Acceptance is but one side of test phase, typically performed in a (sim- 04 the coin, Operational Acceptance is the other. ulated) operational environment by op- To address these questions, the project Introduction 02 erations and/or systems administration team needs to perform the Operational The International Software Testing staff focusing on operational aspects, aspects of acceptance testing and eval- Quality attribute evaluation Qualifications Board (ISTQB) defines E.g. recoverability, resource-behavior, in- uation commonly known as Operational ‘Acceptance Testing’ and ‘OAT’ as: stallability and technical compliance. Acceptance Testing (OAT). This whitepaper 06 evaluates the quality characteristics associ- ated with operational acceptance test scope Scope of Operational Acceptance 03 from the perspective of the newly released Testing Main test phases of Operational software testing standard ISO 29119. Acceptance Testing Types 14 Facets of Operational Acceptance 04 Testing The future of Operational Acceptance Testing 15 Conclusion 05 Automated Network Testing 19 References & Annexes 06 AtoS Bridge & its Services For Reference please contact: 07 Sriini Vaasudevan Paladin – Business Process Oriented Test Lead & Operations Manager/ Product Owner – Telco CRM Monitoring Eemsgolaan 7, 9727 DW Gron- ingen, Netherlands +31 (0)683624587 08 [email protected] Opscode Chef 09 Puppet Labs Operational Acceptance Testing version 1.0 2 3 Operational Acceptance Testing Introduction Initially it used to be a practice for testing to This evolution in Information, Technology, a new understanding Operational Acceptance The present paper gives an overview of how be part of the Software Development Lifecycle Infrastructure, Library provoked the fact that Testing (OAT) came on board. By 2010 the soft- to use ISO 25000 (ISO/IEC JTC 1/SC 7 Software (SDLC) wherein the development teams did In a project, when the project teams completed the devel- Business continuity cannot be only assured by ware quality industry published the ISO 25000 and Systems Engineering, 2010) to scope out a test analysis on their own product leading opment of the software, it was released and handed over to purely functional quality assurance, this created SQuaRE [REF-5] series of standards that outlined OAT systematically and how to apply test meth- to impartially within their own team. Through a need for introducing various non functional the scope of OAT, in respect to the evaluation ods successfully by having application owners experience and growing sophistication in the operation team and the application owner. lt immediately aspects borne out of the ITIL processes (Infor- of quality requirements, at a framework and and operation teams involve other stakeholders the software, software units evolved multiple became a part of the business processes as it was launched mation Technology Infrastructure Library). This strategic level. In 2013 the Testing industry like architects, developers, or infrastructures. phases for testing to improve the quality of new paradigm signaled a clear evolution from published the ISO 29119 Software Testing series the end products produced. During the 1980’s into the production environments. Consequently, known and a purely functional scope to a more holistic of standards, which enabled effective and Paul Rook designed the V-Model methodology unknown defects of the software will directly impact business scope of acceptance testing – encompassing efficient testing by qualified/quantified means to to improve the overall efficiency and effec- both functional and non-functional aspects support the delivery of reliable, robust IT assets. tiveness in software development processes. continuity and potentially cause damage to a greater or lesser to ensure business continuity. After this new By this time, the differing test phases had extent. In addition, responsibility typically is transferred from paradigm of acceptance had been accepted grown to include: Unit Testing, Component by the development and testing communities, Testing, System Integration Testing (SIT), the development units to two stakeholders: System Test (ST) and Acceptance Testing. References to ‘acceptance testing’ were (then) Application Owner understood and interpreted as to mean busi- Figure 1 : Main Activities and OAT stakeholders ness or User Acceptance Testing (UAT). As a result of this interpretation, UAT was The single point of contact for business units concerning the often perceived to be the last, or one of operation of dedicated applications. These units are the the last, lines of defense between a soft- ware development and its implementa- internal part of line organization managing the maintenance, tion into the production environment. and constitute the interface to operation teams. But this misconception started back- tion Te firing. Hereby below a SOC. pera ams O Operation Team An internal or external team deploying and operating software Standard following well-defined processes, e.g. tailored by Information, Transition Operation Technology, Infrastructure, Library. These units are equipped with system and application utilities for managing the opera- tion, they will contact the application owners if bigger changes Crisis Operation should be necessary to guarantee business continuity. Application Owners Analysis Design Implemetation Functional Testing Operation Operational Acceptance testing Operational Acceptance Testing 4 5 Operational Acceptance Testing Scope of Operational Acceptance Testing In traditional sequential development method, the Operational Acceptance testing (OAT) is like- Delivery Models ly to be executed near the end of the software The relevance of the different types of OAT is development life cycle. This would enable the The essentiality of OAT can be achieved through the process of incorporating it in the delivery mod- derived from the individual artefacts and their test to be executed on a like or near-production els. In the growing evolution of software development, the delivery models have matured. From the quality attributes. The major test types are allo- instance of the asset that is to be implement- primitive Waterfall to Agile which is rationalized now in the growing rage to go Devops. cated, and decisions about test execution follow ed. This contributes to the idea of Operational Below a definition scale of all 3 and if OAT is coherent with them. from the risk evaluation - the scope of OAT is Readiness and Assurance. Let’s define them : ultimately defined by selecting columns from Waterfall or V Model - The waterfall model is a sequential design process, used in software devel- the table shown in Figure 3 and substantiation » Operational Readiness is the process of opment processes, in which progress is seen as flowing steadily downwards (like a waterfall) through of these types in coherence with the delivery preparing the future asset owner, and the the phases of conception, initiation, analysis, design, construction, testing, production/implementation models is explained below as part of section 3.1. support team, so that, at the time of imple- and maintenance. mentation/cutover, they are fully ready to as- sume ownership and operation of the asset. Agile Model - Agile development model is also an incremental model. Software is developed in incre- Figure 2 : Quality attribute evaluation » Assurance, in this context, refers to the act mental, rapid cycles. This results in small incremental releases with each release building on previous of re-assuring the project and organization functionality. Each release is thoroughly tested to ensure software quality is maintained. It is used for stakeholders that their (iterative) developing time critical applications. asset, and their organization, is in a state of Operational Code Rehearsal Installation Framework/ SLA/OLA Load/ Security Backup & Failover Recovery DocumentationAnalysis Testing Testing Platform Monitoring Performance testing Restore Testing Testing operational readiness – or by which to pro- Dev Ops - A combination of Development & Operations – is a software development methodology Review Upgrade test Testing Testing vide a measure of assurance that