TVNZ Reduces Costs,Optimizes Efficiency,And Accelerates Business Agility

TVNZ Case Study

TVNZ reduces costs,optimizes efficiency,and accelerates business agility

Tosca Testsuite’s 3rd generation architecture separates business logic and technical layers reducing these unproductive costs [associated with ongoing maintenance of test scripts coupled with specialist skills]”

John Mulrennan Sales Systems Manager at TVNZ

About TVNZ by business subject matter experts. For the The Challenge TVNZ is New Zealand’s national broadcast- SME’s this was unproductive and repetitive. The manual regression test suite er, operating four channels - analogue and We needed to work smarter.” says John Mul- for MediaBook, a system used for digital – and an online “catch up TV” service rennan, Sales Systems Manager at TVNZ. booking and managing TV channel called ‘TVNZ OnDemand’. TVNZ’s Strategic advertising and associated revenue, consisted of 2,500+ test cases. Due to Plan, titled “Inspiring New Zealanders on The Situation the amount of effort required and Every Screen”, emphasises TVNZ’s strong The manual test suite for MediaBook con- the disruption to the business when- focus on providing viewers, consumers sisted of 2,500+ test cases across the 16 ever a manual test suite had to be executed, TVNZ were not able to and customers with the content and ser- different functional modules and - 15 in upgrade MediaBook as frequently vices they want and expect from New Zea- terfaces to internal/external broadcasting as they would have liked in order to land’s leading television and digital media systems, giving system coverage of close benefit from newer releases. company. to 90%. About 90% of TVNZ’s revenue is gained Many of the tests require such an ex- The Objectives through commercial activity such as ad- tensive data set-up component prior to Establish a maintainable, scalable and cost effective test automation vertising, licensing and merchandising and the start of test execution that it took 15 solution, and being able to ‘dial-in’ hiring out production resources. The other person-days per test cycle to prepare the resources as and when required 10% of revenue comes from government data. Additionally, given the nature of the without having to rely on specialist sources. The system used for booking and business, the data used for TV broadcast- with years of experience. managing TV channel advertising and as- ing decays over time (as soon as it is aired), The Solution sociated revenue is a product called Me- so it cannot be reused. Model-based testing with TOSCATM diaBook. elevates test automation to a busi- The Challenge ness discipline and eliminates main- “TVNZ broadcast sales systems are responsi- Each new release had a test cycle which tenance issues inherent in script- based technologies. ble for $1m/day revenue broadcast nation- lasted 3-4 weeks. Due to the complex na- ally & regionally to 1/25sec precision. These ture of the application a test cycle would deliverables required exact manual software require 5 business users and 5 testers, to testing managed by Technology supported set aside blocks of time during which they Tricentis | TVNZ Case Study | Page 2

would first set up the test data, and then introduction of automated testing tools for run through the manual tests. This had a many years. However the costs associated The Results significant negative impact on TVNZ’s reg- with ongoing maintenance of test scripts cou- In only 4 months the core regres- ular work schedules. Typically the full test pled with specialist skills required were not sion suite for MediaBook consisting suite would not be completed within the justified. TOSCA’s 3rd generation architecture of 1,268 test cases was automated, allocated time frame, resulting in a lower separates business logic and technical layers totalling 27,859 test actions and covering 12,215 verification points. than desired coverage and an increased reducing these unproductive costs” says John risk of undetected defects. Also, due to the Mulrennan. • Efficiency - execution effort is down amount of effort required and the disrup - by 94% to 4 days compared to over tion to business users whenever a manual The Implementation 70 days to run equivalent tests manually, as measured by the total test suite had to be run, TVNZ were not A team consisting of one core TOSCA number of verification points. able to upgrade the system as frequently Technical Test Lead and three TOSCA new- as they would have liked in order to benefit bies was established. Automation priority • On demand staffing – TVNZ no from newer releases. was given to the labour-intensive data set- longer needs to rely solely on automation test specialists but engages up tasks, of which there are multiple forms functional testers without technical The Objectives (e.g. creating TV programme schedules, skills to develop automated test The primary objective of automation at customers, products, adverts, etc.). These scenarios. TVNZ was to reduce the duration and cost tasks were then encapsulated into reus- • ROI - TVNZ’s average annual cost of a test cycle and therefore increase the able test case libraries, which can now eas- reduction of testing MediaBook frequency with which upgrades could be ily be added as pre-requisites to any and exceeds 58% (Figures are based on made. Additionally TVNZ sought to reduce all tests as required. Over the course of savings in test execution only and the negative impact of the extended work four months an automated test suite solu- do not include any other benefits resulting from quality and/or turn load on its business users, especially taking tion was established and continues to be around improvements). The antici- into account the extensive and inefficient expanded upon, with a strong emphasis pated investment break-even is 1.7 “one-off test data” setup task required for on reuse and scalability. Implementing a years. every test iteration. Importantly, being able reusable test case design within Tosca has to ‘dial-in’ resources as and when required enabled an expansion to the scope of ex- without having to rely on SME’s or testers isting manual test cases, and has resulted with years of system experience, allows for in higher risk coverage than was previously stream-lining of the test team. achieved.

The Solution The Results TVNZ conducted an on-site proof of con- After four months the test artefacts pro- cept with TOSCA Test Suite in order to as- duced stood at 1,268 unique test cas- sess its ability to effectively steer the comes, covering a total of 12,215 verification plex MediaBook application. The proof of points - three to five times more than concept successfully met all of the goals would be checked manually by the tester/ set out prior to commencement of the as- business user. For all the common parts, sessment, within the allotted period of 5 including the data setup pre-requisites, days to demonstrate the automated steer- these test cases make extensive use of the ing of MediaBook. reusable test step blocks. With 521 reusable test step blocks and 3,448 instances “TVNZ planned and budgeted to comple- of reuse, this indicates an average of al- ment our manual software testing with the most seven reuses per block so far. Tricentis | TVNZ Case Study | Page 3

experts when a test failed, freeing up the business users from the bulk of the testing effort. With such a high coverage now being achieved through automation, TVNZ is able to benefit from speedier test cycles and less use of business resource effort. At the time of writing, TVNZ await an immi- nent release of the latest version of Media- Book, and once again they will commit to a full test cycle, The first time TOSCA was used -to exe points than the manual suite), a TOSCA but this time with an even greater percent- cute test cases against a new release of proficient tester with absolutely no age TVNZ of automation. MediaBook, it was in conjunction with business knowledge was brought in. He manual testing. After an effort of- 70subsequently per ran through 1,545 automat- “Implementing TOSCA will enable TVNZ busi- son-days, the manual team had executed ed test cases - including the automated ness SMEs to focus on strategic goals, in- 1,450 original non-automated tests. To setup of test data - in crease the integrity and agility of our software execute the automated equivalent in TO- only 4 days, covering large areas of the testing discipline, at an acceptable ROI.” says SCA (covering 3-5 times more verification system and only calling on subject matter John Mulrennann

Appendix

About Tricentis

Tricentis, the Continuous Testing Company, specializes in market leading agile software testing tools for enterprises. We help Global 2000 companies adopt DevOps and gain success by achieving automation rates of over 90%. Top analysts recognize Tricentis as a leader in Software Test Automation, with Model-based Test Automation and Test Case Design as standout features.

Our 400+ customers include global names from the Top 500 brands such as A&E, Allianz, Deutsche Bank, HBO, JetBlue, Orange, Swiss Re, Telstra, Toyota, UBS, Vantiv, Virgin Airlines, and Zurich Insurance. Tricentis has offices in Austria, United States, Germany, Switzerland, UK, Netherlands, Poland, and Australia.

www.tricentis.com