Accord Global Technology Solutions Importance of Testing Test Automation and Automated Test Equipment Abstract Testing is one of the major activities in a development life cycle of a product. Automation of testing will help reducing manual intervention and increasing the productivity. Automated Test Equipment (ATE) helps in automating the testing of the system/LRU during development and production. ATE’s can be a simple bench top setup or a set of test equipment arranged in a rack. 1 Introduction – Testing 737 MAX 8 Crashes (March 2019, NASA’s Mars orbiter crash US Northeast Black Out (Aug Oct 2018) (Sept 1999) 2013) Overexposure of radiation Nissan recalled 9.9lac cars for therapy patients in Software Configuration error Airbag malfunction (March 2014) caused Airbus A400M crash (Jan National Cancer Institute, 2015) Panama City (Jun 2000) www.accord-global.com Copyright © 2020 Accord Global Technology Solutions Private Limited. All rights reserved. Accord Global Technology Solutions What is common in all above disasters? Yes... Software was the root cause of the issue in each case. All accidents are preceded by a chain of events that trigger failure responses which cascade into a disaster. In most cases rigorous testing should be able to exercise these fault conditions and eventually avoid disasters. Testing is one of the major activities of any development and change in any software or hardware. Tests are usually performed to verify and validate the expected behaviour against the intended behaviour in the form of requirements and specifications. The overall objective of testing is not to find every software bug/hardware issue that exists, but to expose situations that could negatively impact the operation, maintainability, and usability. In today’s fast-growing world, customer’s expectation is to have products to be developed, implemented, installed and serviced faster than their competitors. Customer wants their product to be more competitive with better features to be implemented in a short time frame (as early as possible), but in the meantime they also want defect free product with defect free software. Releasing software/product to customer with issues may affect the user experience which eventually impacts the company reputation in the market. Therefore, severe emphasis is laid on the rigor of testing in Software/product/system Development Life Cycle. Testing can be in various forms, for example early on in the development phase it could be reviews of artifacts such as requirements and design and later on can include traditional testing methods of verification and validation. It is common knowledge in product development that cost of finding and fixing defect early on is always cheaper to finding and fixing defects later on, this paper also emphasis this point. 2 Importance of Testing • In today’s competitive market, the survival of a product depends on its quality and it is imperative that adequate tests are performed on the product to make it successful in the market. • Some of the avionics or automotive application requires installation of the product with customer electronic system and hence if any defect in the product involves additional costs of repair and downtime of the system. This necessitates the product to be robust, reliable and free from defects. www.accord-global.com Copyright © 2020 Accord Global Technology Solutions Private Limited. All rights reserved. Accord Global Technology Solutions While developing systems for critical applications, several design considerations related to its operational environment also needs to be considered. Some of these are lightning, ESD, EMI/EMC, Temperature, Vibration etc. It is also vital to test the equipment for robustness conditions in addition to normal tests. After all, for the growth of any business the product should speak for itself and testing plays a key role in making this happen. But to make this happen one has to plan it properly before executing it. 3 Test Automation We have seen that for any product (especially avionics or an automotive product), typically there are thousands of requirements and all these requirements are tested for normal and robustness cases. In some cases, equivalence classes conditions are also tested (as per DO-178 guidance). When it comes to production testing, few of the avionics products may have several thousand tests to be covered as part of Acceptance testing. It is also expected that 100% of products manufactured are required to be tested for its performance and operation compliance before delivery. Sometimes a single product production testing may consume more than 10 hours!! In the event where such large amount of testing is required to be performed, it becomes a necessity to automate the tests to improve the production efficiency. TEST AUTOMATION or AUTOMATION TESTING means using an automation tool/equipment to execute the test cases or tests. On the contrary, Manual Testing is performed manually by a human going through the test procedure sequentially. The test automation software can be structured to not only run the tests but also collate the results and generate a detailed report of the tests. Test Automation demands considerable onetime investment which is well worth when products have to be tested with high level of reliability. Automation allows one to execute a defined set of tests repeatedly and no human intervention is required to control or monitor the test execution. The goal of Automation is to reduce the number of test cases to be run manually and not to eliminate Manual testing altogether. 3.1 Importance of Test Automation Automated testing is important for the following reasons: www.accord-global.com Copyright © 2020 Accord Global Technology Solutions Private Limited. All rights reserved. Accord Global Technology Solutions • Manual Testing of all workflows, all fields, all negative/abnormal test scenarios is time and effort intensive. • Automation does not require Human intervention or requires minimal human intervention. One can run automated tests unattended (overnight) too. • Automation increases the speed of test execution (e.g. 10 hours manual test could be done in 2 hours using automation). • Automation helps increase Test Coverage. E.g. If a new feature is added to a product on every release of a software, then, only automated testing can guarantee that every part of the application is sufficiently tested as it grows with each new release. Basically, it is efficient to automate regression tests. • Manual Testing can become monotonous and hence prone to error. • Uncovers hard to find bugs which may slip while performing manual testing. E.g. Same test can be run 100 times to uncover many issues. • If production testing (Acceptance testing) or End of line testing is required to be done, then using automation testing, multiple units can be tested in parallel utilizing the test equipment to its maximum capacity and thereby increase productivity too. • Automated testing Improves accuracy. The repetitious nature of software updates and environment/hardware changes during the development cycle increase the chances of human error. Repeatable and predefined test automation delivers correct results every time. • Generate test reports for each and every test run. 4 Automated Test Equipment Automated test equipment (ATE) is a test equipment that performs testing on a device, which is unit under test (UUT), using automation to quickly perform measurements and evaluate the test results. An ATE can be a simple computer-controlled reader/data acquisition unit or digital multimeter, or a complicated system containing more than a dozen complex test instruments (real or simulated electronic test equipment) capable of automatically testing and diagnosing faults in sophisticated electronic LRU’s including system on chips, integrated circuits and PCBA’s (Printed Circuit Board Assembly). ATE includes the control hardware, sensors, and software that collects and analyzes the test results. ATE becomes cost effective for high-volume testing. www.accord-global.com Copyright © 2020 Accord Global Technology Solutions Private Limited. All rights reserved. Accord Global Technology Solutions In some cases, Automatic test equipment can be expensive, and therefore it is necessary to ensure that the correct philosophy and the correct type or types of automatic test equipment is used. Only by applying the use of automatic test equipment correctly can the maximum benefits be gained. 4.1 ATE’s – Simple to Complex Most of the manual work in a factory or an industry can be automated, be it programming a controller mounted on the PCBA, PCBA initial assembly checks (impedance test, ICT etc.) or production testing/End of line testing of an Unit produced in a production line or complex system on chip testing. Sometimes ATE’s may involve mechanical jigs and fixtures along with complex electronic equipment for testing for e.g. mechanical jigs may ease the mounting of the Unit under test to the ATE/test equipment. The ATE can be a simple bench top equipment or can be a complex equipment assembled in a rack. Below are some examples of ATE’s ranging from simple to complex. 4.1.1 Flashing the software to the Unit For any embedded electronic device to work, software/firmware needs to be loaded onto the unit. Often, this process consumes significant amount of time and involves selection of a particular software/firmware binary executable and download it into the unit through an appropriate UI/IDE. User also needs to select the IO link (can be RS-232, Ethernet etc.) through which the software/firmware file will be transferred to the unit and also should configure the IO link parameters. All of this process can be automated to reduce the manual intervention. FIGURE 1: FLASHING THE SOFTWARE TO THE UNIT With automation, user has to connect the unit to be programmed and just need to click a button in the PC. This programming automation can also be extended to program multiple units at a time. www.accord-global.com Copyright © 2020 Accord Global Technology Solutions Private Limited.