Accord Global Technology Solutions Importance of Testing Test Automation and Automated Test Equipment

Accord Global Technology Solutions

Importance of Testing

Test Automation and Automated Test Equipment

Abstract Testing is one of the major activities in a development life cycle of a product. Automation of testing will help reducing manual intervention and increasing the productivity. Automated Test Equipment (ATE) helps in automating the testing of the system/LRU during development and production. ATE’s can be a simple bench top setup or a set of test equipment arranged in a rack.

1 Introduction – Testing

737 MAX 8 Crashes (March 2019, NASA’s Mars orbiter crash US Northeast Black Out (Aug Oct 2018) (Sept 1999) 2013)

Overexposure of radiation Nissan recalled 9.9lac cars for therapy patients in Software Configuration error Airbag malfunction (March 2014) caused Airbus A400M crash (Jan National Cancer Institute, 2015) Panama City (Jun 2000)

What is common in all above disasters? Yes... Software was the root cause of the issue in each case. All accidents are preceded by a chain of events that trigger failure responses which cascade into a disaster. In most cases rigorous testing should be able to exercise these fault conditions and eventually avoid disasters.

Testing is one of the major activities of any development and change in any software or hardware. Tests are usually performed to verify and validate the expected behaviour against the intended behaviour in the form of requirements and specifications. The overall objective of testing is not to find every software bug/hardware issue that exists, but to expose situations that could negatively impact the operation, maintainability, and usability.

In today’s fast-growing world, customer’s expectation is to have products to be developed, implemented, installed and serviced faster than their competitors. Customer wants their product to be more competitive with better features to be implemented in a short time frame (as early as possible), but in the meantime they also want defect free product with defect free software. Releasing software/product to customer with issues may affect the user experience which eventually impacts the company reputation in the market. Therefore, severe emphasis is laid on the rigor of testing in Software/product/system Development Life Cycle.

Testing can be in various forms, for example early on in the development phase it could be reviews of artifacts such as requirements and design and later on can include traditional testing methods of verification and validation.

It is common knowledge in product development that cost of finding and fixing defect early on is always cheaper to finding and fixing defects later on, this paper also emphasis this point.

2 Importance of Testing

• In today’s competitive market, the survival of a product depends on its quality and it is imperative that adequate tests are performed on the product to make it successful in the market. • Some of the avionics or automotive application requires installation of the product with customer electronic system and hence if any defect in the product involves additional costs of repair and downtime of the system. This necessitates the product to be robust, reliable and free from defects.

While developing systems for critical applications, several design considerations related to its operational environment also needs to be considered. Some of these are lightning, ESD, EMI/EMC, Temperature, Vibration etc. It is also vital to test the equipment for robustness conditions in addition to normal tests.

After all, for the growth of any business the product should speak for itself and testing plays a key role in making this happen. But to make this happen one has to plan it properly before executing it.

3 Test Automation

We have seen that for any product (especially avionics or an automotive product), typically there are thousands of requirements and all these requirements are tested for normal and robustness cases. In some cases, equivalence classes conditions are also tested (as per DO-178 guidance). When it comes to production testing, few of the avionics products may have several thousand tests to be covered as part of Acceptance testing. It is also expected that 100% of products manufactured are required to be tested for its performance and operation compliance before delivery. Sometimes a single product production testing may consume more than 10 hours!!

In the event where such large amount of testing is required to be performed, it becomes a necessity to automate the tests to improve the production efficiency.

TEST AUTOMATION or AUTOMATION TESTING means using an automation tool/equipment to execute the test cases or tests. On the contrary, Manual Testing is performed manually by a human going through the test procedure sequentially. The test automation software can be structured to not only run the tests but also collate the results and generate a detailed report of the tests. Test Automation demands considerable onetime investment which is well worth when products have to be tested with high level of reliability.

Automation allows one to execute a defined set of tests repeatedly and no human intervention is required to control or monitor the test execution. The goal of Automation is to reduce the number of test cases to be run manually and not to eliminate Manual testing altogether.

3.1 Importance of Test Automation Automated testing is important for the following reasons:

• Manual Testing of all workflows, all fields, all negative/abnormal test scenarios is time and effort intensive. • Automation does not require Human intervention or requires minimal human intervention. One can run automated tests unattended (overnight) too. • Automation increases the speed of test execution (e.g. 10 hours manual test could be done in 2 hours using automation). • Automation helps increase Test Coverage. E.g. If a new feature is added to a product on every release of a software, then, only automated testing can guarantee that every part of the application is sufficiently tested as it grows with each new release. Basically, it is efficient to automate regression tests. • Manual Testing can become monotonous and hence prone to error. • Uncovers hard to find bugs which may slip while performing manual testing. E.g. Same test can be run 100 times to uncover many issues. • If production testing (Acceptance testing) or End of line testing is required to be done, then using automation testing, multiple units can be tested in parallel utilizing the test equipment to its maximum capacity and thereby increase productivity too. • Automated testing Improves accuracy. The repetitious nature of software updates and environment/hardware changes during the development cycle increase the chances of human error. Repeatable and predefined test automation delivers correct results every time. • Generate test reports for each and every test run.

4 Automated Test Equipment

Automated test equipment (ATE) is a test equipment that performs testing on a device, which is unit under test (UUT), using automation to quickly perform measurements and evaluate the test results. An ATE can be a simple computer-controlled reader/data acquisition unit or digital multimeter, or a complicated system containing more than a dozen complex test instruments (real or simulated electronic test equipment) capable of automatically testing and diagnosing faults in sophisticated electronic LRU’s including system on chips, integrated circuits and PCBA’s (Printed Circuit Board Assembly). ATE includes the control hardware, sensors, and software that collects and analyzes the test results. ATE becomes cost effective for high-volume testing.

In some cases, Automatic test equipment can be expensive, and therefore it is necessary to ensure that the correct philosophy and the correct type or types of automatic test equipment is used. Only by applying the use of automatic test equipment correctly can the maximum benefits be gained.

4.1 ATE’s – Simple to Complex Most of the manual work in a factory or an industry can be automated, be it programming a controller mounted on the PCBA, PCBA initial assembly checks (impedance test, ICT etc.) or production testing/End of line testing of an Unit produced in a production line or complex system on chip testing. Sometimes ATE’s may involve mechanical jigs and fixtures along with complex electronic equipment for testing for e.g. mechanical jigs may ease the mounting of the Unit under test to the ATE/test equipment. The ATE can be a simple bench top equipment or can be a complex equipment assembled in a rack. Below are some examples of ATE’s ranging from simple to complex.

4.1.1 Flashing the software to the Unit For any embedded electronic device to work, software/firmware needs to be loaded onto the unit. Often, this process consumes significant amount of time and involves selection of a particular software/firmware binary executable and download it into the unit through an appropriate UI/IDE. User also needs to select the IO link (can be RS-232, Ethernet etc.) through which the software/firmware file will be transferred to the unit and also should configure the IO link parameters. All of this process can be automated to reduce the manual intervention.

FIGURE 1: FLASHING THE SOFTWARE TO THE UNIT

With automation, user has to connect the unit to be programmed and just need to click a button in the PC. This programming automation can also be extended to program multiple units at a time.

4.1.2 Development Test System Development Test Systems are the automated system used for performing testing of the Unit during the development phase. This Test system can be used to perform some of the developmental and software qualification testing. In case of avionics system, the software in the unit needs to undergo DO-178 V&V and as part of that Hardware Software integration to be performed. This Development Test System automates this test process. The complexity of the system depends on the complexity of the unit under test. A simple and complex Test Systems developed are shown in Figure 2. A simple Test system shown is developed using python scripting and COTS hardware modules for acquiring the data from the UUT. Since UUT will have to be tested for different software releases, these kinds of Development Test System will help to reduce the test time and human effort.

FIGURE 2: A SIMPLE AND COMPLEX DEVELOPMENT TEST SYSTEM

4.1.3 PCBA Functional Testing PCBA functional testing is important to rule out any issues originating from the PCB assembly process. PCBA test starts with simple check for the short circuit test (checking impedance between ground and all voltage regulator output points) to complicated functionality tests. Manual testing of PCBA will require more time and effort. Automation of the tests saves time, manpower and increases the throughput. This can be a bench top equipment with simple jig mounted on it (This is also called as in circuit tester) for the assembly of the PCB and pogo pin/probing points to probe the test points on the PCBA.

FIGURE 3: IN CIRCUIT TESTER

FIGURE 4 : ATE FOR THE PCBA FUNCTIONAL TESTING

Using automation for the PCBA functional testing, multiple PCBA’s can be tested in parallel. Complex PCBA’s includes both RF and digital functionalities to be tested as part of functional testing. Figure 4 shows a tester for testing RF and digital functionalities of the PCBA. This tester is also capable of testing multiple CCA’s simultaneously.

4.1.4 Production Test Automation/End of Line Tester These Test Automation Equipment are built to test functionality test of the LRUs before dispatching to customer. Hence this is also called as End of Line tester. The development and design of these kinds

of Automation Equipment/Tester depends on the production test procedure, number of units to be tested during the production etc. The ATE can be constructed using a rack with wheels so that the Test system can be deployed in any area of the production line or this can be a tabletop tester, but again this depends on the complexity of the UUT. Avionics UUT’s may have multiple I/O’s such as ARINC, Ethernet, Audio/Video, RF I/O, Discrete I/O, Analog I/O interfaces. Testing all these I/O’s could involve more than thousands of tests. These tests may require more than 8 hours sometimes if run manually. Having an Automated Production tester can reduce the test execution to about one hour per unit. On the other hand, in the case of automotive UUT’s the main interfaces are CAN, Discrete I/O, Analog I/O, RS-232 etc. These units may also have Bluetooth, Wi-Fi, GSM functionalities as well. The production test equipment has to be developed to test all these interfaces along with the wireless interfaces. Testing wireless interface is a challenge and needs special faraday cages (Shown in Figure 7) to prevent outside radio interference from affecting the tests on the UUT.

FIGURE 5: ATE’S DEVELOPED FOR TESTING AVIONICS PRODUCTS

FIGURE 6: ATE’S DEVELOPED FOR TESTING AUTOMOTIVE UUT’S

FIGURE 7: RF SHIELDED BOX FOR PLACING THE AUTOMOTIVE UUT FOR TESTING

5 Summary

It is imperative to identify issues with a product and fix it before it can cause issues after installation. Therefore, testing plays a very important role in finding defects. ATE’s are used to automate the tests and reduces human effort in discovering the issues with the product or software and thereby saving cost to the company.

6 References

1. https://outfresh.com/knowledge-base/6-famous-software-disasters-due-lack-testing/ 2. Blog by Kyle McMeekin: https://www.qasymphony.com/blog/test-automation-automated- testing/ 3. https://en.wikipedia.org/wiki/Automatic_test_equipment

Acronyms

ARINC: Aeronautical Radio, Incorporated

ATE: Automated Test Equipment

CAN: Controller Area Network

CCA: Circuit Card Assembly

COTS: Commercial Off the Shelf

EMC: Electromagnetic Compatibility

EMI: Electromagnetic Interference

GSM: Global System for Mobile Communications

ICT: In-circuit test

IDE: Integrated Development Environment

IO, I/O: Input Output

LRU: Line Replaceable Unit

PC: Personal Computer

PCBA: Printed Circuit Board Assembly

PDU: Power Distribution Unit

RF: Radio Frequency

SCPCB: Signal Conditioning PCB

V&V: Verification and Validation

UI: User Interface

UUT: Unit Under Test

Wi-Fi: Wireless Fidelity

WTS: Wireless Test Station

About the Author

Suhas HN, Manager-Systems, is associated with Accord for more than 16 years with vast experience in the area of Avionics, GNSS and Automation testing Photo