View metadata, citation and similar papers at core.ac.uk brought to you by CORE SCIENCE & TECHNOLOGY TRENDS 2 Technical Trends and Challenges of Software Testing TOSHIAKI KUROKAWA MASATO SHINAGAWA Affiliated Fellow this system was originally planned to be in effect 1 Introduction from October 2007, testing showed that there were numerous problems. Therefore, the operation of Software tests are (herein referred to as software the system was delayed for 3 months, and it started testing) commonly known as “a test to verify to be in use from January 15, 2008. The Tokyo if there is no problem with software”. Software Stock Exchange also had an accident in November testing, in this sense, may seem to be similar to 2005, in which their system failed and all stock tests of various “artifacts (products)” such as ones exchanges were suspended. to check the safety of confectioneries, whether In 2007, there was also an inconvenience with batteries are produced as specified, or whether an the automatic gate system at train stations in and aircraft has any problems concerning its flight. In around the Tokyo metropolis. There have been fact, many problems have been revealed concerning many reports and opinions on this issue.[2-5] For inspections and testing of these products in recent example, a TV program reported that it was caused years. by a comparatively simple error (“+offset” was On the other hand, unlike such tangible products, missing). In this case, the bug was not noticed software would have different types of problems until a certain condition is met. Once this condition because problems in software are hard to see by occurred, it caused errors to all connected users until such problems actually are visualized by automatic gate machines. In addition, as this error a series of defects. In the case of tangible products, affected the entire rail industry of the metropolis you may notice some indication before the serious area, it was reported as a significant social problem problem occurs. In other words, unless the certain with several million people suffered. Such events conditions are tested for the defects, the software is should not happen if some test were conducted to considered and treated to be at normal state, or “no check these specific conditions of this accident. problem”. In a more extreme statement, it can be It was told that only such limited number of tests said that problems attributed to software are caused were conducted for big numbers as 1000, 5000 and by insufficiency of its testing. 10000, so they missed the special case relating to Recent software related problems are largely the buffer which caused failures. caused by lack of testing, and have actually Due to the increasing complexity and brought social problems and financial losses. proliferation of software, the importance of tests is For example, there was a failure with the New increasing. However, the difficulties of testing are Derivation Trading System, which deals with also increasing. There have been various attempts derivative products, of Tokyo Stock Exchange to solve these software testing problems. Recent on February 8, 2008. Due to this problem, the trends in software testing have been frequently trading system operation have been suspended reported in technical community.[6] For instance, a until the 12th of the month. The reported reason group for testing security software products named of the failure was an “initialization error with the Anti-Malware Testing Standards Organization memory within the server, and no initialization (AMTSO) was formed on February 5, 2008. As were conducted under a certain condition”.[1] While malware has become more diverse and products 34 QUARTERLY REVIEW No.29 / October 2008 has become more complex, the appropriate aims to eliminate bugs of programs while testing evaluation of the tests for current security software is to identify any defects in the whole program had become difficult. This organization is to including the system aspects where programs are improve testing methods and appeal the practical used. Therefore, it should be noted that these two standards and guidelines to the world. tasks are completely different in nature. In this report, first, the meaning of software For example, even if there are no bugs in a testing will be made clearer, especially for the so- program, defects may be found by testing. In other called “embedded systems”, so that the testing words, debugging is to identify whether a program becomes important for the technology related to satisfies the predetermined specifications, while the safety and security of citizens. Next, the testing testing is to identify any inconvenience for users of technologies are sketched, and the emergence the program. of companies and organizations specialized in Here is an example outside areas from software. software testing is reported. As the biggest issue Suppose some users may put “waterproof” products of software testing, this paper reports the current into the ocean or bath with some chemicals put as treatments of software test designers and engineers, bath agents. In these situations, the product needs as well as the current state of their education. to not only be waterproof against regular water, but The development of a quality assurance industry also to have certain durability against salt or other based on software testing technology is discussed chemicals. It is hardly said to be “waterproof” from along with the evaluation of software quality and the users’ viewpoint, that only the regular water is social/legal systems for software quality. We also permitted. need innovations to make all of these issues more Defects identified by debugging are attributed commonly understood by general public. This to cases that logic of an intended program was paper summarizes that the efforts to overcome such not achieved with the actual program, or there are challenges would contribute to create a “quality” mistakes in contents. On the other hand, system industry in a long run. failures to be identified by software testing are inconvenience or risk for users so that the software 2 Tests in over-all software should be designed and made to prevent such development processes– problems. differentiation from debugging – Buggy programs are written by the lack of skills of the programmer, and the improving individual In traditional software engineering, software programming skills is the responsibility of the testing was not emphasized much, compared to programmer. If software testing were considered other parts, partly because people did not recognize to be a process for debugging, programmers must the differences between software testing and be in charge of software testing, which is not a program debugging as described below. good practice. Putting programmers for testing may Software program failures itself may have been eventually lead to a higher risk of overseeing bugs seen even in the early days of programming. undetected. Efforts to eliminate failures are called debugging As an empirical rule in software engineering, since such failures in programs are called bugs. there should be at least some bugs unfound for While software tests are related to debugging, years in large programs with over ten-thousand they should be considered as a separate task from lines. Even new bugs (sometimes even crucial debugging. This idea was also emphasized in the ones) could be introduced during the program “Art of Software Testing” by Myers (published upgrading (or debugging) tasks. In lengthy and in 1979), which is said to be the classic book on large-scale programs, it is important to conduct software testing.[7] software testing from the viewpoint of fail-safe, The differences between debugging and software that is to prevent the system from being damaged testing may be summarized as shown in Table 1. significantly or critically even though some bugs Although both debugging and testing have the are remained and they are hit to do critically common goal of quality advancement, debugging damaging function. 35 SCIENCE & TECHNOLOGY TRENDS Table 1 : Difference of debugging and software test Debugging Testing Evaluate quality risks and Purpose Eliminate bugs guarantee an expected quality Subject Program System (including software) Work Correct faults Identify faults Worker Programmer Test engineer A test plan is created from Start After writing a program the phase of requirements specifications Judgment of the person in charge End (None = eternity) of testing Prepared by the STFC Engineering Center (SEC) of the Information 3 Definition and contents of current Processing Association (IPA), Japan. It is necessary software testing to investigate the testability from the upstream operations of requirements specifications, not There are still confusions in definitions of only to check if the users’ needs are met, but also software testing. According to [8], software testing to see how the tests will be effectively done. In is commonly known as a test to evaluate and then other words, the quality assurance process should guarantee the level of quality of software, or to start from the point in the upstream process where verify if there are no risks. With this definition, demands and requirements are studied, where the software testing needs to be started from the person in charge of the test must participate from upstream operations of software development. the start. According to the standard which stipulates the Moreover, as introduced in the Science and life cycle