Research into Testing Service Oriented Architectures: Final Report, July 2007 Prepared For: Avaya, Inc Contract Monitor: David Weiss Project Manager: Joann Ordille Principal Investigator: Je® O®utt George Mason University EXECUTIVE SUMMARY This is the ¯nal report for the Avaya Inc. sponsored project on testing service oriented archi- tectures (including web applications and web services), which started August 2006. This report includes results presented in the three previous reports, plus new results. The purpose of this project is to identify improved methods to test software built with service oriented architectures (SOA) and web application technologies. This report includes an accompanying compressed rar ¯le. The rar ¯le contains a web page (index.html) and two directories (bypass-tests/ and isp-tests/) that contain ¯les generated as part of this project. This includes tests, results from testing, input domain models, and prototype software tools. The major results of this project are summarized as: ² Techniques for automating hand-generated ad-hoc tests for VIA. (Section 3) ² Deploying the AutoBypass tool on Avaya's server. (Section 3) ² Recommending Selenium as a tool to automate non-bypass web application tests. (Section 3) ² Speci¯c recommendations for improving the project. (Section 3) ² A complete process for designing, implementing, automating and developing bypass tests. (Section 4) ² Adapting the bypass testing approach to the unique characteristics of the VIA software. This represented a signi¯cant engineering challenge, and a paper is under development. (Section 4) ² 184 example bypass tests for six web-based VIA screens (Points of Contact, Noti¯cation Filter, Noti¯cation Pro¯le, Time Pro¯le, Change Pin and Create Account). (Section 4) ² 92 of these tests failed, including 63 unique failures. Some of these failures are severe. This represents a publishable case study. (Section 4) ² A process for modeling web-based user interface screens to generate input space partition (ISP) tests. (Section 5) ² A method for designing an input domain model (IDM) for user interface screens. (Section 5) ² Ways to handle the special input issues that arise from the technologies used in web-based user interfaces. (Section 5) ² Example input domain models for six VIA web-based user interface screens. (Section 5) ² An XML schema for representing IDMs in XML ¯les. (Section 5) ² Prototype automation tools for converting IDMs into HtmlUnit tests. (Section 5) ² Two papers are in preparation for submission to refereed conferences. The ¯rst will feature the case study of bypass testing on VIA software, the second will describe the concept of using ISP to test web applications. Both papers will acknowledge Avaya and be co-authored with Avaya scientists. They should be completed during the summer and fall of 2007; plans are to submit at least one to the International Conference on Software Testing, Veri¯cation and Validation. Although the PI's opinion is that this project resulted in substantial and signi¯cant results, this project had several problems. Following up on the discussion during the meeting at Avaya on April 20, the PI has identi¯ed several sources for these problems, which are analyzed below. 1. Problem: Student did not understand bypass testing and the PI did not supply enough over- sight. Impact: Moderate Solution: Once the problem was identi¯ed by the contract monitor, the PI instituted im- mediate procedural and management changes. The GMU team worked very hard to catch up with the expected tests. This was successfully handled and resulted in a positive solution and closeout of the project. Although delayed, the results for task 8 are quite strong and have resulted in signi¯cant problems found in VIA software as well as an empirical paper, in progress. 2. Problem: The task list was modi¯ed at the beginning of the project, by adding a new major task{automating hand-generated, ad-hoc tests. Impact: Major Solution: The GMU team continually worked to adjust the schedule throughout the project. This led to the following problems: ² Confused expectations{It took several months to clarify whether the GMU part of the team was responsible for designing and implementing all VIA bypass tests, or developing processes with sample tests and teaching Avaya personnel how to develop the remaining tests. it was eventually clari¯ed that GMU was to develop the process and provide sample tests. ² It was impossible to complete tasks 5 and 6 once the student left Avaya. ² Because we were not able to complete tasks 5 and 6 in August, tasks 8, 10 and 11 took signi¯cantly more time than expected. 3. Problem: Unconventional use of Javascript in VIA web pages. Impact: Major Solution: The technique of packing client data into XML messages before transmitting to the server was not described or documented. Once the technological decisions were understood, the GMU team worked very hard to ¯nd a way to automate tests. This eventually led to some positive results. The problem was exacerbated when Avaya personnel were not available to 2 the student at the beginning of the project (when the student was on-site). This had a major impact on tasks 6 and 7 and made task 10 impossible to complete. 4. Problem: Communication di±culties between the PI and student. Impact: Major Solution: Weekly meetings, followed by written documentation of the discussion helped, but this problem was never resolved. 5. Problem: Missing schemas for XML-transmitted data. Impact: Moderate Solution: After being told there were no XML schemas, the GMU team generated schemas by studying the individual XML messages. Much later, Avaya personnel provided GMU with the original schemas. 6. Problem: O®utt's account on Avaya's system never worked. Impact: Minor Solution: This meant that all ¯les were handled by the student. 3 1 Summary of Goals and Plans This is the fourth and ¯nal report for the Avaya Inc. sponsored project on testing service oriented architectures (including web applications and web services), August 2006 through May 2007. The purpose of this project was to identify improved methods to test software built with service oriented architectures (SOA). Avaya Research Labs software technology's research department is currently building SOA software. Several speci¯c problems were identi¯ed in the Statement of Work (SOW): (1) How to save and reuse tests of the web and voice interactions with people, particularly in automating test scripts for regression testing; (2) How to validate correct behavior of web service components with regards to requirements; (3) How to detect failures when the incorrect behavior is only reflected in server-side data stores, such as databases and ¯les; (4) Is it possible for external users to subvert web service components and violate security protocols; and (5) How to ensure that the system will recover correctly after loss of connectivity or failure of one or more components. The GMU project is working on aspects of problems 2-5, as speci¯ed in the SOW, but not on problem 1. The original task list, with annotations, is in Appendix A. 2 Overview of VIA Technology VIA pages make heavy use of Javascript (JS). The VIA web pages use Javascript for three activities on the client's computer: 1. Validate inputs. 2. Respond to user events and dynamically modify the screen by manipulating the document object model (DOM) to add, remove, and modify input ¯elds and other selection elements. 3. Encode the input data into XML messages, which are then sent to the server. In most web applications, the user data from the client are sent as part of the HTTP request and accessed on the server by retrieving pairs of variables and values. The VIA system uses the novel approach of encoding input data into XML messages on the client, then submitting the XML messages. Javascript is used to encode the data into XML. JS is commonly used to validate inputs (1) and respond to user events. Most development organization discourage developers from using JS to make more than small, minor, changes to the DOM. When heavily modifying the DOM (2), it is very di±cult to analyze the e®ects, implement the JS correctly, test the JS, and avoid browser compatibility problems. Such web pages almost certainly cannot be made 508 accessibility compliant. Most importantly, this use of JS creates maintenance problems; the JS becomes so complicated that it is di±cult for anyone but the original developer to modify. JS is commonly used in this way to o®-load execution overhead from servers to clients. Clients are notoriously unreliable and inherently slow, thus this goal is seldom achieved. Ajax is a technol- ogy that would achieve almost the same e®ects without introducing these problems. Recommendation: It is strongly recommended that the JS in the VIA user interfaces be replaced with Ajax technology. The third use of JS, to encode the input data into XML messages, is original. Despite extensive research using textbooks, trade books, research articles, course materials, and discussions with numerous highly quali¯ed J2EE / JS developers, the PI was not able to ¯nd any examples of this type of use. In fact, every expert developer who considered this approach was extremely negative. Using JS to pack data into XML introduces extra execution burden on both client and server, 4 introduces many chances for errors, and introduces maintenance problems. Browsers and web servers already have very e±cient mechanisms for packing and transmitting data. If the data needs to be in XML on one of the server components, it is easier, more e±cient, and more reliable to store the data into XML on the server. Recommendation: It is strongly recommended that the JS code in the user interfaces that packs data into XML be replaced by server-side software.