Final Report, July 2007

Research into Testing Service Oriented Architectures: Final Report, July 2007

Prepared For: Avaya, Inc Contract Monitor: David Weiss Project Manager: Joann Ordille

Principal Investigator: Jeﬀ Oﬀutt George Mason University

EXECUTIVE SUMMARY

This is the final report for the Avaya Inc. sponsored project on testing service oriented architectures (including web applications and web services), which started August 2006. This report includes results presented in the three previous reports, plus new results. The purpose of this project is to identify improved methods to test software built with service oriented architectures (SOA) and web application technologies. This report includes an accompanying compressed rar file. The rar file contains a web page (index.html) and two directories (bypass-tests/ and isp-tests/) that contain files generated as part of this project. This includes tests, results from testing, input domain models, and prototype software tools. The major results of this project are summarized as: • Techniques for automating hand-generated ad-hoc tests for VIA. (Section 3) • Deploying the AutoBypass tool on Avaya’s server. (Section 3) • Recommending Selenium as a tool to automate non-bypass web application tests. (Section 3) • Specific recommendations for improving the project. (Section 3) • A complete process for designing, implementing, automating and developing bypass tests. (Section 4) • Adapting the bypass testing approach to the unique characteristics of the VIA software. This represented a significant engineering challenge, and a paper is under development. (Section 4) • 184 example bypass tests for six web-based VIA screens (Points of Contact, Notification Filter, Notification Profile, Time Profile, Change Pin and Create Account). (Section 4) • 92 of these tests failed, including 63 unique failures. Some of these failures are severe. This represents a publishable case study. (Section 4) • A process for modeling web-based user interface screens to generate input space partition (ISP) tests. (Section 5) • A method for designing an input domain model (IDM) for user interface screens. (Section 5) • Ways to handle the special input issues that arise from the technologies used in web-based user interfaces. (Section 5) • Example input domain models for six VIA web-based user interface screens. (Section 5) • An XML schema for representing IDMs in XML files. (Section 5) • Prototype automation tools for converting IDMs into HtmlUnit tests. (Section 5) • Two papers are in preparation for submission to refereed conferences. The first will feature the case study of bypass testing on VIA software, the second will describe the concept of using ISP to test web applications. Both papers will acknowledge Avaya and be co-authored with Avaya scientists. They should be completed during the summer and fall of 2007; plans are to submit at least one to the International Conference on Software Testing, Verification and Validation.

Although the PI’s opinion is that this project resulted in substantial and signiﬁcant results, this project had several problems. Following up on the discussion during the meeting at Avaya on April 20, the PI has identiﬁed several sources for these problems, which are analyzed below.

1. Problem: Student did not understand bypass testing and the PI did not supply enough over- sight. Impact: Moderate Solution: Once the problem was identiﬁed by the contract monitor, the PI instituted im- mediate procedural and management changes. The GMU team worked very hard to catch up with the expected tests. This was successfully handled and resulted in a positive solution and closeout of the project. Although delayed, the results for task 8 are quite strong and have resulted in signiﬁcant problems found in VIA software as well as an empirical paper, in progress.

2. Problem: The task list was modified at the beginning of the project, by adding a new major task–automating hand-generated, ad-hoc tests. Impact: Major Solution: The GMU team continually worked to adjust the schedule throughout the project. This led to the following problems: • Confused expectations–It took several months to clarify whether the GMU part of the team was responsible for designing and implementing all VIA bypass tests, or developing processes with sample tests and teaching Avaya personnel how to develop the remaining tests. it was eventually clarified that GMU was to develop the process and provide sample tests. • It was impossible to complete tasks 5 and 6 once the student left Avaya. • Because we were not able to complete tasks 5 and 6 in August, tasks 8, 10 and 11 took significantly more time than expected.

3. Problem: Unconventional use of Javascript in VIA web pages. Impact: Major Solution: The technique of packing client data into XML messages before transmitting to the server was not described or documented. Once the technological decisions were understood, the GMU team worked very hard to ﬁnd a way to automate tests. This eventually led to some positive results. The problem was exacerbated when Avaya personnel were not available to

2 the student at the beginning of the project (when the student was on-site). This had a major impact on tasks 6 and 7 and made task 10 impossible to complete.

4. Problem: Communication diﬃculties between the PI and student. Impact: Major Solution: Weekly meetings, followed by written documentation of the discussion helped, but this problem was never resolved.

5. Problem: Missing schemas for XML-transmitted data. Impact: Moderate Solution: After being told there were no XML schemas, the GMU team generated schemas by studying the individual XML messages. Much later, Avaya personnel provided GMU with the original schemas.

6. Problem: Oﬀutt’s account on Avaya’s system never worked. Impact: Minor Solution: This meant that all ﬁles were handled by the student.

3 1 Summary of Goals and Plans

This is the fourth and final report for the Avaya Inc. sponsored project on testing service oriented architectures (including web applications and web services), August 2006 through May 2007. The purpose of this project was to identify improved methods to test software built with service oriented architectures (SOA). Avaya Research Labs software technology’s research department is currently building SOA software. Several specific problems were identified in the Statement of Work (SOW): (1) How to save and reuse tests of the web and voice interactions with people, particularly in automating test scripts for regression testing; (2) How to validate correct behavior of web service components with regards to requirements; (3) How to detect failures when the incorrect behavior is only reflected in server-side data stores, such as databases and files; (4) Is it possible for external users to subvert web service components and violate security protocols; and (5) How to ensure that the system will recover correctly after loss of connectivity or failure of one or more components. The GMU project is working on aspects of problems 2-5, as specified in the SOW, but not on problem 1. The original task list, with annotations, is in Appendix A.

2 Overview of VIA Technology

VIA pages make heavy use of Javascript (JS). The VIA web pages use Javascript for three activities on the client’s computer:

1. Validate inputs. 2. Respond to user events and dynamically modify the screen by manipulating the document object model (DOM) to add, remove, and modify input ﬁelds and other selection elements. 3. Encode the input data into XML messages, which are then sent to the server. In most web applications, the user data from the client are sent as part of the HTTP request and accessed on the server by retrieving pairs of variables and values. The VIA system uses the novel approach of encoding input data into XML messages on the client, then submitting the XML messages. Javascript is used to encode the data into XML.

JS is commonly used to validate inputs (1) and respond to user events. Most development organization discourage developers from using JS to make more than small, minor, changes to the DOM. When heavily modifying the DOM (2), it is very difficult to analyze the effects, implement the JS correctly, test the JS, and avoid browser compatibility problems. Such web pages almost certainly cannot be made 508 accessibility compliant. Most importantly, this use of JS creates maintenance problems; the JS becomes so complicated that it is difficult for anyone but the original developer to modify. JS is commonly used in this way to off-load execution overhead from servers to clients. Clients are notoriously unreliable and inherently slow, thus this goal is seldom achieved. Ajax is a technology that would achieve almost the same effects without introducing these problems. Recommendation: It is strongly recommended that the JS in the VIA user interfaces be replaced with Ajax technology. The third use of JS, to encode the input data into XML messages, is original. Despite extensive research using textbooks, trade books, research articles, course materials, and discussions with numerous highly qualified J2EE / JS developers, the PI was not able to find any examples of this type of use. In fact, every expert developer who considered this approach was extremely negative. Using JS to pack data into XML introduces extra execution burden on both client and server,

4 introduces many chances for errors, and introduces maintenance problems. Browsers and web servers already have very efficient mechanisms for packing and transmitting data. If the data needs to be in XML on one of the server components, it is easier, more efficient, and more reliable to store the data into XML on the server. Recommendation: It is strongly recommended that the JS code in the user interfaces that packs data into XML be replaced by server-side software. Analysis of the Javascript and the attempt to apply bypass testing revealed a significant flaw: The VIA user interface screens have 100% failure rate under normal bypass testing. Thus the approach had to be significantly adapted, as detailed in Section 4.

3 Automating Web Tests for VIA

A significant contribution of this project was an effective technique to automate non-bypass, hand- generated, tests for the VIA web-based user interface screens. Although this activity was not in the original task list, it took an extensive amount of time (more than two months early in the project) and resulted in positive results for the VIA project. This effort involved extensive and intensive studies of various publicly available scripting tools. Because of the way VIA screens use JavaScript, several standard tools were not appropriate for automating these tests. The most suitable tool was found to be Selenium1. Recommendation: The publicly available Selenium is the most appropriate tool for automating non-bypass test scripts within the VIA project. Selenium can be accessed from its website: http://www.openqa.org/selenium/. An analysis of several open-source potential test scripting tools follows. GMU evaluated four test scripting tools for VIA: HttpUnit, JUnit, HtmlUnit and Selenium. HttpUnit is a suite of Java classes to test web applications and web services over HTTP. It is written in Java and emulates the essential portions of a browser’s behavior; including form submission, JavaScript, basic HTTP authentication, cookies and automatic page redirection. It also allows Java methods to examine pages returned from the server as containers of forms, tables, and links. HttpUnit can be combined with a JUnit testing framework and it is fairly easy to design and implement tests. Unfortunately, HttpUnit is inappropriate for VIA screens. The problem is that HttpUnit does not support all features of the JavaScript, so the heavy use of JavaScript in the VIA screens means that HttpUnit tests can only test a small part of the functionality. HtmlUnit is similar to HttpUnit in that it works within a JUnit framework to automate web application tests. The biggest difference between HtmlUnit and HttpUnit is the style of writing test cases. HttpUnit is centered on the hypertext transfer protocol, making it easy to create requests and send them to the server independently of whether the technology is HTML, servlet, PhP, or JSP pages. HtmlUnit knows more about HTML and works at a higher level of abstraction; testers can ask an HTML Page for HTML anchors and virtually “click” them or set the values for form fields such as text inputs and select box, then submit the inputs. With more research, we determined that both HtmlUnit and HttpUnit use an open-source implementation of JavaScript called Rhino. Rhino is written entirely in Java and is designed to be embedded into Java applications to provide scripting to end users. Unfortunately, Rhino does not support all features of JavaScript, and HtmlUnit has the same limitations with VIA that HttpUnit has. The final tool analyzed was Selenium. Selenium tests run directly in a browser, just as real users enter data. The tests run in several browsers, including Mozilla Firefox on Windows, Linux,

1Detailed information about Selenium can be found at: http://release.openqa.org/selenium- core/0.8.0/reference.html.

5 and Macintosh, and Internet Explorer. Thus, Selenium can be used to test conformance to diﬀerent browsers and operating systems. Selenium also has builtin features for regression tests. Techno- logically, Selenium uses JavaScript and Iframes to embed test automation engines into browsers. Theoretically, this technique should work with any JavaScript-enabled browser. Selenium extracts Javascript from the web pages and places them into separate ﬁles. Selenium’s ability to support all JavaScript features makes it the best choice for the VIA project. It can be combined with HttpUnit and HtmlUnit by using HttpUnit or HtmlUnit to retrieve controls from web pages, analyze their features and create Selenium test cases. Selenium can be accessed from its website: http://www.openqa.org/selenium/. A good reference can be found at: http://release.openqa.org/selenium-core/0.8.0/reference.html. The following steps can be used to run Selenium tests on Avaya’s network: 1. Open the link: http://ovid.research.avayalabs.com:8000/selenium/core/TestRunner.html 2. Enter “../avaya/AvayaTestSuite.html” in the text box shown in the page and then click button “Go” 3. Run tests (a) Click the button “All” on the right side of the page to run all test cases Or (b) Select one test from left side of the page and then click “selected” to run the test case selected 4. The results should be displayed. All of the hand-generated, non-bypass tests were delivered to Avaya in October.

4 Bypass Tests for VIA

The flow of control through web applications depends on user inputs and interactions, so input validation is used commonly to validate data before submitting them to the server. Though several techniques can be used to validate inputs on the client, users on the web have the ability to bypass this validation and submit invalid data to the server. This can cause unexpected behavior, and even allow unauthorized access. Bypass testing is a technique that sends invalid data to the server by bypassing client-side validation [16, 17]. Web application interfaces use browser events and script languages to react to user’s actions and block submission of data if the data do not meet some requirements. The interfaces may also provide feedback to the users. However, the web server cannot completely control users’ interactions. There are two reasons for this. First, HTML source and scripts are available to the users and can be modified and resubmitted to the server. Second, the Hyper Text Transfer Protocol (HTTP) is stateless. Thus, despite the constraints in the user interface, users can submit arbitrary requests and inputs to the server. Bypass testing is based on specific constraints that HTML can impose on data and on rules that are encoded in JavaScript. Rules are defined to create data that explicitly violate the constraints and rules, then invalid data is sent to the server. Bypass testing has been realized in an automated tool called AutoBypass [20, 15]. AutoBypass is a web application that accepts a URL to another web application under test (the subject) and automatically generates test cases based on the subject’s HTML form fields. AutoBypass has been tried on a mix of commercial, open source, and proprietary web applications, including a detailed empirical study on 16 commercial web applications (including amazon, google, Wells Fargo Life,

6 phpmyadmin, myspace, and yahoo). AutoBypass found failures in all but two applications, and signiﬁcant problems with several. AutoBypass intentionally “bypasses” JavaScript. Because VIA screens use JavaScript to construct parts of the screen and to package data, every AutoBypass test fails. Thus AutoBypass could not be used with the VIA application. Sending data to the server as XML messages imposes special constraints on the automation of VIA tests. As documented above, this meant that HttpUnit and HtmlUnit could not be used to automate ad-hoc testing, so Selenium was used. One of the goals of bypass testing is to avoid the JS that performs client-side error checking. Because of the unique characteristics of the VIA software, this was carried further. After some experimentation, the team at GMU came up the plan of completely ignoring the VIA screens (HTML pages), directly creating XML messages and submitting them to the server. The diﬃculty of this approach is determining the syntax of the XML messages. An advantage is that the tests, once designed and constructed, are easy to automate. This allowed the simple tool of HtmlUnit to be used to submit the XML messages.

4.1 HtmlUnit Bypass Tests HtmlUnit is a way to simulate a browser for testing purposes and is intended to be used within another testing framework such as JUnit. Programs can interact with web applications at a high level by dealing with documents instead of requests and responses. An HtmlUnit test is a Java class that inherits from the HtmlUnit abstract class TestCase. Each class includes one or more test cases, plus class variables that are common to all tests and methods that help setup and run tests. The tests that were created by this project include several elements as standard:

• Variables userID and password : Strings to log into the application. • Method setUp() : Retrieve a login form from the application and log in. • Methods testPC##() : Each test is implemented in a separate method that starts with the string “testPC” and includes the test case number. • Method Test Suite() : Uses the JUnit standard method TestSuite() to collect all methods in the class whose names start with the string “test” and create a test suite. These tests are then run automatically by JUnit.

Test case values are initially written as XML documents, then converted to HtmlUnit tests. Tests for six VIA web screens, NotificationFilterTestCase, NotificationProfileTestCase, PointsOf- ContactTestCase, TimeProfileTestCase, ChangePinTestCase, and CreateAccountTestCase are included in the accompanying rar file. The XML files, Java files, and a schema definition file for the XML test files are included. The following process for running the VIA bypass tests was delivered directly to the tester at Avaya (Xiao Ma) during the project.

1. Set up the execution environment to run the automated tests.

• C:\Program Files\Java\jdk1.5.0\_xx Java Development Kit (xx is the update number) • C:\Program Files\Java\jre1.5.0\_xx Java Runtime Engine (xx is the update number) • C:\JUnit JUnit library • C:\HtmlUnit HtmlUnit library • C:\ant Apache Ant library

7 The JUnit library can be downloaded from http://sourceforge.net/project/downloading.php? groupname=junit\&filename=junit3.8.1.zip\&use\_mirror=superb-west. After downloading, extract the files to C:\JUnit. The HtmlUnit library can be download from http://sourceforge.net/project/showfiles.php?group\_id=47038 After downloading, extract the files to C:\HtmlUnit. The Ant library can be downloaded from http://ant.apache.org/bindownload.cgi. After downloading, extract the files to C:\ant. 2. Initialize the environment variables

• JAVA_HOME Java installation directory, for example, C:\Program Files\Java\jdk1.5.0\_06. • JUNIT_HOME JUnit installation directory, for example, C:\JUnit. • HTMLUNIT_HOME HtmlUnit installation directory, for example, C:\HtmlUnit. • ANT_HOME Ant installation directory for example, C:\ant. • PATH Add %JAVA\_HOME%\bin;%ANT\_HOME%\bin to the path.

3. Create directories for the testing project (a) Create a project directory, for example, C:\testingVIA\ (b) Under the project directory, create a subdirectory \src\ (c) Download all the Java files that contain tests to the directory C:\testingVIA\src\. (d) Download the build.xml file to the project directory. (e) Execute ant to compile all of Java files. (f) Execute ant run.all to run all test suites. 4. Compile and run the tests (a) Execute ant to compile all of Java files. (b) Execute ant run.all to run all test suites.

4.2 Bypass Test Results on the Six Tested Screens A request from Avaya was to illustrate bypass testing on several VIA screens. Six screens were used, Points of Contact, Notification Filter, Notification Profile, Time Profile, Change Pin and Create Account. 184 tests were created for the six screens; 92 of which resulted in failure. Some failures were found by several tests, so only 63 of these failures were unique. The numbers are summarized below and detailed descriptions follow. Some tests revealed more than one failure; these are not included in the summary but are noted in the details below.

Web Screen Java File Tests Failing Tests Unique Failures Points of Contact PointsOfContactTestCase.java 42 23 12 Notification Filter NotificationFilterTestCase.java 26 16 7 Notification Profile NotificationProfileTestCase.java 34 12 6 Time Profile TimeProfileTestCase.java 53 23 23 Change Pin ChangePinTestCase.java 5 1 1 Create Account CreateAccountTestCase.java 24 17 14 TOTAL 184 92 63

8 4.2.1 Test Results for Points of Contact Summary – Tests: 42, Failures: 23, Unique Failures: 12

Description of Failures # Description Tests 1 A blank contact name should be invalid, but was saved successfully. testPC1 2 A blank email address should be invalid, but it was saved. testPC11, testPC12, testPC13, testPC14, testPC23, testPC24, testPC25 3 An invalid email address such as wqxiang@[email protected] was testPC7, testPC8, saved as valid data. testPC9, testPC10, testPC21, testPC22 4 For a voice type contact, 1234567c890 was saved as a valid phone testPC32 number. 5 For a voice type contact, [email protected] was saved as a testPC31 valid phone number. 6 When entering a blank name as contact, the server does not return testPC34 an error message or a correct message (no message at all). 7 When entering a contact with the full name value as “”, the server testPC36 does not return an error message or a correct message (no message at all). 8 When entering a contact and setting the old name value to be an testPC35 invalid name (“Con,tact”), the server indicates the test message was successfully sent to the contact. 9 When entering a contact and setting the full name value to be an testPC37 invalid name, the server indicates that the test message was successfully sent to the contact. 10 When deleting a correct contact, the server returns the message “An testPC38 error occurred while deleting contact “EmailContact1,”” 11 When entering a contact and setting the old name value to be “”, testPC39 the server does not return an error message or a correct message (no message at all). 12 When deleting a contact and setting the full name value to be an testPC41 invalid full name, the server does not return an error message or a correct message (no message at all).

9 4.2.2 Test Results for Notiﬁcation Filter Summary – Tests: 26, Failures: 16, Unique Failures: 7 (Tests 3, 4, 5, 6, 7, 8, 9 and 12 found more than one failure.)

Description of Failures # Description Tests 1 A blank filter name should be invalid, but was saved success- testNF2 fully. 2 A blank application name was saved. testNF3, testNF4, testNF5, testNF6, testNF7, testNF8 3 A blank application keyword was saved. testNF3, testNF4, testNF5, testNF6, testNF9, testNF12 4 For filter rules, a blank type was saved. testNF3, testNF4, testNF5, testNF6, testNF7, testNF8, testNF9, testNF10, testNF12, testNF13 5 For filter rules, a blank subject was saved. testNF14, testNF17, testNF20 6 When deleting a filter and setting the oldname to be “”, the testNF24 server does not return an error message or a correct message (no message at all). 7 When deleting a correct filter, the server does not return an testNF25 error message or a correct message (no message at all).

4.2.3 Test Results for Notiﬁcation Proﬁle Summary – Tests: 34, Failures: 12, Unique Failures: 6

Description of Failures # Description Tests 1 A blank profile name should be invalid, but was saved successfully. testNP2 2 A profile name that starts with a period “.” should be invalid, but testNP6 was saved successfully. 3 A profile name that contains the greater-than character “>” should testNP7 be invalid, but was saved successfully. 4 The days field for Initial Notification should be 0 or positive; a testNP11 negative number “-1” should be invalid, but was saved successfully. 5 The days field for Reminder should be 0 or positive; a negative testNP25 number “-1” should be invalid, but was saved successfully. 6 When creating a notification profile and setting invalid data for Re- testNP26 minder, the server does not return an error message or a correct message (no message at all).

10 4.2.4 Test Results for Time Proﬁle Summary – Tests: 53, Failures: 23, Unique Failures: 23 All failures for Time Proﬁle were found with test case 1 through 34. Subsequent tests all reported the same failure.

Description of Failures # Description Tests 1 A proﬁle name that starts with the character “.” should be invalid, but was testNP6 saved successfully. 2 A proﬁle name that contains the greater-than character “>” should be in- testNP7 valid, but was saved successfully. 3 Min of TimeZone should be a positive number, but a string “aa” was testNP12 saved successfully. 4 Min of TimeZone should be a positive number less than 60, but the num- testNP13 ber 100 was saved successfully. 5 Start day of Time Intervals should be a valid day of the week (one of testNP14 the strings “Monday,” “Tuesday,” ..., “Sunday”), but the string “ssss” was saved successfully. 6 Start day of Time Intervals should be a valid day of the week (one of the testNP15 strings “Monday,” “Tuesday,” ..., “Sunday”), but a blank string was saved successfully. 7 Start day of Time Intervals should be a valid day of the week (one of testNP16 the strings “Monday,” “Tuesday,” ..., “Sunday”), but a number was saved successfully. 8 Start hour of Time Intervals should be a number between 0 and 23, but testNP17 the number 60 was saved successfully. 9 Start hour of Time Intervals should be a number between 0 and 23, but testNP18 a blank string was saved successfully. 10 Start hour of Time Intervals should be a number between 0 and 23, but testNP19 the string “tttttt” was saved successfully. 11 Start minute of Time Intervals should be a number between 0 and 59, testNP20 but the number −1 was saved successfully. 12 Start minute of Time Intervals should be a number between 0 and 59, testNP22 but a blank string was saved successfully. 13 Start minute of Time Intervals should be a number between 0 and 59, testNP23 but the string “ttttt” was saved successfully. 14 End day of Time Intervals should be a valid day of the week (one of testNP24 the strings “Monday,” “Tuesday,” ..., “Sunday”), but the string “ssss” was saved successfully. 15 End day of Time Intervals should be a valid day of the week (one of the testNP25 strings “Monday,” “Tuesday,” ..., “Sunday”), but a blank string was saved successfully. 16 End day of Time Intervals should be a valid day of the week (one of testNP26 the strings “Monday,” “Tuesday,” ..., “Sunday”), but a number was saved successfully.

11 Description of Failures ... Time Proﬁle continued. # Description Tests 17 End minute of Time Intervals should be a number between 0 and 59, testNP27 but the number −1 was saved successfully. 18 End minute of Time Intervals should be a number between 0 and 59, testNP29 but a blank string was saved successfully. 19 End minute of Time Intervals should be a number between 0 and 59, testNP23 but the string “ttttt” was saved successfully. 20 End hour of Time Intervals should be a number between 0 and 23, but testNP31 the number 60 was saved successfully. 21 End hour of Time Intervals should be a number between 0 and 23, but testNP32 a blank string was saved successfully. 22 End hour of Time Intervals should be a number between 0 and 23, but testNP33 the string “tttttt” was saved successfully. 23 Test case testNP34 returned a testNP34 com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter error. Further accesses to the Time Proﬁle web page resulted in the page not being displayed correctly.

4.2.5 Test Results for Change Pin Summary – Tests: 5, Failures: 1, Unique Failures: 1

Description of Failures # Description Tests 1 The length of password can be longer than the stated limit of 16. testCP5

12 4.2.6 Test Results for Create Account Summary – Tests: 24, Failures: 17, Unique Failures: 14

Description of Failures # Description Tests 1 A blank employeenumber should be invalid, but was saved suc- testCA4 cessfully. 2 A blank Common Name(s) should be invalid, but was saved suc- testCA7 cessfully. 3 The invalid telephone number “telephonenumber121,tact” was saved testCA8 successfully. 4 The invalid mobile number “mobile1,tact” was saved successfully. testCA9 5 The invalid fax number “1,tact” was saved successfully. testCA10 6 The invalid email address testCA11 “wqxiang!@#$%*^%(*)^[email protected]” was saved successfully. 7 The invalid title testCA12 “title InvalidW!!!!~))((**^^%%$$##@@!>?ffsfsd*(_@$*) #_@$()#@$WERIOPWER_)@#$_* ##)*#@_($*(#@_*$_"/>” was saved successfully. 8 The invalid Manager name testCA13 “manager InvalidW!!!!~))((** ^^%%$$##@@!>?ffsfsd*(_@$*) #_@$()#@$WERIOPWER_)@#$_* ##)*#@_($*(#@_*$_” was saved successfully. 9 The invalid Department number testCA14 “Departmentnumber InvalidW!!!!~))((**^^%%$$##@@!>?ffsfsd *(_@$*)#_@$()#@$WERIOPWER_)@#$_* ##)*#@_($*(#@_*$_” was saved successfully. 10 The invalid Organization name testCA15 “Organization InvalidW!!!!~))((**^^%%$$##@@!>?ffs fsd* (_@$*)#_@$()#@$WERIOPWER_)@#$_* ##)*#@_($*(#@_*$_” was saved successfully. 11 Invalid values for Room number, Street, City, State and Postal testCA16, testCA17, Code were saved successfully. testCA18, testCA19, testCA20, testCA21 12 The name of an application should be “Sample” or “information,” testCA22 but a blank application name was saved successfully. 13 The name of an applications should be “Sample” or “information,” testCA23 but the string “aba@#* ” was saved successfully. 14 The property for Has Applications should be “true” or “false,” testCA24 but a blank string was saved successfully.

13 defineIndependent model partition choose Input Input Domain Testable Blocks Values Parameters Functions

Figure 1: General Input Space Partitioning Process

5 Input Space Partitioning for VIA

Input space partitioning (ISP) evolved from a black-box system testing technique known as the category-partition method, which was introduced by Ostrand and Balcer in 1988 [2, 18], and based on ideas in an earlier paper by Ostrand [19]. The idea is to identify inputs to a program and design a finite number of subdomains (partitioning) for each input. ISP is useful when the input space is, or can be considered to be, finite and well ordered in the sense that values can be grouped into a few partitions of reasonable size. The method assumes that any value from the same partition will be “just as good” as any other value. The process is summarized in Figure 1. The input domain is defined in terms of independent testable functions (sometimes called characteristics). Each function is modeled in terms of input parameters, which are then partitioned into regions called blocks, and specific values are selected from each block. The input parameters can be parameters to methods, global variables, objects representing current state, or user-level inputs to a program, depending on what kind of software artifact is being tested. The result is called a parameter model, or more generally, a domain model. The values from blocks are combined using criteria called combination strategies, which can be adjusted to generate more or fewer tests. This design process is generally called input domain modeling (IDM) (and sometimes input parameter modeling). The initial ideas by Ostrand and Balcer led to a combinatorial explosion in the number of tests. QN If all combinations of values are used to create tests, the number of tests is combinatorial ( i=1 vi, where vi is the number of values for each one of the N parameters). Combination strategies are used to decrease the number of test cases. Criteria include pair-wise, each-used, base-choice and t-wise as surveyed by Grindal et al. [10]. Pair-wise testing, introduced in 1994 as part of the AETG system by Burroughs, Cohen, Dalal, Erickson, Fredman Jain, Kajla, Parelius, and Patton [3, 4, 6, 5], pairs values from each block with each value from each other block. Each-used, introduced by 1994 by Ammann and Offutt [1], ensured that a value from each block is used at least once. Base-choice, also introduced in the same paper [1], asks the tester to use domain knowledge to choose a “base” block for each input parameter. One value from each base block is combined to form one base test, then each other value is used exactly once by changing one value at a time in the base test. An extension of pair-wise testing is t-wise, defined by Williams and Probert [22] in 1996, which requires every possible combination of values of t parameters be included in at least one test case. Work in developing efficient algorithms for pair-wise, t-wise and other criteria is ongoing [14, 21]. In 1993, Grochman, Grimm and Wegener [13, 12] introduced the use of classification trees to organize the information used by ISP. More recently, Grindal has looked at the process for creating domain models as a modeling problem [7, 9] and techniques for resolving conflicts in domain models [11]. Terminology in this area varies; category partition (CP) was the original term, but as the topic has evolved, this general technique has been called specification-based testing [1], combination strategies (usually in the context of test criteria), and input space partitioning. This report uses the most recent and general term, input space partitioning (ISP). The literature does not contain any papers on applying ISP to graphical user interfaces (GUIs) or to web-based user interfaces (WUIs). This project has identified several interesting decisions that must be made in applying ISP to WUIs, and created a complete domain model for several VIA screens. Domain models for other VIA pages are relatively straightforward, and the possibility of automating the process of designing domain models is exciting. This has always been assumed to be an activity that must be done by hand.

14 Design an Program to Embed the IDM HTMLUnit Web Screen input domain model automatically generate into an XML file executable tests (valid & invalid) HTMLUnit tests

Figure 2: Input Space Partitioning Process for Web-based User Interfaces

5.1 The Process of Applying Input Space Partitioning Input space partitioning starts by creating an input domain model (IDM) of the inputs to the software under test. The IDM must then be used to create tests. To enhance portability and ease maintenance, this project embeds the IDM into an XML ﬁle, then uses a program that automatically generates HtmlUnit tests (Java classes) from the XML ﬁles. This process is depicted in Figure 2.

5.1.1 Input Domain Modeling The most human-intensive portion of applying input space partitioning is usually creating the input domain model. With most programs, the tester starts by identifying functions that need to be tested (testable functions). This is straightforward with WUIs; each screen is a testable function. The tester then identifies the input parameters (characteristics) and divides the domain of each input parameter into sets of values (blocks), which collectively form a partition of the parameter’s input domain. Test inputs are combinations of values from one block from each parameter. Most scientists [6, 23, 8] who have worked in this area have used what is called an interface-based approach, which develops characteristics directly from input parameters to the program under test. Grindal [9, 7] suggested an alternative functionality-based approach, which develops characteristics from a functional or behavioral view of the program under test. The interface-based approach considers each particular parameter in isolation, which is almost mechanical to follow. However, some decisions of the software designer can not be used with an interface-based approach. The functionality-based approach allows the tester to incorporate some semantics or domain knowledge into the IDM. Although a functional-based approach could be used with WUIs, it is natural and mechanical to create interface-based IDMs. Each input element in a web page can be viewed as a separate parameter. Moreover, many input elements have pre-defined, finite, sets of values. For example, a list box can be partitioned with one element from the list per block. This means that most of the work of creating an IDM can be fully automated for web-based user interfaces! For string and textbox inputs, a default categorization of {blank, non-blank valid, non-blank invalid} can be used. Then testers can supply additional blocks and strings as needed. This last step brings in a valuable, but very simple, manual step into a process that is mostly automated. Although most of these decisions are straightforward, a web-based user interface introduces new issues in designing the domain model. As indicated in the previous paragraph, each input HTML element (text input box, drop-down box, check box, radio button, text area, password input box, option button, hidden input, list item and input button) that appears on the web page is modeled as a separate IPM parameter. This is a straightforward approach, allowing the IDM parameters to be easily identified and partitioned. However, several issues are not adequately resolved with this approach.

1. From an external perspective, it is impossible to determine whether the eﬀect of clicking a button will be to submit the HTML form data to the server or to change the web page on the client. 2. HTML buttons do not really have values in the traditional sense; they can only be clicked. 3. Only one HTML button can be clicked at a time.

15 Number of filters (10 max) : 3

Filter 1 Name : Filter 1 Value :

Filter 2 Name : Filter 2 Value :

Filter 3 Name : Filter 3 Value :

Figure 3: Simple Example of Dependent Form Elements

4. Some web pages have multiple forms, and only one form can be submitted to the server at a time. This means that combinations of values from diﬀerent forms are invalid. 5. Web-based user interface screens often have links that are used to navigate to other screens in the application and pages in the website. 6. Some HTML elements only appear after the value of another element is changed. That is, their existence depends on other elements. 7. Some HTML elements are removed from the screen when the value of another element is changed.

After some consideration, it seems reasonable to deal with issue 1 by ignoring it, that is, not caring whether the button submits the form data to the server or not. This is in keeping with the black-box intent of input domain modeling: The user should not care whether an interaction requires a round-trip to the server. Some of these issues can be handled by straightforward adaptations of the model. For example, issues 2 and 3 are modeled by calling one parameter the action parameter, and having the various buttons be mutually exclusive values for the parameter action. Issue 4 is handled by logically breaking the page into multiple testable functions. That is, each separate HTML form is considered to be a separate testable function, and they are tested independently. Issue 5 is unique to user interfaces. Some of the navigation links go to other screens in the web application, and others go to static HTML pages that are not part of the application UI. Such pages should not be considered in this model because they do not send inputs to the server. Navigation links should be tested using conventional testing methods; viewing the website as a graph and trying all links, or some combinations of links. Issues 6 and 7 are unique to graphical and web-based user interfaces, and we are not aware of techniques to model dependent form elements in the literature. Report III in April proposed using a hierarchical modeling approach. Consider the example in Figure 3, where one box allows the user to choose the number of filters, and the number chosen determines how many rows of filter entries are available. In the hierarchical modeling approach, the value for one parameter (in this example, num- Filters) is determined first, then the model for the dependent parameters is included, and values chosen accordingly. Thus, the domain model for the above example looks like (assuming valid values for the filters include any positive integer):

Level 1: numFilters : 0, 1, 2-9, 10 Assuming numFilter=3: Level 2: fName1 : blank, valid-name, invalid-not-blank

16 Name : ( N )

Phone number : ( F )

(dropdown) Phone type : ( L )

work (L1 yields X ) home (L2 yields Y ) mobile (L3 yields Z )

The choice for L causes one of the following three to appear (X is 3 input boxes) (Y is a textarea box) (Z is another dropdown list) a :

b : 1 2 c : 3

Figure 4: Example of Dynamic Form Elements

fValue1 : 1, 2, >1 fName2 : blank, valid-name, invalid-not-blank fValue2 : 1, 2, >1 fName3 : blank, valid-name, invalid-not-blank fValue3 : 1, 2, >1

While this approach is satisfactory for small simple examples, it gets complicated very quickly. With this approach, values for some parameters must be chosen before the rest of the domain model can be completed. The domain should be completely modeled before choosing any values.

5.1.2 Modeling Dynamically Changing Screens Figure 4 shows another dynamic web screen. The screen initially has three input elements, Name, Phone number, and Phone type (above the dashed line). Depending on which choice is made for Phone type, one of the three possibilities below the dashed line appears. If Phone type = work, the three input boxes under X appears. If Phone type = home, the text area box under Y appears. If Phone type = mobile, the dropdown list under Z appears. Three possibilities for modeling this situation have been considered. The ﬁrst is to model all six input elements {N, F, L, X, Y, Z} separately, then use constraints to restrict invalid combinations. This would mean that no value for L1 can be used with any value for Y or Z, yielding constraints such as: L1 != Y1, L1 != Y2, ... L1 != Z1, L1 != Z2, ... L2 != X.a1, L2 != X.a2, ..., L2 != X.b1, L2 != X.b2, ... Having to keep track of this many constraints is expensive at best and probably impractical. The next possibility is to use a functional parameter. This is based on the abstract parameter constraint handling approach from Grindal and Oﬀutt [11]. In the above example, this approach would combine L1 and X to create one new functional parameter, L1:X, L2 and Y to create another, L2:Y, and L3 and Z to create a third, L3:Z. If X.a, X.b, and X.c each has three values, X.a1, X.a2,

17 page baseurl testrequirement loginurl pageurl identifier

characteristic characteristic characteristic

input param name inputparam name ( form ) type ( selectfield ) type

inputparam name input param name ( inputfield ) type ( xml ) nodetype

value

value field name

fieldvalue

Figure 5: XML Tree for the Via Project

X.a3, X.b1, ..., then L1:X would have nine values. This method means that one parameter has a lot of values, which creates problems with some criteria. The third possibility is to use a submodel parameter, based on the submodel parameter constraint handling approach from Grindal and Oﬀutt [11]. Instead of explicitly modeling L, we split it into separate input domain models, based on the values of L: IDM 1: N, F, X IDM 2: N, F, Y IDM 3: N, F, Z This essentially eliminates L from the model by implicitly taking the assumption that L = work in IDM 1, L = home in IDM 2, and L = mobile in IDM 3. This approach is simple to use and does not result in a lot of messy information to compute. Thus it is clearly the best approach.

5.2 Automating Input Space Partitioning for VIA As shown in Figure 2, input domain models are saved in XML files. The hierarchical structure of the XML files is illustrated in Figure 5. The root element in the XML file is the testrequirement element. It has five attributes: page, baseurl, loginurl, pageurl and identifier. The page attribute designates which screen is going to be tested; the baseurl attribute is the absolute address of the web application under test; the loginurl attribute is the relative URL used to login to the web application before the test cases are executed; the pageurl attribute is the relative URL to navigate to the web page under test and the identifier attribute is used to distinguish the test cases from tests from other screens. The testrequirement contains one child element, the characteristic. Each characteristic element

18 corresponds to one testable function identified in the IDM. Test cases are generated for each characteristic based on the combination strategy selected and the final test suite is constructed by taking the union of the test cases generated from each characteristic. The examples provided use the simplest coverage criterion, each-choice [1]. The element characteristic has one child element, inputparam. It describes the physical parameter and values for the parameters. Each inputparam element is assigned at least two attributes. One attribute is name, which represents the physical name of the HTML elements in the web screens. The other attribute is type, which describes what kind of operation will be executed when generating the real test suite from the test requirement. Four types of inputparam elements have been defined. Their child elements are different for each type. The first type of inputparam is “form,” which corresponds to the HTML form in web pages. This type of inputparam does not contain any child element. The name of the HTML form is presented by the name attribute. The second type of inputparam element is “inputfield.” It represents textual HTML input fields (text input, text area, password and hidden input) that appear on the web page. The inputfield element has one child element, value, which is explained below. The third type of inputparam element is “selectfield.” It represents the HTML select element and also has one child element, value. The fourth and final type of inputparam is a special “xml” type. In the VIA project, some HTML elements are encoded into XML data, which is then assigned to a specific hidden input field. So the input parameters identified in the IDM, as encoded in XML, are mapped to the field child element instead of the inputparam element. The name attribute of this type of inputparam indicates which hidden input field is assigned to the XML data. The XML schema that describes the structure of the XML data is presented by another attribute, xmlns. The field element corresponds to the input parameters that are encoded in XML. It has the attribute name and the child element fieldvalue. Depending on how the parameter is sent to the server, the values for each parameter in the IDM are presented by either the value element or the fieldvalue element. If the parameter is sent to the server directly, each value is mapped to a value element as a child element of inputparam. Otherwise, each value is mapped to a fieldvalue element as a child element of field.

5.3 Example : Points of Contact Screen As an example, consider the Points of Contact (POC) screen from the VIA project, shown in Figure 6. The POC screen has several sections (called panes in UI parlance); this analysis restricts itself to the forms in the middle. The ﬁrst form, Contacts, has one drop down list and four buttons, Test, Modify, Load, and Delete. The second form, Change Contacts, has two buttons, Clear and Save, a Help navigation link, and three input elements. Name is a textbox, Contact Type is a dropdown list with the four values shown, and Address/Number is a textbox. When a value Contact Type is selected, other input elements appear on-screen (not shown in Figure 6). The rest of this discussion will focus on the Change Contacts form (characteristic), as it has dependent input elements.

5.3.1 Input Domain Model for POC The input domain model for POC includes an action parameter with values Save, Clear and Help and the parameter Name with blocks blank, non-blank unique valid, non-blank non-unique valid, and non-blank invalid. This part of the model uses domain knowledge to elaborate the default values of strings by splitting non-blank valid into two blocks, one where the name is unique and one where it is not. The Contact Type has four possible values that dynamically change the screen. The Address/Number input element is used diﬀerently depending on the Contact Type value, and other ﬁelds are added. Thus we employ a submodel parameter to split the IDM into four separate

19 Figure 6: Points of Contact Screen submodels, one for each value of Contact Type: IDM1 : Contact Type = Electronic Mail IDM2 : Contact Type = Numeric Pager IDM3 : Contact Type = SMS IDM4 : Contact Type = Voice Table 1 deﬁnes the IDM for the POC screen. It is divided into ﬁve sections, one for each submodel of the Change Contact characteristic, plus another for the Contacts characteristic.

5.3.2 Representing POC’s IDM in XML As indicated in the process shown in Figure 2, IDMs are embedded into XML files. Part of the XML file for the POC IDM is shown in Figure 7 (the complete XML file is in the accompanying rar file). The root element is testrequirement, which has five attributes that are common to all the tests. The attribute page defines which HTML page (screen) is being tested, baseurl defines the initial URL for all the pages involved in the test, loginurl defines the relative URL to the login page for the application, pageurl defines the relative URL to the page being tested, and identifier is a unique string for this page (PC is an abbreviation for Points of Contact). Each sub-domain from Table 1 is embedded into a separate characteristic tag in the XML. The first input parameter in the IDM is Action (acctform), which is mapped to an inputparam element. The type attribute inputfield and the name attribute cmd together indicate that the physical parameter of Action is an HTML input field whose name is cmd. This parameter has three value elements, one for each block in the IDM. The next parameter is Name, which has four values. Because the second block requires a unique valid string, the corresponding value has the attribute unique=“true,” indicating that if this value is used in multiple test cases, its value must be different each time. The POC web page encodes the other four input parameters into XML and stores them in a hidden field named state. This is modeled by creating an inputparam element with type XML

20 Points of Contact IDM Characteristic Change Contact, Sub-IDM 1 : Contact Type = Electronic Mail Parameter Block 1 Block 2 Block 3 Block 4 Constraints Action Save Clear Help Name blank non-blank, non-blank, non-blank, unique, valid non-unique, invalid valid Contact Type Electronic Mail Address blank Valid address Invalid address Content Parameter HTML Plain text External Parameter TRUE FALSE Characteristic Change Contact, Sub-IDM 2 : Contact Type = Numeric Pager Parameter Block 1 Block 2 Block 3 Block 4 Constraints Action Save Clear Help Name blank non-blank, non-blank, non-blank, unique, valid non-unique, invalid valid Contact Type Numeric Pager Number blank Valid phone Invalid phone number number Characteristic Change Contact, Sub-IDM 3 : Contact Type = SMS Parameter Block 1 Block 2 Block 3 Block 4 Constraints Action Save Clear Help Name blank non-blank, non-blank, non-blank, unique, valid non-unique, invalid valid Contact Type SMS Address blank Valid address Invalid address External Parameter TRUE FALSE Characteristic Change Contact, Sub-IDM 4 : Contact Type = Voice Parameter Block 1 Block 2 Block 3 Block 4 Constraints Action Save Clear Help Name blank non-blank, non-blank, non-blank, unique, valid non-unique, invalid valid Contact Type Voice Number blank Valid phone Invalid phone number number Characteristic Contacts Parameter Block 1 Block 2 Block 3 Block 4 Constraints Action Test Modify Load Delete oldname blank non-blank, non-blank, valid invalid fullname blank non-blank, non-blank, non-blank, oldname not oldname, not oldname, valid invalid

21 Table 1: Input Domain Model for POC and name state. This parameter has four children, one for each parameter in the IDM. A value for each block in the IDM is stored in a fieldvalue element. The remaining characteristics are in the accompanying documents, in isp-tests/IDM-PointsOfContact.xml. Also included are example IDMs for Notification Filter, Notification Profile, Time Profile, Change Pin and Create Account. The accompanying documents also include three Java files; XmlEncodingInterface.java and Test- CaseXmlGen.java translates an IDM to a test case, and TestCaseCodeGen.java generates an HTM- LUnit file from a test case file. These programs, although functional, should be considered example prototypes. It is important to note that creating IDMs is a type of design modeling. Just as when modeling software behavior, the ability to effectively model tests is enhanced by experience, education, and knowledge of the software under test. Not surprisingly, different testers will create different IDMs. Some will be better than others, some will be worse, and some will simply be different.

References

[1] Paul Ammann and Jeff Offutt. Using formal methods to derive test frames in category-partition testing. In Proceedings of the Ninth Annual Conference on Computer Assurance (COMPASS 94), pages 69–80, Gaithersburg MD, June 1994. IEEE Computer Society Press. [2] M. Balcer, W. Hasling, and T. Ostrand. Automatic generation of test scripts from formal test specifications. In Proceedings of the Third Symposium on Software Testing, Analysis, and Verification, pages 210–218, Key West Florida, December 1989. ACM SIGSOFT 89. [3] K. Burroughs, A. Jain, and R. L. Erickson. Improved quality of protocol testing through techniques of experimental design. In Proceedings of the IEEE International Conference on Communications (Supercomm/ICC’94), pages 745–752, New Orleans LA, USA, May 1994. IEEE. [4] D. M. Cohen, S. R. Dalal, A. Kajla, and G. C. Patton. The automatic efficient test generator (AETG) system. In Proceedings of Fifth International Symposium on Software Reliability Engi- neering (ISSRE’94), pages 303–309, Los Alamitos, CA, USA, November 1994. IEEE Computer Society. [5] David M. Cohen, Siddhartha R. Dalal, Michael L. Fredman, and Gardner C. Patton. The AETG system: An approach to testing based on combinatorial design. IEEE Transactions on Software Engineering, 23(7):437–444, July 1997. [6] David M. Cohen, Siddhartha R. Dalal, Jesse Parelius, and Gardner C. Patton. The combinatorial design approach to automatic test generation. IEEE Software, pages 83–88, September 1996. pair-wise, input space partition, combination choice. [7] Mats Grindal. Evaluation of Combination Strategies for Practical Testing. PhD thesis, Skovde University / Linkoping University, Skovde Sweden, 2007. [8] Mats Grindal, Birgitta Lindström,Jeff Offutt, and Sten F. Andler. An evaluation of combination testing strategies. Empirical Software Engineering, 11(4):583–611, December 2006. [9] Mats Grindal and Jeff Offutt. Input parameter modeling for combination strategies. In IASTED International Conference on Software Engineering (SE 2007), Innsbruck, Austria, February 2007. ACTA Press. [10] Mats Grindal, Jeff Offutt, and Sten F. Andler. Combination testing strategies: A survey. Software Testing, Verification, and Reliability, 15(2):97–133, September 2005.

Save Clear Help Contact Contact Con,tact email [email protected] wqxiang@[email protected] html plain true false

Figure 7: XML for POC IDM

23 [11] Mats Grindal, Jeff Offutt, and Jonas Mellin. Conflict management when using combination strategies for software testing. In Australian Software Engineering Conference (ASWEC 2007), pages 255–264, Melbourne, Australia, April 2007. [12] M. Grochtmann, K. Grimm, and J. Wegener. Tool-supported test case design for black- box testing by means of the classification-tree editor. In Proceedings of the 1st European International Conference on Software Testing Analysis & Review (EuroSTAR 1993), pages 169–176, London, Great Britain, October 1993. [13] Matthias Grochtmann and Klaus Grimm. Classification trees for partition testing. Software Testing, Verification, and Reliability, 3(2):63–82, 1993. [14] Y. Lei and K. C. Tai. In-parameter-order: A test generation strategy for pair-wise testing. In Proceedings of the Third IEEE High Assurance Systems Engineering Symposium, pages 254–261. IEEE, November 1998. [15] Jeff Offutt, Vasileios Papadimitriou, and Upsorn Praphamontripong. An automated tool for bypass testing of web applications. In Submitted for conference review, 2007. [16] Jeff Offutt, Ye Wu, Xiaochen Du, and Hong Huang. Bypass testing of web applications. In 15th International Symposium on Software Reliability Engineering, pages 187–197, Saint-Malo, Bretagne, France, November 2004. IEEE Computer Society Press. [17] Jeff Offutt, Ye Wu, Xiaochen Du, and Hong Huang. Web application bypass testing. In Work- shop on Quality Assurance and Testing of Web-Based Applications; Associated with COMP- SAC 2004, Hong Kong, PRC, September 2004. [18] T. J. Ostrand and M. J. Balcer. The category-partition method for specifying and generating functional tests. Communications of the ACM, 31(6):676–686, June 1988. [19] Thomas J. Ostrand, R. Sigal, and Elaine J. Weyuker. Design for a tool to manage specification- based testing. In Proceedings of the Workshop on Software Testing, pages 41–50, Banff, Al- berta, July 1986. IEEE Computer Society Press. [20] Vasileios Papadimitriou. Automating bypass testing for web applications. Master’s thesis, De- partment of Information and Software Systems Engineering, George Mason University, Fairfax VA, 2006. (Available on the web at http://www.ise.gmu.edu/∼offutt/). [21] K. C. Tai and Y. Lei. A test generation strategy for pairwise testing. IEEE Transactions on Software Engineering, 28(1):109–111, January 2002. [22] A. W. Williams and R. L. Probert. A measure for component interaction test coverage. In Pro- ceedings of the ACSI/IEEE International Conference on Computer Systems and Applications (AICCSA 2001), pages 304–311, Beirut, Lebanon, June 2001. [23] H. Yin, Z. Lebne-Dengel, and Y. K. Malaiya. Automatic test generation using checkpoint encoding and antirandom testing. Technical Report CS-97-116, Colorado State University, 1997.

24 A Appendix: Original Task List

The following task list was initially proposed by Offutt (PI) on 9 August and agreed to by Or- dille and Weiss. Tasks 1-7 were originally designated for the graduate research assistant (GRA), Qingxiang Wang (QW) to be carried out at Avaya, with the purpose of laying the groundwork to successfully finish tasks 8-13 by the GRA and PI at GMU in Fairfax. The GRA was assigned to a different task while at Avaya; primarily creating hand-generated, ad-hoc, non-bypass tests for Diamond/VIA. This task was not on the original task list. This change had a significant impact on the schedule, and his work on the tasks listed below started more than a month after his return to GMU. Each original task is presented verbatim from the August list in bold-face, with annotations in italics.

1. Become familiar with AutoBypass. Give demo to Joanne, others - week of 14 Aug. (QW) Task was delayed by QW’s reassignment at Avaya, then further delayed by the travel schedule of the author of AutoBypass, Vasileios Papadimitriou. Completed in September. AutoBypass is not directly helpful for the VIA project, because of the unusual use of Javascript in the VIA pages. 2. Gain experience with JUnit. (QW) - by 18 Aug. Task delayed by QW’s reassignment at Avaya, completed in September. 3. Gain experience with HTTPUnit (QW) - by 18 Aug. Task delayed by QW’s reassignment at Avaya, completed in September. (Also HtmlUnit and Selenium.) Selenium was identified as the best tool to automate VIA’s non-bypass tests and HtmulUnit was identified as the best tool to automate VIA’s bypass tests. 4. Evaluate Javascript unit testing tools. • www.jsunit.net • jsassertunit.sourceforge.net/docs/ • jsunit.berlios.de Determine which will work best for us. (QW) - by 18 Aug. Task delayed by QW’s reassignment at Avaya, completed in September. 5. Understand the string-based communication between contexts. Uses a singleton class in the shared class loader. Find the schemas that define the strings. (See step 11 below). (QW) - by 18 Aug. Task delayed by QW’s reassignment at Avaya. This task required active communication between Hamilton Slye and QW; Hamilton was never available thus this was never completed. 6. Study the JavaScript-generated VIA HTML pages. Work with Hamilton to understand how some of them work. (See step 10 below). (QW) - by 25 Aug. Task delayed by QW’s reassignment at Avaya. Hamilton was never available to help with this task, thus it took significantly more time than expected (several weeks instead of a few hours). In August, Hamilton said the XML messages involved were not defined by schemas, but in March found the schemas. Hamilton also made no mention of the very unusual ways in which JS was being used; in particular the use of JS to pack client-side data into XML messages before transmitting to the server. Completed in January.

25 7. Run AutoBypass on VIA pages. (a) Install AutyByass at Avaya. (b) Faults found ? (c) How to check results ? (d) Can it ﬁnd JS-generated form elements ? (QW) - by 25 Aug. This task was rendered obsolete by the unusual use of JavaScript in VIA pages.

*** Qingxiang back to Fairfax ***

8. Construct bypass-style hand-generated tests for VIA pages. Write as HTTPUnit tests. (QW), (AJO) - by 29 Sept. Task delayed by QW’s reassignment at Avaya, and further delayed by theoretical misunder- standings by QW. Completed March 2007 as HtmlUnit tests. 9. Use Input Space Partitioning (ISP) to generate tests for screen interactions. 31 screens : 14 user screens, 3 user utilities, 7 account screens, 3 debug, 4 popups • Analyze form elements (characteristics) • Partition into values (select boxes, radio boxes, text fields, ...) • Choose “base” elements • Choose combination criterion • Combine to form tests • Write as J,HTTP,JSUnit tests (QW), (AJO) - by 13 Oct. Task delayed by delay in previous tasks, begun February and a process and example input domain models for six VIA screens completed in May. 10. Design ways to test JavaScript in VIA HTML pages. • How can coverage be assured? • Do tests have to be created and run by hand? • Can we use HTTPUnit or JSUnit? (QW), (AJO) - by 27 Oct. Significantly more difficult than expected because of the unusual ways that JavaScript is used in VIA pages. Existing tools (such as JSUnit) cannot be used. Very little progress was made on this task. 11. Analyze string-based communication between contexts. • Study schemas. • Construct tests based on XML modification rules (Wuzhi’s research). • Write as HTTPUnit tests.

26 (QW), (AJO) - by 10 Nov. This task was made impossible because of Hamilton’s unavailability (see task 5). 12. Deliver all tests to Avaya. (QW), (AJO) - by 17 Nov. Tests were delivered as created from November through April. All tests are included with this final report (in a separate rar file). 13. Write paper as case study of testing web application. Every step and decision must be documented!!! (QW), (AJO), (JOrd), ... - Spring 2007. This task is still in process and will continue through the remainder of 2007 (unfunded). Two papers are planned, a case study using bypass testing and input space partitioning applied to web applications. Preliminary notes for these papers are included as separate documents in the rar file.