DOCSLIB.ORG

Final Report, July 2007

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Final Report, July 2007 Research into Testing Service Oriented Architectures: Final Report, July 2007 Prepared For: Avaya, Inc Contract Monitor: David Weiss Project Manager: Joann Ordille Principal Investigator: Je® O®utt George Mason University EXECUTIVE SUMMARY This is the ¯nal report for the Avaya Inc. sponsored project on testing service oriented archi- tectures (including web applications and web services), which started August 2006. This report includes results presented in the three previous reports, plus new results. The purpose of this project is to identify improved methods to test software built with service oriented architectures (SOA) and web application technologies. This report includes an accompanying compressed rar ¯le. The rar ¯le contains a web page (index.html) and two directories (bypass-tests/ and isp-tests/) that contain ¯les generated as part of this project. This includes tests, results from testing, input domain models, and prototype software tools. The major results of this project are summarized as: ² Techniques for automating hand-generated ad-hoc tests for VIA. (Section 3) ² Deploying the AutoBypass tool on Avaya's server. (Section 3) ² Recommending Selenium as a tool to automate non-bypass web application tests. (Section 3) ² Speci¯c recommendations for improving the project. (Section 3) ² A complete process for designing, implementing, automating and developing bypass tests. (Section 4) ² Adapting the bypass testing approach to the unique characteristics of the VIA software. This represented a signi¯cant engineering challenge, and a paper is under development. (Section 4) ² 184 example bypass tests for six web-based VIA screens (Points of Contact, Noti¯cation Filter, Noti¯cation Pro¯le, Time Pro¯le, Change Pin and Create Account). (Section 4) ² 92 of these tests failed, including 63 unique failures. Some of these failures are severe. This represents a publishable case study. (Section 4) ² A process for modeling web-based user interface screens to generate input space partition (ISP) tests. (Section 5) ² A method for designing an input domain model (IDM) for user interface screens. (Section 5) ² Ways to handle the special input issues that arise from the technologies used in web-based user interfaces. (Section 5) ² Example input domain models for six VIA web-based user interface screens. (Section 5) ² An XML schema for representing IDMs in XML ¯les. (Section 5) ² Prototype automation tools for converting IDMs into HtmlUnit tests. (Section 5) ² Two papers are in preparation for submission to refereed conferences. The ¯rst will feature the case study of bypass testing on VIA software, the second will describe the concept of using ISP to test web applications. Both papers will acknowledge Avaya and be co-authored with Avaya scientists. They should be completed during the summer and fall of 2007; plans are to submit at least one to the International Conference on Software Testing, Veri¯cation and Validation. Although the PI's opinion is that this project resulted in substantial and signi¯cant results, this project had several problems. Following up on the discussion during the meeting at Avaya on April 20, the PI has identi¯ed several sources for these problems, which are analyzed below. 1. Problem: Student did not understand bypass testing and the PI did not supply enough over- sight. Impact: Moderate Solution: Once the problem was identi¯ed by the contract monitor, the PI instituted im- mediate procedural and management changes. The GMU team worked very hard to catch up with the expected tests. This was successfully handled and resulted in a positive solution and closeout of the project. Although delayed, the results for task 8 are quite strong and have resulted in signi¯cant problems found in VIA software as well as an empirical paper, in progress. 2. Problem: The task list was modi¯ed at the beginning of the project, by adding a new major task{automating hand-generated, ad-hoc tests. Impact: Major Solution: The GMU team continually worked to adjust the schedule throughout the project. This led to the following problems: ² Confused expectations{It took several months to clarify whether the GMU part of the team was responsible for designing and implementing all VIA bypass tests, or developing processes with sample tests and teaching Avaya personnel how to develop the remaining tests. it was eventually clari¯ed that GMU was to develop the process and provide sample tests. ² It was impossible to complete tasks 5 and 6 once the student left Avaya. ² Because we were not able to complete tasks 5 and 6 in August, tasks 8, 10 and 11 took signi¯cantly more time than expected. 3. Problem: Unconventional use of Javascript in VIA web pages. Impact: Major Solution: The technique of packing client data into XML messages before transmitting to the server was not described or documented. Once the technological decisions were understood, the GMU team worked very hard to ¯nd a way to automate tests. This eventually led to some positive results. The problem was exacerbated when Avaya personnel were not available to 2 the student at the beginning of the project (when the student was on-site). This had a major impact on tasks 6 and 7 and made task 10 impossible to complete. 4. Problem: Communication di±culties between the PI and student. Impact: Major Solution: Weekly meetings, followed by written documentation of the discussion helped, but this problem was never resolved. 5. Problem: Missing schemas for XML-transmitted data. Impact: Moderate Solution: After being told there were no XML schemas, the GMU team generated schemas by studying the individual XML messages. Much later, Avaya personnel provided GMU with the original schemas. 6. Problem: O®utt's account on Avaya's system never worked. Impact: Minor Solution: This meant that all ¯les were handled by the student. 3 1 Summary of Goals and Plans This is the fourth and ¯nal report for the Avaya Inc. sponsored project on testing service oriented architectures (including web applications and web services), August 2006 through May 2007. The purpose of this project was to identify improved methods to test software built with service oriented architectures (SOA). Avaya Research Labs software technology's research department is currently building SOA software. Several speci¯c problems were identi¯ed in the Statement of Work (SOW): (1) How to save and reuse tests of the web and voice interactions with people, particularly in automating test scripts for regression testing; (2) How to validate correct behavior of web service components with regards to requirements; (3) How to detect failures when the incorrect behavior is only reflected in server-side data stores, such as databases and ¯les; (4) Is it possible for external users to subvert web service components and violate security protocols; and (5) How to ensure that the system will recover correctly after loss of connectivity or failure of one or more components. The GMU project is working on aspects of problems 2-5, as speci¯ed in the SOW, but not on problem 1. The original task list, with annotations, is in Appendix A. 2 Overview of VIA Technology VIA pages make heavy use of Javascript (JS). The VIA web pages use Javascript for three activities on the client's computer: 1. Validate inputs. 2. Respond to user events and dynamically modify the screen by manipulating the document object model (DOM) to add, remove, and modify input ¯elds and other selection elements. 3. Encode the input data into XML messages, which are then sent to the server. In most web applications, the user data from the client are sent as part of the HTTP request and accessed on the server by retrieving pairs of variables and values. The VIA system uses the novel approach of encoding input data into XML messages on the client, then submitting the XML messages. Javascript is used to encode the data into XML. JS is commonly used to validate inputs (1) and respond to user events. Most development organization discourage developers from using JS to make more than small, minor, changes to the DOM. When heavily modifying the DOM (2), it is very di±cult to analyze the e®ects, implement the JS correctly, test the JS, and avoid browser compatibility problems. Such web pages almost certainly cannot be made 508 accessibility compliant. Most importantly, this use of JS creates maintenance problems; the JS becomes so complicated that it is di±cult for anyone but the original developer to modify. JS is commonly used in this way to o®-load execution overhead from servers to clients. Clients are notoriously unreliable and inherently slow, thus this goal is seldom achieved. Ajax is a technol- ogy that would achieve almost the same e®ects without introducing these problems. Recommendation: It is strongly recommended that the JS in the VIA user interfaces be replaced with Ajax technology. The third use of JS, to encode the input data into XML messages, is original. Despite extensive research using textbooks, trade books, research articles, course materials, and discussions with numerous highly quali¯ed J2EE / JS developers, the PI was not able to ¯nd any examples of this type of use. In fact, every expert developer who considered this approach was extremely negative. Using JS to pack data into XML introduces extra execution burden on both client and server, 4 introduces many chances for errors, and introduces maintenance problems. Browsers and web servers already have very e±cient mechanisms for packing and transmitting data. If the data needs to be in XML on one of the server components, it is easier, more e±cient, and more reliable to store the data into XML on the server. Recommendation: It is strongly recommended that the JS code in the user interfaces that packs data into XML be replaced by server-side software.
Load more

Recommended publications
  • Testing Java EE 6 Applications: Tools and Techniques
    Testing Java EE 6 Applications: Tools and Techniques Reza Rahman Expert Group Member, Java EE 6 and EJB 3.1 Resin EJB 3.1 Lite Container Developer Author, EJB 3 in Action [email protected] Testing and Java EE Testing critical in enterprise development Pain-point that was not addressed well by J2EE Java EE 6 helps enormously by providing a number of enabling features geared towards testing Still somewhat a patchwork of evolving solutions Focus on developer (unit and integration) testing Focus on JUnit More about new tools rather than new techniques Testing Java EE Applications Testing Servlet 3 JUnit, HttpUnit, HtmlUnit great choices See if your container can be embedded into a unit test Cactus and Selenium can be good choices too Simulated Servlet containers like ServletUnit or mocking Servlet objects with EasyMock or Mockito options for very simple cases Servlet 3 Testing Demo! Testing JSF 2 JSF 2 project stages invaluable for debugging, test configuration, component development For simple cases, generic Servlet testing tools could be used, especially Selenium JSFUnit ideal for more complete JSF testing JSFUnit uses Cactus, JUnit, HttpUnit, HtmlUnit under the hood JSF 2 Testing Demo! Testing EJB 3.1/CDI CDI increases Java EE middle-tier testability by leaps and bounds EJB 3.1 embedded containers, generic dependency injection, @Alternative, portable extensions, XML deployment descriptors key enablers Arquillian/ShrinkWrap ideal cross-vendor CDI/EJB 3.1 testing tools Apache MyFaces CODI Testing another option Testing EJB 3.1/CDI Check to
    [Show full text]
  • End to End Testing Using Integrated Tools THESIS Presented in Partial
    End to end testing using integrated tools THESIS Presented in Partial Fulfillment of the Requirements for the Degree Master of Science in the Graduate School of The Ohio State University By Da Zhang Graduate Program in Computer Science and Engineering The Ohio State University 2012 Dissertation Committee: Rajiv Ramnath, Advisor Jay Ramananthan Thomas Bitterman Copyright by Da Zhang 2012 Abstract Automated functional testing for web applications is a well-researched area. Most organizations confront serious challenges in testing their websites, and these challenges are becoming more and more serious because the number of clients who rely on the web to perform e-commerce activity is increasing. Therefore, thorough, automatic, regressive and lean website testing technology is required to maintain website quality. In this paper, we describe an environment for testing with Selenium and Nagios, as well as customization we develop to incorporate Selenium script into a Nagios executable library. Nagios is an open source framework for monitoring network hosts, services and other hardware conditions with the purpose of failure detection [29]. Based on plug-in mechanisms, each service within the Nagios executable library can be executed as a Nagios plug-in. Selenium is a set of different open source software tools, each with a different approach to supporting web application test automation and agile process automated testing [1]. In this paper, we introduce in the how we combine the Nagios monitoring tool and Selenium testing tool to realize end-to-end testing using integrated tools. ii Dedication This document is dedicated to my family and my friends. iii Acknowledgments I sincerely thank my professors, Dr.
    [Show full text]
  • Testing Web Applications in Practice
    Testing web applications in practice Javier Jesús Gutiérrez, Maria José Escalona, Manuel Mejías, Jesús Torres Department of Computer Languages and Systems. University of Seville. {javierj, escalona, risoto, jtorres}@lsi.us.es ABSTRACT Software testing process is gaining importance at same time that size and complexity of software are growing. The specifics characteristics of web applications, like client-server architecture, heterogeneous languages and technologies or massive concurrent access, makes hard adapting classic software testing practices and strategies to web applications. This work exposes an overview of testing processes and techniques applied to web development. This work also describes a practical example testing a simple web application using only open-source tools. 1. INTRODUCTION Internet gives to developers a new and innovative way to build software. Internet also allows the access of millions of user to a web application [4]. Thus, problems in a web application can affect to millions of users, cause many costs to business [2] and destroy a commercial image. The Business Internet Group of San Francisco undertook a study to capture, assess and quantify the integrity of web applications on 41 specific web sites. According to the report, of those 41 sites, 28 sites contained web application failures [14]. Thus, software testing acquires a vital importance in web application development. First web applications had a simple design based on static HTML pages. Nowadays, web applications are much more complex and interactive with dynamic information and customized user interfaces. Design, support and test modern web applications have many challenges to developers and software engineers. This work is organized as follow.
    [Show full text]
  • Integration Tests
    Software Development Tools COMP220/COMP285 Sebastian Coope More on Automated Testing and Continuous Integration These slides are mainly based on “Java Tools for Extreme Programming” – R.Hightower & N.Lesiecki. Wiley, 2002 1 Automated Testing • Testing software continuously validates that - the software works and - meets the customer’s requirements • Automating the tests ensures that - testing will in fact be continuous • Without testing, a team is just guessing that its software meets those requirements. 2 Need for automation Humans make mistakes Humans sometimes don’t care! Manual testing is slow, hard to gather statistics Automated testing can be done at all hours Automated testing is fast, 10,000 tests/second Regression testing builds up with the project size 3 Tests and refactoring • Refactoring is changing existing code for simplicity, clarity and/or feature addition. - cannot be accomplished without tests. • Even the most stable or difficult-to-change projects require occasional modification. • That is where automated testing comes in. 4 Tests and refactoring • Comprehensive tests (running frequently) - verify how the system should work, - allow the underlying behaviour to change freely. • Any problems introduced during a change are - automatically caught by the tests. • With testing, programmers - refactor with confidence, - the code works, and - the tests prove it 5 Types of Automated Testing 1. Unit Testing • testing of a unit of a code - everything that could possibly break • usually exercises all the methods in public interface
    [Show full text]
  • Testing Web Applications
    Testing Web Applications Jeff Offutt http://www.cs.gmu.edu/~offutt/ SWE 642 Software Engineering for the World Wide Web Joint research with Ye Wu, Xiaochen Du, Wuzhi Xu, and Hong Huang Outline 1. Testing Web Applications is Different 2. HtmlUnit 3. User Session Data 4. Atomic Section Modeling 5. Testing Date State Management 6. Testing Web Services 4/27/2009 © Offutt 2 1 General Problem • Web applications are heterogeneous, dynamic and must satisfy very high quality attributes • Use of the Web is hindered by low quality Web sites and applications •Web applications need to be built better and tested more 4/27/2009 © Offutt 3 New Essential Problems of Web Software 1. Web site software is extremely loosely coupled – Coupled through the Internet – separated by space – Coupled to diverse hardware and software applications – Web services will dynamically couple with other services after deployment – without human intervention ! 2. Web software services offer dynamically changing flow of control – Web pages are created by software on user request – The interaction points (forms, buttons, etc.) vary depending on state: the user, previous choices, server-side data, even time of day – Examples : amazon.com, netflix.com, washingtonpost.com 4/27/2009 © Offutt 4 2 Extremely Loose Coupling • Tight Coupling : Dependencies among the methods are encoded in their logic – Changgyqggges in A may require changing logic in B • Loose Coupling : Dependencies among the methods are encoded in the structure and data flows – Changes in A may require changing data
    [Show full text]
  • (Microsoft Powerpoint
    PharmQuest’s Functional Testing Harness By Joe Glenny Thomas Importance of Functional Testing Enhanced software quality Unit testing is not enough We need to test the software as an actual user will use it Automated functional testing Speed Enables functional testing to be part of the build process PharmQuest Corporation Writing Test Code Approaches for writing test code Functionality driven Data driven PharmQuest Corporation Functionality-Driven Approach to testing Code/Script is written to execute every functionality in the software Various parameters are checked as the code executes This is a good approach if we need to test for parameters beyond the data displayed on the screen Large amount of code needs to be written if a screen has more than one data view The data to test for must also be encoded PharmQuest Corporation Data-Driven Approach to testing Contains a generic script to reach an end point The path to be taken and the final data to test is dictated by a specially formatted data file Incorporating a new view can be done by adding a data file This is a good approach if only the end data has to be verified Ease of use PharmQuest Corporation Available functional testing tools FIT SuiteRunner (www.artima.com) Canoo WebTest WinRunner PharmQuest Corporation Our experience using Canoo Webtest Canoo is a testing framework built on top of HttpUnit It uses scripts written in xml Data to be compared with is represented as xml The comparison is done using xpath PharmQuest Corporation Our experience using Canoo -
    [Show full text]
  • Agile Security Testing of Web-Based Systems Via Httpunit
    Agile Security Testing of Web-Based Systems via HTTPUnit A. Tappenden, P. Beatty, J. Miller A. Geras, M. Smith University of Alberta University of Calgary Edmonton, Alberta, Canada T6G 2E1 Calgary, Alberta, Canada T2N 1N4 {andrewt, beatty, jm}@ece.ualberta.ca {ageras, smithmr}@ucalgary.ca Abstract direct loss of profit, in the case of an attacker manipulating prices to their advantage, or an indirect loss The technological advancements of web-based systems of profit through a denial of service attack. In 2003 Dell’s and the shift to iterative and evolutionary development U.S. home & home office division generated $2.8 billion processes have given rise to the idea of agile security or approximately 50% of its revenues through its online testing, where the principles and practices of agile testing storefront [3]. The security of web-based systems is are applied to the domain of security testing. This paper critical for Dell operations, as a breach in the security explores common vulnerabilities for web applications and could literally lead to billions of dollars in lost profits. proposes two synergistic approaches for mitigating them. Security concerns are also of increased importance to The first approach is to employ a highly testable any software system that risks leaking private information architecture in the development of web-based systems, about individuals to the public or a malicious attacker. and the second is to support the security testing process These types of security threats may involve both large using the open source unit testing framework HTTPUnit. amounts of money, such as when a bank or other financial The overall testing strategy mingles well with agile institution is involved, and potential litigation, in the case development efforts and gives the development team an of the exposure of private information.
    [Show full text]
  • Launching Automated Security Attacks Through Unit-Level Penetration Testing
    Launching Automated Security Attacks through Unit-level Penetration Testing Michael Gegick and Laurie Williams Department of Computer Science, North Carolina State University Raleigh, NC 27695 1-919-513-4151 {mcgegick, lawilli3}@ncsu.edu Abstract 1. INTRODUCTION Penetration testing is a software security practice typically Penetration testing is the most commonly used software security conducted late in the software development lifecycle when a best practice [3]. However, penetration testing is most often system has been completed. A penetration test is an authorized considered a black box, late lifecycle activity [3]. A penetration attempt to violate specific constraints stated in the form of a test is an authorized attempt to violate specific constraints stated security or integrity policy. However, penetration testing should in the form of a security or integrity policy [4]. A penetration start earlier in the development lifecycle when corrective action test is designed to characterize the effectiveness of security is more affordable and feasible. To support earlier penetration mechanisms and controls to prevent attack by attempting testing, we are building SecureUnit, a framework of reusable, unauthorized misuse of, and access to, target assets [3, 4]. automated, and extendable Java-based unit test cases. We Penetration testing should start at the feature, component, or unit present the initial installment of SecureUnit comprised of a test level, prior to system integration [3] for more affordable and to detect cross-site scripting (XSS) vulnerabilities. We illustrate robust fortification. the effective use of the XSS test case via its use to identify vulnerabilities in WebGoat, an open source test bed. In the In this paper, we present initial steps in the building of SecureUnit, a testing framework, similar to JUnit1 and the other future, we will extend the test cases for the identification of a 2 wide variety of security vulnerabilities.
    [Show full text]
  • The Grinder 3
    The Grinder 3 Table of contents 1 Project........................................................................................................................... 5 1.1 The Grinder, a Java Load Testing Framework....................................................... 5 1.1.1 What is The Grinder?........................................................................................ 5 1.1.2 Authors...............................................................................................................6 1.1.3 Credits................................................................................................................ 6 1.2 The Grinder License................................................................................................6 1.2.1 The Grinder........................................................................................................6 1.2.2 HTTPClient........................................................................................................7 1.2.3 Jython.................................................................................................................7 1.2.4 jEdit Syntax........................................................................................................7 1.2.5 Apache XMLBeans............................................................................................7 1.2.6 PicoContainer.....................................................................................................8 1.2.7 ASM...................................................................................................................8
    [Show full text]
  • Unit Testing JSF Applications
    Testing JSF Applications Stan Silvert JBoss Core Developer Overall Presentation Goal Learn about approaches to teting JSF Applications with emphasis on JBoss JSFUnit and SeamTest. Pronouncement “JBoss JSFUnit is the first open source community dedicated to JSF testing.” How can we test JSF? Static Analysis Performance Testing Client-Centric Black Box Mock-Centric White Box Comprehensive “acrylic” Box Static Analysis Tests performed on an application without running it For JSF, this means testing the composite configuration Find problems early For JSF, only available in JSFUnit If you do no other testing, at least do this. It's free. JSFUnit Static Analysis Managed Proper Interface Beans/EL Implementations Managed Bean Missing methods getters/setters Serializable Duplicate scopes Managed Beans faces-config.xml JSF Tags typos Managed Bean Class Does it implement java.io.Serializable? <managed-bean> <managed-bean-name> shoppingCart </managed-bean-name> <managed-bean-class> com.foo.demo.ShoppingCart </managed-bean-class> <managed-bean-scope> session </managed-bean-scope> </managed-bean> junit.framework.AssertionFailedError: Managed bean 'shoppingCart' is in session scope, so it needs to implement interface java.io.Serializable Duplicate Managed Beans <managed-bean> <managed-bean- name>shoppingCart</managed-bean- name> <managed-bean- class>com.foo.ShoppingCart</managed- bean-class> <managed-bean- scope>application</managed-bean- scope> </managed-bean> <managed-bean> <managed-bean- name>shoppingCart</managed-bean- name> <managed-bean-
    [Show full text]
  • Httpunit - Are You Getting the Right Response?
    HttpUnit - Are You Getting the Right Response? Advanced Topics in Java Khalid Azim Mughal [email protected] http://www.ii.uib.no/~khalid/atij/ Version date: 2006-09-04 ATIJ 1/13 Overview • Installing HttpUnit • Functional Testing with HttpUnit • Creating Test Scenarios using HttpUnit • Example illustrating a Test Scenario ATIJ HttpUnit - Are You Getting the Right Response? 2/13 Installing HttpUnit • HttpUnit can be downloaded from: http://httpunit.sourceforge.net/ • Unpack the distribution zip file there you want to install HttpUnit. • An environment variable, called HTTPUNIT_HOME, with the following entry can facilitate using HttpUnit: HTTPUNIT_HOME = C:\httpunit-1.6 •The CLASSPATH environment variable should include the following entries in its definition: %HTTPUNIT_HOME%\lib\httpunit.jar;%HTTPUNIT_HOME%\jars\js.jar;%HTTPUNIT_HOME%\jar s\nekohtml.jar;%HTTPUNIT_HOME%\jars\servlet.jar;%HTTPUNIT_HOME%\jars\Tidy.jar;%H TTPUNIT_HOME%\jars\xercesImpl.jar;%HTTPUNIT_HOME%\jars\xmlParserAPIs.jar • Add the above libraries to the Eclipse project containing the test cases. ATIJ HttpUnit - Are You Getting the Right Response? 3/13 Testing a Web Site with HttpUnit • Functional tests are performed to test that a web site/application is providing the correct response to client requests. • HttpUnit is a framework, based on JUnit, that can be used for functional testing (aka block-box testing) of a web application. – It emulates the web-client interaction with a web application. – It is a HTTP client simulator that: • simulates a web browser's GET and POST requests, and maintains state • provides an object model of the response that can be verified by the test cases. • The entity body of the response is primarily HTML content.
    [Show full text]
  • Web Application Testing Challenges
    Web Application Testing Challenges Blaine Donley and Jeff Offutt Software Engineering George Mason University [email protected], off[email protected] September, 2009 Abstract A website is a static collection of HTML files that are linked together through tags on the World Wide Web. A web application, however, is an arbitrarily complex program that happens to be deployed on the World Wide Web. Web applications use new technologies that change on a regular basis, are almost always distributed, often employ concurrency, are inherently component-based, usually built with diverse languages, and use control and state behaviors that are not available to traditional software. This paper explores the technological-based differences between web software applications and traditional applications, with a specific emphasis on how these differences affect testing. Both applied and research issues are considered. 1 Introduction Since the inception of the Internet, the complexity of hypertext-based web applications has dramatically increased. These complexities are masked by a seemingly simplistic client-server model based on standardized languages and protocols. The language and protocol “standards” are interpreted in various ways by a myriad of diverse browsers, languages, and frameworks. These languages and frameworks provide web developers with toolsets to create highly complex functionality and stateful behavior. The client-server model and stateless transportation protocol used by Web applications create a disconnect that introduces additional complexities. Web applications have adopted unique control flow and state management techniques to get around the stateless nature of HTTP. The client-server model used by web application is quite different from other client-server models. A major difference is that the client interface is generated by the server at run-time.
    [Show full text]