TestRig: A Platform independent system testing tool. Candidate: Kaul Vaibhav Advisor: Shasha Dennis Abstract The goal of the TestRig software is to give a test engineer a fixed interface to help him with system/integration testing of software systems. TestRig is platform independent and can be utilized to test software systems coded with any programming language. In addition to doing that, it provides templates and examples of using various Open Source testing tools to help a user design their test cases. TestRig has been designed keeping in mind the current scenario in software development where complex systems are often created using multiple programming languages across different platforms. The challenge is to have a defined set of rules that are able to test any such system. The software makes use of various open source testing tools to run tests and verify results, which enables a user to test a system at different levels such as Performance Testing, Blackbox Testing, and User Acceptance Testing. TestRig is open source and utilizes a programmer’s creativity to test across multiple scenarios. The thesis will show how different software systems have been tested using TestRig. [NYU COURANT INTITUTE OF MATHEMATICS] [TestRig] [A Platform Independent System Testing Tool] Vaibhav Kaul N11167470 [Fall 2010] A thesis submitted in partial fulfillment Of the requirements for the degree of Master of Science in the Department of Computer Science, New York University Approved: ________________________________ (Dennis Shasha, Research Advisor) Approved: ________________________________ (Lakshminarayanan Subramanian, 2nd Reader) 1 Acknowledgements I would like to thank Professor Dennis Shasha for his continued support and guidance. I am also grateful to Arthur Meacham, PhD candidate in CS for his help with understanding the Outsafe Project and for his inputs on testing it. A special thanks to Professor Lakshminarayanan Subramanian for taking the time to read the thesis text and for his feedback. Research and Development TestRig is a platform independent System testing tool developed over Fall 2010. It started out as an application to test the Outsafe application in a cloud based platform but has been extended to incorporate other platforms as well. 2 Contents Introduction .......................................................................................................................... 5 Background ........................................................................................................................... 6 The Software Development Lifecycle ................................................................................................................................. 6 Traditional Waterfall Model....................................................................................................................................................... 6 Iterative Development .................................................................................................................................................................. 7 Software Testing ........................................................................................................................................................................ 8 Approach ............................................................................................................................................................................................ 8 Black Box Testing ................................................................................................................................................................................. 8 White Box Testing ................................................................................................................................................................................ 8 Types of Testing ............................................................................................................................................................................... 8 Functional Testing ............................................................................................................................................................................... 8 Non-Functional Testing ..................................................................................................................................................................... 9 Automation Testing ....................................................................................................................................................................... 9 Test Driven Development (TDD) ................................................................................................................................................. 10 Automation Testing Tools [2] ....................................................................................................................................................... 11 Automation Testing Frameworks ............................................................................................................................................... 12 TestRig ................................................................................................................................ 13 Goal .............................................................................................................................................................................................. 13 State of the Art ......................................................................................................................................................................... 14 The TestRig Operation Contract ............................................................................................................................................ 14 What sets it apart .................................................................................................................................................................... 16 Examples ............................................................................................................................. 18 Test whether a number is prime or not ........................................................................................................................... 18 Context ............................................................................................................................................................................................. 18 Tools Used ....................................................................................................................................................................................... 18 Operation Contract Implementation ................................................................................................................................... 19 JUnit Test Case ......................................................................................................................................................................... 19 Context ............................................................................................................................................................................................. 19 Tools Used ....................................................................................................................................................................................... 20 Operation Contract Implementation ................................................................................................................................... 20 UI Testing in Windows (C#) .................................................................................................................................................. 21 Context ............................................................................................................................................................................................. 21 Tools Used ....................................................................................................................................................................................... 21 Selenium ................................................................................................................................................................................................. 21 How it Works .................................................................................................................................................................................. 21 Sending the Facebook Message .................................................................................................................................................... 26 Performance Test For WCF Web Service ......................................................................................................................... 36 Context ............................................................................................................................................................................................. 36 Tools Used ....................................................................................................................................................................................... 36 3 Microsoft WCF [6] .............................................................................................................................................................................