Critical Analogy of Automated Testing Tools

www.ijemr.net ISSN (ONLINE): 2250-0758, ISSN (PRINT): 2394-6962

Volume-6, Issue-2, March-April 2016 International Journal of Engineering and Management Research Page Number: 187-192

Shivani Varshney1, Himani Gupta2 1, 2Research Scholar, Department of Computer Science, Galgotias University, Greater Noida, U.P., INDIA

ABSTRACT correct behavior, plan that leads them through a set of Software testing aims at evaluating an attribute or important test cases. A test case in software testing is a set capability of a program or system and determining that it of conditions written for particular applications & tester meets its required results. Testing automation tools enables run all these conditions to ensure the correct functionality developers and testers to easily automate the entire process of of the software applications. Manual testing is a laborious testing in software development. It is to audit & customize activity that requires the tester to possess a certain set of source code. Performing testing in an effective manner produces arete software. With agile practices getting more qualities; to be patient, observant, speculative, creative, popular Test Automation is an important factor in achieving innovative, open-minded, resourceful, opinionated, and continuous delivery. There are so many tools available in skillful. Manual testing helps discover defects related to market for test automation. It includes both Open Source and the usability testing and GUI testing area. Any new Proprietary Tools/Framework. It is a very challenging task to application must be manually tested before its testing can pick up best tool for automation testing. The objective of the be automated. Manual testing requires more effort, but is paper is to conduct a comparative study of automated testing necessary to check automation feasibility The problems tools such as Selenium free source, HP Quick test professional with manual testing are, it is very time consuming process, (QTP) , Telerik Test Studio , Watir and Sahi. This work not reusable, has no scripting facility, great effort required, computes and compares these automated software testing tools to determine their application and effectiveness. and some errors remain uncovered . Automation testing covers all the problems of Keywords--- Selenium free source, HP Quick test manual testing. It increases the test execution speed, more professional (QTP), Telerik , Watir and Sahi. reliable, repeatable, programmable, comprehensive. [2]Automation Testing is running test cases where manual intervention is not required to run each one. It uses special I. INTRODUCTION software to write & execute test cases to compare the actual outcome with the predicted outcome. Aim of present

paper is to perceive the foremost automated tool.[3] “Software testing is a process used to identify the The following section highlights a brief review of correctness, completeness and quality of developed automated testing tool. The remaining sections are computer software”. It is the process of exercising and organized as follows. Section 3.0 gives an insight to the evaluating a system or system components by manual evolution of Automated Testing Tools and its components. automatic means to verify that it satisfies specified Section 4.0 gives a description of HP QTP, Selenium, requirements or to identify differences between expected Sahi , Telerik Test Studio and Watir. A critical analogy of and actual results. The fundamental goal is to analyze the these automated tools is presented in Section 5.0. The features supported by various functional testing tools that results are depicted in Section 6.0 along with the scope for aid in minimizing the resources[1]. future work. and Section 7.0 presents the conclusion. There are two ways of testing that are manual or automation. Manual testing carried out by the testers. Manual II. TESTING TOOLS Testing is a process where a tester often follows a written test plan that leads them through a set of important test In software testing, test automation is the use of cases It requires a tester to play the role of an end user, and special software (separate from the software being tested) use most of all features of the application to ensure its to control the execution of tests and the comparison of 187 Copyright © 2016. Vandana Publications. All Rights Reserved. www.ijemr.net ISSN (ONLINE): 2250-0758, ISSN (PRINT): 2394-6962 actual outcomes and predicted outcomes[4]Here the classification of software testing tools has been depicted.[5]

Manual Testing Automation Testing

Acceptance Selenium testing Black Box QTP testing Whitebox Watir testing Unit testing Sahi

System Telerik Test testing studio

Fig. I showing classification of testing tools

MANUAL TESTING AUTOMATION TESTING If you have to run a set of tests repeatedly automation is If the Test cases have to be run a small number of times a huge gain. its most likely to perform manual testing It allows the tester to perform more ad-hoc(random Helping performing “compatibility testing”-testing the testing).. software on different configuration. Short term cost are reduced It gives you the ability to run automation scenarios to perform regressions in a short time The more time tester spends testing a module the greater It gives you ability to run regressions on a code that is the odds to find real use bugs.. continuously changing. Simultaneously testing on different machine with Automation testing can be carried out simulataneously different OS platform combination is not possible using on different machine with different OS platform manual testing. combination Executing the Build Verification Testing(BVT) is very Automation testing is very useful for automating the mundane and tiresome in Manual Testing. Build Verification Testing(BVT). Table I Showing the difference between Manual Testing and Automated Testing.

III. HP QUICK TEST PROFESSIONAL the test procedure and to manipulate the objects and controls of the application under test. Although HP QuickTest Professional is the solution for QuickTest Professional is usually used for "UI Based" Test functional test and regression test automation. In other Case Automation, it can also automate some "Non-UI" words, QuickTest Professional is a tool used for based Test Cases such as file system operations, database automation of functional and regression tests for various testing or Web services testing. HP QuickTest Professional software applications and environments. It performs 11 is current and latest available stable release from HP functional and regression testing through a user interface Software Division.[7] such as a native GUI or web interface.[6] QuickTest SELENIUM: Professional uses VBScript scripting language to specify 188 Copyright © 2016. Vandana Publications. All Rights Reserved. www.ijemr.net ISSN (ONLINE): 2250-0758, ISSN (PRINT): 2394-6962

Selenium provides a rich set of commands for SAHI: fully testing your web-app in virtually any way you can Sahi is an open source testing tool for web imagine. The command set is often called selenese. These applications, with the facility to record and playback commands essentially create a testing language.[8] scripts. Developed in Java and Javascript, this tool uses In selenese, one can test the existence of UI simple Javascript to execute events in the browser. elements based on their HTML tags, test for specific Sahi runs as a proxy server which intercepts content, test for broken links, input fields, selection list traffic from the web browser and records the web browsing options, submitting forms, and table data among other actions. Sahi can play back those recorded actions by things. In addition Selenium commands support testing of injecting Javascript into the browser so it can access window size, mouse position, alerts, Ajax functionality, elements in the web page. This makes the tool independent pop up windows, event handling, and many other web- of the website/ web application.[11] application features. The Command Reference lists all the TELERIK TEST STUDIO: available commands.[9] Telerik Test Studio is a automated software A command is what tells Selenium what to do. Selenium testing tool that offers a Visual Studio plug-in and a commands come in three “flavors”: standalone application to help you build functional web Actions, Accessors and Assertions.Actions are and desktop tests, customize them in code within Visual commands that generally manipulate the state of the Studio, or convert them to unit tests. [9] The tool provides application. They do things like “click this link” and industry-best test automation support for any application - “select that option”. If an Action fails, or has an error, the Silverlight, WPF, AJAX, HTML5, and MVC included, as execution of the current test is stopped.Many Actions can well as support for Visual Studio 2010. What’s more, Test be called with the “And Wait” suffix, e.g. “click And Studio is the best way to test RadControls for ASP.NET Wait”. This suffix tells Selenium that the action will cause AJAX, Silverlight and WPF. The test recorder the browser to make a call to the server, and that Selenium automatically detects the controls and exposes commonly should wait for a new page to load. Accessors examine the used verifications. Use Test Studio standalone app for state of the application and store the results in variables, performance and load testing too.[12] e.g. “storeTitle”. They are also used to automatically generate Assertions. Assertions are like Accessors, but IV. EVOLUTION AND CRITICAL they verify that the state of the application conforms to ANALOGY OF AUTOMATED TESTING what is expected. Examples include “make sure the page title is X” and “verify that this checkbox is checked”.[10] TOOLS WATIR: Watir, is an open-source (BSD) family of Ruby Evolution of testing tools evolve from 1980 libraries for automating web browsers. It allows you to However the evolution of the Automated testing tool write tests that. It is simple and flexible. Watir drives evolve from 1999 to 2013. Today, assuring the quality of browsers the same way people do. It clicks links, fills in any software has become very challenging. The size and forms, press buttons. [10] .Watir also checks results, such complexity of applications have increased exponentially, as whether expected text appears on the page. Watir has a while end-users are becoming more and more demanding. multi browser (& OS) support. Watir has a rich Under these circumstances, verification and validation API(Application Programming Interface).It is very activities tend to lose their momentum and make convenient as the user do not need to have supplementary compromises this lead to the evolution of Automated knowledge of programming. .Watir-classic makes use of Testing Tools. the fact that Ruby has built in Object Linking and With every second new technology is changing Embedding (OLE) capabilities. As such it is possible to and new technology is overcoming the problems that were drive Internet Explorer programmatically. Watir-classic present in the old technology. operates differently than HTTP based test tools, which There are so many test automation tools (both operate by simulating a browser. [10] open source and commercial) available in the market and it Watir-webdriver is a modern version of the Watir is extremely important to choose the right tool that best API based on Selenium. Selenium 2.0 (selenium- suits your overall requirements. But it is always wise to set webdriver) aims to be the reference implementation of a bench mark by comparing HP Quick Test Pro, Selenium, the Web Driver specification. In Ruby, Jari Bakken has Sahi, Telerik Test Studio and Watir. The Goals of implemented the Watir API as a wrapper around the Automation testing can be Selenium 2.0 API. Not only is Watir-webdriver derived from Selenium 2.0, it is also built from the 1.Speed up Test Execution HTML specification, so Watir-webdriver should always be 2.Achieved Improved Test Coverage compatible with existing W3C specifications. 3.Ensures Consistency, Accuracy and Efficiency 4.Reliability and Transparency 5.Return on Investment (ROI) 189 Copyright © 2016. Vandana Publications. All Rights Reserved. www.ijemr.net ISSN (ONLINE): 2250-0758, ISSN (PRINT): 2394-6962

6.Quality Repeatability and Re usability

S NO. YEAR TESTING SPECIFICATION TOOL 1 1999 Software test Book by Mark Fewster and Dorothy Graham automation 2 1999 First version Early 1999mercury interactive release Quick test as an easier approach to creating and of quick test maintaining SAP automated test.It later become Quick test Professional

3 1999 Test Frame The Test methodology Test Frame is published by Hans Buwaldo ,Dennis Jansen and published Iris Pinksterworking for British/Dutch IT company Logica CMG.Test Frame focuses on the uses of Test Automation

4 2000 Agile A conference in Utah by 17 representatives from different development methodologies Manifesto is leads to the Agile Manifesto in which the Twelve Principles of Agile Software are published published. 5 2002 JIRA 1.0 The first version of the popular bug tracking tool JIRA is released by the Australian released software company Atlassian Software. 6 2003 Agile Testing On his weblog Brian Marick publishes the Agile Testing Quadrant in August 2003. The Quadrant quadrant combines two distinctions in testing; the distiction between business facing or technology facing testing and the distinction between support programming or critique the product. 7 2004 Selenium The popular web application test tool Selenium is developed at Thought Works developed (Chicago) by Jason Huggins. 8 2005 Sahi It is released under an Apache License 2.0 Open Source License 9 2012 Telerik Test Telerik Test Studio is a Windows-based software testing tool for web and Studio desktop functional testing, software performance testing, load testing and mobile application testing developed by Telerik.. 10 2013 Watir Watir (Web Application Testing in Ruby, pronounced water), is an open-source (BSD) family of Ruby libraries for automating web browser. Watir was primarily developed by Bret Pettichord and Paul Rogers Table II Evolution of Automated Testing Tools

Success in any Test Automation (TA) effort lies Test Professional, Selenium,Sahi, Watir and Telerik Test in identifying the right tool for automation. There are so Studio. The comparison that is provided in this table is many test automation tools (both open source and based on Licensing Cost, Script Creation time, Object commercial) available in the market and it is extremely oriented language support and scalability, support for important to choose the right tool that best suits your operating system/platform, database framework, database overall requirements. But it is always wise to set a bench application, report generation mobile support,client and mark by comparing HP Quick Test Pro, Selenium , Sahi , drawbacks. On the above mentioned features we have Telerik Test Studio and Watir. provided the analogy of this work. Our work has involved the analogy of the automated testing tools .This paper has compared Quick

CRITICAL ANALOGY OF AUTOMATED TESTING TOOL

FEATURE HP QTP SELENIUM SAHI TELERIK TEST WATIR S STUDIO Licensing QTP License cost varies It is open Open source The cost of telerik The pricing of Cost from country to country, source .No software. No studio varies from sahi in India is Rs depending upon the licensing cost licensing cost is version to version Test 22,275 per user license type it is between is required required. Studio Functional cost and the cost of $8000 - $12000. The best . is $1,799. Test Studio sahi outside India place to inquire would be load cost is $1,799 and is $495 per HP support. the cost of Test Studio user.BSD license Ultimate is $2,999. required

Script Less High Less Less Less Creation Time Object Scripts can be Supports Sahi C#,ASP.net,VB.net Ruby Oriented Developed only in Java, .Net, script,Java,Ruby Language VBScript or JavaScript Perl, sahi script support PHP, and Python and Scalability Ruby Support for QTP supports Supports Operating system Windows Server 2003 Operating System operating Windows XP Windows and browser (users can run load cross platform system/plat PC/MAC/UN independent tests against Windows forms IX Platforms Server 2003 but not on it); Windows Server 2008; Windows Server 2012; Windows Vista; Windows XP (minimum for load testing is XP SP3), Windows 7 Framework Easily integrated with HP Selenium + It easily NET 4.0 / .NET 4.5 as RSpec Quality Eclipse + automates of version 2013.1.806 / ,Cucumber,Test Center or HP ALM Maven / ANT applications built .NET 3.5.1 SP1 for by unit (separate + Jenkins / using ExtJS, ZK, Visual Studio plug-in commercial products) Hudson & its Dojo, YUI or any plugins / other framework. Cruise Control + TestNG + SVN Database It supports relational Access Data Sahi supports Microsoft SQL Server Ruby libraries for application databases, so anything base using database 2008, 2008 R2, and database support s that uses standard SQL is JDBC. connectivity, 2012. Express, and it does not supported – like where script can Developer, work with MS SQLServer, Oracle, connect to Professional, and SQL,MY SQL MySQL, Access, database during Enterprise versions are runtime all supported. to execute any queries. Report Quality Center has in- Report is Impressive Very rich image based Impressive Generation built awesome supported reports and logs reporting reports dashboards through test 191 Copyright © 2016. Vandana Publications. All Rights Reserved. www.ijemr.net ISSN (ONLINE): 2250-0758, ISSN (PRINT): 2394-6962

runner and various logs. Clients Hewelette Packard Blackberry10 Fiserv,HCL,Tesc Sony,IBM,Toshiba, Facebook,SAPYa ,ios o, Mphasis Nestle ,Microsoft hoo ,Bell driver,windo ws phone Mobile Different commercial Android , ipad Supports mobile Supports ios and Support product i.e. iPhone & devices android mobile HP UFT Mobile (formerly iPad , known as Mobile Cloud Blackberry for QTP). ,Headless WebKit Drawbacks Cost is high – License and Identifying Only used for Some features of Doesnot support maintenance locators that Web Testing for Telerik Just Code are browser plugin support now. Limited use, available only in like Java common thus not many particular languages applets,adobe attributes like people know flash and id, about it. microsoftsilverlig names,XPAT ht H,javascript DOM etc Table III Critical Analogy of Automated testing tools

V. CONCLUSION AND FUTURE [3] Mohd. Ehmer Khan, “Different Forms of Software WORK Testing Techniques for Finding Errors,”IJCSI International Journal of Computer Science Issues, Vol. 7, Testing automation tools enables developers and Issue 3, No 1, May 2010. testers to easily automate the entire process of testing in [4] Jovanovich and Irena, “Software Testing Methods software development. Automated Software testing is the and Techniques,” May 26, 2008. best way to increase the effectiveness, efficiency and [5] Prof VN Maurya,Er. Rajendra Kumar “Analytical coverage of software testing .Through this research paper study on Manual Testing vs Automated Testing using with a comparative and concise study of the evolution of simplistic cost model”International journal of Electronics automated testing tools has been given. We discussed and Electrical Engineering Volume 2 Issue 1 Jan 2012 various tools such as QTP, Selenium, Sahi, Telerik Test [6] Manjit Kaur ,Raj Kumari“Comparative study of Studio and Watir .One can use the tool according to their Automation Testing tools Test Complete and Quick Test need and budget. All tools have their own advantages and pro” IJCA volume-24, no 1,June 2011 disadvantages. Analyzing the different tools makes us [7] Richa Rattan “Comparative study of Automation clear that the best tool among these tools is Telerik Test Testing Tools Quick Test Pro and Selenium”International Studio. It is best for performing the Automation Testing. Journal of Computer Science and Information While other tools QTP, Selenium ,Sahi,and Watir have Technology”- Vol 3 No june 2013 more snag as compared to Telerik Test Studio .In the [8] Nidhi Uppal,Vinay Chopra“Design and Telerik Test Studio more advancement is required as per Implementation in Selenium IDE with Web IDE” the demand of testing. Test Studio provides better value if International Journal of Computer Applications Vol 46- we do cost benefit analysis. No-16 May 2013 [9] Prasanth Yalla,Dr.L.S. Reddy,M.Srinivas,T.Subha Mastan Rao“Framework for Testing web application using REFERENCES Selenium testing tool with respect to integration testing” IJCST Vol. 2,Issue 3,September 2011. [1] ”Innovative approaches of automated tools in software [10] Nishi Gogna“Study of browser based Automated Test testing and Innovative approaches of automated tools in Tools WATIR and Selenium” International Journal of software testing and current technology as compared to Information and Education technology Vol. 14,No 4,Aug manual testing “-jan 2009 2014 [2] Vishawjyoti, Sachin Sharma “Study and analysis of [11] Sahi entry in Wikipedia: Http: //en.wikipedia.org automation testing techniques” JGRCS Journal of Global [12] Telerik Test Studio entry in Wikipedia: Http: Research in Computer Science Volume 3, No. 12, //en.wikipedia.org. December 2012 192 Copyright © 2016. Vandana Publications. All Rights Reserved.