Table of Contents
Getting Started ...... 3 Total Cost of Ownership ...... 5 License Size & Cost ...... 7 Buying Strategies ...... 8 Product Demonstration ...... 10 Feature Matrix ...... 11 About RadView...... 13 Getting Started
This paper will assist your evaluation process and VHOHFWLQJDORDGWHVWLQJVROXWLRQWKDWLVWKHULJKWȴW for your organization. How often do you Many tools and options are available today for run load testing? performing load testing. When coupled with the abundance of features and technologies that must be evaluated and compared, the selection process Start by evaluating the frequency of can be quite confusing. In addition to technological running performance tests. parameters, you must also consider issues like Will you be running tests once a quarter? technical support, cost of ownership, and the Monthly? Daily? skills of your testing team. This guide will help you If you run load tests often, then you’ll navigate through these issues in a methodical way. want to look at a professional, higher-end We begin by explaining how to get started with the type of solution. If you plan to load test evaluation process. Then, we continue to discuss infrequently, a “lightweight” or even open WHFKQLFDOIHDWXUHVWREHHYDOXDWHGDQGȴQDOO\ VRXUFHVROXWLRQPD\VXɝFH provide tips on how to negotiate your best deal. 7KHȴUVWVWHSLQFKRRVLQJDORDGWHVWLQJVROXWLRQ is evaluating your needs and resources. Here are some basic questions you should consider.
3 Getting Started
On-Premise, cloud Which technologies (SaaS) or hybrid? and protocols?
Is your organization comfortable (from a security Gather your list of requirements and then stand point) with a cloud solution, or do you need populate the spreadsheet matrix at the end of DQRQSUHPLVHWRRO"0DQ\VROXWLRQVRHUDK\EULG this document, to decide which features you approach where the load testing software runs on need and the exact frameworks and protocols servers within your network, but also allow you your development teams work with. This is to generate load via cloud based hardware (like an excellent way to evaluate and compare Amazon AWS, Rackspace, etc.). Some lightweight which vendor can best meet your needs. You cloud solutions are not capable of generating load can even add weight to the technologies or on premise. Identifying your needs will eliminate features so you can score solutions based on many potential load testing solutions from your list. your needs.
4 Total Cost of Ownership
Before we dive into license size and cost, let’s touch upon TCO – Total Cost of Ownership. Total Cost of Ownership in load testing involves a few factors – from support to powerful features that should be built-in. You can get a free, open source tool. But you and your team won’t have anyone to call, will be spending a lot of time reading and posting questions on sites OLNH6WDFN2YHUȵRZ JUHDWWRROȂEXWFDQ\RXDRUGWRUHO\VROHO\RQWKHSXEOLFGRPDLQ" 7KHUHDUHOLJKWZHLJKWWRROVZKLFKPD\RHUJRRGVXSSRUWEXWZLOOODFNFRUUHODWLRQDQGDGYDQFHGUHSRUWLQJΖI\RXUORDG testing tool doesn’t take you directly to the root cause and help you to identify bottlenecks, what good is it? Do you need an open source, lightweight or powerful load testing tool? Consider these factors:
Category Explanation
Support You should expect easy access to tier 2 professionals during business hours. Support should help you and your team onboard and be very knowledgeable when it comes to scripting and scenario creation. When you can’t rely on support, you will lose time and money on your side.
Scripting $FRPSOHWHVROXWLRQVKRXOGRHUDFRPELQDWLRQRIDQHDV\WRXVH*8ΖDQGQRQSURSULHWDU\ capabilities scripting language like JavaScript. The intuitive user interface will allow non-technical individuals to quickly create scripts. Without it, you will spend more money on training vs. GUI \RXUQRQWHFKQLFDOVWD6FULSWLQJZLOODOORZGHYHORSHUVDQGDGYDQFHGORDGWHVWHUVWR address more complex scenarios. Look for tools that use standard scripting languages that also match the expertise of your team.
5 Total Cost of Ownership Category Explanation
Automatic An automatic correlation engine should identify and replace dynamic values which are correlation unique for each run of the script such as session IDs, time, and others, and replace them automatically. Poor or no correlation is common in lightweight solutions. The money saved on a lightweight tool will be spent on labor-intensive script updates and manual correlation editing.
Integrations, A huge number of frameworks and protocols are available today to your development frameworks team. Consequently, you must ensure that your load testing solution supports the widest range of frameworks and protocols. Down the road, it’s very likely that your development and protocols WHDPZLOOEHXVLQJDGLHUHQWWHFKQRORJ\WKDQWRGD\$ORDGWHVWLQJWRROWKDWSURYLGHV supported EXLOWLQLQWHJUDWLRQZLWKWKHWHFKQRORJ\XVHGZLOOPDNH\RXUWHVWLQJPXFKPRUHHɝFLHQW than tailoring an integration from scratch. Don’t get locked into a long-term maintenance contract with a load testing solution that doesn’t support a multitude of technologies.
Server side The purpose of load testing is to evaluate how your system behaves under intense usage. monitoring $QHQWHUSULVHVFDOHORDGWHVWVROXWLRQZLOORHUDZLGHUDQJHRIVHUYHUSHUIRUPDQFH monitors and technologies to collect data during load testing. Without server side monitoring you are missing a key piece of load test information.
Comprehensive Reporting is often overlooked, but is often a weak area in many tools. However, extensive reporting reporting capabilities will save you time and money identifying problems and performance bottlenecks. Ideally, reporting should provide a rich set of out-of-the box reports, but DOVRWKHDELOLW\WRHDVLO\FXVWRPL]HUHSRUWVVR\RXFDQSUHVHQWGDWDLQDQ\VSHFLȴFDQJOH Another important capability is a web interface to enable collaboration and the monitoring of results from any location in real-time.
6 License Size & Cost
Licensing Options Determining the number of virtual users There are two purchasing models that The license cost of most load testing tools will be determined are common in load testing: by the number of concurrent virtual users you’ll be running, and to a lesser degree, the number of testers. Perpetual License It is therefore important you carefully consider the number of You purchase and own the software with “x” number virtual users to be purchased. of virtual users. As long as you pay for maintenance, To determine the number of virtual users you’ll need, it is best you receive support and upgrades. Even when you stop to closely examine the number of real users you expect to paying maintenance, you own the software and can use your system in peak times. For example, if you’re testing keep using it up to the number of virtual users that you DQHFRPPHUFHSODWIRUP\RXPD\ORRNDWWKHWUDɝFGXULQJ paid for. If you need more virtual users for one or more Christmas time (or any other peak time in your industry). months, you can “rent” them. The number of virtual users for your load testing should be Subscription License 30% above the expected number of users. This will ensure you You do not own the software, but are “leasing it” for a can stress your system beyond expectations as well as address SHULRGRIWLPH7KLQNRIKRZ0LFURVRIWLVQRZRHULQJ any future growth. 2ɝFH Once you’ve selected your tool, your next step is price negotiation. Here are 6 tips to help you save money and avoid unnecessary expenditures when procuring a load testing solution. 1 Larger buy upfront 2 Multi-year maintenance contract The larger the license of virtual user you buy, the more Another way to save money is to consider a multi-year bargaining power you have when negotiating price. Also PDLQWHQDQFHFRQWUDFW8VXDOO\VRIWZDUHFRPSDQLHVZLOOQRW think about whether your management team will be discount on a single year maintenance contract. However, willing to spend money again in one or two years from if you sign a multi-year commitment, you’ll often be able to now. Management is often reluctant to spend money get discounts on maintenance cost. twice in a short period of time, so your best bet might be 3 Avoid hidden costs to consider purchasing future needs now. For example, WKH GLHUHQFH EHWZHHQ D DQG YLUWXDO XVHUV Make sure the price you get includes all features and options might not be that large because the discount for 2000 you need. Watch for hidden costs when procuring a load virtual users will be higher. testing solution. 8 Buying Strategies 4 2HUDFDVHVWXG\ 6 Competitive upgrade Hi-tech companies are always looking for endorsements ΖI\RXȴQG\RXUVHOIXQKDSS\ZLWK\RXUFXUUHQWORDGWHVWLQJ of their solutions. You can sometimes negotiate an solution either because it lacks features, the quality of additional discount if you agree to provide a case study support is unsatisfactory, or perhaps due to the high cost of the maintenance - you can sometimes leverage these issues after you've installed and used the load testing software in a new purchase. Some load testing solution providers for a period of time. ZLOO RHU D FRPSHWLWLYH XSJUDGH WR FHUWDLQ FRPSHWLWRUȇV products. 5 Maintenance fee percentage Maintenance fees are typically calculated as a percent of WKHOLVWOLFHQVHIHHDQGUDQJHEHWZHHQDQGΖI your potential load testing solution provider is charging a higher rate, it may be negotiable, or you might want to look at an alternative solution. 9 Product Demonstration Once you have boiled down your load testing options to two or three choices, getting an online demonstration by each of the vendors is a good idea. START DEMO Of course, you should also try the tools yourself, but a one-hour online demo can provide you a thorough functional overview of the capabilities DQGVSHFLȴFIHDWXUHVΖWZLOODOORZ\RXWRDVVHVV more accurately the capabilities of each of the tools, beyond just a checkmark indicating the existence of a feature. 10 Feature Matrix WebLOAD WebLOAD WebLOAD Platforms Kerberos Weblogic Windows OS Basic Websphere 8VHUΖQWHUIDFH &RQWUROOHU Databases ColdFusion Execution Agent (Load Generator) JDBC (Java database connect) JBoss 8QL[/LQX[([HFXWLRQ$JHQW /RDG*HQHUDWRU ODBC (Open Database Connect) JOnAS Protocols & Technologies Oracle Oracle GlassFish Web 2.0 & Rich Internet Applications (RIA) Microsoft SQL Server Content Management Systems HTTP/HTTPs PostgreSQL WordPress +70/ MySQL Drupal Adobe Flex/AMF MDB Joomla XML/SOAP Applications Magento JSON SAP NetWeaver Wireless & Mobile Web Services SAP Web, Web Dynpro iOS WebSockets SAP BusinessObjects Android Rest API Infor Lawson Windows Phone AJAX (Asynchronous JavaScript and XML) Microsoft Dynamics Blackberry WebDAV (Web distributed authoring and versioning) Ellucian Banner/Luminis Multimedia (Voice & Streaming) Java over HTTP PeopleSoft RTMP Push Technologies (streaming, polling, long polling, PrimaVera (Oracle) async requests, reverse ajax, comet) RTSP/RTP Kaazing JDEdwards Performance Monitoring (Server-Side) - System Resource Oracle Applications – Oracle Forms LightStreamer Microsoft perfmon Oracle 2-tier Angular JS /LQX[8QL[66+ Oracle e-business suite React /LQX[8QL[UVWDWG Oracle Siebel Adobe Air Solaris Sage SPA (Single Page Application) IBM-AIX Kronos Network Protocols +38; )73 ȴOHWUDQVIHUSURWRFRO Salesforce Performance Monitoring (Server-Side) - Databases IMAP (Internet messaging) Oracle Hyperion Microsoft SQL Server SMTP (Simple mail transfer protocol) Sharepoint Oracle 323 3RVW2ɝFH3URWRFRO HR Access ODBC/JDBC JMS IBM Maximo PostgreSQL LDAP (Lightweight Directory Access Protocol) EMC Documentum MySQL 7&3Ζ38'3 Server Technologies Performance Monitoring (Server-Side) – Protocols DNS (domain name service) J2EE Telnet .Net PerfMon NNTP Node.js SNMP SSH (Secure shell) PHP SSH Authentication Apache JDBC SSL Tomcat RSTATD TLS IIS JMX NTLM NGINX 11 Feature Matrix WebLOAD WebLOAD WebLOAD Performance Monitoring (Server-Side) Interoperability/Extensibility Synchronization/Rendezvous Points – Application/Web Servers Java Think Time (Sleep) ANY via JMX COM Transactions ANY via SNMP General Features 8VHUGHȴQHG7LPHUV Adobe LiveCycle ES Protocol-Level Testing 7HVW&RQȴJXUDWLRQ 6FKHGXOLQJ Ariba (SAP procurement) Browser-Level (Emulation) via Integrations Goal-Oriented Load Testing ATG Dynamo (Oracle eCommerce) Distributed Architecture /RDG6FKHGXOLQJ3URȴOHV Broadvision (eCommerce) Page Elements/DOM Access Constant ColdFusion (Adobe) JSON Object Access/Parser Linear Fujitsu INTERSTAGE XML Object Access/Parser Random GlassFish AS (Oracle) Java Object Access Intervals Ζ%0:HE6SKHUH Real Page/Browser View Steps iPlanet (NAS) Full HTTP Headers View Ramp 8S JBoss (RedHat) HTML (Response) View 8VHUGHȴQHG JOnAS Response Validation Test Execution Microsoft Active Server Pages Cookies Support Real-time server-side monitoring Microsoft ASP.NET Cache Support Real Time View of ALL Performance Measurements SAP NetWeaver 0XOWLSOHΖ3$GGUHVV Ζ36SRRȴQJ HTTP statuses reporting and logging SilverStream (Oracle Application Server) Network Emulation Integrated Reporting Manager WebLogic Recording Features MIX execution Apache Browser-based recording Cloud load generation Microsoft IIS Native Mobile Recording Probing Client SunOne (Sun + NetScape) Insert Transactions during recording Throttle control Microsoft Windows Media Server Developing/Debugging Environment Freezing tests during execution Real Networks RealServer Access to the script 6HQG1RWLȴFDWLRQRQGHȴQHGWULJJHUV Integration & Extensibility Quick scripting building blocks Test Results Analysis & Reporting Command Line Interface Intellisense/Auto-completion editor Web Dashboard Open Architecture (Public SDK/API) Context-sensitive Help/Tooltips Reporting Engine and Analytics 3rd Party Tools Integration/Plug-in Debugging Dynamic and easily customizable reports AppDynamics Breakpoints 3UHGHȴQHG5HSRUWLQJ7HPSODWHV Dynatrace Watch (Variables Value) &XVWRPXVHUGHȴQHG5HSRUWV New Relic Step-by-Step Reports Exporting and Publishing Nagios Scripting Language Comparison and Regression Analysis PerfectoMobile JavaScript Support & Maintenance Jenkins Java Online (customer portal) Selenium – browser Emulation Automatic Data Correlation of dynamic values ([SHUWOHYHO WLHU VXSSRUWRQDȴUVWFDOO Ranorex – browser Emulation Manual Data Correlation of dynamic values Phone Original Software TestDrive – browser Emulation Parameterization Email Git )URPȴOHV6WULQJV1XPEHUV'DWHV 'RFXPHQWDWLRQWXWRULDOVKHOSȴOHV%HVW3UDFWLFHV 5DQGRP6HTXHQWLDO8QLTXH Trainings 12 About RadView RadView is a pioneer of web and mobile load testing VROXWLRQV6LQFH WKHFRPSDQ\KDVEHHQRHULQJ WebLOAD, an enterprise-grade software for large- scale load testing, helping companies launch internet DSSOLFDWLRQVZLWKFRQȴGHQFH'HSOR\HGDWOHDGLQJ RUJDQL]DWLRQV:HE/2$'RHUVDEHVWYDOXHORDGDQG performance testing solution. For more information, visit www.radview.com RUFDOO5$'9Ζ(: North American Headquarters +LJKZD\:HVW6XLWH %ULGJHZDWHU1- www.radview.com Email: [email protected]