CASE STUDY Vistaprint – how a global marketing services giant created a unified automated testing platform with Telerik

With three different QE automation platforms being used by their OVERVIEW 100-strong worldwide testing team, Vistaprint needed to create a strategic solution that would support their future development REGIONAL Global needs. Here Clay Bogusky, QE Technology Manager and Harish Narayan, Director, Technology at Vistaprint, tell how they tackled INDUSTRY the mammoth task and brought it to a successful conclusion – with E-commerce, online printing, micro business marketing a little help from Telerik.

CUSTOMER PROFILE Vistaprint (NASDAQ: VPRT) provides affordable, professional print and online marketing Test Unified Quality Other tools for small businesses in Management Automation Reporting Quality 130 countries worldwide. System Platform Tools (TMS) (UAP) Engineers SA CHALLENGE Inspired by the vision of a Quality Platform to support business Processes activities across the organization, (Risk Based methodology, Project Execution, Best Practices) Quality DB Vistaprint set out to replace its three existing QE automation platforms with a single, unified QE Technology Taxonomy automation platform.

The Vistaprint Quality Platform SOLUTION Vistaprint chose to leverage the Telerik Testing Framework, part of the family, a framework for writing automated tests for HTML5, AJAX, Silverlight and WPF applications, ahead of automation The Telerik Testing Framework has been an invaluable resource frameworks from Microsoft and Selenium for us in creating automated tests for Vistaprint.”

BENEFITS Clay Bogusky, QA Technology Manager at Vistaprint • Support for major browsers as well as WPF and AJAX applications • Professional assistance delivered through the Telerik support package BACKGROUND To give us a sense of scale, how •• A few QE engineers inside various many tests and how many people QE teams that develop automated Why did you need to undertake are involved? tests as needed. this project? Automated tests •• 5 software engineers including a A couple of years ago, Vistaprint had solution architect developing the three separate and independently •• 6,000 on the legacy platforms. platform and tools for automation. operating automation platforms that •• A few thousand and counting on were used to validate our solutions. How long did the project take? the new unified platform. Two of the three platforms were The development of the unified used for the majority of the testing. •• Each test can have many platform started in April 2011, its first These two platforms used the configurations and parameters, version was deployed by April 2012 WebAii/Telerik framework. Each of leading to 3 to 5x more actual and migration of automated test these platforms had advantages and test runs. scripts began shortly thereafter. disadvantages but none could scale Staff involved for the long-term without extensive increase in maintenance cost. In total, •• Vistaprint has hundreds of there were over a million lines of developers worldwide. code and over 6,000 automation test •• Approximately 100 of them are programs across all three platforms. members of the Vistaprint QE An initiative began to replace these department. three systems with one unified •• 5 “core” automation test script automation platform. This was part developers on one team. of a wider vision to create a quality platform to support business activities throughout the organization. THE CHALLENGE At the heart of the platform are the common frameworks that are We chose Telerik for Building a new platform on this used to automate the applications its extensive technology scale is a daunting prospect. Can under test. We have defined a support, including various you give us a high-level overview single UI automation interface that of the technology challenge? browsers, WPF, AJAX and is used to test console as well as JavaScript. Our browser Our automation platform consists of browser based user applications. This and windows automation several modules and subsystems or interface is used to test all Vistaprint layers extensively frameworks to ensure flexibility and developed applications. scalability. The main modules in the leveraged the Telerik Test Each automation script uses the platform are: Framework, enabling us common framework to test a to deliver our automation •• Automation script “runner” Vistaprint application. At this point, platform on-time with we have completely migrated off of •• Scheduler services and UI the legacy platforms. multi-technology support”. •• Automation dispatcher What technologies have you used? Harish Narayan, Director, •• Automation triage services and UI We are principally a Microsoft shop Technology at Vistaprint •• Admin services and UI (Windows OS) and use the .NET stack. We code in C# and use the •• Management and monitoring Powershell scripting language for The main subsystems are: some administrative tasks. Our web site is developed in ASP.NET and MVC •• Common framework technologies for generating orders. •• Test framework The backend systems are coded in WinForms and WPF. •• Business-layer frameworks As a result of acquisitions and •• Scheduling framework expansion, open source technologies •• Triage framework are being used but automation projects on those applications have •• Admin not yet started. •• Management framework

The platform uses HP’s ALM product as the test management system CHOOSING TELERIK for designing and executing tests manually or by using automation. How did you decide to use Test We have used Telerik’s RadControls Studio and the Telerik Testing for ASP controls for increased Framework? What were your capabilities, performance, and criteria? reduced development time. Presently, We are using the Telerik Testing we have over 100K lines of Vistaprint Framework because it fits our needs code in the platform excluding test to develop automated tests for the scripts. This excludes the Telerik .NET technology. It was the only Testing Framework. framework that claimed to support How did you adjust your project’s the most widely used browsers (IE, timeline or scope to include time FF, Chrome, and Safari), as well as for automation? Were your initial WPF and AJAX applications. Telerik estimates close? also offers a support contract Our initial estimate to develop which has been helpful in resolving the new unified platform was bugs and answering questions in a approximately one year with 4.5 timely manner. engineers and another year to rewrite What other tools and/or automation and retire the legacy frameworks did you evaluate? systems using 5 engineers.

We reviewed the Microsoft and Our estimates were fairly accurate. Selenium automation frameworks. The most difficult part was finding TELERIK’S DEEP EXPERTISE IN The Microsoft framework was limited qualified people for the project since TESTING to the products Microsoft sells (ie it was a unique venture. We have WPF). When we evaluated Selenium retired all of the legacy systems. its features were largely limited to Telerik Testing Framework has its How did you set expectations roots in the respected WebAii testing browser based automation. framework, dating from 2006. WebAii with your team and was created to meet the growing leadership regarding the work need for testing to be able to handle for automation? AJAX and Rich Internet Applications. Involvement with Telerik began with GETTING IT DONE native support for Telerik’s UI controls Fortunately, we set expectations in 2009. When the team joined Telerik in What impediments did you appropriately during the funding 2010, WebUI Test Studio was released, identify ahead of time? cycle of the project. After the project evolving into today’s Test Studio was approved, we set quarterly with the introduction of support for Since our staff was very WPF applications, as well as load and milestones with clear interim knowledgeable with the WebAii performance testing. deliverables, and provided frequent project and had already automated updates on the status of the project. thousands of tests, we were well The Telerik Testing Framework and Test prepared to seed the Unified All stakeholders (development team Studio were developed hand-in-hand by the same dedicated team. Telerik Automated Platform. and management) were very well Test Studio provides an intuitive UI, informed. Everyone played a part in However, we knew that we had to sophisticated recording capabilities the process until we rolled out the and the ability to share test results build a platform that could scale first deployment of the new platform directly between QE and developer and grow over time. To do that, we teams, speeding the development cycle. and automated tests. Of course, we grew a team of software engineers Test Studio follows Telerik’s aggressive had early adopters that helped pave release cycle with regular updates and with a solution architect. We made the road for the project. new features, and Test Framework is the case and secured budget to do updated alongside Test Studio with each release. so. In addition, we secured budget How closely did your automation to procure a VMware “farm” of developers work with your automation hosts that can execute system developers? Test Studio and the Telerik Testing automated tests for years to come. Framework are used extensively by The Vistaprint QE organization was Telerik product teams in perfecting the primary customer for the Unified Telerik’s legendary components and widgets. Building on Telerik’s drive for Automation Platform. During the product reliability and outstanding beginning stages of the development customer care, the Telerik testing solutions offer best-of-breed support and learning resources to ensure customer success in the field of test automation. of the platform, the QE organization process. We built group-specific played a major role in providing helper methods into our business the requirements with the solution frameworks to make automation architect. As the project progressed easier. For example, one such helper into coding phase of the project, we function is a PayPal page handler that kept the QE organization abreast of logs in with a specific PayPal user ID. changes in meeting the requirements. What sorts of infrastructure/ As we deployed the unified platform backing frameworks did you in phases, the automation team develop to help support your created automation programs/ automation efforts? scripts to test the system. After we We created several infrastructure had a “bake off” between the legacy frameworks and some are and new automation platform, the listed below. automation team fully engaged with the new platform writing •• Browser UI framework: Offers new, efficient and well-designed an abstraction layer for testing automation programs. browser based applications. Uses the Telerik testing framework. Did you alter your system in any way to make it more •• Windows UI framework: Offers automation-friendly? an abstraction layer for testing WinForms or WPF applications. Not so much, though we are now developing more interfaces and •• Common Framework: Offers APIs, which can be leveraged for class libraries with utilities of such automation. One of the guiding generic value as to be useful principles for the new platform was to automation scripts across to increase the efficiency of creating functional areas. automation scripts. Therefore, we •• Administrative framework: This created robust frameworks that subsystem administers the were easy to piece together. These UAP automation farm and also frameworks were designed in layers UAP services. based on unique subsystems. At the top layer, we created the business Mobile Automation framework: frameworks that used one or Offers an abstraction layer for testing more foundational/infrastructure browser based mobile devices such as frameworks. Android and the iPhone.

To test our website, our QE What were the two biggest organization is roughly divided into impediments to implementing separate functional groups that tests automation on your project and specific website capabilities. For how did you overcome them? example, one group tests payments Two challenges were creating/ and another tests the ordering designing modular effective tests that needed to be automated, and compared to our legacy systems, accurately defining the business layer and based on the architecture and frameworks. Overcoming the test design of our unified platform, design problems required significant overall cost of maintenance has amounts of training on effective test reduced significantly. design and with the new platform. It was a significant cultural change for most of the people in QE since most THE RESULT teams had embedded practices for creating tests that they were using What were the biggest gains you for several years. We overcame found using Telerik? the business layer requirements By using the Telerik Testing by continuously meeting with end Framework: users of the system, through the development process and refining the •• We saved an enormous amount design and code over time. of time and effort, since we were familiar with the product, and it What major problems did you find met our varied needs. with maintainability/stability of your test suite? •• We incorporated this framework into our single UI framework. There are always changes with applications under test. The •• We were able to test using recent automation that deals with regression versions of IE and FF. tests changes relatively infrequently. How do your stakeholders feel However, we initially spent significant about the value of automation? maintenance time updating element IDs, triaging network problems, The organization places a high browser differences, and other value on automated testing as it is misconfigured tests. Over the past efficient and helps with faster time year, we have implemented various to market. Everyone is supportive fail-safe mechanisms to assure of automation efforts since they stability. The automation farm uptime identify approximately 8-10 percent of has been upwards of 98 percent. defects. The new platform has been adopted successfully. We are starting How much effort was required for to see a reduction in time to maintain, ongoing care and feeding of your thereby increasing the ROI of our test suite? automation efforts. There are always updates based on The first version of our triaging tool application changes that require is helping in timely triaging and changes to the automated tests will continue to reduce the cost of and/or business frameworks. ownership of automation. The new These changes take up most of the platform is also extensible and can team’s maintenance time. However, test web services APIs as well as UI based applications. What kind of ROI are you seeing? 2. Capture and confirm requirements, tests are designed and executed. design, code, review and repeat Since we have a three week release Though still early, here’s what we in small increments. In building cycle, it took more time than have seen so far: a new platform, ensure that it is expected for people to adapt. As we •• Use of templates and architected for flexibility, scale move forward, we will focus more on configurations, has helped many and robustness. “white box” testing vs. “black box” teams see up to 75 percent 3. Tool selection is critically testing, and extending our platform reduction in number of tests important. Features, flexibility, to support continuous integration written and automated. meeting the requirements, and development-level testing. •• Automation team’s effectiveness and having a roadmap are very important. has doubled over the past 18 months. 4. Ensure that you have stable test THE VERDICT environments. It is difficult to •• Time spent on maintenance To sum up, how would you execute automation successfully in reduced by 10 percent. describe your experiences a highly dynamic environment. with Telerik? What were the biggest lessons What would you have done you learned in this project? The Telerik Testing Framework has differently, and are you applying been an invaluable resource for us The following are some that to future automation efforts? in creating an automation solution key takeaways: Overall, we have successfully for Vistaprint. Without it, our efforts 1. Management must realize the deployed a quality automation would have been much more limited effort involved in building such platform that scales well with our and would have taken significantly a platform. It requires significant business. Using automated UI testing longer to accomplish. Telerik’s investment in people, capital has been done at Vistaprint for support for the framework has been budget, and cooperation across many years. The new platform has very valuable and has considerably the company. standardized the way QE automated reduced time required to create automated tests.

FIND OUT MORE

ABOUT VISTAPRINT ABOUT TELERIK TEST STUDIO AND THE TESTING FRAMEWORK Vistaprint N.V. (Nasdaq: VPRT) empowers Go to more than 15 million micro businesses The Telerik Testing Framework is the free and consumers annually with affordable, test automation framework from Telerik, www.telerik.com/test-studio makers of Test Studio – the one-stop professional options to make an Call us 1-888-365-2779 impression. With a unique business model solution for functional, performance, supported by proprietary technologies, load and mobile testing. The Telerik Email [email protected] high-volume production facilities, and Testing Framework helps automate AJAX, direct marketing expertise, Vistaprint Silverlight and WPF applications. Users offers a wide variety of products and benefit from rich API, LINQ support, services that micro businesses can use to browser abstraction, special wrappers for expand their business. A global company, Telerik RadControls and integration with Vistaprint employs over 4,100 people, unit testing frameworks such as VS Unit operates more than 25 localized websites testing, NUnit, MbUnit and XUnit. globally and ships to more than 130 countries around the world. Vistaprint’s broad range of products and services are easy to access online, 24 hours a day at www.vistaprint.com. ABOUT TELERIK Telerik is a market-leading provider of end-to-end solutions for application development, software testing, agile project management, reporting, and content management. Telerik’s award-winning software development products enable enterprises and organizations of every size to generate tangible productivity gains, reduce time-to-market, and stay on time and under budget. With tens of thousands of users in more than 90 countries around the world, Telerik’s customers include numerous Fortune 2000 companies, academic institutions, governments, and non-profit organizations.

USA AUSTIN, TX 33 Alexander Malinov Blvd. 221 W 6th Street, Suite 850 1729 Austin, TX 78701 phone: +359.2.8099850 phone: +1.512.535.2428 option 3 fax: +359.2.8099888 fax: +1.512.722.7748

UK BOSTON, MA 15 Bedford Square 201 Jones Rd London WC1B 3JA Waltham, MA 02451 phone: +44.20.7291.0580 phone: +1.888.365.2779 fax: +1.617.249.2116 AUSTRALIA Sydney Balanstrasse 73, HOUSTON, TX Suite 705, 80 Mount St 81541 Munich 10200 Grogans Mill Rd, Suite 130 North Sydney, NSW 2060 phone: +49.89.2441642.70 The Woodlands, TX 77380 phone: +61-2-8090-1465 fax: +49.89.2441642.98 phone: +1.888.365.2779 e-mail: [email protected] fax: +1.617.249.2116 Copenhagen INDIA HUDSON, OH Vesterbrogade 149 #505, Tower A, Spaze I-TechPark 1790 Stoney Hill Dr. phone: +45-70-208-875 Gurgaon Sohna Road, Sector 49 Hudson, OH 44236 DK-1620 Copenhagen V Gurgaon, Haryana. 122 002 phone: +1.888.365.2779 e-mail: [email protected] phone: +91.9899115375

Copyright Telerik 2013. All rights reserved. Test Studio is a registered trademark of Telerik AD. All other trademarks are properties of their respective owners. All information believed correct at time of printing.