THE 2018 DZONE GUIDE TO Automated Testing YOUR END-TO-END ECOSYSTEM
VOLUME II
BROUGHT TO YOU IN PARTNERSHIP WITH THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
Dear Reader, Table of Contents
Welcome to DZone's 2018 Guide to Automated Testing. Automated 3 Executive Summary BY MATT WERNER testing has grown significantly from its early purpose of conducting simple tests ensuring that developer code has not regressed from build 4 Key Research Findings to build, to being of product-level importance for engineering software BY JORDAN BAKER at the quality and speed necessary to succeed in today's software 8 Continuous Testing and Progressive Web Apps business. Automated testing is what powers today's Continuous BY CARLO CADET Delivery and Deployment pipelines and is a non-negotiable part of a 12 How Your Project Will Take a Leap With Test-Driven modern software development lifecycle. Development BY STEFAN THORPE In this Guide, we will focus on automation methodologies, automation classifications, how to stratify automation, and the cultural changes 16 AI in Test Automation necessary to build a successful pipeline and an automation-first BY TAMAS CSER engineering team. 20 Embracing the Chaos of Chaos Engineering BY CHRIS WARD In the yesteryears of software, i.e. Waterfall, automation was a task that was usually left for the time-crunched testing phase of the SDLC. 22 Infographic: Treasuring Delightful Development If the team was fortunate, they would gather functional tests, primarily 26 Setting Up Automation for Webhooks Testing focusing on the happy paths, and fold them into the build process. BY SLAVEN SLUGIC If the team was very fortunate, they might also have a suite of unit, performance, and security tests. Why not do this at the end, right? 29 Diving Deeper Into Automated Testing There is always a support team that will pick up the pieces, right? That 32 End-to-End Tests: The Pinnacle in Test Automation world has changed. BY CHRIS KRAUS
In the services world, there is little room for buggy software and there 34 Executive Insights on Automated Testing are a lot of options for the customer to choose from. You need to ship BY TOM SMITH
high-quality software the first time — not version 3.0. Customers expect 36 Solutions Directory DZONE.COM/GUIDES software to just work, to be available in the way we expect electricity to flow when a switch is flicked. Customers don't just request quick 41 Glossary turnaround on broken issues — they demand it. If your cycle time to get a defect fixed is in the frame of weeks or even months, then you have DZone is... lost your customer. Cycle times need to be days, hours, or possibly even
minutes. Automated testing is the only way to satisfy these needs. BUSINESS & PRODUCT EDITORIAL SALES MATT TORMOLLEN CAITLIN CANDELMO CHRIS BRUMFIELD CEO DIR. OF CONTENT & COMMUNITY SALES MANAGER As such, automated tests and automation frameworks are tightly MATT SCHMIDT MATT WERNER FERAS ABDEL PRESIDENT SALES MANAGER coupled with your automated pipeline. Join us as we look into types PUBLICATIONS COORDINATOR JESSE DAVIS SARAH DAVIS ALEX CRAFTS EVP, TECHNOLOGY DIRECTOR OF MAJOR ACCOUNTS of automated tests, types of frameworks, where to integrate these PUBLICATIONS ASSOCIATE KELLET ATKINSON JIM HOWARD MEDIA PRODUCT MANAGER MICHAEL THARRINGTON SR. ACCOUNT EXECUTIVE into your pipeline for maximum coverage and throughput, and how to CONTENT & COMMUNITY MANAGER II JIM DYER SR. ACCOUNT EXECUTIVE evolve to a hands-off testing world. PRODUCTION KARA PHELPS CONTENT & COMMUNITY ANDREW BARKER CHRIS SMITH MANAGER SR. ACCOUNT EXECUTIVE DIR. OF PRODUCTION TOM SMITH BRIAN ANDERSON Thank you for downloading this Guide. As the software industry ANDRE POWELL RESEARCH ANALYST ACCOUNT EXECUTIVE SR. PRODUCTION COORD. continues to push forward at breakneck speeds, check back at DZone MIKE GATES SARA CORBIN ASHLEY SLATE CONTENT TEAM LEAD ACCOUNT EXECUTIVE DESIGN DIRECTOR BRETT SAYRE to catch up with latest in automated testing trends and technologies. JORDAN BAKER ACCOUNT EXECUTIVE G. RYAN SPAIN CONTENT COORDINATOR PRODUCTION COORD. As with any great community, your feedback matters, and we would BILLY DAVIS ANNE MARIE GLEN CONTENT COORDINATOR PRODUCTION COORD. love to hear it. As much as automation has grown, it is but a piece in MARKETING NAOMI KROMER ANDRE LEE-MOYE SUSAN WALL CONTENT COORDINATOR the software engineering machine. Be sure to check out more awesome SR. ACCOUNT MANAGER. CMO JASON BUDDAY LAUREN FERRELL AARON TULL articles at DZone.com. ACCOUNT MANAGER CONTENT COORDINATOR DIRECTOR OF MARKETING MICHAELA LICARI LINDSAY SMITH LAUREN CURATOLA ACCOUNT MANAGER CONTENT COORDINATOR MARKETING SPECIALIST KRISTEN PAGÀN By Rick Seversen MARKETING SPECIALIST DIRECTOR OF ENGINEERING AT DZONE JULIAN MORRIS MARKETING ASSOCIATE
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 2 OF 44 THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
Executive Summary
BY MATT WERNER - PUBLICATIONS COORDINATOR, DZONE
While "automate everything" has long been a catchphrase of TDD in certain teams, and 28% use TDD throughout the whole DevOps advocates, it's still not considered the de facto way of organization. Only 5% of those who use TDD said they were doing things, despite the amount of evangelism on the topic unsure of the benefits. and advancements in technology. For our second Guide to Implications: 95% of survey respondents who use TDD in some Automated Testing, we wanted to find out if attitudes towards capacity have seen some benefit from adopting the practice. The automation had changed within our audience. This year, 541 most significant benefits were improved code quality (76%), less DZone members told us what tests they were automating, as well time spent debugging (60%), easily maintainable code (46%), as other tactics they were using to improve the quality of their and the ability to use tests as documentation (45%). software at all stages of development.
Recommendations: The amount of people who recognize that SHIFTING LEFT TDD has had some kind of benefit for their organization is a Data: 54% of respondents are starting to automatically strong indicator that the practice is worth looking into for anyone test software in development, compared to 52% in 2017. In who has yet to adopt it. For more information, check out our fun addition, 51% are manually testing software in development Test-Driven Development Infographic to learn about some of the compared to 48% in 2017. Also, more DZone members are basic tenets and why they're important. performing both kinds of testing. 11% reported they did not
perform automated testing in 2018, compared to 23% in 2017, DEVOPS TEAMS AND TESTING and 2% reported that they did not perform manual testing in Data: Those who have a DevOps team in their organization 2018, compared to 23% in 2017. have reported that they perform more integration tests (57%), component tests (53%), and performance tests (55%) compared Implications: There is a slow growth in the amount of testing to those that do not. Those who do not have designated DevOps that is happening during development, and a sharp drop in teams in their organizations perform these kinds of tests by 14%, respondents that reported that they did not perform any kind 6%, and 10% less, respectively. of either automated or manual testing. The sharp drop in those who were not performing any testing could be attributed to an Implications: While there is some debate on whether dedicated increased knowledge of what kind of testing is happening as well DevOps teams are necessary for adopting DevOps or not, they as the utilization of testing tools and best practices. are clearly encouraging the adoption of both automated and manual testing practices. Recommendations: "Shifting left" is a common term for performing more tasks earlier in a product pipeline, specifically Recommendations: Even if there is not a designated DevOps development. The earlier testing can happen, the better code team in your organization yet, culture can be influenced from generally is, and less time is spent getting it into production. the bottom up. Speak to your immediate managers or upper leadership about adopting DevOps tools and best practices TEST-DRIVEN DEVELOPMENT that could help your organization develop and release better Data: Only 36% of respondents reported that they performed software. There are several articles on the culture of DevOps no test-driven development (TDD) at all. Another 36% use adoption in our DevOps Zone and Research Guide.
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 3 OF 44 THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
Key Research Findings
BY JORDAN BAKER - CONTENT COORDINATOR, DZONE
they begin automating their tests, 54% of respondents told us DEMOGRAPHICS development, 31% in staging/QA/testing, and 4% in deployment. For this year's Automated Testing Guide Survey, we received 821 responses, with a 64% (526) completion rate. Here's how the basic The most popular tests to automate in the development demographics from this group look: pipeline were integration tests (77%), component tests (52%), • 41% work as a software developer, 19% are developer team and performance tests (51%). Comparing this data to last year's leads, and 13% are architects. automated testing survey, we see that integration tests are on the • 22% are employed by enterprise-level (1,000-9,999 employ- rise, while the frequency of component and performance testing ees) organizations. is falling. In last year's survey, 61% of respondents automated • 65% work on immediate teams of 10 or less people. their integration tests, which, when compared with this year's • 87% identify as male, and 6% identify as female. data, constitutes a 16% increase. Automated performance tests, • There were three main geographic areas reported: however, suffered a 5% decrease year-over-year, and automated –– 18% from the USA. component tests saw a 6% year-over-over decrease. DZONE.COM/GUIDES –– 20% from Europe. DZONE.COM/GUIDES –– 26% from South Central Asia. Since manual testing still has its place in the development pipeline, and there's a small, but not insignificant, faction of AUTOMATED VS. MANUAL TESTING organizations who do not use automated testing, let's quickly look Only 11% of respondents told us that their organization does not at where and how manual tests are used and how this compares practice any kind of automated testing. Out of the 89% who do to automated tests. When asked at what stage they begin manual automate tests somewhere within their organization and SDLC, testing, 51% reported development, 40% said staging/QA/testing, there were varying opinions as to when it's best to automate tests, and 7% told us deployment. Not too surprisingly, manual tests are and which tests should be automated. When asked at what stage used in much the same places in the pipeline as automated tests.
GRAPH 01. Do you have an officially designated DevOps GRAPH 02. What programming language ecosystems does team in your organization? your company use?
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 4 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM Integration for atleast someprojects. degree, and65%saidtheirorganization Continuous hasachieved theirorganizationbelieve to some ContinuousDelivery hasachieved easily maintainable code (46%).Additionally, 59%of respondents improved code quality(77%),lesstimespentdebugging(60%),and process.development Thebiggest benefitsreported from TDDwere onlyuseTDDatcertainwith 36%sayingthey timesduringthe work for organizations thathave implemented TDDinsomecapacity, methodologiesDevOps reported inthissurvey. 64%ofrespondents Test-driven andCI/CDwere development someotherpopular tests and55%for performance tests inthisyear's survey. performance tests rose from 51%in2017to 53%for component teamswith DevOps whohave automated theircomponent and to 57%thisyear. Additionally, thepercentages oforganizations their integration tests increased from 49%inour2017survey expected number, thepercentage teams ofDevOps whoautomate teamDevOps intheirorganization. Despite thislower-than- adopted. Surprisingly, only48%reported havingadesignated we asked ourcommunity practices whatotherDevOps have they As automated testing isacomponent oflarger trends, DevOps CONTINUING THEDEVOPS TREND manual component tests fell from 40%in2017to 29%in2018. performance tests decreased from 45%in2017to 34%in2018;and integration tests, whilein2018thisnumberfell to 38%;manual decreased. In2017,41%ofrespondents performed manual of manualcomponent, performance, andintegration tests all component andperformance tests noted above, theinstances Interestingly, despite theyear-over-year decrease inautomated tests, and38% integration tests. said useracceptance tests, 50%usabilitytests, 41%story-level organization’s pipeline(s)are currently performed manually?', 60% between manualandautomated. Whenasked, 'Whichtests inyour When itcomes thetypesoftests, however, we seesomedivergence are currently automated? GRAPH 03. Which tests in your organization’s pipeline(s) THE DZONE GUIDETO AUTOMATED TESTING: decline, falling 20%(71-51%). (61-58%), andautomated component tests witnessedaprecipitous (77-83%), whereas automated performance tests declinedby 35% automated integration tests sawa6%year-over-year increase "Automated vs.ManualTesting" section.AmongSeleniumusers, we earlier seeasimilartrend astheonewe inthe observed of automated test (automation, component, andperformance) If we compare thesethree maintools to thethree maintypes workedthat they withJMeter, butthisyear thisnumberfell to 45%. use ofJMeter. In2017,50%ofJava ecosystem developers reported Selenium accompanied (orperhapscaused) aninverse trend inthe of Java ecosystem developers in2017to 55%thisyear. Thisrisein however, sawadecent riseinitsadoption,increasing from 48% in 2018thisnumberstayed relatively stable at69%.Selenium, developers actively worked withJUnitto automate various tests; trend interms ofadoptionrates. In2017,68%ofJava ecosystem ecosystem withthesethree populartesting tools, we seeasimilar When we compare thenumberofdevelopers working intheJava (down from 45%lastyear). ranked third witha41%adoptionrate amongourcommunity respondents using it(upfrom 46%in2017),andApache JMeter this tool intheir testing. Seleniumfinishedsecond with52%of our audience with60%ofrespondents claimingto isJUnit, use clear favorites emerged. Themostpopulartesting tool among (36%). Interms oftools, three(69%), Python(37%),andNode.js programming language ecosystems were client-side JavaScript respondents actively work intheJava language. Otherpopular that theircompany usestheJava ecosystem, and65%of among thisyear's reported respondents. 86%ofthosesurveyed Java was far-and-away themostpopularlanguage andecosystem TOOLS, LANGUAGES, ANDAUTOMATED TESTS are currently performed manually? GRAPH 04.Which tests in your organization’s pipeline(s) YOUR END-TO-END ECOSYSTEM PAGE 5OF44 DZONE.COM/GUIDES DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM SPONSORED OPINION PARTNER SPOTLIGHT I’ve seen,” said Batraski. “And it’s real technology, not vaporware.” not technology, real “And it’s Batraski. said seen,” I’ve product testing QA all-in-one best the has “Functionize platform. testing autonomous Functionize the chose Batraski platforms, in-one all and tools, testing Selenium-based solutions, open-source some evaluating After team. QA the on pressure the ease and accurate, more testing make throughput, testing the raise could that solution testing aQA for looking began Batraski Batraski. said scalable,” not definitely was approach testing manual “Our testing. the all do to people three of team aQA on relying platform—and software HR its for processes testing QA manual using was Zenefits bonuses. and raises, taxes, deductions, including payroll and insurance; medical including enrollment benefits termination; and onboarding, hiring, employee as such things include functions can’t-break These flawlessly. work must platform software HR its of functions certain succeed, to Zenefits For States. United the throughout businesses midsize and small 10,000 than more by used platform (HR) software resources ahuman provides Zenefits Containers Production-Grade technologies to automate thepainstaking software testing process. Functionize delivers anautonomous testing platform thatincorporates AIandmachinelearning Functionize Continuous Development Is theCatalyst of Autonomous Testing 4. 3. 2. 1. ways: following the in engine AI our through areality testing continuous making is platform cloud autonomous today. Functionize’s enterprises in 40% under well remains automation test of adoption The CASE STUDY CATEGORY Self-healing test cases via Adaptive Event Analysis Event Adaptive via cases test Self-healing browsers. all across parallel in cases test of thousands of Execution data. user live on based creation test autonomous Completely case. test afunctional into converted automatically is that English plain in journey auser submit to testers allows modeling +AI NLP EMAIL [email protected] THE DZONE GUIDETO AUTOMATED TESTING: Continuous, SaaS delivery RELEASE SCHEDULE TM . TWITTER @functionize
inherent in current QA testing approaches. testing QA current in inherent risks the reducing satisfaction—while customer higher and releases software faster enables Functionize cycles. QA by hampered are which processes deployment agile reexamine to confronted is C-suite the delays, massive from suffer projects large-scale When 4. 3. 2. 1. manifold: are strategies DevOps their into automation testing self-healing Functionize’s implement that enterprises to benefits The YOUR END-TO-END ECOSYSTEM data feedback allows better resource utilization. utilization. resource better allows feedback data Executive insights brand. damage irreparably can which introduced, be won’t errors major that UX Better time. on deployed be to product enables maintenance and execution, creation, for : Scale Faster time-to-market are depreciated. tasks maintenance manual of majority the and scripting Significant costSignificant reductions FOUNDER ANDCEO-FUNCTIONIZE WRITTEN BY TAMAS CSER : Self-healing allows enterprises to have confidence confidence have to enterprises allows : Self-healing • • • • • • • • No OPEN SOURCE? STRENGTHS NOTABLE USERS mobile web mobile including browsers, all across scalability Elastic maintenance Self-healing Autonomous testcreation creation +AI-test NLP Intuit Intuit Salesforce Zenefits Hyundai : Actionable insights provided by continuous BLOG functionize.com/blog : Labor hours are reduced as reduced are hours : Labor
PAGE 7OF44
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
QUICK VIEW
01. It takes automated testing to make continuous delivery possible — and practically speaking, the right test automation at the best possible time.
Continuous Testing and 02. Continuous testing is a critical part of gaining efficiencies throughout DevOps.
03. Progressive web apps Progressive Web Apps technology has arrived, and when implemented well, proves to streamline SDLC while ensuring great UX.
BY CARLO CADET SENIOR DIRECTOR PRODUCT MARKETING AT PERFECTO MOBILE
Continuous testing (CT) is all about getting feedback to those tests pass, automated end-to-end tests can run in order to developers more quickly. In theory, this should increase their ensure that things still work as expected. Other testing — stress productivity and allow them more time to innovate. Whether tests, performance tests, or other large tests — may also be run you subscribe to an "Agile" philosophy, a DevOps methodolo- before the code is deemed ready for release. After this point, gy, or the principles of Continuous Delivery, in practice, it takes monitoring and alerts are (for some devs, anyway) other types of DZONE.COM/GUIDES automation to make CT work. testing that may happen during production. DZONE.COM/GUIDES
In the current era of digital transformation, organizations are Basically, CT is lots of tests, each running at the most always looking for new ways to offer a better user experience appropriate point in the development cycle. (UX) in their online offerings. Traditional web and native app technologies have steadily given way to responsive web design Testers want to create tests that will find issues with the (RWD); for a few years now, it may have looked as though RWD customer experience. They also want to get feedback from was the best hope for developing a universal cross-platform tests as quickly as possible; to make this happen, they target experience for end users. Recent developments (especially their tests to find issues at the earliest possible time. Bugs Apple's decision in March of this year to finally back "service that can be found by unit tests should be found by unit tests. workers" and Google's solution for adding more app-like capa- The same is true for acceptance and integration tests and bilities to the web experience) mean that developers now have is especially important with end-to-end tests. While a lot of a new technology with which to create a uniform experience automation efforts focus on end-to-end tests, they should only across all platforms: progressive web applications (PWA). exist for bugs that can only be found with end-to-end tests.
PWAs are a powerful new tool for ensuring a great UX within the Software products with good CT systems enable teams to confines of mobile web technology; however, they do require a deliver changes and updates to customers frequently and rethink regarding how to implement continuous testing. safely and to use testing at every single integration point to help us determine whether we're heading in the right Continuous testing begins with the developer, who can run unit direction. In the context of CT, one promising aspect of PWAs is tests as part of every local build. Once the code is checked in, that teams can release as frequently as they see fit — directly integration and other system-level tests are run automatically. If to the user's device/browser without having to go through the
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 8 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM responsiveness, makingtheweb appahitwithusers. iOS app!Naturally, thisenormouslyimproved load timesand 99.84% smallerthantheir148MB (that's megabytes) mobile a PWA,thefinalversion oftheirPWAclocked inat233KB— Simplybyproject. thenature ofthespecificrequirements of resultsTheir PWAachieved well themainscope beyond ofthe progressive web: that are starting to seebigpayoffs from making theswitch to great effect. Take aquicklookat few examples of companies codebase ways whiledevising to cutpage load times-andto companies are usingPWAsasaway to start withafresh/lean features. For thetime being,thedata seemsto suggest thatbig technology we develops, can expect ofcreativity levels new and ining everyone's internet presence, onallplatforms. Asthe Apple's change ofheart,PWAsbecame aviableway ofreimag - Instead ofremaining anichetool for theAndroid market, with fits ofPWAsandtaking notice oftheirsuccess andefficiency. native appswilltransition to PWAs.Teams are seeingthebene ing thetransition. Gartnerhasstated thatby 2020,50%ofall the next 12months,withanadditional32%actively research - to theirsites already, followed by aresponsive web designover by Perfecto, 41%ofrespondents planto addPWAcapabilities adoption isexpected to increase. Inarecent conducted survey enriching userexperiences. seemsto agree: PWA Everyone and GPS. These capabilities are propelling PWAadoptionand features like cameras, microphone, speakers, data storage, receiving pushnotifications, andhavingaccess to platform now, were restricted to native apps: browsing whileoffline, conversions. PWAsoffer users access to features until that, apps improve userexperience, grow engagement, andincrease Across theboard, early reports confirm thatprogressive web business decision. customers, andvarious risksallcombine to helpyou make this types oftesting you do, your desired shippingfrequency, your you need inorder to produce agood Theamountand product. find issues.For someweb services, automated tests maybeall ing therightautomation runningatthebestpossibletimeto CT isnotjustabouthavingalotofautomation. It's abouthav Play Store. clearing itthroughof process theAppleAppStore orGoogle CHALLENGES STARBUCKS PROGRESSIVE WEBAPPS:NEWTECH, THE DZONE GUIDETO AUTOMATED TESTING: - - AND CONTINUOUS TESTING A CONVERGENCE OFEFFICIENCIES: PWAS TINDER they have comethey to expect. always-available appsthatgive themtheinstant gratification ogies are going to bebetter able to meet userdemandsfor fast, that pullPWAsinto theircontinuous testing process methodol- of intricacy for theentire build-release process. Thoseteams teamsdoubt thatDevOps are subjectto ever-increasing levels need to besupported/tested/developed/monitored. There's no inaddition,there arebut, othernative/RWD appsthatstill Not onlyare there slightlydifferent features/behaviors to test process pipeline. isanotheradditive elementintheDevOps rising to meet —butworking PWAsinto thecontinuous testing aPWAseemstoDeveloping beachallenge thatteams are a delightfully fast androbust experience withtheirlean apps. for are iOS(asof11.3), they well-positioned to offer allusers have allgotten anearly jumponPWAs,andnow, withsupport The teams atStarbucks, Tinder, andPinterest (andothers) "time to interactive" by astaggering 75%,from 23sto 5.6s. engagement andconversion. ThePinterest team reduced the ate afast, lightexperience thatwould ultimately lead to better all. Pinterest set out(muchlike theprevious examples) to cre to wait asmuch23seconds before anyUIwas usableat With Pinterest's previous mobileweb app, users often had seconds to ~4.5seconds. right impressive. Additionally, cutload they timesfrom ~12 native appto a2.8MBprogressive web app—whichisdown- or data-scarce market. managed They to shrinktheir30MB percent oftheformer data costs for someoneinadata-costly of theirefforts isdelivering the core Tinder experience at ten progressive web appbased onReact andRedux. Theresult Tinder took three monthsto buildandimplementtheirMVP University. Heserved asaCaptain intheUnited States AirForce. School ofManagement andaBScinComputer Science from Tufts and CISSP certifications. HeearnedanMBAfrom theMITSloan responsibilities at RSA Security. Duringthat timeheacquired CISA Conferences, aswell ashosting webinars. Previously, heheldsimilar speaks at industry events such asStarEast, STP CON, JenkinsUser as well asthecompany’s Customer Advisory Board. He strategy, andmanages theAnalyst &PublicRelations ing DevOps pipelines.Heisresponsible for go-to-market continuous testing andmonitoring platform for optimiz- CARLO CADETleadsProduct Marketing at Perfecto, a PINTEREST YOUR END-TO-END ECOSYSTEM PAGE 9OF44 - DZONE.COM/GUIDES DZONE.COM/GUIDES SPONSORED OPINION THE DZONE GUIDETO AUTOMATED TESTING: DEVELOPMENT AND MANAGEMENT PARTNER SPOTLIGHT Automated Testing Automated Functional UItesting madereliable andeasy mabl and ease of use. of ease and stability its for mabl QAwith all do and completely Selenium from away to move planning is James running. of 24 hours first the within found mabl that error aJavaScript on based flow login apps the in abug to fix able was James cases; test written of scope the beyond bugs uncovers proactively mabl’s crawler link and production, monitoring for useful is mabl from hourly in coming data The deployment. on and schedule, adaily on tests mabl running now is James application. Jobcase the of parts dynamic especially untested, remained functionalities many and production this, of Because tests. new writing than tests Selenium broken fixing time more to spend used He Jobcase. at 3teams for engineer automation QAand the is James Intelligent Testing Gives Riseto Modern Development tools will make it easy to create and maintain tests, they will will they tests, maintain and to create easy it make will tools These QAtools. next-generation for groundwork the have laid automation test and intelligence machine in Innovations MACHINE INTELLIGENCE ISHERE on. head challenges these to confront emerging are QAtools of generation However, anew QA. for challenges real present all quality end-to-end for accountability unclear and requirements, evolving cycles, release shorter development, feature Faster years. 10 past the over development software in advances with pace not kept has testing software of state The THE STATE OFSOFTWARE TESTING ISINPERIL CASE STUDY CATEGORY WEBSITE mabl.com THE DZONE GUIDETO AUTOMATED TESTING: Continuously RELEASE SCHEDULE
TWITTER @mablhq • • • • • • • • • • No OPEN SOURCE? STRENGTHS NOTABLE USERS IndoorMedia PassPortal JobCase Simpology Turo pipeline delivery software existing your with Integrate Easily of a test step every at diffs visual compare Automatically and screenshots logs test comprehensive with quickly issues repair and Diagnose changes UI your when tests repair Automatically scripting without tests UI functional automated Create reproduce and fix issues quickly. issues fix and reproduce developers to help context useful provide and language, plain in results and tests describe will they importantly, Most tests. their into learn they what incorporate success—and customer and QA, development, product, team—including entire the from training and feedback take will They channels. communications common other and email, Jira, Slack, through insights and results test share will They pipeline. delivery the into integrated fully be and cloud, the in run to change, seamlessly adapt innovation across the software industry. software the across innovation and efficiency of era new in a ushering of QA, effectiveness the improve dramatically will QAtools next-generation workflow, development modern the into embedded and cloud the from delivered intelligence, machine cutting-edge incorporating By challenging. more even it made has development software of pace quickening the and easy, been never has QA Software THE ERAOFINTELLIGENTTESTING DEVELOPMENT ANDMANAGEMENT CO-FOUNDER -MABL WRITTEN BY DAN BELCHER BLOG mabl.com/blog
PAGE 11OF58 THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
QUICK VIEW
01. Examine the fundamentals and rules behind test-driven How Your Project Will development (TDD). 02. Look at how TDD fits into the Take a Leap With Agile, Lean, and DevOps spaces. 03. Consider the pros and cons of TDD as well as tips on how to successfully implement Test-Driven Development the process.
BY STEFAN THORPE CTO AT CAYLENT
WHAT IS TEST-DRIVEN DEVELOPMENT? opment process and encourage teams to produce products with reliable Test-driven development (TDD) refers to an approach in software devel- scalability and desired quality. The TDD approach is highly applicable to opment geared towards reducing errors and improving flexibility when greenfield projects being developed from scratch. designing applications. It is a process that encourages a quick, rapid, and If the project is the development of a basic app or building on proof-of- fearless testing development style. This approach to programming enables concept, TDD is not necessarily justifiable. It can admittedly be time-con- tests to drive the design of the whole system. suming. On the other hand, it's not realistic to buy a car that hasn't been The following set of rules (designed originally by Robert C. Martin or "Uncle through any quality control (QC), so there's no real argument for leaving QC Bob") defines the best practice guidelines for TDD: out of a software development process, either.
1. You are not allowed to write any production code unless it is to make In Lean Software Development, the integral aim is to reduce waste in proj- DZONE.COM/GUIDES DZONE.COM/GUIDES a failing unit test pass. ects. The application of test-driven development reinforces this core aspect. 2. You are not allowed to write any more of a unit test than is sufficient Furthermore, TDD is useful for building quality into the process by elimi- to fail; and compilation failures are failures. nating defects from the very beginning rather than at the end of the system 3. You are not allowed to write any more production code than is suffi- where most QC takes place. Though building quality in is a central Lean prin- cient to pass the one failing unit test. ciple, it isn't always easy. Implementing TDD will help overcome this hurdle.
Abiding by these rules will provide IT teams with a tight feedback loop TDD IN THE DEVOPS SPACE between production code and test code. In DevOps, high-performing IT teams have turned to TDD to produce high- quality end products. While the DevOps umbrella can cover many workflows As highlighted earlier, the aim is to first write a unit test before any pro- and ideas, here are the ideal practices you can incorporate with TDD involved. duction code is started. The aim is to develop the system (and high-quality code) gradually through an ongoing sequence of tiny test improvements. EMBRACE AUTOMATION By rule two, you obviously can't write much of the unit test itself other than DevOps processes advocate breaking down heavy complex projects for is required for the test code to fail. This approach is generally referred to everyone involved in the project. All stakeholders are continuously involved as red, green, and refactor. Write a small failing test (red), and then help in the whole software development process and DevOps promotes a flow it to pass by writing good production code (green). Then, the final step is of work that reduces any negative impact on anyone else down the value to refunction the code (refactor) before it's possible to restart the process. stream. In this case, TDD can be implemented by first creating the initial (Callaway and Hunt, 2018.) tests and running them manually (as suggested in The DevOps Handbook) and once they're written, they can be added to an automation pipeline to DOES TDD FIT IN THE AGILE, LEAN, AND save time in the future. DEVOPS SPACES? WHERE TDD FITS IN THE AGILE AND LEAN SPACE BRING END-USER FEEDBACK FRONT AND CENTER For projects already affiliated with Agile methodologies of software devel- The DevOps methodology entails fostering a culture of collaboration, opment, implementing TDD is fairly simple. It is ideally suited for building especially with the ultimate end-users. Involving the end-users early in the complex projects with many features to clearly defined requirements — an process by constantly getting their feedback is crucial to make the final integral ideal of Agile. TDD will help to improve the efficiency of the devel- product meet their expectations. Teams can use TDD to engage in initial
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 12 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM ADVANTAGES OFTDD ADOPTION OF AN ENTERPRISE TEST MANAGEMENT TOOL (ETMT) TOOL MANAGEMENT TEST ENTERPRISE AN OF ADOPTION TIPS ONHOW TO IMPLEMENTTEST-DRIVEN the exact specifications (orhave timeto generate these)orwhere you're ate approach for your TDDisideal project. for products where you know also beafalse source ofsecurity. ascertain First, that TDDistheappropri TDD can provide amuch-neededboostofconfidence to teams, but itcan (Kimetenvironment. al.,2016.) TDD concept, aswell astheentire team, receive allthebenefitsofaTDD concept. —atrulyDevOps of development ETMTs to helpthosenew the receivesantees thateveryone allthenecessary information atallstages crucial supportinhandlingallthenecessary tests inTDD.Suchatool guar A robust ETMTtool like Jenkins,XLTestView to useful develop initialtests. understanding oftheneedsend-users. Havingthisknowledge willhelp tests neededto helpdesigntheend-product in accordance withathorough 1. ENSURETDDISTHERIGHT FITFORYOUR PROJECT DEVELOPMENT DEVELOPMENT PROS ANDCONS OFUSINGTEST-DRIVEN DISADVANTAGES OFTDD 4. 3. 2. 1. 6. 5. 4. 3. 2. 1. product designandplanningcan address the issueofgoing off-project.) wasted; e.g.writingtests for features thatmayget dropped later. (Good Thesechangeswhich can slow can development. result inalotoftime If there isarapid change ofdesign,changingtests isalsonecessary, and create. In complicated scenarios, test cases can betough to calculate can mitigate someofthisstep.) approaches(DevOps such aspeerreviews andpaired programming rics suchascode coverage, whichdoesnotactually assessquality. It doesnotnecessarily guarantee unittest quality. Focus goes onmet time andeffort to write unit tests. TDD slows atthebeginning, development asitnecessitates ample writing unittests —it's builtinto theprocess from thestart. suite. There isnoneedto spendtimetesting implementation code after Implementing TDDencourages theuseofanautomated regression test be fixed withouthavinganyimpact down thevalue stream. requirement issueshappensrightatthestart when itisearly enoughto TDD helpsreduce thedefects rate. Theidentification ofdesignor end-users' POVs. TDD offers developers greater insightinto theperspective ofthe a safety cushionduringtherefactoring step. tates clarityduringtheimplementation stage. Inaddition,itprovides TDD simplifiesthemaintenance andrefactoring ofcode. Italsofacili addressed swiftly. Thus,TDD facilitates good architecture. Inherent code structural problems surface muchearlier andcan be tals ofgood modulardesign. assists inunderstanding, learning, andinternalizing thefundamen Coding becomes modularwhenexecuting smalltests atatime.TDD THE DZONE GUIDETO AUTOMATED TESTING: , orSeleniumcan provide
- - - - - 4. DECIDEONONESMALLASPECT OFFUNCTIONALITY bring thefunction closer to becoming aworking end-product. the smallestamountofcode for each step ofyour process. Thiscode can team closer to theendgoal. Using simplebehaviorcan assistto determine Figure outthemoststraightforward possiblebehaviorthatdraws your writing, decidingtheparameters andreturn value isanecessary step. Thisstepresult. issimilarto writingcode withoutTDD.Generally, incode function takes andtheexpectations thatthefunction willreturn acertain vital to choosethefunction signature. Theseare theparameters that the Having established whatdata goes inandtheoutcome to it's expect, now parameter to consider before writinganycode ortest. to runandtheresults thatthisfunction willreturn. Thisstep isanintegral function aswell oftheproduct, asdata thatthefunction willrequire outline thenecessary outputs/feature steps to reach Considertheoverall it. goal. Decidetheinputsnecessary to thatproduct achieve goal andthen Start theprocess from ofdetermining ahighlevel your eventual product the bestroute to aqualityproduct. to stabilizetrying anexisting one—inwhichcase, thisprocess as can serve and growth. leads andfine-tunes development teams to optimize for innovation solutions to supportstrategic business objectives. Stefan management andhands-onexperience providing technical professional andDevOps evangelist with20+years of consultancy management platform Caylent . HeisanIT STEFAN THORPE • • • application flexibility, extensibility, andmaintainability. down theline.Test-driven isthe ideal development approach to achieving cess early onwillboostproductivity, andmarket output, value muchlater and address project improvement through TDD.Spendingtimeonthepro To make your organization successful, adaptto thechangingenvironments eliminating tedious andhelplesscoding after aproduct launch. a loteasier to asyour achieve team gets usedto theprocess, andithelpsin function, continue withthesesmallsteps, repeating from Step 4.TDDgets test asoutlinedatthebeginning ofthisarticle.To advance onthesame You shouldpractically write theslightest amountofcode to pass each unit closer you get to theend-product. When you beginstep, testing behavioratevery thetesting gets easier the conditions. Your focus isto test how step. thefunctions behave atevery function. Thisisopposed to how thefunctions behave concerning certain without TDD.Thesignificant difference isthe focus onimplementingthe It isworth notingthatallprevious steps were similarto programming 2. DECIDETHEINPUTS ANDTHEOUTPUTS 3. SELECTTHEFUNCTIONSIGNATURE 5. IMPLEMENTTEST 6. IMPLEMENTCODE REFERENCES Jun. 2018). Martin, R.(2005).TheThree RulesOfTDD . Available here. (Accessed 14 book. Portland: ITRevolution Press. Kim, G.,Humble,J., Debois,P., andWillis,J. (2016).TheDevOps Hand C# 7.Packt Publishing. Callaway, J. C.(2018). Practical andHunt, test-driven using development YOUR END-TO-END ECOSYSTEM is CTO for theDevOps-as-a-Service PAGE 13OF44
- - DZONE.COM/GUIDES DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM SPONSORED OPINION PARTNER SPOTLIGHT Continuous QualityLab Web AppTesting Automated Progressive with Perfecto to Support Leading Retailer Grows automated testing of desktop and mobile browsers in parallel. parallel. in browsers mobile and desktop of testing automated enabled platform testing cross-browser Perfecto’s to transition The on all platforms. executed tests regression nightly by complemented platforms eight on commit every executed is test smoke A15-minute platforms. twenty-four on parallel in executed is testing CI-triggered solution. cloud-based Perfecto’s to VMs managed internally from Protractor using testing their shift to decided They climbing. devices mobile from traffic web saw it design, responsive embraced team web later, the Ayear as development. app mobile native its for Lab Quality Continuous Perfecto’s selected originally retailer A leading Perfecto supports key automated PWA test scenarios. test PWA automated key supports Perfecto delivers better experiences and increases conversions. capabilities mobile native and access offline of adding combination the that demonstrate results Early technology. new(ish) this support to rallying now are providers browser other that is news good The (PWA). Apps Web Progressive championing is Google Testing Platform Automated Seek Perfection CASE STUDY CATEGORY • • • • User condition test scenarios websites. to devices mobile of capabilities Test audio, camera, and scenarios location-based limitations. Appium and Selenium overcomes This supported. fully is flow asingle in objects Android and iOS native and web both with interacting and identifying aresult, As test. under application an in objects all of view asingle provides –Perfecto &execution Test authoring automation. Launching PWAs WEBSITE perfecto.io – Launching PWAs from device home screens creates a new challenge for test test for challenge anew creates screens home device from PWAs –Launching THE DZONE GUIDETO AUTOMATED TESTING: – Offline testing is easy with Perfecto’s Wind TunnelPerfecto’s with easy is testing –Offline Continuous Deployment TWITTER @perfectomobile RELEASE SCHEDULE – PWAs bring the rich engagement engagement rich the bring –PWAs successfully automated. successfully be to scenarios all enabled screen) home the from (lauched devices mobile on PWA testing support announced recently Perfecto’s testing. App Web Progressive automating tackling on start ahead got team The bars. five to one from transitioning users simulate to conditions network degrading programmatically and utilizing visual scenarios, analysis familiar with camera-based already was team The PWAs. of testing automating navigate them help to able was Perfecto that clear became it However, testing. manual to revert to need the given slow might velocity development that suggested capabilities of PWA investigation Early access. offline customers offering for specifically (PWA), Apps Web Progressive with experimentation initial supporting for foundation the laid also team dev the toolchain, itsDevops into Perfecto Integrating sprint. same the within deploying and testing coding, successfully team the to contribute helping defects, triaging accelerate helped side-by-side results test compare to ability The YOUR END-TO-END ECOSYSTEM SENIOR DIRECTOR, PRODUCTMARKETING-PERFECTO WRITTEN BY CARLO CADET TM capability. • • • • • Yes OPEN SOURCE? NOTABLE USERS BLOG blog.perfecto.io Toyota Lloyd's of London IBM GE AT&T
PAGE 15OF44 THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
QUICK VIEW
01. If artificial intelligence doesn’t excel beyond automating simple, repetitive tasks, then it’s not going to meaningfully impact ROI.
AI in Test 02. True AI should empower personnel to be more effective, as their focus is shifted away from less repeatable tasks and more toward Automation creativity and innovation. 03. With AI, the C-Suite can focus on real-time analytics that impact the bottom line.
BY TAMAS CSER THE FOUNDER AND CEO AT FUNCTIONIZE
Today, almost all IT projects are faced with the challenge of operation- advancement in empowering developers to take further control of QA. alizing and deploying software and services with greater speed and However, the challenges of Selenium and popular record/replay frame- accuracy, creating an unrelenting, high-pressure environment for the works became readily apparent to developers who sought to use the project team. Requirements shift daily and there are never enough recorder in complex environments, as the selectors used for identifying engineers to make it all happen perfectly. A major part of the burden on elements had to be continuously updated with every code change. project teams is the need for continuous testing. TEST CREATION GUIDED BY MARKET DEMAND In this article, I will explore the opportunities I've discovered by Automated test creation has often been limited to three methodologies: applying artificial intelligence (AI) to test automation. AI is meant to manual, scripted languages (some with greater degrees of modularity make businesses far more capable and efficient. The best companies DZONE.COM/GUIDES than others), and record/replay tools. Each on their own offers value, DZONE.COM/GUIDES are using AI to enhance customer and client interactions, not eliminate but they are all constrained in conspicuous ways: them. Big data collection and algorithmic advances are vastly extending the scope of testing automation, making it possible for non-technical 1. Manual testing is slow and not designed for today's CI/CD pipe- team members to define and scale tests with levels of capability and lines because it can’t scale with complex applications. sophistication comparable to or even greater than developers'. In short, 2. Scripting is laborious, error-prone, and expensive, as engineer- AI is transforming all facets of test automation by streamlining creation, ing resources are usually required. execution, and maintenance, and providing businesses with actionable insights in real-time that directly affect the bottom line. 3. Record/replay tools struggle to capture sophisticated user workflows, and editing those workflows often requires having to BACKGROUND re-record everything. More than 12 years ago, I launched a consulting business serving both startup and enterprise clients alike. As I served my clients and looked to As I began listening to the market and our customers, it became clear streamline the time between committing a change to an application and that different options for test creation were desirable, but often not the change being deployed into live production, I found that the need presented within the same tool. With Functionize, I sought to offer both to ensure quality and reliability demanded greater and greater shares conventional and new modes of test creation, all enhanced by AI: of resources. There is an ever-increasing variety of combinations of • Simply writing a user journey in plain English or submitting a innovation, application components, and protocols that interact within a sequenced set of desired tests to our NLP engine, which uses AI single event or transaction. Over time, I realized that there was a need for to analyze and model the data. something more. Then, in the fall of 2013, I saw how AI could shape the testing landscape, and I wrote the first line of code for Functionize. • Training our AI modeler that learns the application.
THE ORIGINS AND LIMITATIONS OF TEST • Using our Developer Mode that enables robot-compatible script- AUTOMATION ing and automatically builds smart Page Object libraries that are Test automation is not new. The advent of Selenium in 2004 was a major modular and portable.
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 16 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM Artificial Intelligence. is confusing jargon AIasAwesome thatmis-defines Integrations, not struggling to integrate AIinto theirautomation stack, andtheresult to apixel/image approach to objectrecognition. Market leaders are tion. There alsoremains ahighdegree ofselector maintenance due tester mustmanuallyselectanddrag thedesired elementfor interac remain constrained. Test creation remains time-consuming, asthe scripting methods andemploy animage- orvisual-based approach However, test even automation frameworks thatgo traditional beyond es. Weset ourselves thefollowing acceptance criteria: highifAIisclaimedtoshould beset beaugmenting thoseprocess very As testers lookfor solutions to execute theirtests atscale, thebar the workload, withinformation shared across allexecution instances. to perform rapid scaling ofapplications, upanddown, dependenton of concurrent tests. Thewholepurposeofcloudcomputing istheability time, based onthenumberofnodesrunning, memory, andthenumber cloud technologies like SeleniumGridare stillhampered by execution ample room for AIto productivity. drive new On-premise andeven The dearth oftruecloudscale test execution optionsreveals there is degree to whichAIispresent intest creation. are ahandful ofexamples asalitmustest for thatserve judgingthe There isalotofnoiseinthemarket around AIintest automation. Below HOW AIIMPACTS TEST EXECUTION HOW AIIMPACTS TEST CREATION • • • • • • • • not hours, let alonedays. Even themostcomplex tests shouldtake minutes to execute — any device, withanybandwidth, andinalltypesofenvironments. Tests shouldbeexecutable from anywhere around theglobe,from more subsequentrunand efficientand release.every reliable with Tests shouldbeexecutable atscale, become inthecloud,sothey language processing andadvanced modeling. Fully autonomous test creation utilizingAItechnologies vianatural mously, increasing test modularityandscalability. Page objectrecognition thathappenscontinually andautono noise andirrelevancies. DOM andapplication states for meaningful information, rejecting AI andmachinelearning to continually scan andanalyze the than justHTMLandCSS. dreds ofselectors. Thisrequires amuchbroader focus andingest Machine visionthatautomatically locates andidentifieshun- cases from live userdata. Fully autonomous test creation thatanalyzes andgenerates test THE DZONE GUIDETO AUTOMATED TESTING: - - - TRANSFORMS BUSINESSES minimizing humanintervention. learn thestructure andorganization thatisuniqueto your application, Applied AImethods suchastheseensure thatyour testing frameworks creation. AIcan helphere: maintenance ultimately resolves to inadequate data modelingduring happening to thedata duringtest creation. Thefailure pointof test the bottleneckinadeployment pipelineisto identifywhatisactually tests. Themostefficient way to ensure that test maintenance isnot Rapid test creation isonlyasviabletheresiliency ofexecuted be identifiedautomatically, anddetails passed back to thedevelopers. these users withthoserunningtheexisting code. Anomalies can then real-world testing code. ofnew AIisusedto compare theexperience of detection through testing canary where asmallset ofusers are usedfor Functionize ispartnering with GoogleCloudto buildadvanced anomaly bottom line. business intelligence previously unavailable thatdirectly impacts the catalyst withinaCI/CDpipeline,butalso, executives are provided with lined whentesting automation changes from thebottleneckto the automation. Notonlyisproduct significantly development stream- capable oftackling themostrepetitive problems presented intesting strategies,within AgileandDevOps assmartalgorithms are now better product results. Developers are renegotiating theirinvolvement areprise level already experiencing greater operational and efficiency Businesses thathave acommitment to implementingAIattheenter HOW AIIMPACTS MAINTENANCE HOW AI-POWERED TESTING AUTOMATION user experiences. need to innovate withtheneedto deliver ever-better cloud. Itallows businesses to effectively balance the the AI-powered autonomous testing platform inthe TAMAS CSERistheFounder andCEOof Functionize, • • • • YOUR END-TO-END ECOSYSTEM ment andshouldn'trequire diggingthrough scripts. identifying broken tests shouldtake seconds inavisualenviron- Computer visiondiagnosis: AImeans visualdiagnosisiseasy: rated and ranked instead ofasingleselector. having elementsidentifiedby hundreds ofdata pointsthatare Data modeling:Selector maintenance shouldbeeliminated by causes for test failure andprovides apath for updates. one-click Self-healing tests : Root cause analysishighlightsallpotential CSS selector, andparent/child elements. previously known size andlocation, visualconfiguration, XPath, of data location points,suchasthesize onapage, ofelement, tained by thecombination ofanexhaustive andautonomous set Self-maintenance: Resultant tests are modeledandthusmain- PAGE 17OF44 - Intelligently AutomateYour End-To-End Testing in the UnitedKingdom, theUnited States, andother countries. Copyright ©2018Eggplant andtheEggplant logoaretrademarks ofEggplant
DZONE.COM/GUIDES DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM SPONSORED OPINION PARTNER SPOTLIGHT applications from the user viewpoint in a stable and proactive manner. manner. proactive and astable in viewpoint user the from applications online live, of monitoring 24/7 through experience user the improve continuously now can DOCOMO NTT Eggplant, With capabilities. monitoring and confirmation service deliver as well as areas, untestable previously test to Eggplant uses DOCOMO day. of NTT time the on based and environment, cloud external an with content, of avariety across network, the within varies that traffic including factors complicated by determined is services internet of quality end-user The networks. wireless advanced via customers million 74 than tomore services mobile secure and convenient, innovative, providing is DOCOMO Japan'sNTT mobile telecommunications company, largest performance and monitoring, website testing, UX AI, Test automation, enhance thequalityandperformance ofthedigital experience. user-centric,An intelligent, test automation suite thatusesAIandmachinelearningto Eggplant Intelligent Automation elements: key five with approach an and products, technology, modern involves automation test true Achieving testing. manual time-consuming, of lots do still teams product Yet DevOps. and toCD turning are Toit. up, organizations keep from want users what does, it what produced, and developed architected, it’s changed—how has software about Everything Approach Requires aModern True Automated Testing CASE STUDY CATEGORY • • • Expand automation beyond test execution using AI, AI, using execution test beyond automation Expand totest. way intuitive amore for tive perspec auser from usability and performance, tionality, func Test UX. digital the of aspects all test Continuously code. the through not do, users way the devices with user. Engage the of eyes the Test through WEBSITE eggplant.io THE DZONE GUIDETO AUTOMATED TESTING: Quarterly RELEASE SCHEDULE TWITTER @eggplantio - - your business objectives. business your exceed and hit workflows and applications your ensuring and experiences digital toimproving key are technologies these All environments and platforms, across browsers. simultaneously models test toautomate lab device on-premise or ahosted Use app. an with interacting when see actually users image search Smart time. prime for ready is arelease if todetermine touchpoints UX various from analytics Predictive coverage. test maximize and faster bugs tofind testing exploratory automated with tests regression andAI directed machine on learning build • • YOUR END-TO-END ECOSYSTEM testing process to boost team productivity. productivity. team toboost process testing the of more machine Automate learning, and analytics. tions to get a true, end-to-end view of performance. performance. of view end-to-end atrue, toget tions condi network real emulate and production in UX Monitor testing. and monitoring to approach Take acoherent rating. 5app of a4out net will now analytics. predictive with user the of terms in quality Report CEO, EGGPLANT WRITTEN BY DR.JOHNBATES Reveal quality specifics in user terms: releasing releasing terms: user in specifics quality Reveal • • • • • • • • No NOTABLE USERS OPEN SOURCE? STRENGTHS Walmart P&G Martin Lockheed Cerner Boeing capabilities Bug-hunting True testing UX analytics predictive learning, machine AI, BLOG blog.testplant.com automates testing of what what of testing automates
combines results results combines PAGE 19OF44
- THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
QUICK VIEW
01. Modern applications are increasingly growing in complexity.
02. A dizzying amount of moving Embracing the parts, layers of abstraction, reliance on external systems, and distributed systems all result in a stack that few Chaos of Chaos truly fully understand. 03. Chaos engineering aims to help you identify the seemingly random errors that plague our complex applications by injecting some if its Engineering own randomness.
BY CHRIS WARD ZONE LEADER AT DZONE
Modern applications are increasingly growing in complexity. Adding 1. Form a hypothesis. 4. Analyze the results. a dizzying amount of moving parts, layers of abstraction, reliance on 2. Communicate to your team. 5. Increase the scope. external systems, and distribution that all result in a stack that few truly fully understand. 3. Run experiments. 6. Automate experiments.
Any developer worth hiring now knows the merits of a thorough testing Early in the lifetime of chaos engineering at Netflix, most engineers regime, but one of the issues with testing is that you are often testing thought chaos engineering was about "breaking things in production," DZONE.COM/GUIDES for predictable outcomes. Despite our "logical systems," show-stopping and it is in part. But while breaking things is great fun, it's not a useful DZONE.COM/GUIDES issues are typically unexpected; situations that no one foresaw. activity unless you learn something from it.
These unforeseen eventualities are what chaos engineering attempts These principles encourage you to introduce real-world events and to account for. It's a reasonably new principle, practiced by Netflix for events you expect to be able to handle. I wonder if fully embracing the several years and then formalized in 2015, setting out its principles in a "chaos" might result in more interesting results, i.e., measuring the time-honored manifesto. worst that could happen. True randomness and extremity could surface even more insightful results and observations. Naturally, there are critics of the practice, and the comments at the bottom
of this TechCrunch article summarize some of them. The typical counter- Let's look at each of these steps in more detail. arguments are that the principle is a band-aid for applications that were poorly planned and architected in the first place, or that it's another buzz- 1. FORM A HYPOTHESIS word-laden excuse to invent shiny new tools that no one knew they needed. To begin, you want to make an educated guess about what will happen in which scenarios. The key word here is "educated;" you need to gather Still, it's proponents are a friendly bunch, so in this article, I summarize data to support the hypothesis that you'll share with your team. my findings on the practice and let you decide.
DECIDE ON YOUR STEADY STATE ORGANIZED CHAOS What is "steady" depends on your application and use case, so decide In many ways, while the term "chaos" is a good eye-catching phrase, on a set of metrics that are important to you and what variance in those it's misleading, summoning images of burning servers and hapless metrics is acceptable. For example: engineers running around an office screaming. A better term is exper- imental engineering, but I agree that is less likely to get tech blog or • When completing checkout, the majority of customers should conference attention. have a successful payment processed.
• Users should experience latency below a particular rate. The core principles of chaos engineering follow similar lines to those you followed in school or university science classes: • A process should complete within a timeframe.
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 20 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM a hypothesisbased onyour steady state, for example: like "let's justbreak things"),it's notaconstructive one.Try to concoct While themosttempting hypothesisis"let's seewhathappens"(much abort atanytime. design each experiment to have minimalimpact andensure you can embarking onchaosengineering,theprinciplesalsorecommend you behind safety-critical systems shouldbeconfident oftheirsetup before Some banks are already following theseprinciples,andwhileengineers argueso they thatrunninginproduction isfundamentally agood thing. possible. Chaosengineeringprinciplesare designedto identifyweakness, in production, orifyou have agenuine reason for notdoingso, asclose The principleswe're working withhere encourage you to undertake tests tion withaproperly pre-tested code base, ittakes onadifferent meaning. untested code base, butaschaosengineeringislikely runincollabora- "Testing inproduction" haslongbeenatongue-in-cheek reference to an mon to mostapplications are: toThe sortsofevents test vary dependingonyour usecase, butcom- andKPIsforSLAs your team orproduct(s). When decidingonthesemetrics, alsoconsider external factors suchas links to find thembelow. There are myriadtools available to help simulate these;I've provided ployments, others by injectingcalls you know willfail, orsimplescripts. The way you introduce your experiments varies, somefrom code de receiving questionsfrom customers. the public,too, suchassupportandcommunity staff whomaystart It's easy to remember engineers, butdon'tforget peoplewhodeal with tion systems, to thenrelevant know. deserve people(ifnoteveryone) to me.Ifyou have ateam ofengineers runningexperiments onproduc As atechnical communicator, thisisperhapsthemostimportant step 2. COMMUNICATE TO YOUR TEAM 3. RUNYOUR EXPERIMENTS INTRODUCE REAL-WORLD EVENTS REAL-WORLD INTRODUCE METRICS RUN IN PRODUCTION • • • • • • • • • • ond longer to complete. If anentire AWS region isunavailable, aprocess willtake 1sec During hightraffic, latency willincrease by 500ms. If PayPal isunavailable, successful payments willdrop by 20percent. Service unavailability Traffic spikes Race conditions issues Dependency Running outofCPU, memory, orstorage space State inconsistency Hardware/VM failure THE DZONE GUIDETO AUTOMATED TESTING: - - - 4. ANALYZE THERESULTS TOOLS ANDRESOURCES Again, theresources sectionbelow listsplaces to findtools to help withthis. change inputparameters for thescope ofyour experiments. factors are introduced into anapplication, butitalsomakes iteasier to as quicklypossible.Thismeans thattheexperiment can runwhennew teams thatfollow continuous deployment) automate your experiments tor theoutcome andabortitifnecessary. Butyou should(especiallywith The first manuallyisfine— you time(s)you can runanexperiment, moni- find outwhere asystem really starts to break down. experimentducing anew orincreasing themetrics ofanexisting oneto issues itidentified,thenmove onto thenext one.Thismaybeintro Once you've learned triedoneexperiment, andpotentially from fixed it, parameters you orthe events test for. ern software besure development, to iterateevents, changing these time to start testing your hypothesis.Muchlike otheraspectsofmod- After defining your ideal metrics andthepotential effects onthem,it's input from anyteams whoseservices were involved intheexperiment. probably already useto helpwiththisstage, butmake sure you involve on whatdata you gathered andto learn There from are it. manytools you There's nopointinrunninganexperiment ifyou don'ttake timeto reflect ment ifneeded,butalsoto analyze results later. Make sure you have alertingandreporting inplace to stop anexperi- insightful debriefingsessionsafter achaosengineering experiment. way) prove your application isasconfident asyou are. Ican imaginemany whereput your money your mouthisand(albeitinacareful andcontrolled application, thenyou shouldn'tfear whatitproposes. force They you to The principlesstate inmanyplaces thatifyou are trulyconfident inyour ones, butrather thatittests andchecksego. engineering, themostinteresting aspectsto meare notthetechnical While engineers anddevelopers are dividedontheusefulness ofchaos QUIETLY CHAOTIC 5. INCREASE THESCOPE 6. AUTOMATE THEEXPERIMENTS podcasts canbefound at gregariousmammal.com. will speakto anyone wholistens. Hisvarious articles and Zone Leader. Heisalways working oncrazy projects and CHRIS WARD isatechnical writer, blogger, andDZone • • • • in thispost form thatcommoditizes manyofthetools andpractices featured dent chaosengineeringresources, runsa'Failure-as-a-service' plat Gremlin, whosestaff are often behindalotofthe Netflix-indepen- The Chaos Community plethora ofuseful tools andresources The comprehensive ChaosEngineering awesome listthatfeatures a The free O'Reilly bookonChaosEngineering YOUR END-TO-END ECOSYSTEM and Meetups, Mailing List, PAGE 21OF44 - - In 2018, 56% of DZone survey respondents reported that they were doing either automatic or manual testing during development. One tactic teams can use to increase testing during development is to implement test-driven development, a strategy that ensures that code can pass unit tests. To demonstrate how adopting test-driven development (TDD) can create better code, let's see how a team might reach some buried treasure when they use TDD and when they don't.
The basic rule of test-driven Because TDD focuses so much on 46% of survey respondents men- 79% of survey respondents report- development is to write code passing tests, it can improve the tion that using tests as documenta- ed that they saw code quality that will pass tests. If the code focus of the development team tion is a key benefit of test-driven improve with test-driven develop- doesn't pass the test, then it is and avoid animosity. If a project development. Reading unit tests is ment, and 62% said they spent not built to solve the intended has very clear acceptance criteria a great way to see how software is less time debugging as a result. By use case or issue. Avoid your (i.e. to pass a test), the team is less intended to be used without using test-driven development, software falling into oblivion likely to be pulled in different having to rely on developers to teams can accomplish their goals by making sure your develop- directions or get distracted with keep documentation up-to-date for quickly, resulting in faster deploys ment is test-driven. different features. 41% of survey every fix. TDD can help you stay with fewer rollbacks because the respondents reported that TDD in the loop so that you avoid software was built to pass auto- improved developer focus. losing track of your ship. mated unit tests.
COPYRIGHT 2018 DZONE.COM from reachingproduction." and preventscriticalerrors is stable, meetsrequirements, needed toensureeveryrelease with thebreadthofcoverage and repeatabletestingprocess us toimplementanautomated "Parasoft SOAtest hasallowed Automated Software Testing
DZONE.COM/GUIDES DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM SPONSORED OPINION PARTNER SPOTLIGHT information chains. chains. information judicial the in necessary is that quality product to commitment strong the compromise to having without functionality new delivering rapidly of level productivity ahigh sustain now they assistance, Parasoft’s With level. quality provable required the of safeguarding effective provided which pipeline, integration continuous their into SOAtest Parasoft integrated team The Parasoft. to turned they DTAP environments, their in later integration continuous the of part as integrated all requirements, non-functional and functional both for verifications and checks automatic maintaining and building required applications stable and reliable delivering quickly that determined Justice and Security of Ministry Netherlands’ the When Solution Testing API with functional,load,performance, andsecuritytesting Bring artificialintelligence into your functionaltest automation to save timeandmoney Parasoft SOAtest the ensure and pyramid testing the of layer each at complexity technical reduce can you tools, automation test Parasoft’s With debug. and maintain to difficult is which testing, UI to-end end- on reliance minimal and tests API/service-level with up backed tests, unit of foundation a with starts This pyramid. testing software a with aligning by scenarios test of portfolio maintainable and ascalable build and automation test of levels greater enable can you environment, your into plug easily that tools automation test Using scenarios. test meaningful maintaining, spent time and creating, of complexity the to due automation test adopt to struggling still is industry software the agile, of speed the at software deliver to continuously automation test your execute to critical it’s Although Testing Strategy Scalable andMaintainable Testing aReality witha Make Continuous CASE STUDY CATEGORY WEBSITE parasoft.com THE DZONE GUIDETO AUTOMATED TESTING: 3x per year per 3x RELEASE SCHEDULE
TWITTER @parasoft • • • • • • • • NOTABLE USERS STRENGTHS CareFirst AT&T Airlines Alaska Easily integrate with service virtualization and test data management. data test and virtualization service with integrate Easily formats. message and protocols 120+ for support with Test broadly scenarios. test complex configure efficiently to intelligence artificial Use security tests. and load/performance to extended be can that tests API scriptless Create end-points. and interfaces multiple across scenarios test end-to-end Automate Continuous Testing areality. Testing Continuous make and automation test of potential the realize customers helps Parasoft ease-of-use, and excellence technical on afocus With feedback ongoing and immediate for continuously executed be can tests • • • • YOUR END-TO-END ECOSYSTEM to the code the to made changes from coming risks understand quickly to ics analyt advanced and visibility complete provide to pyramid testing the down and up from data DTP,Parasoft aggregating continuously executed be can automation test the ensure and vironment en test the within constraints mitigating Virtualize, Parasoft “API use” and design” both “API testing are you ensure and testing API of efficiency increase dramatically to AI leveraging SOAtest, Parasoft 50% to up by cases test JUnit create to developers for time the reducing Jtest, Parasoft VP OFPRODUCTS, PARASOFT WRITTEN BY MARKLAMBERT No OPEN SOURCE? • • DELL Comcast BLOG blog.parasoft.com PAGE 25OF44
- - THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
QUICK VIEW
01. Webhooks are a new API concept that many companies have begun to adapt
02. Write web-service separate Setting Up from the app so you can easily scale.
03. Python is light way and rather easy way to get your web service Automation for running.
04. Create session keys for each automation run so you can easily Webhooks Testing retrieve data. 05. Consider using third party tools if that is going to save you time.
BY SLAVEN SLUGIC AUTOMATION ARCHITECT/MANAGER AT SMARTSHEET
Webhooks are a new API concept that many companies have need to write a web service that will be running in background begun adapting. Unlike an API, which responds when you make independent of your automation. Once it is running, automation a request, Webhooks are triggered when an event occurs in your will subscribe to the Webhooks API by providing the callback URL DZONE.COM/GUIDES application. Automating this process might be a little confusing to of your web service. Once you have the web service in place, your DZONE.COM/GUIDES those unfamiliar with it, so I will provide some high-level informa- automation is free to make changes in your app through the API tion on how to set up automation for this type of testing. or UI and then verify if the information is correct by pulling that data from your web service. HOW IT WORKS From a high-level perspective, Webhooks simply allow applica- tions to provide real-time information as events occur. Imagine From a high-level that you have created a spreadsheet and you would like to be notified any time information is added, updated, or deleted. This perspective, Webhooks is where Webhooks come in handy — they send a callback back to your service that notifies you that a change has been made with simply allow applications to relevant data pertaining to the change. This allows you to receive real-time updates rather than constantly having to make API calls to check if something has been changed. provide real-time information
HOW TO AUTOMATE as events occur. To automate this process, you will need your framework to actively listen for callbacks from the API service so that you can then verify if the data received is correct. In short, every time However, if you plan to have multiple automation runs at the same you run automation, you need to run a web service along with time, then you will need to keep track of that information in your it in order for Webhooks to work. This approach will work — but web service. So, in short, you will need to assign a unique identifier scaling issues will become apparent quickly. In order to deal with to each run. In our automation, we call this the session key; each these issues, you will need to decouple your web service from run needs to create its own session and use that session key to grab your automation framework. What this means is that you will the callback data received from the Webhook API server.
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 26 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM 4. 3. 2. 1. Here isanexample ofhow thismaywork: overview IMPLEMENTATION Here isasampleofweb service definition: the latest callback: something highlevel, likevery thiscould bedoneto retrieve information back to thecallback URLthatyou provided. At a changes andverifying that Webhooksare sendingyou correct At you can thispoint, start testing your application by making this: mysrvcurl/service/sessions/listen?session_id={id}. your callback URL.Your callback URLwilllooksomething like be your APIendpointalongwiththepayload, whichincludes back URLalongwiththesessionkey provided earlier. Thiswill Call your APIto subscribeto Webhooksby providing your call- will return asessionID. Send arequest to your service to create asession.Theservice THE DZONE GUIDETO AUTOMATED TESTING: order for Webhooks to work. a web service alongwithitin automation, you needto run In short,every timeyou run at minimum: is anexample ofalltheendpointsyou mayneedto write At we can thispoint, start defining ouroperations. The following 80 by default soyou can navigate to to localhost:80 access it. Hello Worldservice runninginPython: The following isasamplecode detailing how to quicklyget your within minutes. light-weightvery andeasy to use,andcan spinupaweb service but ours was designedinPythonusingthewebpy module.Itis There are various ways thatthisweb service can beimplemented, To runthiscode, run YOUR END-TO-END ECOSYSTEM ) '/service/sessions/configure','confighook', '/service/sessions/listen','posthook', '/service/sessions/clearall','clearcache', '/service/sessions/getall', 'getallcache', '/service/sessions/getlast', 'getcache', '/service/sessions/create','create', '/service/sessions','sessions', '/','index', urls =( # definitionofavailableendpoints app.run() app=web.application(urls,globals()) if __name__=="__main__": return"HelloWorld" defGET(self): class index: '/','index') urls =( import web python filename.py . Thiswillrunonport PAGE 27 OF44 THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
Once the endpoints are defined, an implementation for each will if result[0]['session_id']==session_id: need to be written. Here is some sample code of what that imple- return json.dumps(result[1], indent=4) mentation may look like: # will return all cached data def getAllCachedData(session_id): class create: datalist = [] def GET(self): for k in cache.keys(): session_id_list = [] result = cache.get(k) session_id = binascii.b2a_hex(os.urandom(15)) if result[0]['session_id'] == session_id: response = {'session_id':session_id} datalist.append(result[1]) session_id_list.append(response)
return session_id As you can see by the definition and examples above, Webhooks can be implemented quite quickly. Webhooks are a newer concept, # post method for webhook server callback; We are but even so, it is a good idea to see if there already exists some- taking the header hook challenge value and returning thing that may fit your needs. Most of what existed at the time it back in our response when our team began using Webhooks was not open-source, so I class posthook: def POST(self): decided to write one for us. A UI can be designed for this, as well, try: allowing your manual testers to have an easy-to-use tool to test session_id = web.input().session_ Webhooks for you, saving everyone time. id.strip() except: return {'Error':'session_id required'}
shc = web.ctx.env.get('HTTP_WEB_HOOK_ Webhooks are a newer CHALLENGE') challenge_key = shc concept, but even so, it is
if(configCache and session_id in configCache): a good idea to see if there flag = configCache.get(session_id) ['setHeader'] already exists something that DZONE.COM/GUIDES body_flag = configCache.get(session_id) DZONE.COM/GUIDES ['setBody'] if(flag!=False): may fit your needs. web.header("Web-Hook-Response", shc)
if(body_flag==False): OTHER CONSIDERATIONS challenge_key=None SSL will be required for this to work and you may not have a else: certificate right away. One way to get around this to use service web.header("Web-Hook-Response", shc) like NGROK. The NGROK service allows you to create a secure URL
payload = web.data() connection to your localhost server and can be used until you get your own certificate. return setData(payload,challenge_key,session_id) If you are planning to have a large amount of concurrent // get last payload automation runs, then you will need to configure your webpy def getLastCachedPayload(session_id): service to handle load more efficiently. There are few ways to cLen = len(cache) do this, such as running it behind NGINX or using Lighthttpd. result = cache.get(cLen-1) Lighthttpd is pretty easy to configure and you can find
if (result == None): instructions for configuring it here. return {'info': 'cache is empty. try to post some payload first'} SLAVEN SLUGIC is an engineer and manager else: working at Smartsheet. He has been in the industry
while cLen > 0: for about 12 years and has worked for companies like cLen = cLen - 1 Microsoft, Expedia, and Walt Disney. Slaven graduated result = cache.get(cLen) from Western Washington University in 2006 with a Computer Science Degree. code continued on next column
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 28 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM thousands ofrepetitive tasks. to doother, more important thingswhileletting thecomputer quickly run Learn how to upyour skillsinautomated testing sothatyou're freed up The Way of theWebTester by-step continuous improvement tutorial, andmore. creating anagiletest suite, improve your programming skillswithastep- Learn how to keep upwiththechangingpace ofyour web application by Selenium DesignPatterns andBestPractices optimal automation. become anecessity. Checkoutthisbookto learn how to achieve Software test automationto hasmoved aluxury beyond Experiences of Test Automation books twitter INTO deeper diving @AutomatedTester @martinfowler @RealGeneKim @eviltester @jcolantonio AUTOMATED TESTING
THE DZONE GUIDETO AUTOMATED TESTING:
@wakaleo @Nikolay_A00 @jezhumble @TourDeDave @angelovstanton
neural networks. applications ofchatbots,deeplearning, knowledge engineering,and Computing. TheAIZone goes thebuzzandprovides beyond practical to MachineLearning, Natural Language Processing, andCognitive The ArtificialIntelligence (AI)Zone features allaspectsofAIpertaining Performance essential hubfor Scrum,XP, Kanban, Lean Startup, andmore. sector.older stylesofworkflow inalmostevery TheAgileZone isyour In thesoftware development world, Agilemethodology has overthrown Agile and muchmore andresourcesnews aboutContinuousDelivery, Puppet,Chef, Jenkins, of developers andIToperations. Zone TheDevOps isyour hotspotfor aimed atencouraging closecooperation teams withincross-disciplinary isaculturalDevOps supported movement, by exciting tools, new thatis DevOps focus onsoftware andqualityassurance. development Get onthecourse to becoming anexpert automation test engineerwitha Automated Testing EngineerMaster's Program Peruse themhere. There are thousandsofcourses related to automated testing onUdemy. Automated Testing Courses onUdemy actionable, free Selenium tutorial. one-week Learn how to start offontheright foot with Seleniuminthissimple, Selenium Bootcamp streamline your technology value stream. out alean software process, development includingbestpractices to This Refcard fillsthegap by covering astep-by-step approach to carrying Lean Software Development utilizing thisconcept. elements ofContinuousTesting methodology, andthebenefitsof aside andlearn exactly whatContinuousTesting is,the10key These concepts are certainly connected. Let’s pushthemisconceptions Continuous Testing explore themanybenefits,andfindreal-world usecases. Download Refcard thisnew to get started withtest designautomation, Test DesignAutomation courses refcardz zones YOUR END-TO-END ECOSYSTEM DZONE.COM/AGILE DZONE.COM/DEVOPS DZONE.COM/PERFORMANCE
PAGE 29OF44 Software TestAutomation The tricentis.com/GartnerMQ Download yourcopynow! ABILITY TO EXECUTE F C Sour igure 1. OMPLETENESS OF ce: Gartner(November2017) MagicQuadrantforSof
Gartner MagicQuadrant NICHE PLA CHALLENGERS VISION Ranorex Pa YERS rasof t tware Microsof SmartBear Te IBM st t Automation As of November2017 T estplant Micro F VISIONARIES LEADERS ocus CA T echnologies T ricentis © Gartner , Inc DevOps reinvented for Software testing
is here!
for
DZONE.COM/GUIDES DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM SPONSORED OPINION PARTNER SPOTLIGHT with significant savings in hours per testing cycle. testing per hours in savings significant with ROI proved Dolby at team TCoE the pilot, initial their during tickets support and incidents of number the in drop aconsistent with Along deliverables. testing its of all for process maintenance arobust and approach astandardized developed quickly Dolby automation, to test approach Tricentis Tosca’s model-based adopting innovative, Since applications. packaged SAP in footprint large their supported that solution automation a test needed also Dolby cadences. release increasing continuously within coverage risk end-to-end providing while cases test to reduce Testing platform Continuous Tricentis Tosca’s end-to-end selected Labs Dolby requirement, akey as approach arisk-based With Testing Platform Continuous reinventing software testing for DevOps. With theindustry’s No. 1ContinuousTesting platform, Tricentis isrecognized for Tricentis Tosca need to: We sufficient. not it’s but essential, is Test automation testing? transform to needed What’s software. innovative of delivery the to accelerate opportunity untapped an is testing Reinventing But it’s notSufficient is Essential… Test Automation CASE STUDY CATEGORY 1. “maintenance trap” created by flaky, brittle tests. tests. brittle flaky, by created trap” “maintenance the them—escaping to maintain way asustainable need also we tests; automated to create ways simpler and faster need only not we To there, get quality@speed. for required improve significantly in order to provide feedback the fast must This enterprises). large for lower (and even 20% adismal around hovering is rate automation test average industry The automation. to test barriers the through Break WEBSITE tricentis.com THE DZONE GUIDETO AUTOMATED TESTING:
Quarterly Release Quarterly RELEASE SCHEDULE
TWITTER @tricentis 3. 2. YOUR END-TO-END ECOSYSTEM well as technologies. as well as transformation people across and deeper processes a requires It tools. different or tools, more of a matter simply not it’s Furthermore, iteration. each in early issues block” “big to expose testing exploratory performing and Integration, Continuous for automation test to stabilize management data test stateful and virtualization service applying risk, business your with testing aligning as such practices involves Testing also Continuous automation. test than more Testing is Continuous that Recognize testing. performance and load as such practices to include testing functional beyond extend also must it technologies—and these of all embrace must Test automation UIs. mobile and to web to APIs, applications, packaged other and SAP from ranging technologies may involve transaction end-to-end a single Today, level. UI web the at testing functional than more is Test automation automation. test of definition our Expand • • • • • • • • • No NOTABLE USERS OPEN SOURCE? STRENGTHS Vantiv (now WorldPay) Vantiv Merck HSBC Experian Allianz candidates release your of risk the into insight clear Gain automation test model-based with maintainability and reuse Maximize test under application any with synchronize that cases test dynamic Create to minutes weeks from testing regression Reduce FOUNDER ANDCHIEFSTRATEGY OFFICER,TRICENTIS WRITTEN BY WOLFGANG PLATZ BLOG tricentis.com/blog PAGE 31OF44 THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
QUICK VIEW
01. End-to-end business process tests ensure not just that your software works but also that the End-to-End Tests: business works. 02. End-to-end tests are not just a series of functional and unit tests The Pinnacle in strung together. 03. Automating end-to-end tests comes with unique challenges that make them especially difficult to Test Automation maintain and scale.
BY CHRIS KRAUS SR. DIRECTOR PRODUCT MANAGEMENT, WORKSOFT
End-to-end testing can mean different things to different people. Re- UIs gardless of the definition, end-to-end tests are more complex, harder Presentation Layer to automate, and many times carry more risks than functional and unit
APIs / tests. Functional and unit tests focus on whether or not a specific fea- SOAP Calls Service Layer ture of a single application works. End-to-end tests ensure a request or ESB Integration DZONE.COM/GUIDES
transaction can flow across an entire system of technologies and appli- Layer DZONE.COM/GUIDES cations. Failure of an order-to-cash process or payroll process not only Enterprise Apps represents a software failure but also a business failure. As companies Data/APP Layer move to adopt Agile across all of their applications and quickly roll out changes across all of their systems, automating end-to-end tests Figure 1: "Vertical" End-to-End Integration Test becomes critical for reducing risks and accelerating change.
WHAT IS AN END-TO-END TEST? Depending on who you ask, end-to-end testing can mean different UIs things. In the custom app development space, end-to-end testing is Presentation Layer often used to describe an integration test. In this case, a "vertical" test Figure 2: "Horizontal" End-to-End Business Process Test that may start at the presentation/services level is used to test the back-end responses from the System Under Test (SUT). The test runs This is a similar debate to test-driven development (TDD) vs. behav- across the various layers used by the SUT. ior-driven development (BDD). TDD execution is writing a failing test for one very specific bit of functionality, generating a unit or "vertical" Business process testing is used to test enterprise business processes test. An end-to-end test or "horizontal" test is similar to BDD when that include multiple steps and, many times, across multiple applica- looking at multiple features working together to complete a workflow tions. In this case, the test is usually driven at the UI layer and is used or epic in Agile terms. to validate that the end-to-end business process works across systems as a "horizontal" test. Unlike vertical tests that measure performance It is also important to note that end-to-end tests are not just several up and down the layers of the technology stack, horizontal end-to- unit and functional tests strung together. The scope of what is being end tests are used to ensure that mission-critical business processes tested and the goal of the tests are considerably different between work. Tests are run at the UI layer and measure whether or not the test functional tests and end-to-end tests. Key differences between func- proceeds to the next step in the process. tional and end-to-end tests include:
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 32 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM ACCESS acts like auserneedsto act like areal userby logginginto thedesktop just spinupapersonal instance. Another bigchallenge isautomation that times inpre-production where thesystems are shared andtesters can't minutes, tests for enterprise applications like SAPneedto berunmany Unlike custom appvirtual devtest environments thatcan bespunupin needs andscale testing to keep pace withupdates. weresolution (WorksoftCertify),they able to meet theirautomation use andease-of-maintenance intheirselectionprocess. Withthenew to continue doingthings manually. focused Thistime,they onease-of- werethey lookingatathree-month patch and there cycle was noway forced Hondato revisit theidea oftest automation. WithSAPHANA, things manually. quicklyabandoned butamove They to it, SAPHANA to create andmaintain theautomation thanitdidto continue to test automation solutionthatwas script-based andfound ittook more time automating end-to-end businessprocess tests. purchased They atest Americas isaperfect example ofchallenges clientshave hadwith A recent case study publishedinSAPInsiderregarding HondaR&D tests ortoo muchtimeto maintain them. abandon theirefforts because iteither takes too long to create the have triedautomation solutionsfor end-to-end testing have hadto does doingthingsmanually. Unfortunately, manyorganizations that it needsto take lesstimeto create, maintain, andruntests thanit the business.Butto besuccessful withend-to-end test automation, Most agree thatimplementingatest automation solution isgood for these challenges have andhow overcome they them. automate, includingacouple ofexamples ofcustomers whohave had Included below are somekey reasons end-to-end tests are hard to scenarios are executed withtest tools. only16percent2017-18 WorldQualityReport, ofend-to-end business mation for anend-to-end test isachallenge. Infact, according to the multiple applications andUIs.For thisreason, justbuildingtheauto dreds —ifnotthousandsoftests thathave to bemanaged across about testing justlike auser, whichmeans there are going to behun- When we talk aboutend-to-end businessprocess testing, we're talking • • WHY AREEND-TO-END TESTS SOHARDTO SCALE? MAINTENANCE End-to-End Tests: Functional Tests: – – – – – – – – – – – – – – – – Testing ensures aprocess continues to work after achange ismade. Validate thateach step intheprocess iscompleted. Test theway multipleusers work across applications. Testing crosses multipleapplications andusergroups. Validate theresult ofeach test for inputsandoutputs. Test theway asingleuserengages withtheapplication. Software is tested to ensure itmeets acceptance criteria. Testing islimited to asinglepiece ofcode orapplication. THE DZONE GUIDETO AUTOMATED TESTING: - defects faster, andbetter meet theneedsof thebusiness. tion faster, were they ableto spendmore timeplanning,detect more automation solutionthatenabledthemto create and runautoma- being ableto orchestrate andruntests atscale. By implementingan business analystsandtesters to better collaborate together alongwith everyone's job. neededto They change theway work andenable they realized thatinorder to besuccessful, automaton neededto become become more Agileandto better supportchanges to SAP. quickly They Cardinal Health recently undertook anautomation effort inaneffort to a given testing window. in parallel across more resources can sothatthey becompleted within Test automation needsto begenerated faster andtests needto berun testing window. Automation becomes acritical part oftheprocess. and more tests have to bedocumented, created, andruninthesame time achange ismade.Aschanges happenmore frequently, more don't justwant oneworkflow tested. want They themalltested each When rolling outachange to amission-critical system, businesses or closelyscrutinized. testing window, especiallyinglobal situations, isoften strictlylimited solution willnotvalidate multiple features are working together. The thousand tests andlet meknow whenthey're done," withouttheright and diverse testing resources. Theabilityto justsay, "Gorunthese quence withcomplex dependencies.It's difficult to manage distributed actually runinproduction. Thetests often need to bescheduledinse process therequest inproduction to validate whether ornotitwill The tests needto runinthesameorder asthereal system would You mustbeableto orchestrate theworkflow ofanend-to-end test. interrupting tests. for thelife ofthetest andprevent thingslike Windows updates from plicated, you alsoneedto figure outaway to keep themachineawake into remote machinesto simulate real users. To make thingsmore com- and runningapplications. Thismeans installing local agents andlogging moving from cowboy to Scrum-basedsprinting. was alsoanearlyadopter ofAgile, working with multipleagileteams architected anddeveloped. Asaproduct manager, he space hashelpedtransform theway testing tools are years ofsoftware experience. His visioninthetesting ment team at Worksoft andbringswithhimover 25+ CHRIS KRAUS currently leadsthe Product Manage - scale inorder to ensure theirlong-term success withtest automation. must selecttools thatare easy to use,easy to maintain, andeasy to end-to-end testing to meet theneedsoftheirbusiness.Companies requires areal machine.ButITmustevolve test automation to include thousands ofsteps thatneedto beorchestrated anddriven viaaUIthat wide variety ofsystems andapplications andcan includehundreds to End-to-end businessprocess tests are complex by nature. cover They a ORCHESTRATION CONCLUSION SCALE YOUR END-TO-END ECOSYSTEM PAGE 33OF44 - 5017CAT_8.5x11.indd 1 automatically, andcontinuously. practices to test early, often, Adopt next-generation testing your software delivery lifecycle. Enable continuoustesting across Continuous Testing Enable your DevOps andcontinuousdelivery practices today. and builduponend-to-end integrations andopensource. Only CAoffers acontinuous testing strategy automated that’s Explore ca.com/continuous-testing 8/3/17 8:13 PM
DZONE.COM/GUIDES 5017CAT_8.5x11.indd 1 Enable your DevOps andcontinuousdelivery practices today. and builduponend-to-end integrations andopensource. Only CAoffers acontinuous testing strategy automated that’s automatically, andcontinuously. practices to test early, often, Adopt next-generation testing your software delivery lifecycle. Enable continuoustesting across Continuous Testing Explore ca.com/continuous-testing 8/3/17 8:13 PM
DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM SPONSORED OPINION PARTNER SPOTLIGHT WEBSITE ca.com/us/products/ca-agile-requirements-designer.html See their story here story their See less cost. for service best the providing of goals business its achieve company the help will and projects, business new for resources up frees which delivery, of software quality and speed the improved has Williams releaseend-to-end management. customization, education, and implementation to provide support provided also Services CA testing. software improve and automate to solutions Technologies CA of asuite implemented Williams this. achieve to ability its hampering were processes testing Manual defects. fewer with and quickly more updates and applications quality higher deliver to needed Williams demands, business with To pace keep US. the across operations through people 5,600 employs it Oklahoma, Tulsa, in Based transportation. and processing gas natural providing company, infrastructure energy 500 aFortune is Williams Testing Continuous CA Agile Requirements Designer tion effectiveness? automa test drive MBT does ways what in becomes, then question The costs. lower at faster delivered are apps high-quality your so, And dramatically. time testing shortening activities, testing automate to you enables It requirements. poor from stem that defects costly avoid you helps (MBT) testing Model-based requirements. from right scripts automation test and data, test cases, test like assets test reusable generate automatically to teams test –enabling automation test towards gravitate they ods, meth development agile test-driven, adopt organizations more as But continuous testing model. agile, today’s with up keep to –unable quality and speed to barrier a remains testing legacy SDLC, the in bottleneck Amajor manually. performed still is testing of 70% years, 30 after that believe to hard It’s Speed ofAgile Drive Testing at the Improve software quality, reduce testing costs, andspeedupapplication delivery. CASE STUDY CATEGORY . THE DZONE GUIDETO AUTOMATED TESTING: Continuous RELEASE SCHEDULE - - • • • • • No • • • OPEN SOURCE? STRENGTHS NOTABLE USERS change and update test cases in minutes. minutes. in cases test update and change a of impact the identify Automatically requirements. changing Manage percent. 30 by cycles test shorten and duplicates, any remove cases, test Import Optimization. Test Case sprint. every in delivered being functionalities all for data test and scripts, automation test cases, test generate Automatically Testing. Agile Enable results. expected and data right the to linked scripts, automation test and coverage functional percent 100 for needed cases test of number smallest the generate Automatically Automation. Test Design percent. 56 by defects software and percent 90 by ambiguity requirement reduce and flowchart, active avisual, to requirements Map Engineering. Requirements GM Financial GM Citigroup Williams speed of agile, enabling you to build better apps, faster. faster. apps, better build to you enabling agile, of speed automation and optimization. These capabilities help you test at the design test through engineering requirements –from activities testing difficult most the automates that solutions comprehensive offers CA artifacts. automation test as rerun or run be can that assets test reusable of alibrary up building –while coverage test maintain to tests repair or create and analyses impact initiate automatically they occur changes as because change, requirements when tests fixing and finding manually on wasted time the cuts This process.’ requirements in ‘change the automate Finally, tests. of set smallest the with coverage maximum at functionality the testing and created, are stories user as flowchart the from right automatically, scripts test and data, test cases, test creating means This automatically. cases test of sets optimized generate Next, communication. and collaboration boosting –while requirements unclear eliminates a flowchart using modeling requirements, poorly-defined from coming defects of 64% With methods. text-based inefficient in them writing versus flowchart, active an as requirements modeling with Start YOUR END-TO-END ECOSYSTEM TWITTER @CAinc PRINCIPAL PRODUCTMARKETINGMANAGER, CATECHNOLOGIES WRITTEN BY GEDEONHOMBREBUENO • • Sprint Level 3 Communications BLOG ca.com/en/blog-highlight PAGE 35 OF44 THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
QUICK VIEW
01. The keys to automated testing are to save time and improve quality while realizing the goals Executive Insights on and objectives of the organization. 02. Increased adoption of automated testing is attributed to the maturation of Agile Automated Testing methodologies and adoption of the DevOps culture.
03. The ROI of automated testing is three-fold: time saved, fewer defects, and greater customer BY TOM SMITH satisfaction/retention. RESEARCH ANALYST AT DZONE
• Gil Sever, CEO, Applitools • Matt Ward, Senior Engineer, Mux
• James Lamberti, Chief Marketing Officer,Applitools • Tom Joyce, CEO, Pensa
• Shailesh Rao, COO, BrowserStack • Roi Carmel, Chief Marketing & Corporate Strategy Officer, Perfecto Mobile • Kalpesh Doshi, Senior Product Manager, BrowserStack • Amit Bareket, CEO and Co-founder, Perimeter 81 • Aruna Ravichandran, V.P. DevOps Products & Solutions Marketing, CA Technologies • Jeff Keyes, Director of Product Marketing, Plutora
• Pete Chestna, Director of Developer Engagement, CA Veracode • Bob Davis, Chief Marketing Officer,Plutora
• Julian Dunn, Director of Product Marketing, Chef • Christoph Preschern, Managing Director, Ranorex DZONE.COM/GUIDES DZONE.COM/GUIDES • Isa Vilacides, Quality Engineering Manager, CloudBees • Derek Choy, CIO, Rainforest QA
• Antony Edwards, CTO, Eggplant • Lubos Parobek, Vice President of Product, Sauce Labs
• Anders Wallgren, CTO, Electric Cloud • Walter O'Brien, CEO and Founder, Scorpion Computer Services
• Kevin Fealey, Senior Manager Application Security, EY • Dr. Scott Clark, CEO and Co-founder, SigOpt Cybersecurity • Prashant Mohan, Product Manager, SmartBear • Hameetha Ahamed, Quality Assurance Manager, FileCloud • Sarah Lahav, CEO, SysAid Technologies • Amar Kanagaraj, CMO, FileCloud • Wayne Ariola, CMO, Tricentis • Charles Kendrick, CTO, Isomorphic • Eric Sheridan, Chief Scientist, WhiteHat Security • Adam Zimman, VP Product, LaunchDarkly • Roman Shaposhnik, Co-founder V.P. Product & Strategy, Zededa • Jon Dahl, CEO and Co-founder, Mux
1. The keys to automated testing are to save time and improve The organization must understand its goals. Planning is quality while realizing the goals and objectives of the important. Many organizations are facing the traditional issue of organization. Value drivers are quality and velocity, and they “we’re too busy to improve, we don’t have time to think about need to be prioritized first. 70% of the time, clients are looking test automation, but we know we are going to fail if we don’t have for both and it can take multiple meetings to decide what to it in place soon.” They need to understand the end goal of doing prioritize. Automation helps customers with a quality assurance this kind of testing. process that runs in minutes and hours versus days and weeks, all while removing errors. There are four key factors of success: 1) level of automation
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 36 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM automated testing isthecorporate culture usedto doing 5. Themostcommon issues affecting theimplementation of simply runningbetter, andthecustomers are happier. a direct correlation withcustomer satisfaction. Theproduct is not havingto lookfor errors. For us,automated testing hashad more accurately saves amilliondollars peryear indevelopers production thanearlier intheSDLC. Catching errors earlier and speaking, ifyou ittakes findadefect, 10timeslonger to fixin to daysandto minutes can yield 20x savings.Conservatively spreading thework across different machines.Goingfrom weeks Organizations can see 20to timeimprovements 40xcycle by defects, andgreater customer satisfaction andretention. 4. TheROIofautomated testing isthree-fold: timesaved, fewer Federal Government. – brokerage andalgorithmic trading; 5)ecommerce; and6)the patient monitoring; 3)telecommunications; 4)financialservices cars; 2)healthcare –pharmaceuticals, medical devices, and automated testing are: 1)automotive, especiallyautomated The industriesmostfrequently mentionedasbenefittingfrom the digital transformation withmicroservices. continuous (CD)andishelpinglegacy delivery enterprises make is helpingorganizations continuous achieve integration (CI)/ automated tests can 100%coverage. achieve Automated testing Manual testing can 6%code achieve coverage, whilewell-written resources (i.e. manualtesters); full and6)achieving coverage. 3) consistency and accuracy; 5)reducing 4)savingmoney; 24/7; 2)reporting withdailyinsightsandaccurate feedback; invaluable for: 1)savingtimeby runningtests automatically testing beingseeninsixkey industries.Automated testing is 3. Thereare so tests can bebuiltquickly. are easier to work with.Enablementtools integrate automatically, changes. Also, thecontinued oftools evolution andframeworks increases, automation istheonlyway to keep upwithallofthe Organizationsfrom development. are realizing ascomplexity culture. teams Agile-driven are nolonger separating automation maturation ofAgilemethodologies andadoptionoftheDevOps year hasbeenincreased adoption.Thisisattributed to the 2. Themostsignificant change to automated testing inthepast Testing needsto shiftleftat conceptualization. be ableto testing doend-to-end DevOps throughout theSDLC. and platform for testing across alldevices. Organizations needto feedback loopwithdata; and4)adoptinganintegrated approach flows, alldatasets, qualityofappandUX; 3)creating anefficient (must be90%+for CI/CDto succeed); 2)maximize coverage ofall more thanadozen benefits to automated THE DZONE GUIDETO AUTOMATED TESTING: or working outat thegym. his spare time, you canfindhimeithereating at Chipotle sharing information ofvalue to helppeoplesucceed. In and qualitative—to drive business results. Hispassion is at gathering insightsfrom analytics—bothquantitative TOM SMITHisaResearch Analyst at DZone whoexcels earlier intheSDLC. methodology from theplanningphase,andbuildrichercode system to betested from thebeginning. Leverage thetesting Get ridofsimple, recurring problems by architecting the it’s doing,andhow thatimpacts APIsandservices. to address.trying Have anoverarching oftheapplication, view what Understand users, are, domains,whothey andwhatproblem they’re Think abouttheusecases first andwhatthepurposeofcode is. write amodeltypeofcode from mentors andprevious products. tests. Know how automated tests are going to bewritten. Learn to the SDLC. Have theabilityto write your own unitandregression different typesof tests you willneed to runatdifferent pointsin Hone test scriptingskills.Write small,simpletests. Recognize the the SDLC. the application; and,3)move left, beginning testing earlier in are: code andapplications perform well withautomated testing willneedto skillsadeveloper 7. Thethree ensure primary their real-time. thereby providing developers accurate vulnerability data innear ML modelsto automate thevulnerability verification process, teams are ableto leverage historical vulnerability data to train ML willalsoimprove automated security testing, assecurity of timeto ensure your code isalways vulnerability anddefect-free. be abletotest useAI/MLtoimaginableintheleast runevery amount and willresult inself-healing tests andself-remediating code. You will models smarter. WithAI/ML,testing willbefaster, more thorough, of thesolutionasteams generate more data whichwillmake the as writingcode whichwillbedoneby machines.AI/MLwillbepart to buildpredictable modelsandpatterns. Testing willbeanatural the process onseveral fronts. Testing willbedesignedfor AI/ML 6. processdelivery to share thefeeback from thedifferent tools. testing andAI.Weneedonecomprehensive, automated, visible testers are notadaptingto, let aloneembracing, automated todevelopers shiftleftand need to learn towrite tests.Oldstyle management doesn’twant to ruffle feathers. Companies need people needto beretrained and turned into programmers —but manual testing. Where manualprocesses have beenused, The future ofautomated testing istheuseofAI/MLto improve YOUR END-TO-END ECOSYSTEM 1) test scriptingskills;2)understanding theusecase of PAGE 37 OF44
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
Solutions Directory
This directory of automated testing frameworks, platforms, and services provides comprehensive, factual comparisons of data gathered from third-party sources and the tool creators’ organizations. Solutions in the directory are selected based on several impartial criteria, including solution maturity, technical innovativeness, relevance, and data availability.
COMPANY PRODUCT PRODUCT TYPE FREE TRIAL WEBSITE
Apache Software Java functional & performance JMeter Open source jmeter.apache.org Foundation testing
Apica Apica Load Test Load testing 30 days apicasystems.com
AI-powered automated visual Available by Applitools Applitools applitools.com testing request
AI-driven, unified test Demo available by Appvance Appvance IQ appvance.com/appvance-iq automation system request
Continuous integration, Atlassian Bamboo continuous delivery, & 30 days atlassian.com/software/bamboo DZONE.COM/GUIDES automated testing DZONE.COM/GUIDES
Automated mobile & web Available by BrowserStack BrowserStack browserstack.com testing request
Buildbot Buildbot Continuous integration Open source buildbot.net
Performance & load testing for CA Technologies BlazeMeter Free tier available blazemeter.com JMeter
CasperJS CasperJS JavaScript navigation testing Open source casperjs.org
Continuous integration & CircleCI CircleCI Free tier available circleci.com continuous delivery
Continuous integration & Cloudbees Jenkins Open source jenkins.io continuous delivery
Acceptance & regression Available by CollabNet VersionOne collab.net testing request
Cucumber Cucumber Automated rails testing Open source cucumber.io
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 38 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM Micro Focus Micro Focus mabl Loadster JUnit JS Foundation JetBrains IBM IBM IBM Hiptest Gridlastic Functionize FitNesse eureQa Eggplant DevExpress COMPANY IBM Rational Functional IBM Rational Functional Performance Tester Performance Urbancode Build Urbancode THE DZONE GUIDETO AUTOMATED TESTING: Enterprise (QC) Enterprise Quality Center Center Quality IBM Rational Rational IBM LoadRunner Functionize PRODUCT Gridlastic TeamCity Eggplant Loadster TestCafe FitNesse Appium eureQa Hiptest JUnit 5 JUnit Tester mabl Performance & load testing &load Performance regression testing regression & functional Automated Selenium-as-a-Service testing framework &mobile web Automated Load & performance testing &performance Load automation intelligence automation &digital management performance Digital Acceptance testing framework testing Acceptance Web app load testing load app Web ML-driven automation testing application release automation release application Continuous integration & Automated web testing web Automated platform &management Test planning Unit testing framework testing Unit security, & performance testing &performance security, load, functional, Automated Load testing Load Functional web testing web Functional management Continuous integration, build PRODUCT TYPE YOUR END-TO-END ECOSYSTEM Demo available by by available Demo Demo available by by available Demo Free tier available tier Free Free tier available tier Free Free tier available tier Free Free solution FREE TRIAL FREE Open sourceOpen Open sourceOpen Open sourceOpen Available by by Available Available by by Available Available by by Available Available by by Available Available by by Available Available by by Available request request request request request request request request 30 days 30 30 days 30 hiptest.net gridlastic.com functionize.com fitnesse.org sayeureqa.com eggplant.io testcafe.devexpress.com management/overview products/quality-center-quality- software.microfocus.com/en-us/ testing/overview products/loadrunner-load- software.microfocus.com/en-us/ mabl.com/features loadsterperformance.com junit.org/junit5 appium.io jetbrains.com/teamcity products/urbancode-build developer.ibm.com/urbancode/ rational-performance-tester ibm.com/us-en/marketplace/ rational-functional-tester ibm.com/us-en/marketplace/ WEBSITE PAGE 39 OF44 THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
COMPANY PRODUCT PRODUCT TYPE FREE TRIAL WEBSITE
Automated mobile & web microfocus.com/products/silk- Micro Focus SilkTest 45 days testing portfolio/silk-test
software.microfocus.com/en-us/ Unified Functional Micro Focus Automated functional testing 60 days products/unified-functional- Testing (UFT) automated-testing/overview
Visual Studio Test Test case management, visualstudio.com/vs/test- Microsoft N/A Professional exploratory, & browser testing professional
Automated mobile & web Available by Mobile Labs Mobile Labs mobilelabsinc.com testing request
Automated performance Neotys NeoLoad Free tier available neotys.com/neoload/overview testing
NUnit NUnit .NET unit testing framework Open source nunit.org
Panaya Panaya Test Center Continuous testing 14 days panaya.com
Available By Parasoft Parasoft C/C++ Test C and C++ development testing parasoft.com/products/ctest Request
Parasoft Continuous Browser-based continuous Available By parasoft.com/products/ Parasoft Testing Platform testing tool Request continuous-testing-platform DZONE.COM/GUIDES DZONE.COM/GUIDES
Available by Parasoft Parasoft dotTEST Test automation platform parasoft.com/product/dottest request
Available By Parasoft Parasoft JTest Java static analysis and testing parasoft.com/products/jtest Request
Available By Parasoft Parasoft SOAtest API testing platform parasoft.com/products/soatest Request
Service virtualization and Trial Available By parasoft.com/products/ Parasoft Parasoft Virtualize virtual test environments Request virtualize
Automated web & mobile Available by perfectomobile.com/solutions/ Perfecto Mobile CQ Lab testing request perfecto-test-automation
PHPUnit PHPUnit PHP unit testing framework Open source phpunit.de
Release & test management Demo available by Plutora Plutora plutora.com platform request
Available by Practitest Practitest QA & test management practitest.com request
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 40 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM SmartBear Software SmartBear Software SmartBear Software SmartBear Sencha Selenium Sealights Screenster Sauce Labs Sahi Robustest Robotium Ranorex Rainforest Radview QASymphony Progress Software COMPANY Telerik Test Studio Test Telerik THE DZONE GUIDETO AUTOMATED TESTING: qTest Platform TestComplete Rainforest QA Sencha Test Sencha Sauce Labs Sauce PRODUCT Screenster Robustest Robotium WebLOAD Selenium Sealights Sahi Pro Sahi Ranorex SoapUI LoadUI Automated mobile testing mobile Automated Android UI testing Android Automated UI testing UI Automated Automated web & API testing &API web Automated testing GUI Automated Automated web testing web Automated Automated web testing web Automated Load testing Load testing framework &mobile web Automated Continuous testing Automated Ext JStesting Ext Automated Continuous testing testing &web mobile Automated API load testing load API Visual UI testing UI Visual load, & API testing & API load, performance, GUI, Automated PRODUCT TYPE YOUR END-TO-END ECOSYSTEM Demo available by by available Demo Demo available by by available Demo Demo available by by available Demo Free tier available tier Free Free tier available tier Free version available version Open-source Open-source FREE TRIAL FREE Open sourceOpen Open sourceOpen Available by by Available Available by by Available Available by by Available request request request request request request 30 days 30 30 days 30 30 days 30 14 days 14 14 days 14 robotium github.com/RobotiumTech/ ranorex.com rainforestqa.com radview.com testing-tools qasymphony.com/software- telerik.com/teststudio testcomplete/overview smartbear.com/product/ soapui.org pro.html soapui.org/professional/loadui- sencha.com/products/test seleniumhq.org sealights.io screenster.io saucelabs.com sahipro.com robustest.com WEBSITE PAGE 41 OF44 THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM
COMPANY PRODUCT PRODUCT TYPE FREE TRIAL WEBSITE
Automated mobile & web SmartBear Software Cross Browser Testing 7 days crossbrowsertesting.com testing
Continuous integration & Solano Labs Solano Labs 14 days solanolabs.com deployment
Test Anywhere Test Anywhere Frontend web testing Free tier available testanywhere.co
Regression and continuous Available by Testcraft Testcraft testcraft.io testing request
Available by Testim Automate Automated regression testing testim.io request
TestingBot TestingBot Automated web testing 14 days testingbot.com
Automated mobile & web Demo available by Testlio Testlio testlio.com testing request
TestNG TestNG Unit testing framework Open source testng.org
Continuous integration & Open-source ThoughtWorks GoCD gocd.org continuous delivery version available
Travis CI Travis CI Continuous integration Free solution travis-ci.org DZONE.COM/GUIDES DZONE.COM/GUIDES
tricentis.com/software-testing- Tricentis Tosca Testsuite Continuous testing platform 14 days tools
Automated mobile & web Demo available by Turnkey Solutions cFactory turnkeysolutions.com testing request
Ubertesters Ubertesters Automated mobile testing 14 days ubertesters.com
Watir Watir Automated web testing Open source watir.com
Windmill Testing Windmill Automated web testing Open source github.com/windmill Framework
Automated mobile & web Demo available by Worksoft Worksoft worksoft.com testing request
xUnit xUnit.net Unit testing framework Open source xunit.github.io
Zephyr Zephyr Real-time test management 7 days getzephyr.com
THE DZONE GUIDE TO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM PAGE 42 OF 44 DZONE.COM/GUIDES THE DZONE GUIDETO AUTOMATED TESTING: YOUR END-TO-END ECOSYSTEM the clientandmanaged by thecloudprovider. resource limitations shouldbeabstracted from becomethat they more run; resilient withevery executing complex tests inamatter ofminutes so CONTINUOUS TESTING: CONTINUOUS DELIVERY: CONTAINERS: CLOUD-SCALE EXECUTION: CALLBACK URL: ming language primarilyby developed Google. ANDROID: AUTONOMOUS TEST CREATION: (AMAZON EC2): AMAZON ELASTIC COMPUTE CLOUD AGILE: GLOSSARY the qualityofacode build. environments to obtain immediate feedback on of thesoftware pipelineacross all delivery executing unattended automated tests aspart requirement sets. virtualization andsufficient for manyisolation access. Muchlighter-weight thanmachine-level memory, I/Orate, root privileges, andnetwork often includefile system, diskquota, CPUand elements vary by containerization strategy and UNIX-based systems) inuserspace. Isolated OS (rather usually(in thanmachine)level, repeatedly withminimalhumanintervention. anddeployeddeveloped rapidly, reliably, and deployment capabilities allow software to be integration, automated testing, andautomated engineering approach inwhichcontinuous method thatyou're calling after you're done. capacity to runapplications. Service thatprovides secure, resizable compute cross-functional teams. the software involves self-organizing, isbuilt; withflexiblecycles requirements thatevolve as methods thatinvolve fairly shortdevelopment and advanced modeling. technologies via natural language processing creation process thatispossibleutilizingAI Agroup ofsoftware development An open-source mobileprogram Resource isolationatthe AnInfrastructure-as-a- AURLinvoked by theAPI Theprocess of Asoftware THE DZONE GUIDETO AUTOMATED TESTING: A way of Atest - and flexible. while remaining standards-compliant, secure, optimized for speed-critical environments oriented programming. supports event-driven, functional, andobject- with HTMLandCSS for web applications that your users. (Source: informIT.com) software from version control into thehandsof manifestation ofyour process for getting official Android language. oriented andfunctional features. Itisalsoan Apache 2.0License, offering bothobject- bydeveloped JetBrains, provided underthe of data information input. used incode to refer to one of theinputpieces PROGRESSIVE WEBAPPS: PARAMETERS: PAGE OBJECT: own certificate. andthatcan beuseduntilyou getserver your a secure URLconnection to your localhost NGROK: MACHINE LEARNING: LOAD TESTING: LIGHTHTTPD: KOTLIN: JAVASCRIPT: FUNCTION SIGNATURE: DEVOPS: DEPLOYMENT PIPELINE: rather thanbeingspecifically programmed. able to learn based onexposure to data new production businessgoals. agilityandachieve deployment ofsoftware to increase software collaborate and onbothdevelopment teamsdevelopment andoperations teams, where allteams intheorganization, especially that are actually regular web pages. mobile applications ortraditional applications applications thatappear to users like native specified load. a software system’s response whenhandlinga whether itwillreturn anexpected outcome. that theendfunction takes thatdetermine objects inthetest code. models parts ofaweb application's UIas Aservice thatallows you to create Alanguage thatrunsontheJVM, AnITorganizational methodology Aprogramming language used
An open-source web server Aparticular kindofvariable Afeature inSeleniumthat Theprocess ofmeasuring AnAIsystem that is Theparameters Anautomated YOUR END-TO-END ECOSYSTEM Web
efficiency andquality.efficiency intodevelopment distinctphasesto improve data to otherapplications inreal-time. browser for testing. automating web applications insideofaweb predicted outcomes the comparison ofactual outcomes with tested) to control theexecution oftests and software (separate from thesoftware being handle thatincreased work. amount ofwork ortheabilityto grow inorder to application, orsystem to process anincreasing (SDLC): SOFTWARE DEVELOPMENT LIFECYCLE and ranked. using hundreds ofdata points,andthenrated eliminated ifelementsare correctly identifiedby Selector-based maintenance shouldbe SELF-HEALING MAINTENANCE: SELENIUM GRID: SELENIUM: SCALABILITY: WEBHOOKS: VIRTUAL MACHINE (VM): UNIT TESTS: UI TEST: TEST-DRIVEN DEVELOPMENT (TDD): TEST AUTOMATION: than theentire application. code to determine whether itisfitfor use,rather testing individualpieces, orunits,ofsource an application's graphical userinterface (GUI). for distributed testing. machines against different browsers in parallel that allows developers to runtests ondifferent different pieces ofhardware. the purposesoftesting for anddevelopment emulates acomputer hardware system for designing applications. reducing errors andimproving flexibility when process insoftware development geared toward
The divisionofsoftware Atest to determine bugsorfailures in Anopen-source framework for Atesting method focused on AnAPIconcept thatdelivers Theabilityofanetwork, Afeature ofSelenium2.0 .
The useofspecial Software that
PAGE 43OF44
A Start Contributing to OSS Communities and Discover Practical Use Cases for Open-Source Software COMMITTERS & MAINTAINERS
Whether you are transitioning from a closed to an open community or building an OSS project from the ground COMMUNITY GUIDELINES up, this Zone will help you solve real-world problems with open-source software.
Learn how to make your first OSS contribution, discover LICENSES & GOVERNANCE best practices for maintaining and securing your community, and explore the nitty-gritty licensing and legal aspects of OSS projects. TECHNICAL DEBT
BROUGHT TO YOU IN PARTNERSHIP WITH