Editor-in-Chief This is LogiGear magazine’s first issue on the big world of DevOps. DevOps is a very Michael Hackett large topic. Just when you thought you were safe from more process improvement for a Deputy Editor while—not so fast. There’s DevOps, Continuous Testing, Continuous Delivery and Christine Paras Continuous Deployment. In this issue, we are focusing on Continuous Testing, the part most concerning Test teams. Graphic Design DevOps is, by one description, Agile for Ops. With closer input and collaboration Tran Dang from the business side, development and operations are using great tools to help Ops be more Agile and migrate code to production faster. But this can be complicated.

Worldwide Offices Now, I am running into organizations that say they do DevOps or are moving to DevOps but have very little in place to do so, or worse, have no idea what they United States Headquarters are doing. This reminds me of a few companies that I knew in the Agile era, that said they were Agile, but weren’t. 4100 E 3rd Ave, Ste 150 I saw firsthand, how many teams limped into Agile then raced to get testing done, while working from significantly Foster City, CA 94404 less documentation, sometimes marginal collaboration, and were still expected to get high automation coverage; some of them are still trying to get their footing. Without the culture change, empowerment, skills and tools to make Tel +1 650 572 1400 it happen, a team that attempts DevOps is headed for disaster. DevOps will highlight the shortcomings of a team Fax +1 650 572 2822 on a larger scale, and faster than Agile ever could.

DevOps is a minefield! To do anything here, you have to know what you are talking about. It’s not just buzzwords. Viet Nam Headquarters Just because you began using Puppet and Docker doesn’t mean you’re ready for Continuous Deployment. 1A Phan Xich Long, Ward 2 Phu Nhuan District We are at the stage in DevOps, and I am greatly reminded of the early stages of Agile, where the use of a single tool or single change had uninformed people make assumptions about an entire paradigm shift. And I’m sad to Ho Chi Minh City see this trend continuing. Tel +84 8 3995 4072 Fax +84 8 3995 4076 If, a dozen years ago, you dropped phased quarterly releases to sets of 2 weeks sprints with user stories instead of requirements and your Dev team started using Jenkins. That, by itself, did not make you Agile. It was a start. If, for example, the team did not have access to the business side/Product Owner daily, while significantly boosting Viet Nam, Da Nang collaboration with early team involvement and significant automation coverage—then that team would be Scrumbutt, or Agile Falls, but still not truly Agile and instead of productivity gains many teams felt more pressure and 346 Street 2/9 uncertainty. Hai Chau District Da Nang It took most organizations years to implement, figure out and tune the new practices. We even already have a few anti-names—DevFlops and DevOops. But let’s not go there. Let’s do it right! There is great progress to be made Tel +84 511 3655 33 here. DevOps, like Agile, is about culture. In DevOps, the whole organization focuses on the business constraints and needs rather than the whole organization working according to development or Operations schedules. www.LogiGear.com Product functionality and cycles are delivered when the business needs it rather than being at the mercy of development and/or IT/Ops. www.LogiGear.vn www.LogiGearmagazine.com DevOps is also more about business change and Operations/IT change than Dev and Testing change. Dev and Testing got turned on our heads with Agile. This time it’s other groups getting turned upside down. Getting Operations involved, shifting their tasks left, earlier in the cycle, and automating as many Ops tasks as possible with Copyright 2016 tools like Puppet, or Chef, and Docker, among many, many others. LogiGear Corporation To get all this business driven product delivered on their schedule, there is even more use of task automation. To All rights reserved. get a good idea of where DevOps is going—everything that can be automated, has to be. From builds using Reproduction without Continuous Integration and test automation we became Agile. Test teams have been dealing with these tasks for permission is prohibited. a long time, but now more tasks, primarily Ops tasks: building and maintaining environments, build promotion, provisioning, monitoring—are all becoming automated.

Submission guidelines are For Test teams, this means a lot. Apart from the team dynamics, tools, and responsiveness to change, this of course, means bigger and more intelligent test automation. How we look at test automation has to evolve and grow. Its located at: use, as well as when, and where to use test automation and how this cycle impacts our regular Dev sprints is http://www.LogiGear.com/ expanding the role of the test teams—Clearly, we have a lot to learn and a lot to change. magazine/issue/news/2015- To be DevOps and not DevFlops—first we have to know what we are aiming for, why, goals and how we can best editorial-calendar-and- support the business. It’s a challenge. submission-guidelines/ I hope we can help you.

In this issue, we feature a two part series by Sanjeev Sharma, Understanding DevOps, and Adopting DevOps. Alister Scott discusses Testing in Production in our Blogger of the Month feature. Sanjay Zalavadia writes about best practices to create a test-driven development environment and Tim Hinds discusses Where QA Fits into DevOps. Steve Ropa also has reviewed the 7 Best DevOps books, and we have an interview with Skytap’s Sumit Mehrotra. I’m pleased to announce that in addition to continuing our new TA Corner series, we also have another new column, Leader’s Pulse, which largely features recommendations on how to manage Test teams. 4

Continuous Testing, Part 1 Michael Hackett 5 DevOps for Test Teams

The DevOps lifecycle 9 Infographic Christine Paras

Understanding DevOps Sanjeev Sharma 10 Continuous Testing and Continuous Monitoring Where Does QA Fit In DevOps? Tim Hinds 12 Fitting QA into a modern DevOps group Struggling with Continuous Testing? 14 Infographic Christine Paras 3 best DevOps practices to create a TDD environment 15 How to ensure a successful test-driven environment Sanjay Zalavadia

Adopting DevOps 17 Aligning the Dev and Ops Teams Sanjeev Sharma

19 How to leverage TestArchitect in DevOps LogiGear Staff

An Interview with Skytap’s Sumit Mehrotra Michael Hackett 20 On what you need to know before making the transition to Virtualization The 7 Best DevOps Books 23 From adopting the culture to implementing Continuous Delivery Steve Ropa

27 The ownership of quality has evolved, don’t get left behind Michael Hackett

Test in production? Alister Scott 30 This post is part of the Pride & Paradev series

32 33 LOGIGEAR LAUNCHES CONTINUOUS TESTING SOLUTION

LogiGear is pleased to announce our new Continuous Testing Service offering. For over two decades, LogiGear has solved complex test automation challenges for our clients. We’ve gained deep experience and plan to use it to guide you through your Continuous Testing, Continuous Delivery and DevOps transformation journey. Our test automation experts and engineers leverage the leading tools and practices to ensure the optimum time to market, cost reduction and peace of mind.

 Read more at: http://www.logigear.com/services/continuous-testing- solution.html

GOOGLE PLANS TO REPLACE SMARTPHONE PASSWORDS WITH TRUST SCORES At Google’s I/O developer conference, Daniel Kaufman, head of Google’s advance technology projects, announced that the company plans to phase out password access to its Android mobile platform in favor of a trust score by 2017. Developer kits will be available by the end of 2016.

 Read more at: https://www.newscientist.com/article/2091203-google -plans-to-replace-smartphone-passwords-with-trust-scores/

AMAZON CEO JEFF BEZOS TEASES AN AMAZON WEARABLE AT CODE CONFERENCE

During his interview with Walt Mossberg, at the Code Conference earlier this month, Amazon CEO Jeff Bezos coyly hinted at the roadmap for Amazon Hardware, which is extensive. When asked about wearbles, Bezos commented “I think it’s a super interesting market, and I obviously can’t talk about our future roadmap, but I think that’s [ the wearables market is] also in its infancy.

 Read more at: http://www.theverge.com/2016/6/8/11879684/walt- mossberg-jeff-bezos-amazon-blue-origin-code-conference-2016 Continuous Testing, Part 1

DevOps for Test Teams

By Michael Hackett

ow that Dev Teams have had a little time to Continuous Testing gets us far along this path and is the settle into the Agile, the new wave of process part of DevOps most concerning to traditional Test optimization has arrived. DevOps. DevOps has teams. That is, Test teams that would execute been described as Agile on Steroids. DevOps N everything from black box, to regression, to UI, to has also been described as Agile for Operations/IT. I like both of those descriptions as well as some others. Many workflow scenario tests. Who generally do not do organizations want Development, Test and Operations security, performance or unit tests and have therefore teams to move to DevOps now. not executed tests in production.

When I think of Continuous Testing I think of the Lean DevOps is a big topic. But DevOps is not the focus of this principle, quality at every step. When developers article. We will not be talking about, for example, containers, commit new code, test it! When the product gets or Docker, or Puppet or Infrastructure as code. In this article, we are going to focus on Continuous Testing. I will focus integrated, test it! When the product gets moved to on what Test teams are responsible for--the practices and any new environment, like the test environment, concerns for testers to have their work become an asset to staging environment or production, test it! the team and not a drag in this exciting new world of DevOps. This is a 2 part series on Continuous Testing. Part 1 is on the rationale, business goals and overview with some specific testing tasks. Part 2 is a deeper dive into the specific ideas and tasks of Test teams that need to be updated or changed to be successful in this next phase of product development.

There are many problems DevOps is trying to solve. The overriding idea is to:

1) Shift Operations or IT tasks left, earlier in the process.

2) To leverage a bunch of newer tools and technologies to automate operations tasks, like provisioning and migrating code, leveraging these tools.

3) Have everyone on the product team communicate There are Continuous Testing activities. and collaborate much more and earlier. Over the last few years, with Agile and Continuous 4) To have the product be ready to be deployed—in a Integration, we have already shifted left and partially consistent state of readiness so that the business can right. Now, we have to learn how to shift testing fully decide when new functionality goes to customers. right. To do this, and test on all the various environments And ultimately, not be held back by Dev or Ops and stages that need to be tested and validated, and being unstable and not prepared to go with for rapid delivery and/or deployment, the testing, of whatever is ready. course, needs to be automated. Another way to look at this is as a system, there are three parts:

1. The build delivered by Development. 2. The deployment on infrastructure handled by OPS/IT. 3. The customers who use the apps/system.

Test teams must have strong knowledge of these three parts, know the goals of testing at each part, be able to design great tests for each part, and have the ability to automate these tests. This is harder and more comprehensive than most organizations think. is for Test teams to automate as effectively, Many Test teams still struggle with Agile, fast test thoroughly, and economically as possible. This will automation and automation maintenance. These not be easy, nor is it an instant change. problems will not help the organization in its mission; DevOps for Testers is focused on Continuous it will only drag it down. Tests can be designed and Testing. built for successful Continuous Integration first, and expand that to Continuous Testing by knowing the The goal of Continuous Testing is that these test right things to do, staying focused on a few key tasks provide immediate, useful feedback to the points and most importantly, automating as smartly whole team on the stability and consistency of the as possible. product, ultimately giving the team confidence before delivery and deployment. Let’s get started with the work.

Test teams must also move seamlessly through the The first task. Know what you are talking about with testing cycles—this is a constant rolling process. For DevOps. most teams who do DevOps, “phases” is not used Go read about DevOps. Familiarize yourself with the to describe where the product is or its readiness. terms currently used in discussions about DevOps, you The product can be in Dev, in Test, tests running on can also reference our Glossary in the following pages. staging server, and tests running in production at

the same time. If the business decides to deliver For example, Continuous Delivery is not Continuous functionality to customers, it can move delivered Deployment. Find out the details, the tools Ops/IT will code to deployment based on feedback from the be using and why they will use them. Learn about tests, instead of the phase the product is in. provisioning, containers, virtualization, and Essentially, making the product is in a constant automatic deployment. We need to be familiar state of test. This will be the primary factor in with the whole DevOps paradigm to be useful to building consistency from coding to deployment. the team and also to design the right tests for each situation, even though they may not be part of our “Testing” is now about all the test efforts. job. For example—understanding containers may Of course, first, you need awesome testing. You will impact our test design for integration testing—but need more of it, it will need to be Agile and it will the how, when and where to use containers is not need to be automated. Automated more than a Test teams task. ever before. Testing includes re-running unit tests, For everyone, DevOps is an automatic leveraging running functional, workflow, validation, and bug- technology, in which everyone’s tasks are automated. finding tests. These along with security tests, load From builds, to provisioning, to deployment, and and performance testing…there is no separation in everything in between—it’s all automatic. The goal terms of running these tests for consistency, Scrum process—fix them before moving any further.

Also, if you are having a difficult time presently keeping up with automation in your sprints, any attempts at Continuous Testing will surely fail. Your testing, test automation and reporting must already be smooth, low maintenance, and effective.

The test automation of new functions has to happen within the sprint; first manual testing in sprint and then by the end of the sprint.

Don’t even try Continuous Testing until you have great, meaningful, effective, fast Continuous Integration. monitoring and reporting. When people describe DevOps as a “shift left,” one reference is to running In every team I have worked with, CI leads to performance tests early and not waiting until the Continuous Testing. CI is the foundation of end of development. This is an important example Continuous Testing. because it illustrates the many pieces of DevOps. Having a great Agile practice means the The production environment can be virtualized and organization understands everyone does Quality spun up at any moment with a VM or in the cloud Assurance, not testing. Quality Assurance means and have full performance tests run early in quality user stories, quality unit tests, and quality development. In the very recent past, good environments. Quality at every step. Developers would run performance tests early but may have had to mock up a limited system and You need a better, more comprehensive test focus on isolated performance tests. strategy.

Different automated suites of combinations of Get your test game on! Great testing skills are still these tests will be run on various environments your most important job skill. throughout the process. Depending on your Where old style “QA” testing tested at the end, organization, some tests “belong to” the QA or Test Continuous Testing takes Test teams and entire team. You may be expected to take on more test organizations to a whole new level. and monitoring responsibilities. Continuous Testing, in theory, frees you from Testing in production is normal and expected. This focusing on big bang deployments. You have may be problematic—be intelligent and careful. automated suites running constantly to monitor Now testing happens everywhere, and is done by system health as more work items are delivered. everyone. To be successful in DevOps, aside from test You will need a strategy for testing in the sprint, a automation, the test leadership should have assigned strategy for what to test in CI, and a strategy for Developers, Test Operations and QA leadership as part what to test in the various environments. of the same team. Testing is now whole, no longer just a part. Test teams need to stay focused on delivering Your team has to be great at Agile. Scrumbutts and Done, potentially shippable User Stories at the end AgileFalls will fail in DevOps. of each sprint or iteration as well as growing and optimizing the various regression suites. As said above, some people describe DevOps as Agile on Steroids. It you aren’t great at Agile You still need to adequately test new functionality. already, Continuous Testing will fail, and DevOps Don’t focus too much on small “acceptance” tests. will fail. If there are problem areas in your Agile/ You need workflows and integration tests, user scenarios and task based tests. Low-level, small A quick, Continuous Testing task overview is: tests may miss workflow/integration/end-to-end 1. Start with an automated smoke test. Move these into CI problems. build process tool. You need speed but don’t forget exploratory, data 2. Build bigger regression suites. Move these into CI build driven, and soap opera tests. Too many teams get process tool. overwhelmed by more automation and focus only 3. Grow in levels of awesomeness of CI; Run smoke and/or on test automation at the expense of a full bug regression on multiple VMs. finding, validation, design collaboration, and customer experience test strategy. 4. Easy and effective reporting back to the organization. 5. Use containers and/or virtualization for data and full production like environments. You may have a separate team building the test 6. Distribute automated tests into different suites with varying automation framework and another team doing goals on different environments. Use VMs for various automation to pick up the pace. Your strategy will environments to grow automation, coverage, speed, need more low-level tests. Integration tests, monitoring and feedback to the team. especially in the new “container” world where Continuous Testing grows from this. containers/components can be easily plugged in, Part 2 of this topic will define and describe the lower are high priority. level practices and automation goals, environment and Another trend in software development over the data problems to solve, virtualization, with more details last few years is Service-Oriented Architecture on testing goals. (SOA). Today, SOA has gone into overdrive, for many reasons not important here, but with the use of containers and microservices as a goal of many Michael Hackett organizations. Whether your team uses or refers to, Michael is a co-founder of APIs, SOA, web services, or microservices; or is using LogiGear Corporation, and a container tool like Docker, more testing is has over two decades of happening now on more lower-level items, and is experience in software more technical. Testing isolated services is great for engineering in banking, finding bugs or breaks fast. Then, separately, securities, healthcare and integration tests need to run for correct integration consumer electronics. Michael and consistency. is a Certified Scrum Master Testing should know a lot about grey-box testing. and has co-authored two books on software This is about grey-box, intelligent test automation testing. Testing Applications on the Web: Test with knowledge to design better tests and to do Planning for Mobile and Internet-Based Systems quicker failure analysis. (Wiley, 2nd ed. 2003), available in English, Chinese and Japanese, and Global Software An analogy of testing in DevOps is similar to an Test Automation (HappyAbout Publishing, assembly line production, like a car manufacturing 2006). line, for example. You need to check and test everywhere—not only at the end. A key difference He is a founding member of the Board of in a car assembly line, though, is that the parts are Advisors at the University of California Berkeley fixed. With software in DevOps, the parts are Extension and has taught for the Certificate in changeable and when the parts change, they will Software Quality Engineering and need automated testing to keep up. Management at the University of California Santa Cruz Extension. As a member of IEEE, his Continuous Testing in DevOps requires culture as training courses have brought Silicon Valley well as process change. Part 1 of this article testing expertise to over 16 countries. Michael focused on culture and understanding. Part 2 will holds a Bachelor of Science in Engineering focus more on process and tasks. from Carnegie Mellon University. The DevOps Lifecycle

By Christine Paras

evOps is really the journey towards the ideals of greater collaboration and getting immediate D feedback thereby achieving greater productivity, and a high quality product. In order for DevOps to be successful, Development, Operations and the Test team must align their duties.

DevOps is an extension of Agile development that aims to streamline the process of software delivery as a whole. By making each development stage interdependent and incorporating Continuous Testing early and through the development cycle, this approach enhances efficiency and reduces risk by allowing Developers to work with Operations much earlier in the process. Understanding DevOps

Continuous Testing and Continuous Monitoring

By Sanjeev Sharma

What is the goal of Continuous Integration? Is it to enable Continuous Delivery of the code developers’ produce out to users? Yes, eventually. But first and foremost it is to enable ongoing test and verification of the code. It is to validate that the code produced and integrated with that from other developers and with other components of the application, functions and performs as designed. Once the application has been deployed to a production system, it is also a goal to monitor that application to ensure that it functions and performs as designed in a production environment, as it is being used by end-users. This brings us to the two practices of DevOps that are enabled by Continuous Integration and Continuous White box Security tests, code performance tests, etc. Delivery. This work would then be delivered to the common Integration area of the team of teams – integrating the Namely: work of all the teams working on the project and all the  Continuous Testing code components that make up the service, application or system being developed.  Continuous Monitoring This is the essence of the process of Continuous Continuous Integration and Delivery are both (almost) Integration. What makes this process continuous is meaningless without Continuous Test. Not having where the individual developers’ code is integrated motoring and hence, not knowing how the application is with that of their team as and when they check-in the performing in production makes the whole process of code and it gets delivered for Integration. (For more DevOps moot. What good is having an streamlined detail, read Part 2 of this Understanding DevOps series). Continuous Delivery process if the only way you find out The important point to note here is the goal of the about your applications’ functionality or performance Continuous Integration process – to validate that the being below par is via a ticket opened by a disgruntled code integrates at all levels without error and that all user? tests run by developers run without error. Continuous Testing starts right with the developers. Let’s break this down. After validating that the complete application (or Continuous Integration and Continuous Delivery facilitate service or system) is built without error, the application is Continuous Testing delivered to the QA area. This delivering of code from the Dev or development environment to the QA Individual developers work to create code – to fix environment is the first major step of Continuous defects, add new features, enhance features, or to make Delivery. the code perform faster – could be one of several tasks (work items) they could be working on. When done, they There is Continuous Delivery happening as the deliver their code and integrate it with the work done by developers’ deliver their code to their teams’ other developers on their team, and with unchanged integration space and to the projects integration code their team owns. They would along the way run Unit space, but this is limited to being within the Dev space. Tests on their own code. There is no new environment to target. When delivering to QA, we are speaking of a complete transition from Once the integration is done, they would do Unit Tests on one environment to another. QA would have its own the integrated code. They may run other tests such as environment on which to run its suites of functional and perform these tasks too – provision staging and performance tests. DevOps principles preach that this production environments and deliver the application. environment be production-like. Continuous Monitoring In addition QA would potentially also need new data sets for each run of the suites of tests it runs. This may be In Production, the Ops team manages and ensures even one or more times every day as Continuous that the application is performing as desired and the Integration leads to Continuous Delivery at a steady environment is stable via Continuous Monitoring. While stream. This means that the Continuous Delivery process the Ops teams have their own tools to monitor their would not only require the processes to transition the environments and systems, DevOps principles suggest code from Dev to QA, but to also ‘refresh’ or provision that they also monitor the applications. They need to new instances of QA’s ‘Production-like’ environment, ensure that the applications are performing at optimal complete with the right configurations, and associated levels – down to levels lower than system monitoring test data to run the tests against. tools would allow. This requires that Ops teams use tools that can monitor application performance and issues. This makes Continuous Delivery a more complex It may also require that they work with Dev to build self- process than just ftp-ing code over. (I will discuss this in monitoring or analytics gathering capabilities right into more detail in a later post of Infrastructure automation the applications being built. This would allow for true and Infrastructure as code). The key point to note is that end-to-end monitoring – continuously. the goal of Continuous Delivery is to get the code ready for test. To get the application to the right The Four Continuous Processes of DevOps: environment – continuously, so that it can be tested – continuously. It would not be much of a stretch to say that the practice of DevOps is really up of the implementation If one extends the process described above to of these four basic processes: delivering the service, application or system to a staging and eventually a production environment, the  Continuous Integration process and goal remains the same. The Ops team  Continuous Delivery wants to run their own set of smoke tests, acceptance  Continuous Testing tests and system stability tests before they deliver the  Continuous Monitoring application to the ‘must-stay-up-at-all-costs’ production environment. That is done using the Staging Of these, the end goal is really Continuous Testing and environment. This is a production-like environment that Continuous Monitoring. Continuous Integration and needs to be provisioned just like the QA Environment. It Continuous Delivery get the code in the right state and needs to have the necessary scripts and test data for in the right environment to enable these two processes. acceptance and performance tests that Ops would Testing and Monitoring are what will validate that you run. Only when this last phase of Continuous Testing is have built the right application, which functions and complete would the application be delivered to performs as designed. In future posts, I will go into more Production. Continuous Delivery processes would detail of the implementation of these four processes and the associated practices that enable them. Where Does QA Fit In DevOps?

Fitting QA into a modern DevOps group

By Tim Hinds

n a traditional software engineering organization, the QA group is often seen as separate from the Development group. Developers and testers have I different roles, different responsibilities, different job descriptions, and different management. They are two distinct entities. However, for folks outside the engineering team – say in Operations – they generally consider Development and QA to be in the same group. From this perspective those teams are working together to do a single job, with a single responsibility: deliver a product that works. So what happens with QA in a DevOps organization? When Development and Operations merge together, where does that leave QA? How does the testing team fit into a modern DevOps group? This article will take a look at exactly that question.

The Reason Behind DevOps: Automated Deployment “Quality” is built into the fabric of DevOps. If you When you look at the trend towards DevOps, it’s pretty couldn’t reliably push high-quality software out the clear that companies are adopting this organizational door, DevOps would fail as a function. model in order to facilitate a practice of automated software deployment. DevOps provides the structure Clearly, there is a critical role for QA in a DevOps that enables teams to push software out as a service on group. So how are people fitting it in? a weekly, or daily, or even hourly basis. The traditional concept of a “software release” melts away into a The Product IS The Infrastructure continuous cycle of service improvement. We asked Carl Schmidt, CTO of Unbounce, what he DevOps is Agile taken through its logical evolution: thinks about QA and DevOps. Unbounce runs a SaaS removing all the obstacles to getting high-quality solution for online marketers, making it really easy to software in the hands of customers. Once you have a build, publish, and A/B test landing pages without IT smooth process for agile development and continuous resources. Unbounce has three development teams, integration, automating the deployment process makes each with a resident QA team member, and practices total sense because it’s fulfilling the objectives that DevOps throughout the organization. business managers crave: Carl states, “I’m of the mindset that any change at all 1. Faster time to market (software or systems configuration) can flow through one unified pipeline that ends with QA verification. In a 2. Higher quality more traditional organization, QA is often seen as being gatekeepers between environments. However, Increased organizational effectiveness in a DevOps-infused culture, QA can now verify the environments themselves, because now infrastructure is code.” But before we move on, let’s ponder that for a moment: The infrastructure is code. It’s a game-changing claim the purpose of DevOps is to get high-quality product out to any traditional development organization. to the market faster – even automatically. The notion of Historically there was a clear separation between what constituted the product and what constituted the Beyond Functional Testing: Automation for Load Testing, operation. You built a product, deployed it into a test Stress Testing, and Performance Testing environment where it could go through some quality Now we are at a very exciting time in the transformation control, and then eventually deployed it onto a live of QA, because while many organizations have mature production system where real users could get at it. If processes for automating functional testing, they are only there was a problem in product, the operations team just beginning to apply these practices to other areas of had to fix it. testing like security and stress testing. But as the lines blur between product and operation In particular, load testing and stress testing are critical – as the very name DevOps implies – it’s no great disciplines for DevOps organizations that are moving at leap to recognize that the environment itself is a part high velocity. A bottleneck introduced to a critical of the product. transaction process on an eCommerce website can bring Carl continues, “It’s QA’s responsibility to actually a business to its knees. You want to do everything you can push new code out to production, so the DevOps to identify scaling problems before a product is pushed team has been providing them with tools that make out to a production environment, and you also want to blue-green deployments push-button easy. Our QA keep a close eye on performance after it’s been team can then initiate deploys, verify that the released. intended change functions as expected, cut over to If you are curious to understand how your process holds the newly deployed code, and also roll back if there up, check out this infographic: How Automated Your is any reported issue.“ Performance Testing Is.

DevOps QA Is About Preventing Defects, Not Finding Creating a DevOps Culture Them Unbounce CTO Carl Schmidt has some wise advice about QA takes a critical role in this organizational structure his DevOps group: “we dislike using that term in favour of because they have the visibility and the directive to saying that we have a DevOps ‘culture.’” DevOps isn’t an push code out when it is working, and roll it back individual, it’s a core value of a development when it is not. This is a very different mindset from QA organization. DevOps is more about trust, people, and organizations of 10 years ago, whose primary teamwork than it is about process. It’s about the creating responsibilities involved finding bugs. Today QA of software as an ongoing service, not a static product. groups are charged with preventing defects from reaching the public site. Although it’s not in the name (DevQuops, anyone?), the only reason that DevOps works is because quality is built This has several implications: into the entire system. You can’t get much more  QA owns continuous improvement and quality important than that. tracking across the entire development cycle. They are the ones who are primarily responsible for identifying problems not just in the product but also in Tim Hinds the process, and recommending changes wherever they can. Product Marketing Manager

 Tests are code, as any test automation expert will Tim Hinds is the Product tell you. It’s a necessity, of course. If your process is Marketing Manager for designed to publish a new release every day (or every hour) there is no room for manual testing. You NeoLoad at Neotys. He has a must develop automation systems, through code, background in Agile software that can ensure quality standards are maintained. development, Scrum, Kanban,  Automation rules. Anything that can be Continuous Integration, Continuous Delivery, and automated, should be automated. When Carl Continuous Testing practices. Previously, Tim was describes Unbounce’s deployment process as “push- Product Marketing Manager at AccuRev, a button easy,” this is what he’s talking about. company acquired by Micro Focus, where he Testers are the quality advocates, influencing both worked with software configuration management, development and operational processes. They don’t issue tracking, Agile project management, just find bugs. They look for any opportunity to continuous integration, workflow automation, and improve repeatability and predictability. distributed version control systems. https:// www.linkedin.com/in/timotheehinds Struggling with Continuous Testing? You’re not alone...

By Christine Paras

The problems around automation have become increasingly complex. And now, automation is much more integrated into the software development process…

We see CI moving onto virtual machines and DevOps running our automation all the time, on all kinds of environments. It is no longer just the test team that runs automation, and reports results, and these tests are occurring whenever a build happens. Many teams are still struggling with getting automated tests into their current sprints, or new sprints. Some teams struggle just to get more tests automated in their development cycle at all, and end up settling for adding new automation after a release, because they just do not have the time. If this is your situation fix it! It may not be an easy fix, but not fixing it has a negative impact on development.

What you need to consider before you make the jump to Continuous Testing…

How to make the jump to DevOps

To make the true leap to DevOps, you have to automate not just your testing, but other tasks that were the responsibilities of other teams. This can get burdensome if you don’t have mature Agile practices in place, and these 3 steps will help you get your automation in shape for the leap.

Once you have efficient Agile practices, good low-maintenance test automation and Continuous Integration processes in place, then Continuous Testing and DevOps is the next step in achieving hyper efficiency. By shifting your testing left, you will uncover bugs earlier, saving both time and funds for your company. 3 best DevOps practices to create a test-driven development environment

How to ensure a successful test-driven environment

By Sanjay Zalavadia

n order to ensure a higher quality product is released in the end, many teams have turned to test-driven development. Under this scenario, quality assurance I metrics professionals first create various QA tests, and then software engineers code based on these tests, typically while using a robust enterprise test management tool.

It's easy to see the benefits of such an approach, as it ensures that QA metrics are kept front and center throughout the entire process. However, it can also introduce problems even for teams following agile development best practices and using free agile test management tools. In particular, test-driven development requires quality assurance metrics professionals to have a very clear understanding of what the final product is supposed to do and all of the steps software development process. There is simply no way they need to take to get there. one person or even a small number of people can In order to make sure QA teams and software engineers manually develop tests, code and run these tests. are always on the same page, it can be extremely Considering how critical the initial tests are in a test- helpful to follow DevOps principles. A portmanteau of driven environment, it is usually advisable for the initial Development and Operations, DevOps describes a tests to be created by hand. But, once the tests are paradigm under which all teams always have a clear established, there's no need for the subsequent steps to understanding of what everyone else is up to throughout also be so labor intensive. Not all components need to the design and delivery process. be coded by hand, and tests can be run Many ideas and workflows can fall under the DevOps automatically. Relying on automation, especially in the umbrella, but these three are absolutely critical best later stages, is crucial for ensuring that work is practices within a test-driven environment: completed quickly without any sacrifices from the DevOps ideals. 1) Embrace of automation 2) Placement of end users front and In any DevOps scenario, a heavy load is placed on the center shoulders of everyone involved. Individuals don't have the luxury to only focus on one or a small number of The name DevOps seemingly puts the focus on Dev and tasks, as they now have to be involved in all parts of the Ops teams, but these are far from the only ones that In a test-driven development environment, it stands to should, and do, fall under this paradigm's umbrella. reason that tests play a large role in the entire DevOps solutions are all about fostering a culture of process. As such, these all-important tests should not collaboration and understanding, including with the be entrusted to a lackluster test case tool. Only a ultimate end users. If the people who will actually be robust enterprise test management tool will ensure using the final piece of software are not involved in its that everyone has the information they need at all development, then it's entirely possible that the final stages of the software development lifecycle. product falls flat of expectations even if internal teams are well aligned. In many scenarios, a test-driven development environment is an ideal way to ensure the final For the quality assurance metrics professionals in charge product is of the highest caliber. Still, a test-driven of creating the initial tests, this means first gaining an environment can be difficult for those new to the understanding of what end users really need from the concept to navigate. By adopting DevOps and a final software, and what issues will cause them to number of associated key best practices, however, abandon it. Armed with this knowledge, they can then these teams can get the most out of a test-driven develop the initial tests to ensure that software development environment. engineers are meeting all of the wants and needs on the end-user side. By adopting DevOps and a number of associated key best practices, teams can get the most out of a test- 3) Adoption of enterprise test driven development environment. management tool suitable for the task

Sanjay Zalavadia

As the VP of Client Service for Zephyr, Sanjay brings over 15 years of leadership experience in IT and Technical Support Services. Throughout his career, Sanjay has successfully established and grown premier IT and Support Services teams across multiple geographies for both large and small companies. Most recently, he was Associate Vice President at Patni Computers (NYSE: PTI) responsible for the Telecoms IT Managed Services Practice where he established IT Operations teams supporting Virgin Mobile, ESPN Mobile, Disney Mobile and Carphone Warehouse. Prior to this Sanjay was responsible for Global Technical Support at Bay Networks, a leading routing and switching vendor, which was acquired by Nortel. Sanjay has also held management positions in Support Service organizations at start-up Silicon Valley Networks, a vendor of Test Management software, and SynOptics. www.getzephyr.com Adopting DevOps

Aligning the Dev and Ops Teams

By Sanjeev Sharma

evOps as a philosophy has had as its centerpiece the principle that Dev and Ops teams need to align better. This is a people and D organizational principle, not a process centric principle. To me this is more important when adopting DevOps than any other capability or tool. My last post focused on the need to better align Dev and Ops and the challenges that such organizational change would address. This post discusses key guiding principles on which this Dev-Ops alignment should be based. They are designed to improve collaboration between these organizations and to break down the proverbial ‘wall’; to end the water-SCRUM-fall, when it comes to the relationship between the Dev and Ops teams. better collaborate across the organizational boundaries, These guiding principles are: these teams should start using the same Change Management and Work Item Management tools or worst

case, use tools that are integrated. Thus allowing 1. Shift Left: seamless visibility across tools and traceability between respective Change Requests. Real-time collaboration Organizationally the goals of DevOps are to bring Dev using a common tool is obviously the best scenario. and Ops closer. Not just at deployment time, as they may do already, but all through the development cycle. It requires Ops to allow Developers to take more 3. Build ‘Application Aware’ Environments responsibility of the operational characteristics of the applications they are building. In turn, it requires As we move towards Software Defined Environments, we Developers to keep Operational considerations in mind have the ability to build, version and manage complex while building their applications. This is referred to in the environments, all as code. All of the benefits of this are DevOps world as ‘Shift Left’. As in shifting towards the left, moot if the environments are not a perfect fit for the some Ops responsibilities. Shifting it to earlier in the applications and more importantly the changes to the software delivery lifecycle, towards Dev. applications being delivered. The goal is hence to build Environments that are ‘Application Aware’ or are fine- tuned for the applications they are designed to run. No more cookie cutter Virtual images for all kids of applications. More importantly, one needs to ensure that the environments are architected in a manner to allow for the evolution of the applications, both as they are developed; as they are projected to change or as they may evolve in the future. This obviously, would require close collaboration between Dev and Ops. Development Architects and Product Managers need to work with the IT Architects and Operations Managers to Architect Environments and their projected evolution to align with 2. Collaborate across all teams: that of the application. Not only that, but also allow The Dev and Ops teams typically use different tools to enough resilience in the environments to allow for manage their projects, for change management and unexpected change. For example, massive change (outside of email) different collaboration tools. In order to caused by a super successful App. Think Instagram and how the founders had to keep changing their server of their Builds and Environments respectively and will environment almost daily as millions joined their service. provide feedback to both at the end of every Sprint, using which Dev can enhance their Apps to improve 4. Environment Sprints? functionality and performance and Ops can enhance the environment for the same. Agile Development Principles prescribe that at the end of every Sprint, developers have a build. An executable that runs, even if does not do much in terms of functionality. It The Human factor has been proposed (by the likes of Gene Kim) that this These principles are designed to foster Dev and Ops concept be extended to environments. This would mean alignment from a teaming perspective, from a people that at the end of each Sprint, the Dev team would have perspective. These however do not replace the need an executable AND the Ops team would have an for good old team building. Whether it is thru face to environment, a production-like environment (potentially face get togethers or in todays distributed worlds, thru with very limited production like capabilities), built that regular virtual meetings and online collaboration in real the executable can be deployed on. This would allow the time. The best teams are the ones where the people build to be tested. That too in a production-like know each other, trust each other, look out for each environment. This would also provide immediate other and when there are challenges, know who to pick feedback to the Ops teams on the behavior of the up the phone (or launch Skype) and talk to. This series application in their environment and use that feedback on Adopting DevOps and my earlier series on to improve the environment. This also provides an Understanding DevOps will continue in future posts. My opportunity for the Ops and Dev teams to align better. thoughts on #DevOps, Software #Architecture, #Agile They will both have to use a single Sprint plan for releases Development, Innovation, Technology, Life…

Sanjeev Sharma Sanjeev is an IBM Distinguished Engineer, and the CTO for DevOps Technical Sales and Adoption at IBM, leading the DevOps practice across IBM. Sanjeev is responsible for leading the worldwide technical sales community for DevOps offerings across IBM’s portfolio of tools and services, working with IBM’s customers to develop DevOps solution architectures for these offerings, and for driving DevOps adoption. He speaks regularly at conferences and has developed DevOps IP, including the DevOps For Dummies book published in 2013.

Sanjeev is a 20 year veteran of the software industry. Sanjeev has an Electrical Engineering degree from The National Institute of Technology, Kurukshetra, India and a Masters in Computer Science from Villanova University, United States. He is passionate about his family, travel, reading, Science Fiction movies and Airline Miles. He blogs about DevOps at http://bit.ly/sdarchitect and tweets as @sd_architect

How to leverage TestArchitect in DevOps

By LogiGear Staff

t is a fundamental role for testing teams to align their test design, test automation, and test case development with DevOps–not only to verify that code changes work but that the changes do not break the product. A key differentiator of DevOps is testing maturity. An organization can automate their integration, testing, delivery, and I monitor, but still have issues with the intelligence of test orchestration and automation, which can lead to a bottleneck if this is not resolved beforehand.

In most projects, TestArchitect can be used in a general DevOps process. Following the Action Based Testing method, TestArchitect offers capabilities to develop and automate test cases quickly in the same sprint as the application under test, thus allowing team members to become actively involved in the testing and automation process, each from their own skill perspective. Apart from QA other team skills/roles will typically be development, operations and product ownership.

There are four continuous processes in DevOps as below.

Once the automated tests is ready, they can be put into test suites, and there execution can be defined batch file, which can be executed by any scheduling or continuous integration tool, and even be used for testing in production. The results of the automation execution can be generated in xUnit format, XML format, or HTML format which can be read and run report against.

You can generate the batch file in the execution in the second tab of the execution dialog: dialog.

The generation of execution information is controlled in the second tab of the execution dialog:

A file is created with an execution command that includes which tests to run, and all relevant settings and options specified in the rest of the execution dialog. Allowing the team to create powerful and maintainable The generation of execution information is controlled tests, and organize their automation like outlined above can help teams achieve an effective and efficient development and release process. An Interview with Skytap’s Sumit Mehrotra ...On what you need to know before making the transition to Virtualization

Interviewed by Michael Hackett

with higher quality.

What is “Environment or Infrastructure as Code?” 2. How can we use this idea in testing? As the name suggests, infrastructure-as-Code is practice of representing infrastructure as code blocks so that they can be executed whenever needed to build an application stack (environment) in a repeatable and consistent manner. Depending on the actual tool used, it can allow you a variety of things including:  Deploy basic infrastructure elements, VMs, disks, networks, etc.  Install and configure operating systems  Deploy and configure application components What are the main differences between cloud-  Populate application data 1. based environments and cloud infrastructure? With a repeatable and consistent process, bugs can be An environment is a collection of infrastructure elements reproduced, fixed, and validated efficiently. working in conjunction to enable an application stack to Developers can recreate the same environment that a work. tester is using to find bugs by using the same For example, a simple 3-tier application, with a web front Infrastructure-as-Code scripts as the tester is using. This -end component, a business logic component and a practice goes a long way in solving the problem of ‘it- database component requires, at a minimum, the works-on-my-machine’ syndrome. following infrastructure components: a few VMs, a few storage disks, and a network. All of these components There are various paradigms available to create are required, and they need to work together as an consistent environments in a repeatable manner. Two environment for the application to be functional. Note, such popular paradigms are “patterns” and “clones.” that as the complexity of the application stack grows, With patterns, you can create the environment from the definition of an environment grows bigger than just scratch each time—allowing you to progress the same infrastructure components. It can include VPNs, public IP exact code at each point of your software delivery addresses, an object store, and other service end-points. pipeline. But it takes time for the environment to be provisioned and can be error-prone especially, for very From an application development and testing complex environments. With clones, you can create perspective, the application environment is the starting another copy of the environment from a pre-existing point of productive work. However, we see teams copy. The process can be very fast, even for complex spending a large amount of time just getting the environments. However, moving the same clone infrastructure components to a working state and hence through your SDLC can be a heavyweight process. losing out on a good portion of their productive work. There is no right or wrong paradigm here. Successful Working with environments as a unit of work can enable teams have chosen a combination of paradigms and these teams to be more productive and hence service have made trade-offs between time and tooling customer and business requirements at a faster pace consistency, to achieve the velocity they need in their software delivery lifecycle.  Configurability - Can the platform deliver test environments that capture the complexity of my How do companies use cloud-based environments application at each stage of testing?  Consistency – Can the platform deliver test to facilitate testing in DevOps? 3. environments that are in the exact state that we Testing especially in the DevOps world has the following need them to be, whenever I want? key requirements:  Collaboration - Can the platform make it easier for  Instant self-service access to application stacks for my team (Devs + QA + Ops) to work together more testing productively?  Support the complexity and scale of the application  Control – Does the platform have controls to ensure stack needed at various stages of testing the right people have the appropriate resources to  Consistent and reproducible ways to create test do their jobs? environments  Ability to collaborate and share environments with How long does it take to build these developers and testers as part of a feedback loop. 5. environments? Some Test teams do not have Constant feedback is a key aspect of Agile dedicated IT members—does the Test team often do development this?  Ability to ensure that the right resources are available The time to get a fully configured environment to test at any time and are being utilized judiciously All of these requirements are hard to meet without a can vary in range from a few minutes to a few weeks. cloud-based platform. So, cloud-based environments are The availability of cloud platforms, and especially IaaS essential to facilitate testing workflows. has reduced the far-end of the range to a few days primarily by taking the acquisition of hardware out of We know that getting the right environments for the equation. This is the lowest layer of the environment 4. testing can be difficult, and sometimes it’s out of a and typically, the one that used to take the longest. Test team’s control. How are innovative organizations Reducing the time to build these environments to solving this challenge? minutes requires more than just infrastructure. It requires a platform that can support the deployment of fully configured environments. Paradigms like patterns and clones, discussed earlier, are essential to achieve that speed of deployment.

Additionally, these paradigms help with reuse of expertise within the organization as well. Environments capture the expertise needed to deploy various aspects of the application, be it infrastructure (IT team), presentation layer (UI team), business logic (app server team), database layer (DBAs). All the tester has to do is click a button or call an API to deploy the environment. The individual components are automatically deployed based on the definition of each component captured in the environment definition.

Organizations are increasingly turning to the cloud for on- How does this impact running an automated test demand and self-service access test environments, but 6. suite? not every cloud platform is suited for the needs of the Automated testing at each stage can be made more applications in an organization’s portfolio. Innovative efficient and predictable by using consistent, fully organizations evaluate the environment needs of their configured environments that can be deployed using applications, and some of the criteria that we have seen API calls. Continuous Integration and Continuous customers use are as follows: Delivery can actually be realized if the environments of desired size and configuration can be deployed within seconds and minutes versus hours and days. efficiencies in their SDLC that they have traditionally done with environments. Additionally, Skytap customers have full Getting the right data to test with, or setting up the control over their container hosts. This is not available in 7. right data is also a problem area for many teams. many container services, but is often desired by our What strategies or tools help with this? enterprise customers. Customers can run containers on host A few strategies can help with test data management: VMs in their Skytap environments. They can also use the  Populating data in base environments which can Skytap driver for Docker Machine to manage containers then be cloned into new copies of the environment running on hosts in Skytap. If you are interested in learning when needed. Once testing is done, the more about Skytap’s support for containers, please contact environment along and data can be thrown away. [email protected]. It is cheap to recreate a new environment with the desired data set.  Creating a database as part of environment How do you see the use of containers impacting creation. In this model, the data is created by using 10. testing? some automation code. This slows the set-up of the environments but ensures that the data creation is Container technology can affect testing in many ways. based on the latest desired state of the test data Some of these are: set.  Using data virtualization and data-masking tools to  Deploying test environments: Application test continuously keep a close-to-production copy of environments that are entirely container-based or those test data for testing. This is especially important in that use containers for some application components the later stages of testing like pre-prod certification require new ways for deploying the test environments testing. while still meeting the requirements of consistency.  Testing both code and deployment: Containers Are cloud-based environments also a popular encapsulate both the code changes and the deployment process by design. Containers deploy fully choice for those doing continuous integration and 8. configured application components across the testing or continuous testing? pipeline. Testing that traditionally only used to focus on code changes and deployment changes in isolation will Absolutely. The key requirement for CI/CD—speed of need to accommodate both. execution and feedback—cannot be achieved until  Integration with tools across the testing toolchain: Consideration must be given to the choice of build, test environments can be deployed on-demand, in an testing, and automation tools that can work with automated fashion, in a consistent state, and with traditional application components side-by-side with desired complexity. container-based components.

We know containers are a stripped down wrapper 9. around software components or services. How Sumit Mehrotra does Skytap use containers? is Sr. Director of Product Skytap views containers as another component of the Management at Skytap, a role in which he is application stack just like all other components: responsible for product infrastructure, middleware and applications. As strategy and roadmaps. customers modernize parts of their application that are Prior to Skytap, Sumit a good fit for container technology, they are able to use worked at Microsoft in them in Skytap environments, along with the other parts different roles and has shipped a number of of their application that are not containerized. This is a products, including Windows Azure and very common scenario for traditional enterprise Windows operating system. applications that tend to incorporate new technologies Sumit holds an MBA from University of Chicago and are modernized over time. Booth School of Business and a Masters in With containers in Skytap, not only are customers able to Computer Science from Boston University. use the container toolchain with container hosts in Skytap, but they are also able to still achieve the same The 7 Best DevOps Books

From adopting the culture, to implementing Continuous Delivery

By Steve Ropa

ith the relative newness of DevOps, there Book Description: Bill is an IT manager at Parts are not yet a ton of DevOps books. That’s Unlimited. It’s Tuesday morning and on his drive into why we’ve assembled a list of the 7 best the office, Bill gets a call from the CEO. The DevOps books based on four criteria: the company’s new IT initiative, code named Phoenix W Project, is critical to the future of Parts Unlimited, but number of ratings from Amazon, the average Amazon rating, number of ratings from GoodReads and the the project is massively over budget and very late. average GoodReads rating. Both Amazon and The CEO wants Bill to report directly to him and fix the GoodReads use a scale of 1 to 5 stars with 5 stars mess in ninety days or else Bill’s entire department will being the best. be outsourced.

We did all the legwork digging through Amazon and With the help of a prospective board member and GoodReads to determine how many reviews each his mysterious philosophy of The Three Ways, Bill starts book has as well as the average rating on each site so to see that IT work has more in common with that you can quickly find the DevOps book that is just manufacturing plant work than he ever imagined. the right fit for your needs! With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and DevOps Books List effectively serve the other business functions at Parts Unlimited. 1. The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win By Gene Kim, Kevin Behr, In a fast-paced and entertaining style, three George Spafford. luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers 4.6 Average Amazon rating (1,012 ratings). will not only learn how to improve their own IT organizations, they’ll never view IT the same way 4.17 Average GoodReads rating (3,350 ratings) again.

2. What is DevOps? By Mike Loukide 3.7 Average Amazon rating (57 ratings) source code repositories, and other tools. But, as Mandi Walls explains in this Velocity report, DevOps is 3.46 Average GoodReads rating (167 ratings) really about changing company culture—replacing traditional development and operations silos with Book Description: Have we entered the age of NoOps collaborative teams of people from both camps. The infrastructures? Hardly. Old-style system administrators DevOps movement has produced some efficient may be disappearing in the face of automation and teams turning out better products faster. The tough cloud computing, but operations have become more part is initiating the change. This report outlines significant than ever. As this O’Reilly Radar Report strategies for managers looking to go beyond tools explains, we’re moving into a more complex to build a DevOps culture among their technical arrangement known as “DevOps.” staff.

Mike Loukides, O’Reilly’s VP of Content Strategy, Topics include: provides an incisive look into this new world of operations, where IT specialists are becoming part of  Documenting reasons for changing to DevOps the development team. In an environment with before you commit thousands of servers, these specialists now write the code that maintains the infrastructure. Even  Defining meaningful and achievable goals applications that run in the cloud have to be resilient and fault tolerant, need to be monitored, and must  Finding a technical leader to be an evangelist, adjust to huge swings in load. That was underscored tools and process expert, and shepherd by Amazon’s EBS outage last year.  Starting with a non-critical but substantial pilot From the discussions at O’Reilly’s Velocity Conference, project it’s evident that many operations specialists are quickly adapting to the DevOps reality. But as a  Facilitating open communication among whole, the industry has just scratched the surface. This developers, QA engineers, marketers, and other report tells you why. professionals  Realigning your team’s responsibilities and 3. Building a DevOps Culture By Mandi Walls incentives 4.2 Average Amazon rating (20 ratings)  Learning when to mediate disagreements and 3.20 Average GoodReads rating (108 ratings) conflicts Download this free report and learn how to the Book Description: DevOps is as much about culture as DevOps approach can help you create a supportive it is about tools. When people talk about DevOps, they team environment built on communication, respect, often emphasize configuration management systems, and trust.

4. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation By Jez Humble, David Farley

4.4 Average Amazon rating (66 ratings)

Winner of the 2011 Jolt Excellence Award

Book Description: Getting software released to users is often a painful, risky, and time-consuming process. This groundbreaking new book sets out the principles and technical practices that enable rapid, incremental delivery of high quality, valuable new functionality to users. Through automation of the build, deployment, and testing process, and improved collaboration between developers, testers, 4.5 Average Amazon rating (2 Amazon ratings)

4.00 Average GoodReads rating (3 GoodReads ratings)

Book Description: A coherent and actionable DevOps framework is now available to businesses through a revolutionary new book, Next Gen DevOps: Creating the DevOps Organisation. Utilising nearly two decades’ experience at firms including AOL, Electronic Arts (EA) and British Gas’ Connected Homes, the book’s author and pioneer of the DevOps movement, Grant Smith, has distilled the essence of DevOps into an easily-implementable framework. Next Gen DevOps merges behaviour- driven development, infrastructure-as-code, and operations, delivery teams can get changes automated testing, monitoring and continuous released in a matter of hours—sometimes even integration into a single coherent process. The book minutes–no matter what the size of a project or the presents an implementation strategy that allows complexity of its code base. firms large or small, start-up or enterprise to embrace the move to DevOps. Jez Humble and David Farley begin by presenting the foundations of a rapid, reliable, low-risk delivery By presenting a new way to look at the operations process. Next, they introduce the “deployment discipline, Next Gen DevOps challenges the old pipeline,” an automated process for managing all idea of a team languishing at the end of the changes, from check-in to release. Finally, they software development lifecycle, forever context- discuss the “ecosystem” needed to support switching between support tasks, security, data continuous delivery, from infrastructure, data and management, infrastructure and software configuration management to governance. deployment. Armed with the lessons learned from history and the Agile software development 5. Next Gen DevOps: Creating the DevOps movement, combined with the latest in Software-as- Organisation By Grant Smith a-Service (SaaS) solutions, cloud computing and automated testing, Next Gen DevOps sets out Grant’s vision for IT in business’ biggest evolution yet. “Every company is now an internet firm – and that means changes in the way we work,” Grant Smith says. “It’s time to drop the silos between our IT teams and work as organisations to improve and develop our products. Using simple theories and practices, Next Gen DevOps: Creating the DevOps Organisation offers a framework that can transform any internet company.

6. The IT Manager’s Guide to Continuous Delivery: Delivering Software in Days By Andrew Phillips, Michiel Sens

4.2 Average Amazon rating (2 Amazon ratings)

Book Description: Turning good ideas into marketable software quickly is now a business imperative for every enterprise. Delivering software features faster and with high quality is the first critical software improvements their businesses require.

Leading-edge companies like Amazon and Google are applying DevOps and Agile principles to deliver large software projects faster than anyone thought possible. But most executives don’t understand how to transform their current legacy systems and processes to scale these principles across their organizations.

Leading the Transformation is executive guide, providing a clear framework for improving development and delivery. Instead of the traditional Agile and DevOps approaches that step. The subsequent step is to rapidly collect focus on improving the effectiveness of teams, this feedback from users to guide the next set of ideas for book targets the coordination of work across teams further improvements. Critical software development in large organizations—an improvement that objectives such as these set the stage for The IT executives are uniquely positioned to lead. Manager’s Guide to Continuous Delivery: Delivering Software in Days, Instead of Months. Conclusion

The book champions the concept of Continuous DevOps is an emerging methodology that is Delivery in enabling organizations to build automated growing and changing quickly. This relative software delivery platforms for releasing high-quality newness and rapid change make it difficult to find applications faster. The book also presents how great DevOps books. I hope our list has made your Continuous Delivery is a set of processes and search a little easier and that you have found some practices that radically removes waste from the DevOps books you are interested in reading! software production process and creates a rapid and effective feedback loop with end users.

7. Leading the Transformation: Applying Agile and DevOps Principles at Scale By Gary Gruver, Tommy Steve Ropa Mouser Has more than 25 years of experience in Book Description: Software is becoming more and software development more important across a broad range of industries, and 15 years of yet most technology executives struggle to deliver experience working with agile methods. Steve is passionate about bridging the gap between the business and technology and nurturing the change in the nature of development. As an agile coach and VersionOne product trainer, Steve has supported clients across multiple industry verticals including: telecommunications, network security, entertainment and education. A frequent presenter at agile events, he is also a member of Agile Alliance and Scrum Alliance. In his personal time, Steve plays principal trombone in a regional orchestra and is an avid woodworker. https://blogs.versionone.com/ agile_management/author/sropa/ Direct your organization into DevOps The ownership of quality has evolved, don’t get left behind

By Michael Hackett

elcome to our new feature in LogiGear Magazine! automated suites of tests, coupled with the reliance on W We will be doing a column in each issue on current speedy creation of environments makes the broad topics and how to manage, deal with, and support your understanding that everyone, at every single step, owns team through them. the quality of the delivered product—an understanding worth repeating. This first installment of Leader’s Pulse is about making the move to DevOps. This is a large topic and will be covered But, if someone felt like arguing that there is a single over a few magazine issues. What I would like to cover in team that owns quality, I would say it is whoever this article are two topics: high-level mindset topic: the manages the product owners in your organization. The growing and evolving ownership of quality, and a low- usual suspects include but, are not limited to: Director of level details topic of DevOps and its impact on test Development, VP of Engineering and the Product environments and data. Manager. Once a product has a product roadmap and the team is sized, the Dev and Test teams have First, who owns quality? Of course, it’s a trick question— their first set of constraints in quality. the answer is everyone owns quality—there is no single owner of quality. And certainly the Test team does not But that is not the subject for today. own quality alone. Testers may be running more tests and even sets of tests that they do not own—like performance Today’s topic of discussion is the growing and evolving tests but the speed of DevOps, the reliance on fast ownership role in quality. DevOps pushes more people reporting on consistency through running all the into the ownership or quality discussion. The move to Agile in the early 2000s showed unmistakably, that Integration process is that performance or security bugs Developers have a clear and big impact on quality by can be found and fixed earlier when they are cheaper. incorporating code reviews, pair programming, unit If Ops is in charge of environments, cloud infrastructure, testing amongst their many other practices. Test teams containers or whatever virtualized services you are using have their practices/role too: requirements and user story for environments and data—then obviously Ops owns analysis, exploratory testing, design collaboration, test pieces of delivering a quality product. automation, bug finding and reporting among them. Now let’s talk about, great test environments and great The Key to being Agile today, is that most companies test data. have implemented various principles of Lean. Lean Software Development (LSD) outlines 7 principles; one is I’ll start with a story of a client LogiGear has been Quality at Every Step. This is sometimes referred to as helping push their development practices into the new “Build quality in’ or “build integrity in”. It means exactly millennium. what it sounds like: you have to build quality in at every step. This means quality user stories, quality code, quality The team supported a complex system with both unit tests, quality test cases, quality bugs, quality legacy and new products running on different automated scripts, quality performance tests, quality environments, they were also completely integrated environments, quality data, among many other with data. The two most important ingredients in the deliverables at every step. test—the environments and the data were a mess.

One of the things DevOps does is put a spotlight on The builds were “pulled” rather than automated. The automating every one of the reproducible quality environments were managed by the IT team; the data practices e.g. re-running unit tests, re-running the test was old, rarely scrubbed, and seldom mirrored team’s many automated suites. This also means things production data. that were traditionally done at the end of the delivery Due to the data having little integrity the number of process, e.g. performance and security tests, now have “bugs” the Test team found was not even close to to happen much earlier. The decision to move those tests confidence level. There were issues not uncovered by will have an impact and often, a cost. That decision is a the team dealing with the test environments. Meaning, quality assurance decision. The impact of moving security Dev went on wild goose chases only to hit dead ends and performance testing even earlier into the Continuous and throw the “bug” back to test teams—essentially ideals of greater collaboration, immediate feedback, undermining the team’s credibility. Clearly there were greater productivity, automating everything possible, many problems to fix. getting your team the tools and resources to have easy, fast, perfect, production-like environments at all The first problem was the manager of the team was times as well as great data to test against, mirrored, fighting for every ounce of help. He was the only person current, live, whatever you need-the data is, like the on the team who really had an understanding of how environment, great high quality, reliable, predictable, productive, responsive and useful Test teams could be— current and as close as is effective to production data. most of the management team had been in place for Collaborate with the IT/Ops teams to solve these too long and had no idea of the impact test teams can problems. Virtualized environments today are as have on the bottom line, and consequently didn’t want common as automated builds. VMs, cloud, “platforms to spend money on them. Instead, his job was mainly as a service”. Fix these problems; there should be no spent on educating management (re: hitting his head excuses, especially when there are such a big number against the wall), protecting his team, making of tools available to help you. incremental change, and only then could he move on to focusing on day-to-day tasks. Leading the organization and/or test team into the DevOps era is a big task. It starts with a change in Ultimately, the environments and data mess was caused culture. Let’s make sure everyone is on the same page by finger pointing between Dev and IT/Ops, which was with Quality Assurance practices throughout the made worse by management’s not caring or willingness development cycle. Also, making significant, to dedicate funds to fix it. Briefly, we fixed this problem incremental change is the key. We can’t change the by auditing/measuring how many “bugs” and wastes-of- world overnight. Incremental change is the way to go. time problems for Developers and testers were caused Tackling environment and data problems is not easy— by testing on bad environments with bad data. We but it’s a great place to start to get the Test team presented these findings to management, and we did much more productive and trusting their results and not let anyone point-fingers or blame, we explained that reporting on a more consistent basis. the fix lied in making sure time testing had a dedicated IT person, that the hardware needed to be brought into this century, and the build process needed to be automated along with a more detailed set of fixes for data. With my guidance, a decade long nagging problem was completely fixed in under one month.

This was not 20 years ago. Even more alarming—it was a fairly recent client. I am happy to say this team is now in much, much better shape. Everyone is happier—Dev, testers, and management. I hope you do not have these problems.

DevOps—or even Agile for that matter—will not tolerate this. DevOps shines a bright light on environments and data. If your team has an environment or data problems, fix them now. We have known about these issues for a long time—they are gone—we hope—from most organizations—but the solution today is more pressing and luckily, easier.

The reality of DevOps is really the journey towards the

Test in production?

This post is part of the Pride & Paradev series

By Alister Scott

With continuous deployment, it is common to wrong. But the key to dealing with these kind of mistakes is not to lock down the process or extend the release new software into production multiple times a breadth, depth and length of regression tests. The day. A regression test suite, no matter how well solution is to enable people to fix their mistakes designed, may still take over 10 minutes to run, which quickly, learn, and get back to creating value as soon can lead to bottlenecks in releasing changes to as possible.” production.

~ Andy Hume on Real-time QA [The Guardian So, do you even need to test before going live? Why not Developer Blog] just test changes in production?

The key to testing changes as soon as they hit  Test changes in production production is to have real time, continuous real user  Test changes before production experience monitoring. This includes metrics like page views and page load time, which directly correlate to advertising revenue, an incentive to keep these Test changes in production healthy.

The website for The Guardian, the UK’s third largest More comprehensive automated acceptance tests newspaper, deploys on average 11 times a day, of can be written in a non-destructive style that means which all changes are tested in production. they can be run in production. This means that these can be run immediately following a fresh production deployment, and as feedback about the tests is “Once the code is in production, QA can really start.” received, any issues can be remedied immediately “Sometimes deployments go wrong. We expect that; into production and tested again. and we accept it, because people (and machines) go

Test changes before production

There are not many businesses that are able to release software without any form of testing into production: whether there be legislative requirements requiring testing, or the risk of introducing errors is too high for its target market.

Whilst automated regression tests do take longer to run than unit or integration tests, there are ways to manage these to ensure the quickest path into production. These strategies include running tests in parallel, only running business critical tests, only running against the single most popular browser, or only running tests that are directly related to your changes. You can set up a deployment pipeline that runs a selected subset of tests before deploying into production then running the remaining tests (in a test environment). Any of the issues found in subsequent tests are judged to see whether they warrant another immediate release or whether they can be included in the next set of changes being deployed into production.

Whilst you definitely should run tests before deploying to production, it doesn’t mean that this has to drastically hinder your ability to continuously deploy.

Alister Scott is an Excellence Wrangler for WordPress.com at Automattic. He has extensive experience in automated software testing and establishing quality engineering cultures in lean cross- functional software development teams. He lives in Brisbane, Australia with his wife and three sons, and writes a popular software testing blog at watirmelon.com."

Summer 2016

TiD 2016 Conference July 17th - 20th, 2016 Beijing, China Speaking Sessions

Hans Buwalda will be giving the keynote speech at TiD 2016, where the conference will focus on “The next generation of Software Development”. The conference covers industries, including the Internet, mobile Internet, finance, insurance, telecommunication, safety and environmental protection and traditional industry software development.

STPCon- Fall September 19th-22nd, 2016 Dallas, TX Speaking Sessions

Hans Buwalda will be giving his presentation Using Keywords to Support Behavior Driven Development (BDD) and leading the workshop “What Makes Automated Testing Successful?” Michael Hackett will be leading two workshops at STPCon-Fall this year. Move into DevOps: Experiences from the Real world, and Test Case Design Intensive. The Software Test Professionals Conference is the leading event where test leadership, management and strategy converge. The hottest topics in the industry are covered including agile testing, performance testing, test automation, mobile application testing, and test team leadership and management.

Software Testing Club Conference July 16th, 2016 Saigon, VN 2016

LogiGear Senior Manager of Testing Center of Excellence Minh Hoang Ngo will be presenting Test Design With Action-Based Testing Methodology at the Software Testing Club Conference this year, and LogiGear is exhibiting at the conference. The Software Testing Club Conference 2016 is the third conference organized by HCMC Software Testing Club with the co-organization of the HCMC Computer Association. The conference aims to promote best practices in software testing industry and inspire software testing careers. The following are terms either found in the accompanying articles, or are related to concepts relevant to those articles.

DevOps Cloud computing

(Term derived from the words “Development” and The use of computing resources (hardware and “Operations”) A software development practice, software) that are delivered as a service over a grounded in agile philosophy, that emphasizes network (typically the Internet). The name comes close collaboration between an organization’s from the use of a cloud-shaped symbol as an software developers and other IT professionals, abstraction for the complex infrastructure it while automating the process of software delivery contains in system diagrams. Cloud computing and infrastructure changes. It aims at establishing a entrusts remote services with a user’s data, culture and organizational workflow in which software and computation. building, testing, and releasing software happens Source: Tagetik rapidly, frequently, and more reliably. Testing in the Cloud Source: Wikipedia Cloud Testing uses cloud infrastructure for soft- Virtualization ware testing. Organizations pursuing testing in Virtualization (or virtualisation) is the simulation of general, load, performance testing, and produc- the software and/or hardware upon which other tion service monitoring in particular are chal- software runs. This simulated environment is called a lenged by several problems like limited test virtual machine. There are many forms of virtualiza- budget, meeting deadlines. High costs per test, tion, distinguished primarily by computing architec- large number of test cases, and little or no reuse ture layer. Virtualized components may include of tests and geographical distribution of users add hardware platforms, operating systems (OS), to the challenges. storage devices, network devices or other re- Source: Wikipedia sources. EaaS (Environment as a Service) Source: Wikipedia Infrastructure as Code (IaC) Environment-as-a-Service, often referred to as IaaS Plus, extends the traditional IaaS ecosystem Infrastructure as Code is the process of managing into the application development space. Here and provisioning computing infrastructure advanced automation is layered on top of the (processes, bare-metal servers, virtual servers, etc.) existing IaaS instance to not only configure servers and their configuration through machine- processable definition files, rather than physical for a particular application, but also to deploy hardware configuration or the use of interactive and test all the other components needed to run configuration tools. The definition files may be in a a given application. version control system. This has been achieved For example in EaaS, automation is in place that previously through either scripts or declarative will simultaneously trigger a software deployment definitions, rather than manual processes, but developments as specifically titled “IaC” are now and IaaS provisioning workflow to run in parallel. focused on the declarative approaches. Infrastruc- This means that while build packages are creat- ture as Code approaches have become increas- ed, tested, and staged for deployment, the ingly widespread with the adoption of cloud infrastructure tool chain is provisioning the multi- computing and Infrastructure as a Service (IaaS). server test environment being requested by the IaC supports IaaS, but should not be confused with build server to support the current version of the it. application. This level of automation enables Source: Wikipedia organizations to deploy entire application The following are terms either found in the accompanying articles, or are related to concepts relevant to those articles.

environments with the same level of consistency For Continuous Testing, the scope of testing and reliability they have come to expect through extends from validating bottom-up requirements IaaS. or user stories to assessing the system requirements associated with overarching business goals. The Source: Infocus.Emc.com goal of continuous testing is to provide fast and continuous feedback regarding the level of Infrastructure as a service (IaaS) business risk in the latest build or release candi- is a standardized, highly automated offering, date. This information can then be used to where compute resources, complemented by determine if the software is ready to progress storage and networking capabilities are owned through the delivery pipeline at any given time and hosted by a service provider and offered to Source: Wikipedia customers on-demand. Customers are able to self- provision this infrastructure, using a Web-based Continuous deployment (CD) graphical user interface that serves as an IT Continuous Deployment is a software develop- operations management console for the overall ment practice in which every code change goes environment. API access to the infrastructure may through the entire pipeline and is put into produc- also be offered as an option. tion, automatically, resulting in many production deployments every day. Source: Gartner.com With Continuous Delivery your software is always Continuous Integration (CI) release-ready, yet the timing of when to push it A software engineering practice in which the into production is a business decision, and so the changes made by developers to working copies of final deployment is a manual step. With Continu- code are added to the mainline code base on a ous Deployment, any updated working version of frequent basis, and immediately tested. The goal is the application is automatically pushed to production. Continuous Deployment man- to provide rapid feedback so that, if a defect is dates Continuous Delivery, but the opposite is not introduced into the mainline, it can be identified required. quickly and corrected as soon as possible. Continu- Source: Electric Cloud.com ous integration software tools are often used to automate the testing and build a document trail. Because CI detects deficiencies early on in Continuous delivery (CD) development, defects are typically smaller, less Is a software engineering approach in which complex, and easier to resolve. In the end, well- teams produce software in short cycles, ensuring implemented CI reduces the cost of software that the software can be reliably released at any development and helps speed time to market. time. It aims at building, testing, and releasing Source: SearchSoftwareQuality software faster and more frequently. The ap- proach helps reduce the cost, time, and risk of Continuous testing (CT) delivering changes by allowing for more incre- The process of executing automated tests as part mental updates to applications in production. A of the software delivery pipeline to obtain immedi- straightforward and repeatable deployment ate feedback on the business risks associated with process is important for continuous delivery. a software release candidate. Source: Wikipedia

OVER