October 2011 | www.thetestingplanet.com | No: 6 £5 Finding the best tester for the job Testing the tester - a short story about practical recruiting methods for testers - see page 22 The mobile environment By Karen N. Johnson When we’re testing a mobile application, most of our focus is centered directly on testing the application. But there are other considerations surround- ing our applications to consider such as the installation of the application, on-going phone and application upgrades, as well as how our application interacts with other variables in the mobile environment. There are still more variables includ- ing firmware upgrades, memory Changes that are out of this world - what does the future hold for the profession of testing? card space issues and device permissions and settings. If you’ve been in software testing for a long time, these ideas (install, upgrade and interaction The future of software testing testing) seem familiar, even a bit like déjà vu. How would something new such as mobile Part one - Testing in production seem old and familiar? Back in the days of By Seth Eliot ness and productivity. almost certainly be wrong. What we client server application testing, • Software development organisa- can do is look at trends in current we had to think about installa- Hey, want to know the future? OK, tions will dramatically change changes - whether nascent or well on tion testing. In the past decade, here it is: the way they test software and their way to being established practice we’ve been primarily focused how they organise to assure soft- - and make some educated guesses. on web testing where installa- • Testing in Production with real ware quality. This will result in Here we will cover Testing in tion testing is rarely needed. users in real data Centres will be dramatic changes to the testing Production. The other predictions will But in the 1990’s we were a necessity for any high perform- profession. be explored in subsequent editions of focused on desktop software ing large scale software service. Testing Planet. and how gracefully applications • Testers will leverage the Cloud to Is this really the future? Well, maybe. could live and perform in a PC achieve unprecedented effective- Any attempt to predict the future will Continued on page 2 environment. ≠For a stretch of time, I worked on a Windows ALSO IN THE NEWS application and I spent a fair amount of time concerned BUILDING APPS FOR BITBEAMBOT - THE 7 CHANGES SCRUM THE EVIL TESTER about Microsoft DLL’s, I wrote an article about installation test- SOCIAL GOOD TESTING ROBOT HAS MADE TO TESTING QUESTION TIME ing and even wrote a blog post If you were thinking of Can your robot play Angry In the last decade, Agile More provocative advice entirely focused on DLL hell. designing or building a Birds? On an iPhone? methods went from for testers who don’t DLL hell is an actual term (not website, you’d be in luck... Mine can. I call it... quirky and novel to... know what to do! just an experience); take a Continued on page 5 Continued on page 14 Continued on page 29 Continued on page 32 Continued on page 4 2 October 2011 | www.thetestingplanet.com | Use #testingclub hashtag Main story continued from page 1 AUTHOR PROFILE - SETH ELIOT Testing in Production (aka TiP) Seth Eliot is Senior Knowledge Engineer for Microsoft Test Excellence focusing on driving best practices for services and cloud development and testing across the company. He previously Software services such as Gmail, Facebook, and Bing have become an everyday part of the lives of was Senior Test Manager, most recently for the team solving exabyte storage and data process- millions of users. They are all considered software ing challenges for Bing, and before that enabling developers to innovate by testing new ideas services because: quickly with users “in production” with the Microsoft Experimentation Platform (http://exp- platform.com). Testing in Production (TiP), software processes, cloud computing, and other top- • Users do not (or do not have to) install desktop ics are ruminated upon at Seth’s blog at http://bit.ly/seth_qa and on Twitter (@setheliot). Prior applications to use them to Microsoft, Seth applied his experience at delivering high quality software services at Ama- • The software provider controls when upgrades zon.com where he led the Digital QA team to release Amazon MP3 download, Amazon Instant are deployed and features are exposed to users Video Streaming, and Kindle Services. • The provider also has visibility into the data center running the service, granting access to system data, diagnostics, and even user data unto themselves with interactions between serv- But in working with teams at Microsoft, as well subject to privacy policies. ers, networks, power supplies and cooling systems as reviewing the publically available literature on (Figure 3). practices at other companies, 11 TiP methodolo- gies have been identified (Table 1). Examples of TiP Methodologies in Action To bring these methodologies to life, let’s delve into some of them with examples. Experimentation for Design and Controlled Test Flight are both variations of “Controlled Online Experimentation”, sometimes known as “A/B Testing”. Experimentation for Design is the most commonly known whereby changes to the user Figure 1. Services benefit from a virtuous cycle experience such as different messaging, layout, or which enables responsiveness controls are launched to a limited number of un- suspecting users, and measurements from both the It is these very features of the service that en- Figure 3. Data Centres are complex exposed users and the un-exposed (control) users are able us to TiP. As Figure 1 shows, if software collected. These measurements are then analysed to engineers can monitor production then they can TiP however is not about throwing any untested determine whether the new proposed change is good detect problems before or contemporaneously to rubbish at users’ feet. We want to control risk or not. Both Bing and Google make extensive use when the first user effects manifest. They can while driving improved quality: of this methodology. Eric Schmidt, former Google then create and test a remedy to the problem, CEO reveals, “We do these 1% launches where we then deploy it before significant impact from the • The virtuous cycle of Figure 1 limits user im- float something out and measure that. We can dice i problem occurs. When we TiP, we are deploying pact by enabling fast response to problems. and slice in any way you can possibly fathom.” the new and “dangerous” system under test (SUT) • Up-Front Testing (UFT) is still important - just Controlled Test Flight is almost the same to production. The cycle in Figure 1 helps mitigate not “Big” Up-Front Testing (BUFT). Up- thing, but instead of testing a new user experi- the risk of this approach by limiting the time users front test the right amount - but no more. ence the next “dangerous” version of the service is are potentially exposed to problems found in the While there are plenty of scenarios we can test tested versus the tried and true one already in pro- system under test. well in a lab, we should not enter the realm of duction. Often both methodologies are executed diminishing returns by trying to simulate all of at the same time, assessing both user impact and production in the lab. (Figure 4). quality of the new change. For example Facebook • For some TiP methodologies we can reduce looks at not only user behaviour (e.g., the percent- risk by reducing the exposure of the new code age of users who engage with a Facebook feature), under test. This technique is called “Exposure but also error logs, load and memory when they ii Control” and limits risk by limiting the user roll out code in several stages: base potentially impacted by the new code. 1. internal release TiP Methodologies 2. small external release 3. full external release As an emerging trend, TiP is still new and the nomenclature and taxonomy are far from finalised. Continued on page 3 Figure 2. Users are weird - They do unanticipated things But why TiP? Because our current approach of Big Up-Front Testing (BUFT) in a test lab can only be an attempt to approximate the true complexities of your operating environment. One of our skills as testers is to anticipate the edge cases and under- stand the environments, but in the big wide world users do things even we cannot anticipate (Figure 2) and data Centres are hugely complex systems Figure 4. Value spectrum from No Up-Front Testing (UFT) to Big Up-Front Testing (BUFT) Tip to myself: Pradeep, test often, your own ideas and conclusions about what good testing is. You would gain significant advantage over lots of other testers just by doing that because they hardly do it. - Pradeep Soundararajan Follow us at www.twitter.com/testingclub 3 test lab are more restricted in production. Proper METHODOLOGY DESCRIPTION Test Data Handling is essential in TiP. Real user Ramped Deployment Launching new software by first exposing it to subset of users then steadily data should not be modified, while synthetic data increasing user exposure. must be identified and handled in such a way as Purpose is to deploy, may include assessment. Users may be hand-picked to not contaminate production data. Also unlike or aware they are testing a new system. the test lab, we cannot depend on “clean” starting Controlled Test Flight Parallel deployment of new code and old with random unbiased points for the systems under test and their envi- assignment of unaware users to each.