2006 Jon Bach & Quardev Laboratories, Inc

Patterns in Brainstoring 1

Patterns in Brainstorming

Jon Bach

OVERVIEW

As a teaching manager at our company, ideas are our business, so it's useful for me to study where ideas come from. I want to find new ways to keep testers thinking and providing value. I’ve taught classes where I’ve shown testers how to work with ideas systematically (heuristic lists and bug taxonomies are one way to do this), but are brainstorms another useful way to cultivate the discipline of using those tools?

For example, are there types of ideas that generate other ideas? Furthermore, what set of ideas that we can control (because it’s on a list) that, when introduced to somebody, can help them produce useful ideas that they otherwise would not have had?

EXPERIENCE REPORT ------

There are trigger words I can give my testers when they are out of ideas, but I have not yet de-mystified the process of how testers get their ideas and what ideas trigger new ones.

At WTST5, I will present details of an experiment to see how software testers can be trained so that when they are out of ideas, they can trigger themselves (or each other) to think of new ones.

The experiment has four stages:

1) Give the student a feature from a sample application to test, accompanied with a blank piece of paper. Tell them “In the next 15 minutes, I want the best set of ideas you can come up with to test this.” At the ten-minute mark, show them something about the feature.

2) At the end of 15 minutes, change the context completely. Give them a brainstorming exercise, saying: “For the next 5 minutes, I want to think of all the yellow fruit you can name.” Identify and debrief the patterns as a group and print a cheat sheet of those patterns.

3) Hand them another piece of paper and ask them in the next 15 minutes to revisit the ideas they brainstormed with the sample software and then add more that are important and useful. Patterns in Brainstoring 3

4) Ask each tester to deliver only their three best ideas. Debrief the testers as a group. For example, has anyone heard an idea that was a surprise to them? Ask what new ideas that came up *only* when doing the group debrief.

I will bring the before-and-after artifacts of this exercise. It’s my hope that it will provoke discussions around the following questions:

1) What patterns of ideas to testers tend to have when they are first given a problem with a short deadline?

2) What thinking patterns emerge when they are allowed to brainstorm with each other?

3) Is there a particular rubric for evaluating their brainstorms?

4) Is it feasible and relevant to evaluate a tester’s level of energy in thinking of ideas?

In my experience in managing testers, I know that to get them to do good work in a professional capacity, I‘ve got to get their minds into shape.

It’s my hope that I can collaborate with you about how to explore the science of brainstorming and produce a "cheat sheet" of patterns to give students, so when they join the working world as professional testers, they have an active, agile mind and can explain their thinking to stakeholders. Patterns in Brainstoring 4

Part 1

Setting: I gathered 5 testers into a conference room that had a projector.

What I told them: “This is an exercise in brainstorming. I want to know where ideas come from. There are four parts to this exercise. This is the first part.”

With my laptop focused on Firefox, I enabled the Find In This Page feature:

“Take out a sheet of paper and write down as many test ideas that you have for this feature. A test idea is a question you ask of the software in order to reveal a particular behavior. You can not interact with it. You have 15 minutes, but at minute #10 I will reveal something about the feature.” Patterns in Brainstoring 5

Test Ideas:

Tester 1: * I vs “I” * 3 * : * - * Image tooltip text * Text embedded in images * Text in revolving images (banners) * Text in page title * Text in button * Null * Script-generated text * ASP * XML * HTML * HML * “New” Superscript * “Next” Java control

{at this point Jon searched for a word on the page}

* < * HTML://

Tester 2:

* Usability: can my mother use it, does it meet disability requirements * Data dictions: what’s accepted, what’s not, UTF, ASCII, foreign, lengths * Security, activex, open reg keys, hidden files, overflows, can I access pages * Automation * Stress/load: how much can it handle? * UI * Functional * Base vs. New features / regression * Matrix out fucntions * Bangshoe / monkey / ad hoc * Validation * Code step through (whitebox) * International, how does it handle right/left up down arrows in those languages * Degrade good? * Error testing, error msg easy to understand, crash safely, what crashes it? * Help / Documentation Patterns in Brainstoring 6

OS compatibility

* Correct search order * Input method: type vs. mouse vs. voice

Tester 3:

* Use the MSN Client Highlight viewer test cases * Enter strings with # symbols, cpas, Lc, spaces * Keyboard only * Mouse only (onscreen keyboard) * Characters in other alphabets (Cyrillic, Asian fonts) * “Match case” checked & unchecked, repeat #2 * Find Next @ end of page * Find Previous @ beginning of page * Copy/Paste into search box * Highlight – colors? * Finds things in headers/footers/captions? * Parts of words? * Put characters in user entry boxes (like search or stock symbol field on MSN Home page) then search for them … does Find In This Page pick up user input here? Should it? * Finds things in address book / frames? * Finds things in ads? * Play with focus * Rotating content … finds something in a headline… how does it cope if the headline changes?

* Can it highlight more than one instance of the string on the page? * Grab source code from a picture, copy into Find box, see if picture gets highlighted * Words in green… pictures in green also? * Red x ... what does it do? Clear green highlight or stop current search? * How to disable / get rid of green highlight? * Go to another page… find value persists? Should it? * Is there a Help entry? Docs? Do they make sense? * Distinguish bold / italic / font? * Find hyperlinks? * Handle frames? * Pop-ups searchable?

Tester 4:

* (match case) Patterns in Brainstoring 7

* Checkbox on/off * Highlight (button) when enabled? Does it highlight spaces? * Find previous (button) works with spaces? * Find Next (Button) * Find (text box) * Exit button Tests: * Can I search local docs in Firefox? * Can I search page source? * Can I search text in ads, buttons, images, etc? * Can I move the position of the search bar? * If I load a different page will the search items automatically highlight? * Is there copy/paste functionality? * Can I use symbols or other languages? * Can I use different text formats (like UTF8) * Does Find integrate with Google search? * Can I drag and drop a file into that text box? * Can I drag buttons?

* Can I modify the output on the bar by clicking within the text? * Should this work with keyboard shortcuts? Do they appear on mouseover? * What happens if I double-click any of the buttons? What about right-clicks? * Is there Help for this tool? Does help spec match usage? * Does highlight color change if text background different color? (like green)

Tester 5:

* Normal text in Find * Numbers in Find * Alphanumeric in Find * DBCS characters in Find * RTL language sets in Find * SQL query text in Find * XML text in Find * C++/C/C# code in Find * Mutated strings in Find * Copy different file strings from Find * Filepath text to DLL’s on system * Highlight when all strings same for long strings * Longest string copy/paste to Find * Resize works/doesn’t Patterns in Brainstoring 8

* Test on different browsers * Find DLL where Find bar & on system – disassemble * Match Case on/off for all text * Find existing result on page * Find non-existing result on page * Test against different pages: XML HTML CSS PHP PH3 ASP P4 * Perl string text in Find * Python text in Find * Find against language pages on long sites using language keyboard * Keyboard API attacks * Automation – C#, Echo/VB/VBA/Astolt/Visual Test * Attempt install in different browsers * Unix feature? Linux feature / Mac feature * Find Next, Find Previous * Tab order / accessibility * High Contrast – standard/non-standard * Themes / All

* Appearance w/ no image tag settings in browser * Reach end of page * Text localized / internationalized -- L10N * Attack surface * Bin scope * App Verifier * FX Cop * File fuzzing * Sound Sentry * Show Keys * Text on page is f l compliant * Change browser settings for text display – largest/large/medium/small/smallest * Wireless network install * IPX/SPX * Port shutoff effects * Pull cable * Hard shutoff / power out effects * Install bad apps * Trojan * Fake virus * Setting Patterns in Brainstoring 9

* Anti-virus effects * Anti-phishing effects * Firewall effects * Security tool effects * VPN Patterns in Brainstoring 10

Part 2

Exercise: Name all of the fruit you can name that are yellow. You have 5 minutes.

Tester 1:

Lemon Banana Star fruit Apples Mango Peppers Tomato Pears Pineapple Salmon berry

Tester 2: (No data)

Tester 3:

Apples Pineapple Bananas Squashes (fruit?) Tomatoes (yellow peer tomatoes) Pears Grapes / raisins Persimmons – sometimes There is a type of yellow berry – kind of tart – argh… don’t remember name Mangoes Lychee Pomegranite Apricots Plums Starfruit Melons – honeydew, cantaloupe Fruit in that “Perelandra” book (CS Lewis)

Tester 4: Lemon Banana Star fruit Grapefruit (?) Apple Pear Cantaloupe Patterns in Brainstoring 11

kiwi Is a fruit considered yellow because of inside or outside coloration? Does seasonal coloring count? (Including process of going bad) What about families of fruit? What about genetically altered fruit?

Tester 5: Bananas Pawpaws Yellow apples Yellow tomatoes Yellow mangoes Pineapples (inside) That starchy thing they grow in Kenya, on that plantation, you know, the one that’s been in the news and was featured on Mother Earth News in the same issue as the one with the solar still and the mini truck that was adapted from the golf cart, come to think of it, that thing was yellow also, or was it the one with the other article about the hovercraft that was written by Diane’s dad that he built in the Moses Lake area and tested, the one that he and Yellow cucumbers

------DEBRIEF: 10 minutes ------

How did people get their ideas? (Group feedback)

* Imagined grocery store (visual order trigger) * Taste (memory trigger) * Shape (visual congruence trigger) * Go through the alphabet (systematic trigger) * Varieties (the what’s-something-totally-different trigger) * Look at your own list (affinity trigger)

I showed them results from previous experiments I have run with different audiences. Patterns in Brainstoring 12

Part 3 – What Else Can You Think Of? (15 minutes)

Exercise: “Now draw a line under your original ideas for Find In This Page and think of more ideas”

Tester 1: [nothing written]

Tester 2: Blindfold testing

Tester 3: A password field – masked. Finds string of asterisks on the password itself?

Tester 4: Does search pick up on words used within the browser’s UI or the URL?

Tester 5: Test on different brands of computers (Dell Gateway Clone, wackjob custom boxes) Overdocked processor > 2 G ram on box Shrink RAM to smallest Shrink HD space to smallest Yank USB Yank Mouse cord Yank KBD Wireless KBD Wireless mouse Patterns in Brainstoring 13

Part 4 – Three Best Ideas (15 minutes)

Exercise: “Now give me your three best ideas” (Group feedback)

Tester 1: Rollover text in images Text inside buttons Script-generated text

Tester 2: Usability Data Dictionary Functional

Tester 3: Site with a password page, does it find asterisks Find an archived site with messy frames (geocities) Find a rotating banner text (e.g. stock ticker)

Tester 4: Symbols Text format (UTF8) Drag and drop

Tester 5: Tab order: accessibility JPN internationalization SQL strings