Focus Groups in HCI: Wealth of Information or Waste of Resources?

Stephanie Rosenbaum (Organizer) Gilbert Cockton Kara Coyne Tec-Ed, Inc. University of Sunderland Nielsen Norman Group +1 734 995 1010 +44 191 515 3394 +1 646 613 1122 [email protected] [email protected] [email protected] Michael Muller Thyra Rauch Lotus Research IBM +1 617 693 4235 +1 919 224 1861 [email protected] [email protected]

ABSTRACT SUMMARY OF EACH PANELIST’S POSITION Many HCI professionals frown on focus groups, while Stephanie Rosenbaum some believe focus group methodology can be successfully Focus groups can be an effective tool for collecting applied to collect usability data. This panel features usability data, if we apply a methodology and guidelines interaction among HCI professionals with very different designed for usability focus groups rather than marketing experiences and opinions. ones. Our primary distinction is that usability focus groups are task-based, so that we can observe the participants’ Keywords actual behavior with products. After an initial group Focus groups, methodology, participatory design, discussion, we typically “break out” into smaller clusters of ethnography, market research, usability two or three people who explore a product in a co- OVERVIEW OF PANEL TOPIC AND FORMAT discovery manner (each cluster with a facilitator), then we Focus groups, with their roots in market research, are also reconvene the full focus group for discussion of the being used in human factors and HCI programs. But this participants’ experiences. Usability focus groups are use is highly controversial; some HCI professionals especially valuable when an organization needs to increase discourage dependence on focus group data for design limited knowledge of its target audiences. Focus groups are decisions. Others continue to conduct focus groups, not well-suited for comparative, competitive, or bench- presumably because we value the data we collect from marking studies, because the co-discovery methods don’t them. Are we succumbing to the influence of marketing produce individual performance data—although focus departments, or have we identified worthwhile applications groups can include individual questionnaires to collect of focus groups to HCI? Our panel addresses these issues: quantitative preference data. One often expressed objection What role—if any—should focus groups play in HCI? to focus groups—concern that extroverted or single-minded participants will bias the data—can be addressed by skilled How much can we trust data from focus groups—and facilitators using good practice, just as they do in contextual what can we do to improve its validity? inquiry, ethnographic interviews, and other qualitative When and why can focus groups be valuable in the methods. product design and life cycle? Stephanie Rosenbaum is founder and president of Tec-Ed, How can focus groups observe behavior as well as Inc., a 15-person firm specializing in usability research and opinions? information design. Stephanie was awarded a Millennium What can HCI professionals do to make focus groups a Medal in 2000 by the IEEE; she recently contributed a more effective method? chapter to the Copenhagen Business School Press volume, Software Design and Usability. The panelists first discuss what’s different about focus groups in HCI. Then the panelists—and the audience— Gilbert Cockton describe their own experiences, telling what worked and I have designed HCI curricula for 17 years. I have included what didn’t in HCI focus groups. mentions (and no more) of focus groups for the last 9 years. I have designed commercial interactive systems over a period of 16 years, including one featured at an Copyright is held by the author/owner(s). international design festival. I have never used a focus CHI 2002, April 20-25, 2002, Minneapolis, Minnesota, USA. group during systems design. I have been using grounded ACM 1-58113-454-1/02/0004. research approaches to interactive systems design for the last 7 years. I have never considered focus groups and their

702

outputs as a source of useful or valid design data. I have the Usability Professionals’ Association (UPA) conferences directed a regional support project for internet and in 2000 and 2001. multimedia companies for the last 3 years. We have never Michael Muller offered, or been asked for, support for the use of focus Traditional focus group methods may tend to extract or groups in the design of interactive media. Focus groups are “mine” knowledge from users rather than to engage them as useful marketing tools, but marketing isn’t just about partners with their own skills and knowledges. In contrast, product design, it’s about the 4 Ps of the marketing mix recent work of our colleagues has transformed focus groups product, price, place and promotion. Focus groups can thus into HCI methods of considerable power, precision, and identify product opportunities and market opportunities, but innovation. Butler’s (Rettger’s) Usability Roundtable, they cannot provide the reliable and detailed data needed to Sato’s and Salvador’s Focus Troupe, and Sanders’s properly ground a product design in its intended context of Strategic Design Workshop inform traditional focus group use. Interactive systems design requires a large range of methods with insights from ethnography, participatory detailed customer data. It is not clear that focus groups can design, dramaturgy, market research, and privacy concerns. provide this more reliably, more cheaply, or more quickly As argued by HCI advocates of focus groups since 1991, than established social science methods (observation, focus groups should be used as “complementary… and interview, questionnaire, diary) and especially not pure HCI converging methods” (Sullivan) - “introductory session[s], approaches such as participative prototyping. laying the groundwork for additional field research” Professor Gilbert Cockton is Research Chair in Human- (Butler) in combination with other necessary methods that Computer Interaction in the University of Sunderland. He can help us work with users to innovate products and work has served in numerous positions for B-HCI-G (the British practices. HCI Group, a specialist group of the British Computer Michael Muller, a research scientist at IBM Research, Society), IFIP, and ACM SIGCHI and will be co-chair for recently completed a chapter on participatory methods as CHI 2003 in Fort Lauderdale. He received IFIP’s Silver hybrid or “third space” practices between the world of users Core Award in 1998 and is a Fellow of the Royal Society of and the world of software professionals, for a new Arts and the British Computer Society. handbook of HCI. Kara Pernice Coyne Thyra Rauch We do not question the merit of focus groups for market Usability focus groups can be used for a variety of purposes research, only the too common usage of them as a usability at all stages of the development process and provide one evaluation method. A group setting could be less threaten- mechanism for gathering feedback from small groups of ing than a lab, and a group setting may stimulate people. Based on the desired goal, they can be used to spontaneous discussion. However, drawbacks of focus generate and vote on ideas, capture and validate user roles, groups include groupthink, the moderator’s potential effect, tasks, and workflows, and validate high-level strategy. and interpretability of results. And, according to marketing Depending on resources available, they may also be used to experts: The method is better suited for the generation of do walk-throughs of designs, prototypes, or code. They can ideas, rather than systematic analysis of well-structured be used from the earliest conceptual discussions to alternatives. Many usability professionals agree that the summary sessions after running usability tests of the best way to learn about usability is to watch users work, at finished product. Small groups (6-8 people) usually provide their own site or in the lab, and analyze their behavior. No the best interaction, but running larger groups is possible matter how smart or quick the user is, you, a designer can with multiple trained facilitators. As with other usability always do this better than they can, because users do not methods, the quality of the data obtained from usability remember everything they do or did, and are often too focus groups is only as good as the quality of the participant forgiving. Participants frequently want to please the facili- selection and questions asked. And, while there are tator, product designers, or even other focus group mem- certainly drawbacks to this methodology, working within bers. (And groupthink can lead to bias, self-censorship, the limitations provides an alternative way to gather data pressure, and rationalization.) Even skilled facilitators have quickly and inexpensively from a large number of users, trouble coaxing valid comments from the meek, and stifling particularly in such settings as professional tradeshows. the dominating members of the group. Also, what looks good, fun, or pleasing is not always easy-to-use. Users can’t Thyra Rauch, an IBM’er for fifteen years, currently works tell this until they actually try to complete tasks. in the Tivoli Software division as a customer research architect. She has been instrumental in validating product Kara Pernice Coyne is the Director of Research at Nielsen and UI direction, user roles, and role/task pairings across a Norman Group; she led the research study about how number of different industries for multiple product journalists use the Web, and co-authored “Designing solutions. Thyra is an active member of the Usability Websites to Maximize Press Relations.” Previously, Coyne Professionals’ Association and is currently the treasurer on established successful usability programs at Lotus its board of directors. She has a Ph.D. in experimental Development, Iris Associates, and Interleaf. Coyne chaired psychology from North Carolina State University.

703