Audience Response Systems As a Data Collection Method in Organizational Research

Audience response systems as a data collection method in organizational research

MATTHEW W MCCARTER College of Business, University of Illinois at Urbana-Champaign, Champaign IL, USA

ARRAN CAZA College of Business, University of Illinois at Urbana-Champaign, Champaign IL, USA

ABSTRACT Audience responses systems are electronic devices allowing audience interaction and they are increasingly being used in educational and business settings to enhance various pedagogical and practical processes. This paper discusses how ARS technology may be used as a method of collecting data for research purposes. Specifically, this paper demonstrates ARS technology’s potential utility by duplicating findings from two organisational studies, it discusses how ARS technology may be used to address three prevalent data collection problems, and it suggests how ARS technology may provide scholars with increased access to certain organisational settings, as well as greater integration between research and service activities.

Keywords: audience response system, personal response system, audience response technology, data collection techniques, data collection technology, organizational research methods

INTRODUCTION discuss how they are traditionally used. Second, ata collection is an important element of we duplicate findings from two organisational Dany research methodology. As scholars strive studies to demonstrate how ARS technology may to understand behavioural phenomena in organi- be used to test theory. Third, we discuss how sations, the need for data collection methods that ARS technology may be used to address three are convenient to both scholars and respondents prevalent data collection problems. Fourth, we and which may be used to gather data from a wide discuss other issues related to using an ARS for variety of organisational settings becomes impor- research, including limitations and benefits, such tant. This is especially true given evidence suggest- as increased access to certain organisational set- ing that individuals are becoming less inclined to tings and a greater integration between research participate in organisational research (Bryman and service activities. 2000). The purpose of this paper is to discuss how audience response system (ARS) technology may AUDIENCE RESPONSE SYSTEMS be used to collect data for research purposes. An audience response system is an electronic This paper is in four parts. First, we review device designed to allow immediate interaction the literature on audience response systems and between an individual presenter and a large audi-

122 JOURNAL OF MANAGEMENT & ORGANIZATION Volume 15, Issue 1, March 2009 Audience response systems as a data collection method in organizational research ence. An ARS typically has two parts. The first attendance, and engagement appear to increase component is a remote control (or ‘clicker’) that when ARS technology is used as compared to audience members use to respond to questions. when it is not (See Fies & Marshall 2006, for a The second component is an electronic receiver review of empirical evidence). (or ‘hub’) that records, and optionally, displays individuals’ responses. An ARS allows for a large ARS technology in organisations number of individuals to respond simultaneously ARS technology is used in organisational settings, (e.g. 1,000 people). Each individual response is as well. There is a growing body of literature in recorded by the hub and can be displayed via pro- the trade and management press advocating ARS jector or exported as a data file for use in other use. A basic search of the EBSCOhost database software. For example, a presenter during a board using the phrases ‘audience response system’ and meeting could use an ARS to present a multiple- ‘personal response system’ found 105 pieces in the choice question to a group of partners, have each practitioner and academic press (search conduct- partner choose an answer, and then immediately ed February 24, 2009). Examination of these display the number selecting each answer. pieces showed ARS use in a variety of organisational contexts. For example, Krantz (2004) ARS technology in education reported on marketing organisations using an ARS technology was originally designed as a ped- ARS to receive feedback from potential customers agogical tool to enhance student learning in ele- at tradeshows. Hatch (2003) described the use of mentary, middle- and high-school, university, and ARS technology in such activities as strategic post graduate settings (Fies & Marshall 2006). planning, brainstorming, monitoring training Caldwell (2006) documents the use of ARS tech- effectiveness, and ice-breaking. ARS use has also nology in fifteen different disciplines of higher been reported to increase organisational efficiency education (e.g. medicine, physics, psychology; and effectiveness in decision making and team also see Duncan 2006). The growing use of ARS planning meetings (Training and Development units reflects the many benefits they can provide 2006). Organisations that have adopted ARS for teachers and students. For example, Homme, technology include Boeing, Academy of the US Asay, and Morgenstern (2004) reported an Federal Bureau of Investigation, IBM, John increase in attendance and enthusiasm of resident Deere, McGraw-Hill, National Academy Founda- doctors using an ARS for board review sessions. tion, Prentice-Hall, Raytheon, Toys ‘R’ Us, Unit- Similarly, Miller, Asher and Getz (2003) found ed States Army and Navy, Walt Disney World, that participants using ARS units reported greater and YMCA (e.g. see http://www.einstruction.com presentation quality, speaker ability, and attention for a list of firms using their particular ARS unit). during professional education programs when The diversity of these organisations suggests the compared to participants not using an ARS. Oth- broad-based adoption and use of ARS systems by ers have reported that use of an ARS benefits a organisations. range of classroom activities, including collecting demographic information, allowing students to ARS technology in academic research share knowledge and experiences pertinent to Despite the growing use of ARS units in academ- course content, polling student opinions on vari- ic institutions, there has been little consideration ous academic and public policy issues, testing of how an ARS may be used to gather data for comprehension of course material, giving in-class research purposes. Some authors have men- quizzes, and facilitating group discussion (Byrd, tioned the possibility, but most have not subject- Coleman & Werneth 2004; D’Arcy, Eastburn & ed the idea to rigorous consideration (e.g. Mullally 2007). Overall, student involvement, Gamito, Burhansstipanov, Krebs, Bemis &

Volume 15, Issue 1, March 2009 JOURNAL OF MANAGEMENT & ORGANIZATION 123 Matthew W McCarter and Arran Caza

Bradley 2005; Draper, Cargill & Cutts 2002). A Two fundamental hypotheses they tested in a notable exception is Bunz (2005), who compared series of studies were: data collection using machine-readable forms Hypothesis 1. The proportion of volunteers and an ARS. The results showed no difference will decrease as the size of the group increases. between methods in time pressure effects on responses. In addition, students were more Hypothesis 2. The proportion of volunteers engaged in answering questions through the ARS will decrease as the payoff for volunteering and found the ARS no more or less difficult than decreases. traditional forms. Overall, Bunz (2005) supports the use of ARS technology as a data collection Hypothesis 1 reflects the logic of free riding method, highlighting the benefit of quick, elec- (Olson 1965), since many volunteer situations tronic data storage, and suggesting that because require only one party to sacrifice (i.e. volunteer) many university and businesses are adopting before the entire group benefits. The presence of ARS units, their availability to scholars and many potential volunteers makes each person less response participants is increasing. willing to volunteer. Hypothesis 2 is based on the assumption that volunteers incur costs that non- USING AN AUDIENCE RESPONSE volunteers do not (Diekmann 1985). As volun- SYSTEM TO TEST THEORY teering becomes increasingly costly, individuals One important limitation in Bunz’s (2005) study are less likely to volunteer. was it did not investigate any behavioural phe- Murnighan and colleagues (1993) ran four nomena or theoretical relationship among con- experiments in a mixed-hybrid repeated meas- structs. Because of the study’s focus on ures design. The first experiment included respondents’ competence and ability to use com- Hypotheses 1 and 2 and the next three experi- puter-mediated technology (Bunz 2005), it did ments replicated these findings and made various not consider whether substantive research find- extensions. The results supported both hypothe- ings could be influenced by ARS technology use. ses: as group size increased, the proportion of In this section, we demonstrate how ARS tech- volunteers decreased; and as the payoff for volun- nology may be used for theory testing. teering decreased, the proportion of volunteers Portions to two organisational studies were decreased. There were no interactions predicted duplicated to show that ARS technology is a or observed. viable and unbiased method for collecting data. The first replicated study concerned volunteer Study 2: The assurance dilemma behaviour (Murnighan, Kim & Metzger 1993). Franzen (1995) used a repeated-measures The second addressed group size effects in collec- between-subjects design to test the effects of tive action (Franzen 1995). These studies are group size on various collective action problems. described below. The hypotheses are summarized Group size was hypothesized, and subsequently briefly, the replication method is detailed, and shown, to reduce cooperation rates in situations then the ARS results are compared with the orig- where mutual cooperation always yielded higher inal findings. payoffs than unilateral or mutual defection. Dilemmas with such a payoff structure are called Study 1: The volunteer dilemma ‘assurance’ dilemmas because coordinated action Murnighan and colleagues (1993) drew on game gives higher payoffs only if all actors cooperate. theoretical, organisational, social psychological Therefore, each party wants to be assured of how and evolutionary perspectives to develop a vari- the other party will act (Sen 1985). For exam- ety of hypotheses about volunteer behaviour. ple, in a union wage dispute, each worker would

124 JOURNAL OF MANAGEMENT & ORGANIZATION Volume 15, Issue 1, March 2009 Audience response systems as a data collection method in organizational research prefer to walkout and lobby for a pay raise participants for both studies. All were third or (Heckathorn 1996); however the walkout strate- fourth year students of various educational back- gy is only attractive if the worker is assured that grounds. The majority of participants had work everyone else will also walkout. Therefore, experience in organisational settings, and all were Franzen (1995) predicted that: familiar with the ARS model being used. Informed consent for using data gathered from Hypothesis 3. Cooperation rates in assurance the ARS technology was acquired during the first dilemmas will decrease as the number of play- lecture of the semester. ers increase. Upon entering, participants were welcomed Group size has a negative effect on coopera- and told they were going to participate in two tion rates in assurance dilemmas because, as the exercises using their ‘clickers.’ Students were first group gets larger, the probability that someone asked to press option ‘A’ on their clicker to assure will fail to cooperate increases (Franzen 1995). that the units were registering with the ARS hub. The fear that at least one person will not cooper- The first of six scenarios was then presented to ate leads others to do the same in self-defense the participants via an electronic projector. Sce- (Liebrand 1983). Franzen’s (1995) results sup- narios were presented in random order. ported the hypothesis; cooperation rates fall as A volunteer dilemma scenario was presented the number of players increased. first, which was adapted from the volunteer dilemma game used in Experiment 3 of Data collection using an ARS Murnighan and colleagues’ (1993) study. This The ARS used in this research was the i-clicker scenario involved a telecommuting work organi- (see Barber & Njus 2007; D’Arcy et al. 2007, sation where the manager needed only a certain for reviews and detailed descriptions). The i- number of staff (e.g. 10 people) to work extra clicker ARS has a hand-held, five-option unit hours on a project for a fixed sum of money (e.g. for respondents (e.g. choose A through E) and $100). However, the workers were unable to talk an electronic hub that collects all respondent to one another, and the task was such that any data. This particular ARS offers the option to excess workers would not be needed or paid. display frequency histograms of responses, Because the work was telecommute-based (i.e. which individuals have responded, and what done remotely), the workers had to complete the each individual response was. Data stored in the task before learning whether they were needed or hub include the breakdown of responses for would be paid. The participants in the experi- each question posed (including the text of the ment took the role of potential workers and question itself), a question-aggregate spread- made the decision to work or not. sheet of how each person responded to a given After reading the scenario aloud, the instructor question, and a person-aggregate spreadsheet of activated the ARS hub to receive responses. Par- how each person responded to various questions ticipants were asked to make their decision. The posed on various dates.1 ARS hub displayed a count of the total number Thirty-three undergraduate students (12% of responses. No time limit was set, but the time female) enrolled in an introductory business required for all participants to respond was less course at a large public university served as the than 40 seconds in every scenario. Once all par-

1 Of course, the researcher should always take caution as to protecting the identity and privacy of the participants using ARS technology. ARS units have the option of displaying participant responses either at the individual or group level and are designed to link responses to individual clickers while still maintaining the privacy of a participant to the other participants and researcher (if desired). In conducting both studies in this paper, the i-clicker was setup so that individual confidentiality was preserved, and only the researcher knew which clicker corresponded to an individual participant.

Volume 15, Issue 1, March 2009 JOURNAL OF MANAGEMENT & ORGANIZATION 125 Matthew W McCarter and Arran Caza

ticipants had responded, the instructor displayed ings consistent with the original studies’ results, the next scenario and repeated the procedure with despite the use of an ARS for data collection. three more volunteer dilemma scenarios. These This suggests that the ARS technology did not scenarios were similar to the example above, introduce methods bias, and that ARS is thus a except for variations in the number of workers potentially useful method for collecting data to needed (10 or 100) and the payment to those test theoretical predictions. whose work was used ($100 or $1000). Following the volunteer dilemma scenarios, AUDIENCE RESPONSE SYSTEMS AND the instructor then presented a series of assurance THREE DATA COLLECTION PROBLEMS dilemma scenarios with payoffs similar to In this section, three prevalent data collection Franzen (1995). The scenarios asked partici- problems are reviewed. These challenges are the pants to imagine themselves as representatives of cost–response problem, the large sample-size various student organisations of the university problem, and the data-entry error problem. We and that they were all involved in a conference then discuss how ARS technology may resolve meeting where students may use their clickers to each of these problems. vote on resource allocation decisions for campus events. The procedure of asking for responses Cost–response problem and waiting until all participants had responded Most traditional data collection methods (e.g. in- was used again. A second volunteer dilemma sce- person and mail surveys) require a tradeoff nario was then read, identical to the first except between cost of implementation and rate of for the number of student organisations involved response (Sproull 1986). Mail surveys make data in the activity (40 instead of 2). collection relatively inexpensive, but generate low response rates. In contrast, laboratory experi- Duplication from using an ARS ments and other in-person approaches enjoy The pattern of ARS-collected responses duplicat- higher response rates, but typically incur greater ed those reported in both of the previous studies. costs in terms of logistics and time. As in the original (Murnighan et al. 1993), vol- Recognizing these tradeoffs, Sproull (1986) unteer payoff had a significant positive effect on suggested that any new data collection method volunteering behaviour. Categorical data analysis intended to alleviate the cost–response problem showed that twelve percent of participants volun- should have three features. First, the method teered in the low payoff condition compared to must be accessible to respondents. E-mail is a 32% volunteering in the high payoff condition good example of an accessible method, since it is 2 (␹ 1 = 6.17, p < 0.05). Group size had a signifi- a very common means of communication in cant negative effect on volunteering behaviour, organisational settings, and one that Sproull with 12% volunteering in the large group condi- (1986) advocated using to administer survey and tion and 32% volunteering in the small group interview questions. The second criterion is that 2 condition (␹ 1 = 6.17, p < 0.05). There was no the proposed new method must engage respon- 2 evidence of an interaction (␹ 1 = 0.01, p = 0.93). dents during data collection. Doing so would Similarly, in the duplication of Franzen (1995), a address the primary challenge of mail surveys, McNemar test was significant (Siegel 1956), which tend to have low response rates because of showing that group size had a significant nega- the respondents’ lack of engagement (Sproull tive effect on cooperation rates: 85% cooperation 1986). Finally, the new method must produce in the two-player case versus 52% cooperation results comparable to those collected by tradi- 2 with 40 players (␹ 1 = 6.67, p < 0.01). tional methods. That is, no method bias should Taken together, these two studies yielded find- be introduced.

126 JOURNAL OF MANAGEMENT & ORGANIZATION Volume 15, Issue 1, March 2009 Audience response systems as a data collection method in organizational research

Large sample size problem To address the data-entry error problem, schol- A second data collection problem facing scholars ars have posed a variety of methods. The three is attaining sufficiently large sample sizes. Small most pertinent to our discussion are double data sample sizes may suffer from low statistical power entry, random selection data checks, and using (Cohen 1988), generate imprecise estimates (Ped- electronic technology for data collection. Double hazur & Schmellin 1991), and have a risk of weak data entry involves having all data entered inde- statistical conclusion validity (Stone-Romero pendently by two persons. The two data sets can 2002). The obvious solution is to increase sample then be compared for differences. This method will size. However, acquiring a large sample typically alleviate data entry error, but it is potentially costly leads to greater costs in time, money, and effort in time and labor (James 1990). Random selection (Shadish, Cook & Campbell 2002). As such, the data checks are similar, but use only a subset of the large sample-size problem is linked with the cost- data: one person enters the data and then another response problem; increasing sample size usually chooses a random sample of entered data points to means expending more resources. compare to the original data. This approach is less In laboratory studies, for example, researchers costly, but also more fallible. A more recent solu- can attain a large sample size, but doing so typi- tion is to use electronic technologies to eliminate cally requires running many waves of partici- the risk of manual data entry error (Hansen & Hill pants. Even the recent solution of using 1998). Current best practice favors this electronic computers to collect lab data can still be prohibi- approach, using such devices as computers and the tive in terms of time costs. Studies using mail or Internet to address data entry error (e.g. Rogelberg e-mail for data collection similarly use multiple et al. 2002; Scandura & Williams 2000). waves of requests and reminders to solicit higher response rates and thereby attain larger sample Using an ARS to address these three sizes (Dillman 2007). Many behavioural scien- problems tists have begun using online surveys as a way to ARS technology can address all three data collec- decrease material costs but often continue to face tion problems. First, ARS technology fulfills the issue of low response rate (Rogelberg, Sproull’s (1986) three requirements for overcom- Church, Waclawski & Stanton 2002). ing the cost–response problem. Since many educational and business organisations are adopting Data entry error problem ARS technology (Duncan 2006; Fies & Marshall Data-entry error has been an ongoing research 2006), the accessibility and convenience of ARS concern for decades (e.g. Smith 1967; Czaja, use is increasing (Draper et al. 2002). ARS units Sharit, Nair & Rubert 1998). Data-entry errors also meet Sproull’s (1986) criterion of participant may occur in any situation where responses engagement: respondents in both education and recorded in one format are transferred to another business settings consistently report higher (e.g. typing hardcopy into a computer; Feng engagement and attention when using ARS tech- 2004). At best, such errors increase research costs, nology than when not (e.g. Byrd et al. 2004; because finding and correcting errors requires Miller et al. 2003). Sproull’s (1986) third criteri- time and effort (James 1990). However, unrecog- on was that the new technique does not intro- nized data-entry errors may have far more devas- duce method bias. The results we presented tating effects, leading to unintentional above, duplicating previous findings, suggest that misrepresentation of results, false conclusions, ARS technology does not introduce method bias. and misdirection of subsequent investigation The second data collection problem, attaining (Rosenthal 1994; Starbuck 2004). Data-entry large sample sizes, may also be addressed through error is a serious source of concern for researchers. ARS technology, because it makes data collection

Volume 15, Issue 1, March 2009 JOURNAL OF MANAGEMENT & ORGANIZATION 127 Matthew W McCarter and Arran Caza

from such large groups both convenient and effi- and Development 2006). Both examples suggest cient. At the authors’ university, for example, there opportunities to use ARS technology for the study are typically two annual sections of the Introduction of intact organisational groups in a way that is to Organisational Behaviour class, each with more seamless, convenient and unobtrusive (also see than 550 students. An ARS is frequently used in Webb, Campbell, Schwartz & Sechrest 2000). If a this class to gather survey questionnaire data on team or department is already using an ARS in attitudes, preferences, and behaviour responses to planning, researchers could easily access that data complex decisions. A key advantage of the ARS and integrate items of their own. Doing so offers technology is that data collection and entry for many advantages, in terms of validity, simplicity, even 1,000 respondents takes no longer than for a non-invasiveness, and ability to give back to group of 30. Of course, ARS units will be most research participants (Eden 2003). useful with respondents who are already familiar These advantages are equally relevant in tradi- with the technology, but Bunz (2005) reports that tional student-based research, which offers the respondents unfamiliar with ARS units can be possibility of creating closer integration between taught to use the technology reliably in 10 minutes. teaching and research. There is a long history of The third data collection problem, data-entry viewing teaching and research as conflicting error, is eliminated by ARS units because all demands (Boyer 1990; Task Force on Teaching and responses are automatically stored in the receiving Career Development 2007), such that teaching hub and exported into traditional computer files detracts from research productivity and prestige for analysis (Bunz 2005). Barber and Njus (2007) (e.g. Fox & Milbourne 1999; Armstrong & Sper- reviewed six of the most commonly used ARS ry 1994; cf. Hattie & Marsh 1996). A common hubs and found that all can store and save response perspective is to treat decisions about teaching and data along with the date and time of the response. research as a zero-sum game, wherein effort allo- Data-entry error is therefore eliminated by use of cated to one must diminish the other (e.g. Kerr an ARS, since manual transfer is not required. 1975; Murphy 1994). However, ARS technology In summary, ARS technology has the potential may provide a way to integrate research and teach- to overcome three of the most common and impor- ing (Magolda 1999). The study replications we tant data collection problems facing researchers: the presented above were also used for an in-class exer- cost–response problem, the large sample-size prob- cise where the students’ data were immediately lem, and the data-entry error problem. used to instruct students on social decision making. The two ends of teaching and research were ADDITIONAL CONSIDERATIONS OF ARS served simultaneously. Similar applications could TECHNOLOGY work in field settings, with scholars simultaneous- An ancillary benefit of using an ARS is its ability to ly collecting data and providing informative feed- integrate with other electronic technologies and back to host organisations. This may help to events that are already present in organisational set- address the growing reluctance to participate in tings. For example, Krantz (2004) used ARS tech- organisational research (Bryman 2000). nology to acquire consumer feedback on products Another potential benefit of using ARS tech- during a tradeshow; relevant research questions nology applies to the study of controversial or might easily have been added to this implementa- sensitive topics in organisations such as sexual tion. Similarly, ARS units are increasing in popu- harassment, sexuality, and whistle blowing.2 ARS larity in strategic decision planning and committee technology could assist in the study such rare voting sessions in business organisations (Training phenomena. For example, it would be possible to

2 We are grateful to Ed Diener for bringing this potential benefit to our attention.

128 JOURNAL OF MANAGEMENT & ORGANIZATION Volume 15, Issue 1, March 2009 Audience response systems as a data collection method in organizational research use an ARS to ask a room of 500 individuals ARS units are alike in terms of options. While (students or business people) if they had ever we do not advocate any particular ARS, we sug- been sexually harassed, and then immediately gest that potential adopters be certain the tech- present aggregate results while providing com- nology fits their specific needs (e.g. collecting plete anonymity to individuals. Such anonymity responses on a 9-item scale is not possible with may elicit more candid responses and less social an ARS unit that only has five key options). desirability bias. Scholars could also potentially Scholars should select the ARS technology to use such data to efficiently and quickly screen match the desired form of data necessary for respondents for further research purposes. answering the research question. For example, Of course, no data collection method is per- the i-clicker only allows responses in the form of fect (Eid & Diener 2005), and ARS technology individual data points (e.g. multiple choice for- has its own limitations. One important challenge mat), while other ARS technology allows partic- of ARS technology is the relatively high upfront ipants to respond to questions using complete cost in money and learning time (Barber & Njus numerical and letter keypads – allowing for the 2007). A second potential challenge has been collection of short open-ended responses(See Fies identified by instructors using ARS units in the & Marshall [2006] for an extensive review and classroom: cheating, wherein one student uses comparison of current ARS technology systems). multiple clickers on behalf of absent classmates Table 1 provides a summary of the potential ben- (Duncan 2006). A third challenge is that not all efits and challenges of using ARS technology.

TABLE 1: BENEFITS AND CHALLENGES OF ARS TECHNOLOGY AS A DATA COLLECTION TOOL Benefits Challenges

Increasing use of ARS technology by organizations Technical problems: poor signal perception; clicker provides researchers greater ease of using existing malfunction ARS units to collect data Ease of using ARS technology allows researchers to Moderate to high upfront costs for ARS technology enter organizations and collect data in many regular business settings

Provides researcher with high response rates at Time required for scholar and participant to learn low costs ARS use Researcher may collect data from large samples Organizations use different ARS technology quickly and efficiently Accurately collects and stores data from audience Not all ARS technology allows participants to answer respondents at their own pace Provides a means for investigating behavioural Double sourcing: a person can answer for others contexts that typically are a challenge to using their clicker researchers; e.g. large group, rare behaviour, field experiments) Possible awkwardness of using ARS technology in various organizational settings (e.g. religious settings, prisoners) Constraint on type of data collected contingent of ARS technology used to gather data.3

3 Although the specific data-input function of a particular system represents a potential constraint (e.g. a 5-button keypad prevents collecting open-ended responses), we note Ray Cooksey’s insightful observation that such constraint may actually serve as an impetus to better research designs, by prompting more careful and/or creative thought about the data collection protocol.

Volume 15, Issue 1, March 2009 JOURNAL OF MANAGEMENT & ORGANIZATION 129 Matthew W McCarter and Arran Caza

In conclusion, we neither expect nor advocate Caldwell J (2006) Clickers in the large classroom: that ARS technology will replace traditional data Current research and best-practice tips, CBE- collection methods such as mail surveys or lab Life Science Education 6: 9-20. Cohen J (1988) Statistical power analysis for the studies. Rather, we see ARS technology as a com- behavioural sciences. Erlbaum, Hillsdale NJ. plementary addition to existing methods, offer- Czaja S, Sharit J, Nair S and Rubert M (1998) ing strengths where other methods have Understanding sources of user variability in weaknesses. As such, ARS is probably best used computer-based data entry performance, in combination with other methods to help con- Behaviour & Information Technology 17: 282-293. D’Arcy CJ, Eastburn DM and Mullally K (2007) duct a more complete study of various behav- Effective use of a personal response system in a ioural phenomena (Eid & Diener 2005). general education plant pathology class, Planet Earth Instructor Retrieved October 10 2007, from ACKNOWLEDGEMENTS http://www.apsnet.org/education/InstructorCo The authors express their appreciation to Geoff mmunication/TeachingArticles/PRSPathology/d Love and Ricardo Flores, who provided invalu- efault.htm. Diekmann A (1985) Volunteer’s dilemma, Journal able assistance with the ARS technology used in of Conflict Resolution 29: 605-610. this paper, and to Ray Cooksey, Pat Laughlin, Ed Dillmam A (2007) Mail and internet surveys: The Diener, and Shirli Kopelman for their com- tailored design method. Wiley, Hoboken NJ. ments, encouragement, and suggestions as this Draper S, Cargill J and Cutts Q (2002) Electronic paper was developed. This research was made enhanced classroom interaction, Australian Journal of Education Technology 18: 13-23. possible through the generous funding of the Duncan D (2006) Clickers: A new teaching aid Mary Jane Neer Scholarship awarded to the first with exceptional promise, Astronomy Education author, who can be reached at 350 Wohlers Hall, Review 5: 70-88. 1206 S. Sixth Street, Champaign IL 61820, Eden D (2003) Critical management studies and USA, E-mail: [email protected]. the academy of management journal: Challenge and counterchallenge, Academy of Management References Journal 46: 390-394. Armstrong JS and Sperry T (1994) Business school Eid M and Diener E (2005) Handbook of prestige: Research versus teaching, Interfaces 24: multimethod measurement in psychology. American 13-43. Psychological Association, Washington DC. Barber M and Njus D (2007) Clicker evolution: Feng S (2004) Detecting errors in the current Seeking intelligent design, CBE-Life Science population survey: a matching approach, Education 6: 1-20. Economics Letters 82: 189-194. Boyer EL (1990) Scholarship reconsidered: Priorities Fies C and Marshall J (2006) Classroom response of the professoriate. Carnegie Foundation for the systems: A review of the literature, Journal of Advancement of Teaching, Princeton NJ. Science Education & Technology 15: 101-109. Bryman A (2000) Research methods and Fox KJ and Milbourne R (1999) What determines organization studies. Routledge, London UK. research output of academic economists? Bunz U (2005) Using scantron versus an audience Economic Record 75: 256-267. response system for survey research: Does Franzen A (1995) Group size and one-shot collective methodology matter when measuring computer- action, Rationality & Society 7: 183-200. mediated communication competence? Gamito E, Burhansstipanov L, Krebs L, Bemis L Computers & Human Behaviour 21: 343-359. and Bradley A (2005) The use of an electronic Byrd GG, Coleman S and Werneth C (2004) audience response system for data collection, Exploring the universe together: Cooperative Journal of Cancer Education 20: 80-86. Quizzes with and without a Classroom Hansen J and Hill N (1998) Control and audit of Performance System in Astronomy 101, electronic data interchange, MIS Quarterly 13: Astronomy Education Review 3: Retrieved 402-413. October 10 2007, from http://aer.noao.edu/cgi- Hatch S (2003) ARS systems: creativity and planning bin/article.pl?id=91%3E%3Cimg%20src=. required, Corporate Meetings & Incentives 22: 35-36.

130 JOURNAL OF MANAGEMENT & ORGANIZATION Volume 15, Issue 1, March 2009 Audience response systems as a data collection method in organizational research

Hattie J and Marsh HW (1996) The relationship 141-161. Blackwell Publishing, Boston MA. between research and teaching: A meta-analysis, Rosenthal R (1994) Science and ethics in conduct- Review of Educational Research 66: 507-542. ing, analyzing, and reporting psychological Heckathorn D (1996) The dynamics and dilemmas research, Psychological Science 5: 127-134. of collective action, American Sociological Review Scandura T and Williams E (2000) Research 61: 250-277. methodology in management: Current practices, Homme J, Asay G and Morgenstern B (2004) Utilis- trends, and implications for future research, ation of an audience response system, Medical Academy of Management Journal 43: 1248-1264. Education 38: 575. Sen A (1985) Goals, commitment, and identity, James C (1990) Evaluating the cost effectiveness of Journal of Law Economics & Organizations 1: hand-held computer technology, Public Product- 341-355. ivity & Management Review 13: 201-213. Siegel S (1956) Nonparametric statistics for the Kerr S (1975) On the folly of rewarding A, while behavioural sciences. McGraw-Hill, New York. hoping for B, Academy of Management Journal Shadish WR, Cook TD and Campbell DT (2002) 18: 769-783. Experimental and quasi-experimental designs for Krantz M (2004) Audience response system = generalized causal inference. Houghton Mifflin, marketing tool, Meeting News 28: 5. Boston MA. Liebrand W (1983) A classification of social dilemma Smith W (1967) Accuracy of manual entries in games, Simulation & Games 14: 123-138. data collection devices, Journal of Applied Magolda MB (1999) Creating contexts for learning Psychology 51: 362-368. and self-authorship. Vanderbilt University Press, Sproull LS (1986) Using electronic mail for data Nashville TN. collection in organizational research, Academy of Miller R, Ashar B and Getz K (2003) Evaluation of Management Journal 29 159-169. an audience response system for continuing educa- Starbuck WH (2004) Why I stopped trying to tion of health professionals, Journal of Continuing understand the real world, Organization Studies Education in the Health Professions 23: 109-116. 27: 1233-1254. Murnighan JK, Kim JW and Metzger AR (1993) Stone-Romero E (2002). The relative validity and The volunteer dilemma, Administrative Science usefulness of various empirical research designs, in Quarterly 38: 515-538. Rogelberg S (Ed.) Handbook of Research Methods Murphy FH (1994) Don’t let shallow reporters set in Industrial and Organizational Psychology, pp 77- the agenda: A comment on Armstrong & Sperry 98, Blackwell Publishing, Boston MA. (1994), Interfaces 24: 25-26. Task Force on Teaching and Career Development Olson M (1965) The logic of collective action: Public (2007) A compact to enhance teaching and goods and the theory of groups. Harvard learning at Harvard. Harvard University Press, University Press, Cambridge MA. Cambridge MA. Pedhazur E and Schmelkin L (1991) Measurement, Training and Development (2006) Improving busi- design, and analysis: An integrated approach. ness decisions and strategic planning, 60: 63-65. Erlbaum, Hillsdale NJ. Webb E, Campbell D, Schwartz R and Sechrest L Rogelberg S, Church A, Waclawski J and Stanton J (2000) Unobtrusive measures: Revised edition. (2002) Organizational survey research. In Sage, London. Rogelberg S (Ed.) Handbook of Research Methods in Industrial and Organizational Psychology, pp. Received 26 March 2008 Accepted 12 December 2008

C A L L F O R P A P E R S COLLABORAT I V E A N D C H A L L E N G E - B A S E D I N N O VATION Guest Editors: Tim Kastelle (UQ Business School) and Kate Morrison (Think Play Do Group) ISBN: 978-1-921314-34-1 ii + 126 pages December 2010 A special issue of Innovation: Management, Policy & Practice _ volume 12 (2010) http://www.innovation-enterprise.com/archives/vol/12/issue/3/call/ eContent Management Pty Ltd, PO Box 1027, Maleny QLD 4552, Australia Tel.: +61-7-5435-2900; Fax. +61-7-5435-2911; [email protected] www.e-contentmanagement.com

Volume 15, Issue 1, March 2009 JOURNAL OF MANAGEMENT & ORGANIZATION 131 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.