CIN: , Informatics, Nursing & Vol. 29, No. 6, 326–334 & Copyright B 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins

CONTINUING

EDUCATION

Effectiveness of an It has been proposed that students’ use of an Audience Response audience response system, commonly called clickers, may promote comprehension and re- System in Teaching tention of didactic material. Whether this method actually improves students’ grades, however, is Pharmacology to still not determined. The purpose of this study was to evaluate whether a lecture format utilizing multiple-choice PowerPoint slides and an audi- Baccalaureate ence response system was more effective than a lecture format using only multiple-choice PowerPoint Nursing Students slides in the comprehension and retention of phar- macological knowledge in baccalaureate nurs- KIMBERLY D. VANA, MS ing students. The study also assessed whether the additional use of clickers positively affected GRACIELA E. SILVA, PhD, MPH students’ satisfaction with their learning. Results DIANN MUZYKA, PhD from 78 students who attended lecture classes LORRAINE M. HIRANI, MSN with multiple-choice PowerPoint slides plus click- ers were compared with those of 55 students who utilized multiple-choice PowerPoint slides only. Test scores between these two groups were not significantly different. A satisfaction questionnaire showed that 72.2% of the control students did not Educators have promoted inquiry, teaching through ques- desire the opportunity to use clickers. Of the tioning, to engage students dynamically in the attainment group utilizing the clickers, 92.3% recommend the use of this system in future courses. The use of and retention of knowledge and skills. Traditionally, stu- multiple-choice PowerPoint slides and an audi- dents have assumed passive roles, observing the instruc- ence response system did not seem to improve tors’ performances, rather than actively participating with the students’ comprehension or retention of phar- 1–4 instructors and peers. Often, lectures present new macological knowledge as compared with those knowledge, but lack dedicated time for teaching higher- who used solely multiple-choice PowerPoint slides. order thinking skills such as analysis, synthesis, and problem-solving.1–4 Inquiry may be used to teach these KEY WORDS higher-order thinking skills and increase student partici- & 4 Audience response system pation. However, encouraging student participation in -assisted instruction & large-lecture courses can be daunting due to time con- & Feedback methods & straints and the reticence of pupils to ask or answer ques- Interactive-voice response system & tions in front of large groups. Student response system Audience response system (ARS) technology can address this need for increased interactivity and critical thinking in a large-classroom environment.1,2,5 Students’ use of an ARS may promote student comprehension and retention Author Affiliation: College of Nursing & Health Innovation, Arizona State University, Phoenix. by offering each learner several chances to participate in The authors do not belong to speaker bureaus that promote au- class, anonymously answering questions posed by the in- dience response system technology. No external funding was re- structor. Researchers have suggested that students are ceived for this project. satisfied with the use of ARSs, but the research is mixed Disclaimer: Authors declare no conflict of interest. on whether the use of these systems actually increases test Corresponding author: Kimberly D. Vana, MS, College of Nursing & Health Innovation, Arizona State University, Downtown Campus, 500 N scores. This study sought to investigate whether students’ Third St, Phoenix, AZ 85004-0698 ([email protected]). use of an ARS to answer multiple-choice PowerPoint DOI: 10.1097/NCN.0b013e3181f9dd9c

326 CIN: Computers, Informatics, Nursing & June 2011

Copyright @ 2011 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. (MCPP) slides (, Redmond, WA) is more effective Students, as well, have said that using the ARS helps in increasing test scores than using MCPP slides alone. them to judge their understanding of the content2,8,11,15 and provides them the safety of answering anony- mously.1,2,6,9,11,23–25 Many students believe that the ARS AUDIENCE RESPONSE SYSTEMS questions help them perform better on quizzes and exami- nations.2,13 Stein et al13 stated that 89% of nursing stu- An ARS is a communication technology that allows par- dents perceived a beneficial effect of discussing distracters, ticipants to interact with the lecturer; many of these systems citing increased comprehension of the lecture content. In seamlessly merge with PowerPoint presentations.2 The addition, the ARS questions may assist students in be- lecturer may post multiple-choice or true-false questions coming familiar with the lecturer’s test-question formats.13 on a PowerPoint slide, and then each student answers by Students may have ‘‘emotional’’ investments in their re- pressing an individual keypad button corresponding to his/ sponses, which increase their attentiveness6 and participa- her answer. These keypads, or clickers,useinfrared,radio tion.5,10,14 Lastly, the use of ARS positively influences class frequency, , or Fidelity (Wi-Fi) tech- attendance.6,22,26 Overall, this technology has been widely nology to communicate with a receiver, which is plugged embraced by students.2,6,8,11,17,20,27,28 This may be due to into a universal serial bus (USB) port on the podium com- the partitioning of lecture content into small, manageable puter.1,6,7 After polling the audience, a bar graph of the units and allowing students intermittent breaks from the classroom responses is displayed on a projected slide. The lecture while increasing student engagement and atten- graph allows the presenter to gauge the level of participant tion.3,6,11–13,27,29 DeBourgh2 suggested that ARS ques- understanding immediately and discuss the correct and tions be used every 20 minutes to prevent ‘‘cognitive incorrect answers.3 The lecturer may choose to review pre- overload.’’2(p81) Miller et al12 demonstrated that students vious content, tailoring the lecture to comprehension assign higher lecturer evaluation scores to instructors who level.1,2,8–15 The responses of each individual may be use an ARS. saved and then later printed in a variety of report for- Some researchers, however, have suggested that sat- mats.16,17 Thus, the responses to the ARS questions may isfaction levels are greater in the freshman-sophomore level be compared with future test scores if desired. courses than junior-senior level courses.20 They suggest that As new communication technologies become available, students may appreciate an ARS more when studying an ARSs continue to advance. Currently, is unfamiliar topic. Some students have registered dissatisfac- preferred to technology because it accommodates tion with the use of an ARS in taking attendance,6 and others larger audiences with less interference and does not re- resent the increased expectationtoparticipateinclass.2 Of quire that the keypads be aimed at the USB receiver while course, students have been known to deceive,6 responding inputting answers.1,16,17 Enterprising researchers are ex- with multiple keypads on quizzes and attendance. perimenting with using PDAs to communicate with the Although students are generally satisfied with audience audience response receivers.1,7 In addition, student ter- response technology, the literature is mixed on whether minals or full keyboards may allow participants the students’ use of ARSs increases student performance to enter their answers to open-ended questions. The in- on test scores. Some researchers have demonstrated clear structor may randomly select a student’s typed response increases in test scores in ARS-enhanced lecture courses to project on the slide.18 Some students, however, may rather than typical lecture courses.19,22,28–30 Crossgrove find too cumbersome to tote to class and too and Curran20 found that students performed signifi- slow to activate.19 Thus, laptop use in large lecture-class cantly better on test questions that had been discussed environments may be seen as burdensome by students. previously in ARS-enhanced lectures. However, reten- The use of ARSs may increase active learning by students tion of course material may be greater in introductory in large lecture classes5,13,20 andinsmallgroups.1,3,6,21,22 versus junior-senior level courses. Preszler et al26 dis- When inserting MCPP slides within a lecture, other con- covered that the test scores improved as the lecturer’s use tent slides must be omitted to stay within the lecture’s of ARS questions increased. Other research showed no allotted period. In addition, the lecturer must allow difference in test scores between courses utilizing didactic more time for each MCPP slide to present the question, lectures versus lectures that incorporated ARS.11–13 discuss the correct and incorrect answers, and possibly The conflicting results on ARSs’ ability to positively reteach the content area. Thus, the use of an ARS may affect test scores could be attributed to how closely ARS decrease the amount of lecture content that may be ad- questions parallel test content, the frequency of MCPP dressed.1,6,9 Increased student comprehension may, slides within the lecture, differences between freshman- however, outweigh this time limitation. In addition, in- sophomore and junior-senior course content and student creased lecture preparation time is required to develop expectations, the instructors’ experience with inquiry, questions that encourage application, synthesis, and or other confounding variables, such as student age. In problem solving.3 addition, the lecturer may have already incorporated

CIN: Computers, Informatics, Nursing & June 2011 327

Copyright @ 2011 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. inquiry or small group work into his/her course; thus, one nursing students in a nursing foundation course. Students more method of encouraging student engagement, ARS, were awarded extra participation points if they used their may not result in further statistically significant score in- clickers in all classes. Initial and midterm assessments of creases.20 Although ARSs may increase scores in tradi- the PRS were completed. Students’ exposure to PRS ques- tional, didactic courses with one-way communication, this tions resulted in 70% or more students correctly an- technology may not add any advantage in courses that swering the same or similar content questions on the already engage the learner in two-way communication. course quizzes; 80% to 92.5% answered these questions Ultimately, the interaction between pedagogy and ARS use correctly on the midterm. The higher percentage of stu- defines learning outcomes,9 and the proposed benefits of dents answering correctly on the midterm may be due to ARSs must justify institutional and student keypad costs.3 the students’ realizations that the quiz questions came from the PRS slides. At midterm, 67% of the students felt that the PRS assisted them in understanding the lecture Use of Audience Response Systems in content, as compared with 56% on the first day of class. Nursing Education Thus, clearer understanding of what constituted core content may have led to higher test scores, rather than Nursing researchers are beginning to investigate the use of PRS use. Studies are needed to separate the effects of ac- ARSs to augment the acquisition and retention of nursing tive participation from that of ARS use. knowledge by nursing students. Three nursing studies Lastly, DeBourgh2 conducted an online survey of 92 evaluated the impact of ARSs on course test scores8,11,13; advanced nursing therapeutics course students on their only the study by Abdallah8 showed increases in test scores. satisfaction levels with ARS use. Sixty-five students com- In 1995, Halloran11 investigated the use of computer- pleted the anonymous questionnaire 1 week before the end managed instruction (CMI) and keypads in a medical/ of the semester. Most of the responding students (75.8%) surgical course taught to baccalaureate junior nursing recommended continued use of clickers, and 66% believed students. The experimental group (14 students) used CMI that the ARS increased their quiz and examination scores. and keypads in the lectures, whereas the control group (14 Fifty-three percent came to class ‘‘better prepared.’’2(p83) students) was taught the same content by the traditional DeBourgh2 did not evaluate the impact of ARS use on lecture format, including discussion, overheads, and oral test scores. questioning. Both groups took the same three multiple- choice tests. The differences in test scores between groups were not significant; no differences in test scores were PURPOSE found between those nursing students who used CMI and The primary objective was to evaluate whether a lecture for- those who participated in traditional lectures. mat using MCPP slides and an ARS was more effective than Stein et al13 investigated the use of an ARS for pretest a lecture format solely using MCPP slides in the compre- reviews in two successive freshman nursing anatomy and hension and retention of pharmacological knowledge among physiology courses (155 and 128 students, respectively). baccalaureate nursing students. In addition, this study sought The students received three ARS test reviews and one to evaluate the students’ satisfaction with the course, their traditional, lecture-format review prior to the examina- perceived comprehension and retention of knowledge, and tions in each course. The ARS test review used a Jeopardy! their satisfaction with the use of ARS technology. game format (a game show in which contestants are presented with answers and must respond with the correct question) with 25 questions. The question for- METHODS mats included multiple-choice questions, true-false ques- tions, and labeling of diagrams. The 92% of students Design voiced positive satisfaction with the ARS technology. Most of the students (94%) perceived a beneficial effect on This quasi-experimental study used convenience samples. their test scores, although the test averages were not The control group consisted of 55 baccalaureate nursing significantly higher in tests that followed an ARS review. If students enrolled in a nursing pharmacology course at the the students answered poorly on the ARS question in class, downtown campus of a large state university in fall 2007. they generally gave significantly more correct responses on This group attended lecture classes that used MCPP slides a similar test question when it was on the examination. interspersed throughout the lectures. The intervention Eighty-nine percent of the students felt that discussing the group comprised 78 baccalaureate nursing students en- distracters, or wrong answers, to the ARS questions as- rolled in the same nursing pharmacology course at the sisted their comprehension of the lecture content. west campus of the same university in fall 2007. This In fall of 2006, Abdallah8 investigated the effects of group attended classes with MCPP slides interspersed using a personal response system (PRS) with 71 junior throughout the lectures and used an ARS to input their

328 CIN: Computers, Informatics, Nursing & June 2011

Copyright @ 2011 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. answers to the multiple-choice questions listed on the ARS participant list. All responses entered into each slides; radio-frequency keypads were used. The students keypad were saved to a computer file after each lecture. were assigned to the only pharmacology course taught on The coded answers for each multiple-choice question their campus; thus, the students were unable to self-select (ARS and test) were then inputted into the study data for the course based on pedagogy. Both groups had the same those students who chose to participate in the study. Each instructor, lecture slides, and content material for all se- correct ARS answer was coded as 1; all other answers mester class periods. were coded as zeros. Data from students who scored zeros for all ARS questions in a lecture were presumed to be absent or nonparticipatory and were dropped from the Procedure analysis of ARS responses for that lecture. For a given Students in both courses were invited to participate in this lecture, 78 to 48 students utilized their clickers (mean, study during the first class periods. They were assured 66.2; mode, 72; median, 70). Because attendance was not that their decision to enroll or decline would not affect mandatory, not all students attended each class period. In their course grades and that all data collected would be addition, students occasionally forgot their clickers. coded to ensure anonymity and confidentiality. They Because some of the students in the intervention group were also advised that they could drop out of the study at had not yet procured their keypads before the first test, any time. Students’ answers to the MCPP slides and the only the ARS data for the lectures covered in tests 2 to 4 test questions were coded so that their responses were not were used. The students in the control group were given linked to their names. In the control group, 56 of 60 stu- MCPP slides interspersed among the lecture’s content slides dents chose to participate, and one later withdrew from and were asked to call out their answers. The instructor the course. Another pupil chose not to complete the satis- read the MCPP questions to both classes and then allowed faction questionnaire at the end of the semester. In the 10 seconds for the students to respond. For each MCPP intervention group, 79 of 81 students agreed to partici- slide, the teacher discussed the correct answer and ex- pate; however, one additional student withdrew from the plained why each distracter was wrong to the control and course. An exemption for this study was obtained from intervention groups. the university institutional review board. Course Tests Demographic Questionnaire Students at both campuses were administered the same All subjects in the control and intervention groups were four-course tests throughout the semester. Each test covered asked to complete the demographic questionnaires during six lecture topics. Each test consisted of six lecture subtests the first class period. Completion of the demographic of 10 questions each, for a total of 60 multiple-choice questionnaire was considered consent to participate in the questions. Each test question was worded differently than study. The demographic questionnaire included questions the ARS practice questions in lecture to encourage under- on age, sex, ethnicity, and prior educational degrees. In standing of the concept versus memorization of the addition, the demographic questionnaire gathered infor- answers. Because the test score distributions were nega- mation about previous ARS experience with courses or tively skewed, the median differences in test scores for tests educational offerings, previous experience with health- 2, 3, and 4 were compared between the two groups using care or information technology, and familiarity with in- the Wilcoxon rank sum (Mann-Whitney) test for indepen- stant messaging and/or text messaging (categorized into 0, dent nonparametric samples. 1–25, 26–50, 51–75, 76–100, and 9100 texts per week). The demographic characteristics between the control and in- tervention ARS group were compared using the Pearson’s #2 Satisfaction Questionnaire test. Student’s t test was used to compare mean age between On the last day of the course, each group was given a e groups. P .05 was chosen for significance for all tests. satisfaction questionnaire asking the student’s satisfaction with the nursing pharmacology course and his/her perceived ARS and MCPP Use comprehension and retention of pharmacological knowl- edge. The survey elicited opinions on the MCPP slides from Each student in the intervention group at the west campus both groups and on the use of an ARS for the interventional purchased an ARS keypad or clicker from the university group. The students in the control group were also asked bookstore, whereas the control group at the downtown whether they wished they had an opportunity to use the campus did not purchase clickers for the pharmacology ARS and would they choose a course that utilized an ARS course. Each student’s keypad was assigned an identifying over one that did not. The control group answered 13 number, which was linked to the student’s name by an questions on the survey; the ARS group responded to 21

CIN: Computers, Informatics, Nursing & June 2011 329

Copyright @ 2011 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. questions. The questionnaire had five answer choices: (1) years and 21.4 [SD, 3.0] years, respectively, P G .0001) strongly agree, (2) agree, (3) neutral, (4) disagree, and (5) (Table 1). The subjects in the ARS group were more strongly disagree. Responses marked as (1) strongly agree or likely to have prior degrees than those in the MCPP (2) agree were coded as yes. All other responses were des- group, reaching a significance level of .007. In addition, ignatedasno.Theauthorschosetoseparatepositivere- the ARS group had more prior ARS course experience sponses, strongly agree and agree from neutral or negative than the subjects in the MCPP group (P =.036).Nosig- responses, because neutral responses were not endorsements nificant differences were found between groups for sex, for ARS use. The Pearson’s #2 test was used to compare the ethnicity, number of text messages per week, and prior differences in percentages between the control and ARS experiences with ARS educational offerings, healthcare groups on their satisfaction with the MCPP slides. employment, and informational technology employment.

Participant Test Scores RESULTS No significant differences between groups were found for Participant Characteristics the overall test scores for examination 2, 3, or 4. The P values were .475, .326, and .207, respectively (Table 2). Subjects in the ARS group were significantly older than the Likewise, no subtest differences were found between the participants in the control group (mean, 27.8 [SD, 9.4] groups.

Table 1 Demographics by ARS and MCPP Groups Demographics ARS,a n (%) MCPP,b n (%) P c Age, mean (SD), y 27.8 (9.4) 21.4 (3.0) G.0001 Age, y G20 1 (1.3) 4 (7.3) .001 20–29 54 (71.0) 50 (90.9) 30–39 11 (14.5) 1 (1.8) Q40 10 (13.2) 0 (0.0) (n = 76) Sex Female 70 (89.7) 48 (87.3) .657 Male 8 (10.3) 7 (12.7) Ethnicity White 60 (77.9) 40 (74.0) .482 Hispanic 9 (11.7) 4 (7.4) Asian 3 (3.9) 5 (9.3) Other 5 (6.5) 5 (9.3) (n = 77) (n = 54) Prior degrees Yes 31 (44.9) 8 (19.5) .007 No 38 (55.1) 33 (80.5) (n = 69) (n = 41) No. of texts/wk 0 1 (1.5) 2 (3.9) 1–25 26 (38.1) 13 (25.5) .630 26–50 12 (17.7) 9 (17.7) 51–75 1 (1.5) 1 (2.0) 76–100 8 (11.8) 5 (9.8) 9100 20 (29.4) 21 (41.1) (n = 68) (n = 51) Prior ARS course experience Yes 31 (39.7) 32 (58.2) .036 No 47 (60.3) 23 (41.8) Prior ARS educational offering experience Yes 1 (1.3) 0 (0.0) .399 No 77 (98.7) 55 (100.0) Prior healthcare experience Yes 28 (35.9) 16 (29.1) .411 No 50 (64.1) 39 (70.9) Prior information technology employment Yes 0 (0.0) 1 (1.8) .232 No 78 (100.0) 54 (98.2) aARS with MCPP slides, n = 78. bMCPP slides only, n = 55. cP value for Pearson’s #2 for all variables except continuous age, which used Student’s t tests to compare means between groups.

330 CIN: Computers, Informatics, Nursing & June 2011

Copyright @ 2011 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. Table 2 Median Difference of Test Scores by Group ARSa MCPPb Median Score Min-Max Scores Median Score Min-Max Scores P c Test 2 questions (tq = 60) 56.0 38–60 54.0 37–60 .475 Inotropic drugs (q = 10) 9.5 4–10 9.0 4–10 .870 Antianginal drugs (q = 10) 9.0 4–10 9.0 5–10 .541 Peptic ulcer drugs (q = 10) 9.0 3–10 9.0 6–10 .439 Opioid analgesics (q = 10) 9.5 6–10 9.0 6–10 .737 NSAIDs (q = 10) 10.0 6–10 10.0 6–10 .439 Antidepressants (q = 10) 9.0 7–10 10.0 7–10 .343 Test 3 questions (tq = 60) 55.0 38–60 55.0 40–60 .326 Anxiolytics, sedatives, hypnotics (q = 10) 9.5 6–10 9.0 8–10 .832 Antihypertensive drugs (q = 10) 10.0 7–10 9.0 6–10 .324 Diuretics (q = 10) 9.0 3–10 9.0 5–10 .735 Lipid-lowering drugs (q = 10) 9.0 4–10 9.0 4–10 .416 Anticoagulant, antiplatelet drugs (q = 10) 10.0 5–10 10.0 7–10 .552 Drugs for thyroid disorders (q = 10) 9.5 5–10 9.0 2–10 .162 Test 4 questions (tq = 60) 55.0 44–59 53 43–60 .207 Adrenocorticoids (q = 10) 8.0 5–10 8.0d 4–10 .726 Drugs for asthma (q = 10) 10.0 6–10 10.0 6–10 .753 Antibiotics I (q = 10) 9.0 4–10 9.0 6–10 .381 Antibiotics II (q = 10) 9.0 4–10 9.0 6–10 .920 Substance abuse (q = 10) 10.0 7–10 10.0 8–10 .301 Herbal therapy (q = 10) 10.0 8–10 9.0 8–10 .074

Abbreviations: NSAIDs, nonsteroidal anti-inflammatory drugs; q, number of questions in each subtest; tq, number of questions in each test. aARS with MCPP slides, n = 78. bMCPP slides only, n = 55. cP value for Wilcoxon rank sum (Mann-Whitney) test. dMCPP group unable to see MCPP slides for adrenocorticoid lecture because of audiovisual malfunction, had PowerPoint handouts without MCPP slides.

Satisfaction Questionnaire Responses In addition, both the ARS and control groups agreed that the MCPP slides influenced content areas studied for Differences in percentages between the control and ARS an examination (statement 7): 56.4% and 66.7%, respec- groups on the satisfaction statements about MCPP slides tively. Both the ARS and control groups denied that cor- are presented in Table 3. The answers differed significantly rect MCPP answers led them to focus less on the content between groups for two statements. The control group when studying for an examination (statement 8): 75.6% was more likely to agree that correct MCPP answers cor- and 75.9%, respectively, but concurred that incorrect responded to correct examination answers (statement 5): MCPP answers led them to focus more on content when 88.9% for the control and 66.7% for the ARS groups studying for an examination (statement 9): 53.9% and (P = .003). However, since the control group’s answers to 64.8%, respectively. Both groups readily agreed that MCPP the MCPP slides were not collected, the accuracy of their slides assisted the students in attaining better examination perception cannot be verified. The ARS group was more scores (statement 10), 78.2% (ARS) and 87.0% (control likely to disagree that incorrect MCPP answers corre- group), and disagreed that the MCPP slides caused the sponded to incorrect examination answers (statement 6), pupils to perform more poorly on the examinations (state- 74.4% versus 57.4% (P = .041). The ARS group had ment 11): 97.4% (ARS) and 96.3% (control group). performed better on the test questions than on the MCPP Interestingly, the control group overwhelmingly disagreed questions. The ARS and control groups both agreed that with two statements pertaining to ARS use (Table 3). The (1) MCPP slides increased understanding of the lecture control subjects rejected statement 12, which declared that content (statement 2): 98.7% and 100%, respectively; (2) a they wished they had the opportunity to use the ARS; discussion of the correct answers to the MCPP slides in- 72.2% said no. In addition, 77.8% rejected the statement creased understanding (statement 3): 98.7% and 98.1%, that they would choose an ARS course over a non-ARS respectively; and (3) a discussion of the incorrect answers course. to the MCPP slides increased understanding (statement 4): Only the intervention group predominantly agreed that 97.4% and 98.1%, respectively. ARS use was beneficial (Table 3). The students felt that

CIN: Computers, Informatics, Nursing & June 2011 331

Copyright @ 2011 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. Table 3 Satisfaction Questionnaire Responses by Group (Statements 1–23) Question No. Answers ARS,a n (%) MCPP,b n (%) P c 1. Course used ARS? Yes 78 (100) No 54 (100) 2. MCPPd increased understanding of lecture content Yes 77 (98.7) 54 (100.0) .404 No 1 (1.3) 0 (0.0) 3. Discussion of correct answers to MCPP slides Yes 77 (98.7) 53 (98.1) .792 increased understanding No 1 (1.3) 1 (1.9) 4. Discussion of incorrect answers to MCPP slides Yes 76 (97.4) 53 (98.1) .787 increased understanding No 2 (2.6) 1 (1.9) 5. Correct MCPP answers corresponded to correct Yes 52 (66.7) 48 (88.9) .003 examination answers No 26 (33.3) 6 (11.1) 6. Incorrect MCPP answers corresponded to incorrect Yes 20 (25.6) 23 (42.6) .041 examination answers No 58 (74.4) 31 (57.4) 7. MCPP slides influenced content areas studied for Yes 44 (56.4) 36 (66.7) .236 an examination No 34 (43.6) 18 (33.3) 8. Correct MCPP answers led me to focus less on Yes 19 (24.4) 13 (24.1) .970 content when studying for an examination No 59 (75.6) 41 (75.9) 9. Incorrect MCPP answers led me to focus more on Yes 42 (53.9) 35 (64.8) .209 content when studying for an examination No 37 (46.1) 19 (35.2) 10. MCPP slides assisted me in attaining better Yes 61 (78.2) 47 (87.0) .196 examination scores No 17 (21.8) 7 (13.0) 11. MCPP slides caused me to perform more Yes 2 (2.6) 2 (3.7) .707 poorly on examinations No 76 (97.4) 52 (96.3) 12. I wish I had the opportunity to use the ARS Yes 15 (27.8) No 39 (72.2) 13. I would choose an ARS course over a non-ARS course Yes 12 (22.2) No 42 (77.8) 14. ARS feedback increased comprehension of material Yes 73 (93.6) No 5 (6.4) 15. ARS increased retention of lecture content Yes 70 (89.7) No 8 (10.3) 16. ARS increased examination scores Yes 50 (64.1) No 28 (35.9) 17. ARS decreased examination scores Yes 2 (2.6) No 76 (97.4) 18. I am glad I had the opportunity to use the ARS Yes 74 (94.9) No 4 (5.1) 19. I recommend use of ARS in the nursing Yes 72 (92.3) pharmacology course No 6 (7.7) 20. I recommend use of ARS in other nursing courses Yes 67 (85.9) No 11 (14.1) 21. I would choose an ARS course over non-ARS course Yes 48 (61.5) No 30 (38.5) 22. I would choose a course that utilized laptops to Yes 28 (35.9) answer MCPP questions over a non-ARS course No 50 (64.1) 23. I would prefer to answer MCPP questions with my Yes 13 (16.7) laptop instead of clickers No 65 (83.3) aARS with MCPP slides, n = 78. bMCPP slides only, n = 54; one subject from MCPP group did not complete questionnaire. cP value for Pearson’s #2. dMCPP slide.

ARS feedback increased comprehension of the material agreed that the use of an ARS increased their examina- (93.6%) and increased retention of lecture content tion scores (64.1%) and did not decrease their scores (89.7%). Furthermore, the students in the ARS group (97.4%). Most were glad to have the opportunity to use

332 CIN: Computers, Informatics, Nursing & June 2011

Copyright @ 2011 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. the ARS (94.9%) and would recommend that the ARS standing of the lecture content and that discussion of be used in nursing pharmacology courses (92.3%) and in correct and incorrect answers to the MCPP increased their other nursing courses (85.9%). In fact, 61.5% of the in- understanding. terventional students would choose an ARS course over Generally, both groups agreed that correct answers to a non-ARS course. Because junior-senior students were the MCPP slides corresponded to correct test answers; noted to be less embracing of ARS technology,20 the the students may have already shown mastery during the authors expected that students with prior degrees might lecture and retained this learning at testing. Correctly not concur; this was not the case. Interestingly, 64.1% answering MCPP slides in lecture did not lead the stu- of the ARS group would not choose a course that used dents in both groups to study less on that content when laptops to answer MCPP questions. Similarly, 83.3% of studying for a test. Pupils may have felt the need to revisit the respondents would prefer to use clickers versus lap- mastered material, believing the MCPP questions would tops in answering MCPP questions. be on the examination. The two groups disagreed, how- ever, that incorrect answers to the MCPP slides resulted in incorrect answers on the examinations. The ARS group DISCUSSION was more likely to disagree that incorrect MCPP answers corresponded to incorrect examination scores. Possibly, Baccalaureate nursing students who attended nursing phar- these older students had more experience studying for macology lectures using MCPP slides and an ARS did not tests and recognized the need to study content areas in demonstrate increased comprehension and retention of which they had difficulty. Students in both groups con- pharmacological knowledge compared with those students curred that incorrect answers to MCPP slides led them to relying solely on MCPP slides. There were no significant focus on that content when preparing for the examination. differences between course test scores or subtest scores This focus may result in postlecture mastery of content. between the MCPP and the MCPP plus ARS groups. The Thus, pupils prepared for examinations by reviewing mas- ARS group, who had older students with more prior tered and nonmastered content. degrees, performed the same as the control group students, Interestingly, the students who attended lectures with who were younger. These results suggested that the use of MCPP slides did not wish for an opportunity to use an an ARS did not provide an additional boost to students’ ARS. Perhaps they preferred to avoid additional costs. The test scores, and these results are similar to the findings costs of ARS USB receivers and site licenses are negotiated of Stein et al13 and Halloran.11 The test scores may have between the university and vendor. Some of these costs been influenced more by the MCPP slides, which in- may be passed to students through keypad purchases at creased student engagement in both the ARS and control the university bookstore. Or the pupils may have been groups. Thus, MCPP use in both groups may have en- pleased with the MCPP-slide format and been unaware couraged active inquiry and positively affected both of the advantages of anonymity and histograms provided by groups’ test scores, resulting in no discernable differences ARSs, the control group having significantly less previous in test scores between groups, as Crossgrove and Curran ARS experience. Future crossover design studies are needed suggested.20 to investigate the reasons for the students’ preferences be- As Stein et al13 observed, the students believed that ARS tween MCPP-enhanced lectures with and without ARSs, use enabled them to comprehend course material and at- as well as to compare these MCPP-enhanced lectures with tain higher test scores, even though the test scores were non-MCPP didactic lectures. In addition, studies are needed not significantly increased. Similarly, Abdallah8 noted that to determine whether cumulative final test scores are in- students felt that the ARS assisted their comprehension of creased by the use of ARSs. lecture content, and DeBourgh2 remarked that students Although the test scores were not significantly different, positively attributed ARS use to achieving higher exami- the histograms of student responses allowed the instructor nation scores. In the current economic climate, the lack of to tailor the lectures to overall student needs, instead of the improvement in test scores may outweigh the increased vocal few. If instructors use existing ARS technology al- student satisfaction levels when weighing the financial ready established at their institution, they may decrease costs of ARSs to the institution and students. However, student costs by requiring one clicker for all courses at that the merits of obtaining immediate student feedback by the institution. In addition, favorable student responses sug- instructor must not be underestimated when concurrently gest that this technology improves the educational cli- tailoring a lecture to their needs. mate, which may be reflected in students’ evaluations of This study supports the benefits of inquiry in lecture faculty.12 Furthermore, if the control group had been al- courses, regardless of ARS use. Students in the ARS lowed to try the ARS, the students might have embraced and control groups extolled the virtues of using MCPP this technology. slides within the lectures. Students in both groups over- A limitation of this study is that ARS participants may whelmingly agreed that MCPP slides increased under- have chosen to switch clickers with other students without

CIN: Computers, Informatics, Nursing & June 2011 333

Copyright @ 2011 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited. the researchers’ knowledge. However, because no points use of an electronic audience response system for data collection. JCancerEduc. 2005;20(1 suppl):80–86. were awarded for attending lecture or for responding to 11. Halloran L. A comparison of two methods of teaching. Computer the MCPP slides, there was no incentive for students to managed instruction and keypad questions versus traditional class- exchange their registered clickers. If the students had an- room lecture. Comput Nurs. 1995;13(6):285–288. 12. Miller RG, Ashar BH, Getz KJ. Evaluation of an audience re- swered for each other, comparison between the ARS and sponse system for the continuing education of health profes- test responses for each student in the ARS group would sionals. J Contin Educ Health Prof. 2003;23(2):109–115. have been invalid. The likelihood of the Hawthorne effect 13. Stein PS, Challman SD, Brueckner JK. Using audience response technology for pretest reviews in an undergraduate nursing course. is low because the students used MCPP slides and the JNursEduc. 2006;45(11):469–473. ARS throughout the semester, which diminishes the effect 14. Uhari M, Renko M, Soini H. Experiences of using an interactive of novelty on the study’s results. In addition, the satis- audience response system in lectures. BMC Med Educ. 2003;3:12. http://www.biomedcentral.com/1472-6920/3/12. Accessed March faction questionnaire has only been tested for content 16, 2009. validity. Only questions deemed valid by all authors were 15. Johnson JT. Creating learner-centered classrooms: use of an audience kept. No other tests for reliability or validity were done. response system in pediatric dentistry education. J Dent Educ. 2005; 69(3):378–381. No validated satisfaction questionnaires on ARS use were 16. Barber M, Njus D. Clicker evolution: seeking intelligent design. found. Lastly, the generalizability of the results is limited CBE Life Sci Educ. 2007;6(1):1–8. because of the use of convenience samples rather than 17. Kelley KA, Beatty SJ, Legg JE, McAuley JW. A progress assess- ment to evaluate pharmacy students’ knowledge prior to begin- random sampling. These outcomes are specific to bacca- ning advanced pharmacy practice experiences. Am J Pharm Educ. laureate nursing students enrolled in a nursing pharmacol- 2008;72(4):88. ogy course. Further research is required on the impact 18. Foegen A, Hargrave CP.Group response technology in lecture-based instruction: exploring student engagement and instructor percep- of ARSs on student learning within other baccalaureate tions. J Spec Educ Tech. 1999;14(1):3–17. nursing courses. 19. Nosek TM, Wang W, Medvedev I, Wile MZ, O’Brien TE. Use of a computerized audience response system in medical student teaching: its effect on active learning and exam performance. In: Reeves T, Yamashita S, eds. Proceedings of World Conference on E-Learning REFERENCES in Corporate, Government, Healthcare, and Higher Education. Chesapeake, VA: AACE; 2006:2245–2250. 1. Collins LJ. Livening up the classroom: using audience response systems 20. Crossgrove K, Curran KL. Using clickers in nonmajors- and majors- to promote active learning. Med Ref Serv Q. 2007;26(1):81–88. level biology courses: student opinion, learning, and long-term 2. DeBourgh GA. Use of classroom ‘‘clickers’’ to promote acquisi- retention of course material. CBE Life Sci Educ. 2008;7(1):146–154. tion of advanced reasoning skills. Nurse Educ Pract. 2008;8(2): 21. Hoopes LL. Educator highlight: Morris Maduro. CBE Life Sci 76–87. Educ.2008;7:3–4. 3.DufresneRJ,GeraceWJ,LeonardWJ,MestreJP,WenkL.A 22. Freeman S, O’Connor E, Parks JW, et al. Prescribed active learning classroom communication system for active learning. JComputHigh increases performance in introductory biology. CBE Life Sci Educ. Educ. 1996;7(2):3–47. 2007;6(2):132–139. 4. Kraft RG. Group-inquiry turns passive students active. Coll Teach. 23. Elliot C. Using a personal response system in economics teaching. 1985;33(4):149–154. Int Rev Econ Educ. 2003;1(1):80–86. 5. Lowry PB, Romano NC, Guthrie R. Explaining and predicting 24. Holmes RG, Blalock JS, Parker MH, Haywood VB. Student ac- outcomes of large classrooms using audience response systems. curacy and evaluation of a computer-based audience response sys- Proceedings of the 39th Hawaii International Conference on System tem. J Dent Educ. 2006;70(12):1355–1361. Sciences, January 4–7, 2006. Los Alamitos, CA: Computer Society 25. Torbeck L. Enhancing programme evaluation using the audience Press; 2006. response system. Med Educ. 2007;41(11):1088–1089. 6. Caldwell JE. Clickers in the large classroom: current research and 26.PreszlerRW,DaweA,ShusterCB,ShusterM.Assessmentofthe best-practice tips. CBE Life Sci Educ. 2007;6(1):9–20. effects of student response systems on student learning and attitudes 7. Menon AS, Moffett S, Enriquez M, Martinez MM, Dev P, over a broad range of biology courses. CBE Life Sci Educ. 2007; Grappone T. Audience response made easy: using personal digital 6(1):29–41. assistants as a classroom polling tool. J Am Med Inform Assoc. 27. Latessa R, Mouw D. Use of an audience response system to augment 2004;11(3):217–220. interactive learning. Fam Med. 2005;37(1):12–14. 8. Abdallah L. Reflective teaching with technology: use of a personal 28. Pradhan A, Sparano D, Ananth C. The influence of an audience response system and publisher’s web site to enhance students’ per- response system on knowledge retention: an application to resi- formance in a nursing assessment and skills course. Online J Nurs dent education. Am J Obstet Gynecol. 2005;193:1827–1830. Inform [electronic resource]. 2008;12(1). http://ojni.org/12_1/abdallah. 29. Conoley J, Moore G, Croom B, Flowers J. A toy or a teaching tool? html. Accessed March 16, 2009. The use of audience-response systems in the classroom. Techniques. 9. Cain J, Robinson E. A primer on audience response systems: 2006;81(7):46–48. current applications and future considerations. Am J Pharm Educ. 30. Schackow TE, Chavez M, Loya L, Friedman M. Audience response 2008;72(4):77. system: effect on learning in family medicine residents. Fam Med. 10. Gamito EJ, Burhansstipanov L, Krebs LU, Bemis L, Bradley A. The 2004;36(7):496–504.

For more than 19 additional continuing education articles related to education, go to NursingCenter.com\CE.

334 CIN: Computers, Informatics, Nursing & June 2011

Copyright @ 2011 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.