International Journal of Teaching and Learning in Higher 2020, Volume 32, Number 2, 293-304 http://www.isetl.org/ijtlhe/ ISSN 1812-9129

Collaborative Learning Techniques, Student Learning Outcomes, and Equal Workload within Groups in Different Teaching Modalities

Megumi Kuwabara, Susan D. Einbinder, Rui Sun, and Roya Azizi California State University Dominguez Hills

A Faculty Learning Community comprised of four faculty members evaluated their work of implementing collaborative learning techniques (CoLTs) into their graduate and undergraduate courses that included different teaching modalities (traditional classroom, hybrid, and online). Two research questions were examined: a) Did students perceive that the implementation of CoLTs facilitated their mastery of course-specific student learning outcomes?, and b) Did students perceive that their group members worked equally? A total of 133 students participated in this study by filling out a survey asking for their evaluation of mastery of student learning outcomes and peer evaluation of group members' collaborative effort. Results show that the implementation of CoLTs facilitated students’ perception that they mastered the course-specific learning outcomes and that the workload was equally distributed among their group members. The contributions of current work and the potential use of the student survey used in this study are discussed.

Collaborative learning is a form of group learning 2014; Kilgo, Sheets & Pascarella, 2015; Nelson, 1994), during which two or more students in a class work enhanced student persistence in math courses after together and share workload equitably to complete failure (Lan & Repman, 1995), increased student’s assignments that are intentionally created to meet the ability to transfer knowledge gained in one class to student learning outcomes of the class (Barkley, Major, another (Loes & Pascarella, 2017; Wright, Millar, & Cross, 2014). Collaborative learning is rooted in two Kosciuk, & Penberthy, 1998), and deepened the level of learning theories: one is social constructivism by learning (Vogt & Skop, 2017). In addition, a study Vygotsky (1978) stating that knowledge is constructed found that collaborative learning generated higher by socially interacting with other individuals; another is benefits for students of color and students with the observational learning theory by Bandura (1977) educational challenges (Loes & Pascarella, 2017). stating that knowledge is gained by imitating and The benefits of CoLTs are not only academic but modeling other individuals. By implementing also extended to social and psychological aspects. For collaborative learning, instructors engage three of the example, Johnson, Johnson, and Smith (2014) showed seven principles for good education practices suggested that the implementation of CoLTs facilitated liking by Chickering and Gamson (1987): encouraging student- among peers in the class, increased psychological faculty contact, facilitating cooperation and learning adjustment (e.g., self-efficacy and self-esteem), and among students, and . Implementing better attitudes toward the university and learning. Laal collaborative learning in the classroom puts an instructor and Ghodsi (2012) showed that collaborative learning in a position of a facilitator and a guide of learning rather increased the awareness of diversity, reduced anxiety, than a deliverer of knowledge, and it put students in and increased positive attitudes toward instructors. charge of their learning (Flannery, 1994). While Courses with collaborative learning also facilitated instructors in higher education have utilized student students’ communication skills, problem-solving skills, in classes, collaborative learning innovative or creative thinking, and better working with techniques (CoLTs) represent planned and intentional their team in engineering courses compared with the methods to facilitate collaboration in the class. The traditional courses (Terenzini, Cabrera, Colbeck, CoLTs range from simple (e.g., think-pair-share) to Parente, & Bjorklund, 2001). complex (e.g., jigsaw) techniques, but all techniques The majority of the studies on the effectiveness of must be individualized and customized to suit the CoLTs focused on STEM courses and traditional contents taught in the class (Barkley et al., 2014). graduate or undergraduate courses. It is not clear Implementing CoLTs in college and university whether the effectiveness of CoLTs can be expanded to courses has consistently been shown to enhance different teaching modalities (traditional classrooms, learning. For example, a meta-analysis conducted by hybrid, and online) and different course levels Springer, Stanne, and Donovan (1999) showed that (undergraduate- and graduate-level courses). Therefore, STEM (Science, Technology, Engineering, and Math) in this study we intend to fill in the research gap by classes in undergraduate-level courses improved examining whether the implementation of CoLTs student exam grades. Other studies have shown that facilitated students’ perception of mastery of student CoLTs improved student’s critical thinking skills and learning outcomes in courses of different modalities engagement in classes (Clem, Mennicke & Beasley, and at different levels. Kuwabara, Einbinder, Sun, and Azizi Collaborative Learning Techniques 294

Table 1 Course Chosen for Evaluation by Level, Modality/Number of Sections, and Response Rate Response Rate (No. of responses/ Course Level Modality/sections No. of enrollment) Undergrad 1 Undergraduate Traditional - 2 98% (59/60) Undergrad 2 Undergraduate Online -1 93% (26/28) Grad 1 Graduate Hybrid - 1 81% (13/16) Grad 2 Graduate Traditional - 2 80% (35/44)

Table 2 Sample Distribution by Sex, per Class and Percentage of the Total Sample (n=133) Undergrad 1 Undergrad 2 Grad 1 Grad 2 Total N % N % N % N % N % Female 58 44 18 14 9 7 30 23 115 86 Male 1 1 8 6 4 3 5 4 18 14 Total 59 44 26 20 13 10 35 26 133 100 Note: percentage differences are due to rounding.

Table 3 Sample Distribution by Race/Ethnicity, per Class and Percentage of the Total Sample (n=133) Undergrad Undergrad 1 2 Grad 1 Grad 2 Total No. % No. % No. % No. % No. % African Americans/non- 3 2 2 2 6 5 7 5 18 14 Hispanic African 7 5 2 2 2 2 3 2 14 11 American/Hispanic Asian/Pacific Islander 3 2 4 3 0 0 4 3 11 8 Caucasian/Non-Hispanic 2 2 5 4 2 2 4 3 13 10 Caucasian/Hispanic 29 22 11 8 1 1 15 11 56 43 Other 15 11 2 2 2 2 2 2 21 16 Total 59 44 26 20 13 10 35 26 133 100

Furthermore, we investigated a part of the definition of over three consecutive semesters in a campus-sponsored, collaborative learning, “sharing workload equitably,” by interdisciplinary Faculty Learning Community (FLC). asking students whether they thought themselves and their The four faculty incorporated CoLTs into their classroom peers in their groups contributed equally to the work during instruction and assessed its effectiveness using a student the CoLTs implemented activities. Based on our best survey in the fall semester of 2017. knowledge, this part of the definition of collaborative learning has not been studied extensively. We found a study Participants mentioning that despite the benefits of collaboration in the classrooms, many students complained about the unequal Table 1 contains details about the four courses workload of group members to complete the collaborative chosen for this study, which included two graduate and activity (Chang & Brickman, 2018). Therefore, our study two undergraduate courses. makes another contribution to the literature about the equal One course was an asynchronous online course, workload aspect of collaborative learning. another was a hybrid, and the remaining two were offered in a traditional classroom setting. Methods A total of 148 students were enrolled in these courses that utilized CoLTs. At the end of the fall 2017 semester, Three full-time and one part-time faculty at an urban, 133 students completed the survey, resulting in a response public university in Southern California, worked together rate of 90 percent overall, with both graduate classes Kuwabara, Einbinder, Sun, and Azizi Collaborative Learning Techniques 295

generating lower response rates of 81 and 80 percent, facilitating the mastery of student learning outcomes based respectively, while both undergraduate classes generated on students’ perceptions. response rates of 93 and 98 percent, respectively. Undergraduate students comprised 64 percent of the Procedure sample, and 86 percent were female, as shown in Table 2. Table 3 shows the race/ethnicity of the sample by The four faculty members who are co-authors of class. This distribution mirrors the diverse student this study originally began collaborating in a one- population of the university. semester FLC convened in spring 2017 after each member responded to a request to participate by the Materials university’s Faculty Development Center. During that period, each faculty member designed an individualized The student survey designed for this study CoLT in his or her respective class. We requested, and contained three sections (see Appendix A). The first were granted permission, to continue our work together section solicited the student’s signature indicating in Fall 2017, during which time we implemented one or whether or not the data could be used in this study (the more CoLTs, which are listed in Appendix B, and the informed consent text was included on the assignment survey for evaluation and assessment of CoLTs were guidelines, distributed to students before the survey). given to the students in each of the courses. The types This section also asked for each student’s year of birth, of CoLTs used in this study were introduced by Barkley gender, and race/ethnicity. The second section asked and colleagues (2014). each student to rate how well the student and other To assess the reliability of the survey instrument, students in the collaborative group collaborated. The Cronbach’s alphas were conducted within each class third section asked each student to evaluate how well and across all of them. Factor analysis was conducted the class-specific CoLTs facilitated mastering class- to test whether all eight questions used in the survey specific student learning outcomes. instrument were necessary or if a more parsimonious Due to the heterogeneity of the courses in this study, number of questions could be used instead. Based on and after reviewing existing instruments, a publicly Cronbach’s alphas, the alpha coefficients were available assessment of student collaboration (Manis, extremely high (the overall alpha was .96; .93 for 2010-2014) was chosen (See Appendix A – Section 2). Undergrad 1, .86 for Undergrad 2, .92 for Grad 1, and In the second section of the student survey, the .94 for Grad 2), suggesting that the survey questions assessment consisted of eight statements about were reliable. Factor analysis of the eight questions collaboration and asked the student to evaluate the evaluating collaborative activities yielded one factor student's work as well as the work of each member of the explaining a total of 78.73% of the variance (see group of students who had worked together on a class Appendix C), thus that the survey questions measured assignment using a rating scale where 1=weak, 2=below one concept, collaboration among students. average, 3=average, 4=above average, and 5=superior. The questions addressed (1) participating in group Results discussions, (2) staying task-focused, (3) contributing usefully, (4) listening to others, (5) professionally and CoLT effectiveness thoughtfully challenging seemingly incorrect responses, (6) noting the quality of contribution, (7) consistently and To test the first research question about whether the actively working toward the group goal, and (8) implementation of CoLTs facilitated students’ perception of collaborating well. Two open-ended questions were mastery of student learning outcomes taught in these included as well. The first asked the respondent to courses, in the third section of survey questions students identify how to improve the collaborative work, and the were asked to rate how well the CoLTs helped them master other asked how the team of students might improve student learning outcomes in each respective course, and their future collaborative efforts. this was used as a dependent variable. Using a Likert scale The third section of the student survey was (0=Not at all; 1=A little; 2=A lot; 3=Totally) across all customized for each class. The section started with the classes and within individual course, students rated that the following: “The purpose of this activity was to facilitate implementation of CoLTs helped them master the student student mastery of the following course outcomes,” and it learning outcomes as seen by the average rating (M=2.53, included a list of student learning outcomes of respective SD=.60 for all courses; M=2.68, SD=.58 for Undergrad 1; courses. This section asked each student to rate how well M=2.56, SD=.62 for Undergrad 2; M=2.15, SD=.63 for the CoLTs helped them master student learning outcomes Grad 1; M=2.42, SD=.53 for Grad 2). Majorities of students in each respective course using a Likert scale (0=Not at also rated this question as 2 (a lot) or more (Undergrad 1 = all; 1=A little; 2=A lot; 3=Totally). A score of 2 or more 98.2%, Undergrad 2 = 88.5%, Grad 1 = 92.3%, Grad 2 = on this question was evidence that the CoLTs helped 80%), suggesting that there was an agreement among Kuwabara, Einbinder, Sun, and Azizi Collaborative Learning Techniques 296

students on how helpful the CoLTs were for them to master ratings as a dependent variable and teaching modality the student learning outcomes. (traditional, hybrid, or online) as an independent To compare students’ perception of effectiveness variable were entered into ANOVA, which yield of CoLTs based on class level, the student rating of significant difference among traditional (M=0.19, student learning outcomes as a dependent variable and SD=.28), hybrid (M=.46, SD=.33), and online (M=.22, course level (undergraduate or graduate) as an SD=.26) courses, F(1, 129)=4.65, p<.05. independent variable were entered into ANOVA, which yield significant difference in students’ ratings between Conclusion undergraduate (M=2.64, SD=.59) and graduate level (M=2.35, SD=.56) courses, F(1,129)=7.66, p<.01. This This study investigated two research questions. shows that students in undergraduate-level courses The first question asked whether the effectiveness of rated the implementation of CoLTs more favorably than CoLTs expands to different teaching modalities did those in graduate-level courses. (traditional, hybrid, and online) and different course To compare students’ perception of effectiveness of levels (undergraduate- and graduate-level courses). For CoLTs based on teaching modality, the student rating as this question, we specifically tested whether the a dependent variable and teaching modality (traditional, implementation of CoLTs facilitated students’ hybrid, or online) as an independent variable were perceptions of the mastery of student learning entered into ANOVA, which yielded marginal significant outcomes. The second question asked whether students difference between traditional (M=2.58, SD=.57), hybrid perceived themselves and their peers in their groups (M=2.15, SD=.63), and online (M=2.56, SD=.62) contributed equally to the work during the CoLTs courses, F(1, 129)=3.02, p=.052. implemented activities. For the first question, we found that although undergraduate students perceived the Equal Workload CoLTs more favorably than did the graduate students, the majority of students across all classes (all modalities To test the second research question, whether students across all levels) perceived the implementation of perceived the implementation of CoLTs made the workload CoLTs to be an effective way to master their course- among group members equally, we used the second section specific student learning outcomes. For the second of survey in which students rated eight questions about question, although there were differences by teaching themselves and their group members on how well they modalities, we found that the majority of students collaborated. To do this, we calculated the standard perceived the workload among themselves and their deviation of the rating given by each student to measure the group members during the CoLTs implemented activity variability in rating, which suggested the variability in to be equal. These findings strongly suggest that the workload or collaborative effort. Therefore, the smaller the CoLTs introduced in these classes facilitated students’ standard deviation given by a particular student was, the perceptions of mastery of course-specific student better the student’s perception of equal workload among learning outcomes and effectively enhanced students’ group members (including themselves) the student had. We abilities to collaborate by equalizing the workload counted how many students’ ratings fell between zero to among their group members. one standard deviation and zero to 0.5 standard deviations This study contributes to the expansion of findings for each course (see Appendix D for an example rating showing the effectiveness of CoLTs in different teaching given by a student). Overall, 97.7% of students’ ratings fell modalities and different course levels. It also adds into the range of standard deviation of zero to one (98.3% knowledge to a new line of research investigating the for Undergrad 1, 100% for Undergrad 2, 91.7% for Grad 1, workload among students during the CoLTs- and 97.1% for Grad 2), and 84.6% of students’ ratings fell implemented activities, which by definition should make into the range of a standard deviation of zero to 0.5 (89.8% the workload equitable. Our findings confirm that by for Undergrad 1, 84.0% for Undergrad 2, 50.0% for Grad 1, implementing CoLTs, students perceived the workload to 88.2% for Grad 2), suggesting that students felt that they be equal among group members. Another contribution and their group members contributed to the activity equally. of this study is the survey instrument, which is reliable To compare students’ perception of equal workload and can be individualized to measure specific course based on class level, the standard deviation of ratings as outcomes for any type of course in any modality. a dependent variable and course level (undergraduate or graduate) as an independent variable were entered into Limitations ANOVA, which yielded no significant difference between undergraduate (M=.20, SD=.28) and graduate There are several caveats to this study. First, the level (M=.26, SD=.31) courses, F(1,129)=1.44, p>.05. students examined in this study were not randomly To compare students’ perception of equal workload sampled: they were students of the instructors who based on teaching modality, the standard deviation of were committed and actively working to improve the Kuwabara, Einbinder, Sun, and Azizi Collaborative Learning Techniques 297

quality of their teaching. Thus, the findings of this Barkley, E. E., Major, C. H., & Cross, K. P. (2014). study cannot be generalized to all students, but they can Collaborative learning techniques: A handbook for probably describe students in classes with faculty who college faculty (2nd ed.) San Francisco, CA: Jossey- devote time to designing and implementing CoLTs and Bass. those who would be just as enthusiastic and invested in Chang, Y., & Brickman, P. (2018). When group work their teaching as the four instructors in this study. doesn’t work: Insights from students. CBE Life Second, this was a non-experimental, post-test only Science Education, 17(3), 1-17. study with a design that relied exclusively on students’ Chickering, A. W., & Gamson, Z. F. (1987). Seven perceptions for the dependent variables. The nature of this principles for good practice in undergraduate research effectively rules out using quasi-experimental or education. American Association for Higher experimental designs. Future studies would benefit from Education (AAHE) Bulletin, 39(7), 3-7 triangulation, including student course grades, written Clem, J. M., Mennicke, A. M., & Beasley, C. (2014). comments in the instructor’s teaching evaluation, and a pre- Development and validation of the experiential test inventory of perceived collaborative skills would learning survey. Journal of Social Work Education, enhance the rigor of the protocol. 50, 490-506. doi: 10.1080/10437797.2014.917900. Finally, the effectiveness of the collaboration Flannery, J. L. (1994). Teacher as co-conspirator: among students might differ based on the demographics knowledge and authority in collaborative learning. New of students, which we did not have enough samples to Directions for Teaching and Learning, 59, 15-23. investigate. For example, a previous study has found Johnson, D. W., Johnson, R. T., & Smith, K. A. (2014). that cultural backgrounds, such as individualistic or Cooperative learning: Improving university instruction collectivistic cultures, influenced the students’ by basing practice on validated theory. Journal on perception of collaboration (Popov et al., 2014). Excellence in College Teaching, 25, 85-118. Another study found that immigration status influenced Kilgo, C. A., Ezell Sheets, J. K., & Pascarella, E. T. the effectiveness of collaboration (Stebleton, Soria, (2015). The link between high-impact practices and Alexixo, & Huesman, 2012). In addition to the student learning: Some longitudinal evidence. demographic of students, we did not measure or Higher Education, 69, 509-525. evaluate how much learning climate might influence doi:10.1007/s10734-014-9788-z the effectiveness of CoLTs. For example, a previous Laal, M., & Ghodsi, S. M. (2012). Benefits of study found that having staff support increased the collaborative learning. Procedia – Social and effectiveness of collaboration in the classroom setting Behavioral Sciences, 31, 486-490. (Lizzio & Wilson, 2015). Lan, W. Y., & Repman, J. (1995). The effects of social learning context and modeling on persistence and Recommendations for Future Research dynamism in academic activities. Journal of Experimental Education, 64(1), 53-67. Replicating this study with other instructors who Lizzio, A., & Wilson, K. (2005). Self-managed learning have designed and implemented CoLTs in their classes, groups in higher education: Students’ perception of and adding course grades, written comments from process an outcomes. British Journal of teaching evaluations, and a pre-test of perceived , 75, 373-390. collaborative skills are recommended. A larger sample Loes, C. N., & Pascarella, E. T. (2017). Collaborative would also enable detailed analyses of student learning and critical thinking: Testing the link. The characteristics to evaluate whether gender, age, race, or Journal of Higher Education, 88(5), 726-753 ethnicity differences emerge in collaborative doi:10.1080/00221546.2017.1291257 effectiveness. Ideally, systematically incorporating this Manis, C. (2010-2014). Cooperative learning: How to survey into all courses that include a CoLT would assign meaningful tasks to group members. Retrieved enable the campus to develop a longitudinal database of from https://www.dailyteachingtools.com/cooperative- CoLT effectiveness. As technological tools continue to learning-tasks.html grow, this database could be linked to overall student Nelson, C. E. (1994). Critical thinking and retention and graduation rates, allowing the university collaborative learning. New Directions for to model its work and to show that CoLTs enhance Teaching and Learning, 59, 45-58. student retention and graduation rates. Popov, V., Noroozi, O., Barrett, J. B., Biemans, H. J. A., Teasley, S. D., Slof, B., & Mulder, M. (2014). References Perceptions and experiences of, and outcomes for, university students in culturally diversified dyads in a Bandura, A. (1977). Social learning theory. Englewood computer-supported collaborative learning environment. Cliffs, NJ: Prentice Hall. Computers in Human Behavior, 32, 186-200. Kuwabara, Einbinder, Sun, and Azizi Collaborative Learning Techniques 298

Springer, L., Stanne, M. E., & Donovan, S. S. (1999). Carson, California, where she has been appointed a Effects of small-group learning on undergraduates Faculty Fellow at the campus Faculty Learning in science, mathematics, engineering, and Community in Fall 2018. Her interdisciplinary technology: A meta-analysis. Review of research explores how social policies can ameliorate Educational Research, 69(1), 21-51. poverty; translating Critical Race Theory into social Stebleton, M. J., Soria, K. M., Aleixo, M. B., & work practices; expanding the use of research methods Huesman Jr., R. L. (2012). Student-faculty and throughout social work education and professional peer interactions among immigrant college students practices; and evaluating different approaches to in the United States. Multicultural learning and improve teaching excellence. Teaching, 7(2), 1-21. doi:10.1515/2161-2412.1122. Terenzini, P. T., Cabrera, A. F., Colbeck, C. L., RUI SUN is a tenured Associate Professor in the Parente, J. M., & Bjorklund, S. A. (2001). Department of Public Administration at the California Collaborative learning vs. lecture/discussion: State University, Dominguez Hills (CSUDH). She Students’ reported learning gains. Journal of received her Ph.D. in Policy Studies from the Engineering Education, 90(1), 123-130. University of Maryland in 2009. Dr. Sun’s research Vogt, B. J. & Skop, E. (2017). The Silverton Field focuses on fiscal federalism, government budgeting, experience: A model geography course for achieving and nonprofit financial management. She has published high-impact educational practices (HEPs). Journal of in peer-reviewed journals and books. Dr. Sun has more Geography in Higher Education, 41(4), 574-589. than 10 years’ experience in teaching public doi:10.1080/03098265.2017.1331421 administration undergraduate and graduate courses both Vygotsky, L. S. (1978). Mind in society: The on campus and online. She is committed to High Impact development of higher psychological processes. Practices and has led the assessment of the public Cambridge, MA: Harvard University Press. administration programs at the CSUDH. Wright, J. C., Millar, S. B., Kosciuk, S. A., & Penberthy, D. L. (1998). A novel strategy for ROYA AZIZI received her doctoral degree in Policy, assessing the effects of curriculum reform on Planning, and Development from the University of student competence. Journal of Chemical Southern California. She is an adjunct assistant Education, 75(8), 986-992. professor at California State University, Dominguez ______Hills and California State University, Fullerton and an advisory council member of the Advancement of MEGUMI KUWABARA earned her Ph.D. in Sustainability in Higher Education (AASHE). Her Developmental Psychology from Indiana University, research interests are in the areas of sustainable policy Bloomington. She is an Associate Professor in Child and planning, mentorship, and collaborative learning Development Department at California State strategies. Dr. Azizi has been featured in KIRN 670am University, Dominguez Hills. Her research Radio programs regarding her work in sustainable investigates cross-cultural differences in cognitive policy objectives. development. In the areas of scholarship of teaching and learning, her research interests are the intersection Acknowledgment between emotion and learning, student learning, collaborative learning techniques, active learning, and We thank the students who participated in this study. evaluation of teaching strategies. We also acknowledge and thank Provost Michael Spagna, MA, Ph.D., Kara Dellacioppa, PhD., Director, SUSAN D. EINBINDER, MS, Ph.D., received her BA and Erika Torres, Assistant Director, Faculty from Barnard College and her MS and Ph.D. from Development Center at California State University Columbia University. She is a Professor and founding Dominguez Hills for awarding faculty grants enabling faculty member of the graduate MSW Program at our ongoing participation in this Faculty Learning California State University Dominguez Hills in Community from January 2017 through June 2018. Kuwabara, Einbinder, Sun, and Azizi Collaborative Learning Techniques 299

Appendix A

Student Survey

[Course Number and Title goes here]

Section 1

Your name______

Date______This evaluation of your class activity is routine component of higher education instruction and may be required by your instructor. You are being asked to allow your instructor to use your feedback in a research study that will include similar feedback from a minimum of 200 CSUDH students in undergraduate and graduate classes.

I agree to allow this feedback to be included in the instructor’s research. I do NOT agree to allow this feedback to be included in the instructor’s research.

Your signature______

Demographic Characteristics Please write down the year in which you were born ______Please pick the one response that best describes your gender. Female Male Trans Other [Please describe______] Please pick the one response that best describes your race/ethnicity, which is presented using the nationally standardized classification system used by the US Census Bureau. African American/Black, non-Hispanic African American/Black, Hispanic Asian or Pacific Islander Caucasian/White, non-Hispanic Caucasian/White, Hispanic Other [Please describe ______]

Kuwabara, Einbinder, Sun, and Azizi Collaborative Learning Techniques 300

Section 2

Instructions: Write the names of your group members in the numbered boxes below. Then, assign yourself a value for each listed attribute. Then, do the same for each of your group members. When you have finished, submit this online on Blackboard. Score: 5=Superior 4=Above average 3=Average 2=Below average 1=Weak Attribute1 Myself 1. 2. 3. 4. 5.

Participated in group discussions

Stayed task-focused (Not distracted or distracting)

Contributed usefully

Listened to others

Professionally and thoughtfully challenged responses that appeared to be incorrect

Quality of contribution

Consistently and actively worked toward a group goal

Collaborated well

TOTAL

Additional comments about collaborating with your classmates (optional):

1 Manis, Chad. (2010-2014). Daily Teaching Tools blog. Copied with permission from http://www.dailyteachingtools.com/cooperative-learning-evaluate.html

Kuwabara, Einbinder, Sun, and Azizi Collaborative Learning Techniques 301

Open-ended qualitative questions

If you were to participate in this activity again, would you do anything differently to improve your work? Why/why not?

______

______

______

How could your team improve its collaboration efforts?

______

______

______

Section 3

The purpose of this activity was to facilitate student mastery of following course outcomes.

• Course Outcome X1: Write the student learning outcome from the course that this activity addresses.

• Course Outcome X2: Write the student learning outcome from the course that this activity addresses.

Using the scale below, please rank how well you think this activity accomplished its goal. 0=Not at all 1=A little Purpose Rank 2=A lot Course Outcome X1 3=Totally Course Outcome X2

Additional comments [optional]:

International Journal of Teaching and Learning in Higher Education 2020, Volume 32, Number 2, 293-304 http://www.isetl.org/ijtlhe/ ISSN 1812-9129

Appendix B

List of CoLTs used in each respective course Undergrad 1 Undergrad 2 Grad 1 Grad 2

Note Taking Pairs Round Robin Test-taking Teams Think-Pair-Share

Affinity Grouping Talking Chips Peer Editing Note-Taking Teams

Round Table Critical Debate Team Matrix Test-Taking Teams

Structured Problem Solving Learning Cell Jigsaw Jeopardy

International Journal of Teaching and Learning in Higher Education 2020, Volume 32, Number 2, 293-304 http://www.isetl.org/ijtlhe/ ISSN 1812-9129

Appendix C

Reliability of Survey Questions: Principal Component Analysis for Eight Survey Questionnaire Loading Participated in group discussion 0.886

Stayed task-focused (not distracted or distracting) 0.864

Contributed useful ideas 0.913

Listened to others 0.874

Professionally and thoughtfully challenged responses that appeared to be incorrect 0.816

Quality of contribution 0.908

Consistently and actively worked toward a group goal 0.918

Collaborated well 0.915

Eigenvalue 6.298

% of Variance 78.725

Kuwabara, Einbinder, Sun, and Azizi Collaborative Learning Techniques 304

Appendix D

Example rating given by a student based on the different standard deviation

SD = 0.5 Attribute Myself Peer 1 Peer 2 Peer 3 Peer 4 Participated in group 5 5 5 4 5 discussions Stayed task-focused (Not 5 5 5 4 5 distracted or distracting) Contributed usefully 5 5 5 4 5 Listened to others 5 5 5 4 5 Professionally and 5 5 5 4 5 thoughtfully challenged responses that appeared to be incorrect Quality of contribution 5 5 5 3 5 Consistently and actively 5 5 5 4 5 worked toward a group goal Collaborated well 5 5 5 4 5

SD = 1.03 Attribute Myself Peer 1 Peer 2 Peer 3 Participated in group 3 3 3 5 discussions Stayed task-focused (Not 3 2 3 5 distracted or distracting) Contributed usefully 3 2 3 5 Listened to others 4 4 3 4 Professionally and 3 3 3 5 thoughtfully challenged responses that appeared to be incorrect Quality of contribution 4 2 3 5 Consistently and actively 5 4 3 5 worked toward a group goal Collaborated well 5 3 3 4