Colleagues Committed to Redesign I Final Report – Introductory Spanish University of North Carolina at Chapel Hill

Submitted by:

Erika Bagley, Assessment Lead Department of Psychology

Bob Henshaw, Project Coordinator Information Technology Services

February 22, 2008

A. Impact on Student Learning

1. Improved Learning

For the fall 2007 pilot, the redesigned Spanish 101 hybrid format was assessed by comparing students’ performance to their counterparts’ in parallel sections of traditionally taught Spanish 101. A brief description of each format is included in Appendix A.

Student learning was assessed by comparing student scores on written and oral exams between the two formats. The written component was assessed using a composite score derived from the common sections on the final exam, including sections on grammar and written composition (common sections were approximately 1/3 of final exam). Results found no statistically significant difference between the scores of the traditional and hybrid sections on the written exam (TRAD 85.04 vs. HYBRID 85.55).

Oral interviews were conducted by instructors using a common protocol and scored using the same rubric. In addition, an outside grader was used to ensure that grading across instructors was consistent and reliable. Students in the traditionally taught sections outperformed hybrid students on the oral exam by a small, yet significant margin (TRAD 83.46 vs. HYBRID 81.11, p<0.05). It should be noted that the Department expected lower scores on the oral portion of the assessment, since the initial redesigned format does not devote as much time to conversation in the target language.

Additional information on the project assessment is included in Appendix B.

2. Improved Retention

WFD rates were higher in the traditionally taught format (TRAD 4.16% vs. HYBRID 1.15%), although it should be noted that WFD represented a very small number of students for both sections.

The complete course completion/retention results are included in Appendix C.

3. Other Impacts on Students

A student’s sense of classroom community, the extent to which he or she feels that the classroom is place of support and belonging, has been shown to relate to positive perceptions of performance and measures of actual performance (Lichtenstein, 2005). The Department was interested in the impact that fewer classroom hours would have on the students’ sense of community in the course. One potential drawback of the redesigned format is that students may be isolated and not feel as if they are a part of a learning community. The small group sessions were designed, in part, to reduce this potential problem.

In the survey given at the beginning of the semester, students were asked to what extent they enjoyed working with others and to what extent a sense of classroom community was important to them. The results showed that students in the two

formats did not significantly differ in their preference for group work or classroom community.

As part of the end-of-semester survey, students completed a Sense of Classroom Community questionnaire (McKinney, 2006). Students in the hybrid sections reported on their experience for the classroom and small group sessions separately. The results from this measure showed that students in the traditional sections had a significantly greater sense of community, as compared to the hybrid classrooms and small groups, although it should be noted that the difference was much smaller for the peer conversation groups (TRAD 7.84 vs. HYBRID classroom 10.59 vs. HYBRID conversation group 8.97). Lower scores on this measure indicate higher levels of sense of community.

B. Impact on Cost Savings

During the fall 2007 pilot, the per-student cost for the five hybrid sections offered was 38% lower than the per-student cost for the five traditional sections. When number of annual sections is increased from 20 to 30 during full implementation, the Department should realize savings of up to 46%. The fully-implemented redesign is expected to decrease the cost-per-student from $415 to $226.

Those savings would be even higher, but are slightly off-set by the Department’s decision to include two traditional sections per semester in the full implementation.

Though difficult to quantify in financial terms, the projected reduction in the use of classroom space will be significant. The hybrid model effectively cuts the use of classroom space for the course in half and consolidates all face-to-face interaction in a single multi-purpose room. After full implementation, the space savings will provide the Department with annual classroom capacity for 26 additional three-credit courses.

C. Lessons Learned

1. Pedagogical Improvement Techniques

• Peer conversation groups

The use of small peer-led conversation groups builds on a growing body of educational research literature on the effectiveness of peer learning groups (Arendale, 2005). The absence of an instructor in this setting lowers the affective filter, and students appeared to be more inclined to take risks and make mistakes in their spoken language practice. Student satisfaction with this component of the redesign was higher than that for both the online software and hybrid classroom sessions with instructors, and students indicated they felt a greater sense of community in these sessions than in the classroom meetings. A more thorough assessment of its pedagogical value will be conducted later this spring.

• Online software and student learning preferences

In the hybrid format, students can submit their online assignments any time before the weekly deadline. In the end-of-semester surveys conducted for the

pilot, many students indicated that they appreciated the ability to work ahead at their own pace. Several students said the ability to work through online assessments as many times as they liked made it easier for them to identify and rectify problem areas in their understanding of course concepts. A small percentage of students worked beyond the assigned material, taking advantage of extra assessments available through the online software.

2. Cost Reduction Techniques

• Restructuring staff time and roles

By reducing the number of contact hours from 4 to 1 per week and standardizing course materials, instructors can teach two sections of the hybrid as the equivalent of a regular course load for one traditional section. Also critical to the success of this model is the availability of the peer tutoring students who conduct the weekly peer conversation sessions (3 per section). These students receive academic credit through the campus Learning Center for the time they spend tutoring.

• Standardizing and re-using course syllabi and learning materials

Roughly 75% of the course materials used during the pilot will be reused in subsequent semesters. Those materials include all online assignments, low- stakes classroom assignments, syllabi, etc..

• Consolidating face-to-face course interaction into one location

Facilities-related savings for this course are contingent on the use of a larger, multi-purpose classroom that can accommodate up to three simultaneous small- group conversations (5-7 students each). The Department was able to convert its old student computer lab for this purpose. It has now effectively reduced the use of classroom space for the course by half, and has the capacity to double the number of annual sections using this facility alone. Note that any traditional sections that continue to be offered will likely occupy regular classroom space, slightly off-setting overall capacity gains realized through the redesign.

3. Implementation Issues

• Importance of qualified instructors

Full implementation of the course will be contingent upon identification of instructors who are a good fit with this format. Based on the results of the pilot, the Department feels that instructors best qualified to teach the hybrid format are those who 1) are comfortable working in online environment, 2) do not need the traditional number of contact hours with students to be satisfied professionally and 3) are experienced enough to be able to quickly establish rapport with students and identify student problems and issues. While there may be some exceptions, most graduate instructors will not be asked to teach the hybrid sections.

• Time devoted to conversation in target language

As noted in the section on learning outcomes, performance on the oral interviews was slightly lower for students enrolled in the hybrid section. Instructors were not surprised, given that less time was devoted to the spoken language in this format. Next fall, however, the Department will likely modify the redesign to provide more time-on-task in this area. One option instructors are considering is utilization of an online conversation feature in the online software (En Línea by Vista Higher Learning) that is already integrated into assignments.

• Pacing coverage of course material

At the end of the pilot semester, instructors teaching the hybrid discovered that a number of students had not gotten enough review during the semester to master key concepts. After budgeting a week and a half for each lesson during the pilot, they are devoting two weeks to each lesson during the spring 2008 semester.

• Student adjustment to course format

Most students enrolled in the Spanish 101 hybrid had little or no experience taking courses in a non-traditional format. Student resistance to new course formats is not unusual (NCAT, 2007), and this redesign was no exception. As a group, students in the traditional sections expressed higher satisfaction (at statistically significant levels) with the course than their counterparts in the hybrid sections. Students in the traditional sections stated that they would recommend the course and found the course to be a good fit for their learning style at higher rates than in the hybrid sections. The students’ self-assessment of their learning was more positive for the traditional format, even though no differences were found on objective measures of written performance or final grades.

In addition to providing an orientation for students on the first day of class, the Department is asking instructors teaching the hybrid sections to offer two mandatory office hours each semester (only one meeting was required during the pilot). This gives instructors additional opportunities to identify problems with the course and reinforce student guidelines for success. The Department is also considering implementation of a more comprehensive orientation program for instructors teaching the course next fall.

• Supplemental assistance

In order to provide students in the hybrid with additional learning supports, the Department offered a drop-in tutoring service where students could get assistance with course assignments. It was staffed 15 hours per week by work- study students who were already being paid to monitor the Department’s Foreign Languages Resource Lab. Though originally intended to support students in the hybrid sections, it was also made available to students in the traditional sections. This resource was largely underutilized by students. At mid-term, fewer than 10% of students in the hybrid sections had taken advantage of the service.

D. Sustainability

The Romance Language Department is satisfied that this model can meet its primary objective of expanding course capacity by making more efficient use of instructor time and classroom facilities. The redesign has not impacted learning outcomes negatively in any ways that were not anticipated. As long as student learning outcomes do not slip, the Department is committed to full implementation.

In fact, the Department has already expanded use of the hybrid model within its Spanish language program. During the spring 2008 semester it is also being used for five sections of Spanish 102, the companion introductory course for Spanish 101.

APPENDIX A

Pilot Course Format Descriptions

Redesigned Sections (5)

a.. 1 50-minute instructor-led classroom meeting per week, 1 50-minute peer-led conversation session per week b.. Online software ( En Línea by Vista Higher Learning) constituted primary learning materials c.. At least one office hour mandatory for all students

Traditional Sections (5)

a.. 4 50-minute instructor-led classroom meetings per week b.. Textbook constituted primary learning materials c.. No mandatory office hours

APPENDIX B

Pilot Assessment Results

Institution: University of North Carolina at Chapel Hill Course Title: Introductory Spanish 101

Measures A, B or C

In the performance sections of the chart, report the mean score and standard deviation for each group of students assessed.

Total # of Written Exam Oral Interview Students

Traditional Course 96 85.04, sd=9.04 83.46, sd=7.85

Timeframe: Fall Semester 2007

Redesigned Course 87 85.55, sd= 8.440 81.11, sd=7.35

Timeframe: Fall Semester 2007

Measures D or E

In the performance sections of the chart, report the percentage of students at each level of performance (for example, the percent earning a grade of “A”, percent “B”, etc.; or the percent rated at each level of a scoring rubric.)

Traditional Course Timeframe Fall semester 2007

Score/Grade Number Percentage

A 40 42% B 37 38% C 15 16% D 3 3% F 1 1%

Total 96 100%

Redesigned Course Timeframe Fall semester 2007

Score/Grade Number Percentage

Number Percentage A 19 22% B 48 55%

C 19 22% D 0 0% F 1 1%

Total 87 100%

2. Were any differences in performance between the two groups statistically significant?

X Yes (oral interviews only). At what level of confidence? p<0.05 __ No

3. Did the two groups of students assessed differ from one another in any important ways (e.g. gender balance, prior preparation levels, motivation, etc.)? If so, please describe these differences briefly:

After taking placement exams for entry into Spanish 101, students were able to enroll in either the traditionally taught or the redesigned hybrid course. To minimize concerns regarding possible bias introduced by self-selection, an extensive survey was administered at the beginning of the semester. In addition to providing data on student demographics and reasons for taking the course, the survey also included a measure of student learning characteristics. The Motivated Strategies for Learning Questionnaire (Pintrich, et al, 1991) included subscales for Intrinsic Goal Orientation, Extrinsic Goal Orientation, Expectancy and Study Skills. Analysis of the survey revealed no significant differences in the learning characteristics of students who chose the traditional versus hybrid sections. Likewise, analysis of the students’ scores on the placement exam found no significant differences in the scores of students in either format. In fact, it was found that the majority of the students were not aware that two different formats of the course were being taught, so it is likely that their decisions concerning which section to enroll in was primarily guided by their schedules, making self-selection bias minimal.

Neither student gender, ethnicity, nor class standing had a significant correlation with student learning outcomes, therefore the assessment team did not control for these student variables. Of the student learning characteristics, only intrinsic motivation and study skills were positively correlated with the written exam scores and only for the hybrid sections.

4. Did you learn anything else about the impact of the redesign on students (e.g. changes in student attitudes toward the subject, better performance in downstream courses in the same discipline, etc.)? If so, please describe these differences briefly:

The project will be in a better position to assess these impacts at the end of the spring 2008 semester. Most of the students who started Spanish 101 in the Fall of 2007 are currently enrolled in Spanish 102, which not only provides the opportunity to continue to track their learning outcomes, but also to see how their perceptions of the course change as they become more accustomed to the new hybrid format. In addition, a new cohort of students enrolled in Spanish 101 in the Spring of 2008. They will be given all of the same surveys/measures as the previous cohort, adding to the assessment’s statistical power. This new cohort also provides the opportunity to assess aspects of the redesign that were overlooked in the initial evaluation.

APPENDIX C

PILOT COURSE COMPLETION/RETENTION RESULTS

Institution: University of North Carolina at Chapel Hill Course Title: Introductory Spanish 101

Traditional Course

Timeframe Fall semester 2007

Score/Grade Number Percentage

A 40 42% B 37 38% C 15 16% D 3 3% F 1 1%

Total 96 100%

Redesigned Course

Timeframe Fall semester 2007

Score/Grade Number Percentage

Number Percentage A 19 22% B 48 55% C 19 22% D 0 0% F 1 1%

Total 87 100%

Your definition of successful completion:

Final grade of D or better.

Your definition of retention:

The Department defines retention as successful completion of the course, with at least a D grade. Many students drop the course simply to shift to another section, but the Department does not track this movement. The College of Arts and Sciences has recently instituted a new policy that prohibits students from dropping courses within the language requirement once they are enrolled. They can shift to another section, but must gain the approval of the Dean to drop the course altogether. The project assessment team will be in a better position to track this data during the spring 2008 semester.

APPENDIX D

References

Arendale, D.R. (2005). Postsecondary peer cooperative learning programs: annotated bibliography. www.eric.ed.gov (accession no. ED489957; accessed February 19, 2008).

Duncan, T.G & McKeachie, W.J. (2005). The making of the motivated strategies for learning questionnaire. Educational Psychologist , 40(2), 117-128.

Lichtenstein, M. (2005). The importance of classroom environments in the assessment of learning community outcomes. Journal of College Student Development, 46(4), 342-356.

McKinney, J. P., McKinney, K. G., Franuik, R. & Schweitzer, J. (2006). The college classroom as a community. College Teaching, 54(3), 281-284.

National Center for Academic Transformation. 2007. Five critical implementation issues. http://www.thencat.org/PlanRes/R2R_Imp_Issues.htm#One (accessed February 20, 2008)

Pintrich, P.R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review , 16(4), 385-407.