Committee Name: Liberal Studies Committee

Submitted: October 5, 2011

For Senate meeting: October 18, 2011

Chair: Mark Paulsen

Subject: Assessment Report for World Cultures Graduation Requirement

This report is informational only. No Senate action is being recommended.

One of the ongoing tasks on the liberal studies committee is to oversee assessment of student outcomes with respect to the goals of each division in the liberal studies curriculum as well as the overall skills and understandings that students who complete their liberal studies courses are supposed to master. The committee has chosen to assess the different components of the liberal studies program on a rotating basis with different division selected annually for assessment. This approach keeps the amount of data collected each year to a manageable level at the cost of relatively infrequent assessment. A division will only be assessed every five to six years. In its current form, the liberal studies program emphasizes the following abilities and understandings:

 Ability to write and communicate clearly and effectively  Ability to evaluate various forms of evidence and knowledge  Ability to engage in analytical reasoning and argumentation  Ability to engage in quantitative analysis  Ability to engage in scientific inquiry and processes  Ability to see across disciplinary boundaries

 Understanding cultural diversity in the United States  Understanding the world as a diverse and interrelated community  Understanding the relationship of the individual to society and its culture and institutions  Understanding the role of the fine and performing arts and the humanities in shaping and expressing a culture’s values and ideals  Understanding natural phenomena and the physical world  Understanding multiple problem-solving perspectives

For 2011, the committee selected the World Cultures requirement for assessment. All Northern Michigan University students must take at least one course of three or more credits designated to meet the world cultures requirement. These courses deal primarily with living cultures of Asia, Africa and the non-Anglo-Saxon Americas. At least two-thirds of the content of each course must emphasize the patterns of thought and action that account for the distinctive ethos of the world culture(s) being studied. Courses may include such topics as literary, artistic and other cultural achievements; religious and ethical values; social, economic and political systems; or intellectual and historical trends.

According to the current bulletin, students completing the world cultures requirement should be able to:  Comprehend and articulate the distinctive world view (e.g., values, norms and beliefs) of at least one culture that varies significantly from Anglo-American and Western European cultures.  Understand how culture is expressed in terms of artifacts, artistic accomplishments, technology, customs and texts.  Understand and respect social and cultural diversity and complexity in a global context;  Understand how factors such as racial, ethnic, gender and class differences affect how groups within a culture relate to each other.  Articulate the important achievements and contributions of other cultures in such areas as the arts, literature, philosophy, ethical values, religion and science.

Assessment Procedure

The assessment plan for world cultures was modeled on the approach used previously for Division V and Division III. Representative student artifacts were collected at the end of the semester from faculty teaching World Cultures courses. An assessment team then scored the artifacts using a rubric developed by the liberal studies committee and the committee would then draft a report based on feedback from the assessment team.

Dr. Brunswick of the liberal studies committee determined that courses in Art and Design, Anthropology, Broadcasting, English Literature, Geography, Health, Human Nutrition, History, Native American Studies, Philosophy, and Modern Languages met the criteria for World Cultures and would be offered in the Winter 2011 term. A list of these courses is included in Appendix A. A total of 1139 students were enrolled in these courses. Discussion with the Office of Institutional Research determined that a random sampling of around 170 students would give us a confidence level of 95%. Institutional Research generated a randomly selected list representing approximately 15% of the students enrolled in each course.

In March of 2011, instructors of World Cultures courses were contacted by the chair of the liberal studies committee and given a list of students for whom they were to submit copies of artifacts turned in by the students. Examples of students' work could include research papers, written essays, projects, investigations, assessments, tests, or final exams. The artifacts were collected by the Chair of the committee and any identifying features were removed as best as possible.

The Liberal Studies Committee generated an assessment rubric based on the goals of the World Culture requirement as outlined in the undergraduate bulletin. A copy of the rubric is included in appendix C. The assessment team included Kia Richmond from English Education, and Judy Puncochar from Educational Research. Mitchell Klett acted as the coordinator of the assessment team but did not participate in ranking any student artifacts. The liberal studies committee would have preferred a three person assessment team which has been past practice but we were unable to recruit a third assessor in a timely fashion.

World Cultures Outcomes

Artifacts representing the work of 94 students were submitted for assessment. Student artifacts were evaluated by the team as either; exceeds expectations (strongly agree), meets expectations (agree), nearing expectations (disagree) or fails to meet expectations (strongly disagree) in the four World Culture outcomes listed in the rubric. Data from the assessment team is tabulated in Appendix B. Several observations can be made based on this assessment

1. None of the artifacts sampled showed students were exceeding expectations 2. Seventy percent of the artifacts sampled showed students were meeting expectations. 3. Eight percent of artifacts sampled showed students were nearing expectations. 4. There were 22% of the artifacts sampled showed students did not meet minimal expectations. 5. Like the other recent assessment, those faculty who participated turned in artifacts promptly 6. Artifacts were received for only 56% of the students for which they were requested

In the opinion of the assessment team, the artifacts indicated that the courses, in aggregate, were most successful in meeting goal 2 (Articulate how culture is expressed and influenced through areas such as ethical values, and social, economic and political systems) and least successful in meeting goal 4 ( the ability to articulated the values norms, or beliefs of the culture being studies). However for all four questions a significant majority of artifacts were rated as meeting expectations. It appears that course instructors are keeping in mind both discipline-specific objectives and the broader goals of the world cultures when designing their courses and their course assignments.

In the opinion of the assessors, not a single artifact was rated as exceeding expectations. This is in contrast to the recent assessments of both divisions III and V. In both of those cases a significant fraction of artifacts were rated as exceeding expectations. This clearly points to either a disconnect between the expectations of the assessment team and the expectations of the course instructors or a disagreement between the assessors and the designers of the rubric. It suggests that some additional training in applying the rubric might have been helpful. In addition, if a third assessor had been involved, there might have been a more appropriate definition of exceeding expectations used which might have resulted in a more meaningful distribution of scores.

Raters determined 8% of the sample artifacts evaluated showed students were approaching expectations and 70% had met expectations. The overall evaluation of students’ sample artifacts also showed that 22% were not meeting expectations. These results are similar to the pattern observed in recent assessments of divisions III and V. An appropriate goal for the World Cultures assessment would be for 100% of the students who complete the graduation requirement to meet or exceed expectations for the expected outcomes. This analysis suggests that some students may be completing the requirement without mastering the desired outcomes. However, since students selected for assessment were chosen randomly prior to the drop deadline, it is likely that some artifacts were submitted for students who in the end did not receive world cultures credit for the course they took in Winter 2011. This characteristic of the assessment procedure would explain why at least some of the artifacts that did not meet expectations. Without additional information, it is not possible to quantify the importance of this effect. Because of the nature of the assessment approached used in which data for all courses is aggregated, it is not possible to identify if particular courses or disciplines were overrepresented in the artifacts that were rated as not meeting expectations.

A serious limitation in the assessment procedure was the lack of participation by faculty. Nearly half the instructors from whom artifacts were solicited submitted no examples of student work at all. This pattern is similar to what was observed in 2009 for Division V. This means that the sample size was far below the target the committee set for a statistically useful sample size. Thus any conclusions drawn in this assessment are skewed by the low participation rate. While it is probable that the missing artifacts would show a similar pattern of meeting expectations, it is also possible that a different picture might emerge if a complete data set had been analyzed. For instance, it is possible that artifacts from the non-compliant courses might show a significantly higher rate of not meeting expectations for some or all of the rubric items or conversely the missing artifacts might have contained samples that would have been rated as exceeding expectations.

Summary

In summary, the assessment indicates that in aggregate, the Winter 2011 World Cultures courses were organized and taught in a manner designed to meet the spirit of the graduation requirement. It also indicated that most of the students completing a World Culture course were rated as meeting the objectives of the graduation requirement. No courses were specifically identified as deficient in meeting the goals of the graduation requirement. The liberal studies committee is concerned about the higher than desired rate of artifacts not meeting expectations. However without additional information, no specific recommendations can be made. It also seems clear based on both this assessment and prior efforts by the committee that a stronger enforcement mechanism needs to be designed in order to insure a high rate of faculty participation. Without better faculty participation, any assessment of student outcomes is incomplete and possibly misleading. Appendix A; NMU courses which meet World Cultures requirement

Title Faculty Name Art & Architecture Of Japan Mitsutoshi, Oba Contemporary Latin American Culture Mary A. Andronis Cultural Food Patterns Mohey A. Mowafy Human Geography Richard H. Eathorne Human Geography Teresa Bertossi Intercultural Communication Shuang Xie International Health Issues Mary J. Tremethick Intro Socio-Culture Anthropology Alex K. Ruuska Intro Socio-Culture Anthropology Alex K. Ruuska Native American Drama, Nonfiction and Short Stories Amy T. Hamilton Native American History William H. Bergmann Regional Studies: World Culture - Latin America Richard H. Eathorne The Native American Experience Shirley A. Brozzo The Native American Experience Aimee L. Cree Dunn The Native American Experience Grace Chaillier The Native American Experience Grace Chaillier Topics in World Literature: S Asian Queer Jaspal K. Singh Traditional Oral Literatures: Native American Jamie Kuehnl Web: Human Geography William A. Imperatore Web: Introduction to the History of Africa Alan S. Willis Web: Regional Studies: World Culture - Latin America Richard H. Eathorne WEB: World Literature in English: Caribbean Bronwyn Mills WEB: World Literature in English: SE Asia Jaspal K. Singh WEB: World Literature in English: SE Asia Jaspal K. Singh World History Alan S. Willis World Literature in English: African Lit. Jamie Kuehnl World Literature in English: African Lit. Bronwyn Mills World Religions Antony Aumann World Religions Antony Aumann Appendix B

How: The data consists of 2 columns; each represents the ratings Cohen’s Strength of given by one of the 2 raters. There are as many rows as there are Kappa agreement subjects, and each row contains the 2 ratings given by the two raters separated by at least 1 space. The total number of rows is entered in 0.00 Poor the first text box, and the data matrix entered in the text area. 0.01-0.20 Slight Explanation: Cohen's Kappa measures concordance between 2 raters 0.21-0.40 Fair on same individuals using an ordinal scale such as the Likert, and the columns and rows represents the scores of the two raters, and the 0.41-0.60 Moderate cells in the matrix the number of cases so scored. In this page, the data is entered raw, and the matrix is drawn up by the program, and 0.61-0.80 Substantial presents as part of the result. 0.81-1.00 Almost Example: Let say researchers are scoring pain as nil (1), little (2), perfect and lots (3). Researchers then present 10 participants to two raters, each rater is asked to assess how much pain each of these participants with have. Let say the two score the participants are as follows: 1,1; 1,1; 2,1; 1,3; 2,3; 3,3; 3,1; 3,1; 2,2; 3,3. The matrix produced would be 2,0,1; 1,1,1; 2,0,2 in the 3 rows, and the Kappa will be 0.14. What is a good agreement? (From Landis and Koch, 1977). The Kappa Statistic is a rough guide is provided, but this should not be taken too seriously.

Cohen’s Strength of Outcomes Based on Understandings Kappa Agreement R1 & R2 1. Articulate how culture (primarily of Asia, Africa, and the non- Anglo-Saxon Americas) is created and expressed through areas 0.43 Moderate such as the arts, literature, philosophy, religion, and science and technology. 2. Articulate how culture is expressed and influenced through areas such as ethical values, and social, economic and political 0.41 Moderate systems. 3. Articulate, within the global or regional context, how factors such as racial, ethnic, gender and class differences affect how 0.28 Fair groups within a culture relate to each other or to other cultures. 4. Articulate the values, norms, or beliefs of the world culture(s) 0.40 Moderate being studied.

Note: Cohen’s kappa overall = 0.391; n = 94; R1 = Rater 1; R2 = Rater 2 World Culture Outcome Assessment

Rater 1 O1 O2 O3 O4

Does not Meet 37 39% 13 14% 21 22% 30 32% Expectations Nears Meeting 4 4% 4 4% 4 4% 8 9% Expectations Meets 53 56% 77 82% 69 73% 56 60% Expectations Exceeds 0 0% 0 0% 0 0% 0 0% Expectations

Rater 2 O1 O2 O3 O4

Does not Meet 19 20% 13 14% 15 16% 22 23% Expectations Nears Meeting 9 10% 6 6% 12 13% 10 11% Expectations Meets 66 70% 75 80% 67 71% 62 66% Expectations Exceeds 0 0% 0 0% 0 0% 0 0% Expectations

1. Articulate how culture (primarily of 3. Articulate, within the global or regional Asia, Africa, and the non-Anglo-Saxon context, how factors such as racial, ethnic, Americas) is created and expressed gender and class differences affect how through areas such as the arts, literature, groups within a culture relate to each other or philosophy, religion, and science and to other cultures. technology. 4. Articulate the values, norms, or beliefs 2. Articulate how culture is expressed of the world culture(s) being studied. and influenced through areas such as ethical values, and social, economic and political systems.

Does not Meet 170 23% Expectations Nears Meeting 57 8% Expectations Meets 525 70% Expectations Exceeds 0 0% Expectations

Appendix C Definition: Students at NMU must take at least one course of three credits or more designated to meet the world cultures requirement. These courses deal primarily with living cultures of Asia, Africa and the non-Anglo-Saxon Americas. They may include such topics as literary, artistic and other cultural achievements; religious and ethical values; social, economic and political systems; or intellectual and historical trends. At least two-thirds of the content of each course concentrates on patterns of thought and action that account for the distinctive ethos of the world culture(s) being studied. Transfer students seeking approval of a substitute course to meet this requirement should contact the Registrar’s Office and provide a course syllabus for the course they wish reviewed.

Division III Learning Assessment rubric and Liberal Studies rubric

Reviewer: Indicate your level of agreement regarding the demonstration of the following components per the learning outcome artifacts reviewed as related to World Cultures.

Does not meet Nears Meets Exceeds Expectations Expectations Expectations Expectations Articulate how culture (primarily of Asia, Africa and the non-Anglo-Saxon Americas) is created and expressed through areas such as the arts, literature, philosophy, religion, and science and technology.

Articulate how culture is expressed and influenced through areas such as ethical values, and social, economic and political systems.

Articulate, within the global or regional context, how factors such as racial, ethnic, gender and class differences affect how groups within a culture relate to each other or to other cultures.

Articulate the values, norms, or beliefs of the world culture(s) being studied.

*At least 2/3’s of the course content concentrates on the patterns of thought and action that account for the distinctive ethos of the world culture(s) being studied.