TO: Peter Whiting, Director of SAGES Kimberly Emmons, Director of CWRU Writing Programs Michael Householder, Associate Director of SAGES

CC: The 2015 Portfolio Assessment Committee: Eric Chilton (English), Georgia Cowart (Music), Colin Drummond (Engineering), Tina Howe (Religious Studies), Jane Marek (Nursing), Todd Oakley (Cognitive Science), Vasu Ramanujam (Weatherhead), Jeffrey Ullom (Theatre), Timothy Wutrich (Classics)

FROM: Megan Swihart Jewell (English) and Erika Olbricht (English), Committee Coordinators

SUBJECT: SAGES Writing Portfolio Assessment Committee Report

DATE: 8 July 2015

EXECUTIVE SUMMARY

The 2015 SAGES Writing Portfolio Assessment Committee, consisting of 11 faculty members from across the university, read and evaluated 854 student portfolios submitted to the SAGES office between May 2014 and May 2015.

The committee’s overall holistic assessment found 68% of students’ portfolios to be Proficient or Acceptable. The committee found that 22% of students were writing at the Proficient, or highest, level. Reaching this level is a considerable accomplishment, given this year’s assessment criteria. The committee observed that students are exposed to a breadth of topics in their First and University Seminars and, in their writing, they have demonstrated the value of the SAGES program and its ability to make them aware of these important subjects. In the holistic scoring section 71% of students scored Proficient or Acceptable in terms of their engagement, a measure indicating an awareness of audience as well as the successful presentation of a problem or question.

Nevertheless, while students explore important topics, they continue to have difficulty making clear and persuasive arguments about them. Therefore, the committee’s major recommendation is that SAGES and the Writing Programs should continue their emphasis on argumentation, giving particular attention to: articulating a clear thesis statement, developing the argument throughout the paper, and using secondary source materials persuasively as evidence. Points of concern to the committee are as follows:

• The number of non-argument driven academic research papers (18%) has increased and seems too high for a program emphasizing argumentation. • A large number (38%, Table 1) of arguments in the researched argument essays were assessed as Developing or Unacceptable in terms of their Reasoning/Development abilities. Only 26% demonstrated proficiency in developing arguments. • Nearly half (44%, Table 1) of thesis statements in the researched argument essays were assessed as Developing or Unacceptable due to the fact that they failed to stake a strong claim or they were not articulated as arguments. • In the categories of both quality and use of evidence in the researched argument essay, the committee found that 27% (Table 1) of essays were Developing or Unacceptable. The committee identified students’ weakness in using evidence as tied to their inexperience with the various roles sources can play to support and strengthen their arguments as well as lack of critical discernment about the quality and types of sources they use in their essays.

Overall, the committee felt that the problems students displayed in using evidence is directly linked to the difficulties they had with articulating and sustaining a clear argument. In other words, students are not perceiving their own arguments with enough complexity to merit a more sophisticated use of sources. Therefore, the committee recommends a stronger programmatic emphasis on the genre and purposes of the researched argument essay, its function in an academic community, and its relationship to the SAGES Learning Outcomes.

HISTORY

Faculty evaluation of student writing portfolios during the SAGES pilot and first years of implementation (2002-2008) focused solely on assessing individual portfolios in order to provide feedback to students who submitted them as a graduation requirement. In June 2009, the portfolio review process was modified to provide programmatic feedback on student writing to SAGES and English Department Writing Program administrators. From 2009-2014 a consistent but evolving process was used to assess portfolios. In that time period, the committee increasingly stressed argument writing rather than report-based writing; subsequent years show more argument writing identified in portfolios. For example, in 2009, readers found that only 66% of researched essays contained arguments; in 2014, that number had risen to 85%. See previous years’ reports for additional details (case.edu/writing).

In 2014-15, SAGES adopted a new set of Student Learning Outcomes (Appendix 1). Therefore, in order to more closely align with the new outcomes, the writing portfolio assessment categories and rubrics were modified significantly (Appendix 2). In response to previous years’ committee concerns about weaknesses in the researched essay, this year the reading process began with a detailed assessment of the researched argument essay from selected students’ portfolios. This essay is a common assignment across university seminars in SAGES, and it is the only document that has a specified genre in the portfolio submission guidelines. The researched argument essay represents the culmination of a student’s work in the first three SAGES seminars. It is the essay included in the portfolio that best tests the SAGES Outcomes; four out of five of the major outcomes should be apparent in it.

The changes to the portfolio reading process this year signify a break in continuity with previous years’ results. This year’s data establish a new descriptive benchmark for the assessment of student writing that will serve as the basis for subsequent committee evaluation.

2015 METHODOLOGY

The reading process this year was divided into two parts. First, each reader assessed the research essay from approximately 25 randomly chosen portfolios. Then, each portfolio was read holistically. (See Appendix 2 for rubrics and instructions to readers.) Committee members met for three full days, based on the schedule developed by the Committee Coordinators.

The first committee meeting day was spent discussing the researched argument essay rubric and sample research papers in order to calibrate reader assessment of the researched argument essays. Readers had five days to read the researched argument essays and enter data into a Qualtrics survey form.

The second meeting day (a week later) was split between discussing initial impressions of the data gathered from the researched argument essay assessments and discussing the holistic reading process, the anchor portfolios (those portfolios designated to define each performance category), and three calibration portfolios. Readers had seven days to read portfolios (an average of 79 per reader) and enter data into a Qualtrics survey form.

The third meeting day was used to discuss the holistic data generated and the review process itself, and to draft the committee’s conclusions and recommendations for classroom instruction and program implementation.

The coordinators drafted this report based on notes from the discussion and reader questionnaires; committee members were asked to offer feedback on the report before it was made public.

RESULTS

Results of the researched argument essay evaluation and holistic assessment are included below. This year’s four performance categories are more closely aligned with those in the SAGES Learning Outcomes, though their designations are more appropriate to portfolio review: Proficient, Acceptable, Developing, and Unacceptable (Appendix 1).

In their assessments, readers were instructed to rank the researched argument essay or the portfolio in the next lower category if it did not fully meet the criteria of the higher category.

Researched Argument Essay Evaluation Results

In their portfolios, students must submit a researched essay from one of their university seminars. The online portfolio submission guidelines explicitly define the genre of this essay: By “research essay,” SAGES means a sustained engagement with an academic conversation - summarizing and critiquing what others have said on your topic - that includes your own novel claim or argument. This paper must do more than catalog the research you have done (i.e., an annotated bibliography is not sufficient); it should demonstrate your ability to synthesize academic research on a specific topic and to offer your own analysis or critical intervention. Such a paper will have a single controlling idea that represents your own thinking about the topic ….The research paper should be approximately 10-12 pages long …. The research essay must integrate and cite primary and/or secondary source material and include a properly formatted bibliography. (sages.case.edu)

Each reader assessed the researched argument essay from approximately 25 of their assigned portfolios, for a total of 271 essays, or 31.7% of portfolios. Readers acknowledged that the researched essays as a whole showed a wide diversity of topics that allowed many students to undertake thoughtful, critical, and creative research projects that explore meaningful questions and problems.

Nevertheless, the evaluative ratings continue to indicate that the quality of argumentation needs significant improvement. The three weakest criteria in the overall assessment of the researched argument essay were: establishing the problem/question guiding the research inquiry, the thesis statement, and reasoning/development. In each case, over a third of papers were in the Developing or Unacceptable categories (35%, 44%, and 38%, respectively; Table 1). Readers acknowledged, though, that these are the most challenging elements in researched writing.

The categories Style and Correctness had the highest scores, at 76% and 84% respectively in the Proficient or Acceptable categories (Table 1).

Thesis Statements Readers commented especially on the data set assessing thesis statements. Only 28% of papers were evaluated as Proficient, having a “clear, precisely stated, argumentative thesis that is insightful, compelling, and appropriate in scope.” Thirty-one percent of papers had Developing thesis statements, meaning that theses were either implicit, only articulated part of the argument, lacked insight, or had an inappropriate scope. A small, but concerning, percentage (13%) had Unacceptable thesis statements: either no argumentative thesis or controlling idea, or the thesis did not address the problem/question being engaged. Because 44% of thesis statements fell in the Developing or Unacceptable categories, readers urge SAGES and the Writing Programs to continue stressing the importance of a clearly articulated, insightful, and debatable central claim for researched writing.

Genre This committee has always been concerned with the genre of the researched essay, preferring to see research presented in the form of an academic argument, which encourages stronger critical thinking and persuasion than a report or other genre. Eighty-one percent of research papers were determined to be Argument essays, down slightly from 85% last year. The number of those classified as exposition/report has been rising, from 12% in 2013, to 14% last year and 16% this year (Table 2). Without exception, readers expressed a preference for argument-driven researched essays.

Types and Use of Evidence The most frequently documented type of source was the academic/scholarly article; 67% of papers had at least one on their works cited page (Table 3). Academic books (52%) were also frequently mentioned, but so were popular journals/newspapers and popular websites, at 43% each. Readers registered disappointment that there were not more scholarly and peer-reviewed sources in students’ papers and that students gave such materials the same weight as non- scholarly, popular web-based resources. Readers overall were concerned with students’ use of many non-scholarly sources, over-reliance on one or two sources, and use of sources for factual evidence or superficial support rather than for setting up a more complex position.

While 84% of papers had at least six sources (Table 6), and 44% of papers listed eleven or more sources on their works cited or bibliography pages, readers expressed a desire for instructors to stress the quality of evidence over the quantity. In addition, a quarter of research essays rated either Developing or Unacceptable in the “use” of those sources (Table 1). Readers posited that fewer sources used more rigorously would yield better researched papers because students would be able to engage in those sources more meaningfully, using them to build their argument rather than to provide facts only. Readers were sometimes given the impression that students included sources merely to fulfill a requirement rather than to genuinely build on them in their argument.

Sixty-four percent of researched argument essays were assessed in either the Proficient or Acceptable categories for the use of evidence. However, this data point seemed at odds with reader perceptions when they were asked to comment specifically on evidence use in the questionnaires they completed after reviewing the data. For example, readers frequently commented that the researched essays reveal student uncertainty (or lack of awareness) about how to use evidence to build an argument rather than only as a source of factual information. When readers described how students were using evidence in their papers (Table 4), they found that the overwhelming majority of papers used the evidence to support the claims of the writer and to provide factual information (82% each). Readers were particularly surprised to see that sources were rarely used to introduce alternative viewpoints (23%).

Citation Style As a way of gathering information about the more mechanical aspects of research writing, readers were asked to note the citation style used and how accurately and consistently the student used it. MLA, Chicago, and APA were the top three styles noted, at 52%, 17% and 15% respectively (Table 7).

Overall, readers were disappointed by citation practices, noting that students should be able to use a style correctly and consistently in order to demonstrate that they understand the conventions of research writing. But more importantly, student writers should understand that acknowledging sources is one way of signaling the academic conversation they are joining.

While 29% of researched essays used the citation format “very correctly and consistently” (Table 8), 29% of the essays used the style either poorly or very poorly, or the reader was unable to tell. The remaining 42% of essays used the style “mostly correctly and consistently.”

Holistic Assessment

Nearly all (95%) portfolios were standard (i.e., did not contain substitutions from non-SAGES classes). For the holistic evaluations, readers were asked to assess the portfolios as a whole, including the three papers as well as the reflective essay. Readers considered the overall effectiveness, impact, and quality of the combined whole as opposed to taking a simple average or sum of the individual parts.

Reflective Essay Readers were asked to answer a series of descriptive questions regarding the Reflective Essay in the holistic assessment of portfolios. Readers answered a prompt regarding students’ perceptions of writing development in SAGES and then identified those aspects of writing in which they cited improvement. Faculty readers also indicated their own perception of students’ writing development in the holistic scoring section. Our major conclusion regarding the reflective essay is that most students do not seem to possess the vocabulary to discuss writing in terms of the “higher-order” skills of critical thinking or argumentation. This finding reinforced to the committee the need for further instruction in those areas.

In the reflective essays, the clear majority of students attributed improvement in writing to their SAGES First and University Seminars, and the clear majority of faculty readers saw some improvement in students’ writing across the portfolios. The majority of students (92%) indicated they had strong or some improvement in their writing, combined, while 84% of faculty readers indicated strong or some improvement across students’ essays (Tables 9 and 10). More students (39%) saw “strong improvement” in writing skills than did faculty readers (17%). This discrepancy shows a potential misalignment in conceptions of writing; i.e., the failure of students to conceive of writing as argumentative or critical.

While readers were pleased with the large number of students (55%) citing an improvement in argument and in use of evidence (46%; Table 11), they were concerned about the high number of students who discussed the more mechanical or “lower-order” skills of citing sources, and sentence-level editing or fluency concerns. When students explicitly mentioned an improvement in argument, they rarely provided appropriate examples or evidence to support their claims. Readers were additionally concerned about the low-percentage (17%) of students who cited revision practices as an area of improvement given that SAGES pedagogy emphasizes the production of multiple drafts.

Finally, readers noted that several students reflected on the sequence of the SAGES classes in productive ways. Students often expressed how they had much apprehension regarding the usefulness of their SAGES classes prior to taking them, and that only in reflection could they realize their full benefit. Many students were grateful for the exposure to different disciplinary topics, with some realizing how some writing in their First Seminar contributed to assignments in University and Departmental Seminars. The committee was also impressed by positive reflections written by students in more technical and professional fields, such as Engineering and Nursing, and who cited the usefulness of SAGES writing instruction to their fields.

Overall Holistic Evaluation Results

Readers evaluated each portfolio according to the revised holistic rubric (Appendix 2). The holistic scores represent readers’ assessments of the entire portfolio, including the Reflective Essay, First Seminar essay, the University Seminar Essay, and the Researched Argument Essay. The holistic scoring results indicate that the solid majority of students are writing at a Proficient (22%) or Acceptable (46%) level (Table 12). Lower numbers in the specific categories of Engagement, Argument, and Evidence continue to reflect readers’ concerns in the Researched Argument Essay regarding students’ abilities in academic writing.

While the aggregate numbers on the holistic rating suggest strong positive programmatic influence on student writing, a number of portfolios (28%) had an overall assessment of Developing. This result is directly correlated to the three categories of Engagement, Argument, and Evidence and supports the view that these skills should be more strongly emphasized.

The majority of students (78%) are Proficient or Acceptable in terms of Readability. While Readability concerned arrangement, correctness and style, several readers remarked giving students a lower score on Readability due to the casual or colloquial tone in their essays.

ESL Holistic Scoring

This year, readers evaluated 69 portfolios from students enrolled in a First Seminar for non- native speakers of English. As is the case with those of native speakers, ESL portfolios show a range of abilities. Similar to results for all students, Argument and Evidence are rated the lowest of the categories, with only 47% and 49% of ESL students, respectively, in the Proficient or Acceptable categories (Table 13), compared to 51% and 63% of native speakers.

More ESL portfolios (16; 23%) were rated Proficient than Unacceptable (6; 9%). Because of the strong ESL curriculum SAGES currently implements, as well as increasing TOEFL admissions standards, readers expect to see this benchmark exceeded in future years.

RECOMMENDATIONS

Recommendations for writing instruction primary focus on strengthening instruction in argumentative writing with a specific focus on articulating a clear thesis, developing the argument through the paper, and using secondary source materials persuasively as evidence.

Recommendations for Seminar Leaders

• Instructors should include the SAGES Learning Outcomes on their syllabi and emphasize them more in class to students. Instructors should particularly emphasize their direct correlation to researched argument writing. • Instructors should spend more time on instructing students how to write appropriate thesis statements. The committee recommended providing students with more thesis statement models in order to make the expectations more explicit. • More instructors should consider using sequenced writing assignments leading to the researched argument; for example, annotated bibliographies promote fuller understanding of—and more sophisticated engagement with—secondary sources. • As noted by last year’s committee, more instructors should employ the Writing Program’s recommended writing texts for First and University Seminars respectively, They Say/I Say (Graff and Birkenstein) and The Craft of Research (Booth, Colomb, and Williams) to promote persuasive rhetorical moves and their relationship to the effective use of evidence. • Instructors might want to consider bringing example student research papers into the seminar in order to illustrate proficient critical writing skills in researched argumentation. • In collaboratively-taught seminars, both faculty should emphasize the importance of writing in their fields and work more closely together to teach argumentative writing. Lead instructors should not leave writing instruction solely up to the writing instructor and writing instructors should take a larger role in assisting them with strengthening their writing pedagogy.

Recommendations to SAGES Administrators

• In order to promote timely submission, SAGES and the Writing Programs should sponsor an end-of-term workshop on portfolio compilation whereby students compile their essays, draft their reflection, and submit their portfolios. • SAGES should continue to investigate establishing a similar assessment process for Departmental Seminars and University Capstone courses. • SAGES should continue to encourage more faculty from across campus to participate in the portfolio evaluation process in order to cultivate additional faculty investment in writing instruction.

TABLES

Table 1. Overall Assessment of Researched Argument Essays

Proficient Acceptable Developing Unacceptable (3) (2) (1) (0)

Engagement

Problem/Question 37% 29% 27% 8%

Audience/Ethos 34% 38% 23% 5%

Argument

Thesis Statement 26% 30% 31% 13%

Reasoning/Development 26% 36% 29% 9%

Evidence

Quality 34% 39% 23% 4%

Use 27% 37% 23% 4%

Readability

Arrangement/Organization 32% 38% 28% 2%

Style 29% 47% 21% 2%

Correctness 34% 50% 15% 2%

Table 2. Researched Argument Essay Genre

Argument/Analysis 81%

Exposition/Report 16%

Personal Narrative 1%

Other 1%

Table 3. Types of sources used in the Researched Argument Essay

What Types of Sources are used in the Research Essay (check all that apply):

Academic monograph or chapter from academic collection 52%

Popular book 31%

Academic/scholarly journal 67%

Popular periodical or newspaper 43%

Government document/source (print or online) 22%

Academic website/online source 15%

Organizational website/online source 31%

Popular website/online source 43%

Personal anecdote/narrative 4%

Interview data collected by student 4%

Observational data collected by student 3%

Reference source (e.g., dictionary, encyclopedia, Facts on File, etc.) 18%

Other: 10%

Table 4. Use of Sources in the Researched Argument Essay

How were sources used (check all that apply):

To provide context or purpose (i.e., to outline the problem/question being discussed) 58%

To supply a definition, theoretical concept, or argument (i.e., that the writer then applied to another 41% object of study)

To support the claims of the writer (i.e., as evidence of similar/complementary arguments or 82% points)

To provide factual information (i.e., examples, statistics, definitions, etc.) 82%

To introduce alternative viewpoints (i.e., other perspectives or counter-arguments) 23%

As objects of analysis themselves 16%

As “filler” or without clear purpose 5%

Other 1%

Table 5. Number of sources cited or paraphrased in the body of the paper Research Essay

How many sources does the Research Essay cite or paraphrase in the body of the paper?

0-1 0%

2-5 14%

6-10 45%

11 or more 40%

Table 6. Number of sources listed on the Works Cited page / Bibliography

How many sources does the Research Essay list in the Works Cited/Bibliography?

0-1 0%

2-5 11%

6-10 40%

11 or more 44%

Works Cited page / Bibliography missing 5%

Table 7. Citation Style used

What citation style does the Research Essay use?

Modern Language Association (MLA) 52%

American Psychological Association (APA) 15%

Chicago Manual of Style (CMS) 17%

American Medical Association (AMA) 3%

Cannot Tell 7%

Other 5%

Table 8. Correct and Consistent use of Citation Style

How correctly and consistently does the author use the citation style you identified?

Very correctly & consistently 29%

Mostly correctly & consistently (minor errors) 42%

Poorly (a few citations missing and/or many errors) 18%

Very poorly (many errors & citations missing or impossible to decipher) 5%

Cannot Tell 6%

Table 9. Student Perception of Development

Strong Some No change Regression in improvement in improvement in in writing writing skills writing skills writing skills skills

Student perception of 39% 53% 7% 1% development in his/her writing skills across SAGES

Table 10. Reader Perception of Student Development

Strong Some No change Regression in improvement in improvement in in writing writing skills writing skills writing skills skills

Reader perception of 17% 67% 14% 2% development in student’s writing skills across SAGES

Table 11. Aspects of writing discussed in reflective essay

Which aspects of writing did the student discuss in his/her reflective essay? (check all that apply)

Identifying questions/problems that motivate writing 19%

Developing/refining argument (thesis, controlling idea, etc.) 55%

Critical thinking/ethical deliberation (incorporating multiple points of view, attending to the 36% implications of one’s own & others’ arguments, etc.)

Use of evidence/sources in writing 46%

Evaluation of evidence/sources (quality, reliability, etc.) 18%

Revision practices (rethinking/refocusing arguments, incorporating additional points of view, 17% responding to additional materials/research)

Writing process (drafting, outlining, rewriting, etc.) 28%

Responding to feedback (of peers and/or instructors) 21%

Organization/arrangement of sentences, paragraphs, etc. 39%

Editing for sentence-level fluency and correctness (word choice, sentence structure/grammar, 44% passive voice, etc.)

Other 18%

Table 12. Results of Holistic Assessment for all portfolios

Proficient Acceptable Developing Unacceptable (3) (2) (1) (0)

Engagement 26% 45% 24% 4%

Argument 20% 37% 36% 7%

Evidence 21% 42% 30% 7%

Readability 32% 46% 19% 3%

Overall Assessment 22% 46% 28% 3%

Table 13. Results of Holistic Assessment for ESL portfolios

Proficient Acceptable Developing Unacceptable (3) (2) (1) (0)

Engagement 25% 33% 35% 7%

Argument 19% 28% 46% 7%

Evidence 19% 30% 39% 12%

Readability 25% 28% 32% 16%

Overall Assessment 23% 22% 46% 9%