EWP Readings Report Fall 2005

Electronic Writing Portfolio Readings Report Fall 2010

Number of Readers: 23 Number of Portfolios: 230 Number of Papers: 642

Method: In Fall 2005, volunteers were solicited from the faculty and staff. Readers were required to be full-time at Eastern, to have completed a master’s degree or higher, and to teach at least one undergraduate course each year. Twenty-three readers were trained at a 1 ½-day workshop by Dr. Robert O’Brien Hokanson, Associate Professor of English and Coordinator of the Communication Ability Department at Alverno College. Readers came from all four colleges, the School of Continuing Education, the Center for Academic Support and Achievement, and Booth Library.

A second training was held in October 2008 and a third in October 2010; Dr. Karla Sanders delivered the training based on the work completed by Dr. O’Brien Hokanson. The following 2010 readers were trained at one of these sessions; they represent all four colleges and a few other administrative units:

Ronan Bernas, Psychology Kaninika Bhatnagar, School of Technology Jay Bickford, Early Childhood, Elementary, and Middle Level Education Rigoberto Chinchilla, School of Technology Tim Croy, Early Childhood, Elementary, and Middle Level Education Jonelle DePetro, Philosophy Bill Feltt, English Rose Gong, Secondary Education Angie Jacobs, Communication Studies Amit Joshi, Physics Karla Kennedy-Hagan, Family and Consumer Sciences Wendy Long, Center for Academic Support and Assessment Jeannie Ludlow, English Amber Lusvardi, Political Science James Ochwa-Echel, African American Studies Dave Pollard, Alumni Services Betsy Pudliner, Family & Consumer Sciences David Richardson, Art Jeff Snell, Management Gordon Tucker, Biological Sciences Traci Worby, Kinesiology & Sports Studies Diana Wyatt, School of Technology Zhiqing Yan, Chemistry

Readers were urged to look at writing patterns across the portfolios rather than focusing on each individual document submitted to the portfolios. The reading guide asks readers to provide an assessment of writing ability for complete portfolios across seven aspects of writing: focus/purpose, organization, development, audience, style, mechanics, and use of sources. Readers were also asked to assess each portfolio overall.

EWP Portfolio Readings Report, FA2010, p. 1 A sample of 10% of the completed portfolios are read each year; readers were given a month to read their portfolios, were given access to a secure web site and were assigned to read 10 portfolios that were chosen at random from the completed portfolios. After reading their set of portfolios, readers were asked to complete a reader’s observation sheet and to attend one of three focus groups held in November to discuss student writing as displayed in the portfolios.

The qualitative data that follow represent the discussions at the focus groups as well as information taken from the readers’ guides and observation sheets; all information has been collated and summarized by the Director of the Center for Academic Support and Assessment. A draft of this report was disseminated via email for readers’ comments. Changes to the report were made based on those comments. The percentages given for each assessment of the various areas of writing are taken from a compilation of scores given by the readers for each portfolio, not each document.

The following data are divided into the categories assessed by the readers. Each section gives the readers’ impressions of the portfolios as a whole and the final section offers potential uses for the data in terms of improving the curriculum/pedagogy. Where appropriate, readers’ written and verbal comments have been quoted to support the general conclusions that have been drawn. Percentages refer to the percentage of portfolios that were rated in the categories described.

Portfolios Overall

Strong Portfolios: 28%

Adequate Portfolios: 54%

Weak Portfolios: 17%

This year showed a marked improvement in the percentage of strong portfolios compared to the 2009 readings with a 6% increase in strong portfolios and a 3% decline in weak portfolios.

Several readers speculated that students’ use of instant messaging and texting makes them think of communication as instantaneous, and time spent on revision and proofreading seems to be suffering because of the very tools that make is easier than it was 20 or 30 years ago. “Not everyone likes to write because to do something really well takes work. It takes a lot of effort. I think some of the conveniences of the technical world take away from the work.”

Because of the close connection between critical thinking and writing, readers were asked whether they could draw any conclusions about critical thinking skills from the portfolios they read. Readers agreed that such conclusions were dependent on the assignment. Many assignments did not appear to be asking for student thought or opinion, but were recountings of information or explanations. “Students are not putting their thoughts into the papers very strongly. Even the research papers start to read like a serial summary. Here is what one person says, and another person says, without a lot of synthesis.”

Strong critical thinking as evidenced by EWP portfolios was the exception rather than the rule in the sampling read. “When you are looking at a student’s critical thinking, you see an awful lot of summary and regurgitation of content—definitely the bottom two levels [of Bloom’s taxonomy]. I also see a lot of analysis and synthesis. But, if you look at Anderson and Crackwell’s re- analysis of Bloom’s taxonomy with the highest levels being original thought, I think there is very, very little of the highest levels of critical thinking.” Another reader agreed, and added, “Many documents are characterized by a lower-order of thinking. For example, reviews that merely

EWP Portfolio Readings Report, FA2010, p. 2 describe or summarize articles and movies areoften in portfolios. A small percentage of documents demonstrate the writer’s ability to bring the audience to a higher order of thinking by such requirements as hypothesis or question, analysis where comparison, contrast, illustration, and inference are used; and evaluation which requires a set of standards to be applied.”

Once again, development was singled out as the biggest issue among the writing skills assessed in the portfolio review. “What seems to be missing in many of the them is the creation of an overall conceptual structure (themes, subthemes), and then methodically working through that structure. Variation of sentence structure would also help to keep our interest.”

The chart below compares the readers’ assessment of the portfolios read across the last five years of readings. The percentage of portfolios judged to be strong overall increased by 6% from 2009 to 2010 marking the first year in four to see an increase. Portfolios deemed adequate have dropped correspondingly 4% and those assessed as weak dropped by 3%. The ratings for the portfolios overall have fluctuated the least of any of the categories scored by the readers over the entire time readers have been evaluating portfolios.

Readers' Assessment of Portfolios--Overall 80% 55% 58% 58% 58% 54% 60%

40% 28% 27% 24% 28% 22% 17% 18% 19% 20% 17% 20% 0% Strong Adequate Weak FA06 FA07 FA08 FA09 FA10

The sections that follow discuss the strengths and weaknesses readers found in portfolios by writing component.

Focus/Purpose

Strong: 31% Consistently strong sense of focus/purpose throughout

Adequate: 40% Clear focus/purpose in most or all submissions

Weak: 26% Some evidence of ability to focus on a purpose

Poor: 2% Very little or no evidence of focus

Overall, the skill of establishing a purpose for writing and staying focused on that purpose was a strength although this year’s numbers indicate a more even distribution than in past years with the percentage falling into the strong category increasing 4% and the weak category increasing 7% over last year’s numbers.

Readers noted that papers without strong thesis statements or that did not employ topic sentences were overall weaker than those that were well executed in this area. “I felt in the ones I read

EWP Portfolio Readings Report, FA2010, p. 3 there seemed to be a lack of some clear topic sentences, a real statement of purpose. Introductory paragraphs were all over the paper.” Most readers agreed that they would like to see students spend more time devloping a well-thought out thesis statement to establish purpose early on in the semester. They also noted that such statements were more likely to occur in papers from English and History courses.

While many readers felt that this area was one of the stronger ones, some noted isses with students meandering through a topic rather than staying focused on one major idea. Some readers suggested that assignments could help students in determing focus and conveying that to the reader. “I think also some papers had a clearer sense of focus, and they knew what to write because of the assignment,” explained one reader.1

Readers' Assessment of Portfolios--Focus 53% 60% 52% 48%51% 40% 31% 40% 28% 28% 27% 26% 23% 18%24%22% 19% 20% 1% 1% 2% 3% 2% 0% Strong Adequate Weak Poor

FA06 FA07 FA08 FA09 FA10

Organization

Strong: 19% Consistent use of structure in ways that enhance presentation of ideas/information

Adequate: 58% Logical organization and/or clearly identifiable structure

Weak: 18% Inconsistent sense of structure and/or lapses in organization

Poor: 2% Very little or no sense of structure or organization

Students’ abilities to organize their thoughts and ideas were some of the weaker elements in this year’s completed portfolios. The percentage of strong portfolios fell by 4% from the previous year for a six-year low at 19% (the highest percentage was 25% in the first year of portfolio readings).

Readers described issues such as non-existent transitions, lack of headings, and other markers to move readers from one idea to another. “Transitions don’t exist. I think they are out of fashion like ‘80s hair.” Several readers noted use of repetition to take the place of transitions. “They had longer papers, and there was nothing to guide me. They kept repeating the last thought. Out and out repetition, ‘as I said in the last paragraph.’”

Other issues described by readers included a lack of strong conclusions and a sense of paragraphing. “Some paragraphs ran on and on. A beginning, middle, and end somehow don’t

1 Starting in FA08, students are asked to describe their assignment, so readers now have access to assignment descriptions for many submitted papers.

EWP Portfolio Readings Report, FA2010, p. 4 translate. They just plunged into it. There is no body and conclusion or explanation.” A second reader concurred and added, “I felt like even if they had an intro, they jumped into the middle of a conversation.” There was general agreement about these organizational issues, and students seemingly unclear about appropriate order of topics when building an argument or displaying knowledge about a topic. Readers felt like student writers often just sat down and started writing without an outline or sense of building toward a conclusion.

Readers' Assessment of Portfolios--Organization 80% 61% 60% 58% 57% 56%58% 40% 22% 23% 20% 20% 19% 21%15%20%19% 18% 20% 1% 1% 2% 2% 2% 0% Strong Adequate Weak Poor FA06 FA07 FA08 FA09 FA10

Development

Strong: 22% Ideas consistently developed in depth and supported with rich and relevant details

Adequate: 43% Ideas developed in depth with appropriate supporting evidence/details

Weak: 31% Some development of ideas and use of supporting evidence/details

Poor: 2% Very little or no development of ideas or use of supporting evidence

While the percentage of portfolios falling into the strong category did rise 3% from the previous year, readers indicated that development was one of the weakest skills among the portfolios, and they linked a lack of development to poor critical thinking skills. One reader suggested that faculty could help students by “enabling you to critically evaluate an ambiguous situation using the best available evidence. . . I would say it is more intellectually immature as opposed to apathy or benevolence. I wouldn’t say it’s not because they don’t want to. I would have to say it’s because they haven’t been challenged enough in difference ways.”

Another reader concurred and added, “Absolutely. . . .And, if your argument isn’t logical, it isn’t going to be clear, and if you want your opinion to be clear, then you are going to have to go out of your way to think like the listener, the reader. The more you think like the reader, the better writer you will be.”

Several readers suggested students were hesitant to take a stand or proffer an opinion and this indecision is evident in their writing. This was especially a problem in research papers where students tended to fill their papers with quotes without making an argument or taking a stand between citations. “A lot of quotes and no original ideas. There was nothing in between. That was the biggest problem in terms of development.” Long-time readers suggested that many of their papers were longer than in previous years, and the length seemed helpful in development and details.

EWP Portfolio Readings Report, FA2010, p. 5 Readers' Assessment of Portfolios--Development 60% 48% 47% 41% 43% 41% 36% 40% 30%33% 30%31% 22% 21%23% 18% 19% 20% 3% 3% 5% 3% 2% 0% Strong Adequate Weak Poor

FA06 FA07 FA08 FA09 FA10

Audience

Strong: 25% Sophisticated sense of audience—e.g., distinctive voice and/or appropriate tone

Adequate: 56% Some awareness of and/or attempt to communicate with audience

Weak: 13% Little or no awareness of audience

Poor: 5% No sense of writing for an audience

With each reading, faculty have indicated that it is often difficult to discern any reader other than the instructor for the course unless the genre of the paper demanded that an audience be identified, such as in a letter. The percentages of portfolios from the various categories remained consistent from 2009 to 2010. Audience awareness is overwhemlingly an adequate skill with most students writing for their faculty members. One reader noted, “I usuallt prescribe the audience, but unless you do that, they write to the instructor.”

Issues in the portfolios related to audience awareness involved use of terminology without definition and feeling as if one was coming into the middle of a conversation with little explanation for the uninitiated.

Readers' Assessment of Portfolios--Audience 80% 67% 59% 59% 57% 56% 60%

40% 24% 25%25% 19% 18% 17% 20% 20% 13% 13%13% 1% 4% 5% 5% 0% Strong Adequate Weak Poor FA06 FA07 FA08 FA09 FA10

Strong: 15% Sophisticated use of language (sentence structure, word choice) that enhances presentation of ideas/information

EWP Portfolio Readings Report, FA2010, p. 6 Adequate: 57% Appropriate use of language that effectively conveys ideas/information

Weak: 24% Use of language that is awkward, unnecessarily complex, and/or overly simplistic

Poor: 2% Use of language that is highly inconsistent or indeterminate

Thoughout the six years of reading completed portfolios, more portfolios are rated as weak or adequate in terms of style than strong. In fact, this is one of the lowest rated skills with portfolios on a downward trend with only 15% of portfolios displaying strong stylistic elements.

Problems with style that were prevalent include using idiomatic expressions, jumping from first person to second person to third person or choosing the wrong voice, and repetitive words or phrases. Readers noted that many portfolios displayed a lack of sophistication in word choice and sentence structure and a poor command of the language. “Even when they are trying to be sophisticated, they get hung up on the words, but I admire them for trying. I give them kudos for using the jargon in their field when they feel awkward for using that jargon, but they tried it out.” Others indicated that academic voice was rare with more portfolios showcasing colloquial language and simplistic sentence structures.

Readers' Assessment of Portfolios--Style 80% 57%57% 63% 60% 59% 57% 40% 18% 23% 22% 23% 24% 20% 18% 15% 21% 20% 13% 1% 2% 1% 2% 0% Strong Adequate Weak Poor FA06 FA07 FA08 FA09 FA10

Mechanics

Strong: 28% Few, if any, errors in mechanics relative to length and complexity of documents

Adequate: 50% Some errors in mechanics that do not interfere significantly with communication

Weak: 19% Patterns of errors in mechanics that affect clarity and/or credibility of writing

Poor: 5% Large numbers of errors in mechanics affecting almost all aspects of the writing

Holistic ratings for the portfolios showed a 4% increase in those rated as strong in mechanics over 2009. The adequate and weak categories dropped a few percentage points, but the poor category was up by 3% to 5% indicating more portfolios had severe mechanical issues. One reader described such issues, “The grammar, the spelling, the sentence structure should have been

EWP Portfolio Readings Report, FA2010, p. 7 fixed. Those things should have been taken care of.” Others noted that students rely on spellcheck as their sole mechanical revision, so they miss issues such as choosing the wrong homynyms. Readers felt that students needed to do a better job proofreading for careless mistakes. And as faculty, we need to assure that this skill is taken seriously by our students.

Readers' Assessment of Portfolios--Mechanics 60% 56%54% 49%49% 50% 50% 34% 40% 28% 29% 24% 30% 24% 22% 18%21% 19% 20% 16% 5% 10% 1% 2% 2% 0% Strong Adequate Weak Poor FA06 FA07 FA08 FA09 FA10

Sources

Strong: 34% Ability to integrate ideas/information from sources into own writing in meaningful and appropriate ways

Adequate: 55% Some effective integration of ideas/information from source

Weak: 11% Inappropriate/ineffective integration of ideas/information

Of the 230 portfolios read, 203 portfolios (88%) included at least one paper incorporating outside sources compared to 87% from the previous year. The portfolios assessed as strong grew by 3% from 2009 and the weak portfolios related to sources dropped 1%. Longtime readers of EWP completed portfolios felt that more papers were now included with sources than those completed under the old EWP requirements.

However, issues still remain that could be improved in students’ use of sources. “I tell [my students] that the reference comes from someone down the line from a university, and you have to get the original quote for it. There are different tiers to referencing; the first tier is jounal articles and the second tier is magazines like Time or Newsweek, and the third tier is wikipedia. They don’t want to take the time to research. They want me to tell them where to look.” Other issues were consistent use of format, correctly paraphrasing, or incorporating quotes smoothly into their own prose.

One reader echoed the sentiments of many, “With sources, it was either black or white. They either used them well or they did very, very poorly.”

EWP Portfolio Readings Report, FA2010, p. 8 Readers' Assessment of Portfolios--Sources 55% 52% 56% 55% 60% 50% 50% 34% 40% 31% 25% 25% 30% 20% 21% 20% 11% 13% 12% 11% 10% 0% Strong Adequate Weak FA06 FA07 FA08 FA09 FA10

Using Data to Develop Curriculum & Improve Pedagogy

After reading all of their portfolios, readers were asked to indicate ways in which the information gleaned from the readings could be used to develop curriculum and/or improve pedagogy. Many readers are quoted in this section to preserve the intent of their suggestions. The predominant theme involved encouraging faculty to make changes in writing requirements/assignments and course curriculum through sharing EWP data from the readers.

One such suggestion was an increased focus on revision, “From a curriculum perspective I think developing student writing in a way that students are constantly revising their work would help improve writing. It often appeared that papers were closer to being a first draft than a final draft. Or, were never proofread at all. If teachers structured classes in a way that would allow for more revision, students might revisit some of their mistakes. . . Across the departments, faculty should be encouraging students to sharpen their language skills.”

One reader suggested that creativity and passion for ideas, for a subject, were largely absent across the portfolios. She suggested, “maybe we need to encourage students to write about what is meaningful to them. For the most part, the papers were adequate. But they aren’t inspired; they aren’t passionate. I don’t see new ideas; I don’t see creativity.”

One reader suggested using the reader’s guide employed by the portfolio readers for faculty use with students. “I think the guidelines sheet would be a benefit for all faculty on campus. It acknowledges the aspects of writing that as faculty we could concentrate on grading our students’ papers with these same guidelines.”

Another reader offered similar ideas, “If the problem is inconsistency, then we might work to provide writing objectives for the campus as a whole and work to help instructors incorporate these into their syllabi in more meaningful ways. If the problem is the higher-level writing (the abstract elements of focus, style, and audience), then we might work to provide studnts with a clearer sense of these by asking them to analyze them in their reading assignments as well as their writing assignments.” Another reader offered a similar idea of faculty assigning good models of writing for students to read, “Deficiencies in organization, development, style, and mechanics are best remedied by exposing students to good models of writing. This can be achieved by assigning more readings in the curriculum. Likewise, more practice in expressing personal thoughts and opinions is necessary. Students seem to be capable of conducting library or archival research and repoting such, but have difficulty forming, integrating, and expressing personal thoughts.”

Readers also suggested making writing workshops available to students to improve the various skills identified as weak. All readers indicated that results from the EWP readings be widely circulated among the faculty.

EWP Portfolio Readings Report, FA2010, p. 9