English Language Writing Assessment: Teacher Practices in Thai Universities

Home , Emoticon

English language writing assessment: Teacher practices in Thai universities

Somruedee Khongput

Thesis submitted in fulfilment of the requirements for the degree of Doctor of Philosophy

School of Education Faculty of Arts and Social Sciences University of New South Wales

June 2014

PLEASE TYPE THE UNIVERSITY OF NEW SOUTH WALES Thesis/Dissertation Sheet

Surname or Family name: KHONGPUT First name: SOMRU EDEE Other name/s: -

Abbreviation for degree as given in the University calendar: PhD

School: Education Faculty: Arts and Social Sciences

Title: English language writing assessment: Teacher practices in Thai universities

Abstract 350 words maximum (PLEASE TYPE)

This thesis explored the assessment practices of teachers of writing for English major students in Thai public universities in different parts of the country. Both teachers' views on assessment practices and their actual rating processes were investigated using an exploratory qualitative research design. First, a questionnaire was used to gather data about teachers' personal and professional backgrounds as well as their views on effective writing instruction and writing assessment in Thai public university contexts. Then, follow-up case studies were conducted to gather more in-depth data from four teachers in four different locations. Semi-structured interviews, think-aloud protocols and stimulated recall interviews were adopted to gather verbal data from each teacher. The think-aloud protocols were used to capture the teachers' thinking processes as they assessed their students' work and to clarify what aspects of writing assessment criteria were emphasised. The data obtained by semi-structured interviews and stimulated recall interviews were then analysed using inductive analysis to identify the major themes relating to the teachers' writing instruction and assessment practices: the think-aloud data were analysed using an adaptation of Cumming eta i.'s (2002) framework to identify teachers' decision-making behaviours during their rating processes.

The overall findings revealed that teachers in Thai public universities noticeably varied in their writing assessment practices. The variability revealed itself in relation to teachers' selection of rating criteria and their components, the degree of cooperation among co-teachers of the same course, the way in which teachers aimed to maintain validity and reliability in their assessment, and the decisions they took in coming to scoring decisions. The findings also revealed a high level of teacher autonomy in the conduct of writing instruction and assessment in the Thai public university contexts. The teacher variability was influenced by various factors: their individual perceptions, their prior or immediate education, their expertise and professional experiences, and contextual factors. This study offers contributions to the relationship between teacher perceptions and writing assessment, a practical basis for universities and teachers in relation to writing instruction and assessment, and further investigations in the areas of writing instruction, writing assessment and related teacher education in Thailand.

Declaration relating to disposition of project thesis/dissertation

I hereby grant to the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or in part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all property rights. such as patent rights . I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation.

I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstracts International (this is applicable to doctoral theses only).

10 June 2014 Signature Witness Date

th<>t th <> r<> m<>v h<> >I r- irr-• r1>r111 iri nn r<>~t r ir-tinn~ nn r-nnvinn nr r-nnrlitinn~ nn ' ~" l'l <>n 11 1>~t~ fnr

circumstances and require the approval of the Dean of Graduate Research.

FOR OFFICE USE ONLY Date of completion of requirements for Award :

ii THIS SHEET IS TO BE GLUED TO THE INSIDE FRONT COVER OF THE THESIS

ORIGINALITY STATEMENT

10 June 2014

AUTHENTICITY STATEMENT

10 June 2014

ACKNOWLEDGEMENT

I would like to thank Professor Anne Burns for her supervision during the writing of this thesis. Professor Burns consistently provided me with timely knowledge, advice and direction throughout my study. I am very grateful to her support, encouragement and patience throughout my journey to complete this project.

I would also like to thank Professor Chris Davison, my co-supervisor, for her valuable advice and support. I am also grateful to Dr. John Knox, my former co-supervisor, for his advice and encouragement during my first year of candidature at Macquarie University.

My thanks go to the Commission of Higher Education, Thailand for their full financial support during my four year candidature. Thanks also to the Department of Languages and Linguistics, the Faculty of Liberal Arts, Prince of Songkla Unviersity Hat Yai Campus, Thailand for granting me permission to take study leave. I would also like to thank Assistant Professor Chonlada Laohawiriyanon for her guidance and encouragement in pursuit of my PhD.

Last but not least, my thanks also go to all teachers who participated in this study, particularly the four case study participants who allowed me to come to their workplace to collect data. Their invaluable data have yielded the body of this research. I must thank a number of people including colleagues in Thailand and my friends who willingly engaged in trialing the questionnaire and the protocols for the case study phase during the early stage of my research design. I owe my special thanks to Mr. Patson Jaihow and Ms. Anchana Rakthong who, despite their busy schedules, provided their time to be the intercoders of the think-aloud data.

Finally, I would like especially to thank my family for their constant moral support and encouragement throughout my journey. I would like to dedicate this thesis to the memory of my father who passed away three years ago. I am certain he is keeping an eye on his youngest daughter’s success from afar.

iii

ABSTRACT

This thesis explored the assessment practices of teachers of writing for English major students in Thai public universities in different parts of the country. Both teachers' views on assessment practices and their actual rating processes were investigated using an exploratory qualitative research design. First, a questionnaire was used to gather data about teachers’ personal and professional backgrounds as well as their views on effective writing instruction and writing assessment in Thai public university contexts. Then, follow-up case studies were conducted to gather more in-depth data from four teachers in four different locations. Semi-structured interviews, think-aloud protocols and stimulated recall interviews were adopted to gather verbal data from each teacher. The think-aloud protocols were used to capture the teachers’ thinking processes as they assessed their students’ work and to clarify what aspects of writing assessment criteria were emphasised. The data obtained by semi-structured interviews and stimulated recall interviews were then analysed using inductive analysis to identify the major themes relating to the teachers’ writing instruction and assessment practices; the think-aloud data were analysed using an adaptation of Cumming et al.’s (2002) framework to identify teachers’ decision-making behaviours during their rating processes.

The overall findings revealed that teachers in Thai public universities noticeably varied in their writing assessment practices. The variability revealed itself in relation to teachers’ selection of rating criteria and their components, the degree of cooperation among co-teachers of the same course, the way in which teachers aimed to maintain validity and reliability in their assessment, and the decisions they took in coming to scoring decisions. The findings also revealed a high level of teacher autonomy in the conduct of writing instruction and assessment in the Thai public university contexts. The teacher variability was influenced by various factors: their individual perceptions, their prior or immediate education, their expertise and professional experiences, and contextual factors. This study offers contributions to the relationship between teacher perceptions and writing assessment, a practical basis for universities and teachers in relation to writing instruction and assessment, and further investigations in the areas of writing instruction, writing assessment and related teacher education in Thailand.

Table of Content

ORIGINALITY STATEMENT ...... i COPYRIGHT STATEMENT ...... ii AUTHENTICITY STATEMENT ...... ii ACKNOWLEDGEMENT ...... iii ABSTRACT ...... iv Table of Content ...... v List of Figures ...... xii List of Appendices ...... xiii

CHAPTER 1 ...... 11 INTRODUCTION ...... 1 1.1 Introduction ...... 1 1.2 English Instruction and Assessment in Thailand ...... 1 1.3 The Assessment of English Writing ...... 5 1.4 Scope and Aim of the Study ...... 8 1.5 Significance of the Study ...... 8 1.6 Structure of the Thesis ...... 9

CHAPTER 2 ...... 11 LITERATURE REVIEW ...... 11 2.1 Introduction ...... 11 2.2 A Paradigm Shift in Assessment ...... 11 2.3 Classroom Writing Assessment ...... 15 2.4 Characteristics of Classroom Writing Assessment ...... 18 2.4.1 Writing task ...... 19 2.4.2 Rating criteria ...... 22 2.4.3 Raters ...... 25 2.5 Teacher Perceptions ...... 33 2.6 Teacher Perceptions and Classroom Writing Assessment ...... 41 2.7 Conceptual Frameworks ...... 46 2.7.1 The Davison (2004) cline of assessor types ...... 46

2.7.2 The Borg (2006) language teacher cognition model ...... 44 2.8 Summary ...... 46

CHAPTER 3 ...... 47 RESEARCH METHODOLOGY ...... 47 3.1 Introduction ...... 47 3.2 Research Questions ...... 47 3.3 Qualitative Research Paradigm ...... 48 3.3.1 Questionnaire ...... 51 3.3.2 Case study approach ...... 53 3.4 Research Methods ...... 55 3.4.1 Questionnaire ...... 56 3.4.2 Semi-structured interviews ...... 59 3.4.3 Think-aloud protocol ...... 60 3.4.4 Stimulated recall interview ...... 61 3.4.5 Documents...... 62 3.5 Case Study Participants ...... 63 3.6 Ethical Considerations ...... 64 3.7 Data Collection Procedure ...... 65 3.8 Data Analysis ...... 68 3.8.1 Questionnaire data ...... 69 3.8.2 Verbal report data ...... 70 3.8.2.1 Think-aloud data ...... 70 3.8.2.2 Interview data ...... 75 3.8.3 Documentary data ...... 79 3.9 Summary ...... 80

CHAPTER 4 ...... 81 TEACHERS’ VIEWS ON WRITING ASSESSMENT PRACTICES IN THAI PUBLIC UNIVERSITIES...... 81 4.1 Introduction ...... 81 4.2 General Background of the Respondents ...... 83 4.3 Teachers’ Views on Features of Good Writing ...... 85 vi

4.4 Teachers’ Views on Rating Practices ...... 89 4.4.1 Views on rating criteria ...... 90 4.4.2 Views on reliability and validity of assessment ...... 95 4.6 Teachers’ Views on Writing Assessment in Thailand ...... 103 4.6.1 Problems of students as writers ...... 103 4.6.2 Problems of teachers as raters ...... 107 4.6.3 Writing course management issues ...... 109 4.7 Overall Pattern of Teachers’ Views on Writing Assessment Practices in Thai Public Universities ...... 110 4.8 Summary ...... 115

CHAPTER 5 ...... 117 TEACHERS’ WRITING ASSESSMENT PRACTICES: LADDA AND NITTAYA 117 5.1 Introduction ...... 117 5.2 Ladda ...... 118 5.2.1 Profile ...... 119 5.2.2 The writing course context...... 119 5.2.3 The nature of students ...... 120 5.2.4 Classroom practices ...... 124 5.2.5 Actual writing assessment practices ...... 130 5.2.5.1 Task profile ...... 131 5.2.5.2 Rating processes ...... 134 5.2.5.3 Product of writing assessment practices ...... 143 5.2.6 Summary of Ladda’s views and practices ...... 145 5.3 Nittaya ...... 146 5.3.1 Profile ...... 146 5.3.2 The writing course context...... 147 5.3.3 Classroom practices ...... 148 5.3.4 Actual writing assessment practices ...... 154 5.3.4.1 Task profile ...... 155 5.3.4.2 Rating processes ...... 157 5.3.4.3 Product of writing assessment practices ...... 169 5.3.5 Summary of Nittaya’s views and practices ...... 172

vii

5.4 Summary ...... 172

CHAPTER 6 ...... 173 TEACHERS’ WRITING ASSESSMENT PRACTICES: REWADEE AND SAK ..... 173 6.1 Introduction ...... 173 6.2 Rewadee ...... 173 6.2.1 Profile...... 174 6.2.2 The writing course context ...... 174 6.2.3 Classroom practices ...... 175 6.2.4 Actual writing assessment practices ...... 181 6.2.4.1 Task profile ...... 182 6.2.4.2 Rating processes ...... 183 6.2.4.3 Product of writing assessment practices ...... 192 6.2.5 Summary of Rewadee’s views and practices ...... 195 6.3 Sak ...... 195 6.3.1 Profile...... 196 6.3.2 The writing course context ...... 196 6.3.3 Classroom practices ...... 196 6.3.4 Actual writing assessment practices ...... 201 6.3.4.1 Task profile ...... 202 6.3.4.2 Rating processes ...... 203 6.2.4.3 Product of writing assessment practices ...... 212 6.3.5 Summary of Sak’s views and practices ...... 215 6.4 Summary ...... 215

CHAPTER 7 ...... 217 CROSS-CASE ANALYSIS: COMMONALITIES AND DIFFERENCES IN TEACHERS’ WRITING ASSESSMENT PRACTICES ...... 217 7.1 Introduction ...... 217 7.1 Teachers’ Profiles ...... 217 7.2 The Writing Course Contexts ...... 219 7.3 Classroom Practices ...... 219 7.4 Assessment Practices ...... 222 viii

7.5 Summary ...... 229

CHAPTER 8 ...... 231 WRITING ASSESSMENT PRACTICES OF TEACHERS IN EFL CLASSROOM CONTEXTS: DISCUSSION ...... 231 8.1 Introduction ...... 231 8.2 Writing Instruction in Thai Public Universities ...... 231 8.3 Writing Assessment Practices of Teachers in Classroom Contexts ...... 236 8.3.1 The employment of rating criteria ...... 237 8.3.2 Reliability and validity of writing assessment ...... 240 8.4 Teacher Perceptions and Writing Assessment Practices ...... 248 8.5 Summary ...... 253

CHAPTER 9 ...... 255 CONCLUSION ...... 255 9.1 Introduction ...... 255 9.2 Summary of Major Findings ...... 255 9.3 Implications for Writing Instruction and Assessment ...... 256 9.3.1 Writing instruction and assessment in Thailand ...... 256 9.3.2 Pre-service teacher education ...... 258 9.3.3 Writing instruction and assessment of in-service teachers ...... 260 9.3.4 Writing instruction and assessment in other East Asian EFL contexts ...... 264 9.4 Limitations of the Study ...... 266 9.5 Recommendations for Further Research ...... 267 9.6 Summary ...... 268

References ...... 271

List of Tables

Table 2.1 A cline of assessor types ...... 47 Table 3.1 Description of the four case study participants ...... 64 Table 3.2 Coding scheme for think-aloud protocols ...... 73 Table 3.3 Example extracts ...... 74 Table 3.4 Summary of categories in relation to Ladda ...... 77 Table 3.5 Summary of categories in relation to Nittaya ...... 78 Table 3.6 Summary of categories in relation to Rewadee...... 78 Table 3.7 Summary of categories in relation to Sak ...... 79 Table 4.1 Respondents’ demographic information ...... 83 Table 4.2 Rating average of writing features ...... 86 Table 4.3 Additional features of good writing ...... 87 Table 4.4 The frequency level of teachers’ views on rating criteria ...... 91 Table 4.5 Teachers’ views on rating criteria ...... 93 Table 4.6 The frequency level of teachers’ views on reliability and validity ...... 96 Table 4.7 Teachers’ comments on the usefulness of training ...... 101 Table 4.8 Teachers’ views on students’ problems ...... 104 Table 4.9 Teachers’ views on teachers’ problems ...... 108 Table 4.10 Teachers’ beliefs and practices adapted from Davison (2004, p. 325)...... 114 Table 5.1 Ladda’s representation of the nature of the students ...... 121 Table 5.2 Ladda’s representations of the classroom practices ...... 124 Table 5.3 Frequencies of overall decision-making behaviours exhibited by Ladda .... 142 Table 5.4 Mean scores of feedback types ...... 144 Table 5.5 Nittaya’s representations of the classroom practices...... 149 Table 5.6 Frequencies of overall decision-making behaviours exhibited by Nittaya ... 168 Table 5.7 Mean scores of feedback types ...... 171 Table 6.1 Rewadee’s rating criteria ...... 175 Table 6.2 Rewadee’s representations of the classroom practices ...... 176 Table 6.3 Frequency of overall decision-making behaviours exhibited by Rewadee .. 191 Table 6.4 Mean scores of feedback types ...... 193 Table 6.5 Sak’s representation of classroom practices ...... 197 Table 6.6 Frequencies of overall decision-making behaviours exhibited by Sak ...... 211 x

Table 6.7 Frequency of feedback types ...... 213 Table 7.1 Description of the four teachers ...... 217 Table 7.2 Summary of major emerging issues in the teachers’ classroom practices ... 220 Table 7.3 Summary of sequences of the rating processes ...... 222 Table 7.4 Frequency of decision making behaviours ...... 227 Table 7.5 Frequency of feedback types ...... 228

List of Figures

Figure 2.1 Components of performance-based assessment...... 18 Figure 2.2 Elements and processes in language teacher cognition...... 45 Figure 4.1 Respondents’ professional background ...... 84 Figure 4.2 Types of criteria in terms of providers ...... 90 Figure 4.3 Overall pattern of teachers’ views on writing assessment practices ...... 111 Figure 4.4 Pattern of teachers’ views on good writing features ...... 112 Figure 5.1 Sequence of Ladda’s decision making behaviours ...... 141 Figure 5.2 Sequence of Nittaya’s decision making behaviours ...... 167 Figure 6.1 Sequence of Rewadee’s decision-making behaviours ...... 190 Figure 6.2 Sequence of Sak’s decision-making behaviours ...... 209 Figure 8.1 The interrelation of assessor types and features of good writing...... 239 Figure 8.2 Interrelation of teacher perceptions and writing assessment practices ...... 252

xii

List of Appendices

Appendix A ...... 287 The questionnaire...... 287 Appendix B...... 292 The semi-structured interview question guide ...... 292 Appendix C...... 294 Think-aloud instructions and training guide ...... 294 Appendix D ...... 296 Stimulated recall interview question guide ...... 296 Appendix E ...... 297 Participant information statement and consent form ...... 297 Appendix F ...... 300 The transferred ethics approval ...... 300 Appendix G ...... 302 Ethics approval for the case studies ...... 302 Appendix H ...... 303 Examples of questionnaire coded responses ...... 303 Appendix I ...... 311 Transcription conventions for think-aloud protocols ...... 311 Appendix J ...... 312 Detailed coding scheme ...... 312 Appendix K ...... 321 Sample of coded think-aloud protocols ...... 321 Appendix L ...... 341 Inter-coder percentage agreement of think-aloud protocols ...... 341 Appendix M ...... 343 Sample excerpts of Ladda’s interview data (Thai) ...... 343 Appendix N ...... 360 Sample excerpts of Ladda’s stimulated recall interview ...... 360 Appendix O ...... 364 Sample of Ladda’s marked writing scripts ...... 364

xiii

Appendix P ...... 367 Sample of Nittaya’s marked writing scripts ...... 367 Appendix Q ...... 368 Rewadee’s writing course syllabus ...... 368 Appendix R ...... 371 Sample of Rewadee’s marked writing scripts ...... 371 Appendix S ...... 372 Sak’s guided essay ...... 372 Appendix T ...... 374 Sample of Sak’s marked writing scripts ...... 374

xiv

CHAPTER 1 INTRODUCTION

1.1 Introduction The present study focuses on the classroom writing assessment practices of teachers in Thai public universities. To provide a foundation for this focus, Chapter 1 provides an overview of the background and context of the study. In addition, Thai teachers’ views on writing assessment practices as well as the processes carried out by teachers in different universities to conduct reliable and valid assessments of their students’ writing will be examined. This chapter will first preview English language instruction and assessment in Thailand before moving on to briefly discuss the assessment of English writing more specifically. The scope and aim of this study as well as the research questions will also be identified. This chapter will conclude with a discussion of the significance of the study.

1.2 English Instruction and Assessment in Thailand In Thailand, English is the main foreign language learnt by students at all levels of education. The English syllabus has been subject to many reforms since it was first made a compulsory foreign language in Thailand in 1921 (Wongsothorn, Hiranburana, & Chinnawongs, 2002). According to the 1999 National Education Act, English is taught as a compulsory course from the first grade (Wiriyachitra, 2002) in the hope that students will have a good command of the English language following school education. As one aim of the 1999 National Education Act is to promote Thailand’s competitiveness in the world of commerce, education, science and technology, the promotion of Thai learners’ English proficiency is regarded as an important endeavour (Wongboonsin et al., 2003 cited in Prapphal, 2008; Wiriyachitra, 2002) Tertiary institutions in Thailand are regarded as integral to the development of students’ English language skills to strengthen the nation’s international competitiveness (Kirtikara, 2001). As a result, the institutions continue to revise their English syllabuses in order to provide students with a greater range of English courses to study. In addition, English teaching methods have undergone changes in response to the various major educational reforms delivered through the 1999 National Education Act. Since 1999, the teaching of

English has transitioned from an emphasis on grammar-translation to a more communicative approach. All four language skills are now emphasised in order to help learners develop their skills to communicate well. Furthermore, the changes to the English syllabuses in tertiary institutions have influenced how teachers assess and evaluate their students’ language ability. The focus on testing traditionally driven by psychometric perspectives of assessment such as translation, pattern drills and structures is now geared towards alternative assessment (Prapphal, 2008; Wongsothorn, et al., 2002). However, it may be argued that the reforms are directed more towards teaching instruction than assessment practices. While the reforms to teaching aim to promote students’ communicative ability, assessment methods corresponding to the reforms to teaching are not highly promoted. As Pillay (2002) remarks, “assessment is a core component of learning but there has been very little if [sic] any work done on how to design and implement methods of assessment that are congruent with the new models of teaching and learning” (p. 51). Instead, objective testing tends to be more commonly employed (Prapphal, 2008). One main reason for this is that many teachers may not be ready to conduct alternative assessment processes in the classroom context. Because education in Thailand has a long history of teacher-centred instruction (Pillay, 2002), a sudden change to student-centred learning may cause confusion and uncertainty among teachers. Due to a lack of professional development, teachers may tend to rely on the teaching and assessment methods with which they are most familiar. As Prapphal (2008) points out, the majority of language teachers do not have knowledge about language testing, assessment and evaluation; even though training programs are launched, they can only be accessed by limited groups of teachers. An explanation may be that only a small proportion of teachers are interested to participating in such training due to their heavy workloads. Further, training programs are normally provided by universities or organisations (Pillay, 2002) located in cities where teachers in remote areas may find it difficult to access. According to Pillay (2002), it is not only teachers who are required to change their methods of assessment, school inspectors and school administrators are also required to increase their knowledge of new assessment methods. This remark appears to suggest a system-wide process of assessment in line with the principle of teacher-based assessment.

Internationally, teacher-based assessment is widely recognised as a successful assessment practice. It is employed as a policy-supported practice in many countries such as Australia, Canada, the United Kingdom, South Africa and Singapore (Davison & Leung, 2009). It covers all aspects of the process of teaching, learning and assessment in educational systems. In this practice, teachers are regarded as important agents because “they are involved at all stages of the assessment cycle, from planning the assessment programme to identifying and developing appropriate formative and summative assessment activities right through to making the final judgments” (Davison & Leung, 2009, p. 401). Based on the literature and the implementation of teacher-based assessment in Hong Kong, Hamp-Lyons (2009) proposes that conducting good assessment practice requires teachers to be capable of conducting fair and valid assessment. Thus, they need to receive extensive professional development. Teachers should work collaboratively both within and across schools to improve and enhance their knowledge and abilities in teaching and assessment. Moreover, there should be a system-wide process to help improve teaching and learning. Quality assurance and validation protocols should also be implemented. In the Thai education context where psychometric assessment is highly dominant, the transition towards more alternative assessment approaches may not be implemented country wide. As a result of Thai teachers’ limited knowledge of testing and assessment, alternative assessment methods are not widely employed in schools. They are, however, more commonly employed in university level English classes. To assess students’ productive skills; namely, speaking and writing, many universities across Thailand now conduct some form of performance assessment (e.g., the use of direct writing tests in writing courses). However, changes to the methods of assessment may place pressure on teachers to adopt more appropriate ways to assess their students’ language ability. Moreover, a teacher’s ability to conduct a reliable and valid assessment becomes more of an issue of concern (Prapphal, 2008). Chinda (2009) mentions in his study on teacher development in testing and assessment in a Thai university that his fellow teachers of the same English course had problems with assessment and there was no expert to help solve the problem in his department. He found the five teachers involved in his pilot study differed in their knowledge of, and practices in, assessment and the development of

rating criteria. Chinda’s study reveals that even in a small context, individual teachers have their own way of understanding assessment and of employing assessment practices. There is no doubt that such teacher variability would be even greater if the same study was conducted on a larger scale. In turn, this variability appears to be influenced to a large extent by teachers’ individual perceptions of assessment. Education quality assurance is also emphasised in the 1999 National Education Act. All levels of education have to improve their systems to meet the new standards, with universities expected to become more globalised and to meet international standards (Kirtikara, 2001; Prapphal, 2008). It appears that one of the ways universities are to meet this aim is to ensure comparability among the education service provided to students. In so doing, improvements to both teaching and assessment practices in universities are called for. As mentioned earlier, teachers have different understandings of assessment as a concept and assessment practices more specifically. Because of this, there is the likelihood that assessment protocols and practices will differ among universities. Diversity may be possible; however, it should be minimised to ensure comparability in educational standards. Teachers have an integral role in helping universities and their students achieve high educational standards (Prapphal, 2008). Indeed, teachers need to be the first stakeholder group to have an accurate understanding of assessment, and to conduct appropriate and efficient assessment practices. Moreover, given that teachers play a key role in shaping the classroom context, they are also required to engage in a number of assessment activities such as:

Report[ing] on subsequent achievements, of individual students in their courses; diagnos[ing] individual learning challenges or problems; determin[ing] student groupings for placements or learning tasks; evaluat[ing] in an informed manner test instruments for their validity and suitability; and interpret[ing] and apply[ing], often in collaboration with other teachers, curriculum policies based on benchmark standards or criteria (Cumming, 2009, p. 516).

To assist teachers to meet these requirements, their professional knowledge and ability, particularly in relation to assessment should be developed (Cumming, 2009). In so doing, their understandings of assessment as well as how they actually conduct their assessment practices may need to be realised first.

1.3 The Assessment of English Writing English writing is considered a difficult skill for students to master and for teachers to teach and assess. In constructing a good written text, writers need to pay more attention to writing conventions at both rhetorical and linguistic levels. They need to focus on the development of ideas and information, the use of vocabulary and certain grammatical patterns belonging to certain text types, as well as sentence structure in constructing texts so that they can build interaction between the text itself and the readers (Celce-Murcia & Olshtain, 2000; Graddol, Cheshire, & Swann, 1994; McCarthy, 1991). This requirement can lead to difficulty in teaching in that teachers need to find appropriate teaching methods to help promote students’ writing ability. In addition to the teaching of writing, assessing students’ writing ability is another concern for many teachers. Two significant constructs in effective assessment; namely, reliability and validity, are what teachers have to bear in mind while assessing their students’ work. Reliability refers to the consistency of the assessment process. Validity concerns the worthiness of the assessment; that is, what assessed is supposed to be assessed by an appropriate instrument of assessment (Anderson, 2003; Popham, 2011). However, the two concepts do not represent two completely unrelated objectives. For instance, while an assessment may be reliable but not valid, a valid assessment needs to be reliable. Moreover, the two concepts can influence the construction and use of tasks and rating criteria throughout the course as well as affect the teacher’s decision making when judging the student’s language ability (Miller, Linn, & Gronlund, 2009; Mousavi, 2002). When assessing a student’s writing, if the teacher has inadequate knowledge of how to assess writing it may undermine the validity and reliability of the rating process. To overcome this problem, objective-type questions may be the preferred choice to assess the writing ability of the student. Wongsothorn (1994 cited in Kongpetch, 2006) mentions that tests in most writing courses emphasise objective-type questions for which students are required to complete sentences, reorder words, reorder sentences, and correct errors. This appears to be an effective way to ensure the reliability of the assessment in that raters can assess writing tasks in the same way. However, it raises questions about the validity of the skill assessed. Students need to plan, generate and revise drafts before completing a writing task if they are to demonstrate their true writing ability (Lumley, 2002).

To assess students’ actual writing ability, direct writing seems to be a more appropriate assessment practice than indirect writing. Direct writing can help teachers to recognise each student’s actual writing performance as it “requires subjects to show what they know by performing specified tasks” (Mousavi, 2002, pp. 492-493). It also facilitates teachers to identify writing problems they can help to solve (Richards, 2003). In classroom contexts, writing tasks can be used for formative and summative assessment depending on the purposes of the course. The former normally aims to help promote learning while the latter aims to help evaluate the learner’s proficiency at the end of the course (K. Hyland, 2003; Miller, et al., 2009). Task, rater and rating criteria are three major factors that can help maintain the reliability and validity of the assessment (McNamara, 1996). The tasks students perform need to reflect the purpose of the course and should be as close to real-life communication as possible. The task should also reflect writing conventions in the text type being taught. Qualified raters are required to assess the writing task using rating criteria appropriate to each text type. Also, consistency of the rating process needs to be ensured (K. Hyland, 2003; Mousavi, 2002; Norris, Brown, Hudson, & Yoshioka, 1998). However, given that human raters are required for the assessment of classroom writing, variability among teachers’ assessment practices, especially in regard to the rating process, may diminish the reliability of the assessment. Teachers are the key agents in designing writing tasks and in employing rating criteria when assessing their students’ work. In the rating process, the rater’s subjectivity may play a role. The rater’s bias towards student writing performances, different perceptions of good writing components, and the cultural and professional background of the rater are all the factors that can influence the assessment of the student’s writing. Also, contextual factors such as course requirements and the time of day the writing tasks are assessed can affect the assessment outcome (Cumming, Kantor, & Powers, 2002; Kobayashi, 1992; Shi, 2001; Wood, 1993; Weigle, 2002). The use of rating criteria may also result in reliability problems in the rating process. Because each writing genre has its own conventions, rating criteria appropriate to one text type may not be appropriate to others. There are various kinds of rating criteria teachers can employ when assessing their students’ writing and each of them has its advantages and disadvantages. It is up to teacher raters to select which type of

criteria is most appropriate to assess a particular type of writing. Thus, the most important component of the rating process is the rater because the rater decides which scale features to focus on, how to adapt scale wording to suit the situation, and how to justify the written texts according to the educational contexts and requirements (Lumley, 2002). As mentioned earlier, raters therefore need to keep many things in mind during the rating process in order to maintain the reliability and validity of the assessment. Research into rating processes can help to broaden our understanding of how teachers in a regular classroom maintain reliability and validity in their assessment. As Connor-Linton (1995b) claims, “if we do not know what raters are doing (and why they are doing it), then we do not know what their ratings mean” (p. 763). The teacher’s mental processes, or what Freeman (2002, p. 1) calls as “the hidden side of teaching”, needs to be made more explicit. Thus, research which investigates the perceptions and practices of teachers in relation to classroom assessment are highly valuable (Leung & Lewkowicz, 2006). Assessment reliability and validity seem to be concepts of greater concern in the context of large-scale assessments than in the smaller scale context of the classroom. Most studies of rating process have been conducted in a context where central rating criteria are used and experienced raters are asked to assess the same set of writing (e.g. Barkaoui, 2007; Eckes, 2008; Lumley, 2002; Milanovic, Saville, & Shuhong, 1996). Nonetheless, the cognitive processes of raters is a growing research domain (Read, 2010). Furthermore, much research contributing to the study of the rating process has been conducted in the contexts of both English as a first language (L1) and English as a second language (ESL) (e.g.Cumming, et al., 2002; Davison, 2004; Lumley, 2002). However, few studies can be found in the EFL context (Cheng, Rogers, & Hu 2004), especially in Thailand. Moreover, most studies in this area focus on rater decision- making in the context of large-scale standardised examinations (Cumming et al., 2002; Lumley, 2002). Little research has been conducted in the classroom context where teacher raters vary in their perceptions, backgrounds and experiences. Thus, this study focuses on the teacher raters’ actual rating processes in their natural settings.

1.4 Scope and Aim of the Study Because direct writing tasks are widely employed in English writing classes, teachers as raters are required to provide students with a fair and valid assessment of their written product. Due to the subjectivity of the assessor, variability in assessment practices may occur naturally. The teachers’ assessment practices may be influenced by their own perceptions and contextual factors such as their previous knowledge, course objectives, and institutional expectations. As such, it is important to explore teachers’ views on writing assessment and to investigate their rating processes and the rating criteria they employ. This study will therefore seek to explore the relationship between classroom writing assessment practices and teacher perceptions of assessment. This study aims to investigate the rating processes used by teacher raters while assessing their students’ writing tasks. Their decision making will also be examined. Qualitative research methods will be adopted to explore the following research questions: 1. How do teachers of writing in Thai public universities view their assessment of students’ writing? 2. How is writing of a specific text type assessed by Thai teachers in different public universities? What influences their selection of the rating criteria? 3. How do teachers’ views of writing assessment influence their actual rating practices?

1.5 Significance of the Study As stated in Section 1.3 and further discussed in Chapter 2, research that links writing assessment to teacher perceptions of assessment is limited. Most studies are conducted in the context of large-scale assessment (e.g., Lumley, 2002) and few studies are therefore found which explore assessment in the EFL classroom context (Cheng, Rogers, & Hu, 2004). This study contributes to theory building in the area of writing assessment and teacher perceptions, provides a practical basis for universities and teachers in relation to classroom writing assessment, and offers a knowledge basis for further investigations. For theory building, the study addresses the relationship between teacher perceptions and writing assessment in order to fill the existing gap between them. Practically, it will be helpful in facilitating understanding of teacher practices in actual assessment situation. This understanding can lead to the improvement of classroom writing instruction and assessment in the Thai university context, both at the

university level and the course level. It can also provide valuable data for teacher education and professional development of both pre-service and in-service teachers in assessing EFL students’ classroom writing. Also, this study can also provide a knowledge basis for future researchers in the area of classroom writing instruction and assessment, and teacher education.

1.6 Structure of the Thesis A summary of the organisation of this thesis is outlined below. Following Chapter 1, Chapter 2 reviews two pertinent areas in the relevant research literature: writing assessment and teacher perceptions. The chapter begins by unfolding the trend in the shift in assessment paradigms. Then, it outlines the key characteristics of classroom writing assessment including writing tasks, rating criteria and rater. Studies that investigate the rating process are also reviewed. The chapter also discusses teacher perceptions, and their relationship to teacher writing assessment practices is also highlighted. Chapter 3 discusses the research methodology employed in this study. The chapter firstly provides the rationale for using qualitative research paradigm. It then presents the research design of the study which includes two phases: questionnaire and case study. The chapter then proceeds to illustrate the methods of data collection. Case study participants, ethics consideration and the data collection procedures are also outlined. The last part of Chapter 3 describes the data analysis methods and procedures through each phase of the study. Chapters 4 to 7 present the findings of this study. Chapter 4 offers findings in relation to the overall views of writing assessment practices held by teachers across different universities in Thailand. Chapters 5 and 6 present the findings from the analyses of the four single case studies conducted. Each case study teacher’s views on writing instruction and assessment as well as their actual assessment processes will be highlighted. Each chapter presents the analyses of two individual teachers. For each case, the section begins by drawing on each teacher’s profile and the context of the writing course. Then, the teacher’s perspectives on classroom practices are presented. Each teacher’s actual rating process is also highlighted.

Chapter 7 provides a cross-case analysis of the four teachers’ assessment practices. Comparisons of the findings with regard to the four individual teachers’ views and practices are demonstrated. Chapter 8 is devoted to a discussion of all findings. The first part discusses writing instruction in Thai public universities. It focuses on the way in which the writing courses are organised and how the teachers conducted their instruction. The second part highlights teachers’ writing assessment practices in the context of their regular classroom practices. It discusses the issues related to the selection of rating criteria, and reliability and validity of assessment. The last part provides a discussion that links teacher perceptions and writing assessment practices. Chapter 9 draws conclusions based on the overall findings from the study. The chapter identifies the implications of this study for writing instruction and assessment in a specific context of Thai higher education and the areas of teacher professional development and teacher education in relation to writing instruction and assessment. Then, a critical evaluation of the limitations of this study is provided. The chapter ends with recommendations for future research.

CHAPTER 2 LITERATURE REVIEW

2.1 Introduction Chapter 1 discussed the background and rationale for the study. This chapter provides a literature review of pertinent areas which support the body of this study. As writing assessment is a crucial component of writing instruction, it can have an influence over teaching content, the evaluation of students and how course effectiveness is judged (K. Hyland, 2003). In a classroom context, fair informative and appropriate assessment needs to be ensured through the employment of writing tasks, rating criteria, and human raters when assessing the writing. Among these three factors, human raters are arguably the main source of variability across different classes and institutions. Moreover, their ratings are not clearly understood (Connor-Linton, 1995b; Sakyi, 2000) as they can be influenced by many factors, particularly their perceptions about the principles of assessment and the practices of their own work. This chapter will discuss the theoretical concepts underpinning the two core focus areas of this study: writing assessment and teacher perceptions. The discussion will provide the conceptual lenses through which to understand the writing assessment practices of teachers in a regular classroom context. In relation to writing assessment, this chapter will begin by highlighting the paradigm shift in concepts of assessment from a testing culture to an assessment culture. Following this, the key characteristics of writing assessment as it is undertaken in a classroom context will be discussed. Subsequently, the topic of teacher perceptions will be discussed with a view to show how they may influence and drive writing assessment practices in the classroom. This chapter will end with two conceptual frameworks by which this study is influenced.

2.2 A Paradigm Shift in Assessment As a result of the influence of communicative learning approaches in language education, the concept of assessment has shifted from a psychometric paradigm to an assessment paradigm, and from a testing culture to an assessment culture (Davison, 2004; Gipps, 2012; Wolf, Bixby, Glenn, & Gardner, 1991). The two paradigms can be described as a dichotomy. The testing culture represents traditional concepts of assessment focusing on standardised testing procedures; whereas the assessment culture

is characterised by more alternative approaches of assessment conducted mainly by teachers (Davison, 2004; East, 2009; Gipps, 2012; McNamara, 2001; Moss, 1992, 1994). The psychometric and assessment paradigms differ in a number of ways, with the key differences lying in their purposes. While the psychometric approach aims at the accountability of students’ ability or ‘assessment of learning’, the assessment approach focuses on students’ learning progress or ‘assessment for learning (G. Brown, Kennedy, Fok, Chan, & Yu, 2009; Gipps, 2008). The purposes of the two assessment paradigms resonate with the concepts of formative and summative assessment. Formative assessment consists of informal and fairly constant activities that gather information on students’ learning; whereas summative assessment focuses on testing the student’s ability in a formal standardised testing situation and mainly provides information about their abilities to interested people such as parents and schools (Harlen, 1996). Formative assessment is considered more essential to teaching than summative assessment in that it provides feedback to teachers and students in order to facilitate the teaching and learning process (Black, 2009; Black & Wiliam, 1998, 2003; Gipps, 2012). The definitions of formative and summative assessment as two distinguishable concepts are predominant in the psychometric paradigm. However, they are considered to be interrelated in the assessment paradigm. Summative assessment can be employed for formative purposes if it is used to provide feedback on the students’ performances and assist with their further learning (Kennedy, Chan, Fok, & Yu, 2008; Kennedy, Sang, Wai-ming, & Fok, 2006). The psychometric and assessment perspectives also differ in relation to the way in which student performances are assessed. In the psychometric views, norm- referenced scoring is emphasised; student performances are assessed by being compared with other student performances (Macken & Slade, 1993). The emphasis on norm referencing indicates that ranking student performances in the group is regarded as more crucial than their actual level of learning accomplishment (Gipps, 2012). On the other hand, alternative views of assessment focus on criterion referencing, producing the assessment of the student’s ability against criterion reference scores (Black & Wiliam, 1998; Macken & Slade, 1993). The employment of criterion referencing places an emphasis on the student as an individual rather than comparing the students, and enables

the identification of the individual’s strengths and weaknesses that can lead to his/her learning progress (Gipps, 2012). The two paradigms influence the implementation of different assessment practices, lying across a continuum of objectivity and subjectivity. The psychometric approach stresses objectivity; whereas the assessment approach allows for subjectivity (Gipps, 2012). In the testing culture associated with the psychometric approach, students are treated as passive subjects whose fixed and innate properties are measured by criteria which are externally created and normally context-free and teacher-free (Davison, 2004; Gipps, 2008, 2012). Alternatively, in the assessment culture students are regarded as active learners who are involved in the process of learning and assessment, including self-assessment and peer assessment (Birenbaum & Dochy, 1996; Gipps, 2008; Inbar-Lourie, 2008). In this context, the assessment may encourage them to self-monitor and self-regulate during their learning (Gipps, 2008). Procedures within the assessment culture are conducted by teachers who play the role of ‘a mentor or a coach’ who formulates and scaffolds student learning through constant feedback (Birenbaum & Dochy, 1996; Inbar-Lourie, 2008). Furthermore, the teacher’s interpretation and negotiation of their judgements are important to maintain reliability and validity of assessment (Davison, 2004). Moreover, the relationship between instruction and assessment in the psychometric and assessment paradigms is different. Birenbaum and Dochy (1996) assert instruction and assessment influenced by psychometric paradigms are regarded as two separate activities. Teachers are responsible for the instruction while measurement experts take control of assessment. This scenario may affect the validity of tests that do not reflect the actual instruction. The tests do not measure actual student performances because they are usually in an indirect form (e.g., paper and pencil type). The product of each student’s performances is assessed as it is without giving consideration to the process. In relation to instruction and assessment following the assessment culture, however, Birembaum and Dochy remark that instruction and assessment are integrated. Their claim is in line with the characteristic of instruction and assessment practices proposed by Lynch (2001). The author summarises teaching and assessment practices in the implementation of alternative forms of assessment as follows:

• Teaching and assessment practices should be considered as integral.

• Students should be active participants in the process of developing assessment procedures, including the criteria and standards by which performances are judged. • Both the process and the product of the assessment tasks should be evaluated. • Reporting of assessment results should usually be in the form of a qualitative profile rather than a single score or other quantiﬁcation (p.360).

As mentioned earlier, the implementation of alternative approaches to assessment such as teacher-based assessment, and performance-based assessment has become more widely employed, particularly in the last two decades (Leung & Lewkowicz, 2006). Teachers have become more active in the integration of both instruction and assessment. In relation to writing assessment in a regular classroom context, the teacher’s underlying principles of assessment often seem to lend themselves towards the assessment culture (Mavrommatis, 1997; Shepard, 2002). However, the long dominance of the psychometric principles in educational assessment may still affect teacher instruction and assessment practices. Instead of adopting and implementing alternative approaches of assessment, teachers may find themselves more comfortable with teaching pedagogy they are accustomed to and may tend to apply psychometric principles in their practices. Rea-Dickins (2004) remarks:

[T]eachers find themselves at the confluence of different assessment cultures and faced with significant dilemmas in their assessment practices: sometimes torn between their role as facilitator and monitor of language development and that of assessor and judge of language performance as achievement. (p.253)

Student performance as the key construct to be assessed is primarily context- bound and construed based on the teacher assessor’s perspectives and values (Gipps, 2012). Rather than acting as an ‘instrument of assessment’, teachers are considered to be ‘agents of assessment’ (Rea-Dickins, 2004) or the major decision makers of pedagogic and assessment activities in the classroom according to their own personal and professional experiences. They may utilise a range of assessment orientations (Teasdale & Leung, 2000), and the principles from both the established psychometric assessment and more alternative approaches of assessment may then influence the teacher’s classroom assessment.

2.3 Classroom Writing Assessment Writing assessment in a classroom context is underpinned by the notion of so- called direct writing assessment. This includes “any writing test that involves actual writing, as opposed to completing multiple-choice items” (Weigle, 2002, p. 46). This definition resonates with the notion of performance assessment put forward by a number of scholars. Grabe and Kaplan (1996) define performance assessment as “a type of testing procedure which stresses authentic and direct measures of abilities” (p. 405). McNamara (1996) defines it as the kind of assessment where “actual performances of relevant tasks are required of candidates, rather than more abstract demonstrations of knowledge, often by means of pencil-and-paper tests” (p. 6). The clearest view of performance-based assessment may be one stated by Norris, Brown, Hudson and Yoshioka (1998). The authors mention that in performance-based assessments “(a) examinees must perform tasks, (b) the tasks should be as authentic as possible, and (c) success or failure in the outcome of the tasks, because they are performances, must usually be rated by qualified judges” (p. 8). From all of these definitions, it can be concluded that performance-based assessment is a type of assessment where students are required to demonstrate their actual ability through tasks that are as authentic as possible. They should be assessed by qualified assessors who may be raters who are trained to assess the performances consistently. Moreover, on the basis of these definitions, in classroom writing assessment authentic tasks (i.e., students’ written productions) are often required of students (H. D. Brown, 2004). These tasks are generally assessed by teachers of the courses. Therefore, classroom writing assessment reflects the assessment of students’ learning activities within learning contexts (D. Smith, 2000) and aims to model the authentic learning activities in which the students are expected to engage (Gipps, 2012). As this form of assessment occurs in the classroom context, it can also be regarded as teacher-based assessment because assessment of students’ writing performances is undertaken by teachers as part of their normal classroom instruction (Harlen, 1996). Student performance is assessed against criterion-referenced rather than norm-referenced decisions. In other words, they are judged against a set of standards instead of being compared with other student performances. Classroom writing assessment serves two different purposes: formative and summative assessment. Formative assessment aims at the process of learning. As noted

above, it provides information about strengths and weaknesses of students’ learning, and feedback for the instruction itself. It is seen as pedagogically motivated because the provided information can assist teachers to adjust their teaching to suit the students’ needs. In contrast, summative assessment aims at the results of learning. It focuses on summing up student achievements at the end of the course or unit, and provides outcomes for the program. It normally reflects the administrative requirements of the institution (Black & William, 1998; Hedge, 2000; K. Hyland, 2003). As mentioned earlier, the formative purpose of assessment seems typically to be more predominant than the summative purpose in the context of a regular classroom instruction. As in any form of assessment, reliability and validity are two important constructs in writing assessment. Reliability is important because it has an impact on equity of assessment for individual learners. In general, it can be defined as “consistency of measurement across different characteristics or facets of a testing situation, such as different prompts and different raters” (Weigle, 2002, p. 49). However, in classroom writing assessment it is argued that the notion of reliability is not appropriate because student differences are not the main focus (D. Smith, 2000). For classroom writing assessment, very high score reliability cannot be easily achieved because there are many variations in the rating process, including writers and human raters (Hamp-Lyons, 2003; D. Smith, 2000). Validity refers to the meaningfulness, appropriateness and usefulness of the inferences or decisions made on the basis of test scores (Anderson, 2003; Messick, 1995a; Mousavi, 2002; Weigle, 2002). It can also be understood to be “the degree to which a test measures what it claims, or purports, to be measuring” (J. D. Brown, 1996, p. 231). Assessment tasks should assess what is meant to be assessed and reflect what is taught (Anderson, 2003; K. Hyland, 2003). To maintain validity, Weir (2005) notes that the trait – or what is assessed – and the method – or how it is assessed – need to be understood. Therefore, in the case of writing in the classroom context, the student’s writing performance is the key trait to be assessed through their written compositions. Validity can be divided into five main types: face validity, content validity, criterion validity, construct validity and consequential validity (K. Hyland, 2003). Face validity is the credibility of the assessment so that students and teachers believe that it assesses what it is meant to assess. Content validity is described as the quality of the test as “a representative sample of the content of whatever the test was designed to measure”

(J. D. Brown, 1996, p. 233). Criterion validity traditionally refers to the relationship of a particular test to other tests within the same skill assessment. The relationship can occur when two or more tests are sat at the same or similar time, and when the student’s future performance can be predicted on the basis of the tests. Construct validity can be considered the most important form of validity because it reflects the qualities that are assessed. Consequential validity concerns ‘washback’ or the effects of the scores that students achieve and their implications for teaching (Hamp-Lyons, 2003; K. Hyland, 2003; Khalifa & Weir, 2009; Mousavi, 2002; Weir, 2005). Traditionally, the different types of validity are seen as discrete isolated categories and focus directly on the test itself. However, they are viewed in a more inter-related perspective proposed by Messick (Messick, 1995a, 1995b) as subsumed and integrated in different facets of a unified concept of validity which includes a broader test context. Based on a socio-cognitive view, Messick (1995a, 1995b) notes that validity involves interrelation of several aspects to form a basis for a more comprehensive construct validity which is concerned with evidential and consequential bases of test interpretation and use. In this unitary view of validity, construct validity is considered the broadest quality constituting context validity, cognitive validity and scoring validity. Context validity is proposed to replace traditional content validity which focuses only on linguistic content parameters of the test while context validity is argued to involve other factors (Weir, 2005). Context validity is seen as being “concerned with the extent to which the choice of tasks in a test is representative of the larger universe of tasks of which the test is assumed to be a sample” (Weir, 2005, p.19). In short, it involves not only linguistic content but also social and cultural factors in relation to the test context. Cognitive validity includes the cognitive processing required of test takers which test developers need to consider before the actual test administration. Scoring validity is demonstrated in all aspects of reliability which is obtained by various methods such as the use of appropriate criteria and rater agreement in marking the test (Khalifa & Weir, 2009). As revealed in the discussion above, validity plays a large part in any kind of test. It includes not only inferences that create the test score but consequential values influencing the test score and interpretation. Based on the unitary view of validity, reliability is also regarded as a contribution to validity. This view confirms the remark that reliability and validity are both very important in assessment and cannot be clearly

separated. Particularly in the classroom context, the two qualities are considered to exist in a highly complex interplay (Rea-Dickins & Gardner, 2000). The assessment cannot be reliable unless the right skill qualities are assessed. In the same way, validity of the assessment cannot be achieved unless reliability is confirmed (Bachman & Palmer, 1996; Hughes, 2003). As Popham (2011, p. 75) asserts, “if an assessment procedure fails to yield consistent results, it is almost impossible to make any accurate inferences about what a student’s score signifies”. Reliability is a predominant quality in high stakes testing situations where rater subjectivity is regarded as problematic (Leung & Rea-Dickins, 2007). However, in classroom writing assessment, it can be argued that subjectivity is acceptable because reliable and valid assessment can be gained through the teacher’s interpretation and negotiation of judgements (Davison, 2004). The concepts of reliability and validity play a crucial role in developing tasks and rating criteria, and in the rating process. These issues will be further discussed in the next section.

2.4 Characteristics of Classroom Writing Assessment Classroom writing assessment involves authentic tasks to assess actual performances of students. Its particular characteristics can be explained by McNamara’s (1996) diagram of performance-based assessment components adapted from Kenyon (1992) as follows:

RATER RATING (SCORE) SCALE

PERFORMANCE

INSTRUMENT

CANDIDATE

Figure 2.1 Components of performance-based assessment (McNamara, 1996, p. 9)

As Figure 2.1 shows, a student’s actual performance is the construct to be assessed, and students need to interact with the instrument or tasks to perform according to their ability. Human raters are then required to rate the performance based on a rating scale in order to reach the score given to the performance. According to McNamara (1996), the student’s ability, task and rater are the sources of variability in the assessment of writing. The abilities of the students naturally vary. The choices of task may also result in variability in the scores obtained by the students. However, it appears the major source of variability comes from the rater (McNamara, 1996). Task, rater and rating scale are also considered by Norris, Brown, Hudson and Yoshioka (1998) to be the three important characteristics of performance-based assessment. These performance assessment components are reflected in classroom writing assessment. The student’s writing is the performance that teachers need to assess using rating criteria. The interplay between the three key components can be represented in the rating process employed by the teachers and is vital to maintaining reliability and validity of assessment. Each of the three characteristics is now discussed in more detail.

2.4.1 Writing task In the classroom writing context, writing tasks are a crucial component because they determine the written product which may represent the student’s actual writing ability. They can also be used as a form of testing which aims to evaluate student writing performance (Reid & Kroll, 1995). They are, as mentioned by K. Hyland (2003), “at the heart of a teaching unit” (p. 112) as they assist students to learn and to develop their writing skills. K. Hyland defines the task as “any activity with meaning as its main focus which is accomplished using language” (p. 112). In a similar vein, Norris et al. (1998) define tasks as directly related to real life activities. They are defined as “those activities that people do in everyday life and which require language for their accomplishment” (p. 331). While these two definitions of tasks are generally broad in meaning, Bachman and Palmer (1996) provide a more specific definition. They note that a task is “an activity that involves individuals in using language for the purpose of achieving a particular goal or objective in a particular situation” (p. 44). Regarding these definitions, a task in a language classroom may focus on either everyday or specific purposes of communication. However, one fundamental key for any tasks used is that students are required to use language to accomplish them. In a writing class, thus,

tasks should be used to encourage students to write and help them to understand how to use language for particular communicative purposes. Different types of writing tasks in the classroom can also reflect the two ends of McNamara’s (1996) continuum of the strong and weak senses of performance-based assessment. In a strong sense, the assessment is based on real-world tasks where task fulfilment is emphasised. Tasks are assessed according to the achievement of the task purpose. In a weak sense, the language use is stressed. Tasks are assessed regarding the appropriateness and the accuracy of language use. In practice, the task can fall somewhere between the two ends of this continuum depending on the course purpose and the writing genre selection, and they are influenced by the social context of writing (Weigle, 2002). The tasks in McNamara’s continuum are echoed in K. Hyland’s (2003) categorisation of two broad groups of tasks: real-world tasks and pedagogic tasks. The real-world task highlights the student’s target communicative purpose; whereas the pedagogic task emphasises the development of the student’s knowledge of genre and composing skills. Still, tasks selected for use in class should encourage students to develop the competence they need to complete real-world objectives in the future. In the context of EFL writing classroom aiming to develop students’ writing skill as in this study, tasks used are, therefore, likely to lead towards pedagogic tasks rather than real- world tasks. Regardless of the writing task types, one fundamental requirement for task design is that tasks should reflect course objectives and content (Reid & Kroll, 1995). According to K. Hyland (2003), to construct a well-written text, students as writers are required to master five areas of writing knowledge; namely, content, system, process, genre and context. Content refers to topical ideas and knowledge. System concerns knowledge of language features needed to construct a text. Process is the knowledge of the writing process. Genre is the knowledge of rhetorical structure and communicative purposes. Context refers to the knowledge of expectations and the beliefs of the reader. This writing knowledge ranges from writing at both microskill and macroskill levels and according to H.D. Brown (2004) it can be categorised into four groups: imitative, intensive (controlled), responsive and extensive writing. Imitative writing focuses more on form than on the context and meaning of writing. Writing mechanics such as spelling and punctuation are emphasised. Intensive or controlled writing focuses on correctness of grammatical features and appropriateness of

vocabulary in a context, and collocations and idioms at a sentence level. Responsive writing refers to writing at a limited discourse level in which writers are required to write logically connected paragraphs in any writing genre. Discourse conventions suitable for different genres as well as context and meaning are addressed. Extensive writing may be considered the longest writing text such as an essay, a term paper, a research project or a thesis. Writers are required to write a well organised text with logical development of ideas. They are engaged in the process of outlining, drafting and revising. Multiple drafts may be required to produce a final draft. Different types of tasks are designed to suit each of H.D. Brown’s (2004) four types of writing. For imitative and intensive writing, indirect forms of assessment such as gap-filling, multiple-choice and short sentences with picture or vocabulary cues are used. However, responsive and extensive writing are more productive. Writers are likely to be more independent in terms of the selection of vocabulary, grammar and discourse. Direct writing that requires writers to perform at their real ability level in writing is employed. Responsive and extensive writing should be assessed by human raters using rating criteria (H.D. Brown, 2004).Variability in assessment seems to occur in the assessment of responsive and extensive writing rather than in the assessment of imitative and intensive writing. In the classroom context, writing tasks can be used as in-class or timed writing, or out-of-class or untimed writing (Weigle, 2002). According to Weigle (2002), in-class writing refers to writing tasks that students are asked to write within the time limit set in class. It is generally used as a testing activity in that teachers can assess their students’ writing ability individually. Alternatively, out-of-class writing tends to be used as a learning activity whereby teachers assign students to write as homework or to practice what they have learned in class. It can promote the student’s writing experience in that students may have to go through all stages in the writing process as well as the use of various resources. Even though the two writing assessment types are different, they can both help teachers to recognise the students’ writing progress and to assess their ability to write. When designing writing tasks, K. Hyland (2003) proposes that the key practice for teachers is consideration for the introduction of linguistic forms in the writing process. Teachers should make decisions on what language features are to be emphasised and when they should be taught. This consideration may be influenced by

the writing pedagogies adopted by the teachers. Two common but contrasting methods of writing instruction are the text-based and the process-based approaches. The text- based approach focuses mainly on text construction in relation to corresponding text types; whereas the process-based approach generally stresses linguistic features at the stage of editing (Coffin et al., 2003; K. Hyland, 2003). The focus on linguistic features at a later stage normally illustrates the teacher’s response to student errors rather than indicating areas of need during the composition process (K. Hyland, 2003). Because the focus of this study is on classroom-based writing assessment at the higher education level, the writing tasks typically employed are likely to vary between responsive and extensive writing depending on the purpose of the course and student ability levels. In an ‘English as a foreign language’ (EFL) class which aims to improve student knowledge and performance in English, various writing text types may be used as a platform for students to practise and develop their writing skills. Typical text types are description, narration, exposition, argumentation and instruction, among which argumentation is considered the most important in academic writing (Björk, 2003). The different writing text types govern different conventions, including rhetorical and linguistic features. The differences may influence the assessment of writing in that particular conventions belonging to a certain text type need to be considered when writing in that text type is assessed. In-class or out-of-class writing assessments are used and teachers are usually the main raters of the writing. However, the assessment of a written product cannot be done consistently without employing the appropriate rating criteria.

2.4.2 Rating criteria As criterion-referenced assessment practices are now more widely used than norm-referenced methods (K. Hyland, 2003), the assessment of a student’s writing performance has changed from making comparisons among student performances to assessing against a set of criteria. A criterion-referenced approach may be regarded as one of the ways to ensure high consistency in teacher judgments (Leung, 2004). It is stated to serve the aims of diagnostic assessment and focuses on the achievement of learning objectives (McNamara, 2001). In order to assess student writing ability reliably and validly, teachers need to set clear objectives or criteria (H. D. Brown, 2004) which describe the required performance level (Shaw & Weir, 2007). Thus, rating criteria

come into play in the rating process as they are a tool of assessment teachers use to assess student performance. They comprise a group of descriptors based on assessment purposes and the assessed construct (Shaw & Weir, 2007). Three to nine performance levels are typically employed (McNamara, 2000) and a score is assigned when a performance meets specific descriptors. Generally, writing can be assessed according to three main rating methods: holistic, analytic and trait-based. Holistic rating or impression rating is an assessment of the overall proficiency level or band based on a student’s writing task. A single score is assigned when the performance level corresponds to a particular set of descriptors. Holistic rating has a number of advantages. Raters can assess what students do well rather than focusing on their deficiencies. It also offers a fast evaluation and promotes high intra-rater reliability; that is, the same rater can consistently rate different writing pieces. Validity of the assessment can be ensured as the overall writing performance construct is assessed (Shaw & Weir, 2007). However, this kind of rating is argued as problematic because a single given score does not provide an exact indication of the student’s proficiency level in relation to different aspects of language. It may also be difficult for raters to assign a score because the student’s work consists of various writing features. Allocating a single number may cause raters to overlook some important writing features, and this may diminish the reliability of the rating. The score assigned may not be interpreted by raters and students in the same way unless they understand the criteria in the same way (Bachman & Palmer, 1996; H. D. Brown, 2004; A. D. Cohen, 1993; Ferris & Hedgcock, 1998, 2005; Hamp-Lyons, 1991; Reid & Kroll, 1995). These problems can be avoided through the implementation of clear rubrics or rating guides; by seeking help from peer-raters to monitor reliability; and by using reliable, explicit assessment practices (Ferris & Hedgcock, 2005). In actual classroom contexts, however, teachers as raters may “prefer a more-finely tuned instrument or more flexible tool for assigning ‘borderline’ marks (i.e., those that fall between bands)” (Ferris & Hedgcock, 2005, p. 312). Therefore, explicit rating criteria can help teacher raters balance their emphases on the multidimensional salient features that constitute effective writing and reduce a tendency for raters to bias their assessment on particular writing features (Hamp-Lyons, 1991). Because holistic rating fails to offer diagnostic feedback essential to student writing improvement, analytic rating is seen as a better option, particularly in ESL/EFL

contexts. Based on the assumption that writing is complex and multifaceted, analytic rating allows raters to assess writing in distinctive categories such as organisation, content, and punctuation. It offers diagnostic feedback to students’ writing in that assessed features are clearly defined by being separated and weighted. Teachers can give consistent and direct feedback to students based on the explicit categories assessed. Profiles of student language proficiency levels can be easily reviewed and teachers can use them as a resource for improving student writing ability. Accordingly, an analytic rating scale seems more appropriate to assess second language writing because ESL learners have different proficiency levels. Some students may be better at using linguistic features; whereas other students may be better at using rhetorical features. Being assessed according to an analytic scale can help to identify student proficiency levels and lead to their writing improvement (Bachman & Palmer, 1996; A. D. Cohen, 1994; Hamp-Lyons, 1991; Shaw & Weir, 2007). Nonetheless, analytic rating does have its disadvantages. The assessment process using analytic rating criteria is time-consuming. Moreover, even though language features are explicitly separated and weighted, it may be difficult to clearly distinguish the assessed language features because rating one feature of writing may have an influence on the rating of other features (Hughes, 2003). Overlapping or ambiguity of the assessed features may also occur. As Ferris and Hedgcock (2005) point out, raters using an analytic rating method may be biased by language features that are easily identified. Grammatical errors may be favoured to other features as they stand out in some students’ writing. This bias may diminish the validity of the assessment in that students may not receive suitable scores or helpful feedback. Unlike holistic and analytic rating methods which are based on ‘standards’ of typical good writing, trait-based rating is an assessment method whereby a set of criteria is created to assess a certain writing task (K. Hyland, 2003; McNamara, 2000). Each task is assessed using a unique set of criteria. Trait rating uses “context-appropriate and task-appropriate criteria” (Hamp-Lyons, 1991, p. 247). It can be divided into primary- trait and multiple-trait ratings. Primary-trait rating is the assessment of one particular trait or feature related to the task without paying attention to other traits. It is regarded as more suitable to assess student ability in specific writing skills rather than their overall writing improvement (K. Hyland, 2003). Multiple-trait rating is the assessment of many different writing features based on a particular set of criteria designed to suit a

specific task. Multiple-trait and analytic rating are similar in that writing features are scored separately, but they differ in terms of their application. Multiple-trait rating can be applied to a specific task; whereas analytic rating is more generalised (Hamp-Lyons, 1991). Due to the fact that clear criteria in assessing writing can help maintain reliability in the assessment, rating scales need to be carefully designed or selected. The selection of criteria should be based on the assessment purpose, learner types, and the task (Grierson, 1995). Theoretically, Teasdale and Leung (2000) assert that criteria “should help make sense of the complex, inter-subjective and on-going teaching and learning processes, since the constructs underlying the assessment process should be those underlying the curriculum” (p. 178). The selection of rating scales may depend on the institutions, programs or the teachers themselves. In the context of higher education, it may be possible that teachers of the course have authority over the choice of rating criteria to be used in class. They may design their own rating criteria or incorporate rating criteria from various published rating scales that they find to be relevant or clearly understood (Teasdale & Leung, 2000). Thus, teachers may have the most influence over the validity and reliability of the assessment.

2.4.3 Raters Teachers do not only play an important role in designing or selecting writing tasks and rating scales, but they are also vital in the rating process. Because teachers are human raters with the variety of perspective that this implies, they may be considered to be the main cause of the variability. As Barkaoui (2007) states, “raters are the major source of variability in terms of the scores assigned and the frequencies of the decision- making strategies used, particularly for multiple-trait rating” (p103). McNamara (1996) also comments that raters “will vary in the standards they use and the consistency of their application of those standards” (p. 3). These two comments support the notion of teacher raters as ‘agents of assessment’ as termed by Rea-Dickins (2004), who suggests teachers make selections about their instruction based on their intuition, expertise and experiences. As mentioned earlier, the way in which teacher raters conduct their assessment is not clearly understood (Connor-Linton, 1995b). This scenario may be because assessing student writing is a complex and multifaceted activity (Hamp-Lyons, 1995),

which leads to great variability in teachers’ assessment practices and may harm the validity of the assessment. Variability is caused by a number of factors such as rater harshness in scoring, rater bias and their different backgrounds (A.D. Cohen, 1994; Wood, 1993). Raters can be different in terms of severity (or leniency) and randomness (error) in the rating. Some raters may be more lenient than others when rating a writing task. They may value and favour writing features differently. A. D. Cohen (1994) explored several studies from both first and second language writing (e.g., Connor & Carrell, 1993; Huot, 1990; McNamara, 1990; Rafoth & Rubin, 1984) and found that raters are likely to focus on grammar and mechanics more than they realise. Rater bias is also a factor that can affect the reliability of the assessment. Students’ handwriting, rating time of day, or teacher preferences towards students are all factors that may lead to bias (Wood, 1993). When employing rating criteria, raters may have different interpretations of writing quality and this may result in a diversity of score intervals given to the students’ writing. There may be variation in the score given to the same piece of writing by different raters, or to different pieces of writing by the same raters. Even though raters use the same rating criteria, they may focus on different features and assign different scores to the same writing (A. D. Cohen, 1994; McNamara, 1996). The issue of different emphasis on writing features is revealed in Milanovic, Saville, and Shuhong’s (1996) study. In their exploratory study, they aimed to investigate raters’ decision-making behaviours in terms of approaches employed in rating EFL compositions and the features which the raters focused on in their rating processes. They looked at various writing features; namely, the length of the composition, legibility, grammar, organisation, communicative effectiveness including coherence and factors affecting comprehension, style of writing, vocabulary, spelling, task relevance and punctuation. Sixteen raters, most of whom were either experienced teachers of English or writing assessors, were asked to assess 40 Cambridge examination scripts written by EFL students using the same set of criteria. Data were collected from the raters’ retrospective written reports, introspective verbal reports of the two writing scripts, and through group interview. Milanovic et al. (1996) discovered that teachers varied in their rating processes and focused on different features of writing. For higher proficiency level writing, raters focused more on vocabulary and content. By

contrast, the raters put greater emphasis on the ability to communicate and task relevance for lower proficiency level writing. Similar findings about rater variability when raters are experienced and the same set of rating criteria is utilised were found in Lumley’s (2002) study. Lumley (2002) examined the rating processes engaged in by four experienced raters while rating the writing in a large-scale test which was part of the immigration procedure in Australia. The raters were asked to rate two series of 24 pieces of writing using an analytic rating scale developed for various forms of test. They rated the first series silently and rated the second series while offering think-aloud protocols to describe their rating process. The results revealed the raters generally followed the same process of rating though they appeared to emphasise different elements of the scale descriptors. It was also discovered that even trained raters may have different rating behaviours. The above studies suggest that the use of the same criteria is not an indicator of reliability in writing assessment. Teachers’ different focus on writing features may be influenced by individual teachers’ expectation of good writing features. Their particular cognitive processes that cannot be easily observed may result in variability in their assessment practices. When raters attend to writing assessment, they can make their own decisions on “which features of the scale to pay attention to, how to arbitrate between the inevitable conflicts in scale wordings; and how to justify her [their] impression of the text in terms of the institutional requirements represented by the scale and rater training” (Lumley, 2002, p. 267). This incident is reflected in a number of research studies, as outlined next. In the context of disciplinary writing, Sakyi (2000) investigated the rating processes employed by six experienced raters using holistic criteria at a Canadian university to identify the factors that influenced their decisions making during the process. Data were collected using concurrent think-aloud protocols while the raters were assessing twelve papers written by students in different disciplines based on standardised holistic rating criteria. The findings from the study revealed four reading style types in relation to what the raters attended to while reading and the criteria employed to judge the quality of the texts. The four reading styles were: focus on errors, focus on essay topic and presentation of ideas, focus on raters’ personal reaction, and focus on scoring guide. Focus on errors referred to the raters who focused on identifying mistakes and made judgements based on the meanings of the text and error frequency.

Focus on essay topic and presentation of ideas emphasised the raters who read the criteria regarding content about the topic and idea presentation and commented directly on the texts rather than on the writer. Focus on raters’ personal reaction represented the raters who reacted to the content, and judged their comprehension of the texts and students’ language ability. This group of raters demonstrated biases and their own reaction influenced the assessment of the writing quality. Focus on scoring guide referred to the raters who directed most of their reading to the scoring criteria. Sakyi (2000) also found the raters’ personal reactions to the text content and the existence of grammatical errors and use of other language features influenced their decisions. Some of the raters in his study compared the quality of different writing to determine their marks. In coherence with Milanovic et al.’s (1996) and Lumley’s (2002) findings, experienced raters in Sakyi’s (2000) study varied in their rating styles even though they employed the same set of criteria. It is suggested in this study that raters’ different rating styles are formed by their individual long experiences of teaching and assessing writing. Without being trained, raters may fall back to their preferred rating styles and lead to variability in their assessment practices. However, even trained experienced raters are also found to vary in their rating behaviours. This phenomenon is evident in a recent study by Eckes (2008) of the standardised entry test responded by foreign students applying for higher education in Germany. He asked 64 experienced raters to rate the test writing section using nine criteria according to their importance in order to examine if rater types could be categorised. Many performance features such as fluency, train of thought and structure were covered in the criteria. He found experienced raters may fall into six classes or clusters, four of which are classified by distinctively important criteria and two of which are grouped according to less important criteria given by the raters. The first four types included Syntax Type, Correctness Type, Structure Type and Fluency Type. Syntax Type refers to raters who focused on vocabulary and syntax. Correctness Type refers to those emphasising the correctness of the text. Structure Type refers to raters who focused on the overall impression and task realisation including argumentation and completeness of the text. Fluency Type refers to raters who focus on text fluency. The final two types included Non-fluency Type and Non-argumentation Type. Non-Fluency Type refers to raters who gave far less weight to fluency. Similarly, the Non- argumentation Type regarded argumentation and train of thought to be less important.

Eckes (2008) also found the raters to have different views of criteria importance and that they perceived rating criteria differently. The Eckes (2008) study suggests that even experienced raters have different perceptions towards rating criteria. These perceptions may be considered as another cause of variability. The existence of different perceptions among raters can be confirmed by a study conducted by Cumming, Kantor, and Powers (2002). Their study aimed at investigating aspects of TOEFL essays rated by experienced raters in Canada. They found the qualities perceived as good writing in the university examination context varied among raters. Ten raters were asked to identify the three most significant aspects of writing. Nine raters identified rhetorical organisation, including introductory statements, development, cohesion, and fulfilment of the writing task as the most important aspects. Expression of ideas, including logic, argumentation, clarity, uniqueness, and supporting points as the most significant were also identified by nine raters. Seven raters gave priority to accuracy and fluency of English grammar and vocabulary, and only two raters emphasised the amount of written text produced. All the above reviewed research studies were conducted in the context that requires the employment of experienced raters and the same set of criteria. ‘Standards’ in test results seem to be essential in this context. However, the study findings indicate that variability in assessment practices exist with both trained and untrained raters. ‘Standards’ could not be universally achieved due to the variability caused mainly by raters’ personal values which may be formed by their experiences or individual preferences. This variability becomes even more evident when different types of rating criteria are employed. This issue is unfolded in Barkaoui’s (2007) study which investigated the impact of holistic and multiple-trait rating scales on rating EFL essays. Sixteen intermediate EFL university students in Tunisia were asked to write two essays on two different tasks, making a total of 32 essays. Four teachers were then asked to rate the written essays using holistic and multiple-trait rating scales. Two sets of four essays, including two essays from each task, were selected for think-aloud protocol while the rest of the 24 essays were for silent rating. The findings revealed holistic rating provided higher reliability; whereas high rater variation was found in multiple-trait rating. The explanation for this difference was that raters were likely to use different criteria when rating the essays. The main cause of the high reliability in using holistic rating scales in this study may result from the nature of this type of scoring method. A

single score given as the holistic rating may represent a composite of unstated scores raters give to different writing features. On the other hand, scores given to each distinct writing feature may be shown when raters employed multiple-trait rating, making rater variation emerge. It was also found in this study that the interaction between raters and rating scales may have impacted essay rating processes, especially the raters’ decision- making behaviours. It appears raters had to explain their decision-making more when using holistic rating than multiple-trait rating due to potential differences in the sets of criteria proportions in mind. This finding seems to confirm the conclusions from the above reviewed studies (e.g., Milanovic et al., 1996; Eckes, 2008) that rater variability is caused by raters’ preferences and personal values. In addition, there are tendencies for teachers from diverse backgrounds to have different perceptions towards good writing and thus tend to focus more on different specific writing features. Lukmani (1996) conducted an experiment in India to compare rating values of three groups of teachers: subject teachers, teachers of English, and English native teachers who were non-subject teachers. The three groups of raters assessed the students’ work impressionistically; that is, they employed their own marking criteria. The study focused on grammatical well-formedness and discourse coherence used in the texts written by the Indian students. It found differences in the criteria employed by different groups of teachers and in the language requirements for students in different fields. Teachers of English and English native teachers focused more on grammaticality while teachers of other subjects attended more closely to the coherence of the texts. Connor-Linton (1995a) discovered similar results in his study, which aimed at comparing the criteria raters from different backgrounds used in rating students’ writing. In his study, American ESL and Japanese EFL teachers were asked to rate ten pieces of writing on the same task written by Japanese EFL students. Five of the writing pieces were rated holistically while the other five were rated analytically. Raters also had to identify the three features they considered most influenced their rating. The study found the two groups of raters were similar in terms of overall scores and rankings of the students’ writing. Moreover, Connor-Linton (1995a) found both groups of raters provided similar reasons overall for their scores, but gave different reasons for allocating the same rank of writing. American raters stressed discourse features and

specific grammatical features within sentences; whereas Japanese raters emphasised accuracy of content, choice of words and grammar. Clearly seen in the above two studies, rater experiences can contribute to variability in the rating processes. This incident is also evident in Cumming’s (1990) and Hout’s (1993) studies. Cumming (1990) examined whether raters implicitly distinguished between the students’ writing ability and language proficiency while assessing their writing. The researcher also set out to describe the thinking processes employed by experienced and inexperienced raters in a Canadian university. Thirteen raters with different experiences were asked to assess 12 essays written by ESL students with two different groups of English ability using a holistic rating scale. Each individual rater conducted think aloud protocols while assessing the essays following their natural rating process. Cumming (1990) found that the two groups of raters implicitly separated the students’ language proficiency from their writing ability. Both groups of raters displayed significantly different decision-making behaviours. He concluded that compared to inexperienced raters, experienced raters appeared to utilise a wider range of criteria and to accommodate their interpretations and judgements of writing features more effectively. Hout (1993) examined how raters read and rated essays written by English native speaking university students in the US. He compared the rating process employed by eight raters who were practicing English teachers with a wide range of teaching experiences. They were classified into two groups: experts with holistic scoring training experiences, and novices without any experience. Both groups of raters were asked to assess four sets of essays written by undergraduate students in the writing course. Two sets were assessed while the raters talked aloud and the other two sets were assessed silently. The findings revealed that without rating criteria to rely on, the novice raters assessed the writing based on their intuition, leading to variability in their rating processes. They also exhibited varied behaviours in their rating sequences. For example, they assessed according to the sequence; ‘Expectations/Follow/Judge’ which showed they developed their own expectations of the writing and followed them through before making judgements about the quality of writing. In contrast, the experts did not display these behaviours. Rather, they relied on the scoring rubric when assessing the writing. Their practices demonstrated how the use of a scoring rubric enhanced their confidence in assessment.

Among the reviewed studies above, Hout’s (1993) study appears to reflect the actual situation of writing assessment practices in a classroom context regarding the employment of novice teachers. In a regular classroom context, teachers could be experienced teachers with many years of teaching or novice teachers without any experience. They may be autonomous in conducting teaching and assessment. This autonomy may allow room for them to adjust the required ‘standards’ in writing assessment practices to suit their individual views, causing variability in the practices. Indeed, rater variability can be the result of a number of factors as revealed in the review above. They appear to be related to the way in which teacher raters maintain reliability and validity of assessment. However, issues of reliability and validity are of greater concern in a large-scale assessment context than in a classroom context, and so a greater number of research studies on the rating process are conducted in the large-scale testing context. In these studies, researchers generally restrict some of the elements in the assessment process such as rater experiences, rating criteria and writing scripts as evident in the studies reviewed above. However, there still appears to be a lack of research studies which investigate these factors as they occur in actual assessment situations, particularly in regular classroom contexts such as the focus of this present study. As Davison (2004) remarks, “the role of the teacher-assessor in classroom-based assessment has received relatively little attention in the research literature in English language education” (p. 306). To maintain reliability and validity in classroom writing assessment, the three characteristics mentioned above play an important part. Among them, the rater has the greatest influence over the reliability of the assessment. Because raters in the classroom context are teachers, they are the key agents in designing or selecting the tasks and the rating scales to use. The variability between them, thus, can certainly affect their assessment practices. As many of the studies referred to above point out the existence of variability between raters is likely to result in large part from their own individual thoughts or perceptions towards writing and the rating process. Recent research shows that what teachers do is driven in complex ways by their social and cognitive perspectives. As Borg (2003) remarks, “teachers are active, thinking decision-makers who make instructional choices by drawing on complex, practically-oriented, personalised, and context-sensitive networks of knowledge, thoughts, and beliefs” (p. 81). Individual teachers have their own sets of knowledge, thoughts and beliefs and this uniqueness can

cause variability in their practices. Because consistency of assessment is required of teachers, research aiming to explore and understand what drives teachers’ perceptions about assessment and how these perceptions relate to their practices is valuable. It is also an area that is under-researched, particularly in relation to writing assessment. It is even rarer to find such studies located in higher educational institutions in Thailand. Thus, to understand the rating processes used by teachers as raters in this study, the area of teacher perceptions in relation to writing assessment practices of in-service teachers needs to be investigated. As the second major theoretical underpinning for this study, it will be discussed below.

2.5 Teacher Perceptions As stated earlier, teacher variability in writing assessment practices can be caused by many factors, one of which is teacher perceptions. Many researchers regard teacher perceptions as a highly influential factor in driving teaching practice (Ashton, 1990; Borg, 2003, 2006; Burns, 1996; Freeman, 1992, 2002; Richards, 1996, 1998). The notion of teacher perceptions is situated in the domain of teacher cognition, first developed in the field of cognitive psychology and later to be applied in the field of teacher education. It relates to “the unobservable cognitive dimension of teaching” (Borg, 2003, p. 81) which can be recognised as a “complex, dynamic, and contextualised” thinking system (Feryok, 2010, p. 277). Feryok (2010) indicates that the complexity of this thinking system lies in its range over diverse areas such as language and teaching practices, and that the thinking system can evolve across time and as a result of a number of influences such as teacher experience and context. Fang (1996) argues that teacher perceptions can take many forms, represented in their expectations of students’ performances and their principles of teaching and learning. In the educational context, teachers are regarded as active agents who bring with them their personal thinking about their work. As Freeman and Johnson (1998) remark, teachers “are individuals who enter teacher education programs with prior experiences, personal values, and beliefs that inform their knowledge about teaching and shape what they do in their classrooms” (p. 401). This proposition is in line with Richards (1998) who suggests the way in which teachers understand and conduct teaching is influenced by their’ implicit instructional principles or “their personal and subjective philosophy and their understanding of what constitutes good teaching” (p. 51). Therefore, teacher

perceptions are considered to be closely related to teacher practices. Individual teachers have their own personal thinking systems which “are founded on the goals, values, and beliefs they hold in relation to the content and process of teaching, and their understanding of the systems in which they work and their roles within it” (p. 51). They thus rely on these thinking systems when making decisions about their work (Richards, 1998). The relationship between teacher perceptions and practices is of increasing interest to researchers (e.g., Ashton, 1990; Burns, 1996; Fang, 1996; Phipps & Borg, 2009; Zheng, 2013). Fang (1996) reviewed a large body of early research studies in the area of reading/literacy in general education and found the teacher’s personal perceptions shaped their instructional practices in a number of reviewed research studies (e.g., Blanton & Moorman, 1987; Brophy & Good, 1974; Haste & Burk, 1977). Some of the studies also revealed inconsistencies in teacher perceptions and their classroom practices (e.g., Kinzer, 1988; Readence, Konopak & Wilson, 1991). These overall findings are now further discussed in relation to some of the key studies in the field of language teaching over the previous two decades. The relationship between teachers’ perceptions and their practices is described in Johnson’s (1994) study. Her study was one of the first to emerge in the field of language teacher research and aimed to investigate pre-service teacher beliefs about second language teaching and learning, and how their conceptions of teaching during the practicum were shaped by their beliefs. In the study, data from four pre-service teachers were collected in various ways, including journal entries, classroom observation and interviews. The findings revealed that the teachers’ beliefs were powerfully affected by their formal experiences as learners and their informal experiences in using language. The two factors also influenced their conceptions of how to conduct their teaching. In terms of their teaching practices, they were found to be different to what the teachers believed and thought they would do. One explanation put forward was that pre-service teachers had to overcome many dilemmas in teaching concerning teaching approaches and the flow of teaching. Their beliefs seemed to be able to shift after they realised their actual teaching practices were different from their initial beliefs. Thus, Johnson (1994) concluded that understanding what teachers think provides a way to understand how conceptions of practice lead to actual practices. As evident in this study, there is a strong interacting relationship between pre-service teacher beliefs and practices. On the

one hand, the teachers’ prior education and background knowledge play a considerable role in informing their practices. On the other hand, their current practices are the key factor in shaping their own beliefs. This interrelationship between teacher perceptions and practices is also revealed in relation to experienced teachers as demonstrated in the following studies. Breen, Hird, Milton, Oliver and Thwaite (2001) examined the relationship between the underlying teaching principles and classroom practices of ESL teachers in an Australian context. The data were collected from 18 experienced teachers using a reflective process between each individual teacher and a researcher. Over a period of five weeks, the researchers conducted two classroom observations which were followed immediately by interviews to elicit the teachers’ underlying principles of observed practices. Three additional interviews were conducted to gradually elicit the teachers’ pedagogic principles underlying their practices. The findings revealed the teachers had their own sets of personal principles that drove particular practices. The authors also found that certain principles cohered with particular practices and, in turn, certain practices were consistent with specific principles. They discovered that even though the teachers varied in their individual underlying principles and practices, there was also a collective pedagogy that involved their specific and different sets of principles. In an EFL context, Zheng (2013) conducted case studies of six experienced EFL Chinese secondary teachers to explore the relationship between their perceptions about EFL teaching and learning and their teaching practices. The teachers’ perceptions about teaching and learning were gathered from semi-structured interviews; whereas their classroom teaching activities involving, for example, grammar teaching and reading comprehension tasks, were identified through classroom observations. Teacher perceptions underlying their specific teaching practices were collected from stimulated recall interviews. The findings revealed the teachers’ perceptions were dynamic in that they changed over time according to their interrelationships with the changing context of educational reform. Zheng (2013) also found that the teachers’ practices were guided by multiple perceptions and the teachers’ perceptions may be interconnected in either a coherent or contradictory way. Nevertheless, the teachers’ choice to engage in certain teaching practice was normally driven by the ‘core’ perceptions they held. As seen in Breen et al.’s (2001) and Zhang’s (2013) studies, teacher perceptions have a great impact on classroom practices. Perceptions held by experienced teachers

are not static. Regardless of teacher experiences, they appear to be developed over time, possibly from their early years of teaching through an interactive process of thinking and action as suggested in Johnson’s (1994) study. They then later form individual teachers’ underlying principles that are reflected in their practices. Various factors including teachers’ educational and professional background as well as contextual factors seem to contribute to this dynamic change. Even though there exists a strong relationship between perceptions and practices, inconsistencies between them are also evident. A number of research studies have revealed this issue. In a study of teachers’ perceptions and practices in the implementation of the communicative approach in language teaching, Karavas-Doukas (1996) developed a five-point numerical Likert scale to collect data on the perceptions of Greek secondary teachers. The attitude statements in the questionnaire covered a broad range of principles related to a communicative learner-centred approach such as group work, error correction, grammar, student needs, and roles of teachers and learners. In the reported findings from the pilot questionnaire phase the responses obtained from 37 teachers revealed the teachers’ stated attitudes were not consistent with their practices observed by the researcher. The researcher indicated the discrepancy was the result of not integrating teacher prior knowledge into the introduction of the communicative teaching approach. Apart from addressing the existence of discrepancy between teacher perceptions and practices, this study also confirms the importance of teacher prior knowledge in shaping their perceptions and practices. In a more recent study, Basturkmen, Loewen, and Ellis (2004) examined the relationship between teachers’ perceptions and their practices with regard to incidental ‘focus on form’ in communicative language teaching. Drawn from a larger project involving 12 teachers from a school in New Zealand, three native English teachers were interviewed to obtain data on their beliefs about ‘focus on form’ instruction. The teachers’ communicative lessons were observed to reflect their actual practices. Following the classroom observation, the teachers were interviewed as well as provided with ‘cued response scenarios’ which required them to comment on their preferred teaching strategies in the given scenarios. Stimulated recall interviews were utilised to elicit the teachers’ decision making during the observed lessons. Findings from the study revealed the three teachers varied in their practices, even though they had the same teaching situation; that is, they taught students with the same level of proficiency

using the same communicative task. The researchers explained that the variability was caused by the teachers’ different teaching styles and the personal beliefs underpinning them. In terms of the relationship between their reported beliefs and practices, the findings also revealed the teaching practices by the three participants were generally inconsistent with their beliefs. One major discrepancy reported was about the teachers’ purpose of focus on form. They believed that form would be focused on when it interfered with comprehension of the students’ message. However, in the teachers’ actual practices they focused on form for linguistic accuracy. With regard to grammar instruction, Phipps and Borg (2009) also examined teacher beliefs and practices. Over a data collection period of 18 months the researchers interviewed three experienced EFL teachers teaching in a university in Turkey. They also observed their classroom practices. The interviews were conducted every four months, with each teacher participating in four interviews to obtain data on the teachers’ beliefs about their classroom practices and the development of their beliefs. Classroom observations were conducted on three occasions, each of which was carried out every three months. The findings revealed teacher beliefs and practices were generally consistent. However, there also existed various tensions between beliefs and practices, especially in relation to grammar presentation, practice and oral group work. In addition, Phipps and Borg found contextual factors such as the expectations and preferences of students, and classroom management issues influenced the discrepancies between teachers’ reported beliefs and actual practices. The above three studies offer different findings on the relationship between teacher perceptions and practices. While Basturkmen et al.’s (2004) and Karavas- Doukas’s (1996) studies offer coherent findings about existing inconsistencies, Phipps and Borg’s (2009) reveal congruence of the overall relationship between the two constructs. These different patterns may result from the research designs employed in each study. Regarding the three studies, inconsistencies may occur in cross-sectional data collection because it provides findings specific to teachers’ perceptions about particular aspects of their practices. This issue is also evident in Phipps and Borg’s (2009) findings on teachers’ specific classroom practices (e.g., grammar presentation). However, when teacher practices are longitudinally observed, the overall pattern of relationship between teacher perceptions and practices could be found to be more consistent as demonstrated in Phipps and Borg (2009).

As revealed in the reviewed studies above, the teachers’ perceptions and practices were likely to be highly varied and dynamic. However, their classroom practices were powerfully shaped by their perceptions (Borg, 2006, 2012; Nunan, 1992; Richards, 1998; Richards, Gallo, & Renandya, 2001; Westerman, 1991). As suggested in numerous studies, their perceptions were informed by a considerable range of influences relating to prior educational knowledge, professional experiences and contextual factors (Ashton, 1990; Barnard & Burns, 2012; Freeman & Johnson, 1998; Richards, 1998). Among these factors, Freeman and Johnson (1998) indicate that teacher training is powerfully impacted by all aspects of the teacher’s prior knowledge which “is constructed through teachers’ experiences in and with students, parents, and administrators as well as other members of the teaching profession” (p. 401). The powerful impact of prior education is also reported in Johnson’s (1994) study reviewed earlier. This study aimed to examine pre-service teachers’ beliefs about second language teaching and learning during their teaching practicum. Based on the teachers’ reflections on their practicum experience, Johnson found that images of the teachers’ prior formal classroom learning were reflected in images of their role as teachers, their instruction, and perceptions of their teaching practices. For example, the teachers employed a teacher-centred teaching approach even though they wanted to adopt a student-centred approach. Their decisions were based on their images of traditional teachers as knowledge sources and authority figures from their prior classroom experiences. The impact of prior education also influences the teaching practices of in-service teachers. Hayes (2005) discovered the three in-service teachers in his study employed teaching methods which reflected the ways in which they were taught in school. These studies confirm that teachers often construct their teaching knowledge through their experiences as learners. In relation to professional experiences, studies have shown that teacher beliefs and practices are influenced by their professional training or development programmes. In Flores’ (2001) study of the relationship between beliefs and self-reported practices of bilingual education teachers, teachers gathered evidence of effective teaching methods from their own instructional experiences. Having gained extensive experiences teaching former students, the teachers could justify their current students’ proficiency level. A similar finding was found in Borg’s (1998) study. Borg’s investigation of classroom practice during grammar teaching by an expert teacher found the teacher developed his

beliefs through his prior professional training. He constructed his knowledge of communicative methodology and a student-centred approach during the training and reflected them in his practice. Apart from the professional training, Barnard and Burns (2012) assert that teacher knowledge and beliefs are also shaped through their contact with other people both in their professional and personal lives. They learn from their previous and current professional experiences as teachers through their interaction with students and colleagues in their professional context. They can also be influenced by authority figures or mechanisms related to their teaching context such as school principals and examination boards, and by people they have contact with in their personal life such as family and friends. Contextual factors are also regarded as a powerful influence on teaching practice. The influence of contextual factors on teaching practice is reflected in Nunan’s (1992) study. He examined the factors that drove teacher decision making processes while teaching. Nine teachers of English, both inexperienced and experienced, were observed while they were delivering a lesson. The findings revealed that teacher decisions were dominated by the demands of classroom management and organisation. Similarly, Johnson (2006) remarks that the most influential factor shaping what teachers do is the context of their operational work. She puts it that teacher knowledge is developed from the knowledge of social activities controlled by social norms. In the teaching context, Richards (1998) includes a range of contextual factors both within and outside the classroom such as language teaching policies, community factors, sociocultural factors, school culture, learner age, learning factors, teaching resources and testing factors. A systematic view of the relationship between context and teaching practices is provided by Burns (1996). According to the researcher, teachers develop their perceptions by way of the interconnection between three interrelated contextual levels: incorporating the institutional context, classroom context and instructional context. The institutional context, which is the broadest level, involves particular organisational philosophies and forms of ‘institutional culture’ that frame teacher perceptions of their instructional work. The classroom context includes the teacher’s personal thinking system they have developed about language, nature and the characteristics of the learners and learning. These thinking systems influence teacher decision making about the way in which they conduct their teaching in the classroom. The instructional context

represents the most specific contextual level where teachers focus on particular areas of distinctive teaching activities in class. The teacher’s beliefs based on the three contextual levels interact with each other and are regarded as fundamental to their classroom practice. Because a teacher is generally recognised as a thinker or decision maker, teaching can be seen as a ‘thinking activity’ (Richards, 1998). This notion of teaching is developed through the teacher’s active engagement in constructing their personal theories of teaching. They have developed tacit knowledge on teaching and learning and as well as theories on classroom practice from their formal training (Richards, 1998). Moreover, it seems what teachers already know may influence their teaching more than what they have learned in their formal education. As Freeman (1992) remarks, “in the greater scheme of their professional lives, formal education in teaching probably has only a minimal impact on what teachers’ [sic] do in the classroom” (p. 4). However, Borg (2011) argues that if teacher education as part of professional experience has an impact on teacher beliefs, it also influences teaching practices. In his longitudinal study investigating the way in which English language teachers’ beliefs were impacted by an intensive eight-week teacher education programme, Borg collected qualitative data from six in-service teachers who attended the programme. For each teacher, he conducted six interviews which were distributed across the programme. He found that the impact of the training on the teachers’ beliefs varied considerably. Three of the six teachers showed a stronger impact on their beliefs than the other three. The teachers showing an impact acknowledged the beliefs underlying their practices and were able to give them expression; whereas the other three teachers recognised the impact on their beliefs to a lesser extent. As reviewed so far, influential factors that impact on teacher perceptions echo what Borg (2006) found in his review of various studies (e.g. Allen, 2002, Farrell, 2003; J.C. Richards, 1996; Richards & Pennington, 1998) conducted in the field of teacher education. These studies confirm that practices of both novice and experienced teachers are influenced by many factors, including the teacher’s school experience, professional education, and contextual factors. According to Borg (2006), the research shows that experienced teachers have more established knowledge of teaching and it is on this knowledge that they rely for their own teaching practices. For example, experienced teachers have a profound understanding of lessons and know how to deliver lessons

appropriately. They also have clear conceptions of what classrooms and students are like from their extensive teaching experience. This knowledge may assist them to predict the likely nature and characteristics of students before they actually meet them. In contrast, novice teachers lack this form of knowledge. Moreover, novice teachers appear to be influenced more powerfully by contextual factors in schools and classrooms. Even though they are equipped with knowledge and principles from their prior teacher education, they can still divert from their beliefs and principles during actual teaching practices due to a number of contextual factors such as encountering large classes, the limited language ability of the students, and pressures from examinations. Professional relationships among teachers and their concerns over how to manage learning may also change their principles of teaching from what they have learned. In the case of experienced teachers, Borg discovered that contextual factors strongly influence what they think and do. Teacher education seemed to be the main source for change to the beliefs of experienced teachers even though the beliefs are influenced by personal experience. Moreover, what teachers believe and what teachers do may differ, primarily because “instructional practices are defined through complex interactions among teachers’ cognitions and situational factors both inside the classroom and in the wider institutional and social context” (p. 107). Drawing on his review of a wide range of research studies of how teacher classroom practices are influenced by social, professional and personal factors, Borg (2006) proposed the model of teacher cognition which will frame the analysis of factors influencing teacher perceptions driving their writing assessment practices in this study. The Borg model will be presented in Section 2.7 below.

2.6 Teacher Perceptions and Classroom Writing Assessment Because teacher thinking systems are now recognised as being an influential factor in motivating their practices, the area of teacher thinking has been widely researched over the last two decades. Most studies have focused on an investigation of teaching practices in general (e.g., Farrell & Bennis, 2013; Feryok, 2010; Freeman & Richards, 1993; Gatbonton, 1999; Golombek, 1998; K. E. Johnson, 1999; Numrich, 1996; Popko, 2005), specific pedagogical domains such as grammar instruction (e.g., Basturkmen, et al., 2004; Borg, 1999a, 1999b, 2005; Johnston & Goettsch, 2000; Phipps, 2010; Phipps & Borg, 2009) and decision-making processes (e.g., K. E.

Johnson, 1992; D. B. Smith, 1996). Very few research studies can be found, however, which link teacher perceptions with assessment practices, especially writing assessment, which is the focus of this research. Among the few studies in the area of assessment in general, M. Yin (2010) explored teachers’ perceptions when conducting a classroom language assessment in an EAP context in a British university. Drawing on her definition of classroom language assessment as; “the collection, synthesis, and interpretation of information about student language use in classroom activities to aid the teacher in decision-making” (p. 178). M. Yin also included the teacher’s observations of the students during teaching activities as one form of assessment in her study. The classroom pedagogy of two teachers was observed across one semester. After the observation of every, or every other, class, the teachers were asked to recall their thoughts during classroom assessment practices as they were revealed to them in video excerpts selected by the researcher. The findings showed the two teachers shared a number of common assessment practices such as reading students’ written scripts and observations of their group discussions. However, they differed in the time they allotted to questioning and in their class observations during student group work. It was also discovered that the teachers held a range of diverse but interconnected cognition types which were divided into two groups: strategic cognitions and interactive cognitions. The strategic cognitions included the teacher’s pedagogical approach and their beliefs about language learning, and contextual factors such as classroom parameters and the course syllabus they were expected to teach. The interactive cognitions included factors such as the assessment principles they adopted and the teachers’ perceptions of the students’ performances in class. M. Yin (2010) stated that the two types of cognitions influenced different phases of the teachers’ assessment practices; that is, the strategic cognitions mainly impacted the planning of lessons while the interactive cognitions primarily influenced teachers’ decisions during the class time. In a study exploring teacher assessment knowledge and practices, Xu and Liu (2009) investigated a Chinese college teacher’s stories of her experience in the reform of assessment in China. Through a narrative inquiry of the teacher’s account, the researchers found that the teacher’s assessment knowledge was constructed by the teacher’s previous experience, her current practices and her future plans for practices. Moreover, the relationship with her colleagues also impacted the teacher’s practices.

This study thus concluded that teacher knowledge is “a highly complex, dynamic, and ongoing process” (p. 508). The two studies confirm other earlier reviewed research in teacher cognition that teacher practices are influenced by their perception. Many factors both internal and external to teachers—their prior experience and contextual factors within their working contexts—are found to impact their perceptions and shape their practices in various aspects. These factors are found to play a predominant part in teacher practices. They are illustrated in the following studies. Focusing on implementation and design of assessment in Iranian secondary schools, Saad, Sardareh, and Ambarwati (2013) distributed an open-ended questionnaire to 35 secondary school teachers of English in Iran to explore their beliefs on and investigate their roles in students’ assessment. They found that the teachers varied in their beliefs on assessment. While most teachers perceived that teacher assessment was useful to develop student learning, some teachers had an opposite view—mentioning assessment was useless. The large gap in their views may largely result from the status of assessment in the Iranian secondary school context. The researchers explained in their study that assessment design was not the teacher task; instead, it was conducted by testing experts. Thus, most teachers were not included in the assessment design process and some of them were not aware of what the students were assessed. Thus, some teachers might not clearly see the importance of assessment in their teaching, leading to deny its usefulness. This scenario suggests that contextual factors such as tests and accountability results of assessment have a strong impact on the teachers’ perceptions. In relation to the area of writing assessment, studies regarding feedback have been largely researched. The focus of the studies lies in the relationship between teacher thinking and practices. In Lee’s (2003) study of teachers’ error feedback on students’ writing, teacher thinking, their practices and the problems they encountered in their practices were investigated. Teachers at both a university and a secondary school in Hong Kong were asked to complete a questionnaire, with some of the questionnaire respondents selected for follow-up interviews. Based on the 206 questionnaire responses and the follow-up interviews, a gap between the teachers’ beliefs and their reported practices was revealed. Lee found that the teachers’ responses to her questions and what they actually did in the classroom were different. She asserts that their real practice may have resulted from the impact of contextual factors which she explains as

“the daily and pressing demands of students, parents, panel chairs, principals, etc.” (p. 226). In an EFL Japanese classroom context, Mori (2011) explored how teacher thinking and knowledge shaped teachers’ practices in corrective feedback during classroom teaching activities. In her preliminary study, Mori focused on two experienced teachers: a Japanese teacher and a native speaker of English teacher. Over a period of four months, Mori collected data from multiple research methods including classroom observation, her field notes reflecting the classroom observations, interviews, and relevant documents. The data were analysed to explore the teachers’ general views about corrective feedback. The findings revealed their practices were driven by their awareness of contextual factors (i.e., the particular school environment and the nature of schooling in Japan), and the knowledge of language teaching and learning they gained through their personal experiences as learner and teacher. The teachers’ goals were to gradually establish values they felt were neglected in their cultural context of teaching such as student confidence and how to communicate well. Clearly seen in the above two studies is a great emphasis on the impact of contextual factors on teacher practices. Based on these studies, contextual factors may be considered to be the most powerful influence that shapes teachers’ practices in an actual classroom context. While other internal factors such as teachers’ prior experiences could form their tacit principles driving their practices, contextual factors are likely to come into play when they actually employ the principles they hold. However, it is evident that teacher practices can change according to a shift in their principles. This issue is illustrated in Min’s (2013) study. Min (2013) investigated her reflective beliefs and practices in giving written feedback to Taiwanese university students’ writing. Employing a self-reflection journal, learning log and written comments on students’ writing over one semester, she discovered four principles underlying her practices at the beginning of the semester. She found she clarified students’ purpose, identified writing problems, explained them and suggested correction. Throughout the semester, these principles changed mainly because of her wider knowledge gained through the consultation with relevant literature. She then changed from giving feedback based on her assumption of students’ intentions to attempting to understand students’ intentions. This study serves to confirm the notion of consistency between teacher beliefs and practices and reveals the dynamic nature of

teacher beliefs. It clearly addresses the teacher’s immediate learning experience as a very powerful factor driving change in her practices. Exploring beliefs and practices of Chinese teachers in students’ peer feedback in EFL writing classroom, Yu (2013) interviewed 26 university teachers from seven cities in China about their views on the implementation of peer feedback in their writing classes. The findings highlighted that teachers varied in their employment of peer feedback. The researcher explained that the variability resulted from teachers’ different beliefs and individual teaching contexts. Consistencies between teacher beliefs and practices were generally discovered. However, some discrepancies were also detected and teachers’ practices could change regarding the change of their beliefs. As in other earlier reviewed studies, this study suggests a close relationship between teacher beliefs and practices. As reviewed so far, research studies that relate teacher perceptions and assessment practices focus primarily on the exploration of what shapes teacher perceptions and how teachers conduct their practices. Some studies also address how teacher perceptions shape their practices and include the investigation of influential factors on the practices. Although these studies offer different findings, all of them demonstrate that teacher thinking systems play an important role in pedagogical practice in terms of both teaching and assessment. Teachers’ previous experiences as learners and teachers, as well as contextual factors relating to the specific educational environments in which they work, have a strong impact on teacher perceptions. As Davison (2004) remarks, teacher beliefs, attitudes and assessment practices may be shaped by the diversity of educational and cultural contexts as well as what the teachers assume about the contexts. Moreover, teacher perceptions seem to powerfully influence what teachers do in the classroom. However, discrepancies between teacher perceptions and the teachers’ actual practices may also exist for reasons which may also be contextually determined. Regarding the methods employed, most of the reviewed studies reported teacher practices gathered from teachers’ self reflected views rather than their actual practices. Even though this method may capture teacher practices based on their own accounts, it is unlikely to unfold what teachers actually do during their assessment practices. Thus, in order to understand the actual practices of writing teachers, this study will employ

multiple methods to illustrate what teachers actually do in their rating processes. These methods will be further discussed in Chapter 3.

2.7 Conceptual Frameworks As demonstrated in the literature reviewed above, teachers in a regular classroom context are important agents who conduct writing assessment influenced by their perceptions. Many factors including teachers’ prior personal and professional experiences as well as contextual factors are found to drive teacher practices. To explore these issues in more details, this study employs two conceptual frameworks: Davison’s (2004) cline of assessor types and Borg’s (2006) language teacher cognition model. The Davison cline of assessor types will frame the way in which teachers’ writing assessment practices are analysed; whereas the Borg teacher cognition will come into play in order to reveal factors influencing teacher perception. These two frameworks are demonstrated below.

2.7.1 The Davison (2004) cline of assessor types The Davison cline of assessor types was developed trough a comparative study of the assessment beliefs, attitudes and practices of teachers in Australia and Hong Kong conducted by Davison (2004). Twelve secondary teachers from each context were asked to assess six written texts based on their usual practices. Applying questionnaire, verbal protocols, interviews, and teacher self-report, Davison found that teachers in both contexts were diverse and could be grouped into a cline of five types of assessors: assessor as technician, assessor as interpreter of law, assessor as principled yet pragmatic professional, assessor as arbiter of ‘community’ values, and assessor as God. The five types differ in terms of the teacher assessor’s views of the assessment task, the assessment process, the assessment product, inconsistencies in judgement and assessor needs. The cline of assessor types identified by Davison is shown in Table 2.1.

Table 2.1 A cline of assessor types (Davison, 2004, p. 325)

Butler (2009) applied Davison’s (2004) framework of teacher assessment beliefs, attitudes and practices to investigate the way in which teachers observe and assess primary students’ foreign language performances in a Korean classroom context. Twenty-six primary and 23 secondary English teachers were asked to watch video recordings of student group activities and assess their performances. Individual teachers had to score each student’s performance according to a holistic rating scale. In addition, they were requested to identify the performance traits they had emphasised in their assessment. The findings revealed the two groups of teachers focused on similar traits. However, the way they interpreted the traits and the way in which they arrived at their judgements were different. As stated by Butler, the student’s behaviours were the factor that influenced the teachers’ judgements. Drawing on Davison’s cline of assessors to frame the variability of the assessment practices demonstrated by the two groups of teachers, Butler found the primary teachers were oriented towards ‘assessor as God’ and ‘assessor as the arbiter of “community” values’; whereas the secondary teachers’ practices tended towards ‘assessor as technician’ and ‘assessor as the interpreter of the law’. The above two studies demonstrate the variability in assessment practices of teachers in two contexts: two different countries in Davison’s (2004) study and two educational levels in Butler’s (2009) study. Davison’s cline of assessors provides a useful tool to frame teacher variability in assessment practices as it illustrates a wide range of practices. In this present study, the focus was on teachers in different universities across Thailand. It is likely that their writing assessment practices will vary in ways that parallel the findings of these studies. Therefore, the cline developed by Davison offers a suitable tool to assist understanding of teacher writing assessment practices in the context of this study.

2.7.2 The Borg (2006) language teacher cognition model As discussed in Section 2.5, teacher practices in the classroom are influenced by teacher perceptions driven by multiple factors including social, professional and personal factors. These influences are illustrated by the Borg (2006) model of language teacher cognition in Figure 2.2 below.

Figure 2.2 Elements and processes in language teacher cognition (Borg, 2006, p. 283)

The figure reveals that teacher cognition is mediated by her or his experiences both as learners and as teachers (through schooling and professional coursework), and the classroom practices situated in contextual factors surrounding the classroom. In turn, teacher cognition can influence teaching experiences throughout the teaching career. According to Borg’s model, teacher cognition and the teacher’s classroom practices are mutually informing; contextual factors surrounding their classroom practices play a key role in mediating the way in which teachers implement their practices in a way that resonates with their perceptions. Moreover, it can be said that the model proposed by

Borg synthesises the influencing factors also identified in the studies reviewed in this section. As this present study focuses on the writing assessment practices by teachers as they emerge in a regular classroom context, it is important to consider how the practices may be influenced by their perceptions and other influential factors in the social, institutional and cultural context. Teacher conceptualisations underpinning assessment practices include the purpose of assessment, its relation to learning and teaching, the role of the teacher in relation to assessment, and the teachers’ pre-existing beliefs about the students and texts they are assessing. To assist this exploration, therefore, it is worth employing Borg’s (2006) concepts of the elements and processes in language teacher cognition to frame the emerging influential factors.

2.8 Summary This chapter has reviewed the literature pertaining to the two major areas of classroom writing assessment and teacher perceptions relevant to the aims of this study. Key issues in both areas have been discussed and the conceptual frameworks guiding this research have also been highlighted. In the Thai context, there have been increasing calls for comparability of institutional standards. However, one potential problem facing the higher education sector in Thailand is the quality of teachers teaching English (Punthumasen, 2007). This problem may result in teacher variability in assessment practices and may harm the comparability of assessment practices among different institutions. Because human raters are a primary reason for variability in assessments, exploring their practices and cognitions is important in order to understand the nature of the variability. As teacher practices are driven by their perceptions, understanding what teachers think and believe may help us to understand what they do in their actual practices. The awareness of multiple factors influencing their perceptions may also assist future development in their writing assessment practices. This study employed a qualitative research approach to investigate teachers’ rating processes and their thinking systems in the Thai context. The detailed methodology for this research study will be discussed in the next chapter. The methodological paradigm and the design of the study will be outlined, together with the research questions, the selection of participants and the procedures for data collection and analysis.

CHAPTER 3 RESEARCH METHODOLOGY

3.1 Introduction The literature review in the previous chapter identified key areas of investigation that relate to this study. In particular, writing assessment and its characteristics in regular classroom contexts – the focus areas of this study – were discussed, along with the notion of teacher perceptions and how they may influence classroom assessment practices. Given that this study aims to explore teachers’ perceptions of assessment and examine their actual assessment practices when assessing students’ classroom writing tasks, it is necessary to collect qualitative data from the participants’ natural workplace setting. Thus, in this study qualitative research methods are adopted to yield rich qualitative data and to “demonstrate the complexity, texture, and nuance involved in how individuals and groups experience themselves and their worlds” (Kamberelis & Dimitriadis, 2005, p. 17). This chapter will start by outlining the aims of the study and the associated research questions. Then, the particular research paradigm will be addressed including the ontology and epistemology underlying the qualitative research approach. Moreover, the characteristics, strengths and weaknesses of the research paradigm will also be presented. The questionnaire and case study approach which are adopted in this study will also be discussed and justified. Then, the chapter will move on to the discussion of the research design of this study. Details of the research methodology, data collection procedure and data analysis methods will be presented.

3.2 Research Questions As mentioned in Chapter 1, this study aims to explore teachers’ views of writing assessment in Thailand and to investigate teachers’ rating processes and the criteria they employ. Various research methods have been utilised to explore the following research questions: 1. How do teachers of writing in Thai public universities view their assessment of students’ writing? 2. How is writing of a specific text type assessed by Thai teachers in different public universities? What influences their selection of the rating criteria?

3. How do teachers’ views of writing assessment influence their actual rating practices?

To answer the research questions most effectively the study was divided into two phases. The first phase aimed to explore teachers’ views on writing assessment in Thailand by obtaining a global picture of how teachers in universities across Thailand view writing assessment in relation to good writing and assessment practices. The second phase aimed to extend the findings from the first phase through an in-depth investigation of the actual assessment practices of four teachers employed at different universities in Thailand. A questionnaire was used during the first phase to collect exploratory data of the views of assessment held by the teachers. A case study approach was then adopted in the second phase of the study in which semi-structured interviews, think-aloud protocols, stimulated recall interviews, and documents were used to collect data on the individual teachers’ actual practices in their respective universities. Discussions of the implementation procedures for the questionnaire and case studies will be presented below. Also, the major characteristics of the qualitative research paradigm employed in this study and the related research methods will be elaborated.

3.3 Qualitative Research Paradigm Qualitative research can be loosely defined as “empirical information about the world, not in the form of numbers” (Punch, 2009, p. 87). Similarly, Dörnyei (2007) defines qualitative research as a type of research which “involves data collection procedures that result primarily in open-ended, non-numerical data which are then analysed primarily by non-statistical methods” (p. 24). Considering its function, Merriam (1998) regards qualitative research as “an umbrella concept covering several forms of inquiry that help [the researcher] understand and explain the meaning of social phenomenon with as little disruption of the natural setting as possible” (p. 5). As such, the perspectives of those people in the actual context of study are vital. Indeed, “How people interpret their experiences, how they construct their worlds, and what meaning they attribute to their experiences” (Merriam, 2009, p. 5) are the main points of focus in this study. These focal points reflect the ontology or the nature of reality, and the epistemology or the nature of knowledge underlying the qualitative research paradigm. A key philosophical assumption underlying this paradigm is that “reality is constructed

by individuals interacting with their social worlds” (Merriam, 1998, p. 6). Study of this reality informs constructivism or interpretive knowledge from which researchers construct knowledge about social phenomena based on “multiple realities, or interpretations, of a single event” (Merriam, 2009, p. 8). The purpose of this form of research is to construct abstraction, concepts, hypotheses and theories in order to describe, understand and interpret a phenomenon according to human perspectives rather than to test a theory (Cohen, Manion, & Morison, 2007; Merriam, 2009). Qualitative research has many characteristics covering research design, sample, data collection, data analysis and how findings are reported (Merriam, 2009; Mertens, 2005). Key issues in research design are flexibility and emergence. The design may evolve in a particular direction or be refined as the research proceeds. This flexibility includes the evolution, change and refinement of research questions and the areas of research focus during the research process. The way in which the research develops is based on the data collected. A wide range of data is obtained through a number of sources such as recorded interviews, texts and images, and this leads to data analysis through words. Data are collected in a natural research setting. Social phenomena in qualitative research are holistically viewed as they naturally occur. No intervention or manipulation from the researcher takes place in the setting because the context of what is observed is very important. Indeed, given qualitative research concerns human behaviour in a natural setting, individual subjective opinions, experiences and feelings are essential. Each participant’s subjective understanding of the issue or event is very important as it reflects his/her interpretation of their experiences and behaviours. Furthermore, context may be regarded as an influential factor in the evolution of events and the participants’ behaviours. Qualitative research data are normally in non-numerical form. Thus, a small sample size may yield rich and thick data leading to locally and contextually interpretive analysis. Researchers can then propose subjective interpretations of the same set of data and report findings based on their interpretations (L. Cohen, Manion & Morrison, 2007; Dörnyei, 2007; Wiersma & Jurs, 2009). Moreover, in qualitative research the researcher is considered the main research instrument for both data collection and analysis as he/she has to build a holistic overview of the situation as well as gain in-depth responses from the participants in the study (Merriam, 1998; Punch, 2009).

Considering these characteristics, the present study was developed based on the qualitative research paradigm. The main aim of the study was to understand the phenomenon which is the classroom writing assessment practices of teachers in Thai public universities. Teachers’ assessment practices may differ due to a range of factors such as their unique perceptions of assessment or the influence of specific contextual factors. In turn, one way to gain insight into the diverse assessment practices employed by teachers is to go into the field and collect data as the assessment practices unfold in their natural setting. In this study, to obtain data on actual assessment practices, the participant teachers were observed while conducting assessment of their students’ written work. In addition, the researcher sought to mediate the meaning of the assessment phenomenon by also collecting data on the participants’ perspectives of assessment (Merriam, 1998). Qualitative research entails various strengths and weaknesses. In terms of strengths, it is generally exploratory in nature and this approach may help to explain complex situations. Moreover, because the qualitative research design is flexible, changes may be accommodated when unexpected problems occur during the data collection process. These changes may also be reported and discussed to improve the validity of the research. Even when the sample size is small, rich data can be obtained (Dörnyei, 2007). Wiersma and Jurs (2009) also assert qualitative research is “the best way we have of getting the insider’s perspective, the ‘actor’s definition of the situation’, the meanings people attach things and events” (p. 294). However, qualitative research has a number of weaknesses. There is the risk that researcher biases and idiosyncrasies will contaminate the results of the study given the data are interpreted mainly by researchers. However, Patton (2002) argues that as qualitative research seeks to establish insights into phenomena based on the participants’ inner perspectives, the researchers’ views are important to capture the depth of understanding though their engagement with the participants under study. To avoid introducing bias during data collection and data analysis, Patton suggests the researcher should look to understand the phenomenon as it is, without passing judgment. Triangulation and respondent validation may also be used to avoid researcher bias. Triangulation includes obtaining various interpretations on the data and the use of multiple methods of data collection. Respondent validation permits research participants

to access the data and check for data authenticity to correct researcher bias (F. Hyland, 1998). Another issue is the use of a small sample size as it is often criticised for not enabling generalisation of the results. Nonetheless, it may be argued that generalisation is not the primary aim of qualitative research; rather, the main aim is depth of understanding of the phenomenon under study. Moreover, the use of a small sample size may allow the researcher to gain a wealth of detailed information about the phenomenon (Patton, 2002). Furthermore, the transferability of research findings can be achieved through the researcher’s full, rich and sufficiently detailed accounts of the study so that readers can evaluate its applicability to their own contexts (Seale, 1999). Another drawback in qualitative research is that data are obtained through written or spoken discourse. As such, the collection or interpretation of the data may be ‘time- consuming and labour-intensive’ (Dörnyei, 2007, p. 42). Furthermore, the term qualitative research paradigm covers numerous approaches. As Punch (2009) asserts, “qualitative research methods is [sic] a complex, changing and contested field—a site of multiple methodologies and research practices…Four aspects of this diversity concern paradigms, strategies and designs, approaches to data, and methods for the analysis of data” (p. 115). In order to obtain a full description of teachers’ writing assessment practices in Thai public universities collectively and to gain insight into their thinking processes and their actual assessment practices when assessing students’ writing tasks in the actual context, this study employs multiple methods including questionnaire and case study approach. Each of them is further discussed below.

3.3.1 Questionnaire A questionnaire is one of the most-widely employed instruments to collect data on respondents’ views, beliefs or attitudes (Borg, 2012; Marshall & Rossman, 2006). J.D. Brown (2001) defines a questionnaire as “any written instruments that present respondents with a series of questions or statements to which they are to react either by writing out their answers or selecting from among existing answers” (p. 6). A questionnaire can be used to obtain a wide range of data including factual data about respondents’ personal backgrounds and demographics, behavioural data about respondents’ actions, life-styles, habits and experiences, and attitudinal data about what respondents think or believe (Dörnyei, 2003).

In this study, the questionnaire was adopted as one of the research methods because it can be used to collect a large amount of data in a short period of time. It can also be used to gather data from “a variety of people in a variety of situations targeting a variety of topics” (Dörnyei, 2007, p. 115). The combination of closed- and open-ended questions may facilitate the elicitation of a wide array of topics in relation to teachers’ writing instruction and assessment practices. Moreover, a questionnaire can assist with the collection of seemingly accurate data from respondents. While closed-ended questions offer practicality in the ease and speed that respondents answer the questionnaire, as well as support the comparison of data, open-ended questions may elicit more creative and varied responses from the respondents (Nunan & Bailey, 2009). The use of closed-ended questions may also enhance convenience in the process of data analysis. Furthermore, questionnaires are recognised as data collection instruments which enable the anonymity of the respondents to be secured (Dörnyei, 2007). As such, the respondents can feel more comfortable answering the questions with honesty and in detail. However, the questionnaire instrument is not without limitations. For instance, the self-report nature of the questionnaire may lead respondents to refrain from providing truly accurate responses as they look to emphasise positive factors and downplay negative ones (Nunan & Bailey, 2009). There may also exist the possibility that respondents respond to the questions according to what they think they know about the researcher’s expectations of the questionnaire results (Nunan & Bailey, 2009). They may, therefore, try to please the researcher by providing responses that show the respondents in a positive light. These issues may therefore result in the questionnaire data not accurately representing the respondents’ actual practices (Dörnyei, 2007). Regarding the exploratory purpose of this study, the questionnaire is considered a suitable data collection instrument as it can help to generate an overview of each teacher’s background, as well as the teachers’ collective views on writing assessment practices in Thai public universities. However, the questionnaire’s inability to generate data to adequately explain the uniqueness and/or complexity of each teacher’s assessment practices meant developing a case study on each teacher was required to fulfil the aim of this study. The case study approach is discussed in the next section.

3.3.2 Case study approach A case study approach to research involves a detailed investigation of a person, place or topic in order to understand it more deeply. In-depth information about the case is gathered in its natural setting in order to understand the complexity and actual context of the case (Punch, 2009; van Lier, 2005; R. K. Yin, 2003). Furthermore, in a case study the complexity of events, relationships between people and other factors can be investigated and reported in a unique way (L. Cohen, Manion, & Morrison, 2007). A case study approach can be employed in situations where a phenomenon cannot be separated from contextual conditions and other research methods are unable to explain them (R. K. Yin, 2003, 2009). Dörnyei (2007) identifies a case under study as any “single entity with clearly defined boundaries” (p. 151). People, a program, an institution, an organisation or a community may be considered as a case worthy of investigation. The definitions of ‘case’ provided by R. K. Yin (2003) and Dörnyei (2007) appear to be slightly different in terms of their relation to contextual conditions or boundaries. Nonetheless, van Lier (2005) provides a compromise between the two perspectives as he remarks, “case study is contextual study, whether or not the contextual boundaries can be easily drawn” (p. 205). According to these definitions, this present study focused on four individual teachers as single cases, and the phenomenon under study was their writing assessment practices as contextually bounded in teachers’ writing instruction in their particular classroom contexts. Stake (2005) categorises the case study approach to research according to three domains: intrinsic case study, instrumental case study, and multiple or collective case study. The intrinsic case study focuses on the representation of the value and uniqueness of a particular case under study. The instrumental case study stresses the insight or understanding of a wider issue that the actual case represents. The multiple or collective case study focuses on the investigation of a phenomenon or general condition by studying several cases. Punch (2009) concludes that intrinsic and instrumental case studies are single case studies focusing on phenomena within one case; whereas a multiple case study focuses on phenomena within and across several cases. In relation to Stake’s (2005) categories, this study employed the multiple case study approach to investigate writing assessment practices of four individual teachers. It is anticipated the multiple case studies will facilitate a robust understanding of the individual teacher’s

writing assessment practices and allow for a comparison of the commonalities and differences across their practices to be made. As mentioned by R. K. Yin (2003), case study research may either be exploratory, descriptive or explanatory in nature. An exploratory case study is often undertaken as a pilot to explore situations, to define the questions and hypotheses of other studies, or to determine the practicality of the potential research process. A descriptive case study aims to offer narrative accounts of events within the specified context. An explanatory case study aims to explain how events occur. Considering the three different purposes for conducting a case study described above, the multiple case study approach employed in the present study aims to be exploratory in order to understand the writing assessment practices of teachers in their natural settings. As such, the four case studies were undertaken to discover a knowledge basis of classroom writing assessment practices in the Thai public university context. In parallel with other studies using the case study approach (e.g., Duff, 2008; F. Hyland, 1998), this study recruited a small number of participants to yield rich information on writing assessment practices employed in regular classroom contexts. The multiple case studies of individual teachers can provide compelling and robust insights to each teacher’s assessment practices within his/her workplace setting. Each case study may also offer substantial data towards the development of a holistic understanding of writing assessment practices in Thai public universities. Moreover, the individual cases can be compared and contrasted (Wiersma & Jurs, 2009) so that the rating processes employed by the teachers, the criteria they used when assessing writing, and each teacher’s influence on the selection of the rating criteria can be clearly examined. As is the case with all approaches to qualitative research, the case study approach has its disadvantages: limited potential for generalisability and the potential for researcher bias, for instance. From the perspective of quantitative researchers (R. K. Yin, 2009), case studies may be criticised for lack of generalisability due to the focus on a small sample. It may be argued, however, that the purpose of the case study is not to achieve generalisation of results, but to stress the understandings of the local context under study. Generalisation may be achieved to some extent by readers as they may relate the findings presented in the case study to their own experiences and understandings (Merriam, 2009; Stake, 2006). Van Lier (2005) comments that

particularisation, or specific insights obtained from a case study, is more important than generalisation. He also remarks that contradictions in the findings of different cases may provide particular insights or lead to further research. Furthermore, theories may be expanded and generalised (R. K.Yin, 2009). For this reason the four cases in this present study are compared and contrasted in order to construct rigorous understanding of assessment phenomenon and to allow for possible knowledge expansion. Researcher bias is considered another potential weakness of the case study approach. In turn, subjectivity may be reduced when various data collection methods are used as evidence to ‘define the “facts” of the case’ (Bickman & Rog, 1998, p. 232). The variety of methods used may help ensure the validity of the present study. Moreover, a well-planned case study research design may also help eliminate researcher bias (R. K. Yin, 2003). On the basis of an awareness of these issues, multiple research methods for data collection were employed in this present study and will be further discussed in the next section.

3.4 Research Methods As mentioned earlier, this study has two phases. The first phase explores the teachers’ views on writing assessment practices in Thai public universities. As such, this phase provides a broad representation of teacher perceptions on assessment practices in relation to their personal and professional experiences, prior knowledge, and the contextual factors influencing their practices. This phase of the study also serves a sampling purpose for recruiting participants to be the subjects of the case studies in the second phase of the study. As previously mentioned, one of the aims of the present study was to ascertain the views of teachers in public universities located in different regions across Thailand regarding assessment practices. As such, a questionnaire was utilised in the first phase of the study to collect a large amount of data over a short period of time (Dörnyei, 2003, 2007). However, it is important to bear in mind that the depth of the data collected from a questionnaire may be limited (Dörnyei, (2007). Indeed, the insights gained through the use of this instrument may only represent a glimpse into teachers’ perspectives and what actually occurs in writing assessment practices in Thailand. As a result, it was deemed necessary to complement the questionnaire data with actual case study examples to gain

an insights into teachers’ thought processes in conjunction with what teachers actually do in their assessment practices were necessary. The four case studies included in this study provide a logical extension to the research conducted in the first phase of the study. They aim to facilitate an in-depth investigation into the assessment practices of writing teachers in Thailand. Four teachers were selected as case study participants in order to obtain the in-depth data (see the detailed account of participant recruitment in Section 3.5). In this second phase, in order to maximise the data collected and to avoid researcher bias during the data collection and analysis, a variety of data collection methods were employed; namely, semi- structured interviews, think-aloud protocols, stimulated recall interviews, and documents. As Borg (2012, p. 22) comments, “judgements about research rigour cannot be made if readers are not provided with adequate detail of how a study was conducted.” The intention in this study is to provide full detailed accounts of how each of the methods was adopted as well as the way in which data were collected (see Section 3.7) and analysed (see Section 3.8). Details of the way in which each research method was developed and employed are presented below.

3.4.1 Questionnaire To obtain the collective views of teachers in terms of their writing assessment practices, a wide range of participants from different geographical areas of Thailand was invited to participate in the questionnaire (further details of recruitment are supplied below). The questionnaire served two purposes: to obtain general information regarding teachers’ personal and professional backgrounds, types of writing being taught in Thailand, and teachers’ views on effective writing and assessment practices; and to ascertain the respondents’ willingness to be participants in the follow-up case-studies. The collection of baseline data from the participants’ responses also assisted the formulation of questions in the follow-up case studies. As evidenced in the literature, teachers’ writing assessment practices can be influenced by various factors such as their personal and professional backgrounds, training experience and their preferred features of good writing (e.g. Lumley, 2005; Shi, 2001; Weigle, 2002). Therefore, the questionnaire developed in this study was designed to highlight some of these key factors. The questionnaire comprised seven sections: background information, teaching experience, training experience on the assessment of

writing, views on good writing, writing at a composition/essay level course, views on assessing compositions/essays, and other comments about writing assessment. The first section was devoted to respondent demographic and general information. The second and third sections were for gathering information about the respondents’ professional backgrounds. The section on ‘views of good writing’ was designed to gain data on the respondents’ opinions about features required in good writing, while the next two sections were for collecting data about the current writing courses the respondents taught and their opinions towards assessing students’ writing in that particular course. The last section allowed the respondents to express an opinion they may have regarding writing assessment in Thailand. At the end of the questionnaire the respondents were asked to provide their e-mail address as an expression of their willingness to participate in the sequential case studies. The questionnaire included both closed- and open-ended questions. Closed questions were used in almost every section of the questionnaire to reduce the response time. Five-point semantic differential scales and six-point Likert scales requiring the respondents to indicate the extent to which they agreed with the provided statements were employed when asking the respondents about their opinions (Dörnyei, 2007). Due to the exploratory purpose of the questionnaire, open-ended questions were also extensively used throughout the questionnaire to gather qualitative data on the respondents’ opinions or views on aspects that cannot be predicted. While options for anticipated responses were provided in many sections of the questionnaire, open-ended questions were also used to obtain additional data to clarify the respondents’ selected options in the close-ended questions (Dörnyei, 2007). Johnson and Christensen (2008) state open-ended questions can help “understand participants’ inner world in their natural languages and categories” (p. 177) and provide information about teachers’ thoughts on aspects that may not be well defined. This procedure was used to provide overview data of the writing assessment practices employed in each of the Thai public university contexts in this study. English language was used throughout the questionnaire to gain data from both Thai and non-Thai teachers of writing (for the final version of the questionnaire used in this study see Appendix A). The questionnaire for this study was developed and piloted with various groups of between five and 20 people before being finalised. Many people in the trial groups shared common understandings with the prospective respondents in this study. To be

more specific, initial versions of the questionnaire were trialled with three individuals: two Thai PhD students with experience in teaching English to university students in Thailand, and a Thai teacher of English at the university level. In addition, the questionnaire was trialled with a group of five PhD students with different personal backgrounds who had experience in teaching English as a second or foreign language, as well as a group of approximately 20 teachers of English at a Thai university. As a result of their feedback, ambiguous items and options in the questionnaire were refined accordingly. When constructing the questions the physical features and content of the questionnaire were also taken into consideration in order to increase the potential return rate. In terms of physical features, the length and the layout of the questionnaire accorded with Dörnyei’s (2003) suggestion that the questionnaire not be longer than four pages with an allocated time of 30 minutes to complete. The layout of the questionnaire was also made more accessible through the provision of appropriate spacing, the use of legible font types and sizes, and suitable highlighting where required (J. D. Brown, 2001; Dörnyei, 2003, 2007). Refinement of the physical features and content was also undertaken following the piloting process. After finalising the questionnaire, consideration was then given to the mode of distribution. A questionnaire may be distributed as a paper-based or web-based survey, depending on the applicability of the data collection procedure. The advantages of a web-based survey over a paper-based survey are stressed in the literature (Dillman, 2000; Gunn, n.d.). Clearly, a web-based survey is more economical and when it is distributed via e-mail responses can be obtained at a faster rate. Also, reminders are more easily sent to participants via e-mail and data can be easily processed because they can be transferred directly to databases ready for analysis (Dörnyei, 2007; Gunn, n.d.). Moreover, with high-level participant anonymity a feature of web-based surveys, the level of honesty in the participants’ responses increases (Dörnyei, 2007). Because of these various advantages, a web-based survey questionnaire was used in this study and distributed online via http://www.surveymonkey.com. After the questionnaire was transferred to a web-based format, additional piloting was undertaken. An email containing a link to the online questionnaire was sent to a Thai teacher who taught English at the university level. The teacher had not seen previous versions of the questionnaire and was not a potential respondent in the study.

The teacher was asked to respond to the questionnaire to see if the format and layout were easy to read, as well as to ascertain if there was any difficulty in accessing and responding to the online version. Her feedback was taken into account prior to the questionnaire’s final adjustment.

3.4.2 Semi-structured interviews In the case studies, semi-structured interviews were employed to obtain information about the four teachers’ views and practices of writing instruction and assessment in their work contexts. Interviews are a frequently used method in qualitative research (Dörnyei, 2007) as they are ways of “accessing people’s perceptions, meanings, definitions of situations and constructions of reality” (Punch, 2009, p. 144). Semi-structured interviews were deemed appropriate for use in this study due to their flexibility. This data collection method allows additional questions to be incorporated into the pre-planned questions, thus increasing the natural flow of the conversation (Dörnyei, 2007; Punch, 2009). The additional questions may be used to obtain further clarification or to seek an elaboration on the participants’ original response. An interview guide including the list of pre-planned questions was developed and used to guide the interviews of the four participants in the case studies. The guide also included pre-rating and post-rating questions. The pre-rating questions were asked prior to the think-aloud protocols in order to build rapport between me, the researcher, and the participants. The post-rating questions were asked in order to have the teachers reflect on the overall process of their general writing assessment practices (for the question guide see Appendix B). The question guide was initially piloted on a PhD student who had experience in teaching writing at a Thai university in order to clarify the questions. The guide asked about training experiences, teaching and assessment practices, and contextual factors such as student profiles and the nature of the course. Additional questions were asked for further clarification and elaboration of the participants’ responses in accordance with each participant’s situation. All participant interviews were audio-recorded to capture the spoken data for the purpose of further analysis. To collect comprehensive data from the participants they were allowed to talk freely and the interviewer was restricted from making comments or providing advice or suggestions.

3.4.3 Think-aloud protocol The think-aloud method of data collection is commonly employed by researchers in the area of writing assessment (e.g. Barkaoui, 2007; Cumming, et al., 2002; DeRemer, 1998; Lumley, 2002, 2006; D. Smith, 2000; Weigle, 1999). These authors used think-aloud protocols to investigate different rating aspects and rater practices (DeRemer, 1998), raters’ decision making during the rating process (Cumming, 1990; Cumming, et al., 2002), and the determination of an appropriate rating scale (Barkaoui, 2007). Although the aims the aforementioned studies were different to the aim of this study, in all cases the data obtained were from the raters’ thought processes. As the principal aim of the follow-up case studies in the present research was to investigate the participants’ unfolding cognitive processes while assessing their students’ writing, concurrent think-aloud protocols appeared to be an appropriate method. This is considered to be a method that can capture “detailed steps of thought processes” (Ericsson & Simon, 1987, p. 24). While performing a task the participants are required to verbalise what goes on in their minds while it is stored in their short-term memory (Ericsson & Simon, 1984; Ericsson & Simon, 1987; S. McKay, 2006). Data obtained using think-aloud techniques are considered direct and immediate as they are obtained before the participant’s thoughts are processed into their long-term memory (Cohen, 1987; van Someren, Barnard, & Sandberg, 1994). Even though the think-aloud technique is regarded as a useful tool to collect on- going thought processes, it is nonetheless criticised for not being a natural process that participants can undertake automatically. Therefore, it is preferable for participants to be instructed and trained prior to the think-aloud process (Dörnyei, 2007). Acknowledging this constraint, a training guide was developed for the think-aloud activity that included an instruction for undertaking think aloud protocols and three practice tasks comprising one simple mathematical task and two practice activities that related to the focus of the study. The guide helped the participants get used to verbalising while thinking (J. D. Brown & Rodgers, 2002; S. McKay, 2006; van Someren, et al., 1994). To ensure the validity of the think-aloud protocol in this study, the think-aloud task was first tested on two individuals from different workplace contexts. One had a teaching background similar to the participants in this study; whereas the other had no

experience in teaching English. Both trial participants were asked to attend a think- aloud training session to determine whether the procedures for introducing the think- aloud protocol and the examples to be used were effective in helping them familiarise themselves with the method. Then, the training guide, particularly the instruction for think-aloud protocols, was edited and finalised (for the final version of the think-aloud instruction and training guide see Appendix C). During the actual data collection period the participants in this study were provided with training prior to being asked to deliver the think-aloud protocols in their actual rating processes. In the training session the participants were provided with a demonstration of how to think aloud while accomplishing the mathematical task. The participants were then asked to practice verbalising their thoughts while doing the two practice tasks: an error recognition task, and the assessment of a short paragraph relevant to the real task they had to do in the main think-aloud activity. In the main think-aloud session the teachers were asked to select and assess two students’ writing scripts based on their usual rating criteria. The think-aloud protocols applied to the two scripts were likely to provide rich information on teachers’ actual rating processes. Video-recording was used to capture both verbal and non-verbal behaviours which are important to show the participants’ thought processes during the two protocols (J. D. Brown & Rodgers, 2002; S. McKay, 2006; van Someren, et al., 1994).The video recordings were also used as stimulus in the stimulated recall interviews undertaken after the think-aloud activity.

3.4.4 Stimulated recall interview Stimulated recall interview is an introspective data collection method used to elicit the thought processes experienced by the participant during his/her involvement in a task or activity. Thus, it may be used to remind participants of what they had in mind during the think-aloud process (Gass & Mackey, 2000, 2007). Stimulated recall interview can also be used to triangulate data obtained from the think-aloud method. As stimulated recall concerns verbalisation of thought processes, the length of time between the recall and the think-aloud activity should be minimised. The recall can be done soon after the think-aloud session so that the participants can recall their thoughts while they are stored in short-term memory (Ericsson & Simon, 1984; Gass & Mackey, 2000; Lyle, 2003). This helps to ensure validity of the recall (Gass & Mackey, 2000).

Recall interference may occur if the time distance between the think-aloud and recall session is too long (Lyle, 2003). Moreover, the recall should be related to what the participants were doing during the task and a strong stimulus should be used to stimulate the participant’s memory. As the use of audio and visual aids are recommended to record the think-aloud activity, they can be played back to facilitate the participant recall. The participants should be able to conduct the recall by themselves with minimal help from the researcher. The timing provided for both researcher- participant interaction and the recall procedure should also be clear. The whole procedure should take equal time and the researcher should be consistent in his/her actions in the recall sessions of all participants (Gass & Mackey, 2000, 2007). In this study, a stimulated recall interview was employed to triangulate data obtained using the think-aloud protocols. The recall interview was undertaken immediately after the think-aloud session. During the recall, a video recording of the participants’ two think-aloud protocols was played. The participants were allowed to pause the recordings whenever they wanted to make comments on their behaviour or to clarify their ideas. They also received constantly reminders to report what they were thinking. Audio recording was used to capture the recall. After the recall interviews based on stimuli from the video recordings, the participants were asked questions to reflect upon the thoughts that emerged while assessing the writing tasks during the think-aloud activity. The questions were intended to obtain more information on the participants’ thoughts in relation to contextual factors such as course objectives. They were also asked to reflect on the rationale for their practices and opinions they held towards the practices they currently employed (for the questions see Appendix D).

3.4.5 Documents As Punch (2009) mentions, documents can be used in conjunction with other methods in case studies for triangulation purposes. In this study, the aim was to use documents to support the understanding of each participant’s instruction and assessment context. All relevant documentary sources of data were collected including the course syllabus, the rating criteria the participants employed in the think-aloud activity, and the students’ marked writing (following permission to do so). The course syllabus and rating criteria were used as supplements to the think-aloud activity and interviews.

While data from the think-aloud activity would show the teachers’ thinking processes during the writing assessment, the students’ marked writing were employed as a representation of the writing assessment product. Apart from the two samples of the students’ writing that were used in the think-aloud activity, other samples of the students’ marked writing in the same text type were collected with the permission of the participants.

This section discussed and illustrated how each research method was employed in this data. The research participants and recruitment method are described in the next section.

3.5 Case Study Participants Fifty-nine teachers from various Thai public universities, including both Thai and non-native Thai teachers, responded to the questionnaire in the first phase of the study. Twenty-four of these teachers expressed their willingness to participate in the follow-up case studies. As this study did not aim to compare the rating processes of teachers with different language backgrounds, only Thai teachers were included as potential participants in the case studies. This criterion narrowed the number of potential participants down to 17 Thai teachers. Among them, four teachers located in four different geographical regions of Thailand were selected as potential case study participants based on their responses in the questionnaire. The four teachers also met the selection criteria: being representative of the teaching population in different regional universities; and teaching at least one of the most common types of writing texts in Thai public universities. However, after making contact with each of the four teachers to arrange a meeting for data collection, it eventuated that some of them were not assigned to teach writing in the semester when I intended to collect data. As a result, other potential teachers from the list of questionnaire respondents were selected. However, because there was approximately one semester time lapse between the completion of the questionnaire phase and the commencement of the case study phase, the potential teachers who expressed their willingness to participate either changed their mind or were not assigned to teach a writing course. As such, only two potential teachers from the list of questionnaire respondents were able to agree to participate. Two other teachers were then contacted through snowball sampling drawn from my

personal contacts. Before the data collection commenced however one of the potential teachers dropped out due to personal constraints, but she nominated her colleague as a replacement. Three female teachers and one male teacher were finally recruited for participation in the study. Collectively, the teachers were teaching in universities located in central, southern, northern and northeastern Thailand and this distribution supported the aim of the research design to include participants from different parts of Thailand. The participants were Ladda1, the only potential teacher from the list of the questionnaire respondents, and Nittaya, Sak and Rewadee, three new teachers. A general description of the four case study participants is provided in Table 3.1 below:

Table 3.1 Description of the four case study participants Name Gender Age Workplace2 Ladda Female 51-60 South University Nittaya Female 41-50 Central University Rewadee Female 31-40 North University Sak Male 26-30 Northeastern University

Because the three new case study teachers participating in the study had not initially answered the questionnaire, they were asked to do so in order for me to collect their demographic data and views on writing assessment practices. Their responses were included in the questionnaire response analysis, bring the total number of questionnaire respondents to 62. All of the respondents were used in the analysis to identify overall patterns of Thai public university teachers’ views of writing assessment.

3.6 Ethical Considerations Given that this study aimed to collect data from and about human participants, research ethics needed to be considered. As Punch (2009) remarks, research ethics are more acute in qualitative research than in quantitative or mixed-methods research because qualitative research may intrude in people’s lives. Ethical issues need to be considered at all stages of the research. Miles and Huberman (1994) state that attention to ethical issues needs to be paid before, during and after the research. Some of the key

1 Pseudonyms were assigned to all case study participants. 2 Pseudonyms were assigned to the workplaces of all participants.

issues are consent, privacy, confidentiality and anonymity of the participants, and the use of data. Researchers need to consider these issues so that participants are prevented from physical and emotional harm or intrusion. In consideration of the ethical issues, various precautions were undertaken when collecting data. The questionnaire employed in the first phase of the study was completely anonymous as no specific questions were asked which could identify respondents. During the case study phase of the study, the four participants were provided with consent and information forms explaining the overall data collection procedures (for the consent and information form for the participants see Appendix E). Data were then collected after the participants had agreed to be involved in the study and had signed their consent. When quotes from the participants’ data sets are used in the following chapters, pseudonyms are assigned to avoid the identification of the participants. When plans for data collection procedures were set, ethics approval for each research phase was sought. Ethics approval for the questionnaire was granted from the Faculty of Human Sciences Ethics Review Sub-Committee, Macquarie University, dated 30 April, 2010. This approval was later transferred to the University of New South Wales Human Research Ethics Committee (for the transferred ethics approval see Appendix F). Ethics approval for the sequential case studies was granted separately by the University of New South Wales Human Research Ethics Committee, dated 13 May, 2011 (see Appendix G).

3.7 Data Collection Procedure Data collection in this study was distributed in two sequential phases. During the first phase the questionnaire was distributed online via http://www.surveymonkey.com from June 2010 to early March 2011. An e-mail containing a direct link to the online questionnaire was also distributed to the heads of department/program whose email addresses were provided in the website of their universities. However, during the search of their email addresses it became evident that not every university provided the department/program heads’ email address on the website. Instead, other contact details such as phone numbers and the mailing address of the department were provided. Contact was made with the university using the phone numbers provided and a request was made for the email addresses of the heads of the relevant departments/programs.

The department/program heads were then contacted via e-mail and asked to forward the e-mail message containing the link to the teachers in their departments/programs. A reminder was sent out every two to three weeks. In the second phase of the study, the four potential participants were contacted via telephone to make arrangements for data collection. The meetings were made based on each individual teacher’s availability and the actual time they assessed their students’ writing assignments. My intention was to collect data on the writing assessment of the same text type (i.e., comparison and contrast) because it was reported in the questionnaire data as one of the most common text types taught in Thai public universities. Moreover, the four participants reported they taught the writing of this text type at a time that was practical for case study data collection in actual situations. However, in the case of Ladda, the meeting time was not convenient and she could not extend her assessment of this text type in her course due to time constraints. It was subsequently decided that her data would be collected while she was assessing writing in another text type (i.e., cause and effect). Because one aim of the case studies was to investigate teachers’ thought processes while rating their students’ writing, the different writing text types would be unlikely to affect their rating processes. Even though this issue is not explicitly addressed in literature, it may be implied from various studies which aim to investigate teachers’ rating processes that writing text types do not affect teachers’ thinking processes. A number of research studies in this area have focused on the differences in rating criteria or the raters’ experiences without paying attention to writing task types. For example, Sakyi (2000) selected writing tests written by students from different disciplines to use in his study. Although their writing samples were taken from the same test, they were asked to complete different task types according to their study discipline. A more recent study conducted by Lumley (2005) also included two different tasks with somewhat distinctive requirements as a means to investigate teachers’ rating processes. The two studies reported the collective raters’ rating processes without differentiating the rating processes employed when assessing different writing tasks, suggesting the distinctive tasks do not have an impact on the rating processes. In this study, each case study participant was involved in two sessions of data collection. In the first session, the participants were asked to give consent to the data collection. The participants were then interviewed to obtain data about their usual

practices in writing instruction and assessment, and, as previously discussed, were trained in how to conduct a think-aloud protocol. Both the interview and the think-aloud training were audio-recorded and video-recorded, respectively. In the second session, the participants were asked to conduct a think-aloud protocol while rating two samples an actual writing task written by their students. The participants were asked to use their actual rating criteria. During the think-aloud rating activity, video-recording was used to record both verbal and non-verbal behaviours made by the participants. After a short break, the participants were asked to recall their thinking while doing the activity using the video recording as a stimulus. They were allowed to pause or forward the video at any time to make comments on their think-aloud or behaviours. Also, the participants were asked additional questions about their think-aloud and behaviours where clarification was needed. The audio recording was used during this activity. Because the participants and I shared Thai as the same first language, all data collection in the case studies was conducted in Thai. The data were also transcribed and analysed in Thai to ascertain validity of the data interpretations. When data extracts were quoted in this research, they were then translated by me into English. Because the case studies were conducted with the teachers in their natural setting, some challenges experienced during the data collection needed to be addressed. In this study it was deemed necessary to collect data at each participant’s workplace in order to maximise the potential to interpret the data according to its natural context (Duff, 2008). Furthermore, given I was an outsider in each participant’s workplace the venues for the case study data collection were selected by the participants. The selected venues had public access which may have inevitably affected the participant’s performance, particularly when the participants carried out the think aloud protocols which required a comfortable quiet place (J. D. Brown & Rodgers, 2002; S. McKay, 2006; van Someren, et al., 1994). Interruptions could not be avoided during onsite data collections undertaken on a normal working day. The participants appeared to be distracted at times by phone calls during the process and other surrounding noises. These distractions may have affected the quality of the data collected, especially the think aloud data which relied mainly on the participants’ short-term memory. The relationship between the researcher and the participants also played a vital role in data collection. If the participants did not feel a rapport with the researcher they may have refused to undertake a certain activity resulting in incomplete data collected.

It was necessary for the researcher to establish a relationship with the participants prior to the actual data collection period to build up rapport and to ensure the process was non-threatening. Due to individual differences, some participants appeared to be more articulate than others in terms of providing data. The participants who had greater experience in teaching English and who shared some common characteristics with the researcher such as age and gender tended to give more detailed and potentially better quality data. As Duff (2008, p. 134) remarks, “chemistry between interviewer and interviewee (in addition to context, purpose, history, etc.) may have something to do with why some respondents provide better data”. For the participant who seemed to be less articulate, the researcher was required to ask more questions to obtain more detailed responses. The data collected from the more articulate participants may provide richer descriptions than the data from the less articulate participant (Duff, 2008). Regarding documentary data, the students’ marked writing could not be collected on the day of data collection. The participants were requested to assess their students’ writing scripts for the data collection so they first assessed their students’ tasks when they undertook the think-aloud protocols. Given that all of the students’ marked writing scripts were required to complement the participants’ think-aloud data, each participant was asked to post all relevant documents including the course syllabus, rating criteria and students’ marked writing to me upon completion of the marking process. Even though this method was practical, it seemed to be less manageable. Because of heavy workloads, some participants could not manage to post all documents when marking was completed. Following a few reminders however the documents (although, at times, not all of those requested) were posted to me.

3.8 Data Analysis To provide a better understanding of writing assessment practices in Thai public universities this study has undertaken an exploratory qualitative approach. As such, the study aims to identify patterns or trends in writing assessment practices rather than to test an existing theory. The analysis approach is both iterative and inductive (Duff, 2008; Scott & Usher, 2010) and data were collected through a variety of methods to “ascertain multiple forms of interpretation (or multiple realities)” (Duff, 2008, p. 30). The initial questionnaire provides both qualitative and quantitative data about teachers’ views on writing assessment practices. To gain further insights into writing assessment

practices in different universities, the four teachers’ individual views of assessment were obtained primarily through semi-structured interviews; whereas their rating processes and practices were obtained though think-aloud protocol and stimulated recall interviews. Documents provided substantial data to support their actual practices. The procedures for the analysis of data from each of these sources are presented below.

3.8.1 Questionnaire data Following the distribution of the questionnaire, the data collected were transferred into a database provided by the SurveyMonkey website to undergo analysis. As the questionnaire was distributed over an extended period of time and many reminders were sent out, there was the possibility that the same respondents answered the questionnaire more than once. To ensure quality of data, data cleaning (Dornyei, 2007) was undertaken by checking the IP addresses of all questionnaire respondents indicated in the SurveyMonkey database. Without any intention to trace the respondents, the provided IP addresses were used to check whether there were repeated responses from the respondents. When repetition was identified the second set of data obtained from the same respondent was omitted. Two types of data were collected: numerical data and word-based data. The numerical data gathered through use of close-ended questions were analysed using simple descriptive statistical methods. There was no intention to use complex statistics. Indeed, the questionnaire was designed to provide exploratory findings which could be used to progress the study and not with statistical analysis in mind. Word-based data gathered through use of open-ended questions were categorised into themes and sub- themes following procedures in interpretive method analysis (Creswell, 2009). Open-ended questions were also included to support additional comments from the respondents throughout the questionnaire. These comments were then used to supplement the respondents’ close-ended answers. All responses were initially labelled under the questions to which they responded as a ‘start list’ of pre-determined categories (Miles & Huberman, 1994). For example, the responses to the question about training were first labelled under the theme of training. Each response was then coded with the same initial theme based on its content. Following the structure of the questionnaire, four initial themes were identified: training, good writing features, rating criteria and writing assessment in Thailand. All responses were then reread to identify

similarities or differences. Any response found to be unrelated to the initial theme was omitted. When the responses under the initial themes were settled the unrelated responses were mixed together and coded based on their content. Following this coding process the groups of responses were further analysed within the same theme to identify the focus of each response. The coded responses were then grouped according to generic features and revised until categories and subcategories were formed. All responses across the different categories were then reread and revised to ensure they cohered with other responses in the same group (for examples of coded responses see Appendix H).

3.8.2 Verbal report data Two types of verbal data were collected in the case study phase of the study: think-aloud data and interview data. The think-aloud data representing the teachers’ concurrent thought processes were collected using think-aloud protocols. Interview data included teachers’ views on their practices and were obtained using semi-structured interviews and the complementary data pertaining to the teachers’ actual rating processes gained through stimulated recall interviews. All verbal report data from the three sources were transcribed, and the transcripts in Thai language were used in the analysis to maintain the original meanings of the verbal report. As mentioned earlier, only the extracts quoted in this thesis were translated into English by me. The two forms of verbal report data were treated and analysed differently. Each of them is discussed below.

3.8.2.1 Think-aloud data Data gained through think-aloud protocols represented the four selected participants’ on-going cognitive processes while rating their students’ writing. The protocols aimed to reveal rating processes employed by the teachers. They were analysed in these stages: pre-coding stage, coding stage and inter-coder agreement.

Pre-coding stage During this stage the data were transcribed using the computer program Nvivo (for transcription conventions see Appendix I). The transcripts were segmented for further analysis, with the segmentation of text units adapted from criteria proposed by

Barkaoui (2008), Cumming et al. (2002) and Lumley (2005). The transcripts were primarily segmented into smaller units of text based on a technical method. Each text unit was based on the following guidelines: a. a text segment that occurs at the beginning or end of a single essay b. teacher identifying the script at the beginning (i.e., student name and topic of essay) c. a writing script segment read aloud by teacher d. teacher interruption while reading a text segment (i.e., teacher writing a comment/marking any part of script) e. a pause of five seconds or more It should be noted that hesitation markers were placed in the same segment as the text which came after them as they did not represent meaning in their isolated occurrence. For example, in the sample; Mmm, however money is seldom important (Text Unit 114, Nittaya, Student 2’s script), the hesitation marker Mmm could not be interpreted and coded on its own and was subsequently placed in the text segment; however money is seldom important. Each long text segment was further divided into smaller text units based on a change in decision-making behaviour by the participants. This aspect of the segmentation process was quite intuitive − as Lumley (2005) mentions, “division of data into text units is ultimately an arbitrary act” (p. 135). Being aware of this challenge, I tried to refine and revise text units in each script many times during the coding process. The text segmentation and coding procedures were thus undertaken in a non-linear way. When parts of the same text segment were assigned to two different codes they were divided into two text units.

Coding stage The coding scheme used in this study was based mainly on Cumming et al.’s (2002) and Barkaoui’s (2008) framework of rater decision-making behaviours3. Cumming et al.’s coding scheme focuses on both rating behaviours and the writing features emphasised by the rater. They divided the decision-making behaviours into three focuses: self-monitoring focus to address raters’ own rating behaviours; rhetorical and ideational focus concerning the use of rhetorical structure and content or ideas in

3 Barkaoui’s (2008) coding framework was adapted mainly from Cumming et al.’s (2001, 2002) preliminary framework. 71

the text; and language focus showing raters’ attention to the accuracy and fluency of the language. Each of these three focuses is further divided into two strategies types: interpretation strategies and judgement strategies. The interpretation strategies focused on the macrostrategies raters employed to comprehend the scripts; whereas judgement strategies represented the raters’ evaluation of the scripts. Regarding the 35 decision-making behaviours in the preliminary framework developed by Cumming et al. (2002), I initially read the text units in each script to identify the applicability of the behaviour statements to the data. Some text units however did not fit into the coding scheme and as a result some behaviour statements were added or changed to make them more suitable to the data. I read, reread and revised the coding scheme by coding the text units of each script until they could be assigned a code. The initial coding at this stage was conducted by hand. During these procedures I made a number of changes to the coding scheme for the purpose of this study. Ten statements originally included in the three focus areas from Cumming et al.’s framework were omitted. In the self-monitoring focus, ‘scan whole composition’, ‘consider own personal responses and bias’, and ‘define or revise own criteria’ were omitted. In the rhetorical and ideational focus, ‘assess interest, originality or creativity’, and ‘assess style, register, or genre’ were omitted. In the language focus, ‘edit phrase for interpretation’, ‘assess quantity of total written production’, ‘assess gravity of errors’, ‘consider error frequency’ and ‘assess fluency’ were omitted. In relation to these omitted statements, some of them such as ‘assess style, register, or genre’ were omitted because they were not relevant to the data while some of them such as ‘assess gravity of errors’, ‘consider error frequency’ were subsumed in a larger category such as ‘rate language’. Eighteen statements of behaviours were then added to support a more detailed analysis of the data. The adapted coding scheme finally included 41 main strategies covering 16 interpretation strategies and 26 judgement strategies as represented in Table 3.2. The added behaviours are noted with an asterisk in the behaviour statements in the table:

Table 3.2 Coding scheme for think-aloud protocols (adapted from Cumming et al., 2002) A. Self-monitoring Focus B. Rhetorical and Ideational C. Language Focus Focus 1. Interpretation Strategies 1. Identify the writer* 1. Interpret ambiguous or 1. Identify mistakes* 2. Identify or interpret unclear phrases 2. Classify mistakes into prompt 2. Identify rhetorical types 3. Identify topic* structure 2.1 Vocabulary 4. Identify outline* 3. Identify mistakes in 2.2 Sentence 5. Read or reread rhetorical structure* structure/Grammar outline/composition 4. Summarise/Translate 2.3 Mechanics (i.e., 6. Refer to teaching* phrases, ideas or spelling, punctuation, 7. Provision for feedback* propositions capitalization) 8. Envision personal 5. Identify topic 3. Identify sentence structure* situation of student development*

2. Judgment Strategies 1. Decide on macrostrategy 1. Assess outline or topic 1. Assess comprehensibility for reading and rating 2. Assess reasoning, logic, 2. Consider/Assess sentence 2. Compare with other or topic development structure or grammar compositions or “anchors” 3. Assess relevance 3. Propose 3. Compare main text with 4. Assess coherence correction/Correct* outline* 5. Identify redundancies 3.1 Vocabulary 4. Summarise, distinguish or 6. Assess text organisation* 3.2 Sentence tally judgments 7. Assess task completion structure/Grammar collectively 8. Propose correction* 3.3 Mechanics (i.e., 5. Articulate overall results* 9. Rate ideas or rhetoric spelling, punctuation, 6. Articulate or revise overall 10. Assess layout capitalization) scoring decision 4. Rate language 7. Identify cause of mistake* 5. Consider/Assess 8. Write/Mark comment in vocabulary text* 9. Articulate (reason for) written comment/mistake* 10. Articulate rating style/problem* 11. Confirmation of correction or decision*

Given the description of the distractions that occurred during the think-aloud protocols was addressed in the transcripts, the code ‘distraction’ was added to facilitate their identification. Some text segments representing incomplete transcripts due to unclear voices were coded as ‘uncodable segment’. After the coding scheme was set, all scripts were then coded using Nvivo software. This software allowed the coded segments to be easily checked across all scripts (for the detailed coding scheme assigned to the decision-making behaviours see Appendix J; and for samples of the coded think-aloud protocols see Appendix K). When all text units were coded, I checked the text segments in all codes to ensure they had the same properties. It is noted

here that some text segments were assigned more than one code due to the overlapping nature of some categories based on coder’s interpretation. This issue is further discussed below.

Inter-coder agreement During the coding session it became evident that codes assigned to some text segments overlapped across focus areas or strategies. For example, one text unit may be labelled as a judgement strategy for the rhetorical and ideational focus or the language focus. Table 3.3 below shows an example extract from Nittaya (Student 2’s writing script) to clarify this point:

Table 3.3 Example extracts (Nittaya, Student 2’s writing script) Text Text segment Unit 52 Most of people don't have much time to meet a neighbourhood 53 Well, the student wants to say people in the city do not have much time to meet. 54 There are many neighbours in country. 55 Okay, okay, this is neighbour.

Text unit 55 above shows the code overlapping. It may be coded as ‘assess reasoning, logic, or topic development’ in the rhetorical and ideational focus category or ‘assess comprehension’ in the language focus category depending on the coder’s interpretation. The overlapping of codes in this study echoes Cumming et al.’s (2002) framework limitation as acknowledged by the authors:

Decision-making behaviours overlap one another logically or categorically ... because of the complexity of human assessment processes, the interrelationships of language, ideas, and rhetorical forms, and the difficulty of distinguishing ... discrete human behaviours from their holistically integrated nature (p. 90)

Being aware of this limitation, inter-coder agreement was sought to ensure reliability in the coding scheme developed. Another coder doing a PhD in English was contacted and asked to code two of eight scripts independently, based on the adapted coding scheme. The percentage agreement was subsequently calculated using a web- based tool provided at http://dfreelon.org/utils/recalfront/recal2/#doc. The agreement

achieved at the strategy level was 65.23% and 76.40% at the focus level4. Even though the percentage agreement at the focus level was acceptable, the agreement at the strategy level was low. A third coder doing a PhD in Linguistics was then contacted to further validate the percentage agreement. The level of agreement among the three coders was calculated using the tool provided at http://dfreelon.org/utils/recalfront/ recal3/. The subsequent percentage agreement at the strategy level was 72.93% and 81.13% at the focus level (for the full result of the inter-coder agreement see Appendix L). Furthermore, due to the low percentage agreement between me as the first coder and the second coder at the strategy level, I then interviewed the second coder to ascertain the reasons for the disagreement. The interview revealed we employed different approaches during the coding process, with my considering each text segment separately and therefore assigning codes based on the content in individual text segment, and the second coder interpreting each text segment in relation to surrounding text segments and assigning codes based on the flow of the participants’ thought processes. The different approaches to coding were likely to have resulted from a lack of shared understanding of the coding scheme developed. Following a discussion of the codes used in one of the two scripts during the inter reliability process, the second coder and I reached a compromise. This issue may imply higher inter-coder agreement may be achieved if coders are trained and have a shared understanding of the constructs in the coding scheme (Kurasaki, 2000).

3.8.2.2 Interview data Data obtained from semi-structured interviews illustrate the four teachers’ perceptions of writing instruction and assessment practices in their mutual working contexts (for sample excerpts of interview data see Appendix M); whereas data obtained from stimulated recall interviews represented complementary data related to the teachers’ thinking processes during the think-aloud protocols (for sample excerpts of stimulated recall interview see Appendix N). The two data sets were transcribed and analysed using Nvivo and each of them was analysed separately according to the following procedures.

4 The strategy level refers to the aggregation of coding frequencies in each kind of strategies (i.e., interpretation and judgment strategies under each focus). The focus level represents the aggregation of coding frequencies in all strategies under each focus. 75

Coding of semi-structured interview data The data set obtained from semi-structured interviews were analysed following iterative interpretive data analysis. The data were divided into text segments, each of which was identified by the flow of content addressing the same concept or topic. A change of topic signalled the end of the previous concept and the beginning of a new concept. Thus, the length of a text segment was not restricted and varied according to the relationship between the content in the interview data. The text segments were assigned codes corresponding to their content which were then used as a ‘start list’ (Miles & Huberman, 1994) for the conceptual themes in the analysis. This start list included a number of concepts produced during the unstructured, free-flowing interviews. Names given to each concept were based on a broad interpretation of each text segment. For example, the concept of ‘co-teacher’ was assigned to a text segment representing information about how case study participants worked with their co- teachers in their contexts.

Co-teacher Researcher: This means you are the only teacher of the course? Nittaya: Yes. Researcher: No co-teacher? Nittaya: Because I have taught this course from the beginning and it seems no one would like to ... Researcher: No one would like to teach? Nittaya: Not really. They know that I’m still teaching it. I told the timetabling organiser that anyone is welcomed to co-teach. However, I prefer teaching alone because I can make my own decision. I’d like to make the three classes the same. I used to teach two classes and another teacher taught one class. It’s like we have to check if we do this and that in the same way. I have to follow up. The test needs to be the same unless some students would be more advantaged than others.

The above extract provides factual information about the co-teaching situation in Nittaya’s context and her views on working with a co-teacher. With this dimension, the data was coded as ‘co-teacher’. Following the assignation of codes to all text segments, I revisited them to compare and contrast their properties. This was undertaken to group together the codes that appeared to represent similar phenomenon and to form an upper level theme. During this process the codes in the initial start list were carefully examined. They were also changed and moved to categories related more directly to the meaning of the text

segments. This process reduced the number of overall codes in the initial start list. Because the interview data contained two forms of information: facts and opinions, the codes were grouped accordingly. The factual data generally represented information about the participant’s context. It included each participant’s profile information and data about the writing course context. This information enabled a better understanding of each teacher’s immediate context to be achieved and provided the background for each teacher’s assessment practices. The codes pertaining to each teacher’s profile and writing course context were subsequently omitted from further analysis. The opinions illustrating each teacher’s perceptions of their practices were rigorously analysed. Following the analysis and interpretation of the first case study participant’s data set, the same analysis procedures was adopted when analysing the other three participants’ interview data. The codes to emerge from the analysis of the first participant were considered to be preconceived categories in the analyses of the three participants’ data sets. The separate analyses of the four teachers offered the emerging of individual teachers’ practices, providing single case analyses and leading to the cross-case analysis. Categories illustrating the individual teachers’ practices were considerably diverse according to the amount of data produced by each teacher. A summary of each category to emerge from each data set is presented on Table 3.4 to Table 3.7 below:

Table 3.4 Summary of categories in relation to Ladda Categories Subcategories Concepts The nature of students Students’ writing Writing quality Influences on writing quality Attitudes towards students Classroom practices Teaching strategies Assessment practices Rating criteria Rating strategies Feedback strategies

Ladda’s data set produced two major categories: the nature of the students, and classroom practices. The nature of the students category revealed Ladda’s comments about her students’ writing in relation to their writing quality and the factors influencing the writing quality. The classroom practices category represented Ladda’s perceptions

of her own teaching strategies and assessment practices. The latter was further classified into her views on the rating criteria she employed, her strategies to assess students’ writing, and her feedback strategies.

Table 3.5 Summary of categories in relation to Nittaya Categories Subcategories Concepts Classroom practices Teaching practices Teaching principles Teaching strategies Assessment practices Rating criteria Rating strategies Feedback strategies Difficulties in assessment

The only major category to emerge from Nittaya’s interview data set was classroom practices. It included her comments on teaching strategies and assessment practices. The teaching practices included attributes in relation to writing instruction: Nittaya’s underlying theory of her instruction and her practices of writing instruction. The assessment practices illustrate her comments on the rating criteria she employed, the way in which she assessed students’ writing, the way in which she provided feedback to the students, and the challenges she experienced in assessment.

Table 3.6 Summary of categories in relation to Rewadee Categories Subcategories Concepts Classroom practices Teaching practices Assessment practices Rating criteria Rating strategies Feedback strategies

Rewadee’s data set revealed her classroom practices and these were further classified into two categories: teaching practices and assessment practices. While the teaching practices pertain to a single category, the assessment practices represent Rewadee’s perceptions of the rating criteria she employed, the strategies she used while assessing students’ writing, and how she provided feedback to the students.

Table 3.7 Summary of categories in relation to Sak Categories Subcategories Concepts Classroom practices Teaching practices Assessment practices Rating criteria Rating strategies Difficulties in assessment

The only major category to emerge from Sak’s interview data was classroom practices. This category demonstrates two further categories: teaching practices and assessment practices. While the teaching practices category illustrates Sak’s views on his writing instruction practices, the assessment practices category shows how he undertook assessment and the three subcategories reveal his perceptions of the process of assessment.

Analysis of stimulated recall interview data The data gathered during the stimulated recall interviews were designed to complement the information collected on the participants’ decision-making behaviours they exhibited during the think-aloud protocols. These data were not analysed in full. The data were examined selectively when text segments corresponded to particular decision-making behaviours displayed by the teachers. When analysing the stimulated recall interview data the whole transcript was initially read through while keeping in mind the decision-making behaviours displayed by the participants. When a match was identified the particular text segment was further interpreted. For example, during Sak’s think-aloud protocols he exhibited the behaviour of reading the overall text before assessing the writing. A text segment in his stimulated recall interview was found to correspond to this behaviour and it was therefore used to support the aforementioned decision-making behaviour.

3.8.3 Documentary data In this present study, three types of documents were collected: course criteria, course syllabuses and students’ marked writing. The course criteria and course syllabuses were used to provide supplementary background information on each case study participant’s context. They were therefore not included in the analysis process. The students’ marked writing was the primary focus of the analysis because it was a

product of each teacher’s assessment practices. The marked writing was analysed to investigate each teacher’s reaction to his/her students’ writing through the feedback strategies employed. In so doing, content analysis was used as the method of analysis (Schreier, 2012). The marked writing scripts were skim read and the emerging three categories of feedback: indirect feedback, direct feedback and end feedback were identified. To clarify, indirect feedback derives from the teacher’s identification of student writing errors without proposing the correction; direct feedback represents the teacher’s correction of the errors (Ferris & Roberts, 2001; Guénette, 2007), and end feedback demonstrates the teacher’s comments about the overall errors made by the students. I then read the marked writing script again and each teacher’s comments on the writing were classified into the predetermined feedback types. Descriptive statistical methods were used to present the frequency of the feedback type.

3.9 Summary This chapter has provided the rationale for adopting a qualitative research paradigm in this study and has explained the design of the research. In addition, all of the data collection methods including questionnaire, semi-structured interviews, think- aloud protocol, stimulated recall interviews, and documentary data were discussed. This chapter also included a detailed account of the participant recruitment and the data collection procedures in the two phases of this study, as well as details of how data were analysed. Chapters 4 to 6 report the findings of this study.

CHAPTER 4 TEACHERS’ VIEWS ON WRITING ASSESSMENT PRACTICES IN THAI PUBLIC UNIVERSITIES

4.1 Introduction Chapter 3 identified the methodologies employed in the two phases of the present study; namely, questionnaire and case studies, respectively. This chapter reports the findings gathered from the questionnaire. As identified through the literature review in Chapter 2, there is variability among teachers as raters. The literature provides evidence that individual teachers have their own set of views or beliefs which are greatly influenced by their personal and professional experiences, prior education, and other contextual factors such as institutional requirements. Understanding these different views or beliefs can shed light on the ways in which teachers conduct their assessment practices. The questionnaire described in Section 3.3.1 explores teachers’ personal views on writing assessment practices for English major students attending Thai public universities. The aim of the data collected from the questionnaire is to answer the first research question for this study:

How do teachers of writing in Thai public universities view their writing assessment practices?

Prior to its wide distribution, the questionnaire was extensively piloted with numerous representative groups with similar professional backgrounds to the potential respondents. The initial versions of the questionnaire were trialled on three individual PhD students with English teaching experience in Thai universities, a group of five PhD students with experience in teaching English as a second or foreign language, and a group of approximately 20 teachers teaching English in a Thai university. An individual Thai teacher currently teaching English at a Thai university was also asked to trial the web-based version of the questionnaire. After refining and adjusting the questionnaire, it was distributed online to teachers of writing for English major students in 68 Thai public universities via http://www.surveymonkey.com. Sixty-two respondents from different universities completed the questionnaire. It is difficult to state what proportion of Thai writing

teachers this response rate represented for several reasons. First, teachers of writing for English major students in each university are generally subsumed in the larger population of the teachers of English. Moreover, the number of teachers of writing skill in each university is probably not static. They may be assigned to teach other skill subjects in different semesters. In addition, the extended period of data collection covering two semesters makes the estimation of the total number of teachers of writing even more difficult. Second, because the questionnaire was not designed to record the respondents’ workplace, because of ethical requirements of anonymity, the exact number of universities represented by the respondents is not available. To facilitate the exploratory objective of the research, both closed- and open- ended questions were adopted to identify respondents’ demographic data, their teaching and training experiences, and their views on features of good writing, rating criteria, and writing practices, as well as their general views on writing assessment in Thailand (see Appendix A for the questionnaire and see Section 3.4.1 for a detailed account of construction of the questionnaire). Both quantitative and qualitative data were collected through the questionnaire. Quantitative data were analysed using descriptive statistical analysis, and qualitative data were analysed using interpretive qualitative data analysis (see the detailed account of the analysis in Section 3.8.1). The findings from the questionnaire will be presented below in predetermined themes based on the structure of the questionnaire. As some responses to open-ended questions could not be grouped into the predetermined themes they were classified according to themes based on the teachers’ views of different issues in writing instruction and assessment in Thai contexts. These themes will also be discussed. This chapter will first present data relating to the respondents’ general background before discussing their views on features of good writing, rating practices, and writing assessment training. Issues pertaining to writing instruction and assessment in Thailand will also be discussed. Following this, the implications of the key reiterated concepts will be presented in a separate section.

4.2 General Background of the Respondents The first section of the questionnaire focused on the respondents’ demographic data. Descriptions of the respondents’ personal information and professional background are presented in Table 4.1 and Figure 4.1 below.

Table 4.1 Respondents’ demographic information (n = 625) Demographic Information % 1. Gender Male 41.9 Female 58.1 2. Age range 20-25 years 1.6 26-30 years 19.4 31-40 years 33.9 41-50 years 17.7 51-60 years 24.2 Over 60 years 3.2 3. Education qualifications Bachelor’s degree 12.9 Master’s degree 54.8 Doctoral degree 32.3 4. First language Thai 83.9 English 12.9 Other 3.2

Table 4.1 shows female teachers who responded to the questionnaire outnumbered male teachers. This seems to reflect the trend in Thai public universities that more female teachers than male teachers are employed as teachers of English. The age range and qualifications of the respondents varied. A minority of teachers were in

5 It is noted that the number of respondents answering each part of the questionnaire was not static. Some respondents could have found particular parts of the questionnaire not relevant to them and decided to finish answering them before they reached the end of the questionnaire. Therefore, the number of respondents represented in different tables is not equal. 83

the 20-25 and over 60 age ranges and this is most likely due to the universities’ academic staff recruitment policies. Because Thai public universities aim to meet international standards (Kirtikara, 2001), there seems to be a demand to recruit teachers with advanced qualifications, particularly a doctoral degree, to increase the number of quality employees working at the universities. Teachers in the 20-25 age range who may hold a bachelor’s or master’s degree tend to be considered last for recruitment as they have little experience in teaching. Furthermore, because 60 years of age is the typical retirement age for Thai people working in government services, teachers over this age are less likely to be employed in the Thai public universities. Even though it is possible for many Thai public universities to enlist retired high calibre academic staff, this trend seems to be slowly changing. Given high quality academic staff tend to be the most common recruits, their expertise may be reflected by their professional background as illustrated in Figure 4.1.

50 45 40 35 30 Teaching English 25 experience (%) 20 Teaching English writing 15 experience (%) 10 5 0 0-3 years 4-10 years more than 10 years

Figure 4.1 Respondents’ professional background (n = 60)

Figure 4.1 shows the range of the number of years the respondents worked as teachers of English and teachers of English writing at their universities. The minority of the teachers of English have less than four years’ experience in teaching English; whereas the minority of teachers of English writing have more than 10 years’ experience. This finding seems to suggest that English writing was taught by teachers

with less experience in teaching English than teachers with extended experience in teaching English. As writing is considered a complex activity that requires multiple skills to master, teachers of writing may have to put much effort into guiding students to write effectively. Assessing writing is also time-consuming. During the assessment process, teachers usually have to engage in students’ written texts in order to judge its quality and to provide feedback to the students regarding their writing performances (Hamp- Lyons, 1995; Speck, 2000). Because both the teaching of writing in English and its assessment require social and cognitive processing by the teacher, it is not likely to be a preferred choice by teachers with extended teaching experience with heavier workloads. Furthermore, given the learning background and professional experiences of teachers may mediate their views (Borg, 2003), acknowledging the respondents’ backgrounds may enhance understanding of the factors influencing their views on writing assessment in Thailand in various areas as presented below.

4.3 Teachers’ Views on Features of Good Writing In the next section of the questionnaire, the respondents were asked their views on what they considered to be features of good writing. They were asked to rate the degree of importance of seven writing features as shown in Table 4.2. The rating was undertaken according to a 5-point numerical scale ranging from 1 (not important) to 5 (very important). Common key features contributing to effective writing cover various constructs, both at a discourse level such as task fulfilment, communicative effectiveness, register and organisation of writing, and at a linguistic level such as linguistic range, lexis and accuracy (e.g. Barkaoui, 2007; Cumming, et al., 2002; Hawkey & Barker, 2004; Knoch, 2009). Based on these common key features, the seven features in this study ranged from those at a discourse level; namely, cohesion, relevant development of ideas, organisation/overall structure of the text, and task completion, to those at a linguistic level including appropriate vocabulary use, appropriate grammar, and mechanics such as capitalisation and spelling. An overview of the teachers’ responses is shown in the table below.

Table 4.2 Rating average of writing features (n = 57)

Writing feature Not Very Rating important important Average (%) (%) Relevant development of 3.5 0.0 3.5 14.0 78.9 4.65 ideas Organisation/Overall 1.8 1.8 3.5 15.8 77.2 4.65 structure of the text Cohesion 1.8 1.8 3.5 24.6 68.4 4.56 Task completion 3.5 5.3 7.0 31.6 52.6 4.25 Appropriate vocabulary 1.8 1.8 10.5 43.9 42.1 4.23 use Appropriate grammar 1.8 3.5 10.5 42.1 42.1 4.19 Mechanics (e.g., spelling, 1.8 5.3 19.3 31.6 42.1 4.07 punctuation, capitalisation)

As seen in the above table, although most teachers indicated they perceived all features to be very important, when considering the importance of ranking, the three writing features regarded as making the most important contribution to good writing were relevant development of ideas, organisation/overall structure of the text, and cohesion. The mechanical aspects of writing were generally perceived by the respondents as being of least importance. These findings indicate that teachers believe they tend to give greater priority to writing features at a discourse level rather than those at a lexicogrammatical level. These findings appear to support Cumming et al.’s (2002) findings on the characteristics of effective writing. The authors found that the raters of ESL/EFL writing in their study most frequently mentioned rhetorical organisation such as development, cohesion and fulfilment of the writing task, and expression of ideas such as logic, clarity and supporting points to be of most importance. In contrast, accuracy and fluency of grammar and vocabulary use, as well as the amount of written text produced, were less frequently mentioned. Apart from the seven writing features focusing on the text itself, some teachers also provided additional comments about other features they perceived as components of good writing. The mentioned features can be classified into two broad categories: product and process, as outlined in Table 4.3. The product-oriented statements related to writing features apparently represented through the students’ written texts. They also

related to the reader’s judgement of the written texts on the basis of text structures. The process-oriented statements concerned the writing process characteristics the writers employed when producing a written text.

Table 4.3 Additional features of good writing Category Subcategory Example comments Product-oriented Genre I tend to give importance to writing aspects used for grading criteria. However, to what extent each aspect should be emphasised really depends on the genre of writing. [Res056] Genre (text type) [Res58] Transfer from L1 Conscious suppression of L1 negative transfers from Thai [Res40] Stylistic impression Rhetorical flair, aesthetics, enjoyability of the text [Res39] Good attention-getting beginning, Good memorably leaving ending [Res46] Creativity/Originality Absence of plagiarism including copying from the Internet [Res16] Originality/individual thinking [Res39]

Process-oriented Writing process Prewriting, draft, edit [Res31] Dealing with errors 1. Error analysis 2. Its correction and 3. How to correct errors [Res14]

Teachers’ perceptions of the important features of good writing not only concerned writing as a product, they also writing as a process. In terms of writing as a product, rhetorical features were stressed. Being able to produce a particular genre, for example, was seen as a positive feature. The teachers appeared to value written texts when they appropriately corresponded to a particular genre, as each genre has its own writing conventions and fulfils a particular purpose.

The issue of language transfer was also seen to affect good writing. The comments imply that the shift from one language to another language may cause

6 The code ‘Res05’ stands for Respondent 05 representing the extract taken from the respondent numbered 5 in the data list. 87

confusion or misunderstanding in the conveyed message due to different writing conventions and language structures. They also suggest that if students poorly transfer the message in their first language to a second language, readers who do not have shared knowledge of the first language may not understand the message. The students may also fail to communicate in the second language. Students’ creativity/originality and the stylistic impression made on the reader were also considered to be features of good writing. These features may reflect the teachers’ views that students should aim to write in a western style as students in a western classrooms are expected to express their opinions, knowledge and judgements in their writing (K. Hyland, 2003). However, consideration for the students’ creativity/originality and the raters’ impressions of the students’ writing style may lead to subjective assessments based on the teachers’ different conceptions of creativity/originality and their personal impression of the students’ writing. These observations correspond with findings presented in other studies that raters vary in their perceptions of rating criteria due to their different personal and professional backgrounds (e.g., Connor-Linton, 1995a; Cumming, et al., 2002; Eckes, 2008; Lukmani, 1996). In terms of writing as a process, the data show teachers tended to perceive writing as an on-going activity that requires cognitive processing in order for students to plan, draft and edit their writing. Also, the way in which they dealt with errors was emphasised. It may be suggested that teachers considered “cognitive processes as central to writing activity and in stressing the need to develop students’ abilities to plan, define a rhetorical problem, and propose and evaluate solutions” (K. Hyland, 2003, p. 10). Overall, the teachers’ responses to the questions on features of good writing show that they tended to value rhetorical features more highly than lexicogrammatical features. However, they clearly perceived writing to be both a product and a process. The findings from these two data sets suggest overall that the teachers perceived writing as complex and multi-faceted requiring a combination of various macro and micro skills to master. They also suggest teachers perceived the process of effective writing construction as requiring more than language skills. Also needed are skills in “thinking, planning, organising, and linking as well as several levels of language manipulation

(sentence and clause levels as well as word and phrase level, plus spelling, punctuation, etc.” (Hamp-Lyons, 2003, pp. 163-164). The features of good writing identified by the teachers as are supported by K. Hyland’s (2003) five key areas of writing knowledge; namely, content, system, process, genre and context. The teachers surveyed were aware of the need to take these five areas into consideration. Knowledge of content, system and genre were likely to be realised in the focus on text-based features such as the use of appropriate vocabulary, grammar and writing mechanics. Students’ knowledge of process may be demonstrated through the entire writing process; that is, from the beginning when they plan their writing through to the editing and polishing stage. Knowledge of context may be reflected through the expectations of the teachers as readers of the writing. This knowledge probably leads to judgements concerning students’ creativity/originality and the teachers’ impression of the text. A discussion of the way in which the features of effective writing are indicators of how teachers view rating criteria selection is presented in the following section.

4.4 Teachers’ Views on Rating Practices In the section on teachers’ views on marking compositions/essays in the questionnaire, the teachers were provided with 13 statements about assessment practices. They were also asked to indicate the level of frequency with which each statement applied to them. The frequency levels ranged from low to high; namely, ‘hardly ever’, ‘occasionally’, ‘often’, ‘most of the time’ and ‘always’. These levels were classified into two groups: low frequency and high frequency to produce more manageable data. The low frequency group included ‘hardly ever’ and ‘occasionally’; whereas the high frequency group included ‘often’, ‘most of the time’ and ‘always’. The 13 statements were grouped into two categories: rating criteria; and issues concerning reliability and validity of the rating practices. The rating criteria statements depicted types of criteria and the nature of the descriptors employed. The issues of reliability and validity were contained in statements about how teachers conducted assessments and how they took potential biases into consideration. This section presents data based on statements in each of the two categories in combination with teachers’ additional comments related to each category.

4.4.1 Views on rating criteria In relation to rating criteria, teachers were asked about types of criteria they employed and how they used the criteria. As it was possible for teachers to employ different types of criteria in their courses, they were asked about criteria types in terms of criteria provider and scoring methods. Types of rating criteria employed by the teachers in relation to criteria provider are demonstrated in Figure 4.2 below.

Self/peer designed criteria (the criteria that the teachers and/or 5.10% 6.40% their co-teachers create to use in their own course)

Course criteria (the criteria that the teachers’ department gives to the teachers teaching a particular writing course) 50% University’s criteria (the criteria 38.50% that the teachers’ university gives to all university teachers teaching any writing course)

Department’s criteria (the criteria that the teachers’ department gives to all teachers in the department to apply in any writing course)

Figure 4.2 Types of criteria in terms of providers (n = 78)7

It can be seen in Figure 4.2 that the majority of the teachers employed self- or peer-designed criteria. This finding suggests the teachers may have a high degree of autonomy in deciding on the criteria to be applied. The autonomy the teachers may indicate teachers of writing in Thai public universities are autonomous to adjust criteria to suit their perceptions of their students’ abilities or the requirements of the course syllabus. They may be free to develop writing tasks and assessment practices without requiring permission from their departments. Nevertheless, the high level of self- or peer-designed criteria employed in the classrooms may indicate only low level support

7 The teachers might employ more than one type of criteria in the same writing course.

for assessment practices is provided by their departments or universities. It may also be assumed that assessment practice training is not common at the universities. The teachers, thus, are likely to prepare their own criteria to be used in their classrooms. The issue of training will be further discussed in Section 4.5. In terms of scoring method, the teachers’ choice of method is presented in Table 4.4.

Table 4.4 The frequency level of teachers’ views on rating criteria (n = 50) Low High N/A Statements frequency frequency (%) (%) (%) 1. I mark compositions/essays by giving a single 34.0 54.0 12.0 score to the overall quality of writing. 2. I compare the quality of all 28.0 66.0 6.0 compositions/essays in the same task and give scores according to their rank orders. 3. I mark compositions/essays by giving scores 12.0 86.0 2.0 to distinct aspects of writing (e.g., task completion, cohesion, mechanics). 4. I mark different types of compositions/essays 26.0 70.0 4.0 by using different sets of criteria. 5. I assign equal weighting to every feature of 54.0 42.0 4.0 writing.

The data in Table 4.4 reveal the analytic scoring method was employed most frequently by the teachers (Statement 3). This finding suggests analytic rating criteria were adopted the most; whereas holistic rating criteria were employed the least (Statement 1). The greater use of analytic scoring methods may be due to the fact that students in this context are EFL students who seem not to have high level exposure to English in their everyday lives outside of the classroom. To promote their English writing skills most teachers may believe that diagnostic feedback is more important to them than the overall score. Scores at distinct categories such as organisation, content and mechanics can help teachers to see the students’ writing proficiency levels, strengths and weaknesses in writing, and guide them to improve their writing (Bachman & Palmer, 1996; A. D. Cohen, 1994; Hamp-Lyons, 1995; Shaw & Weir, 2007). Holistic

rating criteria, on the other hand, cannot distinguish between the strengths and weaknesses of students who have mid-range scores (K. Hyland, 2003). With the ability to classify students’ performances in distinct aspects, analytic rating criteria, thus, may be preferred by the teachers. However, the teachers did not appear to use just one type of rating criteria in an assessment. Genre or writing text type may be an important indicator of what criteria are to be used. A teacher commented on this issue as follows:

Writing is complicated, and in each writing mode, we need different criteria to evaluate students' works. One approach cannot fit all. [Res34]

In this teacher’s view, a particular type of writing should be assessed using criteria corresponding to it. As writing in different genres is governed by different conventions, the students are required to employ particular forms of rhetorical and linguistic features in order to convey the meaning in a particular genre. Therefore, teachers may consider employing criteria which corresponds to a specific genre as one possible way to assess a specific writing type. The data in the Table 4.4 also reveal teachers tended to value features of good writing differently when they assessed their students’ writing. A comment from one respondent seems to be a clear example of this finding:

As mentioned earlier, holistically, whether or not the task is achieved comes first, followed by organisation, and equally important, language, without which the reader will find it hard to understand what the writer wants to say. [Res21]

This comment shows the teacher weighed the writing features differently, even in holistic scoring. The different weightings may be based on the students’ level of study and the kind of writing tasks the students write. Because EFL writing at the undergraduate level ranges from one paragraph to a five-paragraph essay, students are normally provided with writing prompts (i.e., writing topic or stimulus) to follow (White, 1994). If the students can complete the tasks required of them, their product may be regarded as achieving the purpose of their writing. The conventions both in terms of rhetorical and linguistic aspects are then examined later. These findings may imply that teachers in Thai universities are free to make their own decisions on what criteria they can use in their course. The autonomy they

seem to have may include choices of criteria features and the sources of the criteria. Some aspects of the freedom of the criteria to be used are elaborated on in the teachers’ views on rating criteria in Table 4.5.

Table 4.5 Teachers’ views on rating criteria Categories Subcategories Example comments Focus Appropriateness [Criteria] focus on how language is used, formal or of language informal, appropriate or inappropriate in business area [Res01] Sources of Several sources Criteria used are based on several sources. [Res37] criteria Standardised tests We also look at the criteria from other standardised tests, such as TOEFL and IELTS. We also aim to prepare our students to be ready for the high stakes tests, too. [Res35] Factor Students’ It [how criteria are used] depends on the students' influencing background background, but I tried to achieve the criteria I stated. criteria selection [Res14] Basis of rating Integration of For the aspect of grammar/structure and vocabulary, I criteria subjectivity and think it is subjective, though with criteria, because of objectivity individual (teachers’) writing style. For the aspect of organization/cohesion, it should be more universal. [Res06] Objective: rubric used to assess mechanics of the composition; subjective: overall assessment of assignment - did they get it? [Res15]

As seen in Table 4.5, the teachers appear to have autonomy over the criteria to be used in some areas; namely, focus and sources of criteria. In terms of the focus of the criteria, the features mentioned above may reflect what the teachers perceive as good writing, namely, appropriateness of language in a specific context such as the business sector. This perception seems to indicate the teacher focus on the genre of the writing. It is likely that the teachers incorporate their own preferences towards writing into their teaching practice. Their perceptions towards good writing features appear to influence their selection of criteria, especially for analytic rating criteria, where a set of criteria

comes from good writing features(K. Hyland, 2003). Also, issues of practicality and assessment purpose may be taken into consideration when selecting the criteria. The criteria used by the teachers seem to have emerged from various sources, one of which may be criteria adopted from standardised tests. Moreover, the choice of what criteria to use may have been based on the purpose of their use. As Respondent 35 mentioned, teachers need to prepare their students for a high stake exam so they adopt criteria from standardised tests. Also, students’ backgrounds may be another consideration when the teachers select the criteria. As Respondent 14 commented, he took his students’ background into account when using the criteria. However, it is likely that a tension exists for these teachers between the subjectivity and objectivity of criteria. Even when criteria are used, the teachers probably perceive some features of writing are still assessed subjectively. On the one hand, subjectivity seems to be employed when assessing the overall impression of the text. Grammar/structure and vocabulary are also viewed as features that are assessed subjectively “because of individual (teachers’) writing style” [Res16]. On the other hand, mechanics and organisation/cohesion are seen as objectively assessed. Though the use of criteria can help to ensure a high level of consistency in teacher judgment (Leung, 2004), the criteria may be perceived as problematic by some teachers, as illustrated in the following comment:

I think there is inconsistency, but I do not believe any more than anywhere else. The problem with criteria is that they tend to invite a mechanical application, as if grading papers is a science; whereas I think it is more of an art. Criteria give the illusion of greater fairness and reliability, but I have reservations. As I pointed out, when I apply criteria, students tend to get lower grades than the overall grade I believe they deserve. We have had cases where teachers apply the same criteria very differently. I think the best probable solution is a (somewhat messy) mixture of criteria, judgment and experience. [Res46]

This comment is of interest in relation to teacher raters’ use of criteria. While the use of criteria may stress the reliability of the assessment, the teachers may feel they are lowering the validity of the assessment. Thus, as in the case of Respondent 46, there may be a belief that integration of “criteria, judgement and experience” is an effective way to assess students’ writing. However, this integration may involve subjectivity in

the assessment as teachers employ their judgement and experience. This finding suggests that even though objectivity is required when assessing writing, the presence of subjectivity may be inevitable. Thus, the inclusion of the rater’s personal judgement and experience in the assessment suggests the complexity of using criteria. Merely having a set of criteria may not guarantee the fairness of assessment. Evaluating writing may greatly depend on raters’ decision making while attending to the assessed text. Text structures and textual features apparently shown in the writing tend not to be the only qualities the teacher raters consider during their assessment. The student writers’ “understanding of the world and their place in it, their exploration of ideas, and their feelings” (Hamp-Lyons, 1995, p. 759) are also likely to be qualities influencing the assessment. However, these latter qualities seem not to be clearly identified in a set of criteria. Instead, teacher raters probably rely largely on their impression and personal understandings of the text to assess them. They may experience a tension in balancing their intuitive impressions towards the students’ scripts and a set of criteria. Even experienced raters, although equipped with rating criteria, may mediate judgement on a written text using their intuition and personal interpretation (Lumley, 2005). Experience may accommodate the balance between them. Because the raters have more exposures to assessment, teachers with extended experience in assessing writing may be familiar with the assessment practices they use and are most likely able to maintain more reliable and valid assessment. What teachers perceived they did to conduct valid and reliable assessment is presented below.

4.4.2 Views on reliability and validity of assessment Reliability or consistency in assessment can be secured through the moderation of teachers’ agreement of rating criteria and the practice of rating writing samples before the actual assessment sessions (inter-rater reliability). It can also be achieved when a teacher maintains consistency of his/her own practices (intra-rater reliability). Validity can be reflected in the extent to which teachers assess students’ writing performances based on the rating criteria (Gipps, 2012). The teachers’ views on the reliability and validity of their assessment are presented in Table 4.6. Statements in the table can be grouped into three main categories: teachers’ views on inter-rater reliability through the cooperation with their co-teacher(s) (statements 1-2); how they maintain reliability and validity when

undertaking assessment (statements 3-6); and what they take into account during the assessment in terms of bias that may influence reliability and validity (statements 7-8).

Table 4.6 The frequency level of teachers’ views on reliability and validity (n = 50) Low High N/A Statements frequency frequency (%) (%) (%) 1. Before marking compositions/essays, my co- 26.0 52.0 22.4 teacher(s) and I discuss marking criteria and mark the compositions/essays based on our agreement. 2. Before marking compositions/essays, my co- 40.0 34.0 26.5 teacher(s) and I select and mark a sample of compositions/essays to ensure score reliability. 3. I adjust the criteria during the marking period without 46.0 28.0 26.5 consulting my co-teacher(s). 4. The final score I give to all compositions/essays is 4.0 92.0 4.1 based on the criteria. 5. I compare different pieces of writing to ensure the 14.0 84.0 2.0 score consistency. 6. I read compositions/essays more than once before 8.0 92.0 0 giving scores. 7. The quality of students’ handwriting affects the grade 74.0 16.0 10.2 I give their compositions/essays. 8. Knowing who the writer is affects my decision of 78.0 12.0 10.2 what score to give.

Regarding inter-rater reliability, the teachers believed the degree of cooperation with their co-teacher(s) varied. Most of the respondents indicated that they cooperated with their co-teacher(s) to some degree, while about one quarter of all respondents (22.4% and 26.5%) did not appear to have co-teacher(s). The majority of the teachers with co-teacher(s) revealed they tended to discuss the criteria and assess their students’ writing based on an agreement they have with their co-teacher(s) (52.0%). However, a considerable number of them stated they were not likely to rate writing samples in order to assure score consistency before the actual rating session (40.0%). These findings indicate that only to some extent do teachers engage in the process of discussing the

criteria with their co-teacher(s) before the actual assessment. This discussion may help the teachers to develop a shared understanding of the assessed qualities of written texts. Nonetheless, the lack of scoring practices for score consistency assurance during group moderation may result in high variability of assessment practices and scores awarded to students’ writing (Clarke & Gipps, 2000). Moreover, it may lead to low inter-rater reliability. The inconsistency in the scoring practices could reduce comparability of students’ writing scores across classes and may also result in less than effective writing pedagogy. However, as writing in this context involves a classroom writing assignment, it may be argued that the low inter-rater reliability should not be seen as a problem. The primary aim of the assessment may not lie in score consistency between raters. Instead, pedagogic purpose seems to be the focus (Fulcher & Davidson, 2007). In order to promote students’ writing ability, feedback on each student’s writing could be more important than scores. While conducting an assessment the teachers reported they attempted to maintain reliability and validity in several ways. As illustrated in Table 4.6, if teachers are equipped with a central set of criteria in agreement with their co-teacher(s), then most of them believed they relied on that agreement. Indeed, they stated, they did not attempt to adjust the criteria while rating written texts (46.0%). The great majority of teachers (92.0%) confirmed this practice by stating they usually adhered to the criteria and gave the final score based on the agreed criteria. Intra-rater reliability was also a quality the teachers emphasised. The majority of the teachers claimed to compare different pieces of writing to ensure the score consistency (84.0%). Validity was also secured by the practice of reading written texts more than once before awarding scores (92.0%). This finding shows teachers tended to ensure their scores were appropriate to the assessed qualities. In the qualitative responses, the teachers also mentioned various other techniques they used to maintain the reliability and validity of the assessment, as stated in the following comments:

[How to use criteria is] most reliant on co-teachers. We calibrate our marking using both a series of criteria (grammar, punctuation, vocabulary etc.) and by using student's writing to 'co-mark' to make sure we give similar grades. [Res16]

For the group of 15 students and below, after you mark students' work, it is a good idea to ask one of your writing teachers to mark again. Then compare your marks and the marks given by your colleague to see if the marking is reliable or not. [Res58]

The comments above show the nature of the teachers’ awareness of the importance of reliability and validity. Construct validity seems to be the focus. The teachers aim to assess what they have taught and what is mentioned in the course objectives. Some teachers also see the importance of student writers’ realisation of the features to be assessed in their writing. This practice can help to make the assessment fair and transparent as well as reduce the level of anxiety felt by the students at having to be assessed (K. Hyland, 2003). These issues are illustrated in teachers’ comments below:

I try to match my teaching and cover all criteria relating to the course objectives. [Res09]

[I] explain [to] students about the criteria and make agreement before we start the course [Res58]

In terms of bias, statements 7 and 8 in Table 4.6 suggest teachers claimed to minimise biases while assessing students’ writing. Most of the teachers reported the scores they gave were not affected by the quality of the students’ handwriting (74.0%) or by the fact that they knew the identity of the writer (78.0%). Moreover, the teachers seem to be aware of the potential for bias to occur during assessment. One respondent commented:

While handwriting would only very rarely influence my decisions, the layout of a typed page could have an influence on how I grade a paper, but this is something I consciously avoid. [Res16]

Some teachers suggested they employed several techniques to avoid bias as stated in the following comments:

When I ask my students to submit their writing, I ask them [to] write only their student ID so that I can avoid knowing who the writers are. [Res03]

In my situation, when teachers mark students' writing examination papers, they do not mark their own students' paper. That means, quite often that they do not know their students. [Res05]

Apart from the de-identification of individual students, employing writing format guidelines were also stated as another technique adopted by teachers to avoid bias towards students’ handwriting. One respondent pointed out:

To reduce bias against poor handwriting, we ask the students to type using Times New Roman font 12, double space. [Res58]

Overall, these findings suggest that to some extent the teachers demonstrated an awareness of the issues of reliability and validity with regard to the assessment, both in terms of inter-rater and intra-rater reliability. In general, they appear to maintain intra- rater reliability by employing the agreed criteria and avoiding bias in relation to the physical features of the writing and the identity of the writer. Regarding these teachers’ views, writing assessment seems to be seen as a technical activity in that students’ writing performances can be assessed following explicit criteria. Moreover, agreement among the raters tends to be highly valued. However, there also appears to be an attempt to incorporate the teachers’ personal judgements into the assessment process. One teacher expressed the balance between the teachers’ agreement and their personal criteria as follows:

The first three sources of marking criteria [university’s criteria, department’s criteria and course criteria] would reduce the subjectivity of intra-rater reliability because we had to rely on the same set criteria. However, personal criteria can be added but justification was needed. It's to help ensure that the strategies taught in class were carefully applied in students' writing task, which may be different from other sections/teachers. Above all, the first three criteria may not cover what I had taught them in class; therefore, my own criteria was [sic] also supplemented. [Res07]

While this teacher clearly sees the importance of maintaining reliability-based criteria, the use of teachers’ personal judgement is also seen as necessary to secure validity. In a classroom context where writing is taught to students in different sections/classes by different teachers, variability in teaching and assessing may occur. Even though writing is likely to be assessed for a formative purpose, score

comparability of students across classes may still be required (Gipps, 2012). As teachers are generally variable in their assessment practices, training may be one way to achieve more consistent assessment practices. The issue of training is presented in the next section.

4.5 Teachers’ Views on Writing Assessment Training In the section of the questionnaire on training, teachers were asked about their experiences in writing assessment training. About half of all respondents stated they had participated in either formal or informal training. However, the type of training the teachers received ranged from highly formal such as training for standardised test raters, to quite informal such as teachers’ group meeting and private tutoring on assessment. The length of the training varied from a one hour meeting to a semester-long course. As reported by the teachers, the training focus could also vary from teaching writing in general with an embedded writing assessment component to writing assessment alone. This variation is illustrated below:

The content was based on academic writing with a certain format and creative writing. I found it very helpful for teaching. [Res26]

A 3-month training course on teaching & testing EFL held at [the name of an institute in the United Kingdom], two 2-week training sessions on test construction part of which was on assessing writing organized by Thailand's Department of Teacher Education and [the name of the institute in the United Kingdom] held in Thailand (taught by an expert from UK), & writing assessment training among the faculty members of School of English, [the name of a university in Thailand] to ensure validity and inter-rater reliability on a trimester basis. [Res21]

When asked to consider the usefulness of training, most teachers perceived the training they had to be useful for them in various ways. Their comments on the usefulness of training are stated in Table 4.7.

100

Table 4.7 Teachers’ comments on the usefulness of training Categories Subcategories Example comments Affectivity Confidence It often gives me confidence before I mark the papers enhancement on my own. [Res05] Knowledge General ideas and I think the seminar arranged was only [a] short period gaining concerns of of time and it included more than 1 skill; though, assessing writing there’re enhanced thoughts and concerns in marking writing. [Res04] The course gives me general ideas and some particular concerns that I have to be cautious when marking writing tasks. [Res08] How to teach and It was helpful for me on how to teach and evaluate assess writing writing skill for secondary and higher education students. [Res14] Writing I joined the writing training very often in the past. It correction helped me a lot for writing fixation. [Res18] Benefits to Positive The training was very useful and had positive assessment washback effect washback effects on L2 writing. [Res21] Fair assessment The training resulted in fair assessment since each rater agreed upon the rating criteria/rubrics and compared his/her evaluation of each piece of writing with the other raters throughot [sic] the training and thereby adjusting himself/herself all the time until a satisfactory inter-reliability was achieved. [Res21]

As shown in the above table, the teachers perceived the training experience to be useful in several ways. It may enhance their affective feelings about assessing writing as well as their knowledge of both teaching and writing assessment. The comments suggest greater confidence may be an outcome from the knowledge gained through training. Training seems to result in the development of broad ideas related to teaching and writing assessment, as well as specific aspects of assessment such as “writing fixation”. Moreover, training was felt to have a positive washback effect and to enhance fairness among different raters. As teaching and assessing writing are difficult for teachers, training may help them to gain both principles and practices from professionals in the field. It was

101

suggested that novice teachers may benefit the most. With limited exposure to actual writing instruction and assessment practices, they can learn how to conduct an assessment. One respondent commented specifically on this issue as follows:

Training could help new comers to evaluate written pieces on a very basic manner. We discussed criteria for assessment like language, idea or organisation, all of which appeared helpful for novice teachers. [Res34]

Even though training is perceived as useful by many teachers, how useful it is during actual assessment depends on what an individual teacher gains from it. This view was expressed by one respondent as follows:

For working professionals, usefulness in training may be sporadic, and depends entirely on the outcome of each session, lecture or ‘light bulb’ moment. Sometimes [it is] extremely useful measured by the acquisition of one actionable new idea. Sometimes [it is] not useful at all. [Res48]

Also, an individual teacher’s experience seems to influence their view. If the teacher finds the training relevant to their experience or their current practice, they may have a “light bulb moment” and consider such training very useful. However, teachers may also find the training irrelevant to their practice. A respondent illustrated her negative view on training in the following way:

I found, based on my teaching experience of more than 30 years, the training was useless since Thai students (most of them) had never paid attention to even the abbreviation used in marking their paper. The only way to help them see and understand their mistakes is to explain and point out the mistakes one by one which is not only time consuming but also tiring but it is worth giving such efforts. [Res19]

This teacher appears to find training ‘useless’ in relation to her experience of the students she taught and their lack of attention during learning. Her preference is “to explain and point out the mistakes one by one” to the students. Context tends to be another indicator of training usefulness as this teacher continued to say:

The knowledge could not be applied with Thai students whose English background was totally different from others. [Res19]

102

In general, these findings imply there is a lack of training support from the universities. Most training for the teachers is likely to be organised by external organisations and the content of the training seems to be generic. Therefore, the teachers who do not have prior knowledge of assessment are believed to appreciate it the most while experienced teachers who cannot relate their own experience to the training content may not find it as useful as it should be. Given that training is believed to be very important to rating processes (Weigle, 1994) and to enhance both inter- and intra- rater consistency among raters (Elder, Barkhuizen, Knoch, & von Randow, 2007; Shohamy, Gordon, & Kraemer, 1992), training that focuses on writing assessment practices in a specific context may be more beneficial to teachers. A range of contextual factors related to writing assessment practices in the context of Thai public universities are presented below.

4.6 Teachers’ Views on Writing Assessment in Thailand The last section of the questionnaire asked teachers to provide additional responses in relation to writing assessment in Thailand. The teachers highlighted issues related to the writing problems demonstrated by students and the problems experienced by teachers in terms of writing instruction and assessment, and writing course management. Each of these issues is presented below.

4.6.1 Problems of students as writers Teachers presented several points of view on the problems experienced by students as writers. The problems are categorised into three main areas: knowledge problems, affective problems and factors affecting problems. They are presented in Table 4.8 below.

103

Table 4.8 Teachers’ views on students’ problems Categories Subcategories Example comments Knowledge Writing skills For as long as I've taught English in the upper problem secondary level, and now in the higher education level, the hard obstacles of Thai students ranging from mechanis [sic] to organisation. It's a hard work [sic] for me and I take time to give each student some written corrective feedback on the task. I have them write a diary or dialogue journals to promote their writing fluency. But it doesn't work. [Res22] Low English As a native speaker, we often hear that we should give proficiency high marks for the idea of the paper. However, I usually find that my students have very poor grammar and very poor sentence structure and very poor and incorrect vocabulary usage which severely limit their ability to complete assignments at a higher level. [Res57] Inappropriate The students who study in my course, they [sic] language use due always write in informal language. In their daily life, to negative they love to chat with other friend all over the world influence from but the language used is computer word, emoticon language used on word, and abbreviate word [sic] and so on. Even the Internet though I try to have them aware of how to write in a business area but when they have a test, some of students use word [in an] inappropriate context. [Res01] By the way, the English language on the Internet, particularly in emails and chat rooms, makes my students confused so much. [Res22] Inappropriate Thai students have some difficulties in finding the collocation appropriate collocational expressions that are used in English. [Res35] Absence of The use of reason and/or logical process is often logical absent in essay writing by non-native speakers. argumentation [Res16]

104

Categories Subcategories Example comments Direct translation A major problem with many of my students' writing is from L1 translation. This is extremely difficult to identify, and therefore we do not use it as a criterion for grading. I wish it [translation] could be integrated as I feel that if students 'write from the heart' or 'write freestyle' they often avoid many grammar and vocabulary usage problems probably caused by translation from their mother tongues. [Res16] Lack of I find EFL students have an extremely difficult time knowledge of grasphing [sic] the idea of writing as a “process”. This writing process is critical in thesis statement development and ultimately in writing a coherent essay. [Res23] Affective Uneasiness with I find that Thai students have a difficult time with problem criticism criticism. They are often reluctant to "rewrite" an essay in order to correct mistakes, but I think this is critical and insist they follow through. I have had students say that even though they don't like this practice, it is helpful in the long run. [Res23] Negative attitude Thai students tend to rely heavily on "letter grades". If towards writing they are given a bad grade in writing, they may stop learning it. This also becomes true to my students. They initially had negative attitude towards writing such as it's a painful process, structure is something mysterious for them, etc. With a low score perhaps they earned might interfere their personal attitude of writing [sic]. Therefore, teachers should be aware of it, esp [especially] for Thai students. [Res07] Negative feeling It [marking] affects to [sic] student's psychological towards feeling. If Ss. [students] got [a] lot of markings, correction especially with a red pen, they won't to [sic] participate in class. [Res59] Factors affecting Broad range of Actually, the problems in writing cover a broad range problems factors of factors; culture, lack of reading, training, and supports from the school. [Res35]

105

Categories Subcategories Example comments Direct translation Most of the time, Thai students translate directly from from L18 their L1 into English. So, some collocations are unnatural. [Res35]

As shown in Table 4.8, teachers perceived students to have a variety of knowledge problems at both the linguistic level and the discourse level. The problems seem to emerge from many causes. Interestingly, some teachers perceived Internet usage by students to be a factor complicating their ability to write for academic purposes. Given that writing on the Internet is generally thought to have shifted towards speech-like writing (Weigle, 2002), students may not clearly identify the distinction between writing and speaking in the traditional sense. Moreover, teachers were critical of their students’ low proficiency in English and perceived this to cause extensive problems when attempting to teach writing. As one respondent commented, rather despairingly:

I felt that I had a lot of obstacles of teaching the weak-English competency students. I don't know how to start with them. They are weak of every aspects [sic] of English language skills. I felt that they sound like they did not learn how to use grammar with writing and how to write up phrases or sentences. [Res14]

This teacher’s perception of his students’ low proficiency in English then affected his approach to the teaching of writing. He goes on to explain:

As I formerly stated, a lot of problems occurred in [sic] higher education students in teaching [them a] writing skill because of their low competencies in English usages and learning abilities. I'd like very much to teach them [the same way] as in other universities having students with good English competency to teach. Instead of teaching the processes of writing, I must revise them on grammar or sentence structure and how to use them in writing [at] every level. [Res14]

Clearly, the students’ language ability did not meet this teacher’s expectations and he felt frustrated at having to revise what he saw as basic language points such as grammar and sentence structure.

8 ‘Direct translation from L1’ was identified as both a student problem and cause of the problem based on the focus of the teachers’ responses.

106

In terms of affective problems, the teachers perceived Thai students to hold negative attitudes towards the way the teacher gave feedback. Use of the “red pen” to mark students’ writing, letter grades and instructions about rewriting the task were said to seriously affect students’ feelings. The teachers even suggested that encountering such feedback “may stop learning” and deter students from “participating in class”. Although these affective problems may have been partly caused by the students’ culture, they are clearly identified as presenting challenges to the teacher’s capacity to help the students improve their writing. There appears to be awareness among teachers that the factors affecting students’ writing are multiple, including, as one respondent stated; “culture, lack of reading, training, and supports from the school” [Res35]. An additional factor noted by teachers both as a problem and a cause of problems is language interference involving direct translation from students’ first language to English. All of the mentioned problems are based on the teachers’ perceptions. However, how teachers conduct the teaching and assessment of writing may also contribute to these problems. This issue, as it is raised by the respondents, will be discussed in the next section.

4.6.2 Problems of teachers as raters The problems associated with teachers in relation to assessment practices can be categorised into two domains: rating problems and knowledge problems. The rating problems represent teachers’ views on the problems they have in relation to conducting assessments; whereas knowledge problems relates to teachers’ knowledge. Both of these domains are presented in Table 4.9.

107

Table 4.9 Teachers’ views on teachers’ problems Categories Subcategories Example comments Rating problems Lack of [How I use criteria is] not with total confidence. Some confidence in criteria are not specific enough (i.e., minor/major interpreting grammatical error: which is which?; "few" error vs. criteria "sever [sic] error" receive different score (how much serious is "few" or "several"; not sure if I interpret; for example, "score 4" the same as other teachers. [Res06] Use of criteria I always have a big problem with criteria although I follow them if they are provided. For some reason they always result in me giving a lower overall grade than the writer deserves. [Res46] Time constraints I believe most teachers would like to make their assessment reliable and valid. They would love to talk to their co-teachers to discuss about the criteria and to try marking papers before they really mark them. Yet, it's time constraint that would limit the possibility. [Res24] Knowledge Lack of Based on my experience, we normally have co-teachers problems knowledge of teach in the same writing course. However, some are assessment and [sic] not know how to mark the tasks; they just use the language methods they used to know based on their learning proficiency experience as students. In addition, even though in a course, teachers agree on the criteria, they tend not to mark in the same way. This is partly because they lack the body knowledge of assessment and their language proficiency e.g., They don't know some grammatical points. [Res08]

In terms of rating problems, Table 4.9 reveals the main problem experienced by the teachers is how to deal with criteria in addition to concerns about the fairness of the assessment. Difficulty in interpreting frequency words used in the descriptors, and therefore, increasing subjectivity is said to be one source of this problem. Another problem is the discrepancy between practice and perception, and whether using criteria reflects the score the student deserves.

108

Although criteria as a mechanical assessment method appear to have been adopted, teachers’ subjectivity cannot be completely avoided. While teachers seem to be concerned about issues of reliability and validity, “time constraint” may limit their ability to employ robust practices such as working with their co-teachers. The co- teacher’s knowledge of assessment is also noted as a potential problem, with the implication being that teachers tend to assess students’ tasks based on their experiences as learners. Criticisms are also levelled at teachers’ English proficiency, in that “they don’t know some grammatical points”.

The rating and knowledge problems identified by the respondents are interconnected with management issues and are discussed in the next section.

4.6.3 Writing course management issues The teachers’ comments on management issues relate to aspects of the allocation of writing teachers and prerequisite grammar courses. The allocation of writing teachers appears to be affected by the fact that writing is considered difficult to teach and assess. One teacher with less than one year experience in teaching English expressed this view by pointing out:

Most lecturers here do not want to teach any writing course. The newest lecturers, with no or little experience, have to teach writing to English major students and have to keep teaching it until they are not the new faces anymore. Then, the new lecturer(s) can take over. The department might think that a person's ability to write well (or even all right) can be equated to the ability to teach writing. [Res52]

In this comment, teaching writing is perceived as so difficult that most teachers are resistant to it. Teaching writing, with the long-term effort that has to be made both in teaching and assessing students, may also be seen as a too great a burden by many teachers who have heavy workloads. As a result, the ‘newest’ teachers, who are not in a position to decline, become writing teachers, even though their skills in teaching writing may be limited. If there is no training or support for new teachers they may have to rely on themselves for teaching strategies and materials. Because the teaching of writing “requires extensive and specialised instruction” (K. Hyland, 2003, p. xiii), it may be

109

considered challenging even for experienced teachers. Therefore, a mentoring system or support with preparation may be needed for inexperienced teachers. Even though teaching writing requires experienced teachers with good training, teaching administrators may overlook this issue as the comment above suggests. They may perceive “a person's ability to write well (or even all right) can be equated to the ability to teach writing.” In one teacher’s view, departments could assist writing teachers by providing a prerequisite course in grammar:

There are no grammar-course prerequisites to the writing classes. This is a big mistake. It places the cart before the horse. Courses in composition writing or even paragraph writing become reduced to coursework-review in basic sentence writing and basic grammar review. Therefore, the term “assessment practices” sometimes begs the question: What exactly is being assessed? [Res40]

From the teacher’s point of view, because there is not a grammar course prerequisite there is the situation whereby writing classes become primarily about teaching “basic sentence writing and basic grammar review” rather than the skills of writing as discourse. Respondent 40 perceives there is no opportunity for real assessment of writing as a connected text to take place under such circumstances. According to the above comments, some of the problems linked to writing instruction in Thai public universities are related to the administration of the universities. How teachers are allocated to writing classes and what preparation is offered to both teachers and students are some of the management factors highlighted by respondents. The overall findings from the questionnaire highlight a wide range of perspectives from teachers on writing assessment practices in Thai public universities. The emerging and recurring themes are further discussed in the following section.

4.7 Overall Pattern of Teachers’ Views on Writing Assessment Practices in Thai Public Universities The exploratory findings gained from the questionnaire respondents suggest the nature of the interrelation between the teachers’ views on writing assessment practices in Thai public universities and their views on other aspects including good writing features, rating criteria, training experience and contextual factors. Influenced by Borg’s

110

(2006) model of language teacher cognition as illustrated in Chapter 2, the interrelated pattern of these aspects is illustrated in Figure 4.3.

Text-based, reader-based and writer-based features

Cooperation with co-teachers;

assessment Types, focus, processes; views on selection and reliability and sources of criteria validity of assessment

Teachers’ personal attitudes, beliefs and Teachers’ experience on expectations of training of writing students and raters; assessment; views on support from Teachers’ usefulness of training department/school; views on course aim students’ background; teachers’ prior knowledge Figure 4.3 Overall pattern of teachers’ views on writing assessment practices

As seen in the above figure, teachers’ perspectives in general are likely to indicate their rating practices are influenced and affected by rating criteria, their training and contextual factors. The data suggest the implementation of rating criteria is mediated by what teachers believe to be good writing features. It is also likely to reflect what the teachers have learned during their training as well as contextual factors such as their expectations of students’ writing ability and support from their workplaces. The contextual factors are interconnected with other aspects of the teachers’ views. They seem to accommodate the teachers’ rating practices in relation to conducting reliable and valid assessments. Factors such as the purposes of the course may significantly influence how rating criteria are employed. Some factors, especially the teachers’

111

perceptions of their students’ background, may affect the teachers’ views of their training experience, particularly training usefulness. Considering the components of the overall pattern, various sub-patterns emerge. In order to inform the discussion of the good writing features demonstrated in Figure 4.3, a sub-pattern of teachers’ perceptions of good writing features is presented according to three main types: text-based, writer-based and reader-based. The sub- pattern of perceptions is illustrated in Figure 4.4.

How teachers interpret texts: How students mediate texts: -creativity/originality -writing process (writing, -stylistic impression drafting and editing)

-dealing with errors (editing) Reader- Writer- based based

Text-based

How writing features construct texts:

- rhetorical features (e.g., genre) - lexicogrammatical features (e.g., vocabulary, mechanics) Figure 4.4 Pattern of teachers’ views on good writing features

As shown in Figure 4.4, text-based features may be identified through the language used by writers when constructing their written text. Reader-based features reflect the teacher readers’ expectations of the written text; whereas writer-based features are concerned with the writing process employed by the students. These three features indicate that writing is seen by the teachers as a complex activity involving both sociocultural and cognitive factors (J. R. Hayes, 1996; Weigle, 2002). The three components mentioned above seem to reflect to some degree the writing models developed by Grabe and Kaplan (1996) and J. R. Hayes (1996). While Grabe and Kaplan put emphasis on text-based features, Hayes gives more weight to writer-based features. In Grabe and Kaplan’s (1996) descriptive model, writing is constituted by seven features: syntax, semantic, lexicon, cohesion, coherence, functional

112

dimensions or text at an interpersonal level, and non-linguistic resources such as writers’ background knowledge and logical organisation. These seven components form a written text as a product that employs language at both rhetorical and linguistic levels. Writer-based features can be seen through J. R. Hayes’ (1996) individual- environmental writing model which consists of two main components: the task environment and the individual. The task environment covers social and cultural factors that may influence writing, the written text itself and the physical writing medium such as computer-based innovations. The individual aspect involves the writer’s cognitive and affective factors including motivation, and working and long-term memory. Based on Hayes, all factors included in the individual component appear to interact when writers engage in the writing process. Writers are driven by their motivation to write. They also employ both working and long-term memory in their cognitive processes when interpreting input texts, reflecting on the interpretation, and when producing output texts. What can be clearly seen in the two models above is that writing construction is a complex demanding activity, as numerous writers have stated (e.g Grabe & Kaplan, 1996; K. Hyland, 2003; Weigle, 2002). Writers are required to employ their cognitive knowledge while composing to serve purposes driven by social and cultural factors. As part of the social and cultural requirements, readers’ expectations may have to be met. When the writer’s and the reader’s expectations cohere, the reader can interpret the text appropriately (Weigle, 2002). Thus, reader-based features come into play. Among the three components of good writing features, text-based features are foregrounded in this study. Indeed, because they include linguistic knowledge, discourse knowledge and sociolinguistic knowledge they are considered “essential for writing and none should be slighted in testing writing” (Weigle, 2002, p. 29). Thus, they seem to play a greater part in rating criteria as highlighted by teachers in this study. What teachers believe to be good writing features appears to drive their rating criteria selections and how they use the rating criteria when undertaking their assessment. As the findings in this study were drawn from teachers of writing in Thai public universities collectively, their views on rating practices varied. To explain the variability of their views and practices, Davison’s (2004, p. 325) ‘cline of assessor beliefs and practices’ was particularly useful and the results are illustrated in the table below:

113

114

Table 4.10 Teachers’ beliefs and practices adapted from Davison (2004, p. 325)

Davison’s Assessor as technician Assessor as interpreter Assessor as principled Assessor as arbiter of Assessor as God orientations of the law yet pragmatic ‘community’ values professional View of the Criterion-bound Criteria-based Criteria-referenced, but Community-referenced Community-bound assessment task localised accommodations View of the Mechanistic, procedural, De-personalised, explicit, Principled, explicit but Personalised, implicit, Personalised, intuitive, assessment process automatic, technical, codified, legalistic, interpretative, attuned to high impressionistic, beyond analysis seemingly universalised culturally detached local cultures/ culturally-bound norms/expectations View of the Text- focused Text- focused, but Text and student focused Student-focused Student-focused assessment awareness of student product View of Seemingly unaffected Inconsistencies a Inconsistencies Inconsistencies a Seemingly unaffected inconsistencies by inconsistencies problem, threat to inevitable, cannot problem, threat to by inconsistencies reliability necessarily be resolved validity, assessor training satisfactorily, teachers needs to be improved need to rely on professional judgment View of Need better assessment Need better assessor Need more time for Need better assessors System not open to assessor needs criteria training (in interpreting moderation and (to uphold standards) scrutiny, not criteria) professional dialogue accountable, (to make basis of operated by the judgments more “chosen” explicit) Trend of teachers of writing in Thai public universities Both objectivity and subjectivity are included in teachers’ assessment.

As demonstrated in Table 4.10, the Davison framework categorised teachers as assessors into five orientations along the cline ranging from assessor as technician, assessor as the interpreter of the law, assessor as the principled yet pragmatic professional, assessor as the arbiter of ‘community’ values and assessor as God (for a full review of the orientations see Section 2.6). The findings in this study suggest the writing assessment practices of teachers of writing in Thai public universities tend to lie in the middle range of the cline, covering assessor as the interpreter of the law, assessor as the principled yet pragmatic professional, and assessor as the arbiter of ‘community’ values. The teachers were found to believe in conducting reliable and valid assessments using several approaches such as rating criteria based on agreement they and their co- teacher(s) have and group moderation to ensure score consistency. Though they tend to focus on assessment as a mechanical activity, they also seem to exert their personal interpretations and judgments based on their views of contextual factors such as their knowledge of the student’s ability. When employing criteria, some teachers are uncertain about how to interpret criteria descriptors and appear to be concerned about the balance between the subjectivity and objectivity of their assessment. This view is likely to indicate teachers need more training in assessment.

4.8 Summary This chapter presented and discussed the findings related to the teachers’ overall views on writing assessment practices in Thai public universities gathered from the online questionnaire. Patterns of key reiterated themes emerged. The findings revealed the interrelationship between teachers’ views of the aspects of good writing features, rating criteria, training experience, assessment practices and contextual factors. The orientations of the teachers’ views of writing in Thai public universities also emerged. The teachers were found to be situated in the middle ground of Davison’s (2004) cline of assessor beliefs and practices. However, these findings emerged from questionnaire data which may be regarded as a type of self-report by the respondents. Thus, they reveal what teachers believe they do rather than what they actually do. In order to obtain a clearer picture of what teachers do to conduct assessment, insights about their actual assessment practices are needed. In order to complement the self-reporting of assessment practices highlighted in this discussion, Chapter 5 will present an in-depth analysis of the data pertaining to

115

writing assessment practices undertaken by four teachers in four different universities. Their individual views on writing assessment practices and the processes of their actual assessment practices will be highlighted and discussed.

116

CHAPTER 5 TEACHERS’ WRITING ASSESSMENT PRACTICES: LADDA AND NITTAYA

5.1 Introduction Chapter 4 presented a holistic view of the writing assessment practices of teachers of writing in Thai public universities as revealed in their questionnaire responses. The questionnaire findings suggested overall patterns with regard to the teachers’ interrelated views on good writing features, rating criteria, training experience, assessment practices and contextual factors. This chapter and Chapter 6 extend these findings by focusing on the writing assessment practices of four teachers: Ladda, Nittaya, Rewadee and Sak9, who each worked at a different university in a different geographical region of Thailand. This chapter provides findings based on single-case analyses of Ladda and Nittaya; whereas Chapter 6 reports the single-case analyses of Rewadee and Sak, followed by a cross-case analysis of the four teachers. The cross-case analysis will draw on and discuss the commonalities and differences of the four teachers’ views and practices. Data on the teachers’ individual views and practices were collected using semi- structured interviews, think-aloud protocols, stimulated recall interviews and documents (see Section 3.4 for a detailed description of the research methods). Individual teacher questionnaire responses and interview data provided the teachers’ demographic information and views on their writing assessment practices. They also provided understandings of the contextual factors in each teacher’s situation which may influence their writing assessment practices. The actual assessment practices were investigated through think-aloud protocols and stimulated recall interviews. Documents such as students’ marked writing scripts and course outlines provided supplementary data to the emerging findings. Both the views and practices of the teachers are presented in Chapters 5 to 6 in order to answer the second and third research questions for this study:

 How is writing of a specific text type assessed by Thai teachers in different public universities? What influences their selection of the rating criteria?

9 Pseudonyms are used to address the participants. 117

 How do teachers’ views of writing assessment influence their actual rating practices?

For each of the case studies presented, the four teachers were first interviewed before being asked to conduct the think-aloud protocols while assessing two examples of their students’ writing. The teachers were then asked to participate in stimulated recall interviews using the video recordings of their think-aloud protocols as stimuli. Data gained through the interviews were analysed using interpretive qualitative data analysis (for a detailed account of the analysis process see Section 3.8.2.2). Data obtained through the think-aloud protocols and the stimulated recall interviews were coded, mainly using an adaptation of the coding scheme developed by Cumming et al. (2002) (for a detailed account of the analysis see Section 3.8.2.1). Documentary data were analysed using deductive content analysis (for a detailed description of the analysis see Section 3.8.3). This chapter will present findings from the analyses of the responses provided by Ladda and Nittaya in two separate sections; one section for each teacher. Each section will begin by providing details of the teacher’s personal and professional information. It will then highlight the teacher’s views in relation to major themes concerning writing instruction and assessment as located in their particular context, as well as illustrate their actual assessment practices. Each section will then conclude with a summary of the key issues regarding the teacher’s views and practices.

5.2 Ladda This section will begin with a profile of Ladda, outlining her demographic data and professional information, including her teaching and training experiences based on the information gathered through the questionnaire and interviews. This will be followed by a description of her writing course. The profile and the account of the writing course aim to contextualise the data presentation and discussion which shape the body of this study. The data gathered in relation to Ladda’s views of the writing instruction and assessment processes will then be presented, framed according to two major themes: the nature of students and representations of classroom practices. The data were sub-divided into sub-categories as shown below. In the section on actual assessment practices, decision-making behaviours displayed by Ladda while she was conducting the think-aloud protocols will be highlighted to illustrate her rating

118

processes. The products of her assessment (i.e., students’ written scripts) will also be discussed in relation to her decision making. The last section will summarise key reiterated issues regarding Ladda’s views and practices.

5.2.1 Profile Ladda is a female teacher in the 51 to 60 year-old age range who works at South University10. She holds a Master’s degree from a Thai university and has 35 years of experience in teaching both general English and English writing. Ladda stated that her colleagues called her the “Goddess of Writing” and although she was modest about the reasons for this title, it may be inferred that she was regarded highly by her colleagues as a person with extensive knowledge in how to teach writing to students at the university. Ladda had previously attended two formal training workshops on writing assessment organised by different organisations. One workshop focused on how to introduce “peer check” into the writing program; whereas the other workshop focused on “writing evaluation and assessment” with an emphasis on the assessment of American postgraduate students. Ladda evaluated the usefulness of both training workshops according to the extent to which the content of the workshop was relevant to the students’ abilities in her own context. For example, in the case of the workshop on peer check, she saw that “the students did not have adequate ability to do it”. In turn, Ladda expressed a negative attitude towards both training workshops as she found that the workshops “were not what she wanted” and “not relevant” to the students in her context. She also found that the assessment methods were “inapplicable to Thai students”.

5.2.2 The writing course context As the main teacher of writing in her department, Ladda reported that she taught two consecutive writing courses for English major students. The preceding course which focuses on writing at a paragraph level was convened in one semester and treated as the prerequisite for the course on writing at an essay level convened in the following semester; each course covered 14 weeks of writing lessons. Before the data collection for this study in 2011, there were approximately 40 students in her writing class; due to

10 Pseudonyms are used to address the participants’ workplaces. 119

this large number assessment of students’ writing became a burden for Ladda. In response, she negotiated the course management with the department and requested that her colleagues co-teach one course with her while she taught the other course alone. The class size was reduced to 24-28 students per class. During the semester in which the data was collected, Ladda co-taught one writing course with two native English speaking teachers. The teachers were assigned to the class on the basis of feedback from former students who indicated they preferred to study writing with a teacher who was a native speaker of English. Ladda and her co- teachers shared the same course outline, teaching material, broad criteria representing overall course score allocation as organised by Ladda, and the agreed rating criteria to assess exam papers. However, the teachers also had the freedom to select specific criteria to assess particular writing tasks in their own classes. The students were required to accomplish two main forms of writing assignments: writing task and translation task. In relation to the writing task, the students were assigned to write four assignments of one-paragraph writing illustrating important writing features such as coherence and cohesion, and components of essays such as an introductory paragraph they had covered in the first half of the semester. Then, they were assigned to complete four essays representing five-paragraph writing in four different text types: classification, cause and effect, comparison, and contrast in the second half of the semester. For the translation task, the students were assigned to translate newspaper articles based on the topic assigned by the teacher. Ten translation tasks were assigned in the whole course. All assignments were treated as self-study activities that the students completed out-of-class. Only the writing tasks were exhaustively assessed because they contributed to the main part of the score allocation of the course. Overall, the students were required to submit approximately two assignments per week.

5.2.3 The nature of students The data from the initial interview illustrated the nature of the students and their writing as perceived by Ladda. One category identified from the analysis related to the students’ writing. Ladda identified what she saw as the quality of their writing and the factors influencing the writing quality. The other category related to her attitude towards

120

the group of students as a whole. The summary of Ladda’s perceptions about the nature of the students is presented in Table 5.1 below:

Table 5.1 Ladda’s representation of the nature of the students Categories Sub-categories Sample quotes Students’ writing Writing quality They still make mistakes that they should no longer be making11. Influences on writing There’s an influence from Thai language. qualities

Attitude towards students They don't care about their learning. They just let things go in one ear and out the other.

Students’ writing Ladda stressed several times that the quality of her students’ writing fell below her expectations. From her point of view, the writing included many indications of a gap in the students’ knowledge in relation to both the levels of discourse and grammar usage. In terms of the discourse level, Ladda reported that the students had problems with how to unite their ideas:

When they write, they normally deviate from the core content. The writing is full of irrelevant sentences which they think they can join together. But they can’t. They do not relate to the topic.

Regarding the problems at the grammatical level, Ladda stated the students demonstrated a wide range of writing problems from the implementation of basic grammar rules such as the correct use of subject-verb agreement, articles and word form, to problems with sentence structure:

They still make mistakes that they should no longer be making. For example, with a singular subject he, she, it, they don’t add -s to a present tense verb. When mentioning a plural noun, we need -s for a countable noun such as many books, a lot of students, they don’t add -s, though.

Their usual mistakes are to use a fragment. They made these mistakes most frequently among three to four mistakes which were identified in research on writing: run-on, fragment and comma splice.

11 The extracts were originally in Thai and translated into English by me. 121

Ladda hypothesised about the factors that influenced the quality of the students’ writing. For example, she pointed to a lack of particular skills which she saw as necessary for effective writing:

The point is that Thai students don't have system thinking. When they outline their writing, they don't think systematically. They can't think properly.

They use a talking dictionary. I don't know. It's a small machine like a calculator. We just key in Thai words and English words will pop up. They don't care about parts of speech. When they construct a sentence, they just pay attention to the word they want. For example, for the word 'expand' they key in [the Thai word which means ‘expand’] and the word 'expansion' pops up. Then they construct the sentence 'my mother wants to expansion her business to a foreign country.' It's wrong because 'expansion' is a noun.

Ladda also identified cross-linguistic factors as having an influence on the writing quality of the students:

There’s an influence from Thai language. In Thai, sometimes we can omit a subject that we can understand from the context.

The students’ writing problems and the influences on writing quality identified by Ladda echo the issues stated by many scholars in relation to writing construction theory; that is, the employment of writers’ cognitive processes (e.g. Cho, 2003; Grabe & Kaplan, 1996; Hamp-Lyons, 2003; J. R. Hayes, 1996; Weigle, 2002). Although the student problems identified relate to text-based writing features, they were likely to be driven by the students’ mental processes. Using J. R. Hayes’ (1996) writing model, which points to the individual and the task environment as components for discussion, the students’ problems seem to be situated in the individual aspect which includes factors relating to writers’ internal attributes: motivation, cognitive processes, working memory and long-term memory. When constructing a text, writer will engage in an interaction with these attributes. In relation to this model, the influences on writing quality identified by Ladda seem to imply that the students had inadequate ability to draw on their language knowledge stored in their memory to produce the texts. As an experienced teacher, and one who had taught writing for many years, Ladda had clearly developed quite strong views about the quality of writing produced

122

by her students. As the data above illustrate, these views were primarily negative, suggesting her overall attitude towards assessing the students’ writing was drawn from a deficit perspective. This theme of negativity was seen to continue into Ladda’s comments about the students themselves.

Attitude towards the students Ladda’s attitude towards the students came through not only in her comments but also in the tone of her voice when she referred to the students and their learning ability. She frequently expressed her opinions using a strong, harsh tone of voice that appeared to express her disappointment in the students and their ability. Her frustration in attempting to get her students to understand how to improve their writing was made obvious in statements such as the following:

I don’t know, don’t understand. What happens in their brains? Why don’t they understand?

She also felt her students were unable to notice errors and act on changes, despite the feedback she provided:

Though I marked their writing, they don’t realise anything. They just correct their writing based on my correction. They don't care about their learning. They just let things go in one ear and out the other.

At the same time, her comments also suggested her sense of disappointment that despite all her teaching efforts she was not able to elevate the students to the standards she expected:

I told them not to be a teacher of English. It will lose my face. A teacher like you teaches students wrong and produces a student like you.

In general, however, it was noticeable that when referring to her students during the interview, Ladda rarely expressed positive views, either about their writing or the students themselves. The strong deficit views Ladda held about the students, their language learning, and their writing ability appeared to be developed through her extensive experience. As J. C. Richards (1998) states, “experienced teachers hence have well-developed mental representations of typical students, of typical tasks, and of

123

expected problems and solutions” (p. 75). These deficit views also seemed even to have permeated her whole view of English teaching in general. Ladda hypothesised; “English teaching of teachers of English in this country [Thailand] might have failed since it was first introduced”. Her deficit views also appeared to influence her selection of teaching and assessment strategies, as discussed in the next section.

5.2.4 Classroom practices Ladda’s comments on her classroom practices, which pertained to the class she taught independently, can be categorised into two main themes: teaching strategies and assessment practices. While teaching strategies constitute one key theme, the statements about assessment practices can be further sub-categorised in terms of the rating criteria employed, the strategies used to assess the students’ writing, and the strategies used for the provision of feedback. The concepts which contributed to Ladda’s views of her classroom practices are presented in Table 5.2 below:

Table 5.2 Ladda’s representations of the classroom practices Categories Sub-categories Sample quotes Teaching strategies I assign the students to attend performances such as a concert and Thai classical play. Assessment Rating criteria What criteria? They are quite subjective. I use practices my expertise. Rating strategies The students can rewrite only once. After that, they will receive whatever marks they deserve. Feedback strategies I take the students’ writing with severe mistakes to show in class.

Teaching strategies Ladda appeared to employ teaching strategies based on her own preferences rather than consideration of the needs of her students. When assigning the two forms of assignments: the writing task and the translation task, she demonstrated a propensity to minimise the opportunities for the students to engage in autonomous and self-directed learning. For example, she reported that the students were assigned a specific task, which was to compose five-paragraph writing pieces, each focusing on different text

124

types. These were all based on a single topic selected by the teacher. Ladda also seemed concerned to keep these pieces of writing tightly structured by dictating the circumstances surrounding the construction of the writing:

It's easy to control students' writing when they watch the same movie.

The employment of authentic resources as a pre-writing stage (K. Hyland, 2003) to stimulate students’ ideas before constructing an essay was stated as another method to control the students’ writing:

I assign the students to attend performances such as a concert and Thai classical play.

The desire to control the students’ writing as much as possible was also reflected in the way Ladda described how she set up the formatting requirements for the news article translation tasks:

You have to translate five paragraphs of this news topic. … You can translate any part of a newspaper article. You have to copy the news and attach its original copy… Then, you have to put border around the translated paragraphs in the news article. The translated part needs to be consecutive paragraphs… Write number 1, 2, 3, 4 and 5 to identify the translated paragraphs and you have to translate each paragraph into Thai and number them.

While these instructions may be regarded in a positive light, as a teacher providing her students with very clear and explicit instructions, they may also be interpreted as continuing the theme of an experienced, but frustrated writing teacher wanting to minimise problems in the way her students presented their writing assignments. The control of the writing topics ensured the students wrote about the same content corresponding to a particular text type. This practice was likely to reduce the diversity of students’ writing and would have allowed Ladda to feel confident she was conducting reliable assessments across the students’ writing in the same text type by assessing the same facets of the texts. Due to the shared content of the students’ writing, the assessment of these tasks would then primarily address the linguistic features of the writing (Weigle, 2002). The teaching strategies selected by Ladda appeared to be partly influenced by her deficit views of the students. For instance, the newspaper article translation task was

125

assigned and assessed on the basis of an attempt to intensively engage the students in their learning:

Ten marks are given to class attendance. Previously, other teachers gave these scores through a roll call. How come? The students may come to class absent-minded. They physically sit in class but their minds may be somewhere else. How can I give them scores of ten? They need to put an effort. So I assign the translation task once a week.

Overall, Ladda appeared to be the exclusive decision-maker in relation to course content and, as a result, the needs of the students were only minimally addressed. She appeared to be aware of her own teaching style which she described as “traditional” and “not paying attention to students’ feelings”; however, she claimed it provided positive effect to students’ learning as the students and their friends “can become aware and learn. ... They can be more careful and don’t make the mistakes again”. Her choice of teaching strategies and course content also appeared to influence the way she conducted her assessment.

Assessment practices In relation to assessment practices, every course convened in the university is required to follow the proposed course criteria. These criteria illustrate the scoring allocation for the whole course as indicated in the course outline. However, teachers of the course have freedom to select their own criteria to assess the particular tasks they agreed to assign the students in their classes. In Ladda’s case, the data indicated she did not appear to employ specific criteria when assessing the writing task. Instead, as the following extract suggests, she assessed the students’ writing holistically, apparently using a set of implicit criteria based on her experiences and her sense of her own abilities as a writing teacher:

What criteria? They are quite subjective. I use my expertise.

When asked what criteria she used, Ladda seemed quite dismissive and critical of the criteria. She reported that during the process of assessing the students’ writing the implicit criteria she used considered the relevance of the content to the outline, prompt and topic. In addition, as far as text construction was concerned, it appeared she

126

considered the writing features at both the rhetorical and grammatical levels, as this statement implies:

Mostly, I emphasise organisation. This means I check if there are a topic sentence, sufficient supporting details and sentence connectors. Are there cohesion, unity and content? Um, the content is unity. Organisation is cohesion. They cover everything. Then, I focus on grammar automatically.

Ladda indicated she looked first at the organisation or textual structure of the writing, as in her references to topic sentence, cohesion and unity of content, and that the grammatical aspects of the writing were secondary. However, among these writing features, she believed she placed more emphasis on grammatical aspects. As she expressed, “deeply, I feel that I gear towards grammar”. There was little mention of the mechanics of writing which suggests these were the least attended features. The fact that Ladda asked the students to submit a typed writing task may have reduced the potential for students to make mechanical mistakes and therefore this was not an aspect of the criteria that was important to her. Also, the typed written product could also yield more reliable assessment than the handwritten product (H. K. Lee, 2004). Ladda’s view of what was fair in allocating the students’ scores was seen as very important. When making a scoring decision she appeared to make judgements which ranked each student’s writing quality compared to all students’ writing rather than judging it against a set of criteria. This suggests her emphasis was on norm-referencing rather than criterion-referencing (Gipps, 2012). This practice was evident in her justification of the score given to the students’ writing:

The highest score is 5.5. I can't give the writing 10 because the students have a chance to rewrite while their friends don't.

One of her reasons for the scoring decision as shown in the above comment seemed to be her distrust of the students’ self-revision of their writing:

They may ask a foreigner or their friends to help. I don't know about it because this task is out of class writing. They may ask someone who is good at English to check their writing before submitting it. Shall I give it ten? No way. It's not fair for others who don't have that chance.

127

In Ladda’s comment above, her deficit views of the students seemed to permeate her view of the degree to which she could rely on the writing being the students own work. Her attitude was expressed in other comments such as:

The students don’t realise anything. They just follow what I corrected. They don’t care for their learning.

This view was also reflected in her strategy of collecting the writing and showing the responses with a substantial number of mistakes to the other students in class. Ladda stated that during this feedback process the name of the student who wrote the example being discussed was not kept confidential. Indeed, as the following extract reveals, she directly identified the students who had made the writing errors:

I don’t care about the students’ names. I told them I would not correct the mistakes. I just circle them. Then I ask the writers to classify their mistakes. If they don’t know, I would ask other students. I treat the mistakes one by one. That’s why it takes a lot of time.

The comment above also points to Ladda’s preferred marking strategy. This appeared to be linear in design in that she claimed she focused attentively on very specific features of the writing, as illustrated in the following comment:

I check, read every sentence … I circle and write a list of their mistake types such as grammar, fragment and run-on. Then, I let them find them out in the texts.

Overall, Ladda was likely to place value on assessing “the mistakes” the students made much more than the writing process. For her, accuracy of the texts seemed to equate to the students’ writing quality. As her students were novice writers who had only recently started to learn to write a five-paragraph essay, “identifying and correcting problems in the student’s control of the language system” (K. Hyland, 2003, p. 4) may be a preferred technique for her assessment. Her focus on the students’ weaknesses in the use of textual feature may have been designed to assist them to become aware of their own mistakes and to self-edit their work. More likely however was the possibility that the public display of their writing errors would result in the students becoming even more anxious about their writing ability and increase their negative attitudes towards writing (e.g. Chanock, 2000; Ertmer et al., 2007)

128

Ladda reported she was aware of the students’ negative opinions towards her feedback approach and that she understood the strategy may embarrass the students. In spite of this, she appeared to be resistant to change her approach. She based her reasons for this on feedback she received from previous students which seemed to have led to the perception that this practice helped the students:

My former students said they were used to it. It was fine for them. Mmm, and they felt that they had learned a lot. I don’t know. That’s what the students wrote.

Ladda’s extensive professional experience as well as her status within her program as the ‘Writing Goddess’ may have significantly influenced her decisions to employ her particular personal assessment strategies. They seemed to inform her confidence in conducting her assessment practices. This finding is in line with Sweedler-Brown’s (1985) study which found experienced raters often assess writing critically because of their extensive experiences and because it enhanced their confidence in making decisions during the assessment process. Ladda also appeared to have formulated her assessment practices primarily from a ‘deficit’ view of students’ abilities as well as from a sense of frustration about students’ ability to progress. These factors seem to be the basis of her ‘core’ principles of assessment and, as a result, she may employ them at a subconscious level due to her extensive experience (Richards et al., 2001). Indeed, references to the cognitive and affective processes students might experience in producing a piece of writing are noticeably absent from Ladda comments. Indeed, score reliability can be seen as the major factor foregrounded in her approaches to assessment. When undertaking assessment and feedback, Ladda appeared to predominantly adopt the role of an evaluator who grades the students’ writing rather than as a supportive coach who focuses on students’ learning (Weigle, 2007).

Up to this point the discussion has focused on Ladda’s views and beliefs about her instruction and assessment practices. However, what a teacher might state about their practices may not reflect their actual practices when they teach or assess. In order to further examine the ways in which Ladda operationalised assessment in her instructional work, her actual assessment practices, as illustrated in the completion of an assessment task, are presented in the next section.

129

5.2.5 Actual writing assessment practices Ladda’s actual assessment practices were investigated through think-aloud protocols produced while she was assessing two writing scripts written by two students in the same class. The two writing scripts were selected randomly by Ladda from the pile of 20 overall scripts and she was asked to assess them based on her usual rating criteria. In the think-aloud session, Ladda was asked to identify the student’s actual name and the task title before conducting the assessing each script. The reason for seeking identification of the student name and the task title was to support data collection procedures and data management in the data analysis process. The students’ names will not be disclosed in any part of this study12. The think-aloud protocols were video-recorded and replayed for the subsequent stimulated recall interviews (for a detailed account of data collection procedures see Section 3.7). It has to be noted that during the stimulated recall interview, Ladda was unwilling to recall her thoughts at some points when asked to clarify her thinking processes because she believed she had provided enough data in the think-aloud protocols. The think-aloud protocols were analysed and both qualitative and quantitative analyses of the data will be presented based on the adaptation of Cumming et al.’s (2002) descriptive framework (for a full detailed account of the analysis of the decision- making behaviours see Section 3.8.2.1). This framework categorises the decision- making behaviours into three major themes: self-monitoring focus, rhetorical and ideational focus, and language focus. The self-monitoring focus represents the organisation of raters’ rating styles. The rhetorical and ideational focus reveals the emphasis on rhetorical structures and content or ideas used by the writers. The language focus displays the accuracy and fluency of the language used throughout the writing. Each focus comprises two major groups of strategies; namely, interpretation strategies and judgement strategies. Interpretation strategies describe macro-strategies employed by the raters to comprehend the writing; whereas judgement strategies illustrate how raters evaluate the writing. This section describes Ladda’s practices during the actual assessment and also reports her thinking processes as she conducted the assessments, drawn out from the think-aloud protocols. First, the task type under assessment will be described. Then, her rating processes will be highlighted through an analysis of the sequences and trends in

12 These procedures were applied to all four teachers. 130

the decision-making behaviours Ladda exhibited. To complement understandings of the actual assessment practices, the product of Ladda’s actual assessment practices (i.e., marked writing scripts) is also discussed. The marked written product was analysed by examining the comments Ladda included on the script (for a complete description of the analysis see Section 3.8.3). These issues are highlighted below.

5.2.5.1 Task profile The task type Ladda assessed for the purpose of data collection in this study was, as she termed it, the writing of a “cause and effect” text. She assigned the students the task of constructing a five-paragraph expository essay on a single topic entitled; “ID cards for children”. In their essays the students were required to submit an outline stating the type of introduction they employed, a thesis statement, three major points and their minor details, and a concluding sentence. The two writing task student samples Ladda randomly selected to assess during the think aloud are shown below:

131

Text 5.1 Outline13

Topic: ID cards of children The introduction paragraph: Anecdote Thesis statement: Maturity, migrant workers and wasted money are the reasons why children shouldn’t hold the ID cards. A: no have enough maturity - Children lose their ID cards. - Children think ID cards are toys. B: migrant workers - state welfare - try to get hold of the illegal cards C: wasted money - parents - Thai Government Conclude: Summarise ID Cards of Children

When I was 15 years old, I went to Phatthalung District Office for getting my first ID card. However, my younger brother who is seven years old now has to have ID card because of The ID Cards Act 2011. In my opinion, maturity, migrant workers and wasted money are the reasons why children shouldn’t hold the ID cards. First, the children don’t have enough maturity. For this reason, they don’t know the ID cards are very important. It’s easy for them to lose ID cards. They can’t be trusted to carry their ID cards and not lost them. They may think the ID cards are toy since the ID cards look like Yo-Gi-Oh cards that are popular for the children. Consequently, the ID cards are worn out easily because of playing of children. Second, there are many migrant workers in Thailand. They want to get state welfare, such as education, healthcare and housing. Therefore, the migrant workers try to get hold of the illegal cards. If the children must have the ID cards, it’s not difficult for the young migrant workers to take this opportunity to get the ID cards. Finally, ID cards of the children are wasted money. Parents normally keep children’s birth certificates and household registration, so the children don’t need to have the ID cards to save parents’ money. According to Thairath Newspaper, Thai Government allocates budget for the ID cards of the children 191 million baht. Therefore, the ID cards of the children are very wasted. In short, there are three reasons why the children don’t need to have the ID cards. One is that children have no enough maturity. Another is that it’s easy to get the ID cards. The last is that it’s useless money.

13 The texts were typed exactly as shown in the original writing scripts. 132

Text 5.2 The ID card for Children

Thesis statement: Convenient, saving time and helping authorities to identify are 3 reasons why children should have ID card Introductory: General to specific A: It is convenient in one card for getting sate welfare - Health service - Education B: Saving time without taking document - Registration - Birth certificate C: It helps authorises to identify when the bad event happen to children - Be lost - Death Concluding: summarise main point The ID cards for children

There are 65 million populations in Thailand. The number of Thai children population is 300,000 people who study in school. Thai government focuses on them because they are important human resource in the future. Therefore, Thai government set a children’s ID card policy that is useful. Convenient, saving time and helping authorities to identify are 3 reasons why children should have ID card. First, it is convenient in one card for getting sate welfare. The state welfare is health service. Instead of taking many documents to hospital, children take only one ID card to show authorities for getting health service. It is easy way and not complicated. Thai government support many things of education in school, such as textbooks, student uniforms and stationeries. The children show their ID card to receive these things from governmental authorities. It is convenient because the information of children has contained in this card already. Second, it saves time. Normally, their teachers want to know more information of children registration. The teachers always call to their parents, so the teacher must wait for their parents taking the children information for long time. It wastes precious time of them. If the teachers want to know the children information, they take ID card to show the teachers. The teachers know about their information suddenly from ID card verifier. It can save the time, furthermore, the authorities want to know the information from birth certificate when the children admit in hospital. If their parents forget to bring the birth certificate, the authorities can take the ID card from children. It saves the time. Third, it helps the authorities for identifying children when the bad event happens to them. The most problems of parent are missing from their children. Because of no ID card, the authorities, such as police and social worker don’t know about children information. Their parents don’t get answer from authorities. On the other side, the

133

children hold ID card when they lost from the parents. The authorities can identify their children and tell their parents to take their children back from assistance centre. Moreover, the authorities know the identifying from children ID card when they die. It is useful to tell their relation correctly and help the authorities work easily. In conclusion, there are 3 reasons why children should have ID card. One is convenient when they get the state welfare from government. Another is storing the time of their parents and their teachers. The last is assistance the authorities who can identify the children from bad incident.

5.2.5.2 Rating processes The data presented below illustrate Ladda’s rating processes through the decision-making behaviours displayed during the process of marking the two scripts. The rating processes were analysed in terms of sequences and trends in the decision- making behaviours. The sequences were drawn from Ladda’s continual behaviours when she attended to different sections of the scripts; whereas the trends represent the overall frequencies of the behaviours exhibited by Ladda. As mentioned earlier, Ladda asked the students to submit a writing task by attaching the essay outline they had prepared with the main text they composed. During her assessment of the task, Ladda attended to the two parts of each essay in the sequence they were presented (i.e., focusing on the outline before moving on to the main text to make scoring decisions). When attending to the outlines of the two scripts, Ladda’s practice appeared to be to mediate her assessment by reading the relevant text segments in the outlines without referring to the main text. The kind of process she used to assess the outline is shown in the following commentary on Student 1’s writing script:

The topic is ID cards for children14. Introduction, new paragraph, this is in a form of anecdote. Thesis statement: maturity, migrant workers and wasted [sic] money. Wasted, how come? It’s a verb. We need ‘waste of money’. [Ladda corrects the language by crossing part of the word ‘wasted’ and adding the word ‘of’15.] Is it better? It’s a noun. A, B, C are reasons why children should not have ID cards. Major supporting detail A, no have enough

14 Phrases in italic bolds represent text segments which were read by Ladda. 15 The discussion of Ladda’s marking on the text will be presented in Section 5.2.5.3 below. 134

maturity. Minor is Children lose their ID cards. Children think ID cards are toys. For major B, it’s migrant workers. Minor to explain it is state welfare and try to get hold of the illegal cards. For major C which will be the third part of the text body, she stated ‘waste of money’. [The phrase ‘wasted money’ was written in the script, so Ladda corrects the language by crossing out part of the word ‘wasted’ and adding the word ‘of’.] It is about parents and Thai government. Then, conclude will be summarise [sic].

In this extract it is evident Ladda mainly employed interpretation strategies that focused on rhetorical and ideational aspects. The comments show she went through the outline primarily to ensure the student had identified the topic and addressed topic development. After reiterating the topic she set the class, she noted the student had chosen to begin with an anecdote, but made no further comment on this choice. She then continued by itemising the key points and their sub-points the student had nominated to become arguments for the text. In reviewing the outline, she gave no indication of her actual evaluation of the structure of the text from the rhetorical point of view. In contrast, when focusing on the lexicogrammatical aspects, she adopted a more evaluative tone; “Wasted, how come? It’s a verb. We need ‘waste of money’”. Overall, her approach in this first set of comments suggests she preferred to focus on correction of students’ grammar, as it is this part of the outline that received the most attention. Ladda’s concern with treatment of the grammatical aspects of writing also emerged when she attended to the outline of Student 2, as shown in the extract below:

The same topic, the ID card for children. This is the outline. Thesis statement, convenient. [Ladda circles the word ‘convenient’.] Here it comes, part of speech. This is an adjective. How can it be used as a reason? Saving time and helping authorities to identify are three reasons why children should have ID card. [Ladda adds -s to the word ‘card’.] He needs to use a general to specific pattern for the introduction. OK. Major A convenient, it is convenient in one card for getting state welfare. They are health service and education. The second topic is saving time without taking document. Should it be ‘without carrying’? It shouldn’t be ‘taking’. Without ‘carrying other’. [Ladda corrects the language by crossing out the word ‘taking’ and writing ‘carrying other’ to replace it.] For example, registration household registration and birth

135

certificate. The third topic is It helps authorise. How come? ‘Authorise’ is a verb. It must be ‘help authority’. [Ladda circles the word ‘authorises’.] To identify when the bad event happen to children. Is it better to say ‘when something bad’? [Ladda crosses out the word ‘bad event’ and writes ‘something bad’ to replace it.] He wasn’t good at vocabulary selection. Lost and death. Conclude by summarising main point. [Ladda adds -s to the word ‘point’.]

The extract shows that although Ladda’s concern here was to review the students’ outline for the written task in the same way she did for Student 1, she, in fact, attended more extensively to the language aspects of the script. The common strategies Ladda employed included identifying errors and proposing correction together with correcting the language errors she detected. The greater focus on the language aspects in the second script compared to the first script suggests Ladda perceived the second outline to be less effective than the first. Her treatment of the language during the assessment process led to an evaluative remark regarding the student’s writing problems, which she summarised as; “wasn’t good at vocabulary selection.” This conception of the student’s weakness was likely to influence her judgement of the text in the scoring decision at a later stage. This conception also seems to reflect the notion of ‘text image’ referred to in Freedman and Calfee’s (1983) model of rating processes. Text image suggests raters judge students’ writing based on the image of the text they construct through reading and the impressions they then store in their working memory. When attending to the main text in both scripts Ladda went through the writing by focusing on certain text segments and by treating each of them independently. She engaged again in the interpretation strategies she employed when focusing on the outlines such as reading the text segments, identifying topic development and identifying grammatical errors. However, she also employed judgement strategies for both the rhetorical and ideational focus, such as assessing reasoning, logic and proposition, as well as for the language focus, such as proposing correction to the lexicogrammatical mistakes. Among the various judgement strategies she employed, the behaviour of writing or marking a comment in the texts was found to be predominant. This behaviour was likely to reflect the view that the purpose of classroom writing assignments is to help students learn, and that comments as feedback are important to improve students’ writing ability (Coffin, et al., 2003; K. Hyland, 2003). Therefore, the

136

comments or corrections Ladda placed on the text suggest she aimed to provide the students with feedback for their writing. Examples of the judgement strategies employed are illustrated in the extracts below.

Student 116

When I was 15 years old, I went …(name of a district office)… for getting my first ID card. However, my younger brother who is seven years old now has to have ID card because of The ID Card Act 2011. In my opinion, maturity, migrant workers and wasted money are the reasons, why children shouldn't hold the ID cards. [Ladda corrects the phrase ‘wasted money’ by crossing -d in the word ‘wasted’ and adding the word ‘of’.] Well, OK. First, this is a body, first. This is OK; there’s a transition, first. [Ladda underlines the word ‘first’.] The children don't have enough maturity. Mmm, OK. This is a topic sentence; ‘maturity’ is a controlling idea. Let’s see how it is modified. For this reason, they don't know the ID cards are very important. It's easy for them to lose ID cards. They can't be trusted to carry their ID cards and not lost them. Lost them? -st, how come? It must be -e, present tense. [Ladda corrects the word ‘lost’ by taking -t out and replacing it with -e.] They may think the ID cards are toy. -Sss [Ladda adds -s to the word ‘toy’.] Toy, see that? Toy is a countable noun. Why isn’t there -s or an article, either of them? What is this? [Ladda writes ‘a’ to identify an article error.] Since the cards look like Yo-Gi-Oh cards that are popular for the children. What is Yo-Gi- Oh? Why isn’t it modified? What is it? [Ladda circles the word ‘Yo-Gi-Oh’ and writes a question ‘what is it?’ at the margin.] How can I know? I’m not a child. There should be an explanation of what Yo-Gi-Oh cards are and how ID cards look like the Yo-Gi-Oh cards.

Student 2

Second, it saves time. Normally, their teachers want to know more information of children registration. The teachers always call to their

16 Student 1 represents the think-aloud protocol while Ladda was attending to Text 5.1 as shown above; whereas Student 2 illustrates the think-aloud protocol as she assessed Text 5.2. 137

parents, so the teacher must wait for their parents...for their parents taking the children information for long time. Oh, this is a direct translation from Thai. It must be written ‘the teachers must wait’…’have to wait’ [Ladda crosses the word ‘must’ and writes the word ‘have to’ to replace it.] No need for past tense. ‘Have to wait’. There’s no adverb of time to modify it. Is it better to say ‘so the teachers’ or the frequency ‘so the teachers sometimes’? [Ladda adds the word ‘sometimes’ to the sentence.] Why not using it? I have taught about it? Sometimes they ... for their parents ... ‘Taking’, how come? ‘Bringing’ [Ladda crosses the word ‘taking’ and writes ‘brining’ to replace it.] the children information for ... It wastes precious time of them. If the teachers want to know the children information, they … Oh, there should be a gerund to modify adverb of manner. [Ladda adds the word ‘simply’ to the sentence.] Take ID card to show the teachers. Is it ‘they ... simply ... show’? [Ladda crosses the word ‘suddenly’ in the script.] ‘Show ID card’ is reluctant; ‘to the teachers’. The teacher know about their ... Oh, redundant. The teacher will know about their information [Ladda adds the word ‘will’ to the script.] Suddenly; is ‘quickly’ better? [Ladda adds the word ‘quickly’ to the script.] ‘Quickly from the ID card verifier.

As can be seen in the two extracts above, Ladda’s overall area of focus when assessing the two scripts was similar to the areas of focus she gave to the outlines; that is, she placed greater emphasis on the rhetoric and ideas in Student 1, and focused more on the language in Student 2. When focusing on the rhetorical and ideational aspects, her typical behaviours were reading a certain text segment, and identifying the rhetorical structure to understand its function in the text before providing an evaluation. It was also found that when interpreting the text segments Ladda related her own personal experience with the student’s proposition. This is illustrated in the Student 1 assessment when she made a comment about ‘Yo-Gi-Oh cards’; “how can I know? I’m not a child.” This finding lends support to findings in previous studies that rater personal background is one factor influencing the rating processes (e.g., Barkaoui, 2010b). It is evident in the extracts there was a discrepancy in Ladda’s practice when she evaluated the rhetoric and ideas of the two texts. In relation to the script from Student 1, she provided an explanation for the errors made and suggestions for correction; “What is Yo-Gi-Oh? Why isn’t it modified? What is it?” However, in relation

138

to the script from Student 2, she briefly articulated her evaluation of a certain text segment without providing further justification; “Oh, redundant”. This discrepancy suggests inconsistency in Ladda’s own practice when focusing on the rhetoric and ideas during an assessment. Regarding the language aspects, Ladda reacted to the language errors instantly after she read the text segments. When she proposed correction to the errors, she typically wrote her comments or placed the correction in the text. Her rapid reaction to these errors is likely to result from her expertise in teaching and assessing writing. This finding appears to support her claim during the interview that she “focused on grammar automatically”. It also coheres with findings from other research studies that experienced raters are likely to give more emphasis to the language aspects of the writing (Barkaoui, 2010a; Cumming, 1990). The focus on the language errors appears to confirm Lumley’s (2002) claim that experienced raters frequently exhibit ‘classifying errors’ behaviours because they rely in their rating on observable writing features, of which one obvious source is the errors. Therefore, the language errors may be seen by Ladda as the clearest evidence of students’ writing quality. The two extracts above also suggest Ladda integrated her decision making in an interactive way by gathering, interpreting and evaluating information. Her decision- making behaviours were not fixed or systematic; rather, they were likely to vary depending on her attention to the texts. She interspersed her decision-making behaviours among the self-monitoring focus, the rhetorical and ideational focus, and the language focus. She appeared to formulate her strategies when focusing on each part of the writing with confidence; hesitation in her decision making was not evident. While assessing the texts, Ladda also took contextual factors such as her personal experience with the writing class and the writing content that had been taught into consideration. This finding appears to align with the finding in Cumming’s (1990) study that experienced raters are likely to integrate “their interpretations and judgements of situational and textual features of the compositions simultaneously, using a wide range of relevant knowledge and strategies” (p. 44). Ladda reported earlier in the interview that she intuitively scored the students’ writing without using explicit criteria. When it came to giving the two scripts a final score, Ladda’s comments were:

139

Student 1 Overall, it should … But he used correct grammar. He doesn’t have any problem about sentence structure. He used appropriate transitions. But the content is problematic. Grammar is little problematic. Mmm,7.5 then.

Student 2 Oh, rewrite. The content is wrong; grammar is wrong. This student had many problems with the content. There’s direct translation from Thai. No, this student has to rewrite. I’m not satisfied.

As shown in the extracts above, Ladda’s general process for making scoring decisions was to summarise her judgements of the texts collectively before articulating the overall results. When attending to Student 1, she judged the text based on the quality of textual features illustrated. The final result tended to be formulated through the attributes of the rhetorical and ideational, and language aspects of the text. Regarding the product from Student 2, Ladda also considered these attributes when making a scoring decision. However, it is clearly seen that she took the crosslinguistic factor from Thai and her overall impression of the text into consideration. Even though Ladda appeared to base her overall decision making on both rhetoric and ideas, and language aspects, her scoring decision appears to be derived from the greater emphasis she placed on language errors in the students’ writing. This finding is in line with the separate finding in studies by Sakyi (2000) and Eckes (2008) that experienced raters paid great attention to the language aspects. One of Sakyi’s key findings on the reading style used by experienced raters who predominantly employed holistic rating criteria was a “focus on errors in the text”. The raters focused on appropriateness of the language used and as a result the frequency of grammatical errors influenced their judgements. Similarly, Eckes discovered that one group of experienced raters in his study focused strongly on the correctness of the language when assessing writing in a large-scale assessment context. The emphasis on linguistic accuracy seems to be influenced by the raters’ training and teaching experiences (Barkaoui, 2010a; Cumming, 2001; Erdosy, 2004; McNamara, 1996; Song & Caruso, 1996).

140

Even though Ladda displayed a variety of decision-making behaviours when attending to each part of the two scripts, her overall decision-making sequence may be illustrated as below:

Attending to the outlines of the essays - Engage in reading text segments and mainly focus on interpretation strategies for rhetoric and ideas

Attending to the main texts of the essays - Engage in reading text segments and mainly focus

on interpretation strategies for rhetoric and ideas and language.

Making scoring decision - Summarise, distinguish or tally judgment collectively and articulate overall results

Figure 5.1 Sequence of Ladda’s decision making behaviours

Overall, Ladda engaged in a complex and interactive process of decision making while assessing the outlines and the main texts of the two scripts. Her decision-making behaviours were highly interdependent. As they exhibited themselves in the data they appear to be greatly influenced by her subjective views on different parts of the text construction and these views progressively formulated her final scoring decisions. As the two scripts were perceived as having different writing qualities, the decision-making behaviours displayed in each text tended to vary. In order to provide triangulation on the patterns identified in the qualitative data, the frequency of individual behaviours for each focus were also quantitatively summarised to illustrate the trends in Ladda’s decision making as shown below:

141

Table 5.3 Frequencies of overall decision-making behaviours exhibited by Ladda Student 1 Student 2 Freq. % Freq. % Freq. % Freq. % A. Self-monitoring Focus 50 50.51 96 52.46 1. Interpretation Strategies 29 29.29 61 33.33 2. Judgement Strategies 21 21.21 35 19.13

B. Rhetorical and Ideational 35 35.35 20 10.93 Focus 1. Interpretation Strategies 27 27.27 18 9.84 2. Judgement Strategies 8 8.08 2 1.09

C. Language Focus 14 14.14 67 36.61 1. Interpretation Strategies 6 6.06 19 10.38 2. Judgement Strategies 8 8.08 48 26.23

Total 99 100 99 100 183 100 183 100

As Table 5.3 shows, noticeable differences can be seen in the decision-making patterns for each text. Ladda exhibited approximately twice as many overall decision- making behaviours when assessing the second writing script as the first one. When frequencies of decision-making behaviours for each focus are considered, the behaviours in the rhetorical and ideational focus were displayed more frequently in the first script than the second one. However, a contrast was found regarding the behaviours in the language focus which were exhibited more frequently in the second script. The writing quality of each script, as reflected by Ladda through the scores she assigned, is likely to account for these differences. This differentiated focus on the writing quality is coherent with Cumming et al.’s (2002) study which found raters attended to the rhetoric and content more exhaustively in high proficient writing, but focused more on the language when judging low proficient writing. Table 5.3 also reveals Ladda tended to be self-consistent in her assessment when considering the frequency of decision-making behaviours in each group of strategies. She displayed more interpretation strategies than judgement strategies when self- monitoring her rating and when attending to rhetoric and ideas. However, when attending to the language, she focused more on evaluating than interpreting the texts. This finding seems to echo her own view, that she was “deeply geared towards grammar”. Ladda’s different emphasis on the strategies in each focus appears to slightly contradict the finding from Barkaoui (2010c) that raters in his study, employing holistic scoring, attended more to interpretation strategies than judgement strategies in the three

142

focuses. This contrast may inform the nature of variability between human raters in the process of assessment. In general, Ladda’s overall rating processes reflect her self-reported views of her assessment practices, particularly in her emphasis on textual features. In the next section, to complement understandings of what Ladda actually did during her assessment of the two scripts, the product of her writing assessment practices as revealed in her marking of students’ writing scripts is highlighted.

5.2.5.3 Product of writing assessment practices Following Ladda’s assessment of the scripts written by all students in her class, the marked writing scripts were collected. The collected scripts comprised the two scripts marked during the think-aloud protocols as well as the 18 scripts Ladda marked alone. The marked scripts were analysed in terms of feedback types in order to reveal the writing features Ladda attended to. Three types of feedback were identified: indirect feedback, direct feedback and end feedback. The indirect feedback shows the identification of writing mistakes without providing the correction; whereas direct feedback represents the correction of the errors (Ferris & Roberts, 2001; Guénette, 2007). End feedback indicates Ladda’s comments about the overall error types made. Examples of the feedback given in the texts are displayed below for Student 11 and Student 19:

Text 5.3 Student 11 v v The last reason, I think that if there is a corruption happens X who will be the one that get a fund? There are millions of children cross country all must to get the identity card cannot X X deny that those actions do not require money. Government fragment can smuggle a huge amount of money from those children, and sure it’s no one will know if it true. It is highly questionable that why need to change can already good X ? system if it were not for corruption.

143

Text 5.4 Student 19 have First of all, children who must make identification card are burden of their parents. Parents must take their children to - What are a make identification card at district office. Parents who live they? - Why? far from district office will waste time and increase expense.

The extracts above show the indirect feedback and direct feedback provided by Ladda on the texts. The indirect feedback included question marks, underlines and circles to locate the errors in relation to the rhetoric, ideas and language. There may be two explanations for the employment of indirect feedback. First, the errors made may have been too complex to easily classify into types (i.e., rhetoric, ideas or language). Second, Ladda may have wanted to challenge the students to classify and correct their own mistakes as part of a self-editing initiative. The indirect feedback also involved codes for correction such as ‘V’ for a verb error in Text 5.3, and clarification questions such as; ‘What are they? Why?’ in relation to content of the texts in Text 5.4. The direct feedback represented Ladda’s correction to language errors such as the deletion of the question mark and the word ‘all’ in Text 5.3, and the replacement of the word ‘have’ and the insertion of an article ‘a’ in Text 5.4. Apart from these two types of feedback, Ladda also provided end feedback to show a summary of the error types made by the students such as a list of ‘grammar, part of speech, plural, fragment, run-on, pronoun, content’ for Student 11 (see Appendix O for a full sample of the actual marked writing scripts). Among the 20 writing scripts overall, seven marked scripts were found to be scored while the remaining 13 marked scripts were required to be rewritten. The mean scores of all feedback types made by Ladda are illustrated in Table 5.4 below:

Table 5.4 Mean scores of feedback types Types of Indirect Direct End writing feedback feedback feedback Total scripts Scored 3.00 11.29 0.00 14.29 (n = 7) Rewritten 15.08 14.69 0.69 30.46 (n = 13) 144

Table 5.4 shows Ladda provided more feedback of all types in the rewritten scripts. This indicates that the writing from this group was perceived by Ladda to be more problematic than the writing in the scored scripts. The predominant writing features found in the feedback, particularly the direct feedback, were those pertaining to language aspects. This finding supports Ladda’s emphasis on the evaluation of language aspects when conducting her assessment of Student 2’s script (which was perceived as low proficient writing as discussed in the section above). Indeed, Ladda’s comments on the students’ writing showed she addressed accuracy of the written products for both rhetorical and lexicogrammatical features of the texts. Her scoring decisions appeared to result from a combination of the errors in terms of language, rhetoric and ideas. If the writers made more error types, their writing was likely to receive a low score or had to be rewritten. This practice was also reflected when Ladda made a scoring decision for the two texts discussed in the previous section. Overall, Ladda’s actual assessment practices suggest she emphasised writing as a product. While conducting assessment, her decision making appeared to be extensively influenced by personal and professional experiences. She also took into account contextual factors such as reflection on course content. In order to synthesise the discussion of the data presented for Ladda in this case study the major themes and trends concerning her views and practices in relation to writing assessment are summarised in the next section.

5.2.6 Summary of Ladda’s views and practices Ladda’s writing instruction and assessment appeared to be largely influenced by two major contextual factors: her extensive professional experience, and her views towards the nature of the students. Even though the formative purpose of the writing course was maintained by employing writing assignments that reflected the course content, the pedagogy and assessment strategies were predominantly product-oriented. Based on the data presented, Ladda assigned the most importance to the written product and its accuracy, mainly at the grammatical level, at the expense of the writing process which was very little addressed. Students’ writing was also assessed subjectively against implicit holistic rating criteria which she appeared to confidently draw on from her many years of experience as a writing teacher. Although score reliability was seen as crucial and was greatly

145

emphasised in her assessment, it was her personal judgement that inevitably drove her assessment and feedback practices. While conducting the actual rating processes, her overall rating sequences appeared to be linear as she focused on writing components (i.e., outline, main text and scoring decision) as shown in the texts. However, when attending to each component, the displayed decision-making behaviours were found to be recursive and interdependent. Ladda’s reported practices appeared to be largely consistent with her actual practices during the think-aloud procedure, particularly in relation to the provision of feedback on the writing. All of these issues will be revisited and discussed further in Section 6.4.

5.3 Nittaya This section presents Nittaya’s views on writing instruction and assessment practices as well as the actual assessment practices she employed. It will first contextualise the data presentation and discussion by describing Nittaya’s profile including her demographic data and professional information (i.e., teaching and training experiences) based on the information gathered through the questionnaire and interviews. Following this, the data gathered relating to Nittaya’s views of the writing instruction and assessment processes will be presented. Her views will be framed according to the theme of classroom practices which was sub-divided into subcategories as shown below. In the section on actual assessment practices, Nittaya’s decision-making behaviours exhibited during the think-aloud protocols will be highlighted to illustrate her rating processes. Her students’ assessed writing scripts will also be discussed in relation to her decision making. The last section will conclude Nittaya’s views and her actual practices with a summary of the major reiterated issues that emerged from the data.

5.3.1 Profile Nittaya is a female teacher in the 41 to 50 year-old age range who works at Central University. She has taught English for 27 years including seven years of experience in teaching English writing. Due to the extent of her teaching experience she is considered an experienced teacher in this study. She holds a Master’s degree from a Thai university and was doing a PhD at a Thai university during the data collection period for this study.

146

Nittaya had previously attended a workshop focusing on setting a “rubric” for assessing writing organised by an international organisation. She expressed that her aim in attending the workshop was “to see if there is something new”, but she found later that the content was coherent with her own knowledge and her usual practice. She stated; “Actually, I know about it. When we assess writing, we have to set rubrics” and “I normally set rubrics”. She also indicated that her knowledge of writing assessment was partly gained through her education; as she put it, “I am at the moment studying so I know the idea.” Even though Nittaya was equipped with this knowledge, she also expressed her positive attitude towards the training workshop as she saw that “it was like a reminder to employ rubrics in assessment”.

5.3.2 The writing course context Nittaya reported that she has been the only main teacher for the two writing courses for English major students since they were first convened in 2005. She stated the courses aimed to achieve different writing levels: the first course focusing on writing at a paragraph level; whereas the second course focusing on writing five- paragraph essays. The writing courses were convened in two consecutive semesters. The students in each course were typically divided into three classes with approximately 30 students per class. Nittaya reported that she was willing to have a co-teacher; however, she preferred teaching alone because she could control the consistency of instruction and assessment practices in the three different classes. As she stated; “I would like to teach all three classes in the same way.” This practice was to serve departmental policy which required students in the same course to sit the same test regardless of teachers or class enrolment The minimal control exercised by the department allowed teachers to have freedom in the their choice of instruction and assessment methods for their own classes. During the semester in which data was collected, Nittaya was the only teacher of the first writing course. She adopted a commercial course book which she planned to continue using in the sequential writing course because it covered lessons on both writing at paragraph and essay levels. The students were asked to complete two types of activities: a writing task and a writing quiz. The writing task represented one-paragraph of writing in relation to different text types taught in class such as descriptive, process, and comparison and contrast. The writing quiz comprised lesson review exercises

147

focusing on sentence construction components the students had covered in class such as head word and word order, and sentence structures such as fragments. The writing task was treated as an activity the students completed out-of-class and was assigned at the end of each chapter. In contrast, the writing quiz was an in-class activity assigned every week. The writing task was assessed to examine students’ writing ability while the writing quiz was marked to check the students’ mastery of language accuracy. Both tasks contributed to the course score allocation. In order to assess the writing task, Nittaya employed the following criteria:

Text 5.5 Nittaya’s rating criteria Marking: 10 marks Language (5 marks) - Sentence structure including tenses, subject-verb agreement - Vocabulary (appropriate and correct usage) - Mechanics (capitalisation, punctuation marks) Content (5 marks) - Organisation of content - Understandable and interesting

5.3.3 Classroom practices The data from the interviews illustrate Nittaya’s perceptions of her classroom practices. They can be categorised according to two main themes: teaching practices and assessment practices. The teaching practices theme related to the teaching principles that guided Nittaya’s teaching strategies. The assessment practices theme comprised four sub-categories: the rating criteria employed, the strategies used to assess the students’ writing, the strategies used for the provision of feedback, and what Nittaya regarded as the difficulties in her assessment. The concepts which contributed to Nittaya’s views of her classroom practices are presented in Table 5.5 below:

148

Table 5.5 Nittaya’s representations of the classroom practices Categories Sub-categories Sample quotes Teaching Teaching principles I don’t want them to be stressed. I want practices them to enjoy learning. Teaching strategies I keep students’ writing in the first semester and show them again in the second semester to let them see their own work. Assessment Rating criteria Generally, I allotted ten total marks. Five practices marks are devoted for sentence structure and grammar...The other five marks are for organisation of content. Rating strategies If I found [that they plagiarise the writing], I’ll tell them to write on their own...There is no score deduction. Feedback strategies I show the writing with serious errors in class but I don’t tell them who the writers are. Difficulties in Sometimes I wonder if native speakers assessment can accept the writing. If they can accept it and I tell the students it’s not acceptable, it will not be good.

Teaching practices Nittaya expressed her own teaching principles underlying her teaching practices. She indicated writing was a complex skill that required students’ comprehension of writing concepts before constructing texts. This belief seemed to influence her attitude towards using the students’ first language (i.e., Thai) as a medium of instruction to accommodate students’ “profound understanding” about writing. It also seemed to lead her to see little benefit from assigning an English native speaking teacher to teach writing:

We don’t agree to have an NES teacher to teach this skill ... Anyway, Thai needs to be used for explanation. I don’t know. I feel that the students need to understand concepts 149

in order to produce writing. If English is used, can it make profound understanding? Even when Thai is used, it’s sometimes hard to understand.

It can be inferred from Nittaya’s views on the students’ comprehension of writing concepts that she believed writing to be a cognitive activity which requires writers to move through various stages before producing a text (Coffin, et al., 2003). Her views also seem to support a ‘writing process instruction’ approach which primarily emphasises content planning and idea generation. In this approach writers are required to depend very much on their cognitive processes in order to plan before transferring their thoughts into texts (Frodeson & Holten, 2003; J. R. Hayes, 1996). During the planning process writers need to consider various factors such as audience, writing topic, writing purpose and the writing conventions they need to use to produce the text. As students in this study context are second language learners they also need to understand how their intended message can be shaped through the use of linguistic features (K. Hyland, 2003; Rankin, 2001). Therefore, these requirements may be regarded by Nittaya as important and something which the students need to completely understanding in order to write quality compositions. When conducting the teaching of writing, Nittaya appeared to consider her students’ affective attitudes towards learning. Throughout the interviews she frequently expressed her efforts to maintain a positive attitude among the students:

I don't want them to be stressed. I want them to enjoy their learning.

These principles were reflected in her stated teaching strategies which seemed to promote learner autonomy and self-directed learning. For example, Nittaya stated that she allowed the students to see all their quiz scores and to select three of the scores they wanted to be part of the course score allocation:

I show them all of their scores and let them underline the scores they choose. They can choose only one quiz from each group I set.

She believed the practice of assigning more quizzes than the actual number needed for the course score allocation may encourage the students to develop their writing ability:

They will feel that if this quiz is bad they can do the next better. They will realise that

150

if they don’t do well this time they can still improve in the next.

Moreover, because Nittaya taught two consecutive writing courses she reported keeping one of the students’ writing tasks composed during the first course to show to the students during the second course. She stated that the aim of this practice was to help the students self-assess their own writing development:

I showed them the writing they wrote in the previous semester. Sometimes the students were suddenly aware and wondered how they wrote their writing that way. Why did they organise the writing like that? They became aware.

In addition, because Nittaya was the only teacher of the two writing courses her emphasis on the students’ learning process likely results in a more flexible approach to writing instruction:

I planned that I would cover descriptive, narrative and process [in this semester] but when we reach that point. ... because both I and the students are flexible and because I will teach them next semester, I am not serious [if I can include all text types as said in this semester.]

As presented above, what is prominent in Nittaya’s teaching strategies is her consideration for students’ learning and their writing development. She seemed to value the students’ learning process to a greater extent than their written product. Her teaching practices in general appeared to support the notion of writing as a process and the way in which it places the writer as the centre of the process and the emphasis it places on a ‘positive, encouraging, and collaborative’ learning environment (Silva, 1990). Nittaya appeared to be self-reflexive and to understand her role as a teacher. Regarding her overall teaching practice, she described herself as a lenient teacher who cared for students’ affective attitudes. She stated; “Normally, I’m not a strict teacher because I don’t want the students to be stressed. I want them to improve with their full potential.” Her belief about herself as a teacher and the principles of teaching she relied on seemed to influence her assessment practices.

Assessment practices In relation to assessment practices, Nittaya had to follow the proposed course outline and course score allocation; however, she had freedom to employ her own rating

151

criteria to assess the students’ writing in her own class and to apply her assessment strategies independently without any control from her department. Nittaya reported she employed trait criteria based on her intuition, as illustrated in Text 5.5 above. She divided the criteria into two main traits covering writing features at the rhetorical and lexicogrammatical levels:

Generally, I allotted ten total marks. Five marks are devoted for sentence structure and grammar. I think I can’t let it go. If errors are found, they have to be deducted. The other five marks are for organisation of content. Five and five; they are equally important.

Nittaya reported she used these criteria to assess writing in all text types in the two courses she taught. Regarding this practice, she felt writing at a sentence level still needed to be assessed in a higher level of writing:

I don’t know. I feel that I still have to check their sentences.

When assessing the writing, Nittaya reported she did not prioritise any particular trait in her criteria; instead, she focused on all of them at the same time:

I consider them [the two traits] at the same time. For example, grammar is obvious. I check it in one go. Then, I consider if I can comprehend the text. If so, it's fine. If I can't, I will deduct score for organisation.

As seen in the above extracts, Nittaya was likely to put a somewhat greater emphasis on lexicogrammatical aspects than rhetorical aspects despite her claim of equal emphasis. This practice seemed to result from the distinctive nature of grammar in students’ writing (Lumley, 2002). In addition, Nittaya’s views on the language aspects in writing assessment appear to support the notion that grammar is an essential component in an ESL/EFL class given the writers’ backgrounds and needs. As Frodesen and Holten (2003) assert, “issues of language are inseparable from content, organisation, and audience considerations, and they should all be given equal weight in ESL composition classroom” (p. 148). Nittaya appeared to be lenient in her assessment, which was reflected in her rating strategies. When the students made serious writing mistakes such as plagiarism she stated she gave them an opportunity to rewrite their work without resorting to score deduction as a punishment: 152

Normally I stress that they have to write their own work. If I found [that they plagiarise the writing], I’ll tell them to write on their own. ... There is no score deduction.

Nittaya’s emphasis on the students’ learning process can also be seen in her feedback strategies. She reported she provided feedback on individual writing scripts by locating errors and letting the students identify them on their own. The students were presumably then required to correct their work based on the lesson she taught:

I don’t tell them how to correct. I just locate errors and let them revise. Mostly, I integrate grammar in my teaching so the students can correct their work.

She also collected sentences with serious writing errors to show to the students in class without identifying the writers’ names:

I show the writing with serious errors in class but I don’t tell them who the writers are.

Nittaya seemed to promote a collaborative learning environment through her feedback approach. She stated she treated the writing errors as a learning resource for the students:

At least, [the whole class feedback is beneficial for] the writers. I tell other students to listen and learn from their friends’ mistakes. It’s likely they are to teach others. I thank the writers who make mistakes for their examples. This is a learning process. It’s fine to make mistakes.

Nittaya’s feedback strategies suggest that she based her assessment on supporting students’ learning. She did not appear to highlight the students’ weaknesses in an evaluative way. Instead, the weaknesses were identified to promote students’ progress and achievement in writing through a secure learning atmosphere that emphasised the commonness of making mistakes (Lee, 2007). Even though Nittaya had extensive experience in teaching English and many years of teaching writing, she expressed her sense of uncertainty when conducting an assessment. Her main concern was in relation to her language knowledge:

Sometimes I wonder if native speakers can accept the writing. If they can accept it and I tell the students it’s not acceptable, it will not be good. ... This is a problem. I’m not a native speaker so I have to be cautious.

153

Nittaya’s uncertainty about her language knowledge, as seen in the comment above, may be viewed positively in that it demonstrates self-reflection and a concern for her own professional development. The fact that she was a non-native speaker of English appeared to make her become aware of her own use of the language and to show caution during assessment practices. A native speaker was likely to be seen as a benchmark for her professional development. Overall, Nittaya’s assessment practices suggest she is primarily focused on learning processes as demonstrated in the way she employs strategies to support the development of the students’ writing ability in an encouraging learning environment. Although she was an experienced English teacher, she seemed to be self-aware of her own language knowledge and how she judged the students’ writing. It can be said that when conducting assessments Nittaya played the role of supportive coach whose main focus was on the students’ learning (Weigle, 2007).

So far the discussion has emphasised the exhibition of Nittaya’s views and beliefs about her instruction and assessment practices. However, it is important to be cautious in assuming that what is stated equates to her actual practices when teaching and assessing writing. In order to provide a complementary perspective on how Nittaya conducted assessment in her instructional work, her actual assessment practices as demonstrated in the completion of an assessment task are highlighted in the next section.

5.3.4 Actual writing assessment practices Using the same procedures of data collection as for Ladda, Nittaya’s actual assessment practices were investigated through think-aloud protocols produced while she was assessing two scripts written by two students in the same class. She selected the two scripts randomly from a pile of 23 scripts and was asked to assess them based on her usual rating criteria. The think-aloud protocols were video-recorded and replayed for the subsequent stimulated recall interviews (for a detailed account of data collection procedures see Section 3.7). Nittaya’s think-aloud protocols were analysed based on the adaptation of Cumming et al.’s (2002) descriptive framework (for a full detailed account of the analysis of the decision-making behaviours see Section 3.8.2.1) and both qualitative and quantitative analyses of the data will be presented.

154

This section describes Nittaya’s practices during the actual assessment of the two writing scripts. It also reports her thinking processes as she conducted the assessments, drawn from the think-aloud protocols. Her thinking processes will be triangulated against the data gathered from the stimulated-recall interviews. This section will begin with the description of the task type being assessed. Then, her rating processes will be presented through an analysis of the sequences and trends in her decision-making behaviours. In order to build complete understandings of the actual assessment practices, the students’ marked writing scripts are also discussed. The scripts were analysed by way of an examination of the feedback types provided by Nittaya (for a complete description of the analysis see Section 3.8.3). These issues are highlighted below.

5.3.4.1 Task profile At the time of data collection for this study, Nittaya assessed students’ writing in a comparison and contrast text type. The writing task was a one-paragraph composition on a single topic entitled “city life, county life,” as given by Nittaya. To accomplish the task the students were asked to fill in the outline form provided by Nittaya and to construct individual sentences based on ideas in the outline before constructing them into a paragraph. It has to be noted here that because the marking of all other writing scripts took place after the period of data collection, a request was made to Nittaya to post a copy of all her marked writing, including the two scripts assessed during the data collection, so that her rating processes and use of marked writing could be analysed. A few weeks after the data collection, Nittaya posted the students’ scripts showing the one-paragraph compositions without their outlines along with the individual sentence constructions. Thus, the rating process products (i.e., students’ marked writing) were not complete. They were nonetheless analysed based on the one-paragraph composition. The two students’ writing samples of the one-paragraph composition that Nittaya randomly selected to assess during the think aloud are displayed below:

155

Text 5.6 Four differences between in city life and in country life17. The first of all, transportation between in city and in country. For example, in city have traffic jam because in city have many car. While, in country have comfortable traffic because in country have many natural. The second, facilities between in city and in country. For instance, in city you can go everywhere by BTS sky train and by taxi. But, in country you can go to buys some things at department store. but in country you cango to buys some things at market. The third, cost of living between in city and in country. For example, in city have high cost of living and many competition. Whereas, in country have low cost of living and least competition. The last, neighbourhood between in city and in country. For instance, in city have individual more in country. Most people in the city live in condo or apartment and insincerity that form different in country live in house. But, most people in country have can depend on help each other and generous. In conclusion, in the city, people’s life is hectic that different from people’s life in the country is peaceful.

17 The texts were typed exactly as shown in the original writing scripts representing the students’ writing. 156

Text 5.7 City –life and Country-life There are differences between city life and country life, first difference is transportation. It is easy to go somewhere in city because there are many public transportation for city life. In contrast, in country life is difficult to find a transportation everywhere you need. Second difference is facilities. There are many facilities for city life. For instance, if you want to call to someone, There are a lot of public phones for you and if you want to buy something, there are a lot of convenience stores for you, too. On the other hand, in country have few or seldom facilities because they don’t need to use it all the times. Third difference is cost of living. City life have to spend much money for living. For example, if you want to go outside you have to pay a fare for transportation. However, money is seldom important for country life such as you can go outside by walking because there are a little cars in country road. Finally difference is neighborhood. Most of people in city do not have much time to meet a neighborhood because they have to spend time to work hard, but there are many neighborhood in country because they do not busy all the time to chatted each other, To summarize, although city life is difference from country life, but both of it is good life.

5.3.4.2 Rating processes The data illustrated below presents Nittaya’s rating processes as elicited from her decision-making behaviours during the process of assessing the two writing scripts. The rating processes were highlighted through sequences and trends in her decision making while focusing on different parts of the two texts. The sequences represent her qualitative judgements of the texts; whereas the trends show the overall frequency of the decision-making behaviours exhibited by Nittaya. In Nittaya’s class, the students’ writing consisted of three main parts: outlines, sentence construction and paragraph composition. Nittaya attended to them in the sequence that they were presented (i.e., focusing on the outlines, the sentence construction and the paragraph composition, respectively before making scoring decisions). As Nittaya did not post the writing pages containing the outlines and sentence construction her comments made on them during the think-aloud protocols were transcribed according to her actions shown in the videos. This was accomplished without reference to the actual writing scripts. When attending to the outlines of the two

157

scripts, Nittaya’s general practice was to read the relevant text segments to mediate her assessment in the outlines without referring to the other parts of the script. The process she employed to assess the outline is shown in the following commentary on Student 1’s writing script:

Er, the topic is city life, country life. [Nittaya translates ‘city life, country life’ into Thai.] At that time, I thought this topic is, er, interesting. I think it’s interesting so I think the students would be interested in it, too. Er, there’s a point of transportation for the students to think. They have to think about how transportation in city life and country life differ. The student says city life traffic jam. For country life, this student says comfortable. Comfortable. OK. For facility, this student made a list of department store. There are sky train, taxi. For country life, there are market, motorcycle, tricycle. Eh, market in country life facility. Well, only market. Well, this student may want to explain that there are department stores in the city but there are only markets in the country. OK. For cost of living, [Nittaya translates ‘cost of living’ into Thai.] This student thought, expensive in the city and cheap for country life. Neighborhood [Nittaya translates ‘neighbour’ into Thai.] What’s this? Unindividual. What does ‘unindividual’ mean? Live in condo or apartment. Insincerity. OK. I understand these two ideas. But what is unindividual? I don’t understand.

As can be seen in the extract above, Nittaya mainly employed interpretation strategies in the aspect of rhetorical and ideational focus. She went through the outline to ascertain whether the student had identified the writing topic and had achieved the stated topic development. Before focusing on each text segment she justified the choice of topic assigned to the students and ensured the way to which the student had to do to complete the task; “They have to think about how transportation in city life and country life differ”. She then itemised the key points and their sub-points on which the assessment of the student’s writing would be based. Nittaya interpreted the students’ ideas by summarising and translating the phrases or sentences into Thai before evaluating her understanding of the key points; “Well, this student may want to identify differences that there are department stores in the city but there are only markets in the

158

country. OK.” It was not evident that Nittaya focused on aspects of language when focusing on the outline provided by Student 1. When attending to Student 2, Nittaya proceeded to assess the work in the same fashion as for Student 1. That is, she focused on identifying key points and sub-points in order to comprehend the meanings of the overall text before attending to the other parts of the script. However, she employed more evaluative strategies to the work by Student 2 in terms of the rhetorical and ideational focus, and the language focus:

For transportation, she says there are many public transportations in city. OK. I get it. It's easy to go somewhere in city. OK, I understand. For country life, It's difficult to go somewhere in country. Idea is OK. Next, facility. There are many convenient markets. Eh, what is ‘convenience markets’? Convenience markets. Is it from convenience store? Convenience markets. Um, I don’t know. I haven’t heard of it. And many. The student forgets again. ‘Many’ needs to be used with a plural noun. No -s again. For shoppers in city life. For shoppers in city life. If the student has ‘shopper’, then there should not be ‘life’. Shopper in city is fine. There are have. Eh, what’s wrong with ‘there are have’; there are both ‘are’ and ‘have’ again. [Nittaya underlines two words in the writing script.]

The above extract shows Nittaya attended to certain text segments and immediately treated them independently. She articulated the evaluation of her comprehension of the text segments immediately after identifying them without giving further explanation. This may have been because judged her comprehension based solely on her impression of the text. In order to comprehend ambiguous phrases, she appeared to relate them to her tacit knowledge: “Is it from convenience store? ...Um, I don’t know. I haven’t heard of it”. When language errors were detected, Nittaya provided extensive explanation to identify the errors and to propose correction. This practice may have resulted from the fact that the language errors are acknowledged as one of the more noticeable writing features which lead experienced raters to frequently display ‘classifying errors’ behaviour during their assessment (Lumley, 2002). This finding is also likely to echo the findings in previous studies which suggest experienced raters assign greater focus to the language aspects of writing (Barkaoui, 2010a; Cumming, 1990; Lumley, 2002).

159

Overall, Nittaya appeared to place greater emphasis on the rhetorical and ideational aspects when she attended to the outlines of the two writing scripts; whereas the language aspects were likely to be addressed when errors were detected. This practice suggests Nittaya’s attempts to comprehend the ideas the students were to develop further into their writing when she first attended to the texts. Comprehending of ideas at the initial stage of the assessment may be important for Nittaya in order to conceptualise and build her internal sense or ‘text image’ (Freedman & Calfee, 1983; Pula & Hout 1993; Wolfe, 1997; Wolfe, 2006) of the students’ writing qualities before making an actual judgement (see the discussion on text image on page 136). As mentioned earlier, Nittaya did not post the scripts which included the students’ sentence construction. Thus, her comments made on this section of the script during the think-aloud protocols were also transcribed based on the actions she displayed in the videos. Thus, no reference could be made to the actual writing scripts. When attending to the section of the script on sentence construction, Nittaya exhibited various decision-making behaviours across the three major focus areas than she did when attending to the outlines. She monitored her assessment by reading relevant text segments and focused on them at both the rhetorical and ideational level, and the language level, simultaneously. Interpretation and judgement strategies were employed by Nittaya in an unsystematic way depending on her personal judgements of the text segments. Her practice can be seen in the extracts below:

Student 1 …but. Instant ‘but’ out of nowhere. But in country you can go everywhere by motorcycle or tricycle. I understand the idea but it’s an instant ‘but’ out of nowhere with a comma [Nittaya circles a word in the writing script.] The student forgets again. Students had recited in class that ‘but’ is one of ‘FANBOY’ and a comma is put in front of the ‘FANBOY’ because they are clause connectors. Now the student used an instant ‘but’ again. In city you can go to buy something, go to buy, can go to buy, can go to buy something. Imitation. The student likes to imitate Thai words again. [Nittaya says the Thai words which mean ‘go shopping’] Thai people say ... (Thai words which mean ‘go shopping’)... The student uses ‘go to buy’ again. [Nittaya circles a word in the writing script.] I have to emphasise [in class] again that they can use an

160

action verb directly; if they want to buy something, just say ‘buy something’. Er, at the department store but in country. But, the student uses ‘but’. The student doesn’t use a comma again. [Nittaya marks something on the writing script.] Er, but in country you can go to buy, go to buy. This needs to be corrected. [Nittaya circles a word on the writing script.] ... in city have high cost of living and many competition. Competition is an abstract noun; many is used. Many. The student has to be careful again, many. [Nittaya circles a word on the script.] whereas. Whereas is used. in country have low cost of living and least, least competition. The student wants to say there is the least competition. Eh, how to order words?

Student 2 Second difference is facilities. The student says there are many facilities for country life. OK. I get it. For instance, giving an example. If you want to call to somebody. Well, why does the sentence start with ‘if’ and end with a full stop? Why? It’s clause. The student forgets again that it is not a complete sentence. [Nittaya marks something on the writing script.] It’s a fragment. It needs to be written in a form of sentence. If you want to call to somebody. Thus, if the student wants to continue the previous sentence, the full stop needs to be taken out and continue with this one. There are a lot of public phones for you and if you want to buy something, there are a lot of convenience stores for you. On the other hand, in the country have, in the country have. The subject is wrong again. The student’s subject is not right. [Nittaya underlines a word on the writing script.] Have few or seldom facilities. This student likes using the word ‘seldom’ very much but it’s wrong. [Nittaya circles a word on the writing script.] Because they don't need to use it all the time. What’s this? Because they don't need to use it all the time. On the other hand, in country have few or seldom facility because they don't need to use it all the time. What is this? Is it to say ‘in the country, there are facilities or there aren’t facilities because they are not used all the time? It’s not clear here.

As seen in the above extracts, Nittaya focused on the rhetoric and ideas, and the language simultaneously. When focusing on the rhetorical and ideational aspects she

161

generally articulated her judgement of comprehension immediately after reading the text segments. When encountering ambiguous phrases however she tried to clarify and interpret them prior to making an evaluation: “What is this? Is it to say ‘in the country, there are facilities or there aren’t facilities because they are not used all the time. It’s not clear here” (Student 2). As reported earlier, this behaviour may reflect Nittaya’s uncertainty regarding the language used and her perceived need to be careful with her assessment: “I wonder if native speakers can accept the writing. If they can accept it and I tell the students it’s not acceptable, it will not be good...This is a problem. I’m not a native speaker so I have to be cautious. Evidence of this may be drawn from the data from her stimulated recall interviews when she asked herself; “how to order words” in Student 1. As she explained, “Sometimes when I mark writing, I think if I am sure about grammar. Is it correct? Sometimes I have to look the words up in a dictionary.” The fact that she is not a native speaker of English seems to cause her to hesitate during her assessment. In spite of her long career as a language teacher, Nittaya’s hesitation appears to contradict the notion that experienced teachers are confident assessors, a notion proposed by Sweedler-Brown (1985) in her finding that raters become more confident when they are more experienced. When responding to the language errors, Nittaya’s strategy appeared to be consistent with those she employed while attending to the outlines. That is, she identified errors and proposed correction by giving an extensive explanation: “Imitation. The student likes to imitate Thai words again. [Nittaya says the Thai words which mean ‘go shopping’] Thai people say ... (Thai words which mean ‘go shopping’) ... The student uses ‘go to buy’ again” (Student 1) and; “Thus, if the student wants to continue the previous sentence, the full stop needs to be taken out and continue with this one” (Student 2). When the errors were found, Nittaya referred to the teaching content relevant to the errors and the feedback she would give the students. She also frequently referred to the student when she addressed the writing problems. This behaviour appears to cohere with Cumming et al.’s (2002) finding that raters’ often envision the writer’s situation while assessing their writing. The authors found there is a tendency for ESL/EFL raters to envision the writer as a language learner or a student enrolled in a university course. In Nittaya’s case, the student thus seemed to be referred to as a representative of the students in her class.

162

When focusing on the paragraph composition, there was a discrepancy in Nittaya’s rating processes while attending to the two writing scripts. During the assessment of the script by Student 1 she interspersed her decision making across all three focuses with a greater emphasis on the self-monitoring strategies; whereas she emphasised the language aspects when assessing the script by Student 2. The process of her assessment on Student 1’script can be seen in the commentary below.

Student 1 Er, when writing this, the student forgets the title again [Nittaya underlines an empty space to locate where the title should be.] She starts with the topic sentence. Where’s the topic sentence? This is not the correct format. Normally, a paragraph has only one indent. This one, the student forgets the paragraph format. [Nittaya writes a comment at the bottom of the page.] Four differences between, ‘between’ and ‘in’ between in [Nittaya underlines the word ‘in’ in the writing script.] The first of all, the first of all. Why does the student use ‘the’? [Nittaya circles the word ‘the’ on the writing script.] Transportation between in city and in country. For example, [Nittaya marks a symbol to locate an insertion needed in front of the word ‘translation’.] This is not a sentence. It’s not a sentence. For example. Er, is it copied [from the sentence construction]? If it’s copied, it will not work. [Nittaya turns to the page with sentence construction back and forth to compare with the paragraph composition.] While in country. Well, this means the student copied from the front pages and put the sentences together. Uh, there’s addition at the end. There’s in conclusion to summarise. In the city, people's life is hectic that different from. Are sentences in the same order? for example, in city have because in city have many car. While in country have. They are the same.

The extract illustrates the way in which Nittaya employed a greater range of decision-making behaviours to monitor her assessment than she did when assessing the outlines and the sentence construction. She began her assessment of this part by examining the paragraph format before focusing on the relevant text segments and treated them on both the rhetorical and ideas, and the language aspects. During the assessment process she was found to compare the sentences written in the paragraph

163

with the previously assessed sentence construction to identify similarities and differences. It can be inferred from her comment during the comparison that Nittaya expected the student to develop their paragraph writing in an effective way and to avoid ‘copying and pasting’ from the sentences the student had previously constructed; “Er, is it copied [from the sentence construction]? If it’s copied, it will not work.” The behaviour of comparing the two writing parts to monitor her assessment was maintained when Nittaya assessed the script by Student 2. However, she put greater emphasis on the language aspects:

Student 2 There are differences. Is this copied [from the front page]? [Nittaya turns to the page with sentence construction back and forth to compare with the paragraph composition.] There are differences between city life and country life. First difference is transportation. OK. First difference it's easy to go somewhere in city because there are many public transportation for city life. Er, ‘many’, when we use ‘many’. The student, er, try again [Nittaya underlines the word ‘many’ on the writing script.] It's easy to go somewhere in city because there are many transportation. In contrast, in city life is a. Er, in [Nittaya marks something on the writing script.] It's difficult to find. Thus, the subject here should be changed. Otherwise, the student should separate ‘in country life’. Then, a new subject ‘it’s easy’ can be used here. [Nittaya circles part of the word ‘transportation’ on the writing script.] OK. If we want to maintain the student’s idea, if I don’t correct the writing, I’m trying not to correct it. [Nittaya erases her own marking on the script and marks a new place.] The student can insert a sentence It is dificult to find transportation here. To find transportation. ‘Transportation’ is abstract. It should be changed into kinds of transportation [Nittaya marks something on the writing script.] Everywhere you need, you need. Is it about ‘you’ here?

This extract shows Nittaya’s consistency in her practice when attending to the language aspects of the scripts, particularly her extensive explanation of errors and their correction, which was maintained in the rating process of the previous parts of the texts. It was also evident that Nittaya tried to maintain the students’ writing style when she

164

proposed correction to the language errors: “If we want to maintain the student’s idea, if I don’t correct the writing, I’m trying not to correct it.” Nittaya justified her preference to propose minimal correction during the stimulated recall interview by stating; “I try to maintain what the students write. Since there are many ways to correct the writing, I try to propose the way with the least correction.” Nittaya further contended this practice was designed to avoid the students developing a negative attitude towards writing as “they would not feel this is incorrect and that is also incorrect.” Moreover, this practice suggests Nittaya’s inclination to preserve the student’s original writing was influenced by her desire for the students to have a positive attitude towards correction. After having read through the three sections of the writing scripts, Nittaya then summarised her judgements and made scoring decisions as shown below:

Student 1 Therefore, I stop my marking now. I’ll let the student see if these are sentences. When I teach in class, students seem to understand. When I asked where the verb of this sentence was, the students could answer correctly...When it comes to writing, why does the student forget everything? What I have emphasised is not employed. This means [my teaching] fails. I have to emphasise again. I can’t score this writing. If I do, the student will fail because it’s wrong. It may be because the student rushed to submit.

Student 2 Er, ideas are OK. The student is trying... Ideas are fine but... Organisation is OK because the student follows the outline. However, she has to improve sentence structures, especially the use of -s in singular, plural and subject verb agreement. This student needs, er, to be emphasised [and possibly retaught the grammatical aspects as said earlier]. OK. That’s all for this student.

As clearly seen in the extracts above, Nittaya employed different strategies in her scoring decision making. In relation to the script by Student 1, she made her final decision to have the student revise the writing without providing an explanation about the writing quality. Rather, she referred to her teaching and to the student’s response in class about particular grammatical aspects they had learned. Nittaya’s decision-making

165

behaviour may reflect her disappointment that the student’s writing did not adequately reflect the skills and knowledge development taught in class. She appeared to stop her assessment because the writing did not meet her expectations. Indeed, the course content is likely to be an influential factor affecting the assessment of Student 1’s script. By contrast, Nittaya assessed the script from Student 2 on the basis of its quality by summarising her overall judgement of the text both in terms of language and content as stated in her criteria. It is also evident in both texts that Nittaya prepared to provide feedback for the students. She further explained during the stimulated recall interview that she would “reemphasise, reteach and extend [the lessons].” In some ways this practice supports formative assessment theory and its objective to facilitate the development of learning and the improvement of teaching (Black, 2009; Black & Wiliam, 1998; K. Hyland, 2003). In addition, it also demonstrates the emphasis Nittaya places on the students’ learning processes rather than on the quality of their product. Although Nittaya did not score both writing scripts during the think-aloud protocols, she explained in the stimulated recall interview that she assigned scores after she had finished rating all writing scripts; “If I’m not in a hurry, I’ll check [assign score] after finishing marking all writing scripts.” It may be inferred from Nittaya’s comment that she gave importance to score consistency by norm-referencing her assessments to draw comparisons of writing ability among the entire student group (Gipps, 1994, 2012). On the basis of the above assessment processes used for the two writing scripts, the decision-making sequence employed by Nittaya can be demonstrated in Figure 5.2 below:

166

Attending to the outlines of the essays - Engage in reading text segments and mainly focus on interpretation strategies for rhetoric and ideas.

ssays - Engage in reading text segments and mainly focus on interpretation strategies for rhetoric and ideas

Attending to the sentence construction - Engage in both interpretation and judgment strategies for self-monitoring, rhetoric and ideas, and language.

Attending to the paragraph composition - Engage mainly in judgment strategies for self- monitoring and language.

Making scoring decision - Summarise, distinguish or tally judgment collectively and articulate overall results. - Refer to course content and provision for feedback.

Figure 5.2 Sequence of Nittaya’s decision making behaviours

Even though Nittaya focused on three different aspects when assessing the different sections of the writing scripts, her overall decision making illustrated the general direction of her rating processes. As seen in Figure 5.2, when attending to the outlines, she mainly employed decision-making strategies relevant to aspects of rhetoric and ideas rather than language. When moving her attention to sentence construction, Nittaya employed decision-making strategies relevant to rhetoric and ideas, and language elements. Nittaya then engaged more frequently in decision-making strategies relevant to language elements when attending to the paragraph composition. The different points of focus for each section of the script may be explained by the fact that Nittaya valued the importance of conceptualising writing ideas before assessing the students’ actual writing ability. The movement from sentence construction to paragraph composition seems to confirm this point. Nittaya’s greater emphasis on language

167

aspects in the paragraph compositions may result from the fact that they were developed from the sentence constructions in which the rhetoric and ideas had been assessed. Her overall rating sequences seem to support Cumming et al.’s (2002) claim that “the raters typically integrated their decision making into complex, interactive episodes of information-gathering, reasoning, and hypothesising, prior to making then evaluating their assessment judgements” (p. 73). Overall, Nittaya’s decision-making processes during the assessment of the three sections of the writing scripts appeared to be multifaceted and interactive. The decision- making behaviours were greatly interdependent and seemed to be influenced by her impression of the different parts of the texts and contextual factors such as the course content. Her judgements of the writing traits as set down in the criteria seemed to be highly subjective. Furthermore, even though Nittaya clearly assessed the writing features as stated in the criteria, she appeared to include other facets during the actual assessment process. This finding reflects in part the way in which raters’ decisions can tend to be based on personal assessment styles when they cannot relate the text to the rating criteria (Lumley, 2002; Vaughan, 1991). In order to triangulate the data presented in the patterns identified in the qualitative data, the frequency of the individual behaviours for each area of focus is summarised below in Table 5.6. This summary demonstrates the trends in Nittaya’s decision making:

Table 5.6 Frequencies of overall decision-making behaviours exhibited by Nittaya Student 1 Student 2 Freq. % Freq. % Freq. % Freq. % A. Self-monitoring Focus 142 62.01 147 58.80

1. Interpretation Strategies 102 44.54 108 43.20 2. Judgement Strategies 40 17.47 39 15.60 B. Rhetorical and Ideational 49 21.40 37 14.80 Focus 1. Interpretation Strategies 37 16.16 28 11.20 2. Judgement Strategies 12 5.24 9 3.60 C. Language Focus 38 16.59 66 26.40 1. Interpretation Strategies 26 11.35 34 13.60 2. Judgement Strategies 1 5.24 32 12.80 Total 229 100 229 100 250 100 250 100

168

Table 5.6 shows Nittaya displayed somewhat different decision-making patterns while assessing the two texts. In general, Nittaya employed slightly more decision- making behaviours when assessing the script by Student 2. In relation to the frequency of the decision-making behaviours for each focus, Nittaya displayed little noticeable difference between the behaviours in the self-monitoring focus on both texts. However, there were considerable differences in the frequency of the behaviours in the rhetorical and ideational, and the language focuses. Nittaya exhibited greater decision-making behaviour in the rhetorical and ideational focus when assessing the script by Student 1; whereas she displayed more decision-making behaviours in relation to the language aspects of the script by Student 2. The noticeably fewer decision-making behaviours in the language focus for Student 1’s script may have resulted from Nittaya’s decision to stop marking the text when she determined that the text fell below her expectation as discussed in the section above. As shown in Table 5.6, Nittaya was likely to be self-consistent in her assessment of the two texts when attending to the self-monitoring and the rhetorical and ideational focus areas. She employed more interpretation strategies than judgement strategies when attending to these two aspects. However, discrepancies were found in the assessment process when Nittaya attended to the language focus. Interpretation strategies were more frequently used than judgement strategies when assessing the script by Student 1; whereas interpretation strategies and the judgement strategies were applied with approximately equal frequency when assessing the script by Student 2. Again, this discrepancy may be the result of Nittaya’s decision to stop marking the text from Student 1 as mentioned earlier. Overall, Nittaya’s rating processes reflect her self-reported views of her assessment practices. The next section will present an analysis of the product of Nittaya’s writing assessment practices as revealed through her marking of the students’ scripts. The aim of this analysis is to complement our understanding of what Nittaya actually did during her assessment of the two scripts.

5.3.4.3 Product of writing assessment practices Following the assessment of the two scripts during the think-aloud protocols, Nittaya finished assessing the remaining 21 scripts written by all students in her class independently. A request was made to Nittaya to post all 23 marked writing scripts

169

including the two scripts marked during the think-aloud sessions upon completion of the assessment. As mentioned earlier, there were three sections to each writing script (i.e., the outline, the sentence construction and the paragraph composition); however, Nittaya only posted the paragraph composition. Therefore, the discussion of the product of her rating process is based solely on the analysis of the paragraph composition. In order to investigate Nittaya’s assessment practices the types of feedback she provided on each script were analysed. Among the three feedback types stated in Section 5.2.3.5, Nittaya employed only indirect and direct feedback. Examples of the feedback given in the marked writing scripts are illustrated below:

Text 5.8 Student 2 In contrast, in country life, is difficult to find a transportation everywhere you need. Second difference is facilities. There are many facilities for city life. For instance, if you want to call to someone, T, here are a lot of public phones for you, and if you want to buy anything, there are a lot of convenience stores for you, too. , Text 5.9 Student 3 On the other hand, in the country life, you will receive a great atmosphere during transportation which is fastest than city life. A second different of city life and country life is facilities. For instance, you will be comfortable to live in the city life, and you can buy a lot of facilities. But in country life can’t you don’t have supermarket and car parking that is enough. That makes you buy everything harder.

The extracts above illustrate Nittaya’s practice of providing indirect and direct feedback to the texts. The indirect feedback includes circles, underlines and insertion symbols to locate the errors in relation to the rhetoric and ideas, and the language aspects; whereas direct feedback by way of the correction of errors includes inserting punctuation marks as seen in Text 5.8 and deleting words as in Text 5.9 (for a full example of the actual marked writing scripts see Appendix P). The frequency of her feedback type across all marked texts is shown below:

170

Table 5.7 Mean scores of feedback types Types of writing scripts Indirect Direct feedback feedback Total Scored 5 or lower 19.00 1.25 20.25 (n = 8) Scored higher than 5 17.73 0.93 18.67 (n = 15)

Table 5.7 shows Nittaya provided slightly more feedback types for compositions that scored five or lower. This slight difference may indicate Nittaya generally gave feedback on the students’ writing regardless of the writing quality. The provision of the two feedback types to the different kinds of scripts followed the same pattern; that is Nittaya provided more indirect feedback than direct feedback. Nittaya’s emphasis on indirect feedback may signify her attempt to promote student learning through self-editing process (Ferris & Roberts, 2001). This practice corresponds to her reported views on the provision of feedback, as demonstrated in her statement, “I don’t tell them how to correct [errors]. I just mark errors and let them revise.” The lack of commentary feedback and codes to identify error types suggests two possible assumptions about Nittaya’s practice. First, she and the students may have shared knowledge of the error types as located by Nittaya and the students can self-edit their work based on that knowledge. Second, Nittaya may provide detailed explanations of all errors to the students in class and use the marked writing as a learning resource. The second assumption appears to support her previous acknowledgement that her practice was to show writing with errors to the students in class: “I tell other students to listen and learn from their friends’ mistakes. It’s like they are to teach others.” Overall, Nittaya’s actual assessment practices indicate she focused on writing as a process. Writing scripts were seemingly used as a resource to promote student learning processes. When engaging in assessment, Nittaya’s decision making appeared to be based mainly on her personal judgements. Contextual factors such as reflection on the course content and provision of feedback were also taken into consideration. To provide a synthesis of the discussion based on Nittaya’s case study, the major themes and trends concerning her views and practices on writing assessment are summarised in the next section.

171

5.3.5 Summary of Nittaya’s views and practices Nittaya’s writing instruction and assessment practices are suggestive of her support for the notion of writing as a process. Indeed, the formative purpose of writing was emphasised in a variety of her teaching and assessment strategies. Based on the data presented, the students seemed to be treated as major agents in the learning process; their affective attitude was likely to be a major influential factor in the writing pedagogy and assessment. Even though clear writing traits were predetermined in the rating criteria, Nittaya’s inevitable personal judgements nonetheless influenced the subjective assessment. Contextual factors such as course content were found to influence her actual assessment of the texts. Moreover, during the actual rating processes, she appeared to employ linear rating sequences by focusing on various writing components (i.e., outline, sentence construction and paragraph composition) as presented in the texts. However, she tended to exhibit various iterative decision-making behaviours when attending to each component of the texts. In general, Nittaya’s reported practices and her actual practices during the think-aloud protocols appeared to be highly consistent. These issues will be revisited and discussed in Section 6.4.

5.4 Summary This chapter presented data pertaining to the single case analyses of Ladda and Nittaya. Each teacher’s views and practices in relation to writing assessment were highlighted and discussed. The major themes to emerge from their individual views on writing instruction and assessment were identified and their actual rating processes were demonstrated. The next chapter will continue the focus on the case study participants by reporting findings from the single case analyses of Rewadee and Sak.

172

CHAPTER 6 TEACHERS’ WRITING ASSESSMENT PRACTICES: REWADEE AND SAK

6.1 Introduction Chapter 5 presented the findings based on the single-case analyses of Ladda and Nittaya. The individual views and practices in their writing assessment were highlighted and discussed. This chapter focuses on findings from two types of analyses: the single- case analyses of Rewadee and Sak and the cross-case analysis of the four teachers. The single-case analyses of Rewadee and Sak were analysed using the same approach to data collection and data analysis presented in Chapter 5. Findings from the analyses of the responses provided by Rewadee and Sak will be highlighted in two separate sections; one section for each teacher. Each section will start with the teacher’s profile providing details of the teacher’s personal and professional information. It will then present the teacher’s views in relation to major themes regarding writing instruction and assessment as situated in their specific context. Their actual assessment practices will also be illustrated. Each section will conclude with a summary of the key issues regarding the teacher’s views and practices. Following this, the cross-case analyses of the four cases will be presented.

6.2 Rewadee This section begins with Rewadee’s profile, outlining her demographic data and professional information, including her teaching and training experiences based on the information obtained from the questionnaire and interviews. This will be followed by a description of her writing course. The purpose of the profile and the account of the writing course are both to contextualise the data presentation and discussion which form the bulk of this study. Then, the data gathered in relation to Rewadee’s views of the writing instruction and assessment processes will be presented and framed according to one major theme which is illustrative of classroom practices. The data were sub-divided into sub-categories as illustrated below. In the section on actual assessment practices, Rewadee’s rating processes will be demonstrated through her decision-making behaviours displayed during the think-aloud protocols. The products of her assessment (i.e., students’ written scripts) will also be highlighted with regard to her decision

173

making. The last section will summarise key reiterated issues according to Rewadee’s views and practices.

6.2.1 Profile Rewadee is a female teacher in the 31 to 40 year-old age range who works at North University. She holds a Master’s degree from a Thai university and has been teaching English for three years including four months of experience in teaching English writing. In terms of the amount of time Rewadee has spent working as a teacher of both English in general and English writing, this study considers her to be a teacher with little experience. Rewadee never attended training workshops focusing on writing assessment. However, she reported she learned about assessment during one of her postgraduate courses. Here she was introduced to the principles related to various aspects of assessment including the employment of rating criteria: “There was a course focusing on teaching writing. The teacher introduced what should be included in rating criteria.” Because Rewadee had previously not received formal training on how to assess writing she expressed her enthusiasm about receiving such training: “Actually, I would like to [attend training workshops in assessing writing]. I’m interested in it.”

6.2.2 The writing course context Rewadee was a new teacher in her workplace and was assigned to teach writing during her first semester at North University. She was one of the two teachers of the English major students whose instruction aimed to develop their writing at the paragraph level. Among the three classes of students in this course she taught one class while her co-teacher taught the other two classes. There were 25 students in Rewadee’s class. During this writing instruction Rewadee and her co-teacher employed the same course book developed by the co-teacher. The course book was self-designed and adapted from commercial course books. Apart from sharing the same course book, the two teachers also shared the course syllabus including the course score allocation. However, they had freedom in conducting their own instruction and assessment. They could design their own supplementary teaching materials, exercises, exam papers and rating criteria to assess writing tasks and exam papers in their own classes.

174

With regard to Rewadee’s class, there were seven writing tasks comprising an exercise on outline writing and five paragraph writing assignments in four text types: narrative, descriptive, process, and comparison and contrast. During the second half of the semester the students were required to submit approximately one writing task each week. Rewadee developed her own analytic rating criteria to assess the writing task across text types as shown in Table 6.1 below:

Table 6.1 Rewadee’s rating criteria Element Max. score Contents (4 points) - The paragraph is interesting to read. 2 - The paragraph is easy to follow. 2 Organisation (9 points) - The paragraph begins with a topic sentence that has both a 2 topic and controlling idea. - The paragraph has unity: each supporting sentence is related 3 to the topic. - The paragraph has coherence: the ideas in the paragraph have 3 been organised logically, and appropriate transitional words are used. 1 - The paragraph ends with an appropriate concluding sentence. Correction (7 points) - The spelling is correct. 2 - Grammar and sentence structure are correct. 3 - Punctuations and capital letters are used correctly. 2 Total 20

6.2.3 Classroom practices Rewadee’s views on her classroom practices in relation to the class she taught independently can be categorised according two main themes: teaching practices and assessment practices. The former category constitutes one key theme; whereas the comments about assessment practices can be further sub-categorised with regard to the rating criteria employed, the strategies used to assess the students’ writing, and the

175

strategies used for the provision of feedback. The concepts contributing to Rewadee’s views of her classroom practices are demonstrated in Table 6.2 below:

Table 6.2 Rewadee’s representations of the classroom practices Categories Sub-categories Sample quotes Teaching practices I want them to write more than one draft. At least two drafts. Assessment Rating criteria I allotted two marks for a topic sentence practices because I consider it as very important to writing. Rating strategies I check sentence by sentence if it’s wrong, or if ideas are smooth. Then, I mark the deducted score. Feedback strategies Mostly, after returning their work with feedback I will make an appointment with the students who made a lot of serious mistakes to see me in person.

Teaching practices With regard to writing instruction, Rewadee appeared to promote writing as a process. This was reflected in her expectation of students’ writing. She expected the students to go through the writing process before producing a composition:

First, you need to know what you are writing. ... In a pre-writing stage, you need to brainstorm your ideas. ... When you have ideas, you then draft the writing. After that, you have to revise and edit. You have to go through many things before a final draft.

The students were also expected to compose multiple written drafts before submitting their work:

Actually, I want them to write more than one draft. At least two drafts. ... In fact, three drafts. The second draft is to copy the first draft into a piece of paper. My intention is for them to ask their friends to revise the writing and submit it to me after the revision.

Apparent in the above comments is Rewadee’s view on the implementation of writing processes within her teaching practices. This view suggests she regards writing as a cognitive activity which requires writers to go through various stages in order to

176

compose a text (Coffin, et al., 2003; Grabe & Kaplan, 1996; K. Hyland, 2003; Weigle, 2002). This belief is reflected in the classroom activities as stated in the course syllabus which Rewadee is required to follow. In each lesson on writing text types the students were required to be instructed on the process, study a paragraph model and its components, and practise writing by going through the process of prewriting, brainstorming and revising (for the full writing course syllabus see Appendix Q). The students were required to go through the writing process including prewriting, brainstorming and revising as they practised their writing. The implementation of the writing process in her instruction was also reflected in her assessment practices.

Assessment practices In relation to assessment practices, Rewadee appeared to have explicit rating criteria. She reported designing analytic rating criteria to assess writing in all text types, as shown in Table 6.2 above. Her selection of writing features to be assessed was likely to be influenced by the knowledge she had gained through her reading and prior education:

When I realised that I had to teach this course, I learned about it by reading my previous learning materials, buying a book in paragraph writing and downloading a book. I also have to study.

In accordance with her rating criteria as presented in Table 6.2, Rewadee divided the assessed features into three major domains: content, organisation and language accuracy. Each domain was sub-divided and weighted differently. Rewadee appeared to weigh different total sub-scores for each of the assessed writing features according to her views on the degree of their importance in the writing task. For example, a topic sentence was weighted two marks:

I allotted two marks for a topic sentence because I consider it as very important to writing.

Rewadee reported she allotted 20 marks for the total score of each writing piece. She stated the purpose behind such a high total score was so that she could be prepared for score deductions due to the student writing errors. She appeared to deduct scores for each writing feature on the basis of what she saw as the seriousness of particular errors:

177

If a topic sentence looks weird, if it’s not complete, I may deduct one point. I also deduct one point for an incomplete conclusion. If there isn’t a conclusion, I’ll deduct two. What is meant by ‘not complete’ is its incomprehensibility. What are you connecting? What are you saying?

Even though Rewadee had minimal experience in the assessment of writing, her view on the criteria suggests she had a clear rationale for writing assessment. The use of analytic rating criteria with different score weightings infer Rewadee aimed to raise student awareness of the features necessary for effective writing. Although the students may have been discouraged by the score deduction system as it emphasised their mistakes, the process also allowed them to clearly see their weaknesses and improve their writing quality in later drafts based on the score given. As Rawadee commented in her correspondence with a student about the deducted score: “Why was I [the student] deducted two? Read it and find out what you lack. Well, I [the student] lack a topic sentence.” This practice, therefore, seems to support the process of providing diagnostic feedback using the analytic rating criteria (Bachman & Palmer, 1996; A. D. Cohen, 1994; Sawaki, 2007; Shaw & Weir, 2007). Rewadee appeared to employ different rating processes when assessing writing at different proficiency levels. She reported she checked the writing sentence by sentence and considered all the writing aspects at the same time. When errors were detected Rewadee stated she deducted marks before summing up the marks deducted when she finished marking the whole piece of writing:

I check the writing collectively. I tick the errors. As I said, I check it sentence by sentence to find out if it’s wrong, or if ideas are smooth. Then, I write down the deducted score. When I finish marking, I deduct the sum of the deducted score from the total score allotted.

However, when Rewadee assessed low proficiency writing her rating process seemed to be recursive. She stated she based her rating on text comprehension when she first read the text and revisited it to consider error types before making judgement:

Sometimes I have to reread the text to find out why I can’t understand it. When assessing writing, I first read for ideas. Eh, not smooth, this is not smooth. I have to reread the writing. When I score it, I have to think why it is not smooth. Is it about ideas or incomplete sentences?

178

It could be argued on the basis of the above details that Rewadee’s marking practices were likely to support learner autonomy and self-directed learning. For example, she allowed the students to make their own decisions on whether to rewrite their work following a low score:

I ask the students if they would like to rewrite. If so, it’s fine. They will get a new score [based on a new draft]. If not, it’s OK. They’ll get whatever score I give [to the submitted draft].

Rewadee reported she scored all submitted writing drafts independently. The students would receive a better score from their revised draft. This practice was justified on the grounds that it supported the students’ examination preparation:

If they would like to rewrite and resubmit their work, they will get a better score.

I want them to learn. I want them to have a chance to revise. So they will learn that they have to put an effort in an actual exam. Writing is not tested only in the exam but it needs to be gradually developed.

The comments above suggest Rewadee had high expectations that the students’ writing would improve as a result of the process of multiple draft writing. This practice seems to support an iterative approach to the writing process that requires the students to have an opportunity to revise their work in response to the feedback they receive (Coffin, et al., 2003). Rewadee’s view of the relationship between written assignments and examinations also appears to suggest the possibility of a positive wash-back effect from her instruction (Weigle, 2002). This is because she tended to provide an opportunity for the students to practise their writing using multiple draft compositions in order to develop their writing ability to meet the requirement of the test they had to take. It also appears Rewadee held the view that was important to conduct fair assessments as part of her practices. She stated she attempted to accomplish the assessment of all writing tasks in the same text type at the same time in order to secure consistency of her marking:

We should mark the whole series of writing at the same time. Why? ... because I am likely to base my rating on norm referencing [sic; Rewadee possibly meant criteria- referencing]. I have 10 points which I have a clear idea of what they constitute. When 179

I assess this writing, I deduct a certain mark. When I assess another, I’ll still remember what point and how many marks should be deducted.

From the above comment it is clear that Rewadee considered scoring consistency when assessing the students’ writing to be very important. The practice of assessing all writing at the same time assists Rewadee to ensure self-consistency in deducting the writing score and maintaining construct validity by adhering to the criteria (Weigle, 2002). Rewadee also stated she devoted much of her time to the provision of feedback to the students both on the submitted script and in person. She commented she applied correction symbols she had previously provided to the students when she gave feedback on the text:

I have correction symbols which I distributed to the students in the first class. ... When I mark writing, I tell them that I would follow the symbols.

For writing with serious errors, she reported she provided individual consultations to the student writers:

Mostly, after returning their work with feedback I will make an appointment with the students who made a lot of serious mistakes to see me in person.

Rewadee’s approach to feedback appeared to place an emphasis on student’s learning process. The use of correction symbols implied she expected the students to self-edit their work based on the codes provided. This practice can be seen as a way to encourage the students to take responsibility for their own learning (Coffin, et al., 2003). Overall, Rewadee’s assessment practices suggest she predominantly viewed writing as a process. Each student was regarded as an active agent in the learning process who was to be given control over his or her learning. When conducting an assessment, Rewadee appeared to play the role of the reader who tried not to contaminate the students’ ideas with her own preferences: “I won’t say, ‘Your idea is wrong.’ I won’t say this. I’ll say, ‘If you want to say this, how should you say?’ … I try not to put my ideas or my preferences into the students’ writing.” The clear analytic rating criteria were likely to be of significant influence over the consistency in her assessment practices.

180

The discussion above demonstrated Rewadee’s views and beliefs about her instruction and assessment practices. However, it may be the case that her actual teaching and assessment practices do not accord with her statements. In order to obtain a deeper understanding of how Rewadee operationalised the assessment process in her instructional work, her actual assessment practices, as illustrated in the marking of the writing tasks, are presented in the next section.

6.2.4 Actual writing assessment practices Rewadee’s actual assessment practices were examined through think-aloud protocols produced during the assessment of two writing scripts written by two students in her class. The two writing scripts were selected randomly from the pile of 23 overall scripts by Rewadee for this purpose. She was asked to assess them based on her usual rating criteria. The think-aloud protocols were video-recorded and replayed for the subsequent stimulated recall interviews (for a detailed account of data collection procedures see Section 3.7). On the basis of the adaptation of Cumming et al.’s (2002) descriptive framework (for a full detailed account of the analysis of the decision-making behaviours see Section 3.8.2.1), the think-aloud protocols were analysed. Both qualitative and quantitative analyses of the data will be presented in relation to the decision-making behaviours which are in three major themes: self-monitoring focus, rhetorical and ideational focus, and language focus (for a full description of the three themes see Section 5.2.5). This section highlights Rewadee’s practices during the actual assessment and also reports her thought processes while conducting the assessments. All information is drawn from the think-aloud protocols and the stimulated recall interviews. First, the task type under assessment will be described. Then, her rating processes will be presented through an analysis of the sequences and trends in the decision-making behaviours she displayed. The product of her actual assessment practices (i.e., marked writing scripts) will also be discussed in order to provide richer understandings of the actual assessment practices. The marked written product was analysed by investigating the comments left on the paper by Rewadee (for a complete description of the analysis see Section 3.8.3). These issues are presented below.

181

6.2.4.1 Task profile The task type Rewadee assessed for the purpose of data collection in this study was the writing of a “comparison and contrast” text. The students were assigned to write one paragraph, writing on any topic they selected independently. As mentioned earlier in the interview data, Rewadee expected her students to conduct their pre-writing stage on their own. Thus, she asked the students to submit a one-paragraph composition without an outline to be regarded as the product of their writing. It needs to be noted here that a request was made to Rewadee to post the two writing scripts together with other student writing samples after she had finished marking all scripts. It became evident that one of the two scripts used during the think-aloud protocols was the same draft as in the think-aloud process (i.e., Text 6.1) while the other script was presumably taken from a revised version of the other student’s script during the think-aloud process (i.e., Text 6.2). The two student writing task samples Rewadee randomly selected to assess during the think-aloud process are shown below:

Text 6.1 Living in country and living in city Living in country is often very different from living in city. When you live in the country, you will have many good and friendly neighbors, and you can create relationship with them. When you have a problem, you can confer with your neighbors, too. In a part of the nature, you will live there among the pure and clean air from the nature of the country. In addition, the nature of the country can make you be a strong health man. In transport, you can go everywhere conveniently because the street in the country have a few car. However, when you live in the city, it is different experience. You will not have a real close friends and close neighbors. You might meet the unfriendly neighbors, and it might make you can not confer with them. When you want to go somewhere, you might feel very bored because the traffic in the city is very bad. About the air, you will got the bad air from many factory, it can make you be a good health man. Although living in the country and living in the city might give the same residence, living in the country and living in the city are give you the different experience of life.

182

Text 6.2 PC and Notebook PC is different from notebook. When you buy a PC, you can see that it is cheaper than notebook. If you want to upgrade performance, your PC, its accessories can buy easily. You can’t carry because PC is bigger than notebook. The PC doesn’t have battery or emergency generator to save electric when power outage. On the other hand, When you buy notebook, it is more expensive than PC. It is hard to buy notebook’s accessories for upgrade performance because it is expensive. You can carry the notebook because it is smaller than PC. Notebook have battery so, you don’t worry when power outage. Although PC and notebook can use for work, entertainment, etc. They are different in size, prices, performance, and usefulness.

6.2.4.2 Rating processes Presented below are the data illustrating Rewadee’s rating processes through the decision-making behaviours exhibited during the assessment of the two writing scripts. The rating processes were analysed in the form of sequences and trends in the decision- making behaviours. The sequences were shown through Rewadee’s continual behaviours when she attended to the two writing scripts; whereas the trends demonstrated Rewadee’s overall frequency of the behaviours. As stated earlier, the students were asked to submit a one-paragraph composition only without an outline. Rewadee was found to attend to the whole composition before making a scoring decision. In addition, when assessing the texts she interspersed her decision-making behaviours in all three main aspects: self-monitoring, rhetoric and ideas, and the language. Furthermore, she appeared to balance her employment of interpretation and judgement strategies of the three focuses. Rewadee’s rating processes are shown below:

Student 1 You will have many good and friendly neighbours. Thus, the first aspect the student presents is neighbours. Er, you will have many good and friendly neighbours and you can create relationship with them, create relationship? There’s an explanation: When you have a problem, [Rewadee underlines the clause ‘when you have a problem’] you can confer with your neighbours, too.

183

This is not quite smooth. It means, er, this student uses the word ‘confer’. Actually, a more frequent word is ‘consult’ which means [Rewadee says the meaning of the word in Thai.] I think students first thought in Thai and looked the word up in a dictionary. The Thai-English dictionary may show the word ‘confer’ even though the students would see ‘consult’ more often. However, the word ‘confer’ is fine. They have the same meaning. You can confer with your neighbors, too. The idea is comprehensible: In a part of the nature. This one may be influenced by Thai. It says [Rewadee translates the clause into Thai]. It’s direct translation. [Rewadee underlines ‘in a part of nature’.] In a part of nature, you will live there among the pure and clean air from the nature of the country. Ah, the second aspect is nature. In addiction, in addiction [sic]. Incorrect spelling, there’s an extra -c [Rewadee underlines ‘in addiction’ [sic] and writes a code of ‘sp -.25’ to indicate score deducted.] The nature of the country can make you be a strong health man. This may also be influenced by Thai [Rewadee translates the sentence into Thai.] Strong health man. Comprehensible but it doesn’t look like English. ‘The nature of the country can make your health strong’ sounds better. [Rewadee underlines ‘be a strong health man’, crosses ‘be a strong’ and ‘man’ and writes ‘strong’ to replace the word ‘man’.]

The above extract shows Rewadee assessed the writing by attending to particular text segments and treating each of them in relation to the relevant aspects of decision- making behaviours interdependently. When attending to the rhetoric and ideas, her general process was to identify the topic and the development of the arguments made in the text: “Thus, the first aspect the student presents is neighbours,” and “Ah, the second aspect is nature.” This behaviour shows Rewadee attempted to conceptualise the ideas before further reading the arguments and evaluating them. Moreover, it appears to echo the notion of ‘text image’ (Freedman & Calfee, 1983; Pula & Hout 1993; Wolfe, 1997; Wolfe, 2006) as discussed in Section 5.2.5.2. To accommodate Rewadee’s understanding, she generally translated the text segments into Thai. This behaviour appeared to be influenced by her view of the problems with the student’s writing which she identified as being caused by direct translation from Thai language. When language errors were detected, she proposed corrections.

184

When she attended to the second student’s text, Rewadee appeared to conduct rating processes in the same vein as for the first student’s writing script

Student 2 You can see that it is cheaper than notebook about 2000-5000 baht. This means the first aspect of the discussion is ‘price’. PC includes mouse, keyboard and case. [Rewadee underlines a word in the script.] PC is a plural; - s is forgotten. [Rewadee writes a comment on the script.] A bit of subject verb agreement. PC keyboard and case. PC have a big case so you can put more accessories for upgrade. I’m confused with the student’s sentence division. I need to recheck it. PC includes mouse, keyboard and case. The sentence is unfinished, isn’t it? There’s no mark. There’s a problem with vocabulary. [Rewadee writes a comment on the script.] PC have a big case full stop, PC have a big case. Then, there’s a full stop. There’s ‘so’ and a comma. It’s incorrect sentence division again. So you can put more accessories for upgrade performance. These two sentences need to be together; they can’t finish by themselves. [Rewadee underlines sentences on the script.] Here I’ll write sentence separation is confusing. [Rewadee writes a comment on the script.] … Now I feel that this text is more difficult to read than the first one because organisation of ideas, idea organisation or the aspects used in comparison are not as clear as the first text. … Ah, this is an error. I have taught many times that when we start a sentence with ‘although’, it [the clause] needs to end with a comma and to be followed by the next [clause]. Thus, when there is ‘although’, it’s unnecessary to have ‘but’. The students overgeneralise the use of both ‘although’ and ‘but’. Therefore, one mark needs to be deducted here because it’s not a correct writing principle and I have taught it many times. As seen in the above extract, Rewadee engaged in the rhetoric and ideas, and the language aspects in the same fashion as for the first text. She was seen to identify topic development and to propose correction to the language errors when they were detected. However, she generally employed a greater variety of strategies when assessing the second text. It was evident Rewadee deducted points on the basis of the importance of a particular error in relation to the course content: “Therefore, one point needs to be

185

deducted here because it’s not a correct writing principle and I have taught it many times.” This behaviour appears to affirm her reported practice of score deduction based on the degree of importance of a particular writing feature as mentioned earlier: “I allotted two marks for a topic sentence because I consider it as very important to writing.” When an ambiguous text segment was found, she attempted to comprehend it by rereading the passage and by breaking it into smaller text segments before considering them in detail: “PC have a big case full stop, PC have a big case. Then, there’s a full stop. There’s ‘so’ and a comma. It’s incorrect sentence division again.” This behaviour appeared to reflect her view of the rating process as previously stated in her interview data “Sometimes I have to reread the text to find out why I can’t understand it.” It was also evident that Rewadee employed a recursive rating process when she revisited the whole text after her first reading of the second script:

I’ll recheck it. This student first talks about PC. The comparison is in a box style pattern, not a point-by-point pattern. It’s the same as the first student [who writes the first text]. In comparison, compare or contrast, there are two writing styles: point-by-point which is comparing one point to another. For example, the aspect of ‘price’ will be discussed for both PC and notebook. The aspect of ‘convenience’ will be talked about for both notebook and PC. But this one, the student talked about all aspects of PC before moving on to notebook. When I revisited the text, the student first talks about price. If he starts with ‘about prices’ it’s likely a transition or a signal word to show that the first aspect would be price and readers can follow it easily. I’ll note it here that there’s no clear signal word. He should mention the aspects for discussion. [Rewadee writes a comment on the script.] If he asks a question, I can explain it. The first aspect is ‘price’. Er, the second one is ‘monitor, mouse, keyboard, case’. The second aspect is accessories. Again, there’s no clear transition or signal word to show that he’s talking about accessories. But I have to guess because he says includes monitor, mouse, keyboard and case. PC has a bigger case so you can put more accessories for upgrade, something like that. Therefore, the second aspect is accessories. For the third aspect, the student says, ‘on the other hand, when the power cut… cut... cut...’ I’m not sure

186

what he’s writing about. It may mean the electric goes out. This is clear. This seems to be about energy, the energy that supports [PC and notebook]. It will be off immediately. From my understanding, there are three aspects. In PC when you buy notebook. Now let’s see what he’s saying about notebook. When you buy notebook that is smaller than PC. When taking about the notebook, the first aspect he’s talking about is ‘size’ because he says that is smaller than PC. But the first aspect he writes is about price, isn’t it? When I consider his writing about PC, he says about price first, no, not price, size. Then, all accessories are combined together. It's easy to use. Mmm, if I ask whether this student shows clear comparison, yes it’s clear but the aspects are confusing.

It is apparent that Rewadee attended to the whole text again and focused mainly on the rhetoric and ideas. She first identified the rhetorical structure of the text to ensure it reflected what was taught in class. Moreover, she greatly emphasised the development of ideas and the need to assess the ideas employed by the students. The emphasis placed on ideas appears to be reflected in her practice of scanning the text for the various aspects of the discussion. These practices also support the notion of ‘text image’, as mentioned earlier. Rewadee’s attempt to understand the text before judging it was likely to result from her need to make her assessment clear for the students: “If he asks a question, I can explain it.” It can be inferred that if she understood the writing thoroughly she would be able to provide detailed explanation of the errors made to the students. In the stimulated recall interview, Rewadee further accounted for her intention to provide to the students with explanations that promote their writing ability before being tested in an exam: “If they ask a question, I can explain it. So they can do it in the exam.” This issue suggests Rewadee was aiming for a positive wash-back effect of the classroom assessment to testing (Weigle, 2002). When it came to making scoring decisions, Rewadee proceeded with the two texts in a similar fashion. Her judgements of the overall texts are presented below:

Student 1 This student organises ideas quite well. Coherence is OK because there’s the use of transitions when moving from one idea to another. There are a few

187

problems with spelling and weird sentences. A weakness is spelling; 0.5 is deducted for this point. 0.5 is deducted for subject verb agreement, a little problem; this is plural. 0.25 is deducted for ‘bore’. 0.5 is deducted for ‘many factories’. Make your health bad. [Rewadee writes/marks something on the script.] To sum up, about 2.5 marks are deducted. [Rewadee counts the deducted score.] This student gets about 18 marks for this piece of writing. Overall, as I said most grammar is correct. I can have a clear picture of what are compared and contrasted. And the ideas are connected. When discussing ‘country’, the three aspects are discussed, living in the country in the three aspects. Then, he moves to talk about living in the city in the same aspects. This makes the writing easy to follow and easy to understand. It’s not confusing. The score is 18, then.

Student 2 Topic sentence and conclusion are OK. I understand what he’s talking about. I have taught that a topic sentence comprises a topic. You need to say what you want to compare. You will have two things and he has them, PC and notebook. However, the topic sentence should include aspects of discussion; he doesn’t include them in this topic sentence. It’s simply saying PC and notebook. Mmm, yes notebook. This student’s writing is not complete. At this point, capitalisation is incorrect, 0.25. Subject verb agreement has to be deducted. 0.5 mark was deducted for subject verb agreement for this student. [Rewadee checks the deducted score in the first student’s writing.] It has to be deducted equally. 0.25 for punctuation, 0.5 for confusing sentence division,.The vocabulary here is a bit akward. I don’t understand. 0.5 for confusing sentence structure. [T writes/marks something on the script.] Mmm, this is incorrect grammar, 0.25 is deducted; 0.25 for spelling. Now, let’s check places with 0.25 mark deduction. [Rewadee adds up all deducted marks.] Idea organisation is confusing. The aspects are unclear. Comparison is not clear. [T counts the points.] I’ll deduct 3.5 score. I’ll deduct score for ideas. I’ll deduct score for ideas because we agreed that there should be three aspects. I’ll deduct three marks: one mark for each aspect because they are not clear. Six score is deducted for this student. He will get 14 out of 20.

188

As seen in the above extracts, Rewadee evaluated the texts by giving consideration to the quality of both the rhetoric and ideas, and the language. Because she had explicit analytic rating criteria, she appeared to adhere to them by summarising the features as stated in the criteria: content, organisation and language accuracy. Rewadee revisited the scores she had deducted during the assessment of the composition and summed them up in order to indicate the overall quality of the writing against the criteria she had. However, when making a scoring decision for the second script it was evident Rewadee compared its quality with that of the first text she assessed: “0.5 mark was deducted for subject verb agreement for this student. [Rewadee checks the deducted score in the first student’s writing.] It has to be deducted equally.” This behaviour appears to support her reported view of the importance of assessing of all writing scripts at the same time: “We should mark the whole series of writing at the same time. ... When I assess another, I’ll still remember what point and how many marks should be deducted.” This point was supported by Rewadee’s stimulated recall interview when she decided to further deduct points for a feature she had not deducted during the think-aloud protocol. She contended: “I feel we should mark all writing at the same time. Otherwise, we will keep coming up with points to be deducted. It will turn out, why is this student deducted? When we deduct score for this student, that student needs to be deducted as well.” Her comment was likely to indicate she considered both norm-referencing as well as criterion-referencing in her assessment, suggesting she valued score consistency across students’ writing scripts and construct validity. This was subsequently reflected in her adherence to the criteria (Weigle, 2002). Overall, Rewadee seemed to be self-consistent in her general assessment practices with regard to the decision-making behaviours during the process of assessment. The self-consistency may result from the employment of the analytic rating criteria. As Barkaoui (2010b) pointed out, one of his findings was that raters were likely to be self-consistent when employing an analytic rating scale. However, considerable discrepancy in the behaviours displayed was found when she assessed the two writing scripts. In the first text which received a higher score, Rewadee appeared to employ a linear rating process (i.e., attending to the text from the beginning to the end before making a scoring decision). By contrast, Rewadee tended to employ a recursive rating process (i.e., considering the overall text again before making a decision for the score) when assessing the second script – which received a lower score. The different rating

189

processes appear to support Rewadee’s interview data regarding the different approaches she employed when assessing writing. This discrepancy was likely to be influenced by her perceptions of the writing quality. In relation to assessment processes employed for the two writing scripts, the pattern of Rewadee’s overall sequence of decision making is drawn out and illustrated in Figure 6.1 below:

Attending to the one-paragraph composition - Engage in both interpretation and judgement strategies for self-monitoring, rhetoric and ideas, and language.

Making a scoring decision - Summarise, distinguish or tally judgement collectively and articulate overall results. - Refer to course content and compare the writing scripts.

Figure 6.1 Sequence of Rewadee’s decision-making behaviours

To triangulate the data illustrated in the rating sequences highlighted in the qualitative data, the frequency of her individual behaviours for each focus has been summarised. This helps to represent the trends in Rewadee’s decision making as shown below:

190

Table 6.3 Frequency of overall decision-making behaviours exhibited by Rewadee

Student 1 Student 2 Freq. % Freq. % Freq. % Freq. % A. Self-monitoring Focus 57 56.44 96 57.14

1. Interpretation Strategies 33 32.67 48 28.57 2. Judgement Strategies 24 23.76 48 28.57 B. Rhetorical and Ideational 24 23.76 38 22.62 Focus 1. Interpretation Strategies 15 14.85 28 16.67 2. Judgement Strategies 9 8.91 10 5.95 C. Language Focus 20 19.80 34 20.24 1. Interpretation Strategies 9 8.91 15 8.93 2. Judgement Strategies 11 10.89 19 11.31 Total 101 100 101 100 168 100 168 100

Table 6.3 reveals Rewadee exhibited similar patterns of decision making while assessing the two texts. In general, the frequency of decision-making behaviours employed by Rewadee when assessing both texts is approximately the same. When considering the frequency of decision-making behaviours for each focus, Rewadee exhibited slightly different behaviours in the self-monitoring focus on both texts. She attended more to interpretation strategies than judgement strategies when assessing Student 1’s text; however, equal frequencies were displayed when Rewadee focused on Student 2’s writing script. The same patterns of decision-making behaviours were found when she focused on the aspects of rhetoric and ideas, and the language of the two texts. She engaged in interpretation strategies more frequently than judgement strategies when she attended to the rhetorical and ideational aspects; whereas she emphasised the judgement strategies more than the interpretation strategies when focusing on the language aspects. These findings suggest that to a considerable extent Rewadee performed consistently in her assessment practices.

In general, it can be argued from the analysis so far that Rewadee’s overall rating processes reflect her self-reported views of her assessment practices. The next section will highlight the analysis of the product of her writing assessment practices shown during her assessment of students’ writing scripts. This will be done to accommodate a deeper level of understanding of Rewadee’s actual assessment of the two scripts.

191

6.2.4.3 Product of writing assessment practices Apart from the two writing scripts Rewadee was asked to assess during the think-aloud protocols, the other 21 writing scripts written by all students in her class were assessed independently after the process of data collection. A request was made to Rewadee to post all 23 marked writing scripts including the two scripts marked during the think-aloud sessions after she finished the assessment. As noted earlier, Student 1’s text was the same draft as used during the think-aloud process; whereas Student 2’s text was presumably taken from a later draft. It was not evident from what draft other writing scripts were taken. Therefore, the discussion of the product representing Rewadee’s rating process must be based on the analysis of the writing scripts provided regardless of what draft the students composed. To examine Rewadee’s assessment practices the feedback types she provided on each writing script were analysed. Examples of the feedback are illustrated below:

Text 6.318 Student 1 -0.25 When you want to go somewhere, you might feel very bore d because the , traffic in the city is very bad. About the air, you will got the bad air from Plural -0.5 health bad -0.5 A pronoun r many factory, it can make you be a bad health man. Although living in should be the country and living in the city might give the same residence, living in used instead of the country and living in the city are give you the different experience of the noun that has life. been stated in order to avoid word - Good organisation of ideas repetition.

- Easy to follow

18 The commentary feedback in the writing scripts was translated from Thai. 192

Text 6.4 Student 3

Football and basketball

Football and basketball are a very different. The football plays in the stadium, but basketball plays in the hard cord. The football has players more than basketball.

Football has 11 people but basketball has 5 people. Football uses foot for playing but basketball uses hand for playing.

good comparison of the differences point by point

x - use repetitive words x - no transitions These make the writing less interesting. x - use the same connectors - There should be a conclusion of aspects of differences such as playing places, players, period and how to play

As seen in the above extracts, Rewadee employed all three types of feedback: indirect feedback, direct feedback and end feedback. The indirect feedback includes underlines, circles and insertion symbols to locate the errors in relation to the rhetoric and ideas, and the language aspects. It also involves correction codes such as ‘plural’ to indicate the correct word form in Text 6.3 and commentary feedback which are attached to error locations to explain the correction to the error as demonstrated in Text 6.3. The direct feedback shows Rewadee’s error correction was accomplished by adding the correct words or letters as seen in Text 6.3, and by deleting words as in Text 6.4. The end feedback represents Rewadee’s overall comments of the whole composition (for a full example of the actual marked writing scripts see Appendix R). The frequency of her feedback types across all marked texts is illustrated in Table below:

Table 6.4 Mean scores of feedback types Types of writing scripts Indirect Direct End feedback feedback feedback Total Scored 15 and lower 5.58 3.42 0.42 9.00 (n = 11) Scored higher than 15 7.45 2.82 0.36 10.64 (n = 12)

193

Note that the highest score assigned was 18 and the lowest score was 12 from the total score of 20.

As illustrated in Table 6.4, Rewadee provided slightly more feedback in the writing scripts which received a score higher than 15. Considering the different types of feedback given to each group of scripts, it is evident Rewadee offered somewhat more indirect feedback in the scripts with 15 marks and higher, while she provided hardly noticeably more direct feedback and end feedback to the scripts scored 15 and lower. This small difference suggests Rewadee generally gave feedback to the students’ writing regardless of the writing quality. The greater emphasis on indirect feedback may imply it aimed to encourage the students to engage in self-editing (Ferris & Roberts, 2001). This aim appeared to be reflected in the use of correction symbols to indicate the types of errors the students made such as the code for ‘plural’ in Text 6.3 above. The employment of correction symbols seems to cohere with her view reported earlier: “I have correction symbols which I distributed to the students in the first class. ... When I mark writing, I told them that I would follow the symbols.” The qualitative indirect feedback and end feedback data show Rewadee provided an explanation of how to correct the errors: “A pronoun should be used instead of the noun that has been stated in order to avoid word repetition,” (see Text 6.3), and that she identified both the strengths and the weaknesses of the students’ writing: “good comparison of the differences point by point, ... no transitions” (see Text 6.4). It may be argued this practice indicates the product of the students’ writing was treated as a learning resource and that Rewadee attempted to promote student learning through her comments on the texts. This claim can be supported by her comment in the stimulated recall interview: “anyway, feedback should be given. The writing cannot be marked and returned [without having feedback]. Otherwise, the students will not improve.” Rewadee’s comment appears to cohere with the employment of feedback to support a formative writing purpose which aims to improve the student’s writing (Speck, 2000). Overall, Rewadee’s actual assessment practice demonstrates that she focused on writing as a process. The students’ learning processes were likely to be promoted through the writing scripts. When assessing the texts, Rewadee’s decision making appeared to be based on her personal judgements which were greatly influenced by the analytic rating criteria. To synthesise the discussion on the case study of Rewadee, the

194

next section will summarise the major themes and trends concerning her views and practices on writing assessment.

6.2.5 Summary of Rewadee’s views and practices Rewadee’s writing instruction and assessment practices appeared to support the assertion that she viewed writing as a process. A range of the teaching and assessment strategies she employed addressed the formative purpose of writing. In relation to the data presented, the students were seen as key agents in the learning process; they were encouraged to engage in self-directed learning activities such as selecting the topic of their writing based on their preferences and conducting the pre-writing stage independently. Due to the employment of clear analytic rating criteria, Rewadee appeared to conduct transparent and reliable assessment. During the actual processes of assessment, Rewadee appeared to employ both linear and recursive rating sequences when attending to the writing scripts with different proficiency levels. She was also self- consistent in the display of her decision-making behaviours across the three areas of focus despite the different rating processes she employed. Overall, Rewadee’s reported practices and her actual practices during the think-aloud protocols were highly consistent. These issues will be revisited and discussed in more detail in Section 6.4.

6.3 Sak This section will begin with Sak’s profile, describing his demographic data and professional information, including his teaching and training experiences based on the data gathered from the questionnaire and interviews. This will be followed by an account of his writing course. The profile and the description of the writing course will be presented to contextualise the data presentation and discussion in this study. Then, the information obtained in relation to Sak’s views of writing instruction and assessment processes will be highlighted and framed according to one key theme representing classroom practices. The data were sub-divided into sub-categories as shown below. In the section on actual assessment practices, decision-making behaviours displayed by Sak during his think-aloud protocols will be presented to illustrate his rating processes. The products of his assessment (i.e., students’ written scripts) will also be shown to demonstrate his decision making. The last section will summarise key reiterated issues according to Sak’s views and practices.

195

6.3.1 Profile Sak is a male teacher in the 26 to 30 year-old age range who works at Northeastern University. He holds a Bachelor’s degree from a Thai university and was doing a Master’s degree during the data collection period for this study. At the time of the study he had been teaching English for six years including one-year experience in teaching English writing. In relation to his working experiences as a teacher for both English in general and English writing, in this study he is regarded as having moderate experience in teaching English, but little experience in teaching writing. With regard to training experience, Sak reported he had not attended any training on writing assessment. However, as a new teacher of English writing he stated he had a former teacher of writing as his mentor. When he encountered problems in teaching the writing course he reported he turned to his mentor for advice: “I consult a senior teacher who used to teach and asked her what this [probably the course content] means.”

6.3.2 The writing course context Sak was the only teacher of the writing course from which I collected data. The course aimed to develop the students’ writing at a paragraph level; that is, the students had to compose four text types: narrative, descriptive, comparison and contrast, and cause and effect. During the semester in which the data was being collected the course had only one class of 41 students. In relation to the four text types taught in class, the students were required to submit one main composition for each text type, and were also expected to develop multiple drafts prior to their final composition. Because Sak was the only teacher of this course he had complete freedom to design all teaching materials and rating criteria to assess the composition. He reported that he developed self- designed materials “based on three to four commercial course books in teaching writing.”

6.3.3 Classroom practices The data from Sak’s interviews illustrate his instruction and assessment practices. They can be categorised into two major themes: teaching practices and assessment practices. The former theme constitutes one category while the second theme relates to rating criteria Sak employed, his rating strategies, and the difficulties he experienced during assessment. The summary of Sak’s perceptions of classroom practices is presented in Table 6.5 below: 196

Table 6.5 Sak’s representation of classroom practices Categories Sub-categories Sample quotes Teaching practices I don’t focus on sentence structure much because they used to learn writing in their first year. Assessment Rating criteria I look at the components of a paragraph. practices Rating strategies I will score the last draft the students write best. Difficulties in Sometimes I don’t understand what the assessment students are saying and about the students’ word choice.

Teaching practices Sak appeared to promote a ‘writing process’ approach in his teaching. This practice was reflected in the teaching strategies he employed. For example, he reported he taught the students in class to engage in a pre-writing stage by having them brainstorm ideas in relation to the topic he gave them:

At the beginning of the semester, I introduced components of a paragraph to the students. ... Then, I had them work in groups, assigned a topic to each group and told them to do pre-writing activities by thinking of anything about the assigned topic. They had to brainstorm as many ideas as they can.

Because the writing course from which the data was collected was the second writing course the students were enrolled in Sak believed the students had covered the necessary language features in the previous course. He tended, therefore, to focus more on rhetorical and ideational aspects of writing than language aspects in his teaching of the current course:

I don’t focus on sentence structure much because they used to learn writing in their first year. So I mainly emphasise types of paragraph.

This practice was likely to be influenced by his concern about the students’ affective attitudes as reflected in the following comment. Also, the emphasis on language was seen as hindering the students’ actual writing ability:

197

If we focus on grammar more than 20-30% [of writing], the students will be anxious about grammar … If they are worried about grammar, they will write less [they may not show their actual writing potential]. We may also have to consider the nature of the students.

Sak’s concern for the students also prompted him to interview the students to identify the causes of their writing problems:

I can find out why they make mistakes. Are they caused by language interference or the lack of grammar knowledge?

As suggested in the above comments, Sak was more likely to address the writing process rather than the writing product. He appeared to engage the students in the process of planning their writing before beginning on the construction of the texts. The students were seen as independent text producers who were at the centre of the learning process (K. Hyland, 2003; Silva, 1990). It can also be inferred from Sak’s emphasis on rhetoric and ideas that he perceived students’ cognitive processing as prominent in students’ writing; whereas the language aspects were likely to be seen as secondary. These conceptions seem to reflect a view of writing as a cognitive activity which requires the students to engage in mental activities in order to construct a text (J. R. Hayes, 1996; Weigle, 2002). They may also influence how Sak conducted assessment practices.

Assessment practices With regard to assessment practices, Sak did not employ any specific rating criteria. Instead, he appeared to assess the students’ writing holistically based on his intuition. He reported that when assessing writing he focused mainly on the rhetorical and ideational aspects rather than the lexicogrammatical aspects:

First of all, I look at the components of a paragraph. I check if there is a topic sentence ... Do supporting details relate to a controlling idea written? Are there transition words?

I focus on the structure of a paragraph.

Punctuation isn’t checked at the moment. I want them to get key ideas [of writing a paragraph].

198

The greater focus on the rhetoric and ideas was also reflected in his practice to promote multiple draft writing. He stated he focused on the rhetorical and ideational aspects in the first draft and would take language use into account after the students’ ideas were settled:

When their ideas are OK, I will mark their next draft. ... When their ideas are correct, I will check grammar.

Sak’s attention to the ideas in the initial drafts of students’ writing appears to confirm his inclination towards the writing process aspects of writing instruction. As such, there is an emphasis on meaning development based on the ideas conceived during the prewriting stage of an initial writing draft, with linguistic accuracy being addressed in later drafts (Ashwell, 2000; Coffin, et al., 2003; John, 1990). This practice was most likely influenced by his concern for the students’ attitudes towards assessment and so that he tended to avoid the correction of errors without a thorough revision of the content of their writing:

In my opinion, if I focus on grammar [in the first draft], the students will be stressed and use my correction right away. They will not reorganise their ideas. ... I want them to rethink. I don’t want them to follow my grammar correction.

Sak did not appear to have in mind a fixed number of writing drafts the students should compose. His marking and the students’ revision of their drafts was described as cyclical, as he indicated they could continue until the writing was perceived as meeting his expectation:

The students who fail are told to rewrite. If the second draft still fails, I will let them rewrite again.

If [the third draft] is not OK, I will let them rewrite again. But this is a rare case. Mostly, the students can write well in their third draft.

The writing score was said to be awarded to the last draft of writing:

I will score the last draft the students write best.

Sak’s emphasis on the use of multiple draft writing suggests he believed improvement in the students’ writing may be achieved through multiple revisions based

199

on his feedback (Coffin, et al., 2003). This concept would also account for his employment of a writing process approach in his instruction and assessment practices. The difficulties in assessment highlighted by Sak are related to his focus on the ideational aspects of writing. Even though he shared the same first language with the students, he reported that trying to understand their ideas was not always straightforward:

Sometimes I don’t understand what the students are saying and about the students’ word choice. ... Perhaps they don’t have a big vocabulary bank. When they write, they have to look at a lot of examples. Sometimes they use a Thai-English dictionary and choose words which are inapplicable in the context.

This difficulty appeared to lead to the kinds of feedback strategies he employed. He stated he preferred giving feedback to the students in person because he could discuss the errors with them:

I have to talk to the students. What do you mean by this? I don’t get it. Can you explain it? When they explain, I will suggest them use this (kind of) sentence.

The above comments relate to the complexity of writing, in particular the production of comprehensible ideas. A considerable level of cognitive processing is required from both writers – to construct a text – and assessors – to evaluate it (Gipps, 2012; Jeffery & Selting, 1999). Writers need to employ a wide range of writing features both at the rhetorical and lexicogrammatical levels in order to deliver their intended messages and to meet the assessors’ expectation. In relation to Sak as an assessor, understanding the students’ thoughts may be seen as necessary before continuing on the assessment of the textual features. Therefore, interviewing the students individually tended to be his preferred technique to assist them with their comprehension. Overall, Sak’s assessment practices indicate that the writing process was emphasised through the promotion of multiple draft writing. The students were regarded as central to the learning process. The nature of the students, particularly their affective attitudes towards learning, was an influential factor influencing Sak’s assessment practices.

The above discussion illustrated Sak’s views and beliefs about his instruction and assessment practices. Nonetheless, his actual practices in teaching and assessing 200

may not equate to what he reported. To provide a richer understanding of how Sak conducted assessment in his instructional work, his actual assessment practices, as shown in the assessment of the writing tasks, are presented in the next section.

6.3.4 Actual writing assessment practices Following the same data collection procedures as for Rewadee, Sak’s actual assessment practices were examined through think-aloud protocols conducted during his assessment of two writing scripts written by two students in the same class. It needs to be noted here that Sak appeared to hesitate in selecting the two writing scripts to be assessed during the think-aloud protocols. Despite being prompted to select any two writing scripts from the pile of 33 writing, he carefully selected them by turning over the scripts and skimming through some of them before making his selection. His hesitation may have partly resulted from the fact he was requested by his senior colleague who acted as his mentor to replace her as a research participant in this study. This issue will be revisited and discussed in Chapter 8. The think-aloud protocols were video-recorded and replayed for the subsequent stimulated recall interviews (for a detailed account of data collection procedures see Section 3.7). Sak’s think-aloud protocols were analysed based on the adaptation of Cumming et al.’s (2002) descriptive framework (for a full detailed account of the analysis of the decision-making behaviours see Section 3.8.2.1) and both qualitative and quantitative analyses of the data will be illustrated. This section describes Sak’s practices while he was undertaking assessment of the two writing scripts. His thinking processes during the assessments are also reported, drawn from the think-aloud protocols. They will be supported by the data obtained from the stimulated-recall interviews. This section will begin with the description of the task type under assessment. Then, Sak’s rating processes will be demonstrated through an analysis of the sequences and trends in the decision-making behaviours he exhibited. The students’ marked writing scripts are also discussed in order to provide more profound understandings of the actual assessment practices. They were analysed by examining the feedback types Sak made on them (for a complete description of the analysis see Section 3.8.3). These issues are presented below.

201

6.3.4.1 Task profile During the data collection for this study, Sak assessed the students’ comparison and contrast text type. The writing was typically set out as one-paragraph; however, there was variation in the format produced by the students. For instance, some students constructed their writing in two or three paragraphs (hereafter referred to as ‘one- paragraph composition’ or ‘composition’ regardless of the number of actual paragraphs shown). The writing topic was set by Sak and focused on the differences and similarities between beaches and mountains. To accomplish the task, the students were required to read a five paragraph text provided by Sak, entitled “Vacationing at the beaches or in the mountains”(see Appendix S for the complete essay). The students had to summarise to text and draw their arguments from the points provided in the essay. They were also required to write an outline comprising the major and minor details to be discussed in the writing. Then, the students had to compose their essays based on their written outline. Because the marking of all other writing scripts took place after the period of data collection, a request was made of Sak to post a copy of all the marked writing including the two scripts assessed during the data collection. As a result they could be included, in the analysis of the product of students’ writing to triangulate with the data relating to his rating processes. Approximately a month after the data collection Sak posted the students’ writing showing the one-paragraph compositions without their outlines. As Sak intended to have the students rewrite their writing task it became evident the posted writing scripts were taken from a later draft and not the original draft that Sak assessed during the think-aloud protocols. Thus, the posted product of his rating processes (i.e., students’ marked writing) was not complete, nor identical to the drafts used in the think-aloud process. The product of the students’ writing was analysed based on the posted draft. The two writing task samples selected by Sak to assess while undertaking the think- aloud activity are shown below without their outlines:

202

Text 6.5 Locations about the Beach and the Mountains19 There are many aspects to see at the locations of beaches and mountains. First, the mountains have a long view with trees and rock. When we look down from the top of mountains, we can see everything around the area. Second, the beaches have a long view with water, sand and sky. Although the two places are different trait, they are beautiful for themselves.

Text 6.6 The beaches and the mountains On vacations, people like to the beaches and the mountains. They can enjoy and happy doing a lot of thing together at both places. The beaches and the mountain is activities of the tourists can do, the climate and the locations. In the beaches, we like to play beach volleyball and football, but in the mountains, we like to do rock climbing, hiking. I like go to the mountain, because it’s cool, and have beautiful view. The beaches is too. When we go to the mountain or the beaches, we can cook food such as barbeque, grilled meat and salad. In climate, the differences is that the mountains are cooler. But the beaches are suarmer [sic]. Location the mountains have trees and have long view, but the beaches have building, restaurants, department, hotel and resort. Location of the mountains are rocky, but the beaches are sandy. Finally, the beaches and the mountains is important for the people at like to travel. Have many activities and climate. Very lucky to have the beaches and the mountains. Sometime, but I like it. Will remain beautiful forever.

6.3.4.2 Rating processes The data presented below illustrate Sak’s rating processes as derived from his decision-making behaviours displayed during the process of assessing the two writing scripts. The rating processes were represented in the forms of sequences and trends in his decision making while attending to the two texts. The sequences show his qualitative judgements of the texts; whereas the trends present the overall frequency of the behaviours displayed.

19 These texts were typed exactly as shown in the original scripts. 203

As mentioned earlier, Sak’s students submitted their main essays with the outlines attached. When attending to the essays however he appeared to pay little attention to the outlines. Instead, he started his assessment by reading the composition before proceeding with the assessment process based on his personal judgements. When attending to the first student’s writing script, Sak initially focused primarily on the rhetorical and ideational aspects of the text. The process he employed is illustrated in the commentary below:

The topic is Locations about the beach and the mountains. I have to read before marking. There are many aspects to see at the locations of beaches and mountains. First, the mountains have a long view with trees and rocks. When we look down from the top of mountains, we can see everything around the area. In contrast, the beaches have a long view with water, sand and sky. But the beaches have not many components can make us impress because of the excellent atmosphere until can't forget it. Although the two places are trait different, they are beautiful for themselves. I have to check how many aspects this student writes about. [Sak turns to the page with the outline and reads it.] He writes about location and environment of the mountains and the sea and says how they differ. The first aspect is the mountains have a long view. The second aspect is beach. He compares that mountains have a long view. This writing is not good. It’s not good because I assigned the students to compare three aspects but this student writes only about the location. I’ll write a comment telling him to write more. [Sak writes a comment on the script while murmuring ‘Compare three things’.] This writing needs to be rewritten. [Sak writes a comment on the script20.]

As demonstrated in the extract above, Sak generally attended to the interpretation strategies in the rhetorical and ideational aspects of writing. He read the overall text in order to monitor his assessment and identified its topic development by itemising ideas in the outline. This practice of reading the text appeared to be affirm his

20 As Sak did not post to me the original copy of the writing task he assessed during the think-aloud protocols, I could not relate his reaction to the writing seen in the videos with specific points he commented on the scripts. Therefore, only a general description of Sak’s reactions is shown in the transcripts as I transcribed his reaction to the scripts according to his actions captured in the videos without a reference to the texts. 204

comment during the stimulated recall: “When I mark writing, I first read the text. Sometimes I read it two or three times to comprehend the writing and see how the students summarise my guided essay.” This finding implies Sak attempted to comprehend the ideas before further assessing the text. He was found to compare the student’s composition with the outline attached. He explained in the stimulated recall interview that this behaviour was so he could check the relationship between the main text and the outline: “I analysed what ideas the student proposed and turned over to the next page to see what ideas he wrote [in order to check their relationship].” His intention to comprehend the overall text before evaluating it suggests he was attempting to conceptualise the image of the text (Freedman & Calfee, 1983; Wolfe, 1997; Wolfe, 2006) as discussed in Section 5.2.5.2. After identifying the development of ideas, Sak evaluated the text immediately after reading it. Task completion appeared to be taken into consideration when judging the writing quality: “It’s not good because I assigned the students to compare three aspects but this student writes only about the location.” This consideration by Sak resulted in his decision to have the student rewrite the text, and he noted a comment to this effect on the script. During the stimulated recall interview, Sak further articulated the details of his comment on the script whenever he requested the student to revise the structure of the paragraph: “I told him to rewrite his work and asked him to check what his topic sentence is, what supporting details are, how many aspects there are, and which sentence is the concluding sentence. I also told him to use transition words to indicate the aspects. I’ll note them down in a piece of paper to tell him again.” Sak also outlined his plan to provide feedback to the students in class by re-emphasising the paragraph structure: “I may have to discuss the structure of a paragraph with the students again.” These comments suggest Sak highly valued the rhetorical aspects of the writing as reported in the earlier interview data: “I focus on the structure of a paragraph.” When Sak found the task was not complete, rather than stopping the assessment process, he revisited the text and started assessing it again by focusing on the language aspects before making a final scoring decision:

There are many aspects to see at the locations of beaches and mountains. First. Here it has to say ‘many differences’. I put a space for him to fill in.

205

First, the mountains have a long view with trees and rock. When we look. Spelling here is incorrect. [Sak underlines a word and writes a comment on the script.] Spelling must be corrected. When we look down from the top of a mountain. [Sak underlines words and draw a line to link them together.] This is wrong about adding -s or a singular, either of them. We can see everything aound the area. In contrast, the beaches have a long view water, sand and sky. But the beaches have not many components... can make us impress because of the excellent atmosphere until can't forget it. [Sak underlines a word.] This is wrong. Until [Sak translates the word into Thai.] There should be replaced by other words. It can make us impress because of excellent atmosphere that we can't forget it. ‘That’ should be OK. [Sak writes a comment on the script.] ‘Until’ is misspelt. Although the two places are different, they are beautiful. Wrong spelling. [Sak underlines a word and writes a comment.] For themselves. He talks about the location only and talks about the view. I’ll have him rewrite and I’ll mark it again.

The above extract shows that when attending to the language aspects of the students’ writing Sak’s general practice was to read relevant text segments and judge them interdependently based on his personal opinion. On some occasions he identified errors and proposed corrections. It was also evident that the writing script above was regarded by Sak as lacking proficiency due to the incomplete ideas in the text. However, Sak’s focus on the language aspects in this writing script appeared to contradict his reported view in the interview data where he claimed he focused on the language after he was confident the ideas were well organised: “When their ideas are correct, I will check grammar.” Finally, having identified problems at the rhetorical and ideational level as well as in relation to aspects of language, Sak confirmed his decision of having the student rewrite the composition before proceeding with his assessment: “I’ll have him rewrite and I’ll mark it again.” When attending to the second writing script, Sak started his assessment in the same vein as in the first script; that is, he read the whole text before focusing on particular text segments:

Vacations for the beach and the mountains. In vacations, people like to the beaches and the mountains. They can enjoy and happy doing a lot of things 206

together at both places. The beaches and the mountain is activities of the tourists can do, the climate, and the locations. In the beaches, we like to playing beach volleyball and football but in the mountains, we like to work climbing, hiking when we go to the mountains. On the beaches, we can cook similar food such as barbecue, grilled meat and salad. In climate the differences is that the mountains are cooler, the beaches are warmer. In aspect of the location between the mountains and the beaches, the mountains have many trees and have long view, but the beaches have building, restaurants, department, hotel and resort. Locations of the mountains are rocky but the beaches are sandy. Topic sentence is ‘on vacation’ [Sak marks something on the script.] People like to the beach and mountains. They can enjoy doing a lot of things together at both places. Mmm, the topic sentence is incorrect. There’s no controlling idea. They can enjoy doing a lot of things together. [Someone is talking loudly in the background. Sak is distracted and looks in the direction of the loud noise.] The beaches [The teacher looks up again in the direction of the noise.] And the mountains is activities of the tourists. This needs to be the topic sentence. [Sak writes a comment on the script.] Topic sentence [Sak underlines a word in the script.] Play, playing. Incorrect. There must not be -ing. Beach volleyball and football but [Sak marks something on the script.] A capital letter in the sentence is wrong. But in the mountains, we like to rock climbing We like to do rock climbing. [Sak writes a comment on the script.] When we go. No –ing, ‘going’. [Sak marks something on the script.] When we go to the mountain or the beaches, we can cook similar food. This is a similarity. Such as barbecue, grilled meat and salad. The climate differences is that the mountains are cooler but. There’s no full stop. [Sak marks something on the script.] But the beaches are warmer. In aspect of the locations between the mountains and the beaches. This is a fragment. [Sak underlines a sentence and writes a comment above it.] It’s a fragment. There’s no verb. The mountains have many trees and have long view but. (inaudible) in a sentence [Sak marks something on the script.] But the beaches have building, restaurant, department, hotel and resort. The location the mountains. It’s ‘of’ [Sak marks something on the script.] The locations of the mountains are rocky but the beaches are sandy.

207

As shown in the above extract, after reading the whole text Sak assessed the rhetorical and ideational aspects by identifying and evaluating the topic development. Then he changed his focus to draw attention to the language aspects. He typically identified the language errors and proposed correction. His approach to error identification and correction was identical to his approach when attending to the first script; that is, his comments on the errors were very brief: “No –ing, ‘going’”. However, when assessing this text he regularly wrote or marked a comment on the text. It may be assumed that his purpose for this behaviour was to provide feedback to specific points in the writing, suggesting an emphasis on promoting students’ writing improvement through written feedback (Speck, 2000). Unfortunately, the location chosen by Sak for the data collection was a room with public access. As a result, the data extract shows he became distracted by loud noises in the room. However, this type of location with its many distractions, seemed to be very different from where he usually conducted his assessment of student scripts. As he explained in his stimulated recall interview, “Mostly, I spend time marking the writing alone. Sometimes I mark it at night. I prefer marking at night. It’s quiet and I can concentrate well.” This comment supports the notion that much cognitive processing is necessary when raters assess writing (Weigle, 2002). In addition, the change from his usual location for assessment may have to some extent affected the validity of the data collection in this study. This issue will be further discussed in Chapter 8. As the writing scripts referred to above were the first of many writing drafts expected of the students, Sak made his final scoring decision to have the student rewrite his work by considering aspects of the rhetoric and ideas in the text:

This needs to be rewritten because there needs to be a topic sentence, supporting details, and a concluding sentence. Transition words are needed. [Sak writes a comment on the script.]

Although the data on Sak’s rating processes revealed somewhat different decision-making behaviours during the assessment of the two writing scripts, the overall decision-making sequence can be illustrated in Figure 6.2 below:

208

Attending to the composition - Engage in both interpretation and judgement strategies for self-monitoring, rhetoric and ideas, and language.

Making a scoring decision - Summarise, distinguish or tally judgement collectively and articulate overall results.

Figure 6.2 Sequence of Sak’s decision-making behaviours

Sak’s rating processes overall appeared to largely depend on his subjective judgements. He primarily focused on the rhetorical and ideational, and the language aspects when assessing the two texts. However, there was a tendency for the rhetorical and ideational aspects to play a more important role in the scoring decision than the language aspects. This practice coheres with his reported focus on holistic rating criteria when conducting an assessment: “First of all, I look at the components of a paragraph. I check if there are a topic sentence. ... Do supporting details relate to a controlling idea written? Are there transitional words?” The focus on the rhetoric and ideas seems to be in accordance with Sakyi’s (2000) and Barkaoui’s (2010a) findings. Sakyi found one rater in his study emphasised essay topic, rhetorical structure and idea development; whereas Barkaoui found a novice rater’s holistic assessment was likely to be influenced by the quality of the content and the ideas developed in the writing. Even though it was evident there was a discrepancy between Sak’s reported practice and actual practice regarding the emphasis on the language aspects in the initial draft of writing, he reported in the stimulated recall interview that he focused greatly on what he saw as the serious language errors which may affect the student’s writing ability: “I only check some parts [of lexicogrammatical aspects] that he wrote incorrectly, the parts that I should not let him continue doing. ... Taking fragment as an example, he needs to know that he has to write in the form of a sentence. Otherwise, he will make the same error when rewriting the composition.” It may be argued that this comment represents the subjectivity with which he approached the assessment process.

209

In his stimulated recall interview, Sak expressed hesitation when making a decision to write a comment on the script. As he stated, “Mostly, I thought, eh, shall I write a comment? If there are a lot of errors, I will read first. Sometimes I write that there are no components [of a paragraph]. There needs to be a topic sentence, supporting details, a concluding sentence. I’ll tell the student about this and let them rewrite.” He also appeared to be in a dilemma about whether to evaluate the language aspects: “I was thinking if I should mark sentence by sentence. If I do, the student has to rewrite the work anyway. ... So I decided to write it down there [at the bottom of the page]. I’ll check some particular distinctive features.” Sak’s attempt not to discourage the students by pointing out an extensive number of errors may have contributed to Sak’s dilemma: “I’ll not write too many comments. If there are a lot of comments, they will feel why their writing has too many errors, something like that. ... I’ll give comments on major details first [probably about the paragraph format].” The tentative nature of Sak’s assessment practice reflected through these hesitations and dilemmas point, as has been noted by others, to the considerable cognitive processing demands of writing assessment (Weigle, 2002). In Sak’s case, the data suggest that the cognitive processing was not only required to comprehend the texts but it was also needed to make decisions about assessment approaches he could employ. Given that Sak had limited experience in assessing writing and had no formal prior education or training on writing assessment, he was likely to rely on his own knowledge and personal experience to implement a preferred method of assessment. For the purpose of qualitative data triangulation the frequency of the behaviours for each area of focus is also summarised to represent the trends in Sak’s decision making as illustrated ion Table 6.6 below:

210

Table 6.6 Frequencies of overall decision-making behaviours exhibited by Sak Student 1 Student 2 Freq. % Freq. % Freq. % Freq. % A. Self-monitoring Focus 31 67.39 33 73.33

1. Interpretation Strategies 16 34.78 19 42.22 2. Judgement Strategies 15 32.61 14 31.11 B. Rhetorical and Ideational 6 13.04 4 8.89 Focus 1. Interpretation Strategies 4 8.70 4 8.89 2. Judgement Strategies 2 4.35 0 0.00

C. Language Focus 9 19.57 7 15.56 1. Interpretation Strategies 5 10.87 5 11.11 2. Judgement Strategies 4 8.70 2 4.44 Uncodable segment 0 0.00 1 2.22

Total 46 100 46 100 45 100 45 100

Table 6.6 demonstrates the spread of Sak’s decision-making behaviours into the three areas of focus. The overall frequency of his decision-making behaviours when assessing the two writing scripts was relatively equal However, there was considerable difference in the decision-making frequency exhibited for each area of focus. Sak focused more on self-monitoring aspects when attending to the second script; whereas he placed greater emphasis on the rhetorical and ideational, and the language aspects when assessing the first writing script. When considering each writing script independently, the decision-making behaviours displayed by Sak followed the same pattern; that is, he focused on the rhetorical and ideational aspects to a lesser extent than the language aspects. This practice may be due to the different rating processes he applied when attending to the two aspects. He was found to read the overall text and mostly evaluate the rhetorical and ideational aspects collectively; whereas he treated individual text segments independently when assessing the language aspects. It is possible, therefore, that during his assessment practice Sak may make a lower number of decision-making behaviours when focusing on the rhetoric and ideas. As also illustrated in Table 6.6, Sak appeared to be self-consistent during his assessment of the two texts when attending to the three area of focus employing more interpretation strategies than judgement strategies across all three. In general, despite the slight discrepancy between his stated view and actual practice when attending to the language focus, Sak’s overall rating processes reflect his

211

self-reported views of his assessment practices. In order to provide complementary data to support a deeper level of understanding of Sak’s assessment process, the next section will discuss the analysis of the product of his writing assessment practices drawn from his assessment of the students’ writing scripts.

6.2.4.3 Product of writing assessment practices Following the assessment of the two writing scripts during the think-aloud protocols, Sak completed the assessment of the other 31 scripts written by the other students in his own time. Sak was also asked to post to the researcher all 33 marked writing scripts including the two scripts marked during the think-aloud sessions. As stated earlier, Sak posted a later draft of his students’ writing without including the outlines he asked the students to submit. Therefore, the discussion of the product of his rating process is ultimately based solely on the analysis of the posted composition. In order to investigate his assessment practices, the types of feedback Sak provided on each writing script were analysed. With regard to the three feedback types stated in Section 5.2.3.5, examples of feedback Sak gave to the students’ writing scripts are illustrated below:

Text 6.7 Student 2 sp In climate, the differences is that the mountains are cooler, but the beaches are suarmer. Location the mountains have many trees and have long view, but the beaches have is building, restaurants, department, hotel and resort. Location of the mountains are rocky, but the beaches are sandy. are who Finally, the beaches and the mountains is important for the people at like to travel. There are fragment

Have many activities and climate. Very lucky to have the beaches and the mountains. Sometime, but I like it. Will remain beautifull forever.

fragment

212

Text 6.8 Student 25

For the late difference the mountains have long view with trees and hills. The location at a lot of the mountains have hill and forest very much. And will have the country around the mountains. But the beaches have wide view with sand and water. And the location near the city have the hotel very much. - Concluding sentence

The extracts above illustrate Sak’s practice of providing different types of feedback in the texts. The indirect feedback includes circles and underlines to locate the errors in relation to the rhetoric and ideas, as well as the language aspects. Correction symbols such as ‘sp’ for a spelling mistake and ‘fragment’ as shown in Text 6.7 were also employed. The direct feedback represents correction to the errors by proposing the appropriate language use such as ‘a lot of’ and the deletion of a word as seen in Text 6.8. The end feedback demonstrates a phrase likely to show the feature missing from the text as in Text 6.8 (for a full example of the actual marked writing scripts see Appendix T). As the scores given to the writing scripts were not included on the paper, the feedback analysis could not be grouped in such a way as to make a comparison between the writing scripts with different proficiency levels. Instead, the frequency with which Sak employed each feedback type across all assessed texts is illustrated in Table 6.7 below:

Table 6.7 Frequency of feedback types Feedback Indirect Direct End Total types feedback feedback feedback Frequency 116 127 15 819 % 14.16 15.51 1.83 100

Table 6.7 does not show a marked difference in the employment of indirect and direct feedback, suggesting Sak’s preference to provide both forms of feedback to the students. The employment of indirect feedback and end feedback seems to confirm his emphasis on multiple draft writing. Both forms of feedback may be used as prompts to

213

assist the students to self-edit their work and improve their writing in later drafts (Ferris & Roberts, 2001). It may also be inferred from the noticeably high number of direct feedback examples that Sak was likely to prefer correcting the language despite his belief in the need for greater focus to be given to the rhetoric and ideas. This finding suggests a discrepancy between Sak’s actual feedback strategies and his own reported view. Moreover, they may also reflect his hesitation during the rating process and be a demonstration of his lack of knowledge in effective writing assessment. It was found in some writing scripts that Sak separated sentences in order to assist him to assess the rhetorical and ideational aspects as demonstrated in Text 6.9:

Text 6.9 Student 21 I [Introduction] /In my mind, the beaches and the mountains are different and similar. There are 1 two points that I will show you why they are different and similar. / First of all is climate. The climate at the mountains is fresh dry air but at the beaches is salt humid air.

/ Do you know why the mountain is cool but the beach is warm? / The reason is the mountains have a great number of plant. In contrast, the beaches consist of a huge sea, 2 sand and few trees. / Whatever, there are a few similarity of climate in two places. That is they are windy. The hot wind blowing all around the mountains same as the beaches. 3 The hot wind mostly blows from the sea to the beaches. / The last thing is view. The mountains have long view with forest flowers, trees, hills and cloud sea, where as the beaches have a giant wind view with sand and water. To sum up, it tells you how the C [Conclusion] beaches differ from the mountains. /

Sak’s practice of separating the sentences can be seen from two different perspectives. On the one hand, the separation of sentences to support the assessment of paragraph structure appears to confirm his reported emphasis on rhetoric and ideas rather than the language. On the other hand, it may be inferred that he was not confident with his evaluation of the text. Because Sak had little experience in the assessment of

214

writing, separating the sentences may have assisted him to ascertain the topic being developed in the text and to build his confidence in conducting assessments.

Overall, Sak’s actual assessment practices suggest he emphasised writing as a process through the promotion of writing multiple drafts and by assessing different features in each of the writing drafts. When engaging in assessment, his decision making appeared to be largely influenced by his intuitive impression of the texts. In order to synthesise the discussion based on the case study of Sak, the major themes and trends in relation to his views and practices on writing assessment are summarised in the next section.

6.3.5 Summary of Sak’s views and practices Sak’s writing instruction and assessment process largely emphasised writing as a process. The data suggest the students were positioned as major agents in the learning process and that their affective attitudes were likely to be taken into consideration in the writing pedagogy and assessment. Due to the intuitive holistic rating criteria used by Sak his subjective assessment was driven by his impression of the texts. His actual rating processes appeared to depend substantially on his personal preferences, particularly when attending to the language aspects of the text. In general, Sak’s reported practices seemed to be somewhat consistent with his actual practices during the think-aloud protocols. These issues will be revisited and discussed in the next section.

6.4 Summary This chapter discussed the single case analyses of Rewadee and Sak, who are regarded as novice teachers in this study. Their views and practices on writing assessment were represented and discussed. The emerging key themes in their individual views on teaching and the assessment of writing were identified and their actual assessment practices were revealed. The next chapter will represent the findings of the cross-case analysis of the four teachers to reveal commonalities and differences in their views and practices.

215

CHAPTER 7 CROSS-CASE ANALYSIS: COMMONALITIES AND DIFFERENCES IN TEACHERS’ WRITING ASSESSMENT PRACTICES

7.1 Introduction This chapter synthesises the findings obtained in the four single case analyses of the teachers presented in Chapters 5 and 6. The cross-case analysis of the four teachers will be presented and discussed. This chapter will begin with a summary of the teachers’ profiles, followed by a comparison of their writing course contexts. The information is included to provide a holistic representation of the four in-service teachers and their mutual contexts of writing instruction and assessment. The commonalities and differences in their views on writing instruction and assessment will be presented to highlight the major issues relevant to their classroom practices. Following this, the teachers’ actual assessment practices will be presented and discussed.

7.1 Teachers’ Profiles The four teachers: Ladda, Nittaya, Rewadee and Sak differed in several ways in terms of their personal and professional backgrounds. A demographic description of their personal characteristics and professional experiences is represented in Table 7.1 below:

Table 7.1 Description of the four teachers Name Gender Age Years of Years of Highest Training on teaching teaching education writing English writing assessment Ladda Female 51-60 35 35 MA Yes (workshops) Nittaya Female 41-50 27 7 MA Yes (coursework, workshop) Rewadee Female 31-40 3 4 months MA Yes (coursework) Sak Male 26-30 6 1 BA No

217

As seen in the above Table, the four teachers represented different age ranges and varied in their experiences of both teaching English in general and, more specifically for this study, in teaching English writing. Although Ladda and Nittaya had been teaching English for a different number of years, they appeared to be highly experienced in both teaching English in general and teaching English writing. They were also both trained in writing assessment processes, although their training varied in nature and duration (i.e., Nittaya received more substantial training than Ladda). Thus, both teachers may be regarded as ‘expert raters’ in this study. By contrast, Rewadee and Sak had limited teaching experience and little training in writing assessment. Despite Rewadee’s training as part of her postgraduate coursework she actually had very limited experience in teaching writing, as she herself explained. Moreover, although Sak had more experience than Rewadee in teaching English in general, his one-year of experience in teaching writing was still considered by him to be somewhat limited. He also lacked experience in writing assessment having received no formal training, unlike the other teachers in this study. Thus, Rewadee and Sak may be considered ‘novice raters’ in this study. The four teachers each worked at a university located in four different regions of Thailand: South University, Central University, North University and Northeastern University. The universities are characterised by different histories and levels of social status; with one being an established public university and the other three referred to as ‘new’ universities. The three new universities include one university transformed from a technical college, one university transformed from a university campus, and one university transformed from a teacher training college. All of the universities have been recognised as having a high level of diversity and a large gap in university developmental level. They are different in many aspects such as size, location, workforce, student quality, university quality and reputation (Office of the Higher Education Commission, 2008). In general, established universities are perceived as being able to enlist more qualified teachers than ‘newer’ universities because they have well-established policies to develop their workforce. They are also regarded more reputably and admit better quality students than newer universities. These characteristics may influence the internal course management in each university included in this study.

218

7.2 The Writing Course Contexts The organisation of the writing course at each university was based primarily on the teachers of the courses, with minimal control from the their respective departments. In general, the teachers had to follow broad criteria including course score allocation and course objectives as stated in course syllabuses. However, they had freedom to conduct instruction and assessment in their own classes based on their preferred teaching and assessment strategies. Ladda and Rewadee worked with co-teachers to deliver the writing course and they and their co-teachers shared the teaching materials, the course outlines and the course score allocation. However, the degree of cooperation between the ‘main teachers’ and the co-teachers on the writing tests varied. Ladda employed the same tests as her co-teachers and followed the same agreed rating criteria to assess the exam papers. By contrast, Rewadee and her co-teacher appeared to be more autonomous. They had complete freedom to design their own tests and assess them using their individual preferred rating criteria. In contrast, Nittaya and Sak were the only teachers of their respective writing courses and had complete freedom in relation to writing instruction and assessment. The variability in their course management suggests that in a regular classroom context the teachers of the courses played the most important role in the process of instruction and assessment.

7.3 Classroom Practices In relation to the four teachers’ views of their instruction and assessment practices, major themes were framed and discussed according to each teacher’s practices. The prominent issues to emerge for each teacher are summarised and presented in Table 7.2.

219

Table 7.2 Summary of major emerging issues in the teachers’ classroom practices Ladda Nittaya Rewadee Sak Minimising Encouraging self- Promoting writing Engaging students students’ directed learning process (e.g., in writing process autonomous (e.g., students students were (e.g., learning (e.g., chose their expected to go implementation of control of writing preferred quiz through writing pre-writing stages) topic and scores) stages) assignment format) Assessing writing Promoting learning Assessing writing by emphasising Assessing writing process (e.g., features at rhetorical aspects features at flexibility of rhetorical and on initial draft and rhetorical and writing content in lexicogrammatical language aspects on lexicogrammatical writing courses; levels later drafts levels with an giving feedback on emphasis on writing and letting Promoting multiple Promoting multiple language accuracy students correct drafting (e.g., drafting (e.g., errors based on the students may students may Emphasising lesson learned) rewrite until getting rewrite until fairness of scores satisfactory scores) meeting teacher’s across writing Assessing writing expectation) scripts features at Promoting learning rhetorical and process (e.g., Reducing concern lexicogrammatical encouraging about student’s levels with student self-edition attitudes (e.g., somewhat more from feedback) revealing student’s emphasis on name when giving grammatical feedback in class) aspects Expecting a single Considering draft of writing students’ positive (i.e., one rewritten attitudes as a draft was allowed primary focus (e.g., for only writing proposing the least under expectation) correction to errors)

Promoting multiple drafting (e.g., students may rewrite until meeting teacher’s expectation)

220

As shown in the above Table, the teachers’ views on classroom practices reflect two prominent concepts of writing instruction: writing as a product and writing as a process. ‘Writing as a product’ is considered a traditional writing instruction approach which primarily emphasises the textual features of writing; whereas ‘writing as a process’ primarily focuses on the process of text construction (Grabe & Kaplan, 1996). The two approaches entail particular principles which were reflected in the four teachers’ reported practices. Ladda’s reflections on the ‘writing as a product’ approach appeared to permeate her teaching practices. The students tended to be seen as passive learners who had to follow decisions about their learning made solely by Ladda. She appeared to place a great deal of emphasis on controlling the students’ writing in terms of the writing topics and format. Ladda also tended to encourage the students to engage in a linear composing model through the process of outlining, composing and editing (Grabe & Kaplan, 1996). One final draft of writing was preferred, but when revision was required the students were expected to demonstrate their writing ability alone without any help from others. When assessing the texts, attention to textual features was seen as a primary objective. Moreover, language accuracy was considered the main indication of writing quality. Implementing this pedagogical approach when teaching students to write seems to mean emphasising textual features such as linguistic knowledge, lexical choices and sentence structures (K. Hyland, 2003). By contrast, the other three teachers reflected the ‘writing as a process’ approach. In their reported practices each teacher appeared to encourage the students to engage in various writing stages: planning, writing, editing and revising. Students were required to employ a number of strategies to facilitate the writing process and to gradually develop their final written product (Coffin, et al., 2003; Hedge, 2000). It was evident in the data that all three teachers emphasised multiple draft writing, suggesting they expected the students to improve their writing through multiple revisions which is a process regarded as integral to writing development (Cho, 2003). Each teacher’s marking of the first draft of writing serves a formative purpose in that they provided feedback to show the student how they could improve their writing. This is a student- centred approach and they are perceived as independent text producers (K. Hyland, 2003; Silva, 1990). Furthermore, the students’ positive attitudes towards the learning process and assessment are taken into account. The teachers’ concerns about the

221

students’ attitudes and needs appeared to guard against imposing constraints on the students in order to avoid a negative impact on the writing process and the writing quality (Cho, 2003). Overall, the four teachers’ classroom practices appeared to be positioned within two different orientations to writing pedagogy. Moreover, they also appeared to be reflected in the teachers’ assessment practices.

7.4 Assessment Practices With regard to the four teachers’ assessment practices, they varied in their organisation of the writing task, rating processes and attention to the writing product. When assigning the writing task, Ladda, Nittaya and Sak asked the students to submit the main text together with supplementary parts such as outline and sentence construction. Rewadee was the only teacher who preferred the submission of the main text as the final product of the students’ writing. In relation to their rating processes, the four teachers exhibited a variety of decision-making behaviours. Their think-aloud protocols during the assessment of the students’ writing were analysed both qualitatively and quantitatively based on an adaptation of Cumming et al.’s (2002) framework to illustrate the sequences and trends in their rating processes. The features of each teacher’s practice regarding their rating sequences are summarised and demonstrated in Table 7.3 below:

Table 7.3 Summary of sequences of the rating processes Rating Ladda Nittaya Rewadee Sak processes Patterns of Linear Linear Linear and Recursive rating recursive processes Reading relevant Reading relevant Reading the text segments text segments Reading relevant whole text and treating and treating text segments before reading them them and treating and treating individually individually them individual text individually segments

222

Rating Ladda Nittaya Rewadee Sak processes Attention Focusing mainly Focusing on Addressing both Focusing on the to writing on interpretation different interpretation rhetorical and strategies in the strategies in and judgement ideational rhetorical and different aspects strategies in the aspects in the ideational when attending self-monitoring, first reading; aspects in the to different parts the rhetorical greatly outlines of writing (i.e., and ideational, emphasising the initial focus on and the language language aspects Interspersing interpretation focuses when attending decision making strategies related to individual text behaviours into to the rhetoric segments the self- and ideas, and monitoring, the later focus on rhetorical and addressing the ideational, and judgement the language strategies related focuses in the to both the main text rhetorical and ideational aspects the language aspects)

Scoring Considering Considering Considering Mainly decision contextual contextual both the considering the factors (e.g., factors (e.g., rhetorical and rhetorical and teacher’s teacher’s tacit ideational, and ideational personal knowledge, the the language aspects experience with course content aspects the writing class and provision and the course for feedback) Adhering to content) rating criteria Considering Considering both the both the rhetorical and rhetorical and ideational, and ideational, and the language the language aspects aspects with a greater emphasis on the latter

223

Rating Ladda Nittaya Rewadee Sak processes Indicating writing quality using language errors

As seen in the above table, the four teachers appeared to vary in their rating process patterns. Ladda and Nittaya seemed to employ a similar pattern, attending to the different parts of the writing task submitted by the students in a linear fashion (e.g., focusing on the outline and the main text before making a scoring decision). Throughout their assessment, both teachers treated text segments individually. Rewadee tended to employ a linear approach when assessing higher proficiency writing. However, when assessing lower proficiency writing her rating approach was likely to be recursive. She revisited the text to confirm her comprehension when encountering unclear text segments. Despite Sak’s employment of a recursive approach, his practice differed slightly from Rewadee’s approach. While Rewadee attended to individual text segments, Sak assessed the writing by initially reading the whole text for comprehension before revisiting it and attending to individual text segments. It can be inferred that the different approaches employed by the teachers resulted from their different level of teaching experience. Ladda and Nittaya, as expert raters, were likely to make their judgements right after reading the texts. Thus, they appeared to be more assured in their rating processes, supporting Sweedler-Brown’s (1985) claim that an experienced rater’s confidence in assessment is gained through his or her extensive experience. By contrast, Rewadee and Sak were likely to reflect their uncertainty about assessment through their recursive marking. As novice teachers they may have lacked the experience to consolidate their assessment practices in the same way as experienced teachers who assess according to their ‘core’ principles and who are far less conscious of their practices (Richards, et al., 2001). Even though the four teachers employed different approaches when assessing the writing, one common practice found across all teachers was their intention to comprehend the rhetoric and ideas before proceeding with the assessment. This practice appeared to support the notion of ‘text image’ as proposed in Freedman and Calfee’s (1983) information processing model of a rater. Following this model, raters have to go through three essential processes: creating text image from reading and comprehending

224

a text, judging the text image and building up an impression, and articulating judgement. This model illustrates that raters create the text image in their working memory before storing it in their long-term memory. They then evaluate the text image using cognitive processing to produce the results. Regarding the practices of the four teachers, they each appeared to create their text image differently. Ladda, Nittaya and Rewadee created the text image from the small text segments they read; whereas Sak tended to develop the text image based on a whole text reading. The different approaches may have resulted from the teachers’ diverse working environment and experiences (Pula & Huot, 1993). During the assessment the teachers also focused on the language aspects of the text. Their general behaviours involved identifying errors and proposing corrections. However, their verbalisation of the behaviours appeared to vary. Ladda, Nittaya and Rewadee made more extensive commentary on each text segment compared to Sak, who produced briefer comments. This difference partly coheres with the finding from Cumming et al. (2002) that expert raters were likely to produce more extensive verbalisation than novice raters. This finding appeared to be true of Ladda and Nittaya, the expert raters in this study; however, it also true of Rewadee, a novice teacher. It may therefore be argued that the extent to which teachers verbalise their decision making is influenced by factors other than rater expertise. In Rewadee’s case, her educational experience in relation to writing assessment may be an influential factor. The knowledge she gained through her experiences as a learner may have shaped the underlying principles she applies to her rating processes, and may also have contributed to the level of confidence she has to act during the rating processes. As a result, Rewadee may therefore produce extensive verbalisations similar to those demonstrated by the expert raters in this study. When making scoring decisions the four teachers all appeared to consider rhetorical and ideational aspects, suggesting content and ideas were the major attributes of their scoring decisions. However, language aspects were found to be taken into account to different degrees. While Nittaya and Rewadee appeared to consider language aspects as one major component of the score, Ladda greatly emphasised this component and relied heavily on language accuracy when considering the scores. Sak appeared to be the only teacher who did not to consider the language aspects in his scoring decision. The disparity between Ladda’s and Sak’s scoring practice seems to support other studies

225

comparing novice rater’s and expert rater’s use of holistic scoring. The quality of content and ideas in the writing is likely to influence the novice rater’s assessment while language accuracy appears to greatly influence expert’s rating processes (Erdosy, 2004; McNamara, 1996; Rinnert & Kobayashi, 2001; Song & Caruso, 1996). With regard to rating criteria, Nittaya and Rewadee were the two teachers who used explicit rating criteria, although they differed in their approach to the allocation of scores. Nittaya was found to allocate scores to two broad rating criteria domains and to judge writing features in each domain holistically; whereas Rewadee allocated fixed total scores to all writing features and used them as references for score deduction. Among the four teachers, Rewadee was the only teacher who explicitly related her assessment to the analytic rating criteria she used. The other three teachers appeared to assess the writing holistically based on their personal judgements and to gradually formulate their scoring decisions based on these judgements. The different practices may have resulted from the types of rating criteria the teachers employed. While Rewadee relied on analytic rating criteria with fixed allocated scores corresponding to particular writing features, the other three teachers assessed the students’ writing holistically based on their impression of the texts. Despite being a novice teacher, Rewadee appeared to be confident in her assessment due to the use of explicit analytic rating criteria, cohering with Hout’s (1993) claim that expert raters in his study did not show anxiety in their assessment because they could rely on the rating criteria. It may be argued in Rewadee’s case that a novice rater can also undertake assessment confidently if they have specific rating criteria to rely on. Overall, the qualitative data on actual assessment practices revealed noticeable variability in the four teachers’ practices in relation to the rating sequences employed by each teacher. Thus, their level of experience and rating criteria may be regarded as important influential factors.

In order to provide supplementary data pertaining to the four teachers’ rating patterns their decision-making behaviours were also represented quantitatively to illustrate trends in their rating processes. The frequency of their decision-making behaviours is represented in Table 7.4 below:

226

Table 7.4 Frequency of decision making behaviours Ladda Nittaya Rewadee Sak (%) (%) (%) (%) A. Self-monitoring focus 51.77 60.33 56.88 70.33 1. Interpretation strategies 31.91 43.84 30.11 38.46 2. Judgment strategies 19.86 16.49 26.77 31.87

B. Rhetorical and ideational focus 19.50 17.95 23.05 10.99 1. Interpretation strategies 15.96 13.57 15.99 8.79 2. Judgment strategies 3.55 4.38 7.06 2.20

C. Language focus 28.72 21.71 20.07 17.58 1. Interpretation strategies 8.87 12.53 8.92 10.99 2. Judgment Strategies 19.86 9.19 11.15 6.59

Uncodable segment - - - 1.10 Total 100 100 100 100

Table 7.4 shows the variability in the teachers’ rating practices. All teachers exhibited decision-making behaviours most frequently in relation to the self-monitoring focus. Attention to the rhetoric and ideas and to the aspects of language varied. Rewadee was the only teacher to emphasise the rhetorical and ideational aspects more than the language aspects, with the other three teachers focusing considerably more on the language aspects. This difference may have emerged due to the different scoring approaches they employed. As mentioned earlier, Rewadee was the only teacher to use explicit analytic rating criteria with a fixed score allocation, with the other three teachers assessing the writing holistically. Thus, Rewadee may have based her assessment on the fixed scores she allocated to each writing feature and then formulated them as ‘core’ principles (Richards, et al., 2001) in her assessment. It is also possible she adhered to the rating criteria and was not tempted to turn her focus towards the language aspects, which are more distinctive features than the rhetorical and ideational aspects. When considering the decision-making strategies in each aspect, Table 7.4 reveals all teachers displayed higher decision-making behaviours in interpretation strategies than judgement strategies in the aspects of self-monitoring, and rhetoric and ideas. However, noticeable differences in the frequency in the language aspects were found. Ladda and Rewadee exhibited higher judgement strategies than interpretation strategies while Nittaya and Sak displayed more interpretation strategies than judgement

227

strategies. The different emphases in these strategies, regardless of the teacher’s experience, demonstrate the variable nature of a teacher’s practice as stated in many studies (e.g., Barkaoui, 2010a, 2010b; Lumley, 2002; Vaughan, 1991). In relation to their attention to the students’ written products, all four teachers provided feedback on the writing. However, further variation was noticeable in their feedback strategies. The frequency of all feedback types made by each teacher is demonstrated in Table 7.5 below:

Table 7.5 Frequency of feedback types Feedback types Teachers Indirect feedback Direct feedback End feedback (%) (%) (%)

Ladda 43.75 54.44 1.81 Nittaya 94.57 5.43 0 Rewadee 66.52 32.14 4.02 Sak 14.16 15.51 1.83

As seen in the above table, all four teachers varied in the frequency of feedback types they assigned to the students’ writing. While Ladda and Sak provided more direct feedback on the writing, Nittaya and Rewadee provided more indirect feedback. As direct feedback represented correction to language errors in the texts, the considerably greater amount of direct feedback than indirect feedback provided by Ladda and Sak suggests their emphasis on correction proposition. It may be argued that this finding further confirms Ladda’s product-oriented approach in her writing pedagogy which primarily focuses on the linguistic features of texts (K. Hyland, 2003). In Sak’s case, however, the greater amount of direct feedback than indirect feedback seemed to result from his inexperience in writing assessment as discussed in Section 6.3.4.3 rather than a distinct tendency to employ a product-oriented approach in his writing pedagogy. One explanation for this assumption is his promotion of multiple drafting as discussed earlier. In the cases of Nittaya and Rewadee, the fact they provided more indirect feedback than other feedback types may indicate their preference for student self- editing, confirming a process orientation approach in their writing pedagogy.

228

The discussion so far has illustrated the variation in the teaching and assessment practices of the four teachers. These differences are the result of various factors, particularly their professional experience and their orientations to writing pedagogy. The variations may also reflect different perspectives towards writing assessment within their particular classroom and institutional contexts. These issues will be further discussed in the next chapter.

7.5 Summary This chapter discussed a cross-case analysis of the four teachers to provide a more rounded and comparative view of their practices. The commonalities and differences among the four teachers’ views and actual practices in relation to the assessment of their students’ writing were demonstrated and discussed. The following chapter will bring together the findings presented in Chapters 4 to 7 in order to synthesise the discussion on writing assessment practices of teachers in regular classroom contexts in Thai public universities.

229

CHAPTER 8 WRITING ASSESSMENT PRACTICES OF TEACHERS IN EFL CLASSROOM CONTEXTS: DISCUSSION

8.1 Introduction This chapter will discuss the overall findings regarding the writing assessment practices of teachers in Thai public university classroom contexts. The findings are drawn from the collective views of writing teachers across the country as presented in Chapter 4, as well as from four individual teachers’ views and actual practices highlighted in Chapters 5 to 7. Presenting the key findings that emerge from the perspectives of both the collective group of teachers and from individual teachers allows for a rounded portrayal of teachers’ views and practices. The collective perspective provides a broad overview; whereas the individual analyses enable deeper understanding of what occurs within specific contexts. Contributions to research in the areas of writing assessment and teacher cognition are therefore provided from a dual perspective. The discussion of the findings is framed around three major themes: writing instruction in Thai public universities, writing assessment practices of teachers in classroom contexts, and teacher perceptions and writing assessment practices. The discussion of writing instruction in Thai public universities will highlight the organisation of writing courses for English major students in the university context and the teaching methods privileged by the teacher. The section on writing assessment practices of teachers in the classroom context will raise the issue of reliability and validity of assessment as reflected in the teachers’ assessment practices. Following this, the interrelation between teacher perceptions and writing assessment practices will be discussed.

8.2 Writing Instruction in Thai Public Universities Drawing on the findings presented in Chapters 4 to 7, a number of major reiterated findings with regard to writing instruction in Thai public universities are revealed:  Writing course management was undertaken independently by teachers with little oversight and support from their respective workplace.

231

 Teachers of writing courses were allocated to classes regardless of their teaching experiences.  Teachers had full responsibility for organising the courses in terms of writing pedagogy including teaching material selection and the degree of cooperation with their co-teachers. These findings suggest a very high level of teacher autonomy in relation to course management in the Thai public university context in that teachers are portrayed as the active agents in course organisation. This level of autonomy seems to be common in Thai higher education across the writing courses portrayed in this research. Even though quality assurance policies to promote international education standards have been implemented in all universities in Thailand (Prapphal, 2008), it seem likely they operate mostly at the institutional level. In contrast, the data in this study suggest that at the instructional level courses operate in a personalised way within their specific and individualised contexts. As such, they reflect traditional practices in universities which support “a high degree of professional autonomy for the people who work in them” (Brennan & Shah, 2000, p. 341). The autonomy experienced by the teachers who participated in this study appears to allow them to bring their own preferences and experiences into their instruction, without having to consider institutional guidelines or criteria. This level of autonomy could be seen in a positive light in that individual teachers can plan writing instruction flexibly and according to personal views. However, as highlighted in Chapter 4, it can also be seen as a challenge, especially for less experienced teachers, as it may result in little support institutionally in terms of professional training or opportunities for teachers to share their knowledge and experience of writing instruction and assessment. In these circumstances teachers have to rely on their own intuitions and resources to develop an appropriate pedagogic orientation that suits their students’ needs and assists their learning. Teaching and assessing writing is complex and challenging and for this reason many teachers, including experienced teachers, are reluctant to become writing instructors. As a result, novice teachers, with little experience in teaching and in teaching writing in particular may be assigned by program directors to teach writing. In Chapter 4, one of the teachers who participated in this research commented explicitly about the allocation of a novice teacher to teach writing in her department. This practice

232

is demonstrated by Rewadee and Sak, the two novice teachers featured in the case studies. Both suggest they had little choice in their class allocation even though neither was skilled in teaching writing. Rewadee, whose co-teacher provided her with teaching materials, appeared to have clearer personal guidelines about how to operationalise her teaching practices. Her approach to teaching the subject is likely to have been influenced by her prior postgraduate education relating to writing instruction and assessment, as well as the structure provided by the prescribed course syllabus she shared with her co-teacher. In contrast, Sak was provided with little support and subsequently expressed a considerable degree of frustration about how he could structure appropriate writing instruction and assessment. The fact that he was the only teacher of the writing course meant he had to rely on his intuition when designing teaching materials and when organising the course. Though he could seek help from his colleagues and a senior mentor teacher, decisions on how to conduct the course were made solely by him. With limited support and lacking professional experience in relation to writing instruction and assessment he thus appeared to conduct his instruction and assessment with considerable uncertainty. Sak’s lack of confidence in his teaching practices is supported in the literature about novice teachers’ anxiety when making decisions about their instruction. Tsui (2003) and Westerman (1991) found that when there is a curriculum to follow, novice teachers are reluctant to deviate their lesson planning from the prescribed curriculum. Instead, they appear to adhere closely to the curriculum they are given. As they lack well-developed instructional theory, their lesson planning is less effective than demonstrated by expert teachers and they spend much more time planning. They also lack the ability to anticipate the problems that may occur in their classrooms and have limited capacity to improvise the lessons during teaching. These challenges are in parallel with Farrell’s (2009) observation of novice teacher experience. He asserts that the anxiety experienced by novice teachers lies in the tension that comes with the process of learning to teach while also trying to be recognised as a ‘real’ teacher in their professional instructional culture. Furthermore, because novice teachers have, by definition, very limited teaching experience, their initial teaching experiences are often unpredictable and highly idiosyncratic (Johnson, 2002 cited in Farrell, 2009).

233

The allocation of novice teachers to writing classes suggests an unsettled status of writing instruction in the Thai university context. Writing is generally perceived as one of the most difficult productive language skills (K. Hyland, 2003) for students to learn. Moreover, it is an important academic skill that impacts all disciplinary courses and students’ further study. Teaching writing involves a complex process of engaging writers’ cognitive processing. Assisting students to master their writing skills therefore requires much more than a basic ability to teach. As K. Hyland (2003) contends, “the ability to teach writing is central to the expertise of a well-trained language teacher” (p. xv). Teachers must have knowledge of writing conventions and appropriate formative assessment if they are to contribute to the success of instruction. Following Bailey’s (2001) argument that effective learning is the result of appropriate teaching, writing classes therefore require expert teachers who are likely to be more effective than novice teachers in conducting writing instruction, especially if they are equipped with previous training in teaching writing. However, this study suggests teaching experience in actual classroom situations in Thailand is not always taken into account when allocating teachers to writing courses. As a result, teachers can be assigned to teach writing courses regardless of their expertise and professional background. They then have to conduct the instruction and assessment based solely on their personal assumptions and limited experiences which is likely to lead to great variability in outcomes of students’ writing performances within and across different universities. The findings also reveal the two commonly repeated writing approaches employed by teachers are writing as a product and writing as a process, as discussed in Chapter 7. Moreover, these approaches are driven by different underlying teaching principles employed by teachers. In addition, the orientations appear to cohere with two recognised teaching paradigms: teacher-centred learning and student-centred learning. Key differences between these two approaches lie in teachers’ perceptions of the nature of knowledge and the nature of learning. The teacher-centred approach tends to consider knowledge as objective and that teachers can facilitate its assimilation with appropriate practices. Furthermore, students are regarded as passive learners whose learning is under the control of dominant teachers. A student-centred approach, in contrast, aims to acknowledge students as active learners who critically and creatively engage in their learning, and who construct their own knowledge through various learning activities

234

(O’Sullivan, 2004; Rowell, 1995). The teacher’s role in these two approaches also differs. Teachers in teacher-dominant pedagogy tend to be knowledge transmitters; whereas they play the role of mediators of learning in student-centred pedagogy (Rowell, 1995). The contrasting role of the teacher in each pedagogical approach, as described in the literature, appears to be affirmed in the roles of the teachers found in this study. The ‘writing as product’ teacher sees a student’s deficiencies as a failure to perform received knowledge about language that is passed on to them and uses an approach that aligns with the teacher-centred view of instruction. In contrast, the ‘writing as process’ teacher exploits writing as a developing skill and views the student as ‘a social being’ (Rowell, 1995) who can generate their own knowledge. A teacher employing this approach is likely to take a student-centred view of instruction. The two common writing pedagogy orientations found in this study are also consistent with the central principles of language teaching proposed by Kumaravadivelu (2006). Indeed, the author introduces three approaches to the teaching of language: language-centred, learner-centred and learning-centred. The language-centred approach can be considered a traditional teaching method addressing the importance of linguistic forms. In this approach, linguistic structures are assumed to lead to language mastery and learners can retrieve them when they want to communicate the target language. Language is conceived as consisting of systematic discrete aspects learned in a linear, additive process in that a single aspect is introduced at a time. Repetitive practice is seen as essential for learners to develop command over the language, often at the expense of the achievement of communicative function. The learner-centred approach is described by Kumaravadivelu as sharing many similar features as the language-centred approach. For example, language development is presented as intentional rather than incidental. Language input in the classroom needs to be preselected and presequenced. In the same vein as the language-centred approach, learning under the learner-centred method is still linear and additive. However, the main difference between the two categories is the primary focus of teaching and learning. While the language-centred approach focuses on language accuracy, the learner-centred approach emphasises both language accuracy and fluency. Learners are expected to use both formal and functional repertoires to achieve communicative functions outside the class. Teachers are required to facilitate learners to develop language ability by using accurate grammar and performing fluent communication.

235

In the learning-centred approach, learner needs and wants are taken into account and language function is emphasised more than form. Moreover, the cognitive processes of learners are taken into consideration and language development is seen as incidental, cyclical and parallel. Language can then be learned through a series of meaning-based activities as learners are believed to acquire the language through their engagement in a series of learning activities. When comparing all three teaching approaches, Kumaravadivelu argues the language-centred and learner-centred approaches aim at the teaching product; whereas the learning-centred approach attends to learning process. Considering the relationship between Kumaravadivelu’s three teaching approaches and the two writing pedagogy orientations discussed in Chapters 5 to 7, it may be argued that the ‘writing as a product’ approach aligns itself with the language- centred teaching approach. The teacher who employed this approach was Ladda as she was found to primarily emphasise the accuracy of students’ writing. Alternatively, the ‘writing as a process’ approach is more likely to reflect a combination of the learner- centred and learning-centred approaches. Analyses of the case studies revealed Nittaya, Rewadee and Sak were the teachers who favoured a ‘writing as a process’ approach as they emphasised both form and function when they assessed their students’ writing. They tended to preserve their students’ ideas while correcting the language used in the texts. The students’ learning processes were emphasised through their engagement with writing activities to develop their writing capacity and through feedback strategies to enhance self-editing. The principles of teaching that teachers based their teaching on may have also influenced their practices in writing assessment.

8.3 Writing Assessment Practices of Teachers in Classroom Contexts As noted earlier, teachers in the Thai public university context appear to have great flexibility when conducting instruction and assessment for writing. They also appear to employ a wide range of assessment practices. The major emerging findings in relation to this aspect are listed below:  Teachers employed different types of scoring methods. The two most common types were holistic scoring and analytic scoring methods. Teachers also addressed different writing features in their scoring.

236

 Teachers aimed to maintain reliability and validity of assessment through multiple practices such as the use of rating criteria and their decision making during rating processes. These findings are presented in two separate discussions: the employment of rating criteria, and reliability and validity of writing assessment.

8.3.1 The employment of rating criteria In this study the variability in teachers’ practices is illustrated in relation to their employment of rating criteria as presented in Chapters 4 to 7. Generally, teachers in each university are expected to follow criteria for course score allocation as stated in their course outlines. However, when assessing particular writing assignments in writing courses, the case study findings revealed teachers of the courses are autonomous in selecting rating criteria based on their own preferences. As such, the teacher’s professional training and experiences as well as their perceptions of students’ learning needs may inform their selection of rating criteria (Teasdale & Leung, 2000). As illustrated in Chapter 4, teachers’ responses to the questionnaires indicated they reported employing analytic rating criteria more frequently than other types of criteria. However, the findings from the case studies show that holistic scoring was actually employed by most of the teachers. The discrepancy in the teachers’ collective views and their individual assessment practices in this study may imply that teachers generally perceive analytic rating criteria to be more appropriate and applicable to EFL students, particularly in the Thai public university context. The reason for this may be due to the usefulness of analytic rating criteria as highlighted in Chapter 2. However, as demonstrated in the actual case study practices, holistic scoring was quite extensively employed. This difference may be because contextual factors other than the usefulness of rating criteria influence the selection of scoring methods. The number of students in a class and time constraints may be two prominent influential factors impacting the selection of rating criteria. Due to the growing number of university enrolments, class sizes worldwide, especially in developing countries, are increasing (Carpenter, 2006; Coffin, et al., 2003; Watson Todd, 2006). Similar to EFL classrooms internationally, class sizes for English language subjects in Thai public universities are quite large, typically ranging from 30 to more than 40 students (Watson Todd, 2006). In a writing class where written texts are required to be assessed by

237

teacher raters, the large number of students increases the amount of time teachers spend on marking writing. Teachers may therefore select holistic scoring which allows for quick marking rather than analytic scoring which can provide students with detailed diagnosis of their writing strengths and weaknesses (Weigle, 2002, 2007). This observation relates to a common problem in regard to rating and time constraints mentioned by teachers in this research and in the literature (Coffin, et al., 2003). As discussed in Chapter 4, teachers’ perceptions of what constitutes good writing features can also influence the types and selection of rating criteria they employ. When assessing writing, teachers may bring their internalised and personalised expectations of standards of effective writing into their rating processes (B. Huot, 1990; D. Smith, 2000). Their unarticulated and unconscious biases towards the writing they mark may also drive assessment based on ‘implicit criteria’ (Hamp-Lyons, 1991; B. Huot, 1990; Reid & Kroll, 1995). This issue seems to emerge particularly when teachers employ holistic rating criteria. In the case studies, Ladda, Nittaya and Sak assessed their students’ writing holistically and were found to include contextual factors such as their individual preferences in the process of assessment in combination with writing constructs identified in their rating criteria. According to the three major components of good writing proposed and discussed in Section 4.7: text-based features, reader-based features and writer-based features, the case study findings in relation to teachers’ actual assessment reveal confirmation of these features. Text-based features were found to be common foregrounding features in the rating criteria employed by the four teachers. All four teachers appeared to attend heavily to these features when they assessed the texts. Also, although reader-based and writer-based features were not clearly illustrated in the rating criteria the teachers used, they were addressed in teachers’ decision-making behaviours during the rating processes. The findings in Chapters 5 to 7 revealed that the four teachers assessed components of writing features in their rating processes differently. While Nittaya, Rewadee and Sak reflected consideration of writing process in their assessment, Ladda was much more likely to emphasise her personal impression of the texts. Due to these different practices, the teachers may be identified as two different types of assessors: the assessor as an evaluator, and the assessor as a supportive coach (Weigle, 2007). As stated earlier, the assessor as an evaluator aims primarily to grade the students’ writing;

238

whereas the assessor as a supportive coach focuses on assisting students to proceed in their learning. These two types of assessors seem to perceive the components of good writing features very differently. Revisiting Figure 4.4, the relationship between types of assessors and the components of good writing features can be represented below.

Assessor as an evaluator Assessor as a supportive coach

How teachers interpret texts: How students mediate texts: -creativity/originality -writing process (writing, -stylistic impression drafting and editing)

-dealing with errors (editing) Reader- Writer- based based

Text-based

How writing features construct texts: - rhetorical features (e.g., genre) - lexicogrammatical features (e.g., vocabulary, mechanics)

Figure 8.1 The interrelation of assessor types and features of good writing

The above figure reflects writing features attended to by the two types of assessors. It suggests text-based features are shared features emphasised by both types of assessors. However, the assessors appear to be different in the focus on reader-based and writer-based features. In relation to the assessor as an evaluator, the focus of assessment tends to be reader-based features which stress how teacher raters interpret writing. Alternatively, the assessor as a supportive coach tends to reflect the writer- based features by emphasising how student writers construct the texts. Indeed, the two types of assessors discussed above seem to reflect two different assessment cultures regarding teachers’ roles in: ‘assessment of learning’ and ‘assessment for learning’ (G. Brown, et al., 2009; Inbar-Lourie, 2008). The former reflects psychometric perspectives of assessment and emphasises learning outcomes; whereas the latter aims towards students’ improved learning. The assessor as an evaluator appears to pay greatest attention to measuring students’ writing based on their

239

implicit and personal writing standards. Students are not seen as agents and decision makers in their learning (Leung & Mohan, 2004), with the result that assessors may see themselves as judges whose responsibility is to direct students’ learning towards their preferred outcomes. Their assessment then tends to stress reader-based writing features, focusing on their own interpretation of writing at the expense of consideration for the student’s writing process. In contrast, the ultimate goal of assessment for assessor as a supportive coach seems to be to support the students to improve their writing performances and their learning. An effective method in this regard may be to engage students actively in their own process of learning by emphasising writer-based features and restricting assessors’ own preferences. These different perspectives of writing assessment seem to also influence teachers’ selection of rating criteria. The diversity in teachers’ assessment practices in relation to rating criteria in this study coheres with Davison’s (2004) findings about teacher raters’ different interpretations of shared rating criteria. The author contends that the different interpretations are influenced by “[teacher assessors’] personal background, previous experience, unconscious expectations, internalised and personalised preferences regarding the relative importance of different criteria and ideological orientation” (p. 308). Even though teacher raters in this study employed self- or peer-designed rating criteria, the above factors also appear to influence their interpretations of rating criteria. The employment of rating criteria is likely to be an influential factor with regard how teachers conduct reliable and valid writing assessment.

8.3.2 Reliability and validity of writing assessment The reliability and validity of writing assessment practices may be reflected in various aspects such as the use of rating criteria and scoring procedures. Throughout Chapters 4 to 7 the findings revealed teachers employed various practices to conduct reliable and valid assessment. In this study, with regard to reliability, intra-rater reliability or self-consistency of raters seems to be given greater emphasis than inter-rater reliability or consistency across raters. As reported in Chapter 4, when co-teachers are utilised in the writing course most teachers in their questionnaire response reported strengthening consistency in their assessment through the use of group moderation (i.e., discussing the criteria and assessing writing based on the shared agreement). However, this finding is in conflict

240

with the findings from the case studies which revealed teachers were autonomous in terms of their use of specific rating criteria to assess their students’ writing and the conduct of their rating processes. This discrepancy suggests that in a regular classroom context inter-rater reliability may not be of much concern, and that comparability of students’ scores or quality of writing performances is not the main focus in this form of assessment. As classroom writing assessment serves a formative purpose, assessment is neither objective nor technical; instead, it is embedded in the cycle of teaching and learning (Arkoudis & O’Loughlin, 2004; Haines, 2004; K. Hyland, 2003; Inbar-Lourie, 2008; Moss, 1994; Shepard, 2000, 2002; Teasdale & Leung, 2000). Therefore, the essence of assessment in this context is the students’ learning process rather than their learning outcomes. This assumption is reflected in teachers’ practices of giving feedback on the writing. The teachers in all four case studies were found to provide students with feedback to promote self-editing of their writing although they emphasised different writing features of the texts. This feedback can support both teachers and students to reflect on the writing performance and can lead to formulation of their learning development (McNamara, 2001). Even though it is minimally addressed in the case studies, inter-rater reliability appears to be taken into account in some aspects of the teachers’ practices. Its prominence in different educational contexts appears to vary considerably, as revealed in the preferences of Ladda, Nittaya and Rewadee. For instance, Ladda advocated the students’ writing exam papers were to be assessed by the teachers using agreed criteria; whereas the students’ writing in class was assessed based on each teacher’s personal judgements. Indeed, this practice suggests different perspectives on inter-rater reliability in two distinct assessment situations. While inter-rater reliability is suppressed in assessment undertaken in class, it seems to be greatly emphasised in an ‘end-of-course’ exam (summative examination). However, in Nittaya’s case, her departmental policy posits that students are required to take the same test regardless of different teachers. Also, her personal views support the promotion of the same teaching practices across classes, especially in circumstances where there is a co-teacher. These views suggest comparability of teaching and assessment standards is prominent in this particular context. While the cases of Ladda and Nittaya demonstrate consideration of inter-rater reliability in their practices, this issue is not evident in Rewadee’s practices. Teachers in Rewadee’s context appeared to have full autonomy in their practices as she reported

241

they had complete freedom in the operation of their instruction and assessment. This observation suggests they do not value comparability of their students’ scores or their teaching and assessment practices across classes. Since inter-rater reliability is more related to large-scale assessment contexts, the different degrees of consideration of this issue in the context of classroom assessment are not surprising. As mentioned earlier, the primary purpose of classroom writing assessment is to improve teaching and learning. For this reason, validity seems to be more important than reliability. This claim is supported by Wiliam (2001) who argues the assessment of students’ writing performance through their written production can improve validity and weaken reliability. The promotion of inter-rater reliability in some particular Thai educational contexts may then reflect the strong influence of psychometric perspectives of assessment in Thailand (Prapphal, 2008). The discrepancy in the extent to which inter-rater reliability is considered in different institutional contexts in this study may suggest a transition from a culture of testing to a culture of assessment in Thai university contexts. In situations where teacher assessors are seen as having the authority to make judgements (Wiliam, 2001), the control of reliable and valid assessment appears to be located with the teachers. As discussed above, the issue of inter-rater reliability seems to be taken into account more in the cases of Ladda and Nittaya, the expert teachers, than in the case of Rewadee, the novice teacher. This observation may suggest expert teachers are more influenced by psychometric approaches to assessment while novice teachers tend to rely on alternative methods. Limited support or opportunities for training in relation to assessment may cause teachers to fall back on assessment approaches they are familiar with and, because a psychometric testing culture has long been implemented in the Thai education system, it is inevitable that experienced teachers are more accustomed to its principles and may believe they must exploit them in their classrooms. The different assessment orientations shown in the teachers’ practices also point to the various tensions they may encounter when undertaking assessment in a regular classroom context. Two fundamental tensions found in this study seem to align with two potential tensions noted by McMillan (2000): tensions between traditional and alternative assessment cultures (i.e., assessment of learning and assessment for learning), and tensions between criterion-referencing and norm-referencing. Assessment for learning is now increasingly emphasised in classroom contexts internationally

242

(Davison, 2004; Koh & Luke, 2009; Leung & Rea-Dickins, 2007). However, assessment of learning principles still dominate the Thai context. When engaging in assessment in the classroom context, teachers seem to be conscious of external pressures of accountability. As seen in the cases of Ladda, Nittaya and Rewadee, they may believe they need to maintain the kind of comparability of students’ writing outcomes normally recognised in high-stakes assessment. Thus, teachers may become overly concerned about employing reliability in low-stakes assessment situations. This concern may also have led teachers to aim for norm-referencing in their assessment practices. Even though criterion-referencing is stated as central to more recent approaches to assessment (K. Hyland, 2003), it is evident in the cases of Ladda and Rewadee that norm-referencing was very much taken into account when they assessed their students’ writing. This aspect of assessment is seen as a weak practice in formative assessment conducted by teachers (Black & Wiliam, 1998) because students may become demotivated when their performances are compared with other students and their deficiencies are stressed. Criterion-referencing is suggested as a much more effective way to enhance students’ learning progress, as Gipps (2012) asserts:

Assessment against clear standards, in a low-stakes programme, with constructive feedback and a focus on the individual’s own performance in relation to the standard and to his/her previous performance, rather than comparison with others, is more likely to maintain the engagement of pupils and retain them in the system (pp.135- 136).

In the Thai context, the value in more widespread use of this approach is that it may lead to greater validity of assessment and promote more effective learning. Ladda’s case study seems to provide an example of another weakness in assessment due to the scoring purpose being over-emphasised and the learning purpose being under-emphasised (Black & Wiliam, 1998). Value tended to be placed by Ladda on score comparability between students rather than on the improvement of individual students (Black & Wiliam, 1998). Her scoring decisions appeared to put emphasis on reporting grades to provide evidence of students’ proficiency levels in line with her expectations. The allocated grades appeared more to illustrate her students’ ability as compared to that of other students rather than represent individual students’

243

performance development to show their writing progress. This emphasis on maintaining reliability with reference to accountability can reduce the validity of the assessment. However, the assessment practices of the other case study teachers appeared to be more concerned with validity. Indeed, their assessments more clearly reflected an emphasis on the students’ learning progress. Based on the teachers’ judgements, students were allowed to submit multiple versions of the same task to demonstrate their writing development. The teachers then assessed the same writing task written by the same student repeatedly. In turn, the repeated assessment of the same writing task enabled the teachers to form a ‘solid and broadly-based understanding of a pupil’s attainment’ (p. 104) and may have also contributed to high content and construct validity of assessment (Gipps, 2012). The case study teachers appeared to maintain both content and construct validity in their assessment practices. During their rating processes, all four case study teachers addressed the relationship between students’ writing content, and course objectives and content. They also appeared to make reference to the quality of the students’ writing features as stated in the rating criteria regardless of their implicitness or explicitness. Thus, these practices revealed the teachers’ attempts to maintain both construct and content validity in this context of assessment (Anderson, 2003; Moss, 1992). Consequential validity was also found to be ensured through the case study teachers’ feedback strategies. In particular, Nittaya, Rewadee and Sak provided students with feedback to assist them to develop their writing in the sequential versions of the texts, supporting the purpose of students’ improved learning in the classroom context (Gipps, 2012). The high validity displayed in the case study teachers’ assessment practices may therefore reflect positive effects of performance-based assessment in the cycle of teaching and learning (Shepard et al., 1996). The way in which the teachers’ assessment practices appeared to be aimed at maintaining self-consistency suggests they were aware of the need to ensure validity (Moss, 1994). As pointed out earlier, awareness of intra-rater reliability is somewhat striking in this study. The findings from both the collective views of teachers and the individual cases revealed teachers kept in mind the issue of self-consistency in their practices. They appeared to provide the students with guidelines for writing in order to avoid possible biases or misunderstanding, such as a set format for writing assignments in Ladda’s case and the practice of assessing a series of writing assignments at a time as

244

suggested by Rewadee. Because teachers in classroom contexts are the key agents in “turning assessment information and processes into improved learning” (G. Brown, et al., 2009, p. 348), their course syllabuses and rating criteria are important influential factors in their decision making. When they engage in the development of course syllabuses and rating criteria they develop personal understandings of what needs to be assessed and maintain consistency in their own judgement during rating processes (Teasdale & Leung, 2000). Overall, the discussion so far has highlighted the variability that exists in the mediation of assessment reliability and validity in the Thai university classroom contexts reported in this study. However, despite the variation in how reliability is considered, validity appears to be regarded by the participating teachers as the most fundamental concept in assessment (Bachman, 1990; Messick, 1996). The study also provides further confirmation that in contexts where variability in assessment is inevitable, teachers are the most prominent decision makers in the process of assessment (Rea-Dickins, 2004). The discrepancy between reliability and validity also appears to support Black’s (1998) finding that teachers do not understand formative assessment well and they are weak in its practices. The tensions encountered by the teachers in this study also cohere with findings from other studies that teachers have difficulty in making decisions about assessment (Brindley, 2001; McNamara, 2001; Rea-Dickins, 2001). Regarding this point, Rea-Dickins (2004) asserted:

Teachers find themselves at the confluence of different assessment cultures and faced with significant dilemmas in their assessment practices: sometimes torn between their role as facilitator and monitor of language development and that of assessor and judge of language performance as achievement (p. 253).

Considering the overall writing assessment practices of teachers in the Thai public university context, this research also reveals the different orientations and practices of the teachers. To explain further the nature of this variability among the teachers who participated in the study it is useful to apply Davison’s (2004) ‘cline of assessors’ model. Using the adaptation originally presented in Table 4.10 (see Section 4.7), it is evident the teachers as a collective group are located in the middle range of the cline. That is, they included ‘assessor as interpreter of the law’, ‘assessor as principled yet pragmatic professional’ and ‘assessor as arbiter of community values’. However,

245

when each of the four case study teachers is considered independently, the teaching practices appear to be more diverse. The practices included an orientation towards the ‘assessor as God’ category, ‘assessor as principled yet pragmatic professional’ and ‘assessor as arbiter of community values’. However, some references to practice also revealed a tendency towards ‘assessor as interpreter of the law’ and ‘assessor as technician’. The teachers’ writing assessment practices can be framed within the cline as illustrated in Table 8.1 below:

246

Table 8.1 Teachers’ beliefs and practices adapted from Davison (2004, p. 325)

Davison’s Assessor as technician Assessor as interpreter Assessor as principled Assessor as arbiter of Assessor as God orientations of the law yet pragmatic ‘community’ values professional View of the Criterion-bound Criteria-based Criteria-referenced, but Community-referenced Community-bound assessment task localised accommodations View of the Mechanistic, procedural, De-personalised, explicit, Principled, explicit but Personalised, implicit, Personalised, intuitive, assessment process automatic, technical, codified, legalistic, interpretative, attuned to high impressionistic, beyond analysis seemingly universalised culturally detached local cultures/ culturally-bound norms/expectations View of the Text- focused Text- focused, but Text- and student- Student focused Student-focused assessment awareness of student focused product View of Seemingly unaffected Inconsistencies a Inconsistencies Inconsistencies a Seemingly unaffected inconsistencies by inconsistencies problem, threat to inevitable, cannot problem, threat to by inconsistencies reliability necessarily be resolved validity, assessor training satisfactorily, teachers needs to be improved need to rely on professional judgment View of Need better assessment Need better assessor, Need more time for Need better assessors System not open to assessor needs criteria training (in interpreting moderation and (to uphold standards) scrutiny, not criteria) professional dialogue accountable, (to make basis of operated by the judgments more “chosen” explicit) Trend of teachers of writing in Thai public universities Both objectivity and subjectivity are included in teachers’ assessment.

Teachers’ actual writing assessment practices

247

As Table 8.1 shows, the case study teachers’ writing assessment practices disperse into different orientations as the teachers do not necessarily exhibit practices related to any single orientation. Rather, their practices appear to be mixed across the two ends of the cline. A teacher’s assessment practices may be ‘text-focused’ when attending to the written product, but may also be intuitive through its use of implicit and personalised criteria. This teacher may also regard inconsistency in assessment practices as problematic as it may be perceived to hinder reliability, as appears to be the case in Ladda’s assessment practices. Although in her rating processes she focused heavily on text accuracy she nonetheless assessed it impressionistically based on her implicit criteria. Her practices therefore appear to suggest underlying principles situated at both ends of the cline. The practices employed by most of the case study teachers suggest a tendency towards the ‘assessor as God’ end of the cline. For example, Nittaya is ‘student focused’ in her consideration of student needs in the rating processes. Rewadee employed explicit criteria but her practices reveal norm-referencing was also taken into account. The data show the four teachers largely base their assessment on their personal judgements as can be clearly seen in the cases of Ladda, Nittaya and Sak. The diversity in practices highlights the complex nature of writing assessment undertaken by human raters who have different backgrounds, experiences and skills in teaching generally and teaching writing in particular. Their practices become driven by their perceptions in relation to what they believe about writing assessment. The relationship between these two constructs is discussed in the next section.

8.4 Teacher Perceptions and Writing Assessment Practices The interrelationship between teacher perceptions and writing assessment practices can be primarily drawn out from the four individual teachers’ reported views and actual practices in relation to writing assessment. The major findings with regard to this issue reveal themselves as:  Teachers’ rating processes were influenced by various factors: their personal thinking, prior/immediate education and professional experiences as well as contextual factors such as the course management.  Teachers’ reported views appeared in general to be consistent with their actual assessment practices.

248

It is widely accepted that teachers normally operationalise their work in the classroom based on their thinking (Sato, Coffey, & Moorthy, 2005). Their classroom practices are primarily shaped by their belief systems which Richards (1998) describes as “the information, attitudes, values, expectations, theories, and assumptions about teaching and learning that teachers build up over time and bring with them to the classroom” (p. 66). These belief systems also impact the assessment practices of teachers in a classroom context (Inbar‐Lourie & Donitsa‐Schmidt, 2009) as they reflect teachers’ ‘implicit constructs’ or the internal standards of quality teachers rely on when assessing students’ writing (Rea-Dickins, 2004; Teasdale & Leung, 2000; M. Yin, 2010). In this research, the findings show the case study teachers’ actual assessment practices appear to be largely influenced by personal belief systems which vary among and within the different teachers. One influential factor relates to their overall pedagogical orientations. Ladda’s teaching was oriented towards a ‘writing as a product’ approach, which seemed to be greatly influenced by her perceptions of the students’ deficiencies in their writing ability and learning skills. Moreover, these perceptions appeared to have emerged from past experiences with former students. She developed a tendency to form ‘stereotypes’ to illustrate her understandings and general preconceptions of what students would be like (M. Yin, 2010). On the basis of these stereotypes, she tended to consider students as passive learners and relied on her personal preferences and opinions of students when assessing writing. In contrast, the other three teachers adopted a ‘writing as a process’ orientation and appeared to emphasise writer-based writing features to enhance student learning. To some extent, these teachers also tended to conserve students’ ideas in writing by not attempting to change them according to their preferences. The diversity among the teachers in this study in relation to the interrelationship between teacher thinking and pedagogy orientations is similar to Cumming’s (2001) study which found a strong relationship between teachers’ views of writing pedagogical orientations and their assessment practices. He found teachers vary in their assessment methods due to their different orientations to writing instruction. Teachers who based their teaching on ‘English for specific purposes’ orientations utilised a narrower range of assessment methods and assessment criteria than those who aligned with to ‘English for general purposes’ orientations. This distinction lies in teachers’ focus on their assessment. The teachers with specific purpose orientations emphasised the form of the written texts rather than students’ writing development due to their rationale for how to

249

organise a curriculum. In contrast, the focus of teachers with general purpose orientations was on individual students and their personal development. They then included a wider range of assessment methods in their contexts. Other common factors influencing teacher perceptions shown in this research are teachers’ experiences, both in terms of their education and professional backgrounds. In relation to their education, their prior or immediate learning experiences appear to shape their decision making in relation to writing instruction and assessment. Indeed, references to teachers’ professional experiences, particularly training in assessing writing, seem to have an impact on how they conduct their writing instruction and assessment. This impact is reflected in the cases of Ladda, Nittaya and Rewadee. Ladda and Nittaya, the two expert teachers, developed their expertise over time through many years of engagement in writing courses and their training in writing assessment. However, the degree of interaction between their professional experiences and assessment practices is diverse. While Ladda’s practices were largely shaped by her long professional experiences, Nittaya’s conceptions of writing assessment appeared to have partly developed from her immediate postgraduate education. She seemed to integrate her formal learning experiences in relation to writing assessment into her actions, particularly in relation to the employment of rating criteria. An education factor also seems to be reflected in Rewadee’s practices. The professional training Rewadee received during her prior postgraduate education and her experiences as a learner clearly influenced her actual assessment practices. Indeed, the interactive combination of prior or immediate education and professional experiences is known to inform teachers’ theoretical frameworks and influence their practices in their work (Breen, et al., 2001). Both prior education and professional experiences appear to be prominent influential factors in the context of this study. As can be seen in the case of Sak, the novice teacher, because he had neither prior education in writing assessment nor experience in its implementation, there was great uncertainty in regard to his teaching and assessment practices. Furthermore, Sak’s lack of experience seemed to result in numerous dilemmas in his decision making during the rating processes. He was then likely to act according to his intuition or what he assumed to be an effective approach to instruction and assessment. Indeed, Sak’s case highlights the kind of difficulties novice teachers may encounter when operationalising their work and when attempting to perform effectively. For example, Richards and Pennington (1998) found that novice teachers in their study struggled to improve their classroom routines and to manage their

250

complex teaching context so that it was difficult to achieve consistency in their approach to teaching. A writing teacher’s lack of experience, as in Sak’s case, is equally likely to have a similar impact on how they manage their assessment practices. Internal and external contextual factors concerning the classroom context also play a crucial role in the teacher’s writing assessment practices. In relation to internal factors, the course environment including course management and the teacher’s understanding of course objectives and content appear to drive their decision making about assessment practices. As seen in Rewadee’s case, the complete freedom given to teachers of the writing course allowed her to design rating criteria and writing tests based on her own personal theoretical framework. The teachers also took internal contextual factors into consideration in their rating processes. This aspect is clearly seen in Nittaya’s case when she considered the content of the curriculum implemented at her university as a construct when constituting her scoring decision for one of her student’s writing. With regard to external factors, the teachers considered the impact of assessment results on the whole group of students by attempting to maintain assessment standards. This practice seemed to be influenced by the pressure the teachers felt to reach the external standards for students’ writing ability set down by the university. As shown in Nittaya’s case, her view of the need for comparability of students’ writing scores and teaching and assessment practices across different classes appeared to be influenced by departmental policy promoting the use of the same test to ensure reliability in practices among different classes. These factors appear to reflect external variables on assessment practices proposed in the literature. Currently, one dominant factor that greatly influences teachers’ instruction and assessment practices is the pressure put on policy makers, school authorities and teachers to reach performance standards set by external tests (Black, 2003 cited in Inbar‐Lourie & Donitsa‐Schmidt, 2009). Pressures from parents and expectations of students’ evaluation are also identified as influential factors in teachers’ assessment practices (Shepard & Bliem, 1995). However, in the context of this study which aims at low-stake formative classroom assessment, external factors may have only a minimal influence on teaching practices. When the overall interrelationship between teacher perceptions and writing assessment practices in the Thai public university context are considered, the process of determining the most significant factors of influence in teachers’ writing assessment can

251

be informed by an adaptation of Borg’s (2006) framework of elements and processes in language teacher cognition as shown in Figure 8.2 below.

Prior/immediate education Professional training

Teacher perceptions

Writing assessment practices

Contextual factors (e.g., course content, course management)

Figure 8.2 Interrelation of teacher perceptions and writing assessment practices

As shown in the figure, the concept of writing assessment practice is situated with the contextual factors with regard to the classroom. The relationship between writing assessment practice and contextual variables reflect the integrality of assessment to the process of teaching and learning. Writing assessment practices are linked to teacher perceptions formulated through their experiences as learners and their professional experiences. The interrelation of these constructs appears in some ways to support Sato et al.’s (2005) description of the professional development of teachers. The authors point out in their study that teachers’ professional development is influenced by their beliefs, and personal and professional experiences. However, they contend the classroom actions of teachers greatly inform their development. Similarly, in this study, the education experiences of the teachers and their professional experiences are identified as two predominant variables to impact the teacher’s assessment practices. Contextual factors mediate teaching practices so that they are congruent with the teacher’s perceptions (Borg, 2006). The case study findings indicated the teachers acted in their classrooms according to their perceptions, suggesting a high degree of consistency between their reported views and their actual assessment practices. One factor influencing this consistency was most likely the high level of autonomy afforded the teachers by their

252

respective institutions. This autonomy may have helped to support and maintain variability among the teachers across the different universities. As such, this also draws attention to the difficulty of comparing standards across universities as part of the quality assurance in the Thai university context. However, the high level of autonomy may also be seen in a positive light in that teachers are allowed to perform according to their perceptions, leading to consistency in their reported actions and actual assessment practices. As revealed in the case studies, Ladda, Nittaya and Rewadee’s actual writing assessment practices greatly reflected their reported views in a number of aspects such as overall patterns of their rating processes and feedback strategies. Sak appeared to be the only teacher who exhibited discrepancies in his reported and actual practices, particularly with regard to his emphasis on students’ writing. These discrepancies may be due to his lack of experience and the lack of support he received in relation to writing assessment processes as discussed earlier. Indeed, as already noted, professional experience appears to be a highly influential factor in regard to a teacher’s assessment practice. Overall, this research shows a strong relationship between teacher perceptions and writing assessment practices. Teachers’ writing assessment practices were found to be formed and framed by “their own beliefs, attitudes, knowledge and experience” (P. McKay & Brindley, 2007, p. 76). Therefore, because different teachers drawn on their own unique experiences and think in their own unique way it is inevitable that there is variability in their teaching and assessment practices.

8.5 Summary This chapter has discussed the findings pertaining to the writing assessment practices of teachers in general classroom contexts. The findings emerged from this study focus on the collective groups of teachers and four individual teachers of writing in Thai public universities. Particular focus was given to the views and practices of the case study teachers. This chapter also looked to situate the findings in the broader context of writing assessment and the area of teacher cognition. The next chapter concludes the thesis by discussing its implications for the Thai context and to some extent other similar EFL contexts in the Asian region. It also includes contributions, limitations and recommendations for further research.

253

CHAPTER 9 CONCLUSION

9.1 Introduction This thesis has explored the writing assessment practices employed by teachers of writing for English major students in Thai public universities. The study was conducted in two sequential phases. Phase 1 involved the administration of a questionnaire to gather data on the writing assessment practices of teachers of writing at Thai public universities. Phase 2 consisted of four case studies focusing on individual teachers in four distinctive universities located in different regions of Thailand. Multiple research methods including semi-structured interviews, think-aloud protocols, and stimulated recall interviews were used to investigate each teacher’s perceptions of writing instruction and assessment in their particular context, as well as their actual writing assessment practices. The key findings to emerge from the two phases of this study were discussed in Chapter 8. This chapter discusses the implications of these findings for the improvement of writing courses in Thai universities and higher education teacher training in general. As such, it will first provide a summary of the major findings from the overall study before discussing their implications. The limitations of this study and recommendations for further research will also be provided.

9.2 Summary of Major Findings This research has revealed a number of key findings regarding writing instruction and assessment in Thai public universities. The main findings to emerge from the questionnaire and the four case studies emphasise, in line with previous research, the importance of teachers as key agents and decision makers in the conduct of teaching and the assessment of writing within regular classroom contexts. When assigned to teach a writing course, the teachers were seen to play a major role at every stage of their practice from planning the lessons to evaluating their students’ writing tasks. In addition, teacher autonomy is revealed as an influential factor in relation to the way teachers organise their writing classes. As evident in the four case studies, despite sharing some classroom practices such as course syllabuses with co-teachers, the individual teachers had considerable room to adjust their instruction and assessment practices based on their personal thinking and preferences. As such, when assessing the students’ writing

255

assignments the teachers could employ their preferred rating criteria without having to consider whether their practices were consistent with the practices of their co-teachers. This level of autonomy appears to account to a large extent for the variability in their practices. The case study teachers were also found to employ a wide range of teaching and assessment practices ranging from the more traditional to the more current approaches. This variability in practice appears to be influenced largely by each teacher’s (and to some extent intuitive) perceptions of their work. Other influential factors were the teacher’s prior or immediate education, her/his professional experience in relation to writing assessment, and contextual factors surrounding the writing courses. Among these other factors of influence, the first two tended to be predominant in the teacher’s respective Thai university context. Indeed, they were found to mediate the way the teachers constructed their personal theories of how to frame and operationalise their work. These findings have various implications for both writing instruction and assessment in the relevant contexts of this study.

9.3 Implications for Writing Instruction and Assessment The implications to emerge from this research are relevant to four key areas: writing instruction and assessment in Thailand, pre-service teacher education, writing instruction and assessment of in-service teachers and writing instruction and assessment in other EFL contexts.

9.3.1 Writing instruction and assessment in Thailand The overall findings of the study offer implications to writing instruction and assessment inthe specific context of Thai higher education. As discussed earlier, writing courses in Thai universities appear to be independently organised and implemented, contrasting with the ultimate goal of promoting a common English curriculum across all public and private universities in Thailand (Foley, 2005). To achieve this ultimate goal, therefore, Thai universities may have an urgent need to critically think about the way in which writing courses are implemented, particularly in the area of policy planning for such courses. One aspect that needs to be considered is the allocation of writing teachers. As suggested in the findings of this study, universities or departments appear toview writing

256

courses simplistically, so that any teacher regardless of their expertise or experiences can be seen to be appropriate to conduct writing teaching and assessment. This current scenario appears to contrast with theoretical concepts about the complexities of writing instruction and assessment and the need for experienced writing teachers, as commented upon by many scholars (e.g., K. Hyland, 2003; Weigle, 2002). University departments, thus, should be aware of these complexities and assign well-prepared or qualified teachers to teach writing course. There is also a need to consider the the issue of ongoing teacher professional development for writing teachers, and particularly the importance of in-service teacher education. This issue is further discussed in Section 9.3.3 below. Another area of interest to Thai universities is the key principles underlying teachers’ practices. The findings from this study reveal a wide range of teacher views and practices on assessment principles. Ladda appeared to fit within a more traditional testing culture. Other teachers in the study tended to employ more current principles of assessment.. As discussed earlier, this teacher variability may partly result from the long predominance of a testing culture in the Thai university context. While assessment for learning is becoming a current trend of classroom assessment worldwide, it seems to be under-promoted in the Thai context. Thai universities could put a greater emphasis on promoting assessment for learning in classroom instruction and assessment, especially in the area of student self-assessment. To promote assessment for learning, the universities could raise teachers’ awareness of classroom formative assessment and encourage them to implement formative assessment in class. One obvious example could be the adaptation of overall course scores to promote activities related to classroom continuous assessment and reduce summative assessment represented by tests at the end of the unit or course. As one aim of the Thai university system is to become standardised and internationalised, universities should encourage cooperation among different universities to upgrade their performance in all pedagogical aspects, including classroom instruction and assessment. Taking writing courses as an example, the university may encourage writing teachers from different universities to engage in dialogue about the way in which they conduct their instruction and assessment. By doing this, writing teachers could gain shared knowledge and new practices in how to conduct effective writing instruction and assessment. This shared knowledge could lead to the development of teacher confidence in their teaching and assessment practices. Central rating criteria or central writing curriculum could also be developed.

257

These implications suggest that course management should be reconsidered and improved at the university level. Also, at the course level writing teachers may need to be engaged in professional development as discussed below.

9.3.2 Pre-service teacher education Taken together, the findings from this present study suggest a lack of adequate preparation for teachers in relation to classroom writing assessment. This problem may result not only from insufficient preparation provided by teachers’ workplaces but may also be caused by the inadequate knowledge teachers have gained before starting their teaching career. As seen in the case of Sak, a novice teacher, he appeared to conduct writing instruction and assessment without any firm principles of writing pedagogy and assessment, suggesting his lack of prior knowledge about writing instruction and assessment. This finding implies the importance of pre-service teacher education in preparing qualified and confident language teachers who have the skills to teach writing. However, there is evidence to suggest teacher training with regard to assessment practices is still somewhat deficient (DelleBovi, 2012; DeLuca & Klinger, 2010). It is generally the case that knowledge of writing assessment is not required in postgraduate programs in writing and teaching English in general. As a result, teachers are not equipped with a deep understanding of assessment issues (Weigle, 2007). Recently graduated novice teachers are likely to lack the knowledge required to employ appropriate and effective assessment practices in their classrooms. They then have to work their way to seek appropriate assessment approaches through their experiences. This circumstance appears to be reflected in the context of this study as discussed in earlier chapters. As revealed in Rewadee’s case study, teaching and assessment practices were, however, influenced to some extent by the experiences and knowledge the teacher gained through their education. Therefore, it is important to raise pre-service teachers’ awareness of appropriate and effective methods for teaching and assessing writing so as to enhance their understandings of aspects related to their future work and to activate positive behaviours in their classroom practices (Liou, 2001). Teachers should be equipped not only with knowledge about the subject matter, but also the pedagogical knowledge to teach it (J. C. Richards & Renandya, 2002). In terms of assessment, because they are raters of students’ texts in the writing course they should also be

258

equipped with knowledge of how to conduct appropriate and effective writing assessment. As suggested earlier, the present study’s findings reveal teachers’ lack of explicit knowledge of writing instruction and assessment before they come into actual teaching. To prepare pre-service teachers to become effective writing teachers and assessors, various aspects of writing teaching and learning as well as assessment should be addressed in pre-service teacher education. As can be seen from the findings in this study, the writing pedagogy and assessment approaches employed by the four teachers ranged from those based on more traditional methods to others more contemporary in their style. Thus, it appears to be important that the education of pre-service teachers is built upon contemporary knowledge and the current paradigms pertaining to writing instruction and assessment. In particular, a number of more specific aspects that should be highlighted in pre-service teacher education are as follows:

 A wide range of different writing instruction approaches should be explicitly taught. Their strengths and weaknesses should be pointed out in order that teachers can select teaching methods most appropriate to their students.  Different text types of writing should be clearly stated and writing conventions corresponding to each text type need to be explicitly identified.  Writing assessment approaches ranging from traditional methods to others that are more contemporary should be emphasised. A wide range of assessment alternatives such as self- or peer-assessment, as well as knowledge of how to appropriately match assessment tools with assessments designed for either formative or summative purposes (Birenbaum & Dochy, 1996; Dunn, Morgan, O'Reilly, & Parry, 2004; Inbar‐Lourie & Donitsa‐Schmidt, 2009) should also be highlighted.  As classroom writing tasks are usually marked by teacher raters, a very important aspect that needs to be included in pre-service teacher education is the way in which these particular writing tasks can be assessed. Pre-service teachers should be trained to be aware of different rating scales employed in the rating process.

259

 Apart from gaining theoretical knowledge , pre-service teachers should also be equipped with practical knowledge through training in rating actual writing. Because considerable cognitive processing is required in the rating processes, this form of training could help pre-service teachers familiarise themselves and/or anticipate how they can conduct the rating processes in their actual teaching. Direct teaching of the abovementioned aspects should be beneficial for pre- service teachers to enhance their competency and confidence in teaching and assessment. It should also facilitate them to become aware of the complexities of writing instruction and assessment in the classroom context and lead to development of their writing instruction and assessment literacy. The implementation of this form of teacher education may enable pre-service teachers to conduct assessment in their future classrooms with confidence. It may also help them to resolve the dilemma of trying to balance traditional and modern teaching methodologies which teachers sometimes encounter in some East Asia countries such as Thailand (Ho, 2002).

9.3.3 Writing instruction and assessment of in-service teachers Writing is widely accepted as a complex activity to teach and assess, and as such, it needs to be taught by skilful teachers (K. Hyland, 2003). As discussed in Chapter 8 however along with references to actual situations in this study, English writing courses in Thailand are often taught by teachers regardless of their expertise or professional experience. This practice appears to significantly impact the instruction and assessment practices conducted by the teachers, especially teachers with little professional experience. Indeed, the lack of support and professional development in writing instruction and assessment may cause novice teachers to experience feelings of anxiety as a result of their writing instruction and assessment conduct. As shown in this study, none of the universities included in the case studies provided the teacher with training or support prior to or during the writing course. Each teacher, therefore, had to rely on her/his personal perceptions of the work to be done. To reduce teacher anxiety and to build confidence, departmental support in terms of clear and detailed guidelines for teaching and assessment may be one possible option. These guidelines may increase internal consistency for individual teachers as well as external consistency across teachers of the same course. Alternatively, professional

260

development in the form of training in relevant areas such as effective writing pedagogy and the employment of assessment criteria should also be conducted. In the Thai university context, it may be inevitable that novice teachers are required to teach and assess writing. Therefore, there needs to be strategies in place to assist them to become more experienced and to understand local course requirements. At the course level, moderation seems to be an effective way to increase consistency in practice (Gipps, 2012) and is particularly essential for assessment processes. Gipps (2012) contends that moderation is predominant in teacher or school based assessment and can improve inter-rater reliability as well as reinforce assessment processes. Teachers’ shared understandings of the context in which their assessment practices are situated can “warrant assessment-based conclusions” (p. 117) without aiming at high reliability. The processes of moderation range from ‘hard’ statistical approaches to ‘soft’ group discussion. While statistical approaches aim to moderate scores without considering other factors such as inconsistency among teachers, group discussions aim to support a shared understanding of assessment processes among teachers. In a regular classroom context, group moderation is likely to be more appropriate than a statistical approach because it involves teachers who are the major assessors in class. However, group moderation appears to focus on improving particular aspects of teaching practice one stage at a time. Equipping teachers with a solid understanding of teaching and assessment practices in writing pedagogy may provide longer-term improvement. In relation to assessment, Weigle (2007) remarks:

A solid understanding of assessment issues should be part of every teacher’s knowledge base, and teachers should be encouraged to equip themselves with this knowledge as part of their ongoing professional development (p. 207)

Continuous professional development appears to be crucial to enable teachers to conduct their writing instruction and assessment sufficiently. As suggested by this study, a teacher’s classroom practice is greatly influenced by her/his own set of assumptions about their work. This is what Freeman (2002) calls “the hidden side of teaching” (p. 1). Therefore, understanding these mental processes in teachers is important to improving the way in which their professional development is supported. This claim is reflected in Chinda’s (2009) study about professional development in language testing and assessment in a Thai university. He found two important factors contributed to the success of training: the coherence of the teacher’s practices and perceptions, and the

261

acknowledgment of teacher needs in their particular context. The latter factor seems to be essential in the context of this study as it was also reflected in Ladda’s comments about her past training experiences. She viewed her previous training in a negative way, deeming it to be inapplicable to Thai students. This negative attitude seems to suggest the importance of mutual contexts of training and actual writing instruction. Chinda also provides suggestions for effective training. He contends training formats should include hands-on experiences through on-going workshops for teachers. Training should cater to small groups of teachers with different backgrounds and experiences, and should be conducted informally to ensure teachers feel comfortable to share their experiences and opinions. Based on his experience in providing a professional development workshop on assessment to the Thai teachers in his study, Chinda summarises the characteristics of professional development training as follows:  Teacher-centred: teachers are key persons in deciding the direction of the [professional development] activities  Discussion-oriented: the activities encourage the participants to share their ideas and experiences; the leader plays a minimal role in the discussions  Empirical-based: the participants have opportunities to have hands-on experiences  Indigenous: the activities and discussions are based on the local needs with minimal intervention of external materials  On-going: [professional development] is done as a series of workshops or on- going in-service training (p. 255)

This form of training suggested by Chinda aims to be context-based as it is applicable to the professional development of in-service teachers in their immediate teaching context. In the context of this study, therefore, teacher training as a form of professional development appears to be crucial to improve the quality of teacher assessment. In Thailand’s higher education sector, quality assurance is implemented to ensure improved educational quality and standards (Prapphal, 2008) and aims to support the internationalisation (Abbas & McLean, 2007) of the Thai education system. To achieve this aim, comparability of standards in instruction and assessment across universities in the higher education sector is required. Moreover, more effective teacher training is needed to improve comparability between teachers’ instruction and assessment practices. As revealed in the findings of this study, teacher variability in the instruction and

262

assessment of writing was substantially the result of their different personal and professional backgrounds. For teacher training to be sufficient, teachers from different universities may need to be recruited to allow for comparability in their practices. Following Chinda’s (2009) guidelines, teachers with different personal and professional experiences can share and learn from each other. This scenario is also advocated by Lee (2010) who asserts that a professional development activity can offer teachers an opportunity to develop principles of teaching, to balance them with practices, and to share ideas and visions with their colleagues, leading to meaningful changes in the classroom practices of teachers. Effective writing instruction and sufficient teacher education in this area can facilitate improved student learning (I. Lee, 2010). In addition to the usual priority in Thailand’s teacher training programs that focus on improving teaching methods and materials (D. Hayes, 2010), training in writing assessment also needs to be covered. Based on the findings of this present study, teacher training should include specific aspects of writing assessment such as the following:

 The implementation of different types of rating criteria and their impact on students’ learning: This issue could assist teachers regardless of their background and experiences to understand and become more aware of the nature of each type of rating criteria and its influences on students’ learning outcome. Teachers should then be able to select appropriate rating criteria to ultimately promote their students’ learning.  The way in which teachers can rate students’ writing in particular text types and their corresponding writing features: Teachers should understand writing conventions specific to different writing text types and be able to identify them in their actual process of rating.  The practice of rating students’ writing: This issue should be included in training on writing assessment as a fundamental aspect. It could provide teachers with an opportunity to discuss their problems and the obstacles they encounter during their actual rating processes, and offer them solutions for overcoming those problems and obstacles. The training should also provide time for teachers to practise rating students’ writing. This practice should guide teachers to stipulate a preferred way to

263

conduct rating processes that maintain reliability and validity of assessment.

As suggested earlier, training on writing assessment could be held at almost every level (i.e., a national level, an institutional level, a departmental level and a course level). However, at the course level group moderation may only be possible when there is more than one teacher in the course. When teachers are the sole teacher of the course as revealed in the findings of the present study, training on writing assessment, especially on how to conduct the actual rating processes, could still be offered by recruiting teachers teaching different courses that have a writing component. This scenario could provide teachers with an opportunity to share their assessment practices and formulate their own ways of assessing students’ writing with confidence. It may also help to decrease the diversity of individual teachers’ practices as revealed in Table 8.1 and form the middle range of the Davison cline which is regarded as a reasonable level of teachers’ assessment practice for classroom-referenced assessment (Davison, 2004, p. 326).

9.3.4 Writing instruction and assessment in other East Asian EFL contexts The findings from this research pertain to a particular group of teachers in Thai public universities. However, it may be argued that the setting of this study has features in common with other EFL contexts throughout East Asian (i.e., countries in Southeast Asia and Northeast Asia (Ho, 2004a). For instance, in a study of the education systems in 16 East Asian countries, Ho (2004a) found that large classes ranging from 30 to 60 students are common in primary and secondary schools, and that many countries have a shortage of qualified teaching staff. These constraints are likely to lead to less proficient English learners in terms of communicative ability – which is the primary focus of contemporary English language education. Many East Asian countries have implemented major educational reforms in the last two decades. While teacher-based assessment has become more common in the education system in some East Asian countries such as Hong Kong and Singapore (G. Brown, et al., 2009; Koh & Luke, 2009; Luke, Freebody, Shun, & Gopinathan, 2005), traditional teaching conditions still remain in other countries such as Cambodia, Indonesia, Lao People’s Democratic Republic and Mongolia (Ho, 2002; Hu, 2005; Keiichi, 2008; Namsrai, 2004; Nur, 2004; You, 2004). The conventional approach to

264

instruction inevitably embeds traditional assessment approaches, leading to the dominance of the principles of ‘assessment of learning’ in these education systems. This phenomenon is also reflected in the education system in Thailand. Therefore, teachers in many of the regional countries probably experience the same constraints as teachers in Thailand. In these regional countries, professional development is widely seen as a crucial aspect in the development of their education systems. For example, Pit and Roth (2004) note that in Cambodia the majority of in-service teachers are untrained or inadequately trained; therefore, the Ministry of Education, Youth and Sports has implemented many teacher training programs/projects to train 10,000 new teachers at all educational levels and to improve both pre-service and in-service teachers (Hattori, 2009; Tan, 2007). The implementation of teacher training may also be required in other countries in Southeast Asia to improve both the quality of the teacher recruit and the quality of their instruction (Goh, 2012; Ho, 2004b). In Lao PDR, widespread teacher training is also regarded as a means to improve the quality of higher education (Keiichi, 2008). However, teacher training in some countries is likely to be accessed only by certain groups of teachers. Hu (2005) reports in China teachers in rural areas are only minimally exposed to recent trends in English language teaching pedagogy. Thus, it is almost unavoidable that they continue to employ traditional teaching and assessment approaches. Even though teacher training and other forms of professional development is now being more widely implemented in the countries in this region, many of the programs tend to focus on teaching English in general (I. Lee, 2010). Training in specific aspects such as writing instruction and assessment is still limited (Cheung, 2011; Hirvela & Belcher, 2007; Weigle, 2007). In general, Hirvela and Belcher (2007) state writing teacher education should emphasise four specific areas of writing instruction: vocabulary/grammar, genre, responding to writing, and writing assessment. The ‘vocabulary/grammar’ and ‘genre’ areas focus on facilitating teacher trainers to train writing teachers in how to assist their students to construct effective writing (Coxhead & Byrd, 2007; K. Hyland, 2007). In contrast, the ‘responding to writing’ and ‘writing assessment’ areas assist with the preparation of teachers to assess students’ writing. Teacher trainees should learn how to provide feedback on students’ texts (Ferris, 2007) and the principles and processes of writing assessment as well as attend courses on how to organise writing instruction and assessment (Weigle, 2007). These areas of teacher education help to prepare writing teachers to play the dual role of

265

facilitator and assessor of writing. Therefore, to promote effective writing instruction and assessment within the overall development of the education sectors in East Asian countries, profession development in relation to the aforementioned aspects needs to be conducted.

9.4 Limitations of the Study Before concluding this chapter it is important to acknowledge the limitations of this research. First, although the questionnaire aimed to provide the platform for strong representation of the cohort of writing teachers for English major students in universities in Thailand, the number of questionnaire responses is small. The small number of respondents may be due to the administrative decision to distribute the questionnaire online. It is possible that some teachers, especially senior lecturers, did not feel comfortable completing an online questionnaire due to their lack of computer skills. A paper-based questionnaire may have recruited more respondents. Despite the low number of questionnaire respondents limiting the generalisability of the conclusions drawn from the questionnaire responses, the responses nevertheless offered adequate evidence and insights into writing instruction and assessment practices in Thai public university contexts. This evidence may then logically be extended in the sequential case study phase of the research. Second, as stated in Chapter 6, Sak’s senior colleague requested he (Sak) participate in the case study in her place and it did appear that he was psychologically impacted by the circumstances of the data collection. During the actual data collection period Sak demonstrated considerable hesitation when participating in the think-aloud protocols, even though he was provided with training in the process. Sak’s hesitations may have been due to his underlying reluctance to participate in the case study. The fact he was asked by his senior and mentor to participate in the study, a man who had more social power in the workplace than he did, very likely meant that Sak felt pressured on to agree. When conducting the rating processes he may have worried his performance would not be regarded positively. His hesitation may have then affected the quality of his think-aloud protocols. As stated in Chapter 3, the case studies in this present study aimed to capture each teacher’s actual assessment practices in their natural settings. Even though Sak’s hesitation may have led to some degree of incompleteness or reduction in the think-aloud data, it may also illustrate the hidden tensions or dilemmas a teacher may naturally experience in their everyday work situation.

266

Third, the key focus of this study is on teachers’ views and actual practices in the assessment of students’ writing. Thus, the data reported were gathered only from teacher perspectives and address practices that occurred when the teachers attended to their students’ written products. Students’ perspectives, which may have provided very different views of the teachers’ assessment practices were not part of the study. Moreover, classroom observation which could have revealed further relationships between teachers’ actual instruction and assessment practices in the classroom and their beliefs about assessment was not included in this study.

9.5 Recommendations for Further Research The present study emphasises the need for further research in the area of writing assessment and teacher education in Thailand, particularly in the university context. Even though there have been a number of teacher education programs in Thailand (e.g. D. Hayes, 1995; Punthumasen, 2007; Zeichner, 1999), most of them have focused on teaching English in general for teachers at primary and secondary levels. Training for teachers at a university level is noticeably lacking. Given the shortage of teacher education research in Thailand that focuses on writing assessment in particular, the findings of this research may be used as the basis for further studies in this area. A number of possibilities are outlined below. This study investigated the actual writing assessment practices of four university teachers only. As such, the findings from the four case studies cannot be generalised to the broader university teacher cohort. However, further investigation into this higher education domain on a larger scale involving a larger teacher sample from different universities may provide findings which are more generally applicable. Alternatively, an in-depth study focusing on one particular institutional context could be conducted. Moreover, a context-based study could aim to recruit teachers who co-teach the writing course to provide constructive findings that may lead to the development of lecturers in the same course (cf. Chinda, 2009). In relation to teacher professional development, a longitudinal case study focusing on writing assessment practices could be conducted to investigate individual teachers’ development of assessment skills, or the changes in their practices over time. This investigation may be especially beneficial for novice teachers. Because this study was exploratory in nature, the findings were based primarily on qualitative data. Further investigations into this higher education domain could employ quantitative research methods and statistical analyses to provide comparable

267

findings. This study revealed that teachers varied in many aspects of their instruction and assessment practices. Further research may focus on investigating and comparing the particular aspects which influence teacher practices. For example, the relationship between different types of rating criteria and teacher rating processes could be investigated. The impact of teacher assessment on writing improvement by the students or on their learning processes in general may be another area worthy of further investigation. The case study findings in this study revealed teachers provided feedback on students’ writing using different strategies. It may be interesting to examine how the various strategies affect the student’s writing development. Because this study focuses only on teacher perspectives in their writing assessment primarily in relation to their rating processes, further research may widen the scope of the study by examining teachers’ overall classroom assessment practices. Both teachers’ views and actual assessment practices could be investigated. Doing this, classroom observation could be employed as a key research method. Moreover, students’ perspectives on teacher classroom assessment may be another area of interest. It may be worthwhile investigating whether students’ views on teacher assessment practices cohere with teachers’ perspectives.

9.6 Summary This thesis has shed light on the writing assessment practices of writing teachers in Thai public university contexts. Teachers’ perceptions were explored and their actual assessment practices were investigated. The findings revealed how teachers in regular classroom contexts came to their scoring decisions on students’ writing and how they aimed to maintain valid and reliable assessment in their particular instructional situations. The findings also demonstrated the various factors impacting a teacher’s actual assessment practices. Overall, the findings of this study suggest the current status of writing assessment in regular classroom contexts in Thailand’s higher education sector is simply taken for granted. What appears to be the case is that many teachers of writing are likely to be ‘thrown into the pond’ to learn how to ‘swim’ on their own. As illustrated in the four case studies included in the study, teachers often have to rely greatly on their intuitions, assumptions and experiences to conduct writing instruction and to carry out writing assessment. This situation is one that is in contradiction with what is now widely

268

recognised in the literature: that writing instruction and assessment are complex cognitive activities which require well qualified teachers. Above all, the findings presented in this study reveal the need to improve the quality of both writing instruction and writing assessment in the classroom contexts in Thai universities. Although there is increasing emphasis on teacher training for primary and secondary teachers in East Asian countries, the need to improve the training of teachers for the higher education sectors seems to be afforded less prominence. Because these teachers are key potential actors in driving up education standards in universities through the improvement of student learning, greater attention needs to be paid to their current and future professional development. It is hoped that the findings in this study help to raise awareness of some of the current practices and future needs.

269

References

Abbas, A., & McLean, M. (2007). Qualitative research as a method for making just comparisons of pedagogic quality in higher education: A pilot study. British Journal of Sociology of Education, 28(6), 723-737. doi: 10.2307/30036251 Anderson, L. W. (2003). Classroom assessment: Enhancing the quality of teacher decision making. Mahwah, New Jersey: L. Erlbaum Associates. Arkoudis, S., & O’Loughlin, K. (2004). Tensions between validity and outcomes: Teacher assessment of written work of recently arrived immigrant ESL students. Language Testing, 21(3), 284-304. doi: 10.1191/0265532204lt285oa Ashton, P. T. (1990). Editorial. Journal of Teacher Education, 41(1), 2. doi: 10.1177/002248719004100101 Ashwell, T. (2000). Patterns of teacher response to student writing in a multiple-draft composition classroom: Is content feedback followed by form feedback the best method? Journal of Second Language Writing, 9(3), 227-257. doi: 10.1016/s1060-3743(00)00027-8 Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press. Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford, England: Oxford University Press. Barkaoui, K. (2007). Rating scale impact on EFL essay marking: A mixed-method study. Assessing Writing, 12, 86-107. doi: 10.1016/j/asw.2007.07.001 Barkaoui, K. (2008). Effects of scoring method and rater experience on ESL essay rating processes and outcomes. Doctor of Philosophy, University of Toronto, Toronto. Barkaoui, K. (2010a). Do ESL essay raters' evaluation criteria change with experience? A mixed-methods, cross-sectional study. TESOL Quarterly, 44, 31-57. doi: 10.5054/tq.2010.214047 Barkaoui, K. (2010b). Explaining ESL essay holistic scores: A multilevel modeling approach. Language Testing, 27(4), 515-535. doi: 10.1177/0265532210368717 Barkaoui, K. (2010c). Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly, 7(1), 54-74. doi: 10.1080/15434300903464418 Barnard, R., & Burns, A. (Eds.). (2012). Researching language teacher cognition and practices: International case studies. Bristol: Multilingual Matters. Basturkmen, H., Loewen, S., & Ellis, R. (2004). Teachers' stated beliefs about incidental focus on form and their classroom practices. Applied Linguistics, 25(2), 243-272. doi: 10.1093/applin/25.2.243 Bickman, L., & Rog, D. J. (1998). Handbook of applied research methods. Thousand Oaks, CA: Sage Publications, Inc. Birenbaum, M., & Dochy, F. J. R. C. (1996). Alternatives in assessment of achievements, learning processes, and prior knowledge. Boston: Kluwer Academic Publishers. Björk, L. A. (2003). Text types, textual consciousness and academic writing ability. In G. B. Lennart A. Björk, Lotte Rienecker, Peter Stray Jorgensen (Ed.), Teaching academic writing in European higher education (pp. 29-40). Dordrecht, The Netherlands: Kluwer Academic Publishers. Black, P. (2009). Formative assessment issues across the curriculum: The theory and the practice. TESOL Quarterly, 43, 519-523. Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7 - 74.

271

Black, P., & Wiliam, D. (2003). 'In praise of educational research': Formative assessment. British Educational Research Journal, 29(5), 623-637. doi: 10.2307/1502114 Black, P., & William, D. (1998). Inside the blackbox: Raising standards through classroom assessment. 1-13. Retrieved from http://www.rappsproject.org/RAPPS/Formative_Assessment_files/InsideBlackB ox.pdf Borg, S. (1999a). Studying teacher cognition in second language grammar teaching. System, 27(1), 19-31. doi: Doi: 10.1016/s0346-251x(98)00047-5 Borg, S. (1999b). The use of grammatical terminology in the second language classroom: A quality study of teachers' practices and cognitions. Applied Linguistics, 20(1), 95-124. doi: 10.1093/applin/20.1.95 Borg, S. (2003). Teacher cognition in language teaching: A review of research on what language teachers think, know, believe, and do. Language Teaching, 36(02), 81- 109. doi: doi:10.1017/S0261444803001903 Borg, S. (2005). Experience, knowledge about language and classroom practice in teaching grammar. In N. Bartels (Ed.), Applied linguistics and language teacher education (pp. 325-340). Boston: Kluwer Academic Publishers. Borg, S. (2006). Teacher cognition and language education: Research and practice. London: Continuum. Borg, S. (2011). The impact of in-service teacher education on language teachers’ beliefs. System, 39(3), 370-380. doi: 10.1016/j.system.2011.07.009 Borg, S. (2012). Current approaches to language teacher cognition research: A methodological analysis. In R. Barnards & A. Burns (Eds.), Researching language teacher cognition and practices: International case studies. Bristol: Multilingual Matters. Breen, M., Hird, B., Milton, M., Oliver, R., & Thwaite, A. (2001). Making sense of language teaching: Teachers' principles and classroom practices. Applied Linguistics, 22(4), 470-501. doi: 10.1093/applin/22.4.470 Brennan, J., & Shah, T. (2000). Quality assessment and institutional change: Experiences from 14 countries. Higher Education, 40(3), 331-349. doi: 10.2307/3448038 Brindley, G. (2001). Outcomes-based assessment in practice: Some examples and emerging insights. [Article]. Language Testing, 18(4), 393-407. Brown, G., Kennedy, K., Fok, P. K., Chan, J. K. S., & Yu, W. M. (2009). Assessment for student improvement: Understanding Hong Kong teachers’ conceptions and practices of assessment. Assessment in Education: Principles, Policy & Practice, 16(3), 347-363. doi: 10.1080/09695940903319737 Brown, H. D. (2004). Language assessment: Principles and classroom practices. New York: Longman. Brown, J. D. (1996). Testing in language programs. Upper Saddle River, N.J.: Prentice Hall Regents. Brown, J. D. (2001). Using surveys in language programs. Cambridge: Cambridge University Press. Brown, J. D., & Rodgers, T. S. (2002). Doing second language research. Oxford: Oxford University Press. Burns, A. (1996). Starting all over again: From teaching adults to teaching beginners Teacher learning in language teaching (pp. 154-177). Cambridge; New York: Cambridge University Press.

272

Butler, Y. G. (2009). How do teachers observe and evaluate elementary school students' foreign language performance? A case study from South Korea. TESOL Quarterly, 43, 417-444. Carpenter, J. M. (2006). Effective teaching methods for large classes. Journal of Family and Consumer Sciences Education, 24(2), 13-23. Celce-Murcia, M., & Olshtain, E. (2000). Discourse and context in language teaching. New York: Cambridge University Press. Chanock, k. (2000). Comments on essays: Do students understand what tutors write? Teaching In Higher Education, 5(1), 95-95. Cheng, L., Rogers, T., & Hu, H. (2004). ESL/EFL instructors’ classroom assessment practices: Purposes, methods, and procedures. Language Testing, 21(3), 360-389. doi: 10.1191/0265532204lt288oa Cheung, Y. L. (2011). Teacher training for effective writing instruction: Recent trends and future directions. Procedia - Social and Behavioral Sciences, 15(0), 531-534. doi: http://dx.doi.org/10.1016/j.sbspro.2011.03.136 Chinda, B. (2009). Professional development in language testing and assessment: A case study of supporting change in assessment practice in in-service EFL teachers in Thailand. Ph.D., University of Nottingham, Nottingham. Cho, Y. (2003). Assessing writing: Are we bound by only one method? Assessing Writing, 8(3), 165-191. doi: http://dx.doi.org/10.1016/S1075-2935(03)00018-7 Clarke, S., & Gipps, C. (2000). The role of teachers in teacher assessment in England 1996-1998. Evaluation & Research in Education, 14(1), 38-52. doi: 10.1080/09500790008666960 Coffin, C., Curry, M. J., Goodman, S., Hewings, A., Lillis, T. M., & Swann, J. (2003). Teaching academic writing: A toolkit for higher education. London: Routledge. Cohen, A. D. (1987). Using verbal reports in research on language learning. In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp. 82-95). Clevedon, Philadelphia: Multilingual Matters. Cohen, A. D. (1993). Assessing language ability in the classroom. Boston: Heinle & Heinle. Cohen, A. D. (1994). Assessing language ability in the classroom (2 nd ed.). Boston: Heinle & Heinle. Cohen, L., Manion, L., & Morison, K. (2007). Research methods in education (6th ed.). London; New York: Routledge. Connor-Linton, J. (1995a). Crosscultural comparison of writing standards: American ESL and Japanese EFL. World Englishes, 14(1), 99-115. Connor-Linton, J. (1995b). Looking behind the Curtain: What Do L2 Composition Ratings Really Mean? TESOL Quarterly, 29(4), 762-765. Coxhead, A., & Byrd, P. (2007). Preparing writing teachers to teach the vocabulary and grammar of academic prose. Journal of Second Language Writing, 16(3), 129- 147. doi: http://dx.doi.org/10.1016/j.jslw.2007.07.002 Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks, Calif.: Sage Publications. Cumming, A. (1990). Expertise in evaluating second language compositions. Language Testing, 7(1), 31-51. doi: 10.1177/026553229000700104 Cumming, A. (2001). ESL/EFL instructors' practices for writing assessment: Specific purposes or general purposes? Language Testing, 18(2), 207-224. doi: 10.1177/026553220101800206

273

Cumming, A. (2009). What needs to be developed to facilitate classroom-based assessment? TESOL Quarterly, 43(3), 515-519. doi: 10.1002/j.1545- 7249.2009.tb00247.x Cumming, A., Kantor, R., & Powers, D. E. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. The Modern Language Journal, 86(1), 67-96. Davison, C. (2004). The contradictory culture of teacher-based assessment: ESL teacher assessment practices in Australian and Hong Kong secondary schools. Language Testing, 21(3), 305-334. Davison, C., & Leung, C. (2009). Current Issues in English Language Teacher-Based Assessment. TESOL Quarterly, 43(3), 393-415. DelleBovi, B. M. (2012). Literacy instruction: From assignment to assessment. Assessing Writing, 17(4), 271-292. doi: http://dx.doi.org/10.1016/j.asw.2012.07.001 DeLuca, C., & Klinger, D. A. (2010). Assessment literacy development: Identifying gaps in teacher candidates’ learning. Assessment in Education: Principles, Policy & Practice, 17(4), 419-438. doi: 10.1080/0969594x.2010.516643 DeRemer, M. (1998). Writing assessment: Raters' elaboration of the rating task. Assessing Writing, 5(1), 7-29. Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York; Chichester: John Wiley. Dörnyei, Z. (2003). Questionnaires in second language research: Construction, administration, and processing. Mahwah, N.J.: Lawrence Erlbaum Associates. Dörnyei, Z. (2007). Research methods in applied linguistics: Quantitative, qualitative, and mixed methodologies. Oxford: Oxford University Press. Duff, P. (2008). Case study research in applied linguistics. New York: Lawrence Erlbaum Associates. Dunn, L., Morgan, C., O'Reilly, M., & Parry, S. (2004). The student assessment handbook: New directions in traditional and online assessment. London: RoutledgeFalmer. East, M. (2009). Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing. Assessing Writing, 14, 88-115. doi: 10.1016/j.asw.2009.04.001 Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155-185. doi: 10.1177/0265532207086780 Elder, C., Barkhuizen, G., Knoch, U., & von Randow, J. (2007). Evaluating rater responses to an online training program for L2 writing assessment. Language Testing, 24(1), 37-64. doi: 10.1177/0265532207071511 Erdosy, M. U. (2004). Exploring variability in judging writing ability in a second language: A study of four experienced raters of ESL compositions. Princeton, New Jersey: Educational Testing Service. Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. Cambridge, Massachusette: MIT Press. Ericsson, K. A., & Simon, H. A. (1987). Verbal reports on their thinking. In C. Færch & G. Kasper (Eds.), Introspection in second language research (pp. 24-53). Philadelphia: Multilingual Matters. Ertmer, P. A., Richardson, J. C., Belland, B., Camin, D., Connolly, P., Coulthard, G., . . . Mong, C. (2007). Using peer feedback to enhance the quality of student online

274

postings: An exploratory study. Journal of Computer-Mediated Communication, 12(2), 412-433. doi: 10.1111/j.1083-6101.2007.00331.x Fang, Z. (1996). A review of research on teacher beliefs and practices. Educational Research, 38(1), 47-65. doi: 10.1080/0013188960380104 Farrell, T. S. C., & Bennis, K. (2013). Reflecting on ESL teacher beliefs and classroom practices: A case study. RELC Journal, 44(2), 163-176. doi: 10.1177/0033688213488463 Ferris, D. (2007). Preparing teachers to respond to student writing. Journal of Second Language Writing, 16(3), 165-193. doi: http://dx.doi.org/10.1016/j.jslw.2007.07.003 Ferris, D., & Hedgcock, J. (1998). Teaching ESL composition: Purpose, process, and practice. Mahwah, N.J.: Lawrence Erlbaum Assoicates. Ferris, D., & Hedgcock, J. (2005). Teaching ESL composition : purpose, process, and practice (2nd ed.). Mahwah, N.J.: Lawrence Erlbaum Associates. Ferris, D., & Roberts, B. (2001). Error feedback in L2 writing classes: How explicit does it need to be? Journal of Second Language Writing, 10(3), 161-184. doi: 10.1016/s1060-3743(01)00039-x Feryok, A. (2010). Language teacher cognitions: Complex dynamic systems? System, 38(2), 272-279. doi: 10.1016/j.system.2010.02.001 Flores, B. B. (2001). Bilingual education teachers' beliefs and their relation to self- reported practices. Bilingual Research Journal, 25(3), 275-299. doi: 10.1080/15235882.2001.10162795 Foley, J. A. (2005). English In…Thailand. RELC Journal, 36(2), 223-234. doi: 10.1177/0033688205055578 Freedman, S., & Calfee, R. (1983). Holistic assessment of writing: Experimental design and cognitive theory. In P. Mosenthal, L. Tamor & S. Walmsley (Eds.), Research on writing: Principles and methods (pp. 75-98). New York: Longman. Freeman, D. (1992). Languague teacher education, emerging discourse, and change in classroom practice. In S. Hsia, M. N. Brock & J. R. Flowerdew (Eds.), Perspectives on second language teacher education (pp. 1-21). Hong Kong: City Polytechnic. Freeman, D. (2002). The hidden side of the work: Teacher knowledge and learning to teach. A perspective from north American educational research on teacher education in English language teaching. Language Teaching, 35(1), 1-13. doi: doi:10.1017/S0261444801001720 Freeman, D., & Johnson, K. E. (1998). Reconceptualizing the knowledge-base of language teacher education. TESOL Quarterly, 32, 397-417. Freeman, D., & Richards, J. C. (1993). Conceptions of teaching and the education of second language teachers. TESOL Quarterly, 27(2), 193-216. Frodeson, J., & Holten, C. (2003). Grammar and the ESL writing class. In B. Kroll (Ed.), Exploring the dynamics of second language writing (pp. 141-161). Cambridge ; New York: Cambridge University Press. Fulcher, G., & Davidson, F. (2007). Language testing and assessment: An advanced resource book. Abingdon, England: Routledge. Gass, S. M., & Mackey, A. (2000). Stimulated recall methodology in second language research. Mahwah; London: Lawrence Erlbaum. Gass, S. M., & Mackey, A. (2007). Data elicitation for second and foreign language research. Mahwah, N.J.: Lawrence Erlbaum Associates. Gatbonton, E. (1999). Investigating experienced ESL teachers’ pedagogical knowledge. The Modern Language Journal, 83(1), 35-50. doi: 10.1111/0026-7902.00004

275

Gipps, C. (2008). Sociocultural perspectives on assessment. In G. Wells & G. Claxton (Eds.), Learning for life in the 21st century: Sociocultural perspectives on the future of education (1 ed., pp. 73-83). Hoboken: Wiley-Blackwell. Retrieved from http://unsw.eblib.com/patron/FullRecord.aspx?p=351246. Gipps, C. (2012). Beyond testing: Towards a theory of educational assessment. London ; Washington, D.C.: Falmer Press. Goh, P. (2012). The Malaysian teacher standards: A look at the challenges and implications for teacher educators. Educational Research for Policy and Practice, 11(2), 73-87. doi: 10.1007/s10671-011-9107-8 Golombek, P. R. (1998). A study of language teachers' personal practical knowledge. TESOL Quarterly, 32(3), 447-464. doi: 10.2307/3588117 Grabe, W., & Kaplan, R. B. (1996). Theory and practice of writing: An applied linguistic perspective. New York: Longman. Graddol, D., Cheshire, J., & Swann, J. (1994). Describing language (2nd ed.). Buckingham ; Philadelphia: Open University Press. Grierson, J. (1995). Classroom-based assessment in intensive English centres. In G. P. Brindley (Ed.), Language assessment in action (pp. iv, 324 p.). Sydney, N.S.W.: National Centre for English Language Teaching and Research, Macquarie University. Guénette, D. (2007). Is feedback pedagogically correct?: Research design issues in studies of feedback on writing. Journal of Second Language Writing, 16(1), 40- 53. doi: 10.1016/j.jslw.2007.01.001 Gunn, H. (n.d.). Web-based surveys: Changing the survey process Retrieved 12 March 2011, 20011, from http://www.accesswave.ca/~hgunn/special/papers/websurv/ Haines, C. (2004). Assessing students' written work: Marking essays and reports. London: RoutledgeFalmer. Hamp-Lyons, L. (1991). Scoring procedures for ESL contexts. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 241-276). New Jersey: Ablex Publidhing Corporation. Hamp-Lyons, L. (1995). Rating nonnative writing: The trouble with holistic scoring. TESOL Quarterly, 29(4), 759-762. Hamp-Lyons, L. (2003). Writing teachers as assessors of writing. In B. Kroll (Ed.), Exploring the dynamics of second language writing (pp. 162-189). Cambridge; New York: Cambridge University Press. Hamp-Lyons, L. (2009). Principles for large-scale classroom-based teacher assessment of English learners' language: An initial framework from school-based assessment in Hong Kong. TESOL Quarterly, 43, 524-529. Harlen, W. (1996). Editorial. Curriculum Journal, 7(2), 129-135. doi: 10.1080/0958517960070201 Hattori, H. (2009). Enhancing aid effectiveness in education through a sector-wide approach in Cambodia. PROSPECTS, 39(2), 185-199. doi: 10.1007/s11125-009- 9121-2 Hawkey, R., & Barker, F. (2004). Developing a common scale for the assessment of writing. Assessing Writing, 9(2), 122-159. doi: DOI: 10.1016/j.asw.2004.06.001 Hayes, D. (1995). In-service teacher development: Some basic principles. ELT Journal, 49(3), 252-261. doi: 10.1093/elt/49.3.252 Hayes, D. (2005). Exploring the lives of non‐native speaking English educators in Sri Lanka. Teachers and Teaching, 11(2), 169-194. doi: 10.1080/13450600500083964

276

Hayes, D. (2010). Language learning, teaching and educational reform in rural Thailand: An English teacher's perspective. Asia Pacific Journal of Education, 30(3), 305- 319. doi: 10.1080/02188791.2010.495843 Hayes, J. R. (1996). A new framework for understanding cognition and affect in writing. In C. M. Levy & S. E. Ransdell (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp. 1-27). Mahwah, N. J.: Lawrence Erlbaum. Hedge, T. (2000). Teaching and learning in the language classroom. Oxford: Oxford University Press. Hirvela, A., & Belcher, D. (2007). Writing scholars as teacher educators: Exploring writing teacher education. Journal of Second Language Writing, 16(3), 125-128. doi: http://dx.doi.org/10.1016/j.jslw.2007.08.001 Ho, W. K. (2002). English language teaching in East Asia today: An overview. Asia Pacific Journal of Education, 22(2), 1-22. doi: 10.1080/0218879020220203 Ho, W. K. (2004a). English language teaching in East Asia today: An overview. In W. K. Ho & R. Wong (Eds.), English language teaching in East Asia today: Changing policies and practices (2nd ed. ed., pp. 1-32). Singapore: Eastern Universities Press Ho, W. K. (2004b). RELC's role in ELT in Southeast Asia (1968 to 2002): Mediating between the centre and the periphery. In W. K. Ho & R. Wong (Eds.), English language teaching in East Asia today: Changing policies and practices (2nd ed. ed., pp. 46-73). Singapore: Eastern Universities Press Hu, G. (2005). Contextual influences on instructional practices: A Chinese case for an ecological approach to ELT. TESOL Quarterly, 39(4), 635-660. doi: 10.2307/3588525 Hughes, A. (2003). Testing for language teachers (2nd ed.). Cambridge; New York: Cambridge University Press. Huot, B. (1990). The literature of direct writing assessment: Major concerns and prevailing trends. Review of Educational Research, 60(2), 237-263. Huot, B. A. (1993). The influence of holistic scoring procedures on reading and rating student essays. In M. M. Williamson & B. A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 206- 236). Cresskill, N.J.: Hampton Press. Hyland, F. (1998). The impact of teacher written feedback on individual writers. Journal of Second Language Writing, 7(3), 255-286. doi: Doi: 10.1016/s1060- 3743(98)90017-0 Hyland, K. (2003). Second language writing. Cambridge, UK ; New York, N.Y.: Cambridge University Press. Hyland, K. (2007). Genre pedagogy: Language, literacy and L2 writing instruction. Journal of Second Language Writing, 16(3), 148-164. doi: http://dx.doi.org/10.1016/j.jslw.2007.07.005 Inbar-Lourie, O. (2008). Constructing a language assessment knowledge base: A focus on language assessment courses. Language Testing, 25(3), 385-402. doi: 10.1177/0265532208090158 Inbar‐Lourie, O., & Donitsa‐Schmidt, S. (2009). Exploring classroom assessment practices: The case of teachers of English as a foreign language. Assessment in Education: Principles, Policy & Practice, 16(2), 185-204. doi: 10.1080/09695940903075958

277

Jeffery, F., & Selting, B. (1999). Reading the invisible ink: Assessing the responses of non-composition faculty. Assessing Writing, 6(2), 179-197. doi: 10.1016/s1075- 2935(00)00007-6 John, A. (1990). L1 composition theories: Implications for developing theories of L2 composition. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 24-36). Cambridge: Cambridge University Press. Johnson, B., & Christensen, L. B. (2008). Educational research : quantitative, qualitative, and mixed approaches (3rd ed.). Los Angeles: Sage Publications. Johnson, K. E. (1992). The instructional decisions of pre-service English as a second language teachers: New directions for teacher preparation programs. In S. Hsia, M. N. Brock & J. R. Flowerdew (Eds.), Perspectives on second language teacher education (pp. 115-134). Hong Kong: City Polytechnic. Johnson, K. E. (1994). The emerging beliefs and instructional practices of preservice English as a second language teachers. Teaching and Teacher Education, 10(4), 439-452. doi: Doi: 10.1016/0742-051x(94)90024-8 Johnson, K. E. (1999). Understanding language teaching: Reasoning in action. Boston; London: Heinle & Heinle. Johnson, K. E. (2006). The sociocultural turn and its challenges for second language teacher education. TESOL Quarterly, 40, 235-257. Johnston, B., & Goettsch, K. (2000). In Search of the Knowledge Base of Language Teaching: Explanations by Experienced Teachers. Canadian Modern Language Review/ La Revue canadienne des langues vivantes, 56(3), 437-468. doi: 10.3138/cmlr.56.3.437 Kamberelis, G., & Dimitriadis, G. (2005). On qualitative inquiry: Approaches to language and literacy research. New York: Teachers College Press. Karavas-Doukas, E. (1996). Using attitude scales to investigate teachers' attitudes to the communicative approach. ELT Journal, 50(3), 187-198. doi: 10.1093/elt/50.3.187 Keiichi, O. G. A. W. A. (2008). Higher education in Lao People's Democratic Republic: Historical perspective. Journal of International Cooperation Studies, 16(1), 105- 129. Kennedy, K., Chan, J., Fok, P., & Yu, W. (2008). Forms of assessment and their potential for enhancing learning: Conceptual and cultural issues. Educational Research for Policy and Practice, 7(3), 197-207. doi: 10.1007/s10671-008-9052- 3 Kennedy, K., Sang, J. C. K., Wai-ming, F. Y., & Fok, P. K. (2006). Assessment for productive learning: Forms of assessment and their potential for enhancing learning Paper presented at the the 32nd Annual Conference of the International Association for Educational Assessment, Singapore. Khalifa, H., & Weir, C. J. (2009). Examining reading: Research and practice in assessing second language reading. Cambridge: Cambridge University Press. Kirtikara, K. (2001). Higher education in Thailand and the National Reform Roadmap. Paper presented at the the Thai_US Education Roundtable. http://www.kmutt.ac.th/pi/backup/fileKK/Higher%20Edu%20Reform%20Road map.pdf Knoch, U. (2009). Diagnostic assessment of writing: A comparison of two rating scales. Language Testing, 26(2), 275-304. doi: 10.1177/0265532208101008 Kobayashi, T. (1992). Native and nonnative reactions to ESL compositions. TESOL Quarterly, 26(1), 81-111.

278

Koh, K., & Luke, A. (2009). Authentic and conventional assessment in Singapore schools: An empirical study of teacher assignments and student work. Assessment in Education: Principles, Policy & Practice, 16(3), 291 - 318. Kongpetch, S. (2006). Using a genre-based approach to teach writing to Thai students: A case study. Prospect, 21(2), 3-33. Kurasaki, K. S. (2000). Intercoder reliability for validating conclusions drawn from open-ended interview data. Field Methods, 12(3), 179-194. doi: 10.1177/1525822x0001200301 Lee, H. K. (2004). A comparative study of ESL writers’ performance in a paper-based and a computer-delivered writing test. Assessing Writing, 9(1), 4-26. doi: http://dx.doi.org/10.1016/j.asw.2004.01.001 Lee, I. (2003). L2 writing teachers' perspectives, practices and problems regarding error feedback. Assessing Writing, 8, 216-237. doi: 10.1016/j.asw.2003.08.002 Lee, I. (2010). Writing teacher education and teacher learning: Testimonies of four EFL teachers. Journal of Second Language Writing, 19(3), 143-157. doi: http://dx.doi.org/10.1016/j.jslw.2010.05.001 Leung, C. (2004). Developing formative teacher assessment: Knowledge, practice, and change. Languge Assessment Quarterly, 1(1), 19-41. Leung, C., & Lewkowicz, J. (2006). Expanding horizons and unresolved conundrums: Language testing and assessment. TESOL Quarterly, 40(1), 211-234. Leung, C., & Mohan, B. (2004). Teacher formative assessment and talk in classroom contexts: Assessment as discourse and assessment of discourse. Language Testing, 21(3), 335-359. doi: 10.1191/0265532204lt287oa Leung, C., & Rea-Dickins, P. (2007). Teacher assessment as policy instrument: Contradictions and capacities. Language Assessment Quarterly, 4(1), 6-36. doi: 10.1080/15434300701348318 Liou, H. C. (2001). Reflective practice in a pre-service teacher education program for high school English teachers in Taiwan, ROC. System, 29(2), 197-208. doi: http://dx.doi.org/10.1016/S0346-251X(01)00011-2 Luke, A., Freebody, P., Shun, L., & Gopinathan, S. (2005). Towards research-based innovation and reform: Singapore schooling in transition. Asia Pacific Journal of Education, 25(1), 5-28. doi: 10.1080/02188790500032467 Lukmani, Y. (1996). Linguistic accuracy versus coherence in assesssing examination answers in content subjects. In M. Milanovic & N. Saville (Eds.), Performance testing, cognition and assessment: Selected papers from the 15th Langugae Testing Research Colloquium, Cambridge and Arnhem (pp. 130-150). Cambridge: Cambridge University Press. Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246-276. Lumley, T. (2005). Assessing second language writing: The rater's perspective. Frankfurt: Peter Lang. Lumley, T. (2006). Assessing second language writing : the rater's perspective. Frankfurt am Main ; New York: P. Lang. Lyle, J. (2003). Stimulated recall: A report on its use in naturalistic research. British Educational Research Journal, 29(6), 861-878. Lynch, B. K. (2001). Rethinking assessment from a critical perspective. Language Testing, 18(4), 351-372. Macken, M., & Slade, D. (1993). Assessment: A foundation for effective learning in the school context. In B. Cope & M. Kalantzis (Eds.), The powers of literacy: a genre approach to teaching writing (pp. 203-230). London: The Falmer Press.

279

Marshall, C., & Rossman, G. B. (2006). Designing qualitative research (4th ed.). Thousands Oaks, Calif.: Sage Publications. Mavrommatis, Y. (1997). Understanding assessment in the classroom: Phases of the assessment process — the assessment episode. Assessment in Education: Principles, Policy & Practice, 4(3), 381-400. doi: 10.1080/0969594970040305 McCarthy, M. (1991). Discourse analysis for language teachers. New York: Cambridge University Press. McKay, P., & Brindley, G. (2007). Educational reform and ESL assessment in Australia: New roles and new tensions. Language Assessment Quarterly, 4(1), 69-84. doi: 10.1080/15434300701348383 McKay, S. (2006). Researching second language classrooms. Mahwah, N.J.: Lawrence Erlbaum. McMillan, J. H. (2000). Fundamental assessment principles for teachers and school administrators. Practical Assessment, Research & Evaluation, 7. Retrieved from http://pareonline.net/getvn.asp?v=7&n=8 McNamara, T. (1996). Measuring second language performance. New York: Longman. McNamara, T. (2000). Language testing. Oxford: Oxford University Press. McNamara, T. (2001). Language assessment as social practice: Challenges for research. Language Testing, 18(4), 333-349. doi: 10.1177/026553220101800402 Merriam, S. B. (1998). Qualitative research and case study applications in education (2nd ed.). San Francisco: Jossey-Bass Publishers. Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. San Francisco: Jossey-Bass. Mertens, D. M. (2005). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods. Thousand Oaks, California: Sage Publications. Messick, S. (1995a). Standards of validity and the validity of standards in performance asessment. Educational Measurement: Issues and Practice, 14(4), 5-8. doi: 10.1111/j.1745-3992.1995.tb00881.x Messick, S. (1995b). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. doi: 10.2307/1176219 Messick, S. (1996). Validity and washback in language testing. Language Testing, 13(3), 241-256. doi: 10.1177/026553229601300302 Milanovic, M., Saville, N., & Shuhong, S. (1996). A study of the decision-making behavior of composition markers. In M. Milanovic & N. Saville (Eds.), Performance testing, cognition and assessment: Selected papers from the 15th Langugae Testing Research Colloquium, Cambridge and Arnhem (pp. 92-114). Cambridge: Cambridge University Press. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks: Sage Publications. Miller, M. D., Linn, R. L., & Gronlund, N. E. (2009). Measurement and assessment in teaching (10th ed.). New Jersey: Merrill/Pearson. Min, H.-T. (2013). A case study of an EFL writing teacher's belief and practice about written feedback. System, 41(3), 625-638. doi: http://dx.doi.org/10.1016/j.system.2013.07.018 Mori, R. (2011). Teacher cognition in corrective feedback in Japan. System, 39(4), 451- 467. doi: 10.1016/j.system.2011.10.014

280

Moss, P. A. (1992). Shifting conceptions of validity in educational measurement: Implications for performance assessment. Review of Educational Research, 62(3), 229-258. doi: 10.3102/00346543062003229 Moss, P. A. (1994). Can there be validity without reliability? Educational Researcher, 23(2), 5-12. doi: 10.2307/1176218 Mousavi, S. A. (2002). An encyclopedic dictionary of language testing (3rd ed.): Tung Hua Book Company. Namsrai, M. (2004). English language teaching in Mongolia. In W. K. Ho & R. Wong (Eds.), English language teaching in East Asia today: Changing policies and practices (2nd ed. ed., pp. 313-319). Singapore: Eastern Universities Press Norris, J. M., Brown, J. D., Hudson, T., & Yoshioka, J. (1998). Designing second language performance assessments. Hawaii: Second Language Teaching & Curriculum Center, University of Hawaii at Manoa. Numrich, C. (1996). On becoming a language teacher: Insights from diary studies. TESOL Quarterly, 30(1), 131-153. doi: 10.2307/3587610 Nunan, D. (1992). The teacher as decision-maker. In S. Hsia, M. N. Brock & J. R. Flowerdew (Eds.), Perspectives on second language teacher education (pp. 135- 165). Hong Kong: City Polytechnic. Nunan, D., & Bailey, K. M. (2009). Exploring second language classroom research: A comprehensive guide (1st ed.). Boston, MA: Heinle, Cengage Learning. Nur, C. (2004). English language teaching in Indonesia: Changing policies and practical constraints. In W. K. Ho & R. Wong (Eds.), English language teaching in East Asia today: Changing policies and practices (2nd ed. ed., pp. 178-186). Singapore: Eastern Universities Press O’Sullivan, M. (2004). The reconceptualisation of learner-centred approaches: A Namibian case study. International Journal of Educational Development, 24(6), 585-602. doi: http://dx.doi.org/10.1016/S0738-0593(03)00018-X Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed.). Thousand Oaks, California: Sage Publications. Phipps, S. (2010). Language teacher education, beliefs and classroom practices: A case study of the development of in-service language teachers' grammar teaching beliefs and practices. Saarbrèucken: Lap Lambert Academic. Phipps, S., & Borg, S. (2009). Exploring tensions between teachers’ grammar teaching beliefs and practices. System, 37(3), 380-390. doi: 10.1016/j.system.2009.03.002 Pillay, H. (2002). Teacher development for quality learning: The Thailand education reform project. Retrieved 8 November 2010 http://www.edthai.com/publication/0005/fulltext.pdf Popham, W. J. (2011). Classroom assessment: What teachers need to know (6th ed.). Boston, Massachusetts: Pearson Education, Inc. Popko, A. J. (2005). How MA-TESOL students use knowledge about language in teaching ESL classes. In N. Bartels (Ed.), Applied linguistics and language teacher education (pp. 365-386). Boston: Kluwer Academic Publishers. Prapphal, K. (2008). Issues and trends in language testing and assessment in Thailand. Language Testing, 25(1), 127-143. Pula, J. J., & Huot, B. A. (1993). A model of background influences on holistic raters. In M. M. Williamson & B. A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 237-265). Cresskill, N.J.: Hampton Press. Punch, K. F. (2009). Introduction to research methods in education. Los Angeles; London: SAGE.

281

Punthumasen, P. (2007). International program for teacher education: An approach to tackling problems of English education in Thailand. Paper presented at the The 11th UNESCO-APEID International Conference Reinventing Higher Education: Toward Participatory and Sustainable Development, Bangkok, Thailand. http://worldedreform.com/pub/paperie13dec07.pdf Rankin, E. (2001). The work of writing: Insights and strategies for academics and professionals. San Francisco: Jossey-Bass. Rea-Dickins, P. (2001). Mirror, mirror on the wall: Identifying processes of classroom assessment. Language Testing, 18(4), 429-462. doi: 10.1177/026553220101800407 Rea-Dickins, P. (2004). Understanding teachers as agents of assessment. Language Testing, 21(3), 249-258. doi: 10.1191/0265532204lt283ed Rea-Dickins, P., & Gardner, S. (2000). Snares and silver bullets: Disentangling the construct of formative assessment. Language Testing, 17(2), 215-243. doi: 10.1177/026553220001700206 Read, J. (2010). Researching language testing and assessment In B. Paltridge & A. Phakiti (Eds.), Continuum companion to research methods in applied linguistics (pp. 286-300). London Continuum. Reid, J., & Kroll, B. (1995). Designing and assessing effective classroom writing assignments for NES and ESL students. Journal of Second Language Writing, 4(1), 17-41. doi: http://dx.doi.org/10.1016/1060-3743(95)90021-7 Richards, J. C. (1996). Teachers' maxims in language teaching. TESOL Quarterly, 30(2), 281-296. doi: 10.2307/3588144 Richards, J. C. (1998). Beyond training: Perspectives on language teacher education. Cambridge; New York: Cambridge University Press. Richards, J. C., Gallo, P. B., & Renandya, W. A. (2001). Exploring teachers’ beliefs and the processes of change. 41-62. Richards, J. C., & Renandya, W. A. (2002). Methodology in language teaching: An anthology of current practice. Cambridge: Cambridge University Press. Rinnert, C., & Kobayashi, H. (2001). Differing perceptions of EFL writing among readers in Japan. The Modern Language Journal, 85(2), 189-209. doi: 10.1111/0026-7902.00104 Rowell, P. M. (1995). Perspectives on pedagogy in teacher education: The case of Namibia. International Journal of Educational Development, 15(1), 3-13. doi: http://dx.doi.org/10.1016/0738-0593(93)E0012-Y Saad, M. R. B. M., Sardareh, S. A., & Ambarwati, E. K. (2013). Iranian secondary school EFL teachers’ assessment beliefs and roles Life Science Journal 10(3), 1638-1647. Sakyi, A. A. (2000). Validation of holistic scoring for ESL writing assessment: How raters evaluate composition. In A. J. Kunnan (Ed.), Fairness and validation in language assessment: Selected papers from the 19th Language Testing Research Colloquium, Orlando, Florida (pp. 129-152). Cambridge: Cambridge University Press. Sato, M., Coffey, J., & Moorthy, S. (2005). Two teachers making assessment for learning their own. Curriculum Journal, 16(2), 177-191. doi: 10.1080/09585170500135996 Sawaki, Y. (2007). Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite. Language Testing, 24(3), 355-390. doi: 10.1177/0265532207077205 Schreier, M. (2012). Qualitative content analysis in practice. London: SAGE.

282

Scott, D., & Usher, R. (2010). Researching education: Data, methods and theory in educational enquiry Retrieved from http://unsw.eblib.com/patron/FullRecord.aspx?p=634557 Seale, C. (1999). The quality of qualitative research. London: SAGE. Shaw, S. D., & Weir, C. J. (2007). Examining writing: Research and practice in assessing second language writing. Cambridge ; New York: Cambridge University Press. Shepard, L. A. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4-14. doi: 10.2307/1176145 Shepard, L. A. (2002). The role of classroom assessment in teaching and learning. In V. Richardson (Ed.), Handbook of research on teaching (4th ed., pp. 1066-1101). Washington, D.C.: American Educational Research Association. Shepard, L. A., & Bliem, C. L. (1995). Parents' thinking about standardized tests and performance assessments. Educational Researcher, 24(8), 25-32. doi: 10.2307/1176891 Shepard, L. A., Flexer, R. J., Hiebert, E. H., Marion, S. F., Mayfield, V., & Weston, T. J. (1996). Effects of introducing classroom performance assessments on student learning. Educational Measurement: Issues and Practice, 15(3), 7-18. doi: 10.1111/j.1745-3992.1996.tb00817.x Shi, L. (2001). Native- and nonnative-speaking EFL teachers' evaluation of Chinese students' English writing. Language Testing, 18(3), 303-325. doi: 10.1177/026553220101800303 Shohamy, E., Gordon, C. M., & Kraemer, R. (1992). The effect of raters' background and training on the reliability of direct writing tests. The Modern Language Journal, 76(1), 27-33. Silva, T. (1990). Second language composition instruction: Developments, issues, and directions in ESL. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 11-23). Cambridge: Cambridge University Press. Simon, B. (1998). Teachers' pedagogical systems and grammar teaching: A qualitative study. TESOL Quarterly, 32(1), 9-38. Smith, D. (2000). Rater judgments in the direct assessment of competency-based second language writing ability. In G. P. Brindley (Ed.), Studies in immigrant English language assessment (Vol. 1, pp. 159-189). North Ryde, N.S.W.: National Centre for English Language Teaching and Research. Smith, D. B. (1996). Teacher decision making in the adult ESL classroom. In D. A. Freeman & J. C. Richards (Eds.), Teacher learning in language teaching (pp. 197-216). Cambridge; New York: Cambridge University Press. Song, B., & Caruso, I. (1996). Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students? Journal of Second Language Writing, 5(2), 163-182. doi: http://dx.doi.org/10.1016/S1060- 3743(96)90023-5 Speck, B. W. (2000). Grading students' classroom writing: Issues and strategies. Washington, DC: Graduate School of Education and Human Development, The George Washington University. Stake, R. E. (2005). Qualitative case studies. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (3rd ed., pp. xix, 1210 p.). Thousand Oaks, CA: Sage Publications. Tan, C. (2007). Education reforms in Cambodia: Issues and concerns. Educational Research for Policy and Practice, 6(1), 15-24. doi: 10.1007/s10671-007-9020-3

283

Teasdale, A., & Leung, C. (2000). Teacher assessment and psychometric theory: A case of paradigm crossing? Language Testing, 17(2), 163-184. doi: 10.1177/026553220001700204 Tsui, A. B. M. (2003). Understanding expertise in teaching: Case studies of second language teachers. Cambridge: Cambridge University Press. van Lier, L. (2005). Case study. In E. Hinkel (Ed.), Handbook of research in second language teaching and learning (pp. 195-208). Mahwah, N.J.: L. Erlbaum Associates. van Someren, M., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think aloud method: A practical guide to modelling cognitive processes. London: Academic Press. Vaughan, C. (1991). Holistic assessment: What goes on in the raters' minds? In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 111-125). New Jersey: Ablex Publidhing Corporation. Watson Todd, R. (2006). Why investigate large classes? . KMUTT Journal of Language Education, 9, 1-12. Weigle, S. C. (1994). Effects of training on raters of ESL compositions. Language Testing, 11(2), 197-223. doi: 10.1177/026553229401100206 Weigle, S. C. (1999). Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing, 6(2), 145-1778. Weigle, S. C. (2002). Assessing writing. Cambridge: Cambridge Unviersity Press. Weigle, S. C. (2007). Teaching writing teachers about assessment. Journal of Second Language Writing, 16(3), 194-209. doi: 10.1016/j.jslw.2007.07.004 Weir, C. J. (2005). Language testing and validation: An evidence-based approach. Basingstoke, UK: Palgrave Macmillan. Westerman, D. A. (1991). Expert and novice teacher decision making. Journal of Teacher Education, 42(4), 292-305. doi: 10.1177/002248719104200407 White, E. M. (1994). Teaching and assessing writing: Recent advances in understanding, evaluating, and improving student performance (2nd ed.). San Francisco: Jossey-Bass. Wiersma, W., & Jurs, S. G. (2009). Research methods in education: An introduction (9th ed.). Boston: Pearson. Wiliam, D. (2001). An overview of the relationship between assessment and the curriculum. In D. Scott (Ed.), Curriculum and assessment (pp. 165-181). Westport, CO: Alex. Wiriyachitra, A. (2002). English language teaching and learning in Thailand in this decade. Thai TESOL Focus, 15(1), 4-9. Wolf, D., Bixby, J., Glenn, J., & Gardner, H. (1991). To use their minds well: Investigating new forms of student assessment. Review of Research in Education, 17(ArticleType: research-article / Full publication date: 1991 / Copyright © 1991 American Educational Research Association), 31-74. doi: 10.2307/1167329 Wongsothorn, A., Hiranburana, K., & Chinnawongs, S. (2002). English language teaching in thailand today. Asia-Pacific Journal of Education, 22(2), 107-116. Wood, R. (1993). Assessment and testing: a survey of research. Cambridge: Cambridge University Press. Xu, Y., & Liu, Y. (2009). Teacher assessment knowledge and practice: A narrative inquiry of a chinese college EFL teacher's experience. TESOL Quarterly, 43, 493-513. Yin, M. (2010). Understanding classroom language assessment through teacher thinking research. Language Assessment Quarterly, 7(2), 175-194. doi: 10.1080/15434300903447736

284

Yin, R. K. (2003). Case study research: Design and methods (3rd ed.). Thousand Oaks, California: Sage. Yin, R. K. (2009). Case study research: Design and methods (4th ed.). London: SAGE. You, X. (2004). New directions in EFL writing: A report from China. Journal of Second Language Writing, 13(4), 253-256. doi: http://dx.doi.org/10.1016/j.jslw.2004.09.002 Yu, S. (2013). EFL teachers' beliefs and practices regarding peer feedback in L2 writing classroom Polyglossia, 24(March 2013), 74-79. Zeichner, K. (1999). The new scholarship in teacher education. Educational Researcher, 28(9), 4-15. doi: 10.3102/0013189x028009004 Zheng, H. (2013). The dynamic interactive relationship between Chinese secondary school EFL teachers' beliefs and practice. The Language Learning Journal, 41(2), 192-204. doi: 10.1080/09571736.2013.790133

285

Appendix A The questionnaire

Survey on Writing Assessment Practices of Thai Public University Teachers

This questionnaire aims to collect data on writing assessment practices of teachers in Thai public universities. It focuses on the practice of teachers of a writing course at a composition/essay level for English major students. The questionnaire comprises 7 sections: background information, teaching experience, training experience on marking writing, views on good writing, writing at a composition/essay level course, views on marking compostions/essays and other comments about writing assessment. The questionnaire is completely anonymous, and the results will be used for research purposes only. The questionnaire should take no more than 15-20 minutes. I thank you for your time and effort. This questionnaire is a part of the study being conducted by Ms. Somruedee Khongput to meet the requirements of Doctor of Philosophy in Linguistics under the supervision of Professor Anne Burns, Tel. No: +61 2 9850 8604 E-mail address: [email protected] and Mr. John Knox, Tel. No: +61 2 9850 8729 E-mail address: [email protected]. Should you have any further questions, please do not hesitate to contact: Ms. Somruedee Khongput, Tel. No. 61 4 3206 9481 E-mail address: [email protected]

Please answer the questions by marking X in the box in front of the appropriate responses or writing on the spaces provided.

Section I: Background information

1. Gender: a. Female b. Male

2. Age:  21-25  26-30  31-40  41-50  51-60  over 60

3. Highest education:  Certificate  Diploma  Bachelor’s degree  Master’s degree  Doctoral degree Please specify where you completed your highest education: ______

4. Nationality: ___Thai______

287

5. 1st language:  Thai  English  Other (Please specify): ______

6. Other languages you have a good command of (Please specify): ______

7. Please answer if you are a non-native speaker of English. The length of time you studied or lived in English speaking countries: ____ years If less than a year, please specify. ______months

Section II: Teaching experience

1. How long have you been teaching English? ______years If less than a year, please specify. ______months

2. How long have you been teaching English in Thailand? ______years If less than a year, please specify. ______months

3. How long have you been teaching writing? ______years If less than a year, please specify. ______months

4. How long have you been teaching writing for EFL students? ______years If less than a year, please specify. ______months

Section III: Training experience on marking writing

1. Have you ever had any training regarding marking writing?  No (Go to Section IV)  Yes. Please select the kind of training you have had. More than one answer is possible.  formal training for standardized test raters  formal training within your department  formal training as a part of certificate/degree qualification  Others (Please specify):______2. Please comment briefly on the training you have experienced (e.g. length of training time, numbers of sessions you joined the training, your view on the training). ______

288

3. How useful have you found the training when marking writing?  Not at all  A little  Very  Extremely Please comment: ______

Section IV: Views on good writing

1. How important are these aspects to good writing? Please circle the number according to their importance of each aspect. 5 equals very important and 1 equals not important.

Writing aspects Very Not important important Cohesion……………………………… 5 4 3 2 1 Task completion……………………… 5 4 3 2 1 Relevance of ideas………………….... 5 4 3 2 1 Logical development of ideas………… 5 4 3 2 1 Appropriate grammar………………… 5 4 3 2 1 Appropriate vocabulary use………….. 5 4 3 2 1 Oganisation/Overall structure of the text 5 4 3 2 1 Mechanics (e.g. spelling, punctuation, 5 4 3 2 1 capitalization) Others:  ______5 4 3 2 1  ______5 4 3 2 1  ______5 4 3 2 1

Other comments on writing aspects (please specify): ______

Please complete Section V and VI based on the highest level of writing you teach.

Section V: Writing at a composition/essay level course

1. Name of the course:______

2. Level of writing (e.g. paragraph, composition level): ______

289

3. What kinds of writing do you have in the class? More than one answer is possible.  cause-effect  problem-solution  comparison-contrast  letter  narrative  procedure  description  classification  persuasive writing  others (Please specify):______

4. What criteria do you use for marking writing? More than one answer is possible.  no criteria  university’s criteria (the central criteria that your university gives to all university teachers to apply in any writing course)  department’s criteria (the central criteria that your department gives to all teachers in the department to apply in any writing course)  course criteria (the criteria that your department gives to the teachers teaching a particular writing course)  your own criteria (the criteria that you and/or your co-teacher(s) create to use in your writing course)  other (Please specify): ______Please comment on how you use each kind of criteria: ______

Section VI: Views on marking compositions/essays

How do these statements apply to you when marking students’ writing? Please circle the number according to your response.

5 = always 4 = most of the time 3 = often 2 = occasionally 1 = hardly ever N/A = not applicable to your situation

Statements 1. I mark compositions/essays by giving a 5 4 3 2 1 N/A single score to the overall quality of writing. 2. I compare the quality of all 5 4 3 2 1 N/A compositions/essays in the same task and give scores according to their rank orders. 3. I mark compositions/essays by giving scores 5 4 3 2 1 N/A to distinct aspects of writing (e.g. task completion, cohesion, mechanics).

290

4. I assign equal weighting to every aspect of 5 4 3 2 1 N/A writing. 5. I mark different types of compositions/essays 5 4 3 2 1 N/A by using different sets of criteria. 6. Before marking compositions/essays, my co- 5 4 3 2 1 N/A teacher(s) and I discuss marking criteria and mark the compositions/essays based on our agreement. 7. Before marking compositions/essays, my co- 5 4 3 2 1 N/A teacher(s) and I select and mark a sample of compositions/essays to ensure score reliability. 8. I read compositions/essays more than once 5 4 3 2 1 N/A before giving scores. 9. I adjust the criteria during the marking period 5 4 3 2 1 N/A without consulting my co-teacher(s). 10. I compare different pieces of writing to 5 4 3 2 1 N/A ensure the score consistency. 11. The quality of students’ handwriting affects 5 4 3 2 1 N/A the grade I give their compositions/essays. 12. Knowing who the writer is affects my 5 4 3 2 1 N/A decision of what score to give. 13. The final score I give to all 5 4 3 2 1 N/A compositions/essays is based on the criteria.

Section VII: Other comments about writing assessment

If you have any comments about writing assessment practices in Thailand, please feel free to express them. ______

Thank you for your time

If you are willing to participate in the follow-up case studies, please give your contact e-mail address: ______

Note that once you provide your e-mail address, your responses to this survey will not be completely anonymous. However, all of your information and data obtained will be confidential. Only the researcher and her supervisors will have access to your identity. When quotes are used, your identity will not be revealed.

291

Appendix B The semi-structured interview question guide

Pre-rating questions Teaching practice 1. In general, how many hours do you teach per week? 2. How many writing classes do you teach? What classes are they? 3. What is the highest level writing class?

Nature of students 1. Please tell me about the students in your class (year, proficiency, number of students, etc.).

Nature of the highest level writing course 1. What is the aim of the course? 2. How many text types do they have to write? Please describe what they are. 3. What form of writing do students submit, written or typed; online or hard copy? 4. How many drafts do you allow students to write before final submission? 5. Which draft of writing do you score?

Rating criteria 1. What criteria do you use? (How do you get them? Who helps you do that?) 2. Do you use the same set of criteria to rate writing in all text types? How do you use them?

Assessment practices 1. Do you have co-teachers? 2. How do you and your co-teacher(s) rate writing? (e.g. rate the same piece of writing, rate the work of students each of you teach) 3. How do you normally rate a piece of writing? Please describe. 4. When you rate writing, how long approximately does it take you to rate one piece of writing? Please give an example. 5. What do you find as challenges when assessing writing? 6. Are there any restrictions in assessing writing?

292

Training (The questions will be asked if the participants have had training.) 1. How often do you have training in assessment? 2. What sort of training have you had in the past? 3. Please describe how training in assessment you have had helps you when you rate the writing.

Post-rating questions Strengths and weaknesses 1. Do you have anything further to say about writing assessment practices in your department/course? (strengths and weaknesses)

Improvements 2. What do you want to see improved in relation to the writing assessment?

293

Appendix C Think-aloud instructions and training guide

Think-aloud protocol requires you to say everything you think while you are doing a task in order to document and understand what you do and think when performing a task. The purpose of think-aloud protocol in this study is to gain a deeper understanding of what you think, decide and do while you are rating sample pieces of your students’ writing. Now I am going to ask you to rate two scripts of your students’ work in the same text type. I would like you to rate them using the criteria you usually use in the usual way. The important thing to remember is, please say anything you are thinking about while you are rating the scripts. I will record what you are doing and saying with video recording. You should first identify each script by the student ID. Then, while rating the scripts, please vocalise your thoughts and explain why you give a particular score or comment. Please keep talking while you are thinking. If you spend time reading the scripts or the rating criteria, please also do that aloud. While you are rating, I will be sitting here without saying anything to interrupt your thoughts and talking. However, if you fall silent for more than five seconds, I will prompt you to keep talking by saying, “Please continue talking”.

Before rating your students’ scripts, I would like you to practise how to talk while thinking. The following three warm-up tasks will help you better understand what think-aloud is. I will demonstrate the first task to you and let you practise the rest.

Task 1: Without a calculator, please work out 1298 + 45683

Task 2: Please read these sentences and correct any mistakes you find. 1. I am certainly what that mean. 2. A grown fully Chihuahua dog may only weigh as little as four kilograms. 3. Today's lecture will be on the subject of problems environmental and air pollution.

294

Task 3: Please rate the following text sample using your usual criteria.

Write a paragraph describing your reasons to study English. Use transitions where necessary.

Reason to study English I have three reasons to study English. First of all, English increases the chance of finding a job. Many jobs have to use English to contact to foreigner. The second reason is that you can learn anything more easily. If you know English well. you can learn many things in the world. because many countires use English to be formal language. if you want to know about Aztec’s tribe you can learn in English historical book from the library of congress in USA. The third reason is that English can learn many places. English is in my places, for instance, guide post , book building , lebel of product and internet. For these three reason , I study English to today.

295

Appendix D Stimulated recall interview question guide

Rating focus 1. What was your main focus while you were rating?

Challenges 2. Could you describe any problems you encountered when you rated your students’ writing?

Assessment practice 3. Did you keep the aim of the course in mind while assessing your students’ writing? 4. Do you normally give this kind of feedback to your students’ writing? 5. How do you think your rating/feedback influence your students’ writing improvement?

296

Appendix E Participant information statement and consent form

Approval No. 11 037

THE UNIVERSITY OF NEW SOUTH WALES

PARTICIPANT INFORMATION STATEMENT AND CONSENT FORM

Title of project: Teachers’ Writing Assessment Practices in EFL Context: Case Studies

Participant Selection and Purpose of Study You are invited to participate in a study of writing assessment practices of teachers of writing in Thai public universities. We hope to learn insights of teachers’ actual practices of rating their students’ compositions/essays. The study will focus on the rating process and teachers’ decision making while rating the compositions/essays. You were selected as a possible participant in this study because you are likely to be a representative of teachers of writing in universities in a particular part/region of Thailand.

Description of study and risks If you decide to participate, we will ask you to participate in a case study which you will conduct a think-aloud protocol while you are rating sample pieces of your students’ writing. Then you will be interviewed in stimulated recall and semi-structured interviews before the think-aloud protocol and after stimulated recall. The procedure of all mentioned activities will take approximately 2 hours 30 minutes.

We will video-record during your think-aloud protocol and audio-record the interviews as well as stimulated recall. The recording may cause discomforts and inconveniences.

This study may not have direct benefits to you. However, you can be more aware of your rating practices. The data obtained from the think-aloud protocol, stimulated recall and the interviews will give us insights of writing assessment practices in Thailand and the need for teacher development in this area may emerge. We cannot and do not guarantee or promise that you will receive any benefits from this study.

297

Confidentiality and disclosure of information Any information that is obtained in connection with this study and that can be identified with you will remain confidential and will be disclosed only with your permission, except as required by law. If you give us your permission by signing this document, we plan to discuss, publish and present the results of this study in conferences, journal articles, book sections and the researcher’s PhD thesis. The information of this study will be furnished for other EFL teachers and researchers in terms of assessing writing and conducting research in this area. In any publication, information will be provided in such a way that you cannot be identified.

Recompense to participants You will be receiving a book voucher as remuneration with the value of 500 THB.

Complaints may be directed to the Ethics Secretariat, The University of New South Wales, SYDNEY 2052 AUSTRALIA (phone 9385 4234, fax 9385 6648, email [email protected]). Any complaint you make will be investigated promptly and you will be informed out the outcome.

Feedback to participants If you require a summary of research findings, please tick a box and provide your email address at the end of the consent form. The summary of the findings will be sent to you through your email.

Your consent Your decision whether or not to participate will not prejudice your future relations with the University of New South Wales. If you decide to participate, you are free to withdraw your consent and to discontinue participation at any time without prejudice.

If you have any questions, please feel free to ask us. If you have any additional questions later, Ms.Somruedee Khongput, phone number +66 81 5996658 will be happy to answer them.

You will be given a copy of this form to keep.

298

THE UNIVERSITY OF NEW SOUTH WALES PARTICIPANT INFORMATION STATEMENT AND CONSENT FORM (continued)

Title of project: Teachers’ Writing Assessment Practices in EFL Context: Case Studies

You are making a decision whether or not to participate. Your signature indicates that, having read the information provided above, you have decided to participate.

…………………………………………… ………………………………………… Signature of Research Participant Signature of Witness

…………………………………………... …………………………………………. (Please PRINT name) (Please PRINT name)

………………………………………….. .………………………………………… Date Nature of Witness

REVOCATION OF CONSENT

Title of project: Teachers’ Writing Assessment Practices in EFL Context: Case Studies

I hereby wish to WITHDRAW my consent to participate in the research proposal described above and understand that such withdrawal WILL NOT jeopardise any treatment or my relationship with The University of New South Wales.

…………………………………………… …………………………………….. Signature Date

…………………………………………………… Please PRINT Name

The section for Revocation of Consent should be forwarded to Prof. Anne Burns, School of Education, The University of New South Wales, SYDNEY 2052 AUSTRALIA.

299

Appendix F The transferred ethics approval THE UNIVERSITY OF NEW SOUTH WALES

10 Januaty 2011

Professor Anne Bums School of Education HUMAN RESEARCH ETHICS 136 Goodsell Building COMMITTEE (HREC)

Dear Professor Bums,

Teachers' writing assessment practices in EFL context: case studies HREC 10439 MQ ref: 5201000304D

Thank you for the above application for ratification of the ethics clearance given by Macquarie University Faculty of Human Sciences Ethics Review Sub-Committee to you dated 5 May 2010.

The Executive noted the above protocol at its meeting held on 14 December 2010, and is pleased to advise it is satisfied that it meets the requirements as set out in. the National Statement on Ethical Conduct in Human Research*. The UNSW Human Research Etl1ics Committee has accepted the request to take over as the primacy ethics committee.

The Deputy Vice-Chancellor (Research) accepted the ethics Committee's recommendation.

Would you please note -:

• approval is valid for tlve years (from the date of approval given by the Macquarie University i.e. 30 April2010);

• you will be required to provide annual reports on the study's progress to the HREC, as recommended by the National Statement;

• you are required to immediately report to the Ethics Secretariat anything which might warrant review of ethicai approval of the protocol (National Statement 3.3.22, 5.5.7) including:

a) serious or unexpected outcomes experienced by research participants (using the Serious Adverse Event proforma on the University website at http://www.gmo.unsw.edu.au/Etbics/HumanEthic$il.PformationForAp.Qiicants!Proforma sTemplates/Cl3 SAE%20Profonna.l1D; b) proposed changes in the protocol; and c) unforeseen events or new infonnation (eg from other studies) that might affect continued ethical acceptability of the project or may indicate the need for amendments to the protocol;

• any modifications to the project must have prior written approval and be ratified by any other relevant Human Research Ethics Committee, as appropriate; .. 1..

UNSW SYDN EY NSW 2052 AUSTRAliA Telephone: +6 1 (2) 9385 4234 facsim i l e: +6 1 ( 2) 9 385 6646 Ema il: e t hics.sec@unsw. edu.au Locali

(HREC 10439 cont'd)

..2 ..

• if there are implantable devices, the researcher must establish a system for tracking the participants with implantable devices for the lifetime of the device (with consent) and report any device incidents to the TGA;

• if the research project is discontinued before the expected date of completion, the researcher is required to infonn the HREC and other relevant institutions (and where possible, research participants), giving reasons. For multi-site research, or where there has been multiple ethical review, the researcher must advise how this will be communicated before the research begins (National Statement 3.3.23 and 5.5.6);

• consent fonns are to be retained within the archives of the School and made available to the Committee upon request.

Yours sincerely,

Professor Michael Grimm Presiding Member HREC

*http://www.nhmrc.gov.au

301

Appendix G Ethics approval for the case studies

Human Research Ethics Advisory Panel B Arts, Humanities & Law

Date: 13.05.2011

Investigators: Ms Somruedee Khongput

Supervisors: Professor Anne Burns

School: School of Education

Re: Teachers' Writing Assessment Practices in EFL Context: Case Studies

Reference Number: 11 037

The Human Research Ethics Advisory Panel B for the Arts, Humanities & Law is satisfied that this project is of minimal ethical impact and meets the requirements as set out in the National Statement on Ethical Conduct in Human Research*. Having taken into account the advice of the Panel, the Deputy Vice-Chancellor (Research) has approved the project to proceed.

Your Head of School/Unit/Centre will be informed of this decision.

This approval is valid for 12 months from the date stated above.

Yours sincerely

Associate Professor Annie Cossins Panel Convenor HREA Panel B for the Arts, Humanities & Law

Cc: Professor Chris Davison Head of School School of Education

* http://WvVW.nhmrc.gov.au/

302

Appendix H Examples of questionnaire coded responses

1. Criteria Categories Subcategories Example comments Focus Appropriateness of Focus on how language is used formal or informal, language appropriate or inappropriate in business area [Res01] Mechanics objective: rubric used to assess mechanics of the Overall impression composition subjective: overall assessment of assignment - did they get it? [Res15] Task completeness As mentioned earlier, holistically, whether or not the task Organisation is achieved comes first, followed by organisation, and Language equally important, language, without which the reader will find it hard to understand what the writer wants to say. [Res21] Grammar Grammartical correct [sic], coherent [sic] and cohesion, Coherence and word choices, spelling, forms, appropriateness of cohesion language used: formal and informal [Res25] Vocabulary Mechanics Appropriateness of language **high level of well researched literature review comprehensible writing** structure/outline language use rich discussion writer' s voice [Res34] Preferred rating Subjectivity For the aspect of grammar/structure and vocabulary, I criteria think it is subjective, though with criteria, because of individual (teachers) writing style. 2. For the aspect of organization/cohesion, it should be more universal. [Res06] I don't believe that the grads [sic] as a result of work evaluation can explain students' competence. We have assessed students' learning by using some methods that we called 'scientific ways', not realizing that human elements are so complex that they cannot be evaluated scientifically as scientists do in their scientific experiments. I feel apathetic [sic] for the students who sit in classes whose teachers are grammar-oriented, who never appreciate nothing but correct, but dead, grammar. These students have well organized, well written pieces, but these pieces are useless in terms of minds and souls. [Res35] Criterion I wish it [translation] could be integrated as I feel that if students 'write from the heart' or 'write freestyle' they often avoid many grammar and vocabulary usage problems probably caused by translation from their mother tongues. [Res16] Integration: I think there is inconsistency, but I do not believe any criteria, judgment, more than anywhere else. The problem with criteria is that experience they tend to invite a mechanical application, as if grading papers is a science, whereas I think it is more of an art. Criteria give the illusion of greater fairness and reliability, but I have reservations. As I pointed out, when I apply criteria, students tend to get lower grades than the overall grade I believe they deserve. We have had cases where teachers apply the same criteria very differently. I think the best probable solution is a (somewhat messy) mixture of criteria, judgment and experience. [Res46]

303

Categories Subcategories Example comments Different criteria to Writing is complicated, and in each writing mode, we need different tasks different criteria to evaluate students' works. One approach cannot fit all. [Res34] Aim To ensure The first three sources of marking criteria [university’s reliability and criteria, department’s criteria and course criteria] would validity reduce the subjectivity of intra-rater reliability because we had to rely on the same set criteria. However, personal criteria can be added but justification was needed. It's to help ensure that the strategies taught in class were carefully applied in students' writing task, which may be different from other sections/teachers. Above all, the first three criteria may not cover what I had taught them in class; therefore, my own criteria was [sic] also supplemented. [Res07] Selection Genre of writing I tend to give importance to writing aspects used for grading criteria. However, to what extent each aspect should be emphasized really depends on the genre of writing. [Res05] Students’ It depends on the students' background, but I tried to background achieve the criteria I stated. [Res14] Types of writing The type of writing may affect the importance of those points [writing features]. [Res48] Students’ need Although I put 4 [of 5 in the rating scale showing the importance of writing features] for grammar, vocabulary and mechanisms, I think they are the aspects that students want me to focus on. [Res06] Sources of Several sources Criteria used are based on several sources. [Res37] criteria Standardised tests We also look at the criteria from other standardised tests, such as TOEFL and IELTS. We also aim to prepare our students to be ready for the high stakes tests, too. [Res35] Standardised tests Though we claimed that the criteria/rubrics are ours, we constructed them based on the writing conventions of the English language and also adopted/adapted from many standardised tests, IELTS, TWE, etc. [Res21] Use Test marking It's normally used when marking unit writing tests. [Res32] Usefulness Help objectivity Using a rubric makes the marking more objective. [Res15]

2. Practice

Categories Subcategories Example comments Bias Bias avoidance When I ask my students to submit their writing, I ask them technique: Write write only their student ID so that I can avoid knowing who student ID the writers are. [Res03] Bias avoidance: In my situation, when teachers mark students' writing Not mark their own examination papers, they do not mark their own students' students’ writing paper. That means, quite often that they do not know their students. [Res05] Bias avoidance: To reduce bias against poor handwriting, we ask the Use of writing students to type using Time New Roman font 12, double format guideline space. [Res58]

Bias awareness While handwriting would only very rarely influence my decisions, the layout of a typed page could have an influence on how I grade a paper, but this is something I consciously avoid. [Res16]

304

Categories Subcategories Example comments Technique to Reliability: For the group of 15 students and below, after you mark maintain Double marking students' work, it is a good idea to ask one of your writing reliability and practice teachers to mark again. Then compare your marks and the validity marks given by your colleague to see if the marking is reliable or not. [Res58] Validity: While correcting students' writing, I will follow my own - Students are criteria students are be informed that I will use this criteria. informed of the [Res03] criteria - Stick to the criteria set Reliability: A team of teachers of the course get together and agree on a Use of central set of rubrics for marking each writing test. [Res05] criteria Reliability: Use of - study and discuss the criteria together before we start the central criteria lessons - readjust the criteria (rubrics) if necessary [Res58]

Reliability: Use of most reliant on co-teachers We calibrate our marking using central criteria both a series of criteria (grammar, punctuation, vocabulary Mark before actual etc.) and by using student's writing to 'co-mark' to make sure marking we give similar grades. [Res16] Validity [I] explain students about the criteria and make agreement before we start the course [Res58]

validity I try to match my teaching and cover all criteria relating to the course objectives. [Res09]

Discrepancy of However, I think what we focus on more is the 1st aspect theory and [the aspect of grammar/structure and vocabulary] I practice mentioned. I think we should focus more on the 2nd aspect [the aspect of organization/cohesion]. [Res06] Co-teaching No co-teacher In my university, there's only writing course for each semester, so I can't find someone that teach the same subject as me. [Res03] I mark and use the criteria basing on my own as I don't have to work with my co-teacher in my course. [Res09] Unawareness of I don't know that [sic] there are any standard writing standard assessment practices where I work. [Res15] Use of additional In addition to criteria, I also use correction symbols for tool students' practices. [Res37] Positive towards I don't have any Example commentss because the current practice assessment practices in Thailand are proper for the implementation of teaching writing. [Res26] Rating style Difficult to say. I use my experience and knowledge of the kind of writing I am assessing. I have a fair idea of what is effective and not so effective. I judge the piece as a whole not giving marks for artificially divided components. [Res39]

305

3. Training

Categories Subcategories Example comments Reflection Quite useful. I think the seminar arranged was only short period of time and it included more than 1 skill; though, there're enhanced thoughts and concerns in marking writing. [Res04] I had some experiences in training of teaching writing. It was a two-day training. The content was based on academic writing with a certain format and creative writing. I found it very helpful for teaching. [Res26] It was a model of writing for university students with low background. [Res26] It was great for me at that time during the training. It exactly depends on the trainee who has to spend more time to practice everyday writing. I’ve known that my writing is the worst skill; I try to find new ways to practice. [Res33] A 3-month training course on teaching & testing EFL held at UCLES, Cambridge, UK, two 2-week training sessions on test construction part of which was on assessing writing organized by Thailand's Department of Teacher Education and UCLES held in Thailand (taught by an expert from UK), & writing assessment training among the faculty members of School of English, Suranaree University of Technology to ensure validity and inter-rater reliability on a trimester basis. [Res21] Usefulness Confidence It often gives me confidence before I mark the papers on my enhancement own. The training also includes the session in which all teachers mark a set of same papers and compare their given scores. [Res05] General ideas and It is just part of a course, language assessment, a semester concerns of marking (how long), which mainly focuses on assess four skills. The writing course gives me general ideas and some particular concerns that I have to be cautious when marking writing tasks. [Res08] How to teach and It was helpful for me on how to teach and evaluate writing mark skill for secondary and higher education students. [Res14] Writing fixation I joined the writing training very often in the past. It helped me a lot for writing fixtion [sic]. [Res18] Positive washback The training was very useful and had positive washback effect effects on L2 writing. [Res21] Fair assessment The training resulted in fair assessment since each rater agreed upon the rating criteria/rubrics and compared his/her evaluation of each piece of writing with the other raters throughot [sic] the training and thereby adjusting himself/herself all the time until a satisfactory inter- reliability was achieved [sic]. [Res21] useful It is always useful to experience training and learn what others in the field are doing. [Res23]

useful I think the training is great because I can use it as a criteria [sic] to score the students' work. [Res30] Helpful for novice Training could help new comers to evaluate written pieces teachers on a very basic manner. We discussed criteria for assessment like language, idea or organization, all of which appeared helpful for novice teachers. [Res34]

306

Categories Subcategories Example comments Sporadic depending For working professionals, usefulness in training may be on each session sporadic, and depends entirely on the outcome of each session, lecture or ‘lightbulb’ moment. Sometimes extremely useful measured by the acquisition of one actionable new idea. Sometimes not useful at all. [Res48] useless I found, based on my teaching experience of more than 30 years, the training was useless since Thai students (most of them) had never paid attention to even the abbreviation used in marking their paper. The only way to help them see and understand their mistakes is to explain and point out the mistakes one by one which is not only time consuming but also tiring but it is worth giving such efforts. [Res19] Applicability Not applicable The knowledge could not be applied with Thai students whose English background was totally different from others. [Res19] The training I received is not applicable to Thai (EFL) writing. [Res57] Characteristics Simple but practical [training] is important and more challenging. [Res58]

4. Management

Categories Subcategories Example comments Prerequisite No grammar There are no grammar-course prerequisites to the writing course course classes. This is a big mistake. It places the cart before the horse. Courses in composition writing or even paragraph writing become reduced to coursework-review in basic sentence writing and basic grammar review. Therefore, the term “assessment practices” sometimes begs the question: What exactly is being assessed? [Res40] Allocation of Novice teachers Most lecturers here do not want to teach any writing course. teachers are assigned to The newest lecturers, with no or little experience, have to teach writing. teach writing to English major students and have to keep teaching it until they are not the new faces anymore. Then, the new lecturer(s) can take over. The department might think that a person's ability to write well (or even all right) can be equated to the ability to teach writing. [Res52]

5. Students’ problems

Categories Subcategories Example comments Language Wrong language The students who study in my course, they [sic] always problem use due to negative write in informal language. In their daily life, they love to (Knowledge influence from chat with other friend all over the world but the language domain) language used on used is computer word, emoticon word, and abbreviate word the Internet [sic] and so on. Even though I try to have them aware of how to write in a business area but when they have a test, some of students use word [in an] inappropriate context. [Res01] By the way, the English language on the Internet, particularly in emails and chat rooms, makes my students confused so much. [Res22] Translation A major problem with many of my students' writing is translation. This is extremely difficult to identify, and therefore we do not use it as a criterion for grading. [Res16]

307

Categories Subcategories Example comments Collocation Thai students have some difficulties in finding the appropriate collocational expressions that are used in English. [Res35] Low English As a native speaker, we often hear that we should give high proficiency marks for the idea of the paper. However, I usually find that my students have very poor grammar and very poor sentence structure and very poor and incorrect vocabulary usage which severely limits their ability to complete assignments at a higher level. [Res57] Use of reason The use of reason and/or logical process is often absent in Logical process essay writing by non-native speakers. [Res16] Process writing I find EFL students have an extremely difficult time grasphing [sic] the idea of writing as a “process.” This is critical in thesis statement development and ultimately in writing a coherent essay. [Res23] Problems ranging For as long as I've taught English in the upper secondary from mechanics to level, and now in the higher education level, the hard organisation obstacles of Thai students ranging from mechanis [sic] to organisation. It's a hard work [sic] for me and I take time to give each student some written corrective feedback on the task. I have them write a diary or dialogue journals to promote their writing fluency. But it doesn't work. [Res22] Psychological Uneasiness with I find that Thai students have a difficult time with criticism. effects criticism They are often reluctant to "rewrite" an essay in order to (Affective correct mistakes, but I think this is critical and insist they domain) follow through. I have had students say that even though they don't like this practice, it is helpful in the long run. [Res23] Negative attitude Thai students tend to rely heavily on "letter grades". If they towards writing are given a bad grade in writing, they may stop learning it. This also becomes true to my students. They initially had negative attitude towards writing such as it's a painful process, structure is something mysterious for them, etc. With a low score perhaps they earned might interfere their personal attitude of writing [sic]. Therefore, teachers should be aware of it, esp [especially] for Thai students. [Res07] Negative feeling It [marking] affects to [sic] student's psychological feeling. due to the use of If Ss. [students] got [a] lot of markings, especially with a red red pen pen, they won't to [sic] participate in class. [Res59] Factors affecting Broad range of Actually, the problems in writing cover a broad range of problems factors: culture, factors; culture, lack of reading, training, and supports from lack of reading, the school. [Res35] training and support from school Direct translation Most of the time, Thai students translate directly from their from L1 L1 into English. So, some collocations are unnatural. [Res35]

6. Teacher:

Categories Subcategories Example comments Rating problem Lack of confidence Not with total confidence. Some criteria are not specific due to unclear enough i.e. minor/major grammatical error (which is criteria and which?); "few" error vs "sever [sig] error" receive different reliability concern score (how much serious is "few" or "several"; not sure if I interpret; for example, "score 4" the same as other teachers. [Res06]

308

Categories Subcategories Example comments Harshness in I always have a big problem with criteria, although I follow marking using them if they are provided. For some reason they always criteria result in me giving a lower overall grade than the writer deserves. [Res46]

Time constraint I believe most teachers would like to make their assessment reliable and valid. They would love to talk to their co- teachers to discuss about the criteria and to try marking papers before they really mark them. Yet, it's time constraint that would limit the possibility. [Res24] Knowledge Lack of knowledge Based on my experience, we normally have co-teachers problem of assessment and teach in the same writing course. However, some are [sic] language not know how to mark the tasks; they just use the methods proficiency they used to know based on their learning experience as students. In addition, even though in a course, teachers agree on the criteria, they tend not to mark in the same way. This is partly because they lack the body knowledge of assessment and their language proficiency e.g. They don't know some grammatical points. [Res08] Cause of Students’ weak I felt that I had a lot of obstacles of teaching the weak- teaching problem English proficiency English competency students. I don't know how to start with them. They are weak of every aspects [sic] of English language skills. I felt that they sound like they did not learn how to use grammar with writing and how to write up phrases or sentences. I mean this point of problem may occur in Rajabhat Universities, espectially [sic] in education field. I'm afraid that when they become teachers of English in every level of education, they cannot educate the students to learn enough English skills, espectially [sic] writing. [Res14] Students’ low As I formerly stated, a lot of problems occurred in [sic] English proficiency higher education students in teaching [them a] writing skill because of their low competencies in English usages and learning abilities. I'd like very much to teach them [the same way] as in other universities having students with good English competency to teach. Instead of teaching the processes of writing, I must revise them on grammar or sentence structure and how to use them in writing every [sic] level. [Res14]

7. Assessment

Categories Subcategories Example comments Effect Washback effect Judging the students' writing alone is not enough. Assessment should have washback effects so that the students can learn from the feedback and revise their writing accordingly to become more effective writers. [Res21] Product-oriented EFL student writers need to understand the English writing convention since each language has its own way of organizing ideas to achieve the writing purposes. Writing samples should be analyzed, perhaps through Swales' move analysis, to see how each text is organized, and the language needed to express the ideas need to be taught as well as mechanics (though the least emphasis). In other words, content/message comes first, then organization, and finally language. Equally important is the writing task assigned must be meaningful, that is, it must have an audience. EFL writers need to learn to be self-editors. [Res21]

309

Categories Subcategories Example comments Process-oriented In student centred methodology, process is more important than product. Judging 'writing' on the whole is a difficult task for an academic or teacher. The 'level' of the writer, and the 'type' of writing always affects the expected outcomes for writing tasks. [Res48]

310

Appendix I Transcription conventions for think-aloud protocols

Symbols Meanings Examples bold italicised Text read directly from the essay Living in country is often very (include single word read from the essay) different from living in city. [ ] Description of rater reaction or behaviour [Rewadee underlines the clause ‘when you have a problem’.] (inaudible) Unclear transcription (inaudible) in a sentence … (three dots) Incomplete or interrupted utterances Sometimes they ... for their parents ...

Hesitation markers: Codes Meanings Examples Eh For short hesitation sounds Eh, market in country life facility. (Nittaya) Er For long hesitation sounds Er, the topic is city life, country life. (Nittaya) Mmm For relatively hesitation sounds Mmm,7.5 then. (Ladda) Oh For exclamation Oh, rewrite. (Ladda)

311

312

Code note: A = Self-monitoring Focus 1 = Interpretation Strategies B = Rhetorical and Ideational Focus 2 = Judgment Strategies C = Language Focus D = Distraction U = Uncodable segment TA1 = Student 1’s script TA2 = Student 2’s script

Detailed codingscheme

Note: one text segment may be applicable to more than one code. AppendixJ Self-monitoring Focus > Interpretation Strategies

Code Category Focus Example text segments (Thai) Example text segments (English)

A1.1 Identify the writer - Assignment ชิ้นที่ 7 ของ …student’s name… The seventh assignment of ……student’s (Ladda TA1) name… (Ladda TA1) A1.2 Identify or interpret - เอ่อ ให้เขียนเปรียบเทียบของสองสิ่งนะคะ หรือ Errr, I asked them to compare two things. promt อะไรสองอย่าง (Rewadee TA1) (Rewadee TA1) A1.3 Identify topic - เด็กคนนี้เขียนหัวข้อ living in country and living This student wrote in the topic of living in in city นะคะ อันนี้ topic นะ (Rewadee TA1) country and living in city. This is the topic. (Rewadee TA1)

Code Category Focus Example text segments (Thai) Example text segments (English) A1.4 Identify outline - ตรงนี้นี่ Outline thesis tatement (Ladda TA2) This is outline thesis statement. (Ladda เอ่อ หน้าแรก นักศึกษา brainstorm idea ตัวเอง TA2) ให้ point มาว่า (Nittaya TA1) Errr, on the first page the student brainstorms his idea. The point is (Nittaya TA1) A1.5 Read or reread - saving time and helping authorities to identify saving time and helping authorities to outline/composition are three reasons why children should have identify are three reasons why children ID card. (Ladda TA2) should have ID card. (Ladda TA2) A1.6 Refer to teaching - เคยบอกหลายครั้งแล้ว บอกหลายครั้งแล้วว่า I told them many times that subject can’t subject นี่ จะขึ้นต้นด้วย preposition ไม่ได้ be a preposition. (Nittaya TA1) (Nittaya TA1) A1.7 Provision for feedback - เอ่อ ตรงนี้ต้องไปเน้นนักศึกษาอีกทีว่า เอ่อ Errr, I have to tell the students again. Errr, ค านามนี่ เวลาเขียนมันต้องมี article ด้วย a noun needs an article. (Nittaya TA1) (Nittaya TA1) A1.8 Envision personal - คนนี้รู้สึกจะชอบใช้ค าว่า seldom เหลือเกินนะ This student seems to like ‘seldom’ too situation of student (Nittaya TA01) much. (Nittaya TA1)

313

314

Self-monitoring Focus > Judgment Strategies Code Category Focus Example text segments (Thai) Example text segments (English) A2.1 Decide on - ดูที่ประโยค topic sentence ก่อนเช่นกันนะคะ Look at topic sentence first. (Rewadee TA2) macrostrategy for (Rewadee TA2) I need to read the whole text before reading and rating ต้องอ่านก่อนรอบหนึ่งแล้วค่อยตรวจ (Sak TA01) marking. (Sak TA1) A2.2 Compare with other - คนนี้ลบไป subject verb agreement 0.5 ก็ต้อง This student was deducted 0.5 for subject compositions or ลบให้เท่ากันนะคะ (Rewadee TA2) verb agreement. I have to do the same. “anchors” (Rewadee TA2) A2.3 Compare main text - อ๋อ แสดงว่านักศึกษาลอกมา ลอกจากด้านหน้า Uh, it means this student copied and with outline มา แล้วเอามาเรียงต่อกันเฉยๆ (Nittaya TA1) pasted this from the front page. (Nittaya TA1) A2.4 Summarise, distinguish - เนื้อหาก็ผิด ไวยากรณ์ก็ผิด คนนี้ผิด content มาก Content is wrong. Grammar is wrong. This or tally judgments เหลือเกิน แปลจากไทยแล้วก็เขียนเป๊ะๆ เลย student has too much wrong content. It’s collectively (Ladda TA2) direct translation from Thai. (Ladda TA2)

A2.5 Articulate results - โอย ท าใหม่ ๆ (Ladda TA2) Oh, rewrite. (Ladda TA2) เด็กคนนี้ลบไป 6 คะแนนแล้วกันค่ะ ก็จะได้ไป 14 This student is deducted 6 points. This เต็ม 20 (Rewadee TA02) makes him get 14 out of 20. (Rewadee TA2)

Code Category Focus Example text segments (Thai) Example text segments (English) A2.6 Articulate or revise - ขอหักคะแนนเรื่องการจัดเรียงไอเดีย เนื่องจากเรา I need to deduct idea organization overall scoring ก าหนดไว้นะคะว่าจะให้มีสามด้าน ขอหักไป 3 because there should be three aspects. I’ll decision คะแนน ด้านละ 1 คะแนน เพราะมันไม่ชัดเจน deduct 3 points, one point for one aspect. (Rewadee TA2) It’s not clear. (Rewadee TA2) A2.7 Identify cause of - อืม อันนี้ต้องคิดเป็นภาษาไทยมาก่อนแน่ ๆ Mmm, this is certainl from Thai. (Rewadee mistake (Rewadee TA2) TA2) A2.8 Write/Mark comment in - [T writes/marks something on the script.] [T writes/marks something on the script.] text A2.9 Articulate written - โน้ตไว้ว่า บางประโยคความหมายไม่ชัดเจน อ่าน I noted here that meaning of some comment แล้วไม่ค่อยเข้าใจ (Rewadee TA2) sentences is not clear. Incomprehensible. (Rewadee TA2) A2.10 Articulate rating - ตรวจไม่ค่อยจะถูกเลย (Nittaya TA01) I don’t know how to mark this. (Nittaya style/problem TA1) Rhetorical and Ideational Focus > Interpretation Strategies B1.1 Identify/interpret meaning อะไรนี่ (Nittaya TA01) What’s this? (Nittaya TA1) ambiguous or unclear phrase

315

316

Code Category Focus Example text segments (Thai) Example text segments (English) B1.2 Identify rhetorical Paragraph เพราะว่าจะต้องมี topic sentence ต้องมี ...because there must be a topic sentence, structure structure supporting detail, concluding sentence แล้ว supporting detail, concluding sentence as ก็ให้ใช้ transition word (Sak TA2) well as transition word. (Sak TA2) B1.3 Identify mistake of - อืม topic sentence ก็ไม่ถูก ไม่มี controlling Mmm, the topic sentence is not correct; rhetorical structure idea (Sak TA2) there’s no controlling idea. (Sak TA2) B1.4 Summarise/translate/ - เพื่อนบ้าน (after T reads ‘neighbor’) (Nittaya Neighbor (after T reads ‘neighbor’) (Nittaya interpret phrases, ideas TA1) TA1) or proposition B1.5 Identify topic - supporting point ที่สอง (Nittaya TA1) The second supporting point (Nittaya TA1) development ให้ตัวอย่างมาว่า (Nittaya TA1) The example is… (Nittaya TA1) Rhetorical and Ideational Focus > Judgment Strategies B2.1 Assess outline or topic - นักศึกษาลืมอีกแล้วว่า topic sentence นี่มันต้อง The student forgets again that a topic เป็นประโยค (Nittaya TA1) sentence needs to be a sentence. (Nittaya TA1) B2.2 Assess reasoning, - OK ideas สองอันนี้เข้าใจ (Nittaya TA1) Okay. These two ideas are logic, or topic comprehensible. (Nittaya TA1) development

Code Category Focus Example text segments (Thai) Example text segments (English) B2.3 Assess relevance - ไม่เห็นจะเข้ากันเลย (Nittaya TA1) They are irrelevant. (Nittaya TA1) B2.4 Assess coherence - โอเค ข้อดีคือมี transition signal ให้ โอเคที่ยังจ า Okay. A good point is the use of ได้ (Nittaya TA1) transitional signal (word). It’s good that they can still remember it. (Nittaya TA1) B2.5 Identify redundancies - พูดกลับไปกลับมา (Ladda TA1) Redundancy. (Ladda TA1) B2.6 Assess text - การจัดเรียงไอเดียค่อนข้างสับสน aspect ไม่ Confusing idea organisation, unclear organization ชัดเจน การเปรียบเทียบไม่ชัดเจน (Rewadee aspects, unclear comparison. (Rewadee TA2) TA2) B2.7 Assess task completion - ข้อความไม่ครบ (Sak TA1) This is incomplete. (Sak TA1) B2.8 Propose correction - ปกติ paragraph มัน เอ่อ ย่อหน้าได้รอบเดียว Commonly, a paragraph has one indent. (Nittaya TA1) (Nittaya TA1) B2.9 Rate ideas or rhetoric - การจัดเรียงไอเดียค่อนข้างสับสน aspect ไม่ Idea organization is confusing. Aspect is ชัดเจน การเปรียบเทียบไม่ชัดเจน หักไป 3.5 นะ not clear, Comparison is not clear. I’ll คะ (Rewadee TA2) deduct 3.5 points. (Rewadee TA2) B2.10 Assess layout - อยู่ดีๆ มันไม่ใช่ format ที่ถูกต้อง (Nittaya TA1) This is not the right format. (Nittaya TA1)

317

318

Language Focus > Interpretation Strategies Code Category Focus Example text segments (Thai) Example text segments (English) C1.1 Identify mistake - อยู่ดี ๆ ก็ขึ้น but (Nittaya TA1) Then, ‘but’ (Nittaya TA1) A, an, the ไม่มีอีกแล้ว (Ladda TA1) There’s no a, an, the. (Ladda TA1) C1.2.1 Classify mistake into Vocabulary Authorise เป็น verb (Ladda TA2) ‘Authorise’ is a verb. (Ladda TA2) types C1.2.2 Classify mistake into sentence subject verb agreement ไม่สัมพันธ์กันอีก Again, subject and verb do not agree. types structure/ แล้ว (Nittaya TA01) (Nittaya TA1) grammar C1.2.3 Classify mistake into Mechanics ตรงนี้ผิดเรื่องของ spelling (Sak TA1) Spelling is incorrect. (Sak TA1) types (i.e., spelling, punctuation, capitalization) C1.3 Identify sentence - parallel structure แล้วนี่ (Nittaya TA1) It’s a parallel structure. (Nittaya TA1) structure

Language Focus > Judgment Strategies Code Category Focus Example text segments (Thai) Example text segments (English) C2.1 Assess - อ๋อ เข้าใจ (Nittaya TA1) Well, I understand. (Nittaya TA1) comprehensibility C2.2 Consider/Assess Sentence ประโยคนี้ยังใช้ไม่ได้ (Nittaya TA1) This sentence is not good. (Nittaya sentence structure or structure or TA1) gramar grammar C2.3 Propose correction Unidentified อันนี้ต้องแก้ไข (Nittaya TA1) This needs to be corrected. (Nittaya types TA1) C2.3.1 Propose correction Vocabulary competition เป็นนาม abstract ใช้ many ‘Competition’ is an abstract noun and /Correct (i.e. word (Nittaya TA1) use ‘many’. (Nittaya TA1) choice, word help authority สิ (Ladda TA2) ‘It should be ‘help authority’. (Ladda form) TA2) C2.3.2 Propose correction/ Sentence เอ่อ ขั้นที่สุดนี่ นักศึกษาจะต้องไม่ลืมว่าต้องมี the Errr, the superlative the student needs Correct structure, นะ (Nittaya TA1) to remember that it needs ‘the’. (Nittaya grammar TA1)

319

320

Code Category Focus Example text segments (Thai) Example text segments (English) C2.3.3 Propose correction/ Mechanics เวลาที่พูดอะไรขึ้นมาที่เป็น series นี่ให้ใส่ When mentioning a series of Correct (i.e. spelling, comma comma แล้วก่อนสุดท้ายให้ใส่ comma something, it needs a comma, a punctuation, and นี่ (Nittaya TA01) comma then put ‘a comma and’. capitalization) (Nittaya TA1) C2.4 Rate language - อืม ตรงนี้ผิด grammar หักไป .25 spelling .25 Mmm, grammar here is incorrect. 0.25 (Rewadee TA2) point is deducted. 0.25 point for spelling. (Rewadee TA2) C2.5 Consider/Assess - Missing เหรอ (Ladda TA2) Missing? (Ladda TA2) vocabulary

Appendix K Sample of coded think-aloud protocols (Ladda, Student 1’s script: Thai)

Text Text segment Code Unit 1 assignment ชิ้นที่ 7 ของ…student’s name… A1.1 2 หน้าแรกเป็น outline A1.4 3 หัวข้อ ID cards for children A1.3 4 Introduction paragraph ใหม่ อันนี้เขาก็จะเล่นแบบ anecdote B1.2 5 thesis statement: maturity, migrant workers and wated money A1.5 6 [Ladda corrects the language: waste of money] A2.8 7 wasted ได้ไงน่ะ นี่มันเป็น verb C1.1 8 เราต้องการเป็น waste of money ดีกว่าไหม เป็นค านาม C2.3.1 9 เพราะว่า A B C นี่ก็คือ B1.5 10 reasons why children should not have ID cards. A1.5 11 major supporting detail A no have enough maturity A1.5 12 minor เขาก็บอกว่า B1.5 13 Children lose their ID cards. Children think ID cards are toys. A1.5 14 ส่วน major B คนนี้ก็บอกว่า B1.5 15 migrant workers A1.5 16 minor ที่จะอธิบาย ก็บอกว่า B1.5 17 state welfare และ try to get hold of the illegal cards A1.5 18 ส่วน major C ที่จะเป็น body ที่สามนี่ เขาขึ้นหัวข้อว่า B1.5 19 waste of money C2.3.2 20 [Ladda corrects the language.] A2.8 21 ก็จะมีเรื่องของ parents แล้วก็ Thai government B1.4 22 แล้วก็ conclude เขาก็จะเอาแบบ summarize B1.2 23 main point...Essay ก็อยู่หน้าต่อไป 24 ตะกี้นี้เป็น outline A1.4 25 ชื่อเรื่อง ID cards for children A1.3

321

Text Text segment Code Unit 26 ย่อหน้าแรกก็เป็น Introductory B1.2* 27 When I was 15 years old, A1.5 28 เขาเขียนว่าไง อ๋อ เขาเขียนแบบ Anecdote B1.2 29 When I was 15 years old, I went …name of a district office… for A1.5 getting my first ID card. However, my younger brother who is seven years old now has to have ID card because of The ID Card Act 2011. In my opinion, maturity, migrant workers and wasted money are the reasons why children shouldn't hold the ID cards. 30 [Ladda writes/marks something on the script.] A2.8 31 เออ ก็โอเคนะ B2.2 32 First, อันนี้ body แล้ว first B1.5 33 อันนี้โอเคๆ มี transition first B2.2, B2.4 34 [Ladda writes/marks something on the script.] A2.8 35 the children don't have enough maturity. A1.5 36 เออ ก็โอเคนะ B2.2 37 เป็น topic sentence มี controlling เป็น maturity B1.3* 38 ไหนดูสิว่าจะขยายความว่าไง A2.1 39 For this reason, they don't know the ID cards are very important. It's A1.5 easy for them to lose ID cards. They can't be trusted to carry their ID cards and not lost them. 40 lost them A1.5 41 -st ได้ไงล่ะ C1.1 42 ต้องเป็น -e สิ present tense C2.3.2 43 [Ladda writes/marks something on the script.] A2.8 44 They may think the ID cards are toy A1.5 45 …s C2.3.2 46 [Ladda writes/marks something on the script while speaking.] A2.8 47 Toy เห็นไหม toy เป็นค านามนับได้ก็ไม่ใส่ –s หรือไม่มี article อีกแล้วนี่ อย่าง C1.1 ใดอย่างหนึ่ง

322

Text Text segment Code Unit 48 อะไรกันนี่ B1.1 49 [Ladda writes/marks something on the script.] A2.8 50 since the cars look like Yo-Gi-Oh cards that are popular for the A1.5 children 51 Yo-Ki-Oh นี่คืออะไรน่ะ ท าไมไม่ขยายความ B1.1 52 What is it? A2.9 53 [Ladda writes What is it? at the end of the line.] A2.8 54 ฉันจะรู้ได้อย่างไรน่ะ ฉันไม่ใช่ children นะ B1.1 55 น่าจะมีการขยายความอธิบายว่า Yo-Ki-Oh cards ว่า ID cards ไป look like B2.8 ไปหน้าตาเหมือนไอ้ Yo-Ki-Oh นี่อย่างไร 56 Consequently, the ID cards are worn out easily because of playing of A1.5 children. 57 บอกว่าไม่มี maturity B1.4 58 [Ladda underlines the word maturity.] A2.8 59 เอ้า ต่อไป ดูสิ ย่อหน้า body ที่ สอง เหตุผลที่สอง ก็คือว่า B1.5 60 มันตรงกันไหมกับ predictor ใน thesis statement A2.1 61 migrant workers A1.5 62 ดูสิ มันเกี่ยวข้องกันอย่างไร A2.1 63 Second, there are many migrant workers in Thailand. They want to get A1.5 state welfare, such as education, healthcare and housing. Therefore, the migrant workers try to get hold of the illegal cards. If the children must have the ID cards, it's not difficult for the young migrant workers to take this opportunity to get the ID cards. 64 อืม ความไม่ค่อยชัดเจนนะ B1.1 65 เพราะถ้าให้เด็กมีบัตรประชาชนแล้วไอ้พวกลูกๆ ของแรงงานอพยพนี่มันจะมา B1.1 เกี่ยวอย่างไรน้า 66 ไม่เห็นชัดเลย B2.2 67 ต่อไป ย่อหน้า body สุดท้าย B1.5

323

Text Text segment Code Unit 68 Finally, ID cards of the children A1.5 89 are waste of money เหรอ C1.1 70 waste a lot of money ไม่ดีเหรอ เป็น verb ไปเลยสิ C2.3.1 71 จะมา are มา waste ได้ไงล่ะ C1.1 72 ก็ ID cards เป็น subject แล้ว แล้ว of ก็เป็นส่วนขยาย ก็เอาเป็น verb แท้ไป C2.3.2 เลย waste a lot of money a lot of เป็นตัวขยาย money 73 [T writes/marks something on the script.] A2.8 74 Parents normally keep children's birth certificates and household A1.5 registration, so the children don't need to have the ID cards to save parents' money. 75 อะไรน่ะ save money อย่างไรอ่ะ ไม่เข้าใจ B1.1 76 ก็ตรงนี้บอกว่า waste a lot of money แล้วตรงนี้มาบอกว่า ถ้ามี birth B1.1 certificate และ house registration นั้น children don't need to have the ID cards แล้วก็ไป save parents' money ตรงไหนน่ะ 77 According to Thairath newspaper, Thai government allocates budget A1.5 for the ID cards of the children 191 million baht. Therefore, the ID cards of the children are very 78 โอ๊ย wasted ได้อย่างไร wasteful ไหมล่ะ ท าเป็น adjective สิ C1.1, C2.3.1 79 ไม่ค่อยชัดเลย idea B2.2 80 In short, there are three reasons A1.5 81 อ้าว ไปสรุปแล้ว B1.5 82 why the children don't need to have the ID cards. One is that children A1.5 have no enough maturity. Another is that it's easy for the young migrant workers to take this opportunity to get the ID cards. The last 83 อืม อย่างไรอ่ะ to get the ID cards How? How? How? B1.1 84 [Ladda writes/marks something on the script.] A2.8 85 ไม่เห็นอธิบายบ้างเลย idea B2.2

324

Text Text segment Code unit 86 ย่อหน้านี้เลย เรื่อง workers น่าจะบอกว่ามัน relate กันอย่างไรกับความ B2.8 คิดเห็นเรื่อง ID cards 87 เพราะว่าพวกเขาจะฉวยโอกาสนี้ท า ID cards ได้อย่างไร ในเมื่อพวกเขาก็ไม่มี B1.1 ใบทะเบียนบ้าน และไม่มีใบเกิดของเด็ก แล้วเขาจะมาสวมรอยได้อย่างไรล่ะ ไม่เข้าใจ 88 เอ่อ แล้วอย่างไรต่อ B1.5 89 Another is that it's easy for the young migrant workers to take this A1.5 opportunity to get the ID cards. The last is that it's useless 90 อะไร useless B1.1 91 it is useless A1.5 92 it is a waste of C2.3.1 93 [Ladda writes/marks something on the script.] A2.8 94 [Ladda turns the page over while humming--making a low noise in A2.8 her throat. The researcher reminds her to keep talking.] 95 ดูภาพรวม น่าจะ… A2.1 96 แต่ว่าเขาใช้ grammar ถูก แล้วก็ใช้ คือเขาไม่มีปัญหา เรื่อง sentence A2.4 structure เลย เขาใช้ตัว transition ได้อย่างเหมาะสม แต่ไปเสียในเรื่องของ เนื้อหา grammar ก็แทบจะไม่มี นิดหน่อย 97 อืม ประมาณว่าให้สัก 7.5 A2.5

325

Appendix K (cont.) Sample of coded think-aloud protocols (Ladda, Student 1’s script: English Translation)

Text Text segment Code Unit 1 Assignment 7 written by…student’s name… A1.1 2 The first page is the outline. A1.4 3 The topic is ID cards for children. A1.3 4 Introduction, new paragraph, this is in a form of anecdote. B1.2 5 thesis statement: maturity, migrant workers and wated money A1.5 6 [Ladda corrects the language: waste of money] A2.8 7 ‘wasted’, How come? It’s a verb. C1.1 8 We need ‘waste of money’. Is it better? C2.3.1 9 It’s a noun. A, B, C are B1.5 10 reasons why children should not have ID cards. A1.5 11 major supporting detail A, no have enough maturity A1.5 12 Minor is B1.5 13 Children lose their ID cards. Children think ID cards are toys. A1.5 14 For major B, it’s B1.5 15 migrant workers A1.5 16 Minor to explain it is B1.5 17 state welfare and try to get hold of the illegal cards A1.5 18 For major C which will be the third part of the text body, he stated B1.5 19 ‘waste of money’ C2.3.2 20 [The phrase ‘wasted money’ was written in the script, so Ladda A2.8 corrects the language by crossing out part of the word ‘wasted’ and adding the word ‘of’.] 21 It is about parents and Thai government. B1.4 22 Then, conclude will be summarise [sic]. B1.2

326

Text Text segment Code Unit 23 main point...Essay is on the next page. A1.4 24 The last part is the outline. A1.4 25 The title is ‘ID cards for children’. A1.3 26 The first paragraph is Introductory. B1.2 27 When I was 15 years old, A1.5 28 What did he say? Well, he wrote in a form of an anecdote. B1.2 29 When I was 15 years old, I went …name of a district office… for A1.5 getting my first ID card. However, my younger brother who is seven years old now has to have ID card because of The ID Card Act 2011. In my opinion, maturity, migrant workers and wasted money are the reasons why children shouldn't hold the ID cards. 30 [Ladda corrects the phrase ‘wasted money’ by crossing -d in the A2.8 word ‘wasted’ and adding the word ‘of’.] 31 Well, OK. B2.2 32 First, this is the body, first. B1.5 33 This is OK; there’s a transition, first. B2.2, B2.4 34 [Ladda underlines the word ‘first’.] A2.8 35 the children don't have enough maturity. A1.5 36 Mmm, OK. B2.2 37 This is a topic sentence; ‘maturity’ is a controlling idea. B1.3* 38 Let’s see how it is modified. A2.1 39 For this reason, they don't know the ID cards are very important. It's A1.5 easy for them to lose ID cards. They can't be trusted to carry their ID cards and not lost them. 40 lost them A1.5 41 -st, how come? C1.1 42 It must be -e, present tense. C2.3.2 43 [Ladda corrects the word ‘lost’ by taking -t out and replacing it with A2.8 -e.]

327

Text Text segment Code Unit 44 They may think the ID cards are toy A1.5 45 -Sss C2.3.2 46 [Ladda adds -s to the word ‘toy’.] A2.8 47 Toy, see that? Toy is a countable noun. Why isn’t there -s or an article, C1.1 either of them? 48 What is this? B1.1 49 [Ladda writes ‘a’ to identify an article error.] A2.8 50 since the cars look like Yo-Gi-Oh cards that are popular for the A1.5 children 51 What is Yo-Gi-Oh? Why isn’t it modified? B1.1 52 What is it? A2.9 53 [Ladda circles the word ‘Yo-Gi-Oh’ and writes a question ‘what is it?’ A2.8 at the margin.] 54 How can I know? I’m not a child. B1.1 55 There should be an explanation of what Yo-Gi-Oh cards are and how B2.8 ID cards look like the Yo-Gi-Oh cards. 56 Consequently, the ID cards are worn out easily because of playing of A1.5 children. 57 There’s no ‘maturity’. B1.4 58 [Ladda underlines the word maturity.] A2.8 59 Well, next. Let’s see. The second paragraph of the body. The second B1.5 reason. 60 Is it the same as the predictor in the thesis statement. A2.1 61 migrant workers A1.5 62 Let’s see. How are they related? A2.1

328

Text Text segment Code Unit 63 Second, there are many migrant workers in Thailand. They want to get A1.5 state welfare, such as education, healthcare and housing. Therefore, the migrant workers try to get hold of the illegal cards. If the children must have the ID cards, it's not difficult for the young migrant workers to take this opportunity to get the ID cards. 64 Mmm, not quite clear. B1.1 65 If students have an ID card, how can migrant workers’ children relate B1.1 to this issue? 66 Not clear. B2.2 67 Next, the last paragraph. B1.5 68 Finally, ID cards of the children A1.5 69 are waste of money ? C1.1 70 Is it ‘waste a lot of money’ better? Change it into a verb. C2.3.1 71 Why using ‘are’ and ‘ waste’? C1.1 72 ID cards are the subject. ‘Of’ is a modifier. So place a verb here. Use C2.3.2 ‘waste a lot of money’. ‘A lot of’ modifies ‘money’. 73 [Ladda writes/marks something on the script.] A2.8 74 Parents normally keep children's birth certificates and household A1.5 registration, so the children don't need to have the ID cards to save parents' money. 75 What? ‘Save money’? How? I don’t get it. B1.1 76 Here it says ‘waste a lot of money’ and it says ‘if there are a birth B1.1 certificate and house registration, children don't need to have the ID cards and how does it save parents' money? 77 According to Thairath newspaper, Thai government allocates budget A1.5 for the ID cards of the children 191 million baht. Therefore, the ID cards of the children are very 78 Oh, wasted, how come? Is it ‘ wasteful’? Use an adjective. C1.1, C2.3.1

329

Text Text segment Code unit 79 Ideas are not clear. B2.2 80 In short, there are three reasons A1.5 81 On, here comes the summary. B1.5 82 why the children don't need to have the ID cards. One is that children A1.5 have no enough maturity. Another is that it's easy for the young migrant workers to take this opportunity to get the ID cards. The last 83 Mmm, How? ‘To get the ID cards’ How? How? How? B1.1 84 [Ladda writes/marks something on the script.] A2.8 85 There’s no explanation of ideas. B2.2 86 This paragraph is about ‘workers’. It should say how they relate to ID B2.8 cards? 87 How can they take advantage to have the ID cards issued? They B1.1 don’t have a house ceritficate and a birth certificate. So how can they have the ID cards issued? I don’t get it. 88 Er, so what? B1.5 89 Another is that it's easy for the young migrant workers to take this A1.5 opportunity to get the ID cards. The last is that it's useless 90 What is useless? B1.1 91 it is useless A1.5 92 it is a waste of C2.3.1 93 [Ladda writes/marks something on the script.] A2.8 94 [Ladda turns the page over while humming--making a low noise in A2.8 her throat. The researcher reminds her to keep talking.] 95 Overall, it should … A2.1 96 But he used correct grammar. He doesn’t have any problem about A2.4 sentence structure. He used appropriate transitions. But the content is problematic. Grammar is little problematic. 97 Mmm,7.5 then. A2.5

330

Appendix K (cont.) Sample of coded think-aloud protocols (Rewadee, Student 1’s script: Thai)

Text Text segment Code Unit 1 ตอนนี้เราจะตรวจงานเขียนนะคะ วิชา paragraph writing เอ่อ ชิ้นงานของ A1.1 นักศึกษาชื่อ…student’s name… เป็นเด็ก major นะคะ ชั้นปีสอง 2 เอ่อ ให้เขียนเปรียบเทียบ ของสองสิ่งนะคะ หรืออะไรสองอย่าง A1.2 3 เด็กคนนี้เขียนหัวข้อ living in country and living in city นะคะ อันนี้คือ A1.3 topic นะ 4 topic sentence ดู topic sentence ก่อนเลย A2.1 5 Living in country is often very different from living in city. A1.5 6 topic sentence โอเค B2.2 7 บอกว่าอยู่สองที่แตกต่างจากกัน B1.4 8 When you live in the country A1.5 9 ต้องเริ่มจาก country ก่อนนะคะ B1.5 10 มี transition ชัดเจน B2.4 11 [Rewadee underlines a word.] A2.8 12 You will have many good and friendly neighbors A1.5 13 เพราะฉะนั้น aspect แรกที่นิสิตเสนอคือเรื่อง neighbors นะคะ B1.5 14 เอ่อ you will have many good and friendly neighbors and you can A1.5 create relationship with them 15 create relationship A1.5 16 ให้ข้อมูลเพิ่มเติมว่า B1.5 17 When you have a problem, you can confer with your neighbors, too. A1.5 18 [Rewadee underlines the sentence.] A2.8 19 ตรงนี้สะดุดนิดหนึ่ง แสดงว่า เด็กๆ เอ่อ นิสิตคนนี้ใช้ค าว่า confer C1.1, A1.8 20 จริงๆ แล้วค าที่เจอบ่อยกว่าคือค าว่า consult ที่แปลว่าปรึกษาหารือ C2.3.1

331

Text Text segment Code Unit 21 คิดว่าอันนี้เด็กๆ น่าจะคิดจากภาษาไทยก่อนแล้ว คิดว่าน่าจะไปหาใน A2.7 ดิกชันนารี ปรึกษาอะไรอย่างนี้น่ะค่ะ แล้วดิกชันนารีภาษาไทยเป็นอังกฤษ มันน่าจะขึ้นค าว่า consult มา ทั้งๆ ที่จริงๆ แล้ว ค าว่า consult เด็กๆ น่าจะ เจอบ่อยกว่า แต่ค าว่า confer มันก็ได้อยู่นะ แปลว่าปรึกษาเหมือนกัน 22 you can confer with your neighbors, too A1.5 23 ไอเดียเข้าใจดีนะคะ B2.2 24 In a part of the nature, A1.5 25 อันนี้น่าจะมาจากภาษาไทย A2.7 26 ที่บอกว่าในส่วนของธรรมชาติ ก็เขียนมาตรงตัวเลย B1.4 27 In a part of nature, you will live there among the pure and clean air A1.5 from the nature of the country. 28 อ่า aspect ที่สองคือพูดถึง nature B1.5 29 In addition A1.5 30 In addition A1.5 31 spelling ผิดนะคะ มี d เกินมาตัวหนึ่ง C1.1, C1.2.3 32 [Rewadee writes/marks something on the script.] A2.8 33 the nature of the country can make you be a strong health man A1.5 34 อันนี้ก็น่าจะคิดเป็นภาษาไทยเหมือนกัน A2.7 35 ธรรมชาติที่ดีจะท าให้คุณเป็นคนที่มีร่างกายแข็งแรง B1.4 36 strong health man A1.5 37 เข้าใจ C2.1 38 แต่มันเหมือนไม่ใช่ภาษาอังกฤษ C1.1 39 The nature of the country can make your health strong น่าจะดีกว่า C2.3.2 40 [Rewadee writes/marks something on the script.] A2.8 41 In transport, A1.5 42 ตรงนี้ไม่น่าจะเป็น in C1.1 43 [Rewadee writes/marks something on the script.] A2.8

332

Text Text segment Code Unit 44 น่าจะเป็น for C2.3.2 45 For transport, you can go everywhere conveniently A1.5 46 มาจากค าว่าอย่างสะดวก B1.4 47 because the street in the country have a few car A1.5 48 the street A1.5 49 ไม่เติม s แต่ใช้ have C2.3.2 50 [Rewadee writes/marks something on the script.] A2.8 51 อันนี้มีปัญหาเรื่อง subject verb agreement เพราะประธานเป็นเอกพจน์ C1.2.2 แต่ใช้ค ากริยาพหูพจน์ 52 [Rewadee writes/marks something on the script.] A2.8 53 However, when you live in the city A1.5 54 อันนี้มี transition ถ่ายโอนไอเดียมา B2.4 55 ตอนนี้เด็กๆ เอ่อ นิสิตจะพูดถึงการอยู่ในเมือง B1.4 56 when you live in the city, it is different experience A1.5 57 ก็มีประโยคเปิดขึ้นมา ใช้ได้เลยนะคะ B2.2 58 you will not have a real close friends and close neighbors A1.5 59 อืม aspect แรก พูดถึง neighbors นะคะ B1.5 60 ก็บอกว่าอยู่ในเมืองเนี่ย จะไม่ค่อยมีเพื่อนแท้หรือเพื่อนบ้านที่แท้จริง B1.4 61 You might meet the unfriendly neighbors, and it might make you A1.5 cannot confer with them. When you want to go somewhere, you might feel very bore. 62 bore เติม d C2.3.2 63 [Rewadee writes something down.] A2.8 64 because the traffic in the city is very bad A1.5 65 พูดถึง transport นะคะ B1.5 66 เมื่อกี้ใน country เขาบอกว่าใน country transport ค่อนข้างจะ convenient B1.4 ตอนนี้เขาก็เปรียบเทียบว่า ถ้าอยู่ใน city การจราจรจะค่อนข้างติดขัด 67 About the air, you will got the bad air from many factory, A1.5

333

Text Text segment Code Unit 68 อ่า many factory แต่ไม่เติม s C1.1 69 [Rewadee writes a comment.] A2.8 70 อันนี้ plural form C1.2.2 71 It can make you be a bad health man A1.5 72 [Rewadee writes something down.] A2.8 73 It can make your health bad. Although living in the country and living A1.5 in the city might give the same residence 74 อันนี้คิดว่าจะเป็นประโยค conclude ของเขานะคะ B1.5 75 ถึงแม้ว่าการอยู่ในเมืองและการอยู่ในชนบทจะมีที่พักอาศัยที่เหมือนกัน B1.4 76 living in the country and living in the city are give you the different A1.5 experience of life 77 อืม ไอเดียในการ conclude paragraph ใช้ได้ดีทีเดียว B2.2 78 แต่ผิดตรงที่ว่าคือพูดซ ้า B2.5 79 เพราะเมื่อกี้พูดไปแล้วว่า B2.5 80 although living in country and living in the city A1.5 81 พูดไปแล้ว เป็นประธานของประโยคไปแล้ว แต่ประโยคติดกันก็ยังใช้ B2.5 82 living in the country and living in the city are give you the different A1.5 experience of life 83 ใช้ติดกันสองประโยคเลย C1.1 84 ตรงนี้น่าจะใช้ pronoun แทนค านามที่กล่าวไปแล้วว่าที่อยู่ที่กล่าวไปแล้วเพื่อ C2.3.1, หลีกเลี่ยงการใช้ค าซ ้าๆ โอเค 85 [Rewadee writes a comment.] A2.8 86 ทีนี้มาให้คะแนนกัน เริ่มให้คะแนน A2.1 87 นิสิตคนนี้ค่อนข้างจะ organise idea ได้ดี มีการ เรื่อง coherence ใช้ได้ A2.4 เพราะว่ามีการใช้ transition ในการที่จะ move idea เดิมไปสู่ idea ใหม่ได้ดี มีปัญหาแค่เล็กๆ น้อยๆ แค่เรื่อง spelling นิดหน่อย พวกประโยคที่แปลกๆ บ้าง 88 spelling นิดเดียวนี่คือจุดอ่อน ตรงนี้ลบ .5 C2.4 89 subject verb agreement ลบไป .5 นิดหน่อยอันนี้ plural C2.4

334

Text Text segment Code Unit 90 bore ลบ .25 C2.4 91 many factories ลบไป .5 C2.4 92 make your health bad A1.5 93 [Rewadee writes/marks something on the script.] A2.8 94 สรุปแล้ว ลบไปประมาณ 2.5 A2.6 95 [Rewadee counts the deducted marks.] A2.8 96 เด็กคนนี้น่าจะได้ประมาณ 18 คะแนนค่ะของงานชิ้นนี้ A2.5 97 คือรวมๆ แล้ว อย่างที่บอกว่า grammar ส่วนใหญ่ค่อนข้างถูก และเห็นภาพ A2.4 ชัดเจนว่าเปรียบเทียบด้านไหนบ้าง แล้วก็คือไม่หยุด เมื่อพูดถึง country ข้างบนก่อน แล้วพูดถึงสามด้าน การอาศัยอยู่ในชนบทสามด้าน ด้านบน ก่อน จากนั้นพูดถึงการอยู่ในเมืองคือ aspect เดิม ก็ท าให้ติดตามได้ง่ายแล้ว ก็เข้าใจได้ง่าย ท าให้ไม่สับสน 98 คะแนนจึงอยู่ที่ 18 A2.5

335

Appendix K (cont.) Sample of coded think-aloud protocols (Rewadee, Student 1’s script: English translation)

Text Text segment Code Unit 1 Now I will mark the writing in Paragraph Writing course. This script A1.1 belongs to …student’s name… 2 Er, I asked the students to compare two things. A1.2 3 This student wrote in the topic of ‘living in country and living in city’. A1.3 4 Topic sentence, check the topic sentence first. A2.1 5 Living in country is often very different from living in city. A1.5 6 The topic sentence is OK. B2.2 7 It says living in two different places. B1.4 8 When you live in the country A1.5 9 Start from ‘country’. B1.5 10 There’s a clear transition. B2.4 11 [Rewadee underlines a word.] A2.8 12 You will have many good and friendly neighbors. A1.5 13 Thus, the first aspect of presentation is ‘neighbors’. B1.5 14 Er, you will have many good and friendly neighbors and you can A1.5 create relationship with them 15 create relationship A1.5 16 Giving more information, B1.5 17 When you have a problem, you can confer with your neighbors, too. A1.5 18 [Rewadee underlines the sentence.] A2.8 19 It’s not smooth here. This means, er, this student uses the word C1.1, A1.8 ‘confer’. 20 In fact, the most frequent word is ‘consult’ which means [Rewadee C2.3.1 says the word meaning in Thai.]

336

Text Text segment Code Unit 21 I think the student may think in Thai and look up the word in A2.7 dictionary. The Thai-English dictionary then shows the word ‘consult’ [Rewadee seems to intend to say ‘confer’ here.] In fact, ‘consult’ is the most frequent word for the student. But the word ‘confer’ is fine. It means [Rewadee says the word meaning in Thai.] 22 you can confer with your neighbors, too A1.5 23 The idea is fine. B2.2 24 In a part of the nature, A1.5 25 This may be influenced by Thai language. A2.7 26 Which says [Rewadee says the thai translation of ‘in the nature’.] B1.4 Direct translation. 27 In a part of nature, you will live there among the pure and clean air A1.5 from the nature of the country. 28 The second aspect is ‘nature’. B1.5 29 In addition A1.5 30 In addition A1.5 31 Wrong spelling; there’s an extra ‘d’. C1.1, C1.2.3 32 [Rewadee writes/marks something on the script.] A2.8 33 the nature of the country can make you be a strong health man A1.5 34 This one seems to be influenced by Thai language. A2.7 35 [Rewadee translates the sentence into Thai.] B1.4 36 strong health man A1.5 37 I understand. C2.1 38 But it’s not English. C1.1 39 ‘The nature of the country can make your health strong’ must be C2.3.2 better. 40 [Rewadee writes/marks something on the script.] A2.8

337

Text Text segment Code Unit 41 In transport, A1.5 42 It should not be ‘in’ here. C1.1 43 [Rewadee writes/marks something on the script.] A2.8 44 It should be ‘for’. C2.3.2 45 For transport, you can go everywhere conveniently A1.5 46 It’s from the word ‘convenient’. B1.4 47 because the street in the country have a few car A1.5 48 the street A1.5 49 No ‘s’ but there’s ‘have’ C2.3.2, 50 [Rewadee writes/marks something on the script.] A2.8 51 This one has a ‘subject verb agreement’ problem because the C1.2.2 subject is singular but the verb is plural. 52 [Rewadee writes/marks something on the script.] A2.8 53 However, when you live in the city A1.5 54 This one has a transition to transit the idea. B2.4 55 Now the student talks about living in the city. B1.4 56 when you live in the city, it is different experience A1.5 57 There’s an opening sentence. It’s OK. B2.2 58 you will not have a real close friends and close neighbors A1.5 59 Mmm, the first aspect is ‘neighbors’. B1.5 60 It says [Rewadee translates the sentence above.] B1.4 61 You might meet the unfriendly neighbors, and it might make you A1.5 cannot confer with them. When you want to go somewhere, you might feel very bore. 62 ‘Bore’ needs ‘d’. C2.3.2, 63 [Rewadee writes something down.] A2.8 64 because the traffic in the city is very bad A1.5 65 It’s about ‘transport’. B1.5

338

Text Text segment Code Unit 66 It was earlier about ‘country’. The student says ‘country transport’ is B1.4 quite convenient. Now she makes a comparison that in the city the traffic is hectic. 67 About the air, you will got the bad air from many factory, A1.5 68 ‘Many factory’ but there’s no ‘s’. C1.1 69 [Rewadee writes a comment.] A2.8 70 This is a plural form. C1.2.2 71 It can make you be a bad health man A1.5 72 [Rewadee writes something down.] A2.8 73 It can make your health bad. Although living in the country and living A1.5 in the city might give the same residence 74 I think this is the concluding sentence. B1.5 75 [Rewadee translates the sentences into Thai.] B1.4 76 living in the country and living in the city are give you the different A1.5 experience of life 77 Mmm, the idea in the concluding sentence is good. B2.2 78 But it’s wrong because it’s repetitive. B2.5 79 She said earlier, B2.5 80 although living in country and living in the city A1.5 81 It’s been said; it’s the previous sentence subject but it’s still used in B2.5 the sequential sentence. 82 living in the country and living in the city are give you the different A1.5 experience of life 83 It’s in the two consecutive sentences. C1.1 84 It should be a pronoun instead of the noun that has already been C2.3.1 mentioned to avoid repetition. 85 [Rewadee writes a comment.] A2.8 86 Now let me score. A2.1

339

Text Text segment Code Unit 87 This student organise ideas quite well. Coherence is OK because A2.4 there’s the use of transitions when moving from one idea to another. There are a few problems with spelling and weird sentences. 88 A weakness is spelling. 0.5 is deducted for this point. C2.4 89 0.5 is deducted for subject verb agreement. C2.4 90 0.25 is deducted for ‘bore’. C2.4 91 0.5 is deducted for ‘many factories’. C2.4 92 make your health bad A1.5 93 [Rewadee writes/marks something on the script.] A2.8 94 To sum up, about 2.5 marked are deducted. A2.6 95 [Rewadee counts the deducted score.] A2.8 96 This student gets about 18 marks for this piece of writing. A2.5 97 Overall, as I said most grammar is correct. I can have a clear picture A2.4 of what are compared and contrasted. And the ideas are connected. When discussing ‘country’, the three aspects are discussed, living in the country in the three aspects. Then, he moves to talk about living in the city in the same aspects. This makes the writing easy to follow and easy to understand. It’s not confusing. 98 The score is 18, then. A2.5

340

2 Coder agreement Intercoder agreement at Strategy Level.csv (i.e. A1, A2, B1, B2, C1, C2) filesize 2432 bytes n columns 2 n variables 1 n coders per var 2 Inter

- coderpercentage agreement thinkof Percent Scott's Pi Cohen's Krippendorff' N N N N Agreement Kappa s Alpha Agreements Disagreement Cases Decisions s Variable 1 (cols 1 & 2) 65.2284264 -0.04878 0.00916 -0.04745 257 137 394 788

AppendixL

Intercoder agreement at Focus Level.csv (i.e. A, B, C) filesize 2038 bytes n columns 2 n variables 1

- n coders per var 2 aloudprotocols

Percent Scott's Pi Cohen's Krippendorff' N N N N Agreement Kappa s Alpha Agreements Disagreement Cases Decisions s Variable 1 (cols 1 & 76.3959390 2) 9 0.045769 0.061256 0.04698 301 93 394 788

341

342

3 Coder agreement

Intercoder agreement at Strategy Level.csv (i.e., A1, A2, B1, B2, C1, C2) filesize 2865 bytes n coders 3 n cases 394 n decisions 1182 average pairwise percent agreement 72.93% pairwise agreement cols 1 & 3 0.812182741 pairwise agreement cols 1 & 2 0.652284264 pairwise agreement cols 2 & 3 0.723350254 fleiss' kappa 0.193858456 FK observed agreement 0.72927242 FK expected agreement 0.664168678 average pairwise cohen's kappa 0.213111611 pairwise CK cols 1 & 3 0.327350329 pairwise CK cols 1 & 2 0.009159829 pairwise CK cols 2 & 3 0.302824675 krippendorff's alpha 0.194540471 Î£cocc 862 Î£cnc(nc - 1) 926744

Intercoder agreement at Focus Level.csv (i.e., A, B, C) filesize 2865 bytes n coders 3 n cases 394 n decisions 1182 average pairwise percent agreement 81.13% pairwise agreement cols 1 & 3 0.860406091 pairwise agreement cols 1 & 2 0.763959391 pairwise agreement cols 2 & 3 0.80964467 fleiss' kappa 0.258886246 FK observed agreement 0.811336717 FK expected agreement 0.74543276 average pairwise cohen's kappa 0.263592675 pairwise CK cols 1 & 3 0.403441156 pairwise CK cols 1 & 2 0.06125586 pairwise CK cols 2 & 3 0.326081007 krippendorff's alpha 0.259513246 Î£cocc 959 Î£cnc(nc - 1) 1040280

342

Appendix M Sample excerpts of Ladda’s interview data (Thai)

RESEARCHER: ปกติแล้วพี่สอนกี่ชั่วโมงต่อสัปดาห์คะ LADDA: ปกติสอนสามชั่วโมงต่อสัปดาห์ คอร์สเดียวนะ ถ้ามีสองคอร์ส Intro to academic writing ก็สาม แล้วก็ Advanced writing ก็สาม ก็จะเป็น 6 ชั่วโมง RESEARCHER: แต่ว่าเทอมนี้นี่สอนวิชาเดียว LADDA: สอง ทั้ง advanced ทั้ง intro RESEARCHER: ก็คือสอน writing อยู่หกชั่วโมง มีสอนวิชาอื่นด้วยไหมคะ LADDA: ไม่มีแล้ว แต่พี่ไปรับจ๊อบข้างนอก ของคณะแพทย์ก็มี สัปดาห์นึงก็เท่าไหร่ล่ะ โห 10 กว่า ชั่วโมงเลยแหละ เฉพาะ writing ที่คณะแพทย์สอง เภสัชสัปดาห์ละสี่ สี่บวกสองเป็นหก บวกที่นี่อีก สาม สาม ก็ 12 ชั่วโมง RESEARCHER: แล้ว writing ระดับที่สูงที่สุดที่พี่สอนนี่คืออะไรคะ LADDA: ก็ Advanced ไง RESEARCHER: เป็น writing ระดับ paragraph หรือ essay คะ LADDA: เป็น essay เข้าสู่ essay RESEARCHER: essay นี่คือ 5 paragraphs LADDA: ใช่ RESEARCHER: แล้วเด็กของพี่ในวิชานี้เป็นอย่างไรบ้างคะ LADDA: ไม่ได้เรื่อง เบื่อ ไวยากรณ์อ่อนแอมาก เขียนก็ไม่ถูก ขนาดว่าเราปูพื้นให้แล้ว และเขาก็ เรียนจบหลักสูตรของคณะศิลปศาสตร์ ซึ่งเขาก็ต้องผ่านวิชาบังคับเลือกอีกตัวหนึ่ง ซึ่งเขาก็ผ่าน วิชา Introduction to writing มาแล้ว ก็ยังใช้...แต่บางคนก็โอเค เห็นการพัฒนา ไม่ใช่ว่าทั้ง 40, 50, 70 คนที่เราสอนนี่แย่หมดก็เป็นไปไม่ได้ แต่เปอร์เซ็นต์ของคนที่เขียนได้ดีจริงๆ ยังมีน้อย ก็ยังมี ปัญหา แต่คนที่เขียนเก่งๆ ก็โอเค เขาก็มีการพัฒนามากขึ้นจากระดับเขียนไม่เป็นเลย ก็มาเป็น ประโยค พอจากประโยคสั้นๆ ก็เริ่มมาเป็นย่อหน้า จากย่อหน้าสั้นๆ เขาก็สามารถแตกเขียน ออกมาให้ได้ 5 ย่อหน้า เขาเองก็ยังรู้สึกมหัศจรรย์ในตัวเองว่าเขาท าได้อย่างไร เพราะก่อนหน้านี้ เขาท าไม่ได้ เขาคิดว่าเขาท าไม่ได้ แต่พอเรียนไป ฝึกไป เรียนไป ฝึกไป ผิดบ้าง ถูกบ้าง ก็ล้มลุก คลุกคลานกันมาจนกระทั่งเขาข้ามมาเรียน advanced ก็มาสามารถ RESEARCHER: ก็ท าได้ LADDA: ก็ท าได้ แต่ก็ยังมีปัญหา คือ แล้วทั้งๆ ที่เขาเรียน grammar ต่างหากเลยนะ ซึ่งเป็นวิชา บังคับเหมือนกัน ผ่านมาแล้วสองตัว

343

RESEARCHER: คือ grammar 1 grammar 2 LADDA: ใช่ เขาก็ยังมีปัญหาในการเขียน โดยเฉพาะ คือ มันมาผิดในสิ่งที่ไม่น่าผิด อย่างเช่นว่า ประธานเป็นเอกพจน์ he, she, it อะไรอย่างนี้ verb ถ้าเป็น present tense เขาก็ไม่เติม s อะไร อย่างนี้ แล้วอีกอย่างหนึ่งเวลาเราพูดถึง plural noun ก็ต้องเติม s เวลาเป็น countable noun เช่น many books, a lot of students ก็ไม่มี s เนี่ย แล้วก็ ที่น่าเกลียดก็เริ่มน่าเกลียด คือมันเริ่มตั้งแต่ เล็กๆ อย่างนี้ก็มี a, an, the นี่ไม่ต้องพูดเลย ประจ า ไม่ค่อยมี article เลย a,an, the ไม่ค่อยมี แล้วก็ข้ามไปจนถึงระดับที่ว่า part of speech ก็ยังใช้ไม่เป็น คือ เขาหันไปใช้ talking dict ไง ซึ่งพี่ ก็ไม่รู้นะ เป็นเครื่องเล็กๆ คล้ายๆ เครื่องคิดเลข แล้วก็จิ้มคาภาษาไทยลงไป แล้วมันก็พรึ่บขึ้นมา เป็นค าภาษาอังกฤษที่เขาต้องการ แต่เขาก็ไม่ได้สนใจ part of speech เลย แล้วเขาก็จะแต่ง ประโยค คือเขาแค่จับว่าเขาแค่ต้องการค าว่า ขยาย เขาก็ท าลงไปในเครื่องของเขา แล้วเขาก็จะขึ้น ค าว่า expansion ซึ่งเป็นค านาม แล้วเขาก็มาผูกประโยคเลยว่า เอ่อ My mother wants to expansion her business to foreign country. ซึ่งมันผิดไง เพราะ expansion มันเป็นค านาม แล้วก็ We hope to get... เอ่อ...We hope to success อย่างนี้ คือเอาค านามมาเป็น verb หรือ อีกคนหนึ่ง ก็ เอ่อ People who live in this social ก็ social มันเป็น adjective เธอเอามาท าเป็น ค านามได้ยังไง society สิ ประจ าน่ะ อย่างนี้ ระดับ part of specch ใช่ไหม แล้วก็ระดับประโยคที่ เป็น fragment ประจ าเลย มากที่สุดเลย ในความผิดสามสี่ mistakes เนี่ยนะ ซึ่งเขา พวกคนที่เขา วิจัยนี่ เขาจะแบ่ง การเขียนผิดนี่ แบ่งออกเป็น run-on, fragment, แล้วก็ comma splice แต่ทีนี้ เด็กไทยน่ะ ที่ท ามากๆ ก็คือ comma splice กับ เอ่อ run-on ไม่ค่อยเท่าไหร่นะ กับ fragment ไม่มี ประจ า คือเข้าใจแหละ มันมีอิทธิพลของภาษาไทยเข้าไปเกี่ยวข้องด้วย เพราะในภาษาไทยเราบาง ทีเราพูดก็ลอยๆ ไม่มีประธาน ละเอาไว้ในฐานที่เข้าใจ เช่น จะจัดงานพิธีพระราชทานเพลิงศพ แล้ว ใครจะจัดน่ะ แล้วมันก็เขียนเป็นภาษาอังกฤษว่า will ขึ้นเลย ตรงภาษาไทยอย่างนี้ แล้วมันไม่มี ประธาน หรือว่าถ้าเวลามันใช้ complex sentence เช่น adjective clause อะไรอย่างนี้ มันก็จะ หลุด เพราะมันมีตัวขยายเยอะ an ambulance which arrived this morning อะไรของมันก็ไม่รู้ แล้วมันก็ full stop ไปแล้ว โดยที่มันไม่มี main verb อะไรอย่างนี้ มันก็กลายเป็น fragment ประ จ าเลย แล้วพูดตั้งไม่รู้กี่ครั้งแล้ว ก็ ก็แก้ไม่หายน่ะ มันก็ยังจะเป็นอยู่ พี่ก็ไม่รู้ ไม่เข้าใจเหมือนกันว่า เกิดอะไรขึ้นในสมองมันน่ะ ท าไมมันเรียนไม่รู้เรื่อง แล้วเราเรียน นะ โอ้โห grammar มันเรียนอย่าง ละเอียดเลยนะ grammar book ตั้งสองคอร์ส เล่มหนาเตอะเลยนะ มันก็ยังไม่เห็น get อะไรเลย โดยเฉพาะ noun clause ใช้ไม่เป็น แต่งไม่เป็น เพราะมันงงมากเลย ว่า noun clause ก็หน้าที่ เหมือนกับ noun ก็คือเป็น subject ก็ได้ เป็น object ก็ได้ แต่ถ้าเป็น object นี่มันพอไหว เพราะว่า มันตามหลัง verb ใช่ไหม We hope that อะไรๆ ก็แต่งเป็นประโยคมา แต่พอจะเอามา เอาเป็น

344

ประธาน เป็น subject ของประโยคงี้ มันเขียนไม่ได้แล้ว what the...what the Prime Minister promised is not true. อะไรอย่างงี้ มันท าไม่ได้แล้ว มันคิดว่าเป็นประโยคแล้วไง มันยังสับสนอยู่ ระหว่าง adjective clause มันไปสับสนกับ noun clause แล้วก็ adjective clause, noun clause แล้วก็ไปสับสนกับ subordinating conjunction มั่วไปหมดเลยน่ะ พี่ก็เลย อุ้ย อะไรเนี่ย ไปๆ มาๆ มันจะท าให้ฉันงงเหมือนกันนะ คนสอนนี่ (หัวเราะ) RESEARCHER: ก็เลยสับสนกันไปใหญ่ LADDA: อืม แล้วเขา เขาก็ไม่ถนัดใช้พวก transition เชื่อมประโยค พี่สังเกต เพราะว่าเวลา ตัวเชื่อมประโยคมีสี่ เอ่อ สาม major ใหญ่ วิธีแรกที่เราจะไม่ค่อยค านึงถึง เพราะเราก็แค่ separate แล้วก็ใส่ full stop เข้าไป ทีนี้ถ้าใช้เป็น conjuction อย่างพวก coordinating conjunction ที่ท าให้ เกิด compound sentence เนี่ย มันเป็นระดับนั้นมันง่ายไง ก็เอา fanboy มาเชื่อมไง for, and, but, also,or, yet ใช่ไหม อันนั้นมัน พอระดับนั้นมันพอเป็น แต่พอเริ่มเป็น complex sentence ใช้ subordinating conjunction มันก็เริ่มสับสน และอย่าไปพูดถึง transition เลย ที่เพราะๆ additionally ใช้ไม่เป็น similarly ใช้ไม่เป็น เอ่อ as a result, consequently นี่ ใช้ไม่เป็น ทั้งๆ ที่เรา บอกสูตรไปแล้วนะ มันไม่ค่อยกล้าใช้ แล้วที่แย่ที่สุด คือ มันใช้ preposition มาเชื่อมประโยคไม่ เป็น due to, because of มันก็ยังตามด้วยประโยค ทั้งๆ ที่ of o-f of นี่โดย common sense มันก็ คือ preposition แล้วสิ่งที่ตามหลัง preposition นี่ เราก็รู้ ๆ กันอยู่ ถ้าไม่เป็นค านาม ก็ต้องเป็น gerund แล้วมันก็ลอกเป็นประโยคมา แล้วพออธิบาย มันก็ อ๋อ อ๋อ แล้วท าไมไม่ปฏิบัติน่ะ พี่ไม่ เข้าใจ คือมันแยกน่ะ แยกระหว่างการเรียนไวยากรณ์กับการใช้จริง เรามา apply จริงๆ มันท าไม่ เป็นอ่ะ เอามาผสม เอามา integrate แล้วใช้ไม่เป็นน่ะ ก็ไม่เข้าใจน่ะว่าสมองมันเป็นอะไร หรือว่า การสอนของเราล้มเหลวมาตั้งแต่มัธยม การสอนภาษาอังกฤษของครูภาษาอังกฤษในประเทศ นี้น่ะ มันล้มเหลวมาตั้งแต่เริ่มต้นแล้วน่ะ แล้วพอมาถึงมือเราซึ่งเป็นขั้น advanced ขั้นที่ต้องเขียน ได้แล้ว ต้อง produce ได้แล้ว ไม่ใช่ต้องให้เรามานั่งสอน a, b, c ใหม่น่ะ มันไม่ใช่ แต่ว่าในความ เป็นจริง เราต้องท าเช่นนั้น เราต้องกลับไปที่ 0 ใหม่ ซึ่งเสียเวลาและเหนื่อยมากๆ ขอบอก RESEARCHER: อันนี้คือท าในคอร์ส writing นี้เลยใช่ไหมคะ LADDA: อืม เพราะพี่มันต้องอัด เพราะไม่งั้นมันก็เขียนไม่ได้อีกแหละ พี่ต้องมาทบทวน ต้องแทรก ตลอด ว่าต้องเขียนอะไร อย่างไรไง นั่นคือความผิดที่น่าเบื่อ จนไม่รู้จะต้องท ายังไงแล้วเหมือนกัน แต่คนที่เขียนเก่งก็เขียนดีนะ รู้จักใช้ แล้วภาษาก็สละสลวยมากเลย แล้วอีกอันหนึ่งก็คือ เรื่องของ idea ไม่ค่อย unity พอเขียนๆ แล้วก็ชอบออกนอกเรื่อง irrelevant sentence เยอะแยะไปหมด เพราะมันคิดว่าน่าจะไปด้วยกัน แต่มันไม่ใช่ไง มันไม่ได้ relate กับหัวข้อที่เขียน อย่างเช่น ให้เขียน ว่าท าไมยูถึงชอบไปกินอาหารร้านนี้ ให้ยกเหตุผลว่าอะไรเป็นแรงจูงใจที่ท าให้ยูติดอกติดใจที่จะมา

345

ใช้บริการอาหารร้านนี้ แล้วก็สอนแล้วนะว่า เออ เวลาเขียนนี่เราจะต้องคิด supporting detail ให้ ได้สาม สามประเด็นใหญ่ๆ ใช่ไหม ก็ช่วยกันคิด เช่น ท าไมถึงชอบ เอ้า หนึ่ง การบริการ โอเค แล้ว ภายใต้ heading การบริการ มีอะไรบ้าง ก็ list ออกมา ถูกไหม แต่ตอนแรกเนี่ย จะไม่บอกก่อน ก็ ให้ heading เฉยๆ แล้วให้ไปคิดเอาเอง โอ๊ย ออกมาสารพัดรูปแบบ service นี่มันจะบอกได้ไงว่า part ของ service นี่มันไปพูด เออ เขียนนะว่า บริการดี ยิ้มแย้มแจ่มใส อาหารสั่งปุ๊บได้ปั๊บ ไม่ต้อง รอนาน เออ โอเค อันนี้รับได้ยังอยู่ในหัวข้อการบริการ แต่มันไปบอกอีกประโยคหนึ่งซึ่งอยู่ภายใต้ หัวข้อนี้ว่าห้องน ้าก็สะอาดสะอ้าน หรือว่าพี่กับมันนี่เข้าใจกันคนละเรื่องว่าค าว่า service นี่ ครอบคลุมอะไรบ้าง อ๋อ ห้องน ้าสะอาด น ้าไหลดี อะไรอย่างนี้ เอ๊ะ มีสบู่พร้อม เอ๊ะ ชักงงเหมือนกัน นะว่า ฉันเข้าใจผิดหรือว่ามันถูก หรือว่ามันผิดแล้วฉันไม่ถูกหรืออะไรอย่างไร แต่มันไม่ใช่น่ะ ฉันว่า มันไม่ใช่น่ะ แล้วก็พูดอะไร คือ นี่ irrelevant เยอะไง แต่จะหลุดมาเป็นบางประโยค คือเขาคิดว่า มันน่าจะ อธิบายเป็นการขยายความ แต่ว่า พอเรามาอ่านลึกๆ จริงๆ มันไม่ใช่ RESEARCHER: แสดงว่าไอเดียส่วนใหญ่เนี่ยมันไม่เข้าเรื่อง แต่จะมีหลุดไอ้ที่เข้าเรื่อง ไอ้ที่ใช้ได้ อยู่บ้าง ใช่ไหมคะ LADDA: บ้าง แต่ เออ ก็จะมี irrelevant บ้าง แล้วที่เรื่องของเรื่องก็คือเด็กไทยยังคิดไม่เป็นระบบ เวลาวางโครงอะไรอย่างนี้นะ มันยังคิดไม่เป็นระบบ มันยังคิดไม่ออก อย่างเช่นเราบอกว่า ข้อดีของ คอมพิวเตอร์คืออะไร มันก็ โอ้โห จารนัยซะจน คือมันไม่แบ่งเป็น group ทันทีเลยว่า หมวดที่หนึ่ง อะไร หมวดที่สอง คืออะไร แล้วหมวดที่สามคืออะไร แต่มันจะเอามาปนกันหมด แล้วมันยังแยกไม่ เป็น ไม่ออกระหว่าง major กับ minor details ซึ่งสอนยากน่ะ เพราะว่าอย่างนี้ คุณต้องคิดเป็น group เป็น group ไอเดียเป็น ไม่เป็น ขาดตรงนี้ด้วยน่ะ แล้วก็เวลาเขียนก็มักจะไม่ค่อยให้ รายละเอียดเท่าที่ควร ก็แค่คิดว่า เช่น ฉันชอบเลี้ยงหมา เพราะว่าหมามันน่ารัก จบ มันไม่ขยาย ความค าว่าน่ารักอย่างไร หรือว่า ฉัน ชอบเลี้ยงหมาก็เพราะว่าหมามันไม่รบกวน เออนะ หรือว่า หรือ สมมติว่าฉันชอบเลี้ยงปลาทองเพราะมันไม่รบกวนอะไรเลย แล้วไม่รบกวนนี่คืออะไร ก็ไม่ เขียน คือมันจะทิ้งไว้แค่นั้นน่ะ แล้วมันก็จะไปขึ้นเรื่องอื่น ขึ้น ขึ้น second หรือ third factors ไป เลย RESEARCHER: ก็เลยกลายเป็นว่าใจความไม่ครบ LADDA: เออ ใจความไม่ครบ อย่างนี้อ่านแล้วก็ไม่ค่อยได้ใจความ คุณให้แต่ major มา แต่คุณ ขาด minor น่ะ ไม่อธิบาย minor ให้มันแจ่มแจ้งเลย จะยกเป็น อธิบายเป็นการยกตัวอย่างหรือ จากประสบการณ์แล้วมาเสริมเข้าไปว่าฉันหมายความว่าอย่างไร ตรงนี้น่ะว่าท าไมฉันถึงชอบเลี้ยง หมา มันว่านอนสอนง่าย เออ แล้วมันหมายความว่าไง ไอ้ค าว่าว่านอนสอนง่าย เช่นว่าอะไรๆ เธอ ก็เล่ามาสิ มันไม่พูดอ่ะ มันแค่บอกว่าว่านอนสอนง่าย แล้วก็ไปแล้ว เออ แล้วมีเด็กคนหนึ่งบอกว่า

346

เนี่ยเขาเขียนถึงงานของพ่อเขาว่า job ของพ่อเขาเนี่ย interesting แล้วเขาก็เล่าไปเรื่อยเปื่อย ก็คือ คือ เลื้อยไปเรื่อยอ่ะ เอ่อ พ่อเป็นครู เพราะงั้นพ่อก็จะได้เจอคนหลายหลากอาชีพ มีนิสัยใจคอ แตกต่าง อ้าว แล้วประเด็นมันอยู่ตรงไหนล่ะ แล้วพ่อก็สอนหนังสือ พ่อก็ เป็นการช่วยชาติไปไปตัว ช่วยเยาวชน อะไรล่ะ มันปนกันไปหมด ท าไมคุณไม่แบ่งเป็นข้อๆ a อาชีพพ่อของเธอนี่น่าสนใจ ตรงที่ว่าหนึ่งคืออะไร a คืออะไร b คืออะไร c คืออะไร มันไม่แยก กี่ครั้งแล้ว แล้วเนี่ยคนเดิมอีก เหมือนกัน แล้วพออ่อนทั้งเรื่อง เรื่องของความคิดนะ ก็ยังอ่อนในเรื่องของไวยากรณ์อีก เออ My father has friendly. ก็ friendly มันเป็น adjective แล้วใช้กับ have, has ได้อย่างไร has friendly อะไรของมันก็ไม่รู้ อืม จนพี่นี่นะหมดปัญญาแล้วเวลาส่งงานนะ ตรวจเสร็จต้องให้บอกท าใหม่ ท า ใหม่ เป็นรอบที่สองที่สาม พี่ไม่ไหวแล้วที่จะมาซ ้า แล้วก็ตรวจ เพราะว่าเด็กตั้ง 40-50 คนต่อ 1 section RESEARCHER: ของพี่นี่ section นี้เท่าไหร่คะ LADDA: อันนี้พูดถึง paragraph writing นะ paragraph writing นี่ 28 คน ส่วน advanced นี่ปีนี้ น้อยหน่อย ได้ 24 คน ซึ่งไม่เหมือนสมัยก่อนๆ เทอมที่แล้วนี่นะ 40 38 แล้วแต่ละคนนี่นะ เท่ากับ ว่าพี่ต้องท างานสองเท่า เพราะว่า ส่วนใหญ่ร้อยละเท่าไหร่ดีล่ะ ร้อยละ 80 ล่ะมั้งหรือร้อยละ 90 ของจ านวนทั้งหมด พี่ต้องส่งงานคืนว่าไปท าใหม่ RESEARCHER: แล้วแต่ละ text type นี่ พี่ให้เขาเขียนงานกี่ครั้งคะ LADDA: สัปดาห์ละ 1 เรื่อง RESEARCHER: แล้วต้องแก้ไหมคะ LADDA: ต้องแก้ทุกคนเลย เกือบจะทุกคนเลย RESEARCHER: แก้นี่คือกี่ครั้งคะ LADDA: เมื่อก่อนนะ 2-3 ครั้งจนกว่าเขาจะ จะ พอใจ ตัวพี่จะพอใจ แต่ตอนหลังพี่ไม่ไหวแล้ว เรื่องอะไรจะต้องท างานซ ้าซากอยู่อย่างนี้ ได้แล้วก็ตรวจ ตรวจแล้วก็เด็กมันไม่ค่อยได้ realise อะไร ก็ได้แต่แก้ไปตามที่เราว่าอะไรอย่างนี้ คือมันไม่ได้ใส่ใจ เรียน เหมือนกับว่ามันทะลุหูซ้าย หูขวา พอ ให้ผ่านๆ ก็แล้วกัน พี่ก็เลยไม่เอา ตอนนี้ก็เลยให้ได้ครั้งเดียว RESEARCHER: ครั้งเดียวส่งเหรอคะ LADDA: ไม่ๆ ให้แก้ตัวได้รอบเดียวเท่านั้น แล้วหลังจากนั้นไม่เอาแล้ว ได้เท่าไหร่ก็เท่านั้น แต่มี กฏเกณฑ์ว่า คืองานทุกชิ้นพี่จะให้ 10 เต็ม 10 แล้วถ้าใครต ่ากว่า 5 เนี่ย ก็ต้องท าใหม่ ส่วนมากก็ ท าใหม่น่ะ คือว่า ไม่ให้คะแนนเลย ให้ท าใหม่มาเลย แล้วก็พอแก้รอบที่สองนี่ สูงสุดคือได้แค่ 5.5 แค่นั้น ไม่ให้เต็มสิบแล เรื่องอะไร เพราะว่าคุณน่ะ มีโอกาสแก้ตัว ในขณะที่เพื่อนที่ส่งมารอบเดียว ผ่าน ได้ 7.5 ได้ 8 แต่คุณนี่ต้องไปแก้ใหม่ทั้งยวงใช่ไหม แล้วคุณอาจจะไปจ้างฝรั่งที่ไหนข้างนอก

347

ซึ่งฉันไม่เห็น เพราะนี่คือการบ้านข้างนอกไง ไปถามใครที่เก่งๆ รุ่นพี่ที่เก่งๆ ให้เขาตรวจให้ แล้วเธอ มาส่งให้ฉันแล้วเธอจะได้เต็มสิบเหรอ เป็นไปไม่ได้ มนั ไมย่ ตุ ิธรรมกบั คนอนื่ ๆ ที่เขาไมม่ ีสิทธิ์แก้ตวั ทั้งๆ ที่เขามีศักยภาพที่จะเขียนดีกว่านี้ แต่ว่า ในเมื่อเราตรวจแล้วเราเห็นว่า เออ เขาผ่านแล้ว แต่ ได้ระดับ 8.5 9.5 ม่เอาแล้ว โอเค ไม่ต้องท าใหม่ แต่ไอ้พวกนี้มันโหลยโท่ย เพราะต ่ากว่า 5 ต ่ากว่า 4 น่ะ ซึ่งไม่รู้จะให้คะแนนยังไง คือจากใจน่ะอยากจะให้ 0 ไข่ไก่ไปเลย แล้วก็ไปท าใหม่มา ก็จะให้ สูงสุดแค่ 5.5 หรือไม่ก็ 5 หรือไม่ก็ต ่าไปกว่านั้น ไม่มีเยอะกว่านี้แล้ว RESEARCHER: แล้วเด็กเขาเขียนงานกี่ชิ้นคะ LADDA: โอ แปดชิ้นต่อเทอม RESEARCHER: นี่คือวิชา basic เอ้อ paragraph writing LADDA: เท่ากัน paragraph writing ก็เท่ากันแปดชิ้น RESEARCHER: advanced ก็แปด LADDA: อืม สัปดาห์ละเรื่อง แปดชิ้น แต่เขาต้องไปอ่านหนังสือพิมพ์อีก เพราะว่าพี่มีวิธีเก็บ คะแนนหลากหลายนะ นี่คือ assignment ทั้งหมดคือ 100% แบ่งเป็น midterm final อย่างละ 30- 30 60 แล้วใช่ไหม วิชา ไอ้คะแนนช่วยน่ะ assignment 30% จากงานแปดชิ้น ชิ้นละสิบ สิบ สิบ แล้วก็ค่อยมาตั้งบัญญัติไตรยางค์แล้วก็เก็บมาแค่ 30 อีก 10 นะ ซึ่งถ้าเป็นเมื่อก่อน คนอื่นๆ เขาก็ แค่ attendance เรียกชื่อๆ เรื่องอะไรอ่ะ แค่โผล่มาบางทีใจมันไม่มาอ่ะ มีแต่ร่างกายมานั่งอยู่ หัวใจไปนั่งอยู่ใต้ต้นไม้ไหนไม่รู้ ปลายตึก ยอดตึกที่ไหนก็ไม่รู้ ฉะนั้น เรื่องอะไรมันจะได้ 10 ไปเลย เสร็จ มันต้อง put an effort ฉะนั้น พี่ก็จะสั่งเลย อาทิตย์ละเรื่อง พี่จะ assign ว่าเอาหละ คือ พี่ก็ จะคอยติดตามข่าวว่าช่วงนี้อะไรดัง เช่น เครื่องบินตก ของทหารใช่ไหม เอ้า คุณไปท า ไปแปลข่าว นี้มา 5 ย่อหน้า เฉพาะข่าวนี้นะ หัวข้อเรื่องนี้เลย เฉพาะไอ้เครื่องบินตกเนี่ย ไปแปลมา 5 ย่อหน้า หรือว่าตอนนี้ hot hot ก็คือ เราก าลังจะตั้งรัฐบาลใหม่ เขาเพิ่งจะเปิดการประชุมสภาทั้งวันนี้ เอ้า คุณไปแปลมา 5 ย่อหน้า ตรงไหนก็ได้ ส่วนไหนของหนังสือพิมพ์ แล้วคุณต้อง xerox ต้นฉบับของ หนังสือพิมพ์ ไม่งั้นเดี๋ยวฉันไม่รู้หนังสือพิมพ์ ว่าเธอนั่งเทียนเขียนหรือเปล่า แล้วเธอต้องตีกรอบให้ ฉันว่าเธอแปลตั้งแต่ แล้วต้องเป็นย่อหน้าที่ติดต่อกันเลย ไม่ใช่เลือกเอาเฉพาะสองบรรทัด จะเป็น อย่างนี้ไง ตอนนั้นเราไม่เคยก าหนด มันจะเลือกบรรทัดไหนที่มีสองบรรทัดน่ะ ก็หนึ่ง แล้วก็หาอีก ตรงไหนที่มีสองบรรทัด มันก็ไม่ต่อเนื่องสิ ไม่เอา คุณเริ่มจากตรงไหน คุณต้องเอามาให้ฉัน แล้วก็ เขียนก ากับหมายเลข 1 2 3 4 5 แล้วก็แปลเป็นภาษาไทย ย่อหน้าต่อย่อหน้า บรรทัดต่อบรรทัด และคุณก็ต้องก าหนดภาษาไทยว่านี่คือย่อหน้า 1 2 3 4 5 เพราะฉันตรวจ เทียบเคียงเลย ส่ง ต้นฉบับมา แล้วฉันก็ดู แล้วก็บังคับว่า ไอ้วัตถุประสงค์ behind อันนั้นนะ ก็คือว่า หนึ่ง ต้องการให้ เด็กฝึกดู sentence structure ที่เด็กเรียนมาแล้วในห้อง simple compound complex

348

โดยเฉพาะ complex ใช่ไหม ซึ่งมีทั้ง noun clause, adjective clause แล้วก็ adverbial clause โดยเฉพาะ adverbial clause แบบ reduced แบบลดรูปนี่ ก็จะได้สังเกตเห็นว่าเขาใช้จริงเนี่ย จาก authentic material เขาใช้กันอย่างไร ให้ฝึก แล้วก็สอง เพิ่มพูนค าศัพท์ อย่างเช่น รัฐสภา คุณ ก็เรียกว่าอะไร การเผาศพ เรียกว่าอะไร เครื่องบินตก เรียกว่าอะไร แล้วก็อะไรที่เกี่ยวข้องสารพัด กับศัพท์ในแวดวง ในหัวข้อนั้นๆ น่ะ สิ่งแวดล้อมเอย อะไรเอย มันก็จะได้เพิ่มพูนค าศัพท์ นี่คือหัวใจ สองอันที่พี่ต้องการให้มันเรียนรู้ แต่ไม่รู้ว่ามันจะเรียนรู้หรือเปล่านะ เพราะเราไม่ได้เช็ค RESEARCHER: แล้วกี่ชิ้นล่ะคะพี่ LADDA: อาทิตย์ละเรื่องเลย ก็ 16 ชิ้นน่ะ ประมาณ 10 ชิ้นน่ะ แล้วก็มาตรวจ เช็คๆๆ มีนะ มัน กระโดดข้ามไง แปลเนื้อ แล้วข้ามไป ฉัน บอกกลับไปแปลมาใหม่ แล้วบางทีก็นั่งเทียน คือไม่ยอม หาค าศัพท์ มันคิดเอา เดาเอา ซึ่งผิดไปจากต้นฉบับเลย พี่ก็จะวงแล้วก็เขียนว่าผิดในภาษา ใน ต้นฉบับไม่มีค านี้ที่คุณแปล มันไม่มี ในต้นฉบับมันไม่มี เธอนั่งเทียนเขียนเองเหรอ ไปแปลมาใหม่ เป็นย่อหน้า ๆ แปลตรงอะไรอย่างนี้ RESEARCHER: แสดงว่าในหนึ่งอาทิตย์นี่เด็กก็มีงานเขียนสองชิ้น LADDA: ใช่ RESEARCHER: คืองานแปลหนึ่งชิ้นแล้วก็งานเขียนหนึ่งชิ้น แล้วถ้าเขียนดีก็ผ่าน ถ้าเขียนไม่ดีก็ แก้ LADDA: ซ ้าอีก อืม LADDA: แล้วเมื่อก่อนนี่นะ พี่ให้ท าเป็นคู่เฉพาะข่าว สองคนไปช่วยกัน ช่วยกันแปล ช่วยกันหา เปล่า ในความเป็นจริง ตอนหลังจับได้ คราวนี้อาจารย์สั่ง แกไปท า แล้วก็พิมพ์ชื่อฉันลงไปด้วย พร้อมรหัส คราวหน้า งวดหน้า คือสลับกันท า มันไม่ได้ช่วยน่ะ ฉะนั้น พวกคุณอยากท า individually ใช่ไหม โอเค fine เอาไปเลย คนละเลย ต่างคนต่างท า ซึ่งมันต้องถูกบังคับ แล้วพี่ก็จะ บอกว่าห้ามซ ้าด้วยนะ ห้ามมีย่อหน้าที่ซ ้ากันนะ เพราะเท่ากับว่ายูไปลอกของอีกคนหนึ่งมาสิ RESEARCHER: หมายถึง ห้ามซ ้ากับเพื่อน LADDA: ใช่ RESEARCHER: ข่าวเดียวกันได้ แต่ส านวนแปล... LADDA: ใช่ เพราะฉะนั้นต้องไปหา แล้วก็มันลงติดๆ กันตั้งหลายวัน พี่ก็ไม่ได้เจาะจงว่าเธอไปหา ฉบับวันที่ 24 สิงหา ใช่ไหม เธอไปหาวันไหนก็ได้ที่เกี่ยวกับเรื่องเครื่องบินตก เพราะมันจะลงอยู่เป็น อาทิตย์ด้วยซ ้าไป เพราะฉะนั้นโอกาสที่จะซ ้ายาก น้อยมาก เพราะแค่ห้าย่อหน้าเอง นั่นคือของพี่ อีกสิบเปอร์เซ็นต์ เห็นไหม กว่าจะได้มาไม่ใช่ง่ายๆ นะ RESEARCHER: แล้วอย่างเวลาพี่ให้คะแนน อ๋อ คือแล้วแต่เด็กส่งมา

349

LADDA: อ๋อ สิบเปอร์เซ็นต์นี้ ชิ้นละ 1 เปอร์เซ็นต์ไปเลย เหมือนสมัยก่อนที่เราส่งการบ้าน FE หรือ ไรไม่รู้หละ RESEARCHER: พวก port folio อะไรอย่างนี้ใช่ไหมคะ LADDA: เออ อะไรอย่างนั้นแหละ ก็ได้ไปเลยหนึ่งแต้มฟรีๆ ไปเลย ก็เท่ากับที่เราเช็คเวลาเขา attendant แต่นี่มันยิ่งกว่า attendance น่ะ ใช่ไหม เพราะเขาต้องท างานส่ง ทุกอาทิตย์ อาทิตย์ละ แล้วก็แนบต้นฉบับมาด้วย แล้วไม่ให้เขียน ไม่ให้เอาลายมือ เพราะเดี๋ยวนี้เด็กสมัยใหม่มันเก่งคอม พิมพ์มาให้เรียบร้อย ทุกอย่างพิมพ์ พิมพ์หมดลย ทั้งตอนเขียน assignment ด้วย สบายฉันไง RESEARCHER: พี่ให้ส่งเป็น hard copy เลยใช่ไหมคะ ไม่ได้ส่ง online LADDA: ไม่น่ะ ไม่เอา เพราะท าไม่เป็น เรียกมาดูไม่เป็น เพราะฉะนั้น ส่งเป็น paper มาเลย แล้ว พอเสร็จก็คืนเขา RESEARCHER: ใช้แบบนี้ทั้งสองวิชาใช่ไหมคะ ทั้ง basic ทั้ง advanced LADDA: ใช่ๆๆ RESEARCHER: แต่สองวิชานั้น เนื้อหาวิชามันต่างกัน LADDA: ต่างกันแล อันนึง level แค่ระดับหนึ่งย่อหน้าไง แค่ basic แต่อีกอันนึงนั้นเป็น essay ไป เลย กระโดดไปห้าย่อหน้า RESEARCHER: แล้วอย่างจุดประสงค์ของสองวิชานี้มันล้อกันไหมคะ LADDA: ล้อ ก็แค่ต้องการจะฝึกให้เด็กเข้มแข็งในการเขียน writing RESEARCHER: แค่ว่าระดับเท่านั้นที่มันต่างกัน LADDA: หนังสือก็ต่างกัน เนื้อหาก็ต่างกัน RESEARCHER: แล้วเวลาพี่ตรวจนี่ พี่ใช้เกณฑ์อะไรคะ LADDA: เกณฑ์อะไรเหรอ ค่อนข้าง subjective มากๆ คือ อาศัยความช านาญ แต่จริงๆ เวลาสอบ นี่เราจะตั้งเกณฑ์ว่า grammar เท่าไหร่ content เท่าไหร่ organisation เท่าไหร่ mechanics เท่าไหร่ แต่ถ้าเวลาส่งงานมาแบบนี้ พี่จะอ่านรวมเลยในภาพรวม ทั้ง content ทั้งอะไรทุกอย่าง และในความรู้สึกลึกๆ พี่จะ gear ไปทาง grammar คือถ้า grammar มันสละสลวย คือมันถูกต้อง ใช่ไหม แล้วเนื้อหา คือมันทั้งสองอย่าง เนื้อหามันก็ตรงตามที่เราต้องการ การมี เพราะเขาจะต้อง วางตั้งแต่ outline outline เขาถูกต้อง major detail, minor detail เขาละเอียด แล้วก็เวลาเขา organisation ก็ดี มี sentence connector ที่สละสลวยดี พี่ก็จะให้ไปเลย 8.5, 9 หรือเลิศๆ ไปเลย พี่ก็ให้ไปเลย 10 แต่น้อยมากที่จะได้ 10 แต่ถ้ามันมี grammar, fragment บ้าบอ ผิดเยอะๆ แล้วก็ irrelevant sentence อะไรหลายๆ อย่าง พี่ก็จะเริ่มลดแล้ว เพดาน โดยอัตโนมัติน่ะ พี่ก็บอกไม่ถูก เหมือนกัน แต่พี่ไม่แบ่งชัดเจนว่า grammar เท่าไหร่ อะไร ขี้เกียจน่ะ เพราะมันแค่ 10 คะแนน

350

RESEARCHER: พี่ให้อย่างนี้จริง แล้วเวลาเข้าห้องเรียนนี่คือพี่จ าประเด็นที่เขาท าผิดๆ ไปเหรอ คะ LADDA: โอ๊ะโอ พีส่องงาน เพราะเดี๋ยวนี้ดีมากเลย hi-tech ช่วยพี่ได้เยอะ สมัยก่อนพี่ต้องมานั่ง เขียนบนกระดาน ประโยคที่ผิดๆ ทั้งหลาย เพื่อให้ทุกคนได้เรียนรู้ แต่ตอนนี้นี่ ใครงานคนนั้น ใช่ ไหม เสร็จ พี่ก็ส่องไปที่ LCD แล้วพี่ก็ช่างหัวมัน แล้วฉันก็ชี้ว่า ฉันจะไม่แก้ไง ไม่แก้แล้วต่อไปนี้ วงๆๆๆๆ แล้วนี่ รู้ไหมว่าผิดตรงไหน ถามเจ้าของ ถ้าเจ้าของไม่รู้ก็ถามคนอื่นๆ นะ แล้วก็ค่อยๆ ไปที ละอันๆๆ อืม เพราะฉะนั้น เสียเวลามากๆ RESEARCHER: แล้วอย่างนี้ ของพี่นี่คือ LADDA: แต่พี่ไม่เอาของทุกคนนะ เอาของไอ้คนที่เด่นๆ ที่แบบผิดจน แบบเน่าทั้งฉบับแล้วหนะ ซึ่ง ก็ประมาณสิบกว่าคนน่ะ เอามาส่องดู RESEARCHER: เยอะเหมือนกันนะคะ LADDA: ก็เยอะสิ ใช้เวลาเกือบครึ่งชั่วโมงน่ะ เวลาไปกับไอ้พวกนี้น่ะนะ RESEARCHER: ตรวจเสร็จแล้ว พี่ก็มานั่งดูอีกรอบเหรอคะเพื่อเลือกกลุ่มพวกนี้ LADDA: อ๋อ แล้วก็กาเลย กากบาทเลย กาที่ชื่อเขาเลยว่า หรือไม่ก็วงไว้ว่านี่ outstanding มาก เดี๋ยวฉันจะเอาไปเป็น lesson โชว์ ซึ่งเด็กบางที feedback มันก็บอกมันอายไง เพราะว่าชื่อเสียง เรียงนาม แล้วเราก็ว่ามัน ด่า เสียๆ หายๆ อะไรอย่างนี้ มันก็ไม่ชอบ ฉีกหน้ามัน แต่รุ่นก่อนๆ เขา บอกเขาชินเสียแล้ว เฉยๆ เออ แล้วก็รู้สึกว่าเขาได้เรียนรู้ขึ้นเยอะไง ไม่รู้นะ เท่าที่เด็กเขาเคยเขียน มา RESEARCHER: พี่ก็ใช้เกณฑ์มาจากประสบการณ์ของพี่นี่ เหมือนกันทุก text type ใช่ไหมคะ LADDA: ใช่ๆ วิธีเดียวกัน RESEARCHER: เพราะความต่างก็คืออยู่ที่ idea กับเนื้อหา แล้วก็ grammar แค่นั้นใช่ไหมคะ LADDA: อืม เออ ส่วนมากพี่จะมุ่งประเด็นไปที่ organisation ซึ่งหมายถึงว่ามี topic sentence ไหม มี supporting detail เพียงพอไหม มีการใช้ sentence connector มี cohesion น่ะ มี cohesion กับ unity นี่ แล้วก็เรื่องของ content เอ่อ content คือ unity ส่วนของ organisation ก็ คือ cohesion ใช่ไหม เรื่องพวกนี้ ครอบคลุมไปหมด แล้วก็ grammar โดยอัตโนมัติน่ะ มันจะ ออกมาเอง RESEARCHER: แล้วอย่าง mechanics ล่ะคะ LADDA: เอ่อ พวก mechanics นี่เขาไม่ค่อยผิดกันเท่าไหร่นะ RESEARCHER: ใช่ เพราะว่าเป็นคอม(พิวเตอร์) LADDA: เอ่อ เขาไม่ค่อยมีปัญหาเรื่อง comma เออ ก็มีบ้างนิดหน่อยอ่ะนะ full stop ไม่เห็นมันมี

351

ปัญหา ขยันใส่ full stop จะตาย เพราะมัน จะได้จบประโยคมันเร็วๆ ใช่ไหมน่ะ แล้วก็ spelling มัน ก็โอเคอ่ะ ไม่ค่อยมีปัญหาอะไรมากนัก เพราะ mechanics มันครอบคลุม เอ่อ แต่ว่า grammar พี่ ก็แยกออกมาต่างหาก เป็นอีกหัวข้อใหญ่ ทั้งที่ในหนังสือเนี่ย mechanics คือรวมถึง grammar แต่ จริงๆ แล้ว ตามหลักจริงๆ แล้ว ของฝรั่งเนี่ย mechanics ของเขา คือ เขาหมายถึงตัว grammar, spelling แล้วก็ punctuation แต่ของพี่นี่พี่แยกเป็นหัวข้อต่างหาก ฉันคิดของฉันเลยว่า grammar คือส่วนใหญ่เลย เป็น head แล้ว mechanics ก็มี punctuation, capitalisation การเขียนอักษร เล็กอักษรใหญ่ แล้วก็ spelling ซึ่งมันไม่ค่อยมีปัญหาสักเท่าไหร่ RESEARCHER: วิชาที่จะตรวจให้ดูวันนี้ คือวิชาอะไรคะ LADDA: อ๋อ นี่เป็น advanced นี่เป็น essay แล้วหละ RESEARCHER: วิชา Advanced นี่พี่มีคนสอนร่วมไหมคะ LADDA: มี เนี่ยเป็นปีแรกที่ใช้คนต่างชาติสอน สอนคนละกลุ่ม RESEARCHER: แล้ววิธีการสอนเหมือนกันไหม LADDA: อันนี้ไม่ทราบ RESEARCHER: แยกกลุ่มไปแล้วก็อิสระเลย LADDA: ใช่ แต่ว่าเรามีหนังสือ แล้วก็มีเกณฑ์มาว่าให้สอนอะไรบ้าง เรามี course description แล้วก็ course รายละเอียดของ course เนี่ย course outline ว่าคุณจะต้อง go over อะไรบ้าง มี หัวข้อ แต่ว่าเทคนิคการสอนก็ตัวใครตัวมันแล เราไปบังคับเขาไม่ได้ RESEARCHER: แล้วอย่างตอนตรวจล่ะคะ LADDA: ก็ตัวใครตัวมัน แต่เฉพาะข้อสอบ midterm กับ final เรามาตั้งเกณฑ์กันเอง ชัดเจนเลย RESEARCHER: การสั่งงานก็แล้วแต่เขา LADDA: อืม เพราะพี่ไปแล้วไง ว่าเกณฑ์การให้คะแนนเป็นอย่างนี้ เขาก็ท าตาม คือต้องมี assignment กี่ชิ้น ก็มี news report translation กี่ชิ้น แต่ในแง่ของเขา เขาก็อ่านภาษาไทยไม่ได้ เขาก็สั่งให้เด็ก summarise แล้วก็แนบต้นฉบับมา แล้วเขาก็จะมาเทียบเคียง แต่ของพี่นี่ เราอ่าน หนังสือไทยได้ ก็ให้เด็กแปลบรรทัดต่อบรรทัด เป็นย่อหน้า แต่ของเขาให้เด็กอ่านแล้วก็ summarise เพราะอ่านภาษาไทยไม่ได้ คือจ านวนงาน ลักษณะงานอะไรเหมือนกัน แต่เทคนิคการ สอนก็ตัวใครตัวมัน RESEARCHER: แล้วอย่างตรวจงานแต่ละชิ้นพี่ใช้เวลานานไหมคะ LADDA: ชิ้นนึงก็ประมาณ 20 นาที RESEARCHER: ตรวจอย่างละเอียดสิคะ LADDA: ใช่ ก็อ่านทุกบรรทัดน่ะ ถึงได้ค้างเป็นกะตักเลย ค้างเยอะมาก แล้วไหนงานที่พอ แล้วเรา

352

ก็สั่งงานไปเรื่อยๆ ทุกอาทิตย์ งานเก่าที่มันไปแก้มาก็ต้องไปส่งเราในคาบต่อไปด้วยนะ ไม่มีการ ข้ามอาทิตย์นะ พี่ส่งไปวันจันทร์ใช่ไหม แล้ววันพฤหัสเจอ งั้นวันพฤหัสคุณต้องไปแก้ฉบับที่สอง มาแล้วส่งฉัน ใครไม่ส่งตรงเวลา ไม่รับ ไม่ต้องมาสอดตามประตูนะ ไม่รับ มีนะมี งานไม่ส่งตรง เวลาเหมือนคนอื่นเขา เอามาส่งให้พี่นะ พี่กาเลย เขียนเป็น หนูขอโทษนะคะ หนูรู้ว่าอาจารย์ต้อง ล าบากในการตรวจอะไรต่ออะไร แต่จ าเป็นอย่างยิ่ง ต้องขออนุญาต ฉันบอก ฉันไม่รับ ค าพูดต้อง เป็นค าพูด เราตกลงกันแล้ว กติกาต้องเป็นกติกา ส่งมาทั้งปึกเลย ดูสิทั้งงานแก้งานปัจจุบัน งาน ข่าวมันไม่ส่ง แล้วมันรวบทีเดียวสี่ห้าชิ้นมาส่งฉัน แล้วเรื่องอะไรฉันต้องมาตรวจให้มันไม่ตรวจให้ RESEARCHER: ตรวจงานเขียน 20 นาทีทุกรายหรือว่ายังไง LADDA: ทุกรายเลย เพราะมันต้องตรวจทุกบรรทัด แต่บางคนที่เขียนดีๆ ก็ท าให้ตรวจเร็วขึ้น เราก็ ตรวจ อ่านทุกบรรทัดเลย แต่ถ้าคนที่ช้า ไอยา แล้วถ้าเรามาแก้ให้ ไม่เอา วงๆๆ แล้วพี่ก็จะกา ข้างบนว่าผิดอะไรบ้าง grammar, fragment, run-on แล้วก็ไปหาเอาเอง ถ้ามันแย่มากก็เอาไป ส่องเลย RESEARCHER: แล้วอย่างเวลาตรวจนี่ ตรวจรวดเดียวทุกคนหรือเปล่าคะ LADDA: โห ไม่ไหว RESEARCHER: ประมาณพี่ชิ้นคะ ต่อครั้ง LADDA: ก็ท าเป็นพักๆ ประมาณ 5 ชิ้น 10 ชิ้น ต่อครั้ง เมื่อวานตรวจได้ 10 กว่าชิ้น ระหว่างคุม สอบสามชั่วโมง RESEARCHER: ถ้าเครียดมากๆ ก็พักก่อนเหรอคะ LADDA: ก็ต้องลุกขึ้นไป ไปรดน ้าต้นไม้ ไปพัก 5 นาที 10 นาที เปลี่ยนอิริยาบถ แล้วกลับมาท าต่อ ถ้าในกรณีที่เอางานกลับไปตรวจที่บ้าน RESEARCHER: พี่สอน writing มากี่ปีแล้ว LADDA: 30 กว่าปีแล้ว LADDA: ใครๆ ก็เรียกฉันเป็นเจ้าแม่ writing ซึ่งไม่รู้ว่าฉายานี้มาจากไหน อันที่จริงทุกคนก็สอนได้ แต่การที่คนปฏิเสธ ไม่รับ หลายคนปฏิเสธ เมื่อก่อนพี่ต้องรับหมดเลยทั้งสองกลุ่มสามกลุ่ม พี่ เหนื่อยมากๆ เลย ตั้งแต่เปิดเป็นคณะศิลปศาสตร์ เปิดเป็นวิชาบังคับ พี่เหนื่อยมากก็เลยบ่นกับ ภาคว่าไม่ไหวแล้วนะ มันเยอะเกิน สอนเนี่ยสอนได้ แต่มันตรวจงานไม่ไหว เพราะมันเยอะ สอง กลุ่มก็ 80 คน ร้อยละ 90 ต้องท าใหม่เป็นรอบที่สอง กลายเป็นว่าถัวเฉลี่ยแล้วต้องตรวจงาน 160 คน ไม่ไหวน่ะ แล้วมันน่าเบื่อด้วย แล้วก็เยอะน่ะ ไม่ไหว ก็เลยยื่นค าขาดกับภาคว่าถ้าไม่ส่งคนมา ช่วยนะก็ไม่สอน ลงไปสอนวิชาพื้นฐานไม่ดีกว่าเหรอ เตรียมครั้งเดียว สอนเหมือนกันทุกกลุ่ม ข้อสอบก็ไม่ต้องออก เพราะมีกรรมการออกข้อสอบชุดหนึ่ง และข้อสอบก็ไม่ต้องตรวจ เพราะใช้

353

ตรวจด้วยระบบคอมพิวเตอร์ แต่วิชานี้ท าไม่ได้ เพราะต้องอ่านทุกชิ้น เพราะจะท าเป็น M/C เหรอ ก็ไม่ใช่ ต้องตรวจทุกชิ้นอีก เพราะมันต้องฝึกเขาไง มันเหนื่อยไง พี่น่ะสอนได้ แต่ถามว่ามีคนช่วย ไหม ก็ไม่มีอีก มันก็ไม่ได้ดั่งใจเราอีก ตอนนั้นก็เลยลาก...[T names two of her colleauges.] มา ช่วยสอน อันนั้น intro to writing แล้ว advanced ก็ยังสอนเอง ประกอบกัเด็กเขาท าการประเมิน ว่าเขาอยากเรียนกับฝรั่ง เพราะฉะนั้นก็เลยให้ฝรั่งช่วยสอน advanced ฝรั่งแท้หนึ่ง ฝรั่งเทียมหนึ่ง เพิ่งไปแอบๆ ถามว่าเป็นไงมั่ง ก็รู้สึกว่า เขาบอกเขาไม่รู้เรื่อง เพราะเขาเรียนเป็นภาษาอังกฤษหมด หนึ่งนะ เพราะ idea มันก็ยากไง มันก็เลยแบบ แล้วฝรั่งก็บ่นว่าเหมือนกับว่านี่เป็นของใหม่ส าหรับ เขาไง ทั้งๆ ที่เป็นเจ้าของภาษา ก็เหมือนกับเรานั่นแหละ เราก็รู้ เข้าใจภาษาไทยดี แต่อยู่ๆ จับให้ เราไปสอนภาษาไทยให้กับฝรั่งหนึ่งคน เราก็งงเหมือนกันว่าเราจะขึ้นต้นอย่างไร ลงท่ายอย่างไร แม่ว่าจะมีหนังสือเป็น guideline ใช่ไหม [T mentions her English native speaking co- teacher.] นี่มีปัญหาประจ า เขาบอกว่าเขาไม่เข้าใจว่าเด็กเกิดอะไรขึ้น ท าไมเขียนไม่รู้เรื่องเลย fragment เต็มไปหมด ซึ่งส าหรับพี่เป็นเรื่องปกติ แต่ส าหรับเขา เขาบอก เขางง แล้วไหนพี่บอกว่า introduction มันก็เรียนมาแล้ว grammar 2 คอร์สก็เรียนจบมาแล้ว แล้วยิ่งอยู่ปีสาม นี่เรียนเป็น วิชาเอก แล้วท าไมออกมาแบบนี้ล่ะเธอ พี่บอกพี่ก็ไม่รู้เหมือนกัน เพราะของฉันก็มี แล้วมันคืออะไร ล่ะ แล้วความผิดพลาดมันอยู่ตรงไหน มันก็เลยท าให้เราย้อนมาดูว่าเออ ความล้มเหลวมันเกิดขึ้น ตั้งแต่ตรงไหนล่ะ ท าไมเราสอนแล้วเด็กไม่ได้แบบที่เราต้องการ แล้วเราก็สอนหมดแล้วทุกสิ่งทุก อย่าง แต่ปรากฏว่าผลผลิตออกมาแย่ และที่น่าตกใจก็คือ เขาจะคิดเป็นภาษาไทย แปลจาก ภาษาไทยเป็นภาษาอังกฤษ RESEARCHER: แปลตรงๆ LADDA: ใช่ แบบตรงๆ เลย เช่น ฉันคิดจะซื้อของให้แม่ I think will ฉันคิดจะ มีจะ ก็ will ก็จะใช่ ไหม ซื้อของขวัญ ก็ buy a gift ให้แม่ give to my mother นี่ไง ให้ give ไง นี่ไง เอามาจาก talking dict น่ะ หรือเขาไม่รู้ว่าตรงนั้น preposition แล้ว for, to ไม่ใช่ให้ give ภาษาไทยเขายังแยกไม่ออก เลยอะไรเป็นบุพบท แต่เราคนที่สอนหนังสือ เราสอนภาษา เรารู้ไง แต่พวกมันเองน่ะ ไม่รู้ว่าอะไร ท าหน้าที่อะไร เพราะมันใช้ค าเหมือนกัน ฉันจะซื้อของขวัญให้แม่ ก็ I will buy a gift for my mother. มันบอกว่าให้ give to my mother แล้วกลายเป็นว่าประโยคนี้มี verb กี่ตัวล่ะ ขึ้นกระดาน ให้ดู แล้วฉันบอกเธอแล้วใช่ไหม ในประโยคมันมี main verb อยู่ตัวเดียว นอกนั้นเป็นส่วนขยาย มันก็ไม่ get น่ะ ไม่รู้จะว่าไงเหมือนกัน RESEARCHER: อันนี้ถือเป็นความท้าทายอย่างหนึ่งในการสอน writing หรือเปล่าคะ LADDA: ท้าทาย แต่เราไม่รู้จะแก้ให้เขาตรงไหน เพราะเขาคิดเป็นภาษาไทย แล้วเด็กถาม แล้ว อาจารย์จะให้หนูคิดเป็นภาษาอังกฤษอย่างไร เออ ฉันก็ไม่รู้เหมือนกันนะ เออ เป็น question ที่ดี

354

ว่าแล้วจะให้คิดเป็นภาษาอังกฤษอย่างไรนี่ เพราะวงศัพท์ก็แคบ ก็เขียนไม่เป็นอีก มันก็ไม่รู้จะ อธิบายความรู้สึกนึกคิดออกมาเป็นภาษาอังกฤษให้ลุ่มลึกเท่าภาษาไทยมันก็ติดค าศัพท์อีก แล้ว พอเอามาเรียบเรียงเป็นประโยค มันก็แปลทื่อๆ ตรงๆ อีก ก็เลย ล้มเหลวหมดทั้งระบบ RESEARCHER: แล้วอย่างนี้เวลาตรวจ writing นี่พี่รู้สึกว่ามีอะไรท้าทายบ้างไหมคะ จากที่สอน มาสามสิบปี LADDA: ไม่เห็นมีอะไรน่าท้าทาย มีแต่น่าเบื่อ มีแต่ปัญหา แล้วฉันไม่รู้จะแก้ปัญหาอย่างไร จนบาง ทีฉันท้อเลยนะ นี่ฉันเสียพลังงานไปเยอะ เสียความตั้งใจนะ เสียความตั้งใจมากๆ เลย เราอยากจะ ถ่ายทอดความรู้ อยากจะให้มันรู้ อยากให้มันเก่งนะ เก่งเหมือนเราน่ะ แต่ปรากฎว่ามันไปไม่ถึง ดวงดาวที่เราตั้งไว้ แต่มันก็ไม่ใช่ทุกคนนะ บางคนก็เออ เห็นแล้วรู้สึกหายเลย หายเหนื่อยว่าที่เรา ลงแรงไปมันไม่เสีย แล้วพี่ก็บอกไอ้พวกนี้ว่าถ้าเธอไม่กล้ามาหาครู ฉันก็เบื่อพวกเธอแล้ว เธอไปหา คนที่เก่งๆ นี่เธอเก่งแล้วเธอช่วยเพื่อนหน่อยสิ ช่วยผ่อนแรงให้ครูหน่อย ครูไม่ไหวแล้ว ก็จะตายแล้ว บางทีเครียดมาก เหมือนจะ control ตัวเองไม่อยู่เลย ว่าอะไรกันนักกันหนา ผิดซ ้าผิดซาก แล้วที่ เธอพูดมามันจะสื่ออะไร ฉันไม่เห็นรู้เรื่องเลย แล้วมันก็บอก ผมต้องการจะพูดว่าอย่างนี้ แล้วท าไม ไม่เขียนมาตรงๆ ตรงนี้อะไรของเธอวกไปวนมาจนฉันไม่รู้เรื่อง แล้วแถมเป็น fragment อีกต่างหาก ประธานก็ไม่มี ลอยไปลอยมา มีแต่กริยา อะไรน่ะ เกิดอะไรขึ้นในสมองของเธอแล้ว แล้วสิ่งที่เรา เรียนมามันไปไหนหมดแล้ว RESEARCHER: แสดงว่าปัญหาของเด็กนี่เป็นอุปสรรคอย่างหนึ่งในการตรวจหรือเปล่าคะ LADDA: แน่นอน อุปสรรคอันใหญ่หลวงเท่าภูเขา RESEARCHER: เรื่องอื่นด้วยไหมคะ เรื่องเวลา งาน workload เกี่ยวไหมคะ LADDA: ด้วย แต่เทอมนี้ดีหน่อย เพราะเขาจัดให้ดีหน่อย เพราะเป็น listening and speaking ก็ดี หน่อย มี listening, role play ในห้องก็จบ ไม่มีการบ้านมาตรวจไง แต่ถ้าเป็นตัวที่สอง ที่เป็น reading and writing ก็มีการบ้านบ้างใน small class เล็กๆ น้อยๆ แต่ถ้าคุณสอนปาไป 4 sections section ละ 40 คน ก็ 160 แล้วก็ไปรวมกับวิชานี้ที่ต้องอ่านอย่างละเอียดอีก ก็ไม่รู้ว่าฉัน ผ่าน survive มาได้ไงก็ไม่รู้ RESEARCHER: พี่คิดว่าวิธีการตรวจแบบนี้มีข้อดีข้อเสียอย่างไรบ้างไหมคะ LADDA: มีข้อดีอย่างเดียว ไม่มีข้อเสีย ยกเว้นการที่เอาไปส่องแล้วบางคนอาจจะไม่พอใจ เพราะ บางทีชื่อก็โชว์ไปทั้งหมด หรือฉันก็จะถามเลยว่า ไหนเจ้าของ ครองขวัญคนไหน ช่วยยกมือหน่อย แล้วเธอเขียนมาอย่างนี้หมายความว่าอย่างไร เพราะว่าอายคน แต่เราไม่ได้อยากด่าอะไรนอกเรื่อง เลย แต่เราโฟกัสที่งานเขียนของเขา เราะเขาจะรู้สึกอึดอัดมากกว่าช่วยอะไรไม่ได้ นี่คือวิธีสอนของ พี่ อาจจะเป็นแบบโบราณ ไม่ค่อยคิดจะถนอมน ้าใจ แต่พี่คิดว่าเขาได้ส านึก ได้เรียนรู้ ไม่ได้เรียนรู้

355

คนเดียว เพื่อนๆ ก็ได้เรียนรู้ด้วย ว่าแบบนี้ผิด ทีหลังจะได้ระวัง ไม่ท าผิดซ ้าอีก เพราะเท่าที่คุย ให้ เขียน (feedback) ไปถามเขาตรงๆ ก็ไม่กล้าพูดหรอก วันปิดคอร์ส พี่แจกกระดาษให้เขียน อะไรก็ ได้ที่อยากเขียนเกี่ยวกับคอร์สนี้ ข้อดีข้อเสีย ชอบไม่ชอบ โดยไม่ต้องเขียนชื่อ เขาเขียนมาว่า ได้ เยอะ รู้อะไรเยอะๆขึ้นมากๆ แต่บางคนก็บอกว่าก็ดี แต่ไม่ชอบ เพราะอายคน RESEARCHER: พี่มีอะไรอยากปรับปรุงไหม LADDA: ปรับวิธีการสอนของเราให้เด็กเข้าใจได้ง่ายขึ้น สิ่งที่พี่ชอบ คือการ assign ให้เด็กไปดูการ แสดง เช่นคอนเสิร์ต โขน ที่มีจัดขึ้น แล้วให้เขียนมาว่ารู้สึกอะไรกับการแสดงนี้ ต้องวิเคราะห์ ความรู้สึกตัวเองได้ สองให้เช่าหนังเรื่องเด็กหอ ดูเรื่องเดียวกันแล้วเอามาเขียน เพื่อจะได้ control ได้ง่าย เพราะเป็นเรื่องเกี่ยวกับวิญญาณที่ยังมีผูกพันอยู่ แล้วมาเพื่อเตือนสติให้เด็กเข้าใจว่าพ่อแม่ ห่วง หนังมีข้อคิด มีสาระ แล้วเขียนวิจารณ์ว่าชอบหนังเรื่องนี้เพราะอะไร หรือได้แง่คิดอะไรจาก หนังเรื่องนี้ คิดมาสามประเด็น เขียนเรียงความ แสดง opinion เยอะๆ บางคนเขียนมาเลยว่าเขา ชอบมาก เขาไม่เคยดูและไม่เคยซาบซึ้งศิลปะอะไรพวกนี้มาก่อน เด็กแพทย์เก่งกว่าเด็กเอก ภาษา จะไหลลื่นกว่า แต่เก่งกว่าเด็กเอก พี่ยังด่าเลยนะ ท าไมเขาเก่งกว่า ทั้งๆ ที่ไม่ได้เรียน grammar มา สองคอร์สแบบเธอ เขาเรียนแค่ FE I, FE II แค่สองตัว แล้วไอเดียดีมาก logic ดี อธิบายได้เป็นเหตุ เป็นผล แต่ค าศัพท์อาจมีใช้ผิดบ้าง เขาบอกเขาไม่เคยสนใจ ตอนนั้นครูบังคับ เขาซาบซึ้ง เขา ขอบคุณพี่ ที่ assign จากตอนแรกที่เขาไม่พอใจว่าไปบังคับเขาซึ่งเขาไม่ชอบโขน ดูหนังอะไรอย่าง นี้ ไม่อยากดู แต่พอกลับมา เขารู้สึกเขาซาบซึ้ง เรามีวัฒนธรรมที่ดีงาม เขาบอกต่อจากนี้ก็จะไปดู ศิลปะวัฒนธรรมเป็นการอนุรักษ์ต่อไป เพราะการไปชมก็เป็นการช่วยอนุรักษ์ เขียนมาดีมาก ซาบซึ้งมาก การแสดง opinion หนึ่งไปดูโขน สองไปดูหนัง อันที่สาม ไปกินอาหารในร้านอาหาร แบบร้านค้าไม่ให้ ต้องเป็นร้านอาหารกับเพื่อน แล้ว brainstorm ว่ามีอะไรที่น่าสนใจบ้าง หรือถ้า ไม่ดี ไม่ดีอย่างไร ชอบไม่ชอบ ดีไม่ดี เขียนบรรยาย reasons ท าไมชอบไม่ชอบเพราะอะไร เช่น atmosphere, service, emotion หรือ food, variety of food หรือถ้าเขียน narrration ไปห้อง ฉุกเฉิน นั่งหนึ่งวันแล้วเลือกเหตุการณ์มาหนึ่งเรื่อง แล้วเขียนบรรยายเกิดขึ้น หรือกลับบ้าน summer ก็ไปเล่ามาว่าเจออะไรบ้าง ใช้ authentic เป็นของจริง แล้วจะเขียนได้เยอะกว่า จะ work กว่า

356

Appendix M (cont.) Sample excerpts of Ladda’s interview data (English translation for the first ten minutes)

RESEARCHER: What is the highest level of writing do you teach? LADDA: Advanced writing. RESEARCHER: Is it writing at a paragraph or essay level? LADDA: It’s the essay level. RESEARCHER: Five paragraph writing? LADDA: Yes. RESEARCHER: How are the students? LADDA: Not good. I’m bored. Their grammar is very poor. Even though I taught them the basics and they have already learned Introduction to Writing Course. They still… But some students are OK; they have improved. It doesn’t mean that all 40, 50 or 70 students are poor. But the number of students who can write well is small. There’s still a problem. Students with good writing ability are OK. They have improved from the stage of not being able to write to the stage of being able to write a paragraph, from a short paragraph to five paragraphs. They are amazed by themselves because they thought they couldn’t do it before. But after learning, practicing and doing trail and errors until passing through to Advanced Writing Course. RESEARCHER: They can do it. LADDA: Yes. But there’s still a problem though they have taken two grammar courses. RESEARCHER: Grammar 1 and 2? LADDA: Yes. They still have a problem in writing; particularly they make errors that they shouldn’t make. For example, for the singular subjects he, she, and it the verb should be in a present tense. But they didn’t add –s into it. Another example is when we talk about a plural noun, we need –s to countable nouns such as many books, a lot of students. They didn’t add –s. They make errors even with small thing such as a, an, the. This is their common error. They hardly use a, an, the. They also make errors with part of speech. They can’t use it appropriately. I think it’s because they use a talking dictionary. I don’t know. It’s a small machine like a calculator. They can just simply key in Thai words and English words will pop up. But they don’t care for part of speech. They construct a sentence by just…. They simply pick the word they want. If a

357

student wants the word ‘modify’, he keys in [the Thai word which means ‘modify’] and the word ‘expansion’ which is a noun shows up. He then uses this word to construct a sentence ‘My mother wants to expansion her business to foreign country.’ This is wrong because ‘expansion’ is a noun. Moreover, he says ‘we hope to get’, er, ‘we hope to success’ something like this. He uses a noun as a verb. Another student says ‘people who live in this social’. Well, ‘social’ is an adjective. How can you use as a noun? It must be ‘society’. This is a common mistake. At the sentence level, they usually make a mistake with fragment. This is the most frequently found mistake among three to four mistakes which are categorised by researchers. They divided mistakes into run-on, fragment and comma splice. For Thai students, comma splice and, er, not much for run- on, and fragment are the common mistakes. It seems Thai interference has something to do with this. This is because in Thai language we can omit the subject and we can still understand it from the context. For example, for the sentence ‘there will be a royal cremation ceremony.’ Students may start the sentence with ‘will’ with no subject. Another example is when they use a complex sentence such as an adjective clause, they will make a mistake because the sentence is full of modifier. They might say ‘an ambulance which arrived this morning.’ I don’t know what it is and they put a full stop to end the sentence without showing the main verb. The sentence then becomes a fragment. I have told them many times but the problem still exists. I don’t know, don’t understand what happens in their brain. Why can’t they understand this? They have learned grammar in detail. They’ve taken two courses of grammar with very thick course books. But it seems they don’t get it, especially about a noun clause. They can’t use it. They can’t construct it. They are confused because noun clause functions as a noun which can be either subject or object. They can use object better because it comes after a verb. ‘We hope that…’ they can add anything in the sentence. However, when it is used as a subject, they can’t construct ‘what the… what the Prime Minister promised is not true.’ Something like that. That can’t do it. They are confused by an adjective clause, a noun clause and subordinating conjunction. It’s so confusing. It makes me confused. RESEARCHER: Confusing. LADDA: Mmm, students are not keen on using transition to connect sentences. I have observed… There are four, er, three major types of connectors. The first one is often neglected because we can just separate sentences using a full stop. If we use conjunctions like coordinating conjunction which helps make a compound sentence, the

358

easiest way is to use ‘fanboy’ (i.e., for, and, but, also, or, yet) to connect sentences. Students can do this but when they have to make complex sentences using subordinating conjunction, they become confused. Don’t talk about transition. They don’t know how to use ‘additionally’, ‘as a result’, and ‘consequently’ even though I have told them how to use them. They don’t dare use them. The worst is that they don’t know how to connect sentences using connectors containing prepositions such as ‘due to’ and ‘because’. They still continue the sentences using ‘of’. In a common sense, prepositions will be followed by a noun or gerund but students follow them using a sentence. When I explain them, students show their realisation. But why didn’t they use the correct form. I don’t understand. Students separate grammar usage and actual use. They can’t apply the rules. They can’t integrate them. I don’t know what happens in their brain. Otherwise, our teaching has failed since the secondary education. English teaching in this country has failed since it was started. When students come to our hands which is at the advanced level, they have to be able to write. It’s not the time to teach a, b, c again. In fact, we still have to do that. We have to teach them from the beginning. It wastes time and very exhausting. RESEARCHER: So this occurs in the writing course? LADDA: Yes, I have to push them; otherwise, they can’t write. I have to review and teach more about what they should write. It’s a boring error. I don’t know what to do. However, students who write well are good. Their writing reads well. Another problem is about the unity of ideas. Students sometimes put irrelevant sentences. They think they would go together but they are not. They do not relate to the topic. For example, I asked them to give reasons why they ate in a particular restaurant. They have to give reasons for what motivate them to eat there. I taught them to give three major supporting details. They brainstormed and came up with 1. Service, OK. Then what are under this heading. Make a list. But I will not them first. I gave them the heading and let them brainstorm by themselves. Then I got a variety of service. How can they say part of the service is…? Er, they wrote good service, smiling staff, quick serving, no need to wait long. OK, these are fine. They are under the heading of service. But they also added ‘clean toilet’ and ‘well flown tap water’ under this heading. Eh, I’m quite confused and wonder if I misunderstood and they were right or they were wrong and I’m right. But it’s not correct. I think it’s wrong. It’s irrelevant. They think that the ideas should be under the heading. However, when I read through, they are not.

359

Appendix N Sample excerpts of Ladda’s stimulated recall interview

Timespan Content 1:00.0 - 1:05.0 RESEARCHER: ท าอย่างนี้ทุกงานเลยหรือคะ LADDA: เขียน outline 2:25.0 - 2:45.0 Ladda is distracted because someone is at the door. 2:50.0 - 2:52.0 LADDA: (comment on the video/student's script?) ข้ามไปข้ามมา ลากไป จริงๆ 3:08.0 - 3:26.0 RESEARCHER: ตรงนี้พี่คิดอะไรคะ LADDA: ก็นี่ไงพูดไปหมดแล้ว RESEARCHER: ก าลังคิดว่าเขาพูดอะไร จะแก้ยังไงใช่ไหมคะ LADDA: พูดแล้ว ว่าต้องแก้ยังไง ผิดยังไง 5:10.0 - 5:12.0 RESEARCHER: ตอนนี้พี่คิดอะไรอยู่คะ ตอนที่เงียบๆ อยู่ตะกี้ LADDA: ไม่รู้ จ าไม่ได้แล้ว 5:16.0 - 5:35.0 LADDA: ก็บอกแล้วไงว่ามันเป็น verb เป็น waste ไม่ถูก ท าไมไม่เปลี่ยนไปเลย เป็น waste a lot of money 5:36.0 - 5:45.0 The researcher urges Ladda to continue listening to the think-aloud protocol. RESEARCHER: พี่ก็ว่าพี่พูดชัดแล้ว 7:34.0 - 7:36.0 The researcher tries to ask a question of what Ladda was doing. LADDA: That's what I said. I tried to score the writing. [Ladda seems a bit annoyed by the researcher's questions.] 8:08.0 - 8:40.0 RESEARCHER: ตอนนี้ก าลังคิดหรือเปล่าคะว่าจะให้คะแนนอะไร เท่าไหร่ดี LADDA: ก าลังดูว่าเนื้อหาทั้งหมดมันเป็นยังไง สอดคล้องกับ outline ไหม แล้วก็ จะให้คะแนนแล้ว RESEARCHER: กวาดตาดูอีกรอบเหรอคะ LADDA: ว่าเขามี point ประเด็นไหน ที่พูดไปไงว่าเนื้อหาดี ใช้ค าถูกต้อง ไม่มี fragment ไวยากรณ์ผิดนิดหน่อย เอา 7.5 แล้วกัน 9:00.0 - 9:06.0 เนื้อหาก็โอเค แต่ว่ามันไม่มี elaboration อธิบายไม่ชัด

360

Timespan Content 11:08.0 - 12:33.0 RESEARCHER: เวลาที่พี่คิดว่าจะแก้ยังไงนี่ พี่หา choice ของค าแล้วเลือกค าที่ ดีที่สุด หรือค ามันจะขึ้นมาในหัวเอง LADDA: อืม ค าจะมาเองว่า structure แบบนี้ ไม่ควรเขียนแบบนี้ ควรจะเขียน อีกแบบนึง ค าจะขึ้นมาเอง แต่พี่จะไม่แก้ให้หมดหรอก ส่วนใหญ่พี่จะวงๆๆ เพราะว่าถ้าแก้ให้ทุกอย่าง เดี๋ยวเขาจะเสียนิสัย แล้วคิดเองไม่เป็น ในระดับนี้เขา เขียนได้แค่นี้ก็โอเคแล้ว อย่าไปคาดหวัง ขอแค่เขาเขียนให้เราอ่านรู้เรื่องก็พอ อย่า เป็น fragment เป็นอะไรก็พอแล้ว แต่บางทีมันแปลจากภาษาไทยเกินไปไง มันก็ รับไม่ได้ไง แต่จะไปบอก ไปเจาะจงว่าให้เขียนเหมือนที่ฝรั่งเขาเขียน มันก็เจาะจง เกินไปไง มันก็ไม่รู้ว่าฝรั่งเขาเขียนกันยังไง การอ่านมันน้อยน่ะ มันต้องอ่านเยอะๆ อ่านหนังสือเยอะๆ จะถึงจะรู้ว่าลีลาการเขียนแบบนี้ ฝรั่งเขาจะพูดว่ายังไง เช่น กินยา แล้วจะบอกว่า eat medicine ก็ไม่ได้ ฝรั่งเขาบอกว่า take ไปเข้าสอบ มัน ก็จะบอก go enter the exam อะไรอย่างนี้ มันก็ไม่ใช่ take the exam, take the midterm test, take the lobal test อะไรอย่างนี้ 12:20.0 - 12:59.0 RESEARCHER: ตอนนี้ก าลังกลุ้มหรือเปล่าคะของคนนี้ LADDA: กลุ้มๆ RESEARCHER: กลุ้มเพราะอะไรคะ LADDA: ก็มันเขียนไม่เป็นภาษาน่ะ 13:08.0 - 13:12.0 Ladda comments on the think-aloud activity: มันไม่เป็นธรรมชาติไง ตรวจ แล้วพูดเสียงดัง 13:18.0 - 13:32.0 LADDA: เพราะปกติพี่จะดูไป แล้วก็คิดไปในใจ แล้วก็วิเคราะห์วิจารณ์อะไรของ พี่ไปในใจไง ไม่พูดออกมา แต่นี่แปลกๆ เพราะมันไม่เป็นธรรมชาติไง แล้วแถมมี กล้องอีกต่างหาก 14:00.0 - 14:04.0 RESEARCHER: LADDA: ไม่รู้ ไม่รู้แล้ว มันเป็น action เป็นอิริยาบถ 15:52.0 - 15:55.0 [Ladda laughs at her gestures in the video.] LADDA: ท าไมมือฉันอยู่ไม่สุข

361

Timespan Content 16:04.0 - 17:17.0 RESEARCHER: ท าไมต้องมี adverb ขยายล่ะคะพี่ LADDA: เอ้า มันจะได้เพราะขึ้นไง ไม่งั้นภาษามันแข็ง มันใช้ไม่เป็นเลยพวกนี้น่ะ adverb of manner, adverb of frequency มันเรียบมากเลย มันท าให้ภาษา สละสลวยขึ้นตั้งเยอะ we sometimes อะไรอย่างนี้ แทนที่จะบอกว่า We go to the market. ก็จะเป็น We sometimes go to the market. ...แทนที่จะบอก They take a card to show the teacher. They simply เพียงแค่อะไรมันใช้ไม่ เป็นไง They simply...show ID card ไปเลย นี่อะไร ภาษาไทย เอาไปให้ครูดู ก็ บอก show ให้เขาดูเลยไง ไม่ต้องบอกเอา take show อะไรอย่างนี้ เยิ่นเย้อ go go sleep, go swim อะไรอย่างนี้ พี่พูดหลายทีแล้ว แต่มันก็ไม่ get 18:00.0 - 18:02.0 Ladda gave up watching the video recording and talked about something else. Ladda finally told the researcher to stop the recording. 18:50.0 - 18:53.0 RESEARCHER: While marking, what did you think about? Course objectives? LADDA: คิดว่าเขาเขียนตรงตาม outline ไหม ตรงตามค าสั่งไหม ตรงหัวข้อไหม ไม่ได้คิดถึงอย่างอื่น เช่น จุดประสงค์รายวิชาหรืออะไรอย่างอื่นเลย 18:54.0 - 20:00.0 RESEARCHER: การให้ feedback ของครูที่โชว์บนจอมีผลกระทบอะไรกับเด็ก หรือไม่ LADDA:น้อย เหมือนเขาไม่รู้ว่าผิดอะไรอย่างไร RESEARCHER: ถึงแม้จะกาให้แล้วเหรอคะ LADDA: อ๋อ อันนี้คงดีขึ้นแหละ อย่างน้อยเขาคงให้เพื่อนช่วยด้วย ให้คนเก่งๆ ช่วย

362

Appendix N (cont.) Sample excerpts of Ladda’s stimulated recall interview (English translation for the first ten minutes)

Timespan Content 1:00.0 - 1:05.0 RESEARCHER: Do you do this with all assignments? LADDA: Write the outline. 2:25.0 - 2:45.0 Ladda is distracted because someone is at the door. 2:50.0 - 2:52.0 LADDA: (comment on the video/student's script?) Forward, backward. 3:08.0 - 3:26.0 RESEARCHER: What were you thinking here? LADDA: I’ve said it all. RESEARCHER: Were you thinking of what they were talking and how to correct it? LADDA: I’ve said what was wrong and how to correct it. 5:10.0 - 5:12.0 RESEARCHER: What were you thinking when you kept silent? LADDA: I don’t know. I can’t recall. 5:16.0 - 5:35.0 LADDA: I said it is a verb, ‘waste’, wrong. Why don’t you change into ‘waste a lot of money’? 5:36.0 - 5:45.0 R urged P to conticue listening to the think-aloud protocol. RESEARCHER: I think I said clear enough. 7:34.0 - 7:36.0 R tried to ask a question of what she was doing. LADDA: That's what I said. I tried to score the writing. [P seems a bit annoyed by the researcher's questions.] 8:08.0 - 8:40.0 RESEARCHER: Were you thinking how much you should score? LADDA: I was thinking about the overall content. How is it? Is it coherent with the outline? Then, I would score. RESEARCHER: Skimming through the writing again? LADDA: To see what points they have. As I said, good content, appropriate word choice, no fragment, a few grammatical errors. Then 7.5. 9:00.0 - 9:06.0 Content is OK but there is no good elaboration. Unclear explanation.

363

Appendix O Sample of Ladda’s marked writing scripts

Student 1

364

Ill Ca1·ds of Children

When I was 15 years old, 1 went to Phatthalung District

Office for getting my first ro card. However, my younger

hrother who is seven years old now has to have 10 card because

of The lD Cards Act 20 I I. In my opinion, maturity, rnigrant

workers and wasl(oney arc the reasons why children

shouldn't hold the lD cards.

.fir st, the children don't have enough matu....----ri\y. For this_... ~- reason, they don' t know the llYcards arc very important It's

easy for them to lose ID cards. T hey can't be trusted to cany <:'\ e/ \ . theirlD cards and not losphem. They may think the 10 cards arc l - - - C!PJ r<::9ince the 1D cards 1 ?0?)?_Jj_~ 't:~~2~a rds that are popular ~Vi~ .,._ for the children. Conseq~~n~ l y , th;jf) cards arc worn out easily J j--·· ·

because of playing of child ren.

Second, there arc many migrant workers in Thailand. They

want to g~ t sl-ate wel fare, such as educati on, hcalthcare and

365

~~~,.,·~'\"'!'"§'---·.. -· ·---·.. -···- -- ... - ... ··-·-.- ... -

housing. Therefore, thC~~~~~~or~~~)------to get hold of the () illegal cards. if the children must have the rD canis, it's not I difficult for the young migrant workers to take this opportunity

to get lbe ID cards. <:' I . J a l.f7vv·· Finally, ID cawltrthe child'1r wast~o ney . Parcn" normally keep children'<; birth cerjfica!es and household

registration, so !he children don't need to have the lD cards to 1~~0>a<< nl' ' money. Ae<:<>rding to ·n.,irath Newspaper, Thai ' Government allocates budget for the lD cards of the children 191

million bah t. Therefore, the 10 cards of the children arc very wasted. L(pv?l.. fJv-(_ . In short, there arc three reasons why the children don' t

need to have the rD cards. One is that children have no enough

maturity. Another is that i t'~ easy for the young migrant workers

to take this opporruniry_.to..geWh.eJ..Q cards. The last is that il's tJ( 1J.C-vl .h;..- ( / -- _us~r oney. '-~._/ 7

366

Appendix P Sample of Nittaya’s marked writing scripts

Student 11

367

Appendix Q Rewadee’s writing course syllabus

Subject code [the code number] Credit 3(3-0) Course title Paragraph writing Course condition - Course status Elective course Curriculum English Degree Bachelor Teaching hours Lecture 45 hours Practicum -- hours Total 45 hours Lecturers [Name of lecturers]

Course description Students practise writing different types of paragraphs for a meaningful communication and writing sentences containing main clauses with proper discourse connectors.

Objectives By the end of the course, students should be able to do the following: 1. Identify the components of a sentence. 2. Build up a simple, compound and complex sentence. 3. Identify component and types of paragraphs. 4. Write different types of paragraphs effectively.

Date/Time Class Contents Activities Materials meeting Week 1-2 1-2 - Pre-test - Question and answer - Computer - Orientation - Writing a 100 word presentation Introduction to the course paragraph - Handbook outline - Doing exercises - Classification of sentences Week 3-4 3-4 - Classification of sentences - Lecture - Computer - Components of paragraph - In class writing presentation - Process in writing workshop - Handbook - Doing exercises Week 5 5 - Introduction to narrative - Lecture - Computer paragraph - Study model paragraph presentation - Writing daily routine and its components - Handbook - Practice prewriting, brainstorming and revising

Week 6 6 - Writing storytelling - Lecture - Computer paragraph - Study model paragraph presentation and its components - Handbook - Practice prewriting, brainstorming and revising

368

Date/Time Class Contents Activities Materials meeting Week 7 7 - Writing autobiography - Lecture - Computer paragraph - Study model paragraph presentation and its components - Handbook - Practice prewriting, brainstorming and revising Week 8 Midterm exam Week 9 9 - Introduction to descriptive - Lecture - Computer paragraph - Study model paragraph presentation - Writing descriptive and its components - Handbook paragraph - Practice prewriting, brainstorming and revising Week 10 10 - Writing descriptive - Lecture - Computer paragraph - Study model paragraph presentation and its components - Handbook - Practice prewriting, brainstorming and revising Week 11 11 - Writing descriptive - Lecture - Computer paragraph - Study model paragraph presentation and its components - Handbook - Practice prewriting, brainstorming and revising Week 12 12 - Introduction to - Lecture - Computer comparison/contrast - Study model paragraph presentation paragraph and its components - Handbook - Writing - Practice prewriting, comparison/contrast brainstorming and paragraph revising Week 13 13 - Writing - Lecture - Computer comparison/contrast - Study model paragraph presentation paragraph and its components - Handbook - Practice prewriting, brainstorming and revising Week 14 14 - Writing - Lecture - Computer comparison/contrast - Study model paragraph presentation paragraph and its components - Handbook - Practice prewriting, brainstorming and revising Week 15 15 - Review - Lecture - Computer - Post-test - Post-test presentation - Handbook Final exam Evaluation criteria 1. Attendance & participation 10% 2. Assignments 20% 3. Quiz 10% 4. Test 20% 5. Midterm test 20% 6. Final exam 20% Total 100%

369

Criteria for grading Grade Score A 80-100 B+ 75-79 B 70-74 C+ 65-69 C 60-64 D+ 55-59 D 50-54 F 0-49

Teaching materials/References [a list of teaching materials and/or references]

370

Appendix R Sample of Rewadee’s marked writing scripts

Student 3

371

Appendix S Sak’s guided essay

Vacationing at the Beaches or the Mountains21 It is common thing that people around the world go to visit tourist attractions on their vacations. They often go to the tourist attractions where they can take their all family members with them for they can do activities together. Nowadays, it seems that the most popular places where a great number of the people go vacationing are beaches and mountains. This is because they can enjoy doing a lot of things together at both places. However, to make a decision to choose one to visit might be difficult. Which one has the right feature and math our favourites? When we compare and contrast the both places, there are some differences and some similarities between the beaches and the mountains in aspects of the activities the tourists can do, the climate, and the locations. The first differences and similarities between the mountains and the beaches are the activities we can do there. The activities which we can do at the mountains are the adventure activities while we do the simple ones at the beach. When we go to the mountain, we usually enjoy some hard activities like rock climbing, hiking, camping, and mountain biking. But the activities we like to do at the beaches are quite simple and soft such as building sand castles, playing beach volleyball and football, drawing or writing on the sand, and collecting shells. On the other hand, there are some similarities of the activities between the both places; cooking, playing guitar, and taking photos. When we go to the mountains or the beaches, we can cook the similar food such as barbeque, grilled meat, and salad. We can play guitar and sing as well on the mountains when we enjoy fire camping at the night time; similarly, we can do that when we go to the beaches. Next, we never missed to bring the camera to take photos when we go on our vacations because we can remind the old days and a good time in the past by looking at the pictures. So, we always take photos on vacationing the mountains and the beaches. The second differences and similarities between the two places are climate. The first different climate is that the mountains have fresh dry air while the beaches have humid air. The air around the mountains is produced by trees, so it is very fresh and

21 This essay is typed as exactly shown in the original script.

372

clean. But the wind around the beach is blown from the sea, so it is very hot and humid. Next, the mountains are very shady but the beaches are very sunny. The mountains consist of a great number of trees. Those trees make shadows in all area of the mountain. In contrast, the beaches are the long plain sand area and have not any shadow since another sides end with the sea. The sun shines directly to the beaches and makes it very sunny. The last difference of climate is that the mountains are cooler while the beaches are warmer. As mention above, there are a lot of shadows in the mountains and the trees produce the fresh air. So, the climate around the mountains is cooler than one around the beaches since the beaches openly receive the hot wind blown from the sea and sunshine from the sky. On the other hand, the similarity of the climate between the two places is that they are windy. In the mountains, there are a lot of trees that always produce the wind blowing all around the mountains. This seems to be similar to the beach. The hot wind always blows from the sea to the beaches and makes the beaches breezy and windy. For the last comparison, there are some differences in aspects of the location between the mountains and the beaches. Firstly, the mountains have long view with trees and hills, whereas the beaches have wide view with sand and water. Another different aspect of location is that the mountains are far away from downtowns while the big cities have been located along the beaches. There are a lot of buildings like restaurants, department stores, hotels, and resorts built along the beaches, while there are not such buildings along the mountains. The last unlike feature of location is that the mountains are rocky but the beaches are sandy. In conclusion, these are the three different and familiar features between the mountains and the beaches; the activities, the climate, and the locations. People can make a decision where to go vacationing between the two places by considering from their differences and similarities. Nevertheless, which one is better; it depends on one’s preference.

373

Appendix T Sample of Sak’s marked writing scripts

374