A HIERARCHY OF GRAMMATICAL DIFFICULTY FOR JAPANESE EFL LEARNERS: MULTIPLE-CHOICE ITEMS AND PROCESSABILITY THEORY A Dissertation Submitted to the Temple University Graduate Board In Partial Fulfillment of the Requirements for the Degree of Doctor of Education By Atsuko Nishitani August, 2012 Examining Committee Members David Beglar, Advisory Chair, CITE/TESOL Yoko Kozaki, External Member, Mukogawa Women’s University Mark Sawyer, External Member, Kwansei Gakuin University Steven Ross, External Member, University of Maryland Edward Schaefer, External Member, Ochanomizu University © Copyright 2012 by Atsuko Nishitani iii ABSTRACT This study investigated the difficulty order of 38 grammar structures obtained from an analysis of multiple-choice items using a Rasch analysis. The order was compared with the order predicted by processability theory and the order in which the structures appear in junior and senior high school textbooks in Japan. Because processability theory is based on natural speech data, a sentence repetition test was also conducted in order to compare the result with the order obtained from the multiple- choice tests and the order predicted by processability theory. The participants were 872 Japanese university students, whose TOEIC scores ranged from 200 to 875. The difficulty order of the 38 structures was displayed according to their Rasch difficulty estimates: The most difficult structure was subjunctive and the easiest one was present perfect with since in the sentence. The order was not in accord with the order predicted by processability theory, and the difficulty order derived from the sentence repetition test was not accounted for by processability theory either. In other words, the results suggest that processability theory only accounts for natural speech data, and not elicited data. Although the order derived from the repetition test differed from the order derived from the written tests, they correlated strongly when the repetition test used ungrammatical sentences. This study tentatively concluded that the students could have used their implicit knowledge when answering the written tests, but it is also possible that students used their explicit knowledge when correcting ungrammatical sentences in the repetition test. The difficulty order of grammatical structures derived from this study was not in accord with the order in which the structures appear in junior and senior high school textbooks in Japan. Their iv correlation was extremely low, which suggests that there is no empirical basis for textbook makers’/writers’ policy regarding the ordering of grammar items. This study also demonstrated the difficulty of writing items testing the knowledge of the same grammar point that show similar Rasch difficulty estimates. Even though the vocabulary and the sentence positions were carefully controlled and the two items looked parallel to teachers, they often displayed very different difficulty estimates. A questionnaire was administered concerning such items, and the students’ responses suggested that they seemed to look at the items differently than teachers and what they notice and how they interpret what they notice strongly influences item difficulty. Teachers or test-writers should be aware that it is difficult to write items that produce similar difficulty estimates and their own intuition or experience might not be the best guide for writing effective grammar test items. It is recommended to pilot test items to get statistical information about item functioning and qualitative data from students using a think-aloud protocol, interviews, or a questionnaire. v ACKNOWLEDGMENTS I would like to express my sincerest gratitude to my advisor, Dr. David Beglar, for his support and guidance throughout my dissertation writing process. He not only helped me with the design and statistical analysis of the study but also read my drafts countless times and corrected many of the stylistic, formatting, and grammatical errors. Without his careful reading and generous support, I would not have finished this project. It has been my great honor to have him as my chief dissertation advisor. I would also like to express my appreciation to the other members of my dissertation committee, Dr. Yoko Kozaki, Dr. Mark Sawyer, Dr. Steven Ross, and Dr. Edward Schaefer. Their insightful comments and invaluable suggestions helped me improve the final draft of my dissertation. I am also grateful to Dr. Noel Houck, who served as a dissertation-proposal committee member. Her suggestion to conduct a sentence repetition test made this study more valuable. I would like to extend my gratitude to all the teachers and students who kindly let me use their precious class time and helped me collect data. Without their cooperation, my study could not have been completed. Finally, I would like to thank my parents for their unconditional love and support throughout my life, and my four Chihuahuas, Lil, Tinny, Mona, and Niña, for quietly sitting around me while I was spending hours at a computer working on this dissertation. I hope I can spend more fun time with them from now on. vi TABLE OF CONTENTS PAGE ABSTRACT ....................................................................................................................... iv ACKNOWLEDGMENTS ................................................................................................. vi LIST OF TABLES ............................................................................................................ xii LIST OF FIGURES ......................................................................................................... xiv CHAPTER 1. INTRODUCTION ...........................................................................................................1 The Background of the Issue ...................................................................................1 Statement of the Problem ......................................................................................... 3 Purposes of the Study ..............................................................................................5 Significance of the Study .........................................................................................6 Theoretical Perspective ............................................................................................7 The Audience for the Study .....................................................................................7 Delimitations ............................................................................................................8 The Organization of the Study ...............................................................................10 2. REVIEW OF THE LITERATURE ...............................................................................11 Morpheme Studies .................................................................................................11 Negatives................................................................................................................14 Interrogatives .........................................................................................................15 Relative Clauses .....................................................................................................16 Natural Order .........................................................................................................20 Implicit/Explicit Knowledge and Grammatical Difficulty ....................................24 vii Implicit/Explicit Knowledge and Processability Theory .......................................27 Fill-in-the-Blank Activities and Processability Theory .........................................31 Timed/Untimed Tests and Implicit/Explicit Knowledge .......................................32 Gaps in the Literature.............................................................................................33 Purposes of the Study and Research Questions .....................................................34 3. METHODS ....................................................................................................................36 Participants .............................................................................................................36 Instrumentation ......................................................................................................38 Test 1 ...........................................................................................................38 Test 2 ...........................................................................................................42 Test 3 ...........................................................................................................43 Test 4 ...........................................................................................................44 Test 5 ...........................................................................................................45 Test 6 ...........................................................................................................47 Sentence Repetition Test .............................................................................52 Procedures ..............................................................................................................55 The Order of Grammatical Structures in Textbooks ...................................60 Analyses .................................................................................................................60 The Rasch