UCL University College London Department of Computer Science EVALUATION OF COMPUTER AIDED INSTRUCTION: ASSESSING THE VALUE AND EFFECTIVENESS OF OPERATIONAL SYSTEMS Arif Mahmud Iqbal A thesis submitted for the degree of Master of Philosophy in the University of London July 2003 ProQuest Number: U642301 All rights reserved INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed, a note will indicate the deletion. uest. ProQuest U642301 Published by ProQuest LLC(2015). Copyright of the Dissertation is held by the Author. All rights reserved. This work is protected against unauthorized copying under Title 17, United States Code. Microform Edition © ProQuest LLC. ProQuest LLC 789 East Eisenhower Parkway P.O. Box 1346 Ann Arbor, Ml 48106-1346 ABSTRACT This thesis investigated a number of performance measures for computer-aided instruction (CAT) systems. These "evaluation metrics" are intended to assess the worth and value of teaching systems. An operational accounting tutor (which teaches marginal costing) was used to develop the metrics and a replication study was conducted on Application Program Tutor (a tutoring system designed to teach courses). Although, CAI is a mature technology which has evolved in a variety of fields and forms since the 1950s, its potential remains untapped. Factors attributed to this include resistance from teachers, lack of student involvement in design, and insufficient imagination in curriculum design. Inadequate system standards and a deficiency of good software tools, lack of documentation, maintenance and education value have also been key limiting factors. The overall picture seems akin to a cottage industry than a co-ordinated enterprise. Evaluation is significant, to developers and users in this field, because in the short-term it improves the usability and life-span of the numerous systems that have been developed and in the long-term it focuses attention (away from the impetus to deliver) towards issues of appropriateness and quality in system design. Different traditions of evaluation are explored, including the selection criteria used in educational technology and the impact of the quality philosophy on software engineering. This research was conducted using the Before-after Two-group design on forty-two accountancy students, where a conventionally taught group was compared with the accountancy tutor group. Performance on a number of marginal (or variable) costing problems was measured before and after both groups were taught. Moreover, the experimental group was given a questionnaire to complete (which was designed to capture their assessment of the system). The results derived from the well- crafted questionnaire were indicative of the system’s strengths and weaknesses and supplied useful criteria for future research. - 2 - DEDICATION I would like to thank my parents, Muhammad Iqbal and Sharifan Iqbal for their love and support. I would also like to dedicate this thesis to them, especially my mother since she was my original teacher and guide (and continues to be). Arif Iqbal, September 1996. - 3 ACKNOWLEDGEMENTS Although my name is the only one on this work there are many other people who directly or indirectly contributed to the creation of this thesis, and just as it takes a team to make a dream come true, so I must acknowledge all my colleagues and seniors who made it possible. I certainly could not have completed this thesis without the support of my supervisor. Professor Paul Samet and many thanks are due to him for taking me through both the agony and ecstasy of the research process. I would also like to thank the following Kinshuk and Ashok Patel of the Department of Accountancy at De Montfort University for their generosity; Paul Wernick for introducing me to the Marginal Costing system and Wanderley Lobianeo for his help in proof-reading and advice; and credit is due to John Cook (at Thames Valley University) and Mark-Elsom Cook (at the Electric Brain Company) for providing help and the loan of their system. Many thanks for the help given by the students who participated and the permission provided by Julian Lavis of the Faculty of Business, Management and Social Studies at University of Westminster, Ian Potts of the School of Accountancy at Thames Valley University to carry out the study and loan the systems for the study and especially for Andrew Scott of the Management Centre at University College London for helping out with the study. Jack Levy and Alan Shaw both of the Information Systems Division at University College London who were instrumental in making my disks run on the hardware used in the study and Naflsa Taylor for helping me with WordPerfect (Version 5.1). - 4 - I am also indebted to my brother for his financial support and encouragement through the good, bad and ugly periods of my writing up. I am consistently reminded of that "Fear can hold you prisoner. Hope can set you free." The work carries the warning "Brains are not included - but are necessary". There are many others, too numerous to mention, including some of my research colleagues within the Department of Computer Science most notably Kapalandu Pal, David Fulton and Angela Sasse who have bestowed their friendship and advice, I am extremely grateful to these unsung heroes. Three special individuals have had the misfortune to call me their friend during my research work. All of them shared my sorrows and happiness and two other things bound them together - a steadfastness belief in me and a true friendship as well as a first name that began with the letter "K" (i.e. Khurshid, Kinshuk and Kamran). I hope they know how much they mean to me (especially the last person), as the British poet, Alexander Pope reminded us "In every friend we lose a part of ourselves, and the best part". This research was carried out under a Science and Engineering Research Council studentship. Arif Iqbal, September 1996 (revised July 2003). - 5 - CONTENTS ABSTRACT ....................................................................................................................... 2 DEDICATION..................................................................................................................... 3 ACKNOWLEDGEMENTS .............................................................................................. 4 CHAPTER ONE ............................................................................................................... 14 1.1 The research problem .................................................................................. 14 1.1.1 The Problem Outlined .................................................................. 14 1.1.2 The Gap in the Literature ........................................................... 14 1.1.3 Why is it im portant? ..................................................................... 15 1.2 Focus of this research .................................................................................. 15 1.2.1 Scope of this th esis ....................................................................... 15 1.2.2 The Main Objectives..................................................................... 17 1.2.3 The Research Approach and its Boundaries ............................ 19 1.2.4 Significance of this R esearch ...................................................... 22 1.3 Structure of thesis ......................................................................................... 23 1.4 Author’s n o te ................................................................................................. 26 1.5 Key Points ...................................................................................................... 26 CHAPTER TWO ............................................................................................................. 28 2.1 The promise of powerful educational technology .................................... 28 2.2 Computers in Education ............................................................................... 31 2.3 A brief history of CAT ............................................................................... 33 2.4 Current realities of education and educational software .......................... 36 2.5 Brave new world of CAI - multimedia and the Internet ....................... 43 2.6 Evaluation of CAI - The Road to Better Software ................................. 49 2.6.1 Pitfalls in Evaluative Design ...................................................... 51 2.6.2 When does the Evaluation end and the Teaching begin? .... 55 - 6 - 2.6.3 Instructor versus Evaluator .......................................................... 57 2.6.4 Establishing the goals of educational software .......................... 63 2.7 Key Points ..................................................................................................... 64 CHAPTER THREE ..................................................................................................... 66 3.1 What are Intelligent Tutoring Systems? ................................................... 66 3.2 History of IT S s .............................................................................................. 68 3.3 Examples of Classic IT S s ............................................................................ 75 3.3.1 SCHOLAR ...................................................................................