Development and Evaluation of Computer‐Aided Assessment in Discrete and Decision Mathematics A Thesis Submitted for the Degree of Doctor of Philosophy by Kinga Zaczek School of Information Systems, Computing and Mathematics Brunel University September 2014 I ABSTRACT This thesis describes the development of Computer‐Aided Assessment questions for elementary discrete and decision mathematics at the school/university interface, stressing the pedagogy behind the questions’ design and the development of methodology for assessing their efficacy in improving students’ engagement and perceptions, as well as on their exams results. The questions give instant and detailed feedback and hence are valuable as diagnostic, formative or summative tools. A total of 275 questions were designed and coded for five topics, numbers, sets, logic, linear programming and graph theory, commonly taught to students of mathematics, computer science, engineering and management. Pedagogy and programming problems with authoring questions were resolved and are discussed in specific topic contexts and beyond. The delivery of robust and valid objective questions, even within the constraints of CAA, is therefore feasible. Different question types and rich feedback comprising text, equations and diagrams that allow random parameters to produce millions of realisations at run time, can give CAA an important role in teaching mathematics at this level. Questionnaires identified that CAA was generally popular with students, with the vast majority seeing CAA not only as assessment but also as a learning resource. To test the impact of CAA on students’ learning, an analysis of the exam scripts quantified its effect on class means and standard deviations. This also identified common student errors, which fed into the question design and editing processes by providing evidence‐based mal‐rules. Four easily‐identified indicators (correctly‐written remainders, conversion of binary/octal/hexadecimal numbers, use of correct set notation {…} and consistent layout of truth tables) were examined in student exam scripts to find out if the CAA helps students to improve examination answers. The CAA answer files also provided the questions’ facilities and discriminations, potentially giving teachers specific information on which to base and develop their teaching and assessment strategies. We conclude that CAA is a successful tool for the formative/summative assessment of mathematics at this level and has a positive effect on students’ learning. II Acknowledgements First and foremost I want to thank my supervisor Dr Martin Greenhow for supporting me during this PhD. His guidance, time and patience with me are much appreciated. He always believed in me and motivated in tough moments. His enthusiasm, smile, good sense of humour and understanding allowed me to complete this research. I want to thank Dr Steven Noble, my second supervisor, for his support with the research, administration staff for friendly atmosphere and finding the necessary data, computer support officers for help anytime something was not working, all the lecturers and fellow PhD students for good word and rich conversations. I also thank my friends and groups I was a part of. I met fantastic people on squash courts, during Polish Society meetings, but also when living in the hall. We had fun times but also very supportive conversations if needed. Friends outside the university gave me not any less support. They motivated me, believed in me and were always for me when I needed them. Last, but not least, I thank my family for all their love and encouragement. For my parents who raised me as a person who respects values. They gave me the opportunity of studying in the UK hoping it will give me a better life in future. They, my sister and my beloved grandparents made me feel loved and special. I am happy they feel proud of me. I would not make it this far without you. Thank you. III Contents Abstract …………………………………………………………………………………………………………………………. I Acknowledgements ………………………………………………………………………………………………………. II Contents ……………………………………………………………………………………………………………………..… III List of Figures ………………………………………………………………………………………………………………... IX List of Tables ………………………………………………………………………………………………………………….. XII 1. Introduction …………………………………………………………………………………………………………... 1 1.1. Types of assessments ………………………………………………………………………………….. 1 1.2. CAA in mathematics in Higher Education ……………………………………………………… 3 1.3. Technology used …………………………………………………………………………………………… 4 1.4. Research questions ………………………………………………………………………………………. 5 1.5. Methodology ……………………………………………………………………………………………….. 5 1.6. Structure of the thesis ……….. ………………………………………………………………………… 7 2. Discrete and Decision Mathematics questions ………………….……………………………………. 9 2.1. Target group …………………………………………………………………………………………………. 9 2.2. Question database and general appearance ………………………………………………… 9 2.3. Question types ……………………………………………………………………………………………... 14 2.4. Randomisation ……………………………………………………………………………………………… 16 2.5. Feedback ………………………………………………………………………………………………………. 18 3. Numbers …………………………………………………………………………………………………………………. 20 3.1. Prime factorisation ………………………………………………………………………………………. 20 3.1.1. Assumed and tested skills …………………………………………………………………. 20 3.1.2. Objective questions ………………………………………………………………………….. 22 3.1.3. Feedback ………………………………………………………………………………………….. 23 3.1.4. Technical content …………………………………………………………………………….. 24 3.2. Modular arithmetic ………………………………………………………………………………………. 26 3.2.1. Assumed and tested skills ………………………………………………………………… 26 3.2.2. Objective questions ………………………………………………………………………….. 27 3.2.3. Feedback ………………………………………………………………………………………….. 27 3.2.4. Technical content …………………………………………………………………………….. 28 3.3. Non‐decimal arithmetic ……………………………………………………………………………….. 28 3.3.1. Assumed and tested skills …………………………………………………………………. 29 3.3.2. Objective questions ………………………………………………………………………….. 30 IV 3.3.3. Feedback ……………………………………………………………………………………………31 3.3.4. Technical content ……………………………………………………………………………… 33 3.4. Scientific notation …………………………………………………………………………………………. 35 3.4.1. Assumed and tested skills …………………………………………………………………. 35 3.4.2. Objective questions ………………………………………………………………………….. 36 3.4.3. Feedback ………………………………………………………………………………………….. 37 3.4.4. Technical content …………………………………………………………………………….. 38 4. Sets …………………………………………………………………………………………………………………………. 39 4.1. Elements ……………………………………………………………………………………………………….. 39 4.1.1. Assumed and tested skills …………………………………………………………………. 41 4.1.2. Objective questions ………………………………………………………………………….. 42 4.1.3. Feedback ………………………………………………………………….………………………. 42 4.1.4. Technical content ……………………………………………………………………………… 43 4.2. Subsets and set equality ……………………………………………………………………………….. 46 4.2.1. Assumed and tested skills …………………………………………………………………. 46 4.2.2. Objective questions ………………………………………………………………………….. 48 4.2.3. Feedback ……………………………………………………………………………………………48 4.2.4. Technical content …………………………………………………………………………….. 49 4.3. Cardinality …………………………………………………………………………………………………….. 52 4.3.1. Assumed and tested skills …………………………………………………………………. 53 4.3.2. Objective questions ………………………………………………………………………….. 53 4.3.3. Feedback ………………………………………………………………………………………….. 54 4.3.4. Technical content …………………………………………………………………………….. 55 4.4. Power sets …………………………………………………………………………………………………….. 56 4.4.1. Assumed and tested skills …………………………………………………………………. 57 4.4.2. Objective questions ………………………………………………………………………….. 58 4.4.3. Feedback ……………………………………………………………………………………………59 4.4.4. Technical content ……………………………………………………………………………… 60 4.5. Cartesian product ………………………………………………………………………………………….. 62 4.5.1. Assumed and tested skills …………………………………………………………………. 62 4.5.2. Objective questions ………………………………………………………………………….. 62 4.5.3. Feedback ……………………………………………………………………………………………63 4.5.4. Technical content ……………………………………………………………………………… 64 4.6. Union and intersection ………………………………………………………………………………….. 66 4.6.1. Assumed and tested skills …………………………………………………………………. 67 V 4.6.2. Objective questions ………………………………………………………………………….. 68 4.6.3. Feedback ……………………………………………………………………………………………70 4.6.4. Technical content ……………………………………………………………………………… 71 4.7. Difference …………………………………………………………………………………………………….. 72 4.7.1. Assumed and tested skills …………………………………………………………………. 73 4.7.2. Objective questions ………………………………………………………………………….. 73 4.7.3. Feedback ……………………………………………………………………………………………74 4.7.4. Technical content ……………………………………………………………………………… 74 4.8. Complement …………………………………………………………………………………………………. 76 4.8.1. Assumed and tested skills …………………………………………………………………. 76 4.8.2. Objective questions ………………………………………………………………………….. 77 4.8.3. Feedback ……………………………………………………………………………………………78 4.8.4. Technical content ……………………………………………………………………………… 79 4.9. Partition ………………………………………………………………………………………………………… 80 4.9.1. Assumed and tested skills …………………………………………………………………. 81 4.9.2. Objective questions ………………………………………………………………………….. 81 4.9.3. Feedback ……………………………………………………………………………………………83 4.9.4. Technical content ……………………………………………………………………………… 84 4.10. Algebra (Set identities) ………………………………………………………………………………….. 85 4.10.1. Assumed and tested skills …………………………………………………………………. 86 4.10.2. Objective questions ………………………………………………………………………….. 88 4.10.3. Feedback ……………………………………………………………………………………………88 4.10.4. Technical content ……………………………………………………………………………… 89 4.11. Venn diagrams ……………………………………………………………………………………………… 91 4.11.1. Assumed and tested skills …………………………………………………………………. 91 4.11.2. Objective questions ………………………………………………………………………….