A Learning Environment for First Year Software Engineers

Mark Ratcliffe, Lynda Thomas and John Woodbury Department of Computer Science University of Wales, Aberystwyth, UK {mbr, ltt, jyw}@aber.ac.uk

Abstract students’ ability to come to grips with the This paper reports on ongoing work in the basic concepts of programming and design Department of Computer Science at the [Thomas, 2000]. This involved her University of Wales. It describes a learning application of ideas similar to Richard environment (Software Engineering Felder’s work with Chemical Engineers Teaching and Learning System - SETLAS) [Felder, 1996] to the problems of beginning that has been successfully used with first Software Engineers. year software engineering students and that Aberystwyth is also the centre of the is the result of a synergy between several MONET network, which focuses on Model groups within the department. Based Reasoning. This community includes The resulting system has exceeded the researchers elsewhere who are also objectives for which it was initially designed interested in the use of models in Software and improved student performance and Engineering [Bredeweg, 1998]. motivation. These groups have all come together to apply their expertise to the creation of an integrated environment for teaching first 1 Introduction year programming and design. In the early 1990’s, the United Kingdom’s Development is still underway, but the move to a more modular approach to system’s existing functionality has proved education led to more frequent examination. enough to improve students’ performance Not surprisingly, the process of revising for and motivation. The next step is the examinations and receiving feedback was incorporation of Model Based Reasoning observed to help students in terms of final techniques. (end of year) marks. It seemed that increasing the frequency of these examinations would be beneficial, but staff 2 The Underlying Pedagogy workloads made this prohibitive and any The environment has evolved and is still changes would need to be accomplished in a evolving and a chronological account would way that made the students feel that they note that the authors’ understanding of the were the real beneficiaries of the process. pedagogical principles on which it is based Although not a new idea, objective testing has also evolved. But certain principles have [Rust 1977] was thought to be the answer. emerged, either initially, or on reflection: Mark Ratcliffe and John Woodbury took in hand the process of developing such an 1. Problem solving is the highest form of objective testing and rapid feedback system. learning [Gagne, 1977] and Software Engineering is about problem solving. Meanwhile, Lynda Thomas had become interested in the effect of learning styles on

Ratcliffe et al. A Learning Environment for First Year Software Engineers page 1 of 7 Research has shown that knowledge We have currently created an environment alone is not sufficient for successful that succeeds in accomplishing all of these problem solving in a domain: the student objectives to some extent. We have made must also choose to use that knowledge, significant progress on the first four and we and to monitor the progress they are are working on improving our coverage of making [Silver, 1987]. the last. 2. Research has also shown that methods of teaching and learning which combine 3 Framework significant student autonomy, In order to accomplish them we needed to conjecturing and articulation with put these disparate objectives into some kind dynamic scaffolding by the teacher are of coherent framework. highly effective [Tanner, 1999]. In her survey of Intelligent Tutoring Systems 3. Ideally, teaching should be “discursive” [Woolf, 1988], Beverly Woolf identifies and “adaptive” [Laurillard, 1993]. The four models which such systems need to aim is that the teacher and the students consider: a knowledge model, a model of the agree on learning goals, making their communication media, a cognitive model of conceptions accessible to each other. By the learner and a model of the tutoring tightening the feedback loop between discourse itself. In figure 1 we show the the teacher and the student, it then SETLAS environment and how it becomes possible for the student to incorporates these 4 models. receive, generate and act on feedback, and for the teacher and the student to adjust the affordances appropriately. Communication media model - SETLAS So, briefly and practically, we sought to address these principles by achieving the Knowledge Cognitive model model following objectives: 1. Giving students opportunities to more … notes improve their problem solving skills. Such opportunities should provide as learning much student autonomy as possible. styles practicals 2. Making feedback fast, frequent and helpful – this means tailoring it Testing lectures individually to the student. & 3. Making learning goals explicit and videos relating them to testing. 4. Helping students reflect on their learning style and making a wide range of Tutoring Model materials available – so that as many learning styles as possible are catered to. Learning Objectives 5. Providing dynamic scaffolding for the student – this means adjusting the level of help in line with the needs of the individual student. Figure 1: SETLAS Models

Ratcliffe et al. A Learning Environment for First Year Software Engineers page 2 of 7 The Knowledge model is currently realised matched very closely the results obtained by the lectures that the student may attend from other subjective testing carried out (these are also videotaped for later viewing), during the semester in terms of ranking of the lecture notes and the practical students. Overall, however, students’ project assignments (these are not assessed but are work was much better. This led the authors supported by demonstrator help). to believe that the system was testing the The model of the media involved is the right things and also improving overall system itself and its interface as experienced students’ learning by the student. The cognitive model, in the current version, consists of the models of student learning styles and the results of the tests. This is individual for each student. In addition an understanding of possible misconceptions informs the multiple choice test questions. Finally, the Tutoring model consists of the identified learning outcomes and also the arrows between the other blocks in the system. This framework provides a good overview of the relationships between the parts of the system and also highlights the areas in which more work is needed. At present we are considering how to make the cognitive model more complete so that we can further address the fifth objective of providing Figure 2: Typical Screenshot of Test better individualised scaffolding for 4.2 Rapid Feedback students. Although the original software had a facility to e-mail students their marks immediately 4 SETLAS Description after completing the tests, this was delayed 4.1 Initial Construction until after all the sessions had been held. At the end of the week the students were SETLAS began with a simple objective automatically e-mailed their results. The e- testing system, initially implemented in mail consisted of two variations. For the QuizPlease [Quiz, 2000], that would provide successful student the contents of the fast feedback for students to monitor their message took the form: progress. In the first, entirely voluntary, test the take up rate was 97%. “ Congratulations, you have scored 80%”. This exercise was successfully repeated 4 weeks later. Students unanimously requested For the unsuccessful, the message looked that the traditional essay-type examination something like: planned for the end of the semester be “ Unfortunately you failed the test replaced by objective testing. gaining only 30%. Please see [the Most encouraging was that the final set of lecturer concerned] to discuss your results produced by the objective tests performance”.

Ratcliffe et al. A Learning Environment for First Year Software Engineers page 3 of 7 Unfortunately this mass e-mailing backfired. overall performance of the students taking a Even after providing a general web page to test. The web pages provide access to results advise on the objective tests, a large of all individuals on the module and each proportion of all the students visited the result is linked directly into the question lecturer for individualized advice on how to bank. The information was used very improve performance. The additional effectively to stimulate appropriate workload was horrendous. teacher/student dialogue relating to The obvious solution might have been to individual student learning difficulties. advise the students of their erroneous 4.5 Improving the Questions answers via e-mail. Indeed this would have Developing the questions has been a heavy been quite straightforward. However, given burden on the lecturers concerned but has the huge investment required for developing had some extremely positive effects. Using the questions, it was important to protect the the Mastery teaching approach has led the question bank. We wished to be able to reuse lecturers to find that the students, without and refine the questions with a particular realizing it, are constantly producing emphasis on questions that required more material suitable for inclusion in new problem solving. questions. Experience has shown that at least 4.3 Making Learning Goals Explicit a couple of new questions were generated in A solution to the problem discussed in 4.2 this way in every lecture. This certainly was to relate the learning objectives helped to populate the question bank. explicitly to each question and to e-mail this During the objective test students are given information to the students instead. This was “Scratch Pads” which are essentially a blank helpful in two ways: students now gain sheet of paper that they may use for rough immediate feedback to the areas that they working. During the trail period students need to revise and also students become were encouraged to write down any more aware of the learning goals of the comments that they had on the questions so course. This allows students to reflect on, that they could be taken into consideration and hence improve, their learning and makes after the test. These comments were then it more likely, at least, that students can help analyzed. This exercise proved useful in influence the course learning goals. identifying any ambiguities that were 4.4 Supporting Mastery Teaching present in the questions and, as occurred in one test, can result in a question being Informing the students on their performance eliminated from a test, with all results being is an important aspect of the system but it is adjusted accordingly. The authors believe just as important to keep the lecturer up-to- that this extra level of involvement helps the date on the students’ progress. By having students become real stakeholders in the access to such statistics, the lecturer is able project. to alter the lectures, assign alternative worksheets and so forth. (This interactive 4.6 Individual Student Profiles approach has been called ‘Mastery An important part of the project has been Teaching’ and has been shown to improve recording information on each of the student performance by as much as 25% students taking the tests and this is now [Bloom, 1984].) being used to build up an individualized web To support this aspect of the system, a set of page for each student. Each page is web pages were automatically generated to password protected, restricting access to present the lecturer with statistics on the students’ own pages. There they can gain access to all of their results, including access

Ratcliffe et al. A Learning Environment for First Year Software Engineers page 4 of 7 to the learning objectives that they have been with the kind of empirical results that we failing (see Figure 3). now have available. In addition demonstrators record information about the practicals that the students have been given. This information becomes available via the individual student profiles. Students can see a record of their attendance at practical sessions It is hoped (bat has not as of this writing been tested) that this will have a positive effect in emphasizing the importance of students taking responsibility for their own learning. In line with the goal of making the learning environment as individualised as possible, the next step has been to profile students through a series of diagnostic tests. Information is captured on individuals’ learning styles, their mathematical ability and their IT proficiency to build up a picture Figure 3: The Student Web Page of each student’s cognitive model. This information can then be analyzed to determine any relationships between this 5 Issues Considered data and individual’s performance on the 5.1 Objective Versus Subjective various worksheets and assignments. Such information is then added to the student There has been considerable work published profile pages. Links are then set up to direct about objective testing and at first many students to the learning resources that best people within the authors’ department were suit their needs. For instance learners with a skeptical of the techniques that underly preference for audio learning might profit SETLAS. Despite initial reservations as to from re-viewing the lectures on RealVideo. how well objective testing would assess a Students who prefer learning from written student’s ability on a module, the authors are materials are directed to the lecture notes. satisfied that they have moved a fair distance down Bloom’s taxonomy [Bloom, 1956]. 4.7 The Next Step Whilst most people would argue that Ultimately our expectation is that students objective testing only addresses knowledge will be given practical non-graded recall, careful consideration of questions assignments that best meet their needs. Each certainly enables accurate assessment assignment will be tagged with the specific involving comprehension, application, learning objectives that it addresses and each analysis and problem solving. Students are will be categorized as to the learning style to being examined on areas of the syllabus that which it is most applicable. the authors originally thought impossible to A more complete cognitive model could be address though objective tests. used to tutor the student in the actual content 5.2 The Evolution of the Environment material of the course. This remains in the Initially QuizPlease provided all the future and relies on building a better functionality that was required, but over cognitive model, but this is only possible time the requirements outpaced QuizPlease

Ratcliffe et al. A Learning Environment for First Year Software Engineers page 5 of 7 and the authors began their own in-house 7 Future Work development. The system that is now The aspect of the SETLAS system that is available is fully generic, flexible and weakest at present is the cognitive model. extensible. This is also the hardest of Woolf’s models to The system is written in Java and accesses produce. Although results in the objective an underlying Postgres database. It has been tests correlate with graded practical developed using an object-oriented approach assignment work, the authors do not believe with emphasis on rapid prototyping. This that they are the same thing – or why bother has given a high degree of user confidence to assign graded practical work at all? providing, at every stage of development, a It has been our experience that the process of working product that can be adapted and constructing an object-oriented program and extended in the light of experience. deciding how to link the appropriate classes is the most difficult part of an introductory 6 Results programming course. This is essentially a creative activity and seems to require, To date the tests have been run 655 times; amongst other things, the ability to create a 280 of these have been used as summative mental model of the system and ‘animate it’. tests, the rest undertaken as formative tests. The results have been very positive. Not The next phase of this research will include only do they correlate very nicely with the an examination of such mental models (in, results obtained in coursework associated for instance, the work of Mary Hegarty with the module but also it is believed that [Narayanan and Hegarty, 2000]) and their they have been a strong motivator for the application in the arena of the first year students. The students seem very happy with programming and design class. We hope that the regular examinations that are now held. this will lead to the modification of SETLAS In fact when an extra session was held to into a truly intelligent and dynamic system load-test the system, 90% of the students that can provide suitable scaffolding for volunteered to take part. students in the completion of the difficult task of creating their first programs. Probably the most important point to appreciate is that in the last session the lecturer had access to performance indicators References. of the students as early as mid October, with Bloom, B. S. (ed) Taxonomy of educational more data becoming available every few objectives, Book 1, Cognitive Domain. weeks. Compare this to the traditional (1956) New York. British system where results are not available until early January. This has Bloom, B. S. The 2-Sigma Problem: The enabled the lecturer to identify potential Search for Methods of Group Instruction as problems much earlier on and where Effective as One-to-One Tutoring, appropriate take remedial action. Educational Researcher 13 (1984) 4-16. Demonstrators associated with the module Bredeweg, Bert, and Radboud Winkels, but not involved in the objective tests noted “Qualitative Models in Interactive Learning a significant improvement in understanding Environments: an Introduction”, Interactive of the material by the students. Marks Learning Environments, Volume 5, (1998) overall have improved. Felder, Richard M., Matters of Style, ASEE Prism,6 (4), (1996)

Ratcliffe et al. A Learning Environment for First Year Software Engineers page 6 of 7 Gagne, R.M., The Conditions of Learning, Holt Rhinehart and Wilson, (1977) Laurillard, D., Rethinking University Teaching, Routledge (1993) Narayanan, N., and M. Hegarty, “Communicating Dynamic Behaviors: Are Interactive Multimedia Presentations better than Static Mixed-Mode Ones”, in M. Anderson, P. Cheng and V. Haarslev (eds.) Theory and Application of Diagrams, Springer (2000) QuizPlease: The fastest and easiest way to create learning resources for the Internet. Online. Internet. [03/09/2000] Available at http://www.quizplease.com/ Rust, W. B. Objective Testing in Education and Training, Pitman Press, Bath, UK (1973) Silver, E.A., “Foundations of Cognitive Theory and Research for Mathematics Problem Solving and Instruction” in A. Schoenfeld (ed.) Cognitive Science and Mathematics Education, Lawrence Elbaum (1987) Tanner, H. and S. Jones, “Dynamic Scaffolding and Reflective Discourse: the Impact of Teaching Style on the Development of Mathematical Thinking”, Proceedings of the 23rd Conference of the International Group for the Psychology of Mathematics Education, Haifa (1999) Thomas, Lynda, “The Implication of Different Learning Styles on the Use of UML Notation”, Poster presented at CSEE&T 2000. (2000) Woolf, Beverly, “Intelligent Tutoring Systems: A Survey” in H. Schrobe and the American Association of Artificial Intelligence (eds.), Exploring Artificial Intelligence, Morgan Kauffmann (1988)

Ratcliffe et al. A Learning Environment for First Year Software Engineers page 7 of 7