Durham Research Online Deposited in DRO: 13 July 2012 Version of attached le: Accepted Version Peer-review status of attached le: Peer-reviewed Citation for published item: Budgen, D. and Drummond, S. and Brereton, P. and Holland, N. (2012) 'What scope is there for adopting evidence-informed teaching in software engineering?', in 34th International Conference on Software Engineering ICSE 2012, 2-9 June 2012, Zurich, Switzerland ; proceedings. Piscataway, NJ: IEEE, pp. 1205-1214. Further information on publisher's website: https://doi.org/10.1109/ICSE.2012.6227022 Publisher's copyright statement: c Copyright 2012 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE. Additional information: Use policy The full-text may be used and/or reproduced, and given to third parties in any format or medium, without prior permission or charge, for personal research or study, educational, or not-for-prot purposes provided that: • a full bibliographic reference is made to the original source • a link is made to the metadata record in DRO • the full-text is not changed in any way The full-text must not be sold in any format or medium without the formal permission of the copyright holders. Please consult the full DRO policy for further details. Durham University Library, Stockton Road, Durham DH1 3LY, United Kingdom Tel : +44 (0)191 334 3042 | Fax : +44 (0)191 334 2971 https://dro.dur.ac.uk What Scope Is There for Adopting Evidence-Informed Teaching in SE? David Budgen & Sarah Drummond Pearl Brereton & Nikki Holland School of Engineering & Computing Sciences School of Computing & Maths Durham University Keele University Durham DH1 3LE, U.K. Staffordshire ST5 5BG, U.K. fdavid.budgen;[email protected] fo.p.brereton;[email protected] Abstract—Context: In teaching about software engineering The models we use in teaching are largely derived from we currently make little use of any empirical knowledge. Aim: expert knowledge, and while we have argued elsewhere that To examine the outcomes available from the use of Evidence- software engineering practices are excessively dependent Based Software Engineering (EBSE) practices, so as to identify where these can provide support for, and inform, teaching activ- upon expert judgement [2], we do also recognise that there ities. Method: We have examined all known secondary studies are good reasons for using this in teaching. In particular: published up to the end of 2009, together with those published • the very wide range of software applications often in major journals to mid-2011, and identified where these makes it difficult to identify ‘representative’ examples provide practical results that are relevant to student needs. Results: Starting with 145 candidate systematic literature of the use of a method or technique; reviews (SLRs), we were able to identify and classify potentially • the (possibly excessive) concern of companies about useful teaching material from 43 of them. Conclusions: EBSE proprietary information makes it difficult for the teacher can potentially lend authority to our teaching, although the to obtain unbiased exemplars from real life; coverage of key topics is uneven. Additionally, mapping studies • empirical research in software engineering has only can provide support for research-led teaching. recently begun to make a significant impact, and even Keywords-empirical; education; evidence-based now, empirical data about major topics may be hard to find or non-existent—and in addition, what is available I. INTRODUCTION may be difficult to interpret in a teaching context. However, while these still remain true, in the case of the third During a history that now spans more than 40 years, the point, there is one development that is likely to influence teaching of software engineering has been largely structured the way that we teach our subject. The emergence of the around descriptions and models of procedures (methods) and evidence-based paradigm has significantly changed the way technical products. To some extent, that emphasis reflects that clinical medicine is taught and practised. Its adoption our research culture. As Glass et al. have observed from (and adaptation) for use in software engineering through their study of the research literature, software engineer- Evidence-Based Software Engineering (EBSE) [3], [4] has ing research is dominated by the use of descriptive and potential to make an impact upon our own teaching too. formulative approaches, with a much lower proportion of The core objective of EBSE (and of evidence-based stud- evaluative studies [1]. Methodologically, our research is also ies in general) is to find all relevant data related to a topic in a overwhelmingly based upon the use of concept analysis systematic, unbiased and objective manner, to aggregate this and concept implementation as research methods. Where data, and then to combine it with experience and context to ‘real’ data is employed (for example, in the formulation provide the best possible evidence about the given topic. We of COCOMO), the details of this are often abstracted out describe some further details about EBSE and its practices in when presented to the user (student). Our textbooks like- the next section, for the moment, this description is sufficient wise present largely descriptive formulations for (mostly for us to identify our research question as being: idealised) models of the topics that characterise software engineering. “What is available to enable evidence-informed Since it is unrealistic to expect that any teacher should teaching for software engineering?” have extensive experience of all aspects of software en- Note that we have preferred to use the term ‘evidence- gineering, one question this leaves is how both teachers informed’ rather than ‘evidence-based’ here, as we recognise and students can be informed about what actually works that the range of factors involved in empirical software engi- in practice, and under what conditions? Indeed, even where neering make it difficult, if not impossible, to provide strong a teacher may have experience related to particular topics and authoritative evidence on most topics, and also that the (e.g. testing), they may not have any means of knowing how use of evidence is likely to be context-specific. Underpinning representative their experience and knowledge actually is. this question there is also, of course, the assumption that providing them with evidence will create a better and more engineering have subsequently been incorporated into the satisfying experience for our students, and also prepare them Guidelines document, updated in 2007 [4]1. more effectively for their future careers, whatever form What particularly distinguishes a systematic review from these might take. For the moment this has to remain an a conventional (expert) review is the use of a defined assumption, since we lack suitable longitudinal studies that methodology that ensures that the review is both fair and would allow a more objective assessment. However, given objective, and that it can also be seen to be so. In particular, its impact elsewhere, not least in the teaching of evidence- the conduct of a systematic review should be: based medicine (EBM) [5], it is not unreasonable to expect • Open in that all the procedures are defined beforehand that the impact of EBSE is likely to be beneficial, even if at in the research protocol and reported with the findings. present we cannot effectively estimate the extent of this. • Unbiased in that as far as possible, all relevant studies To address our research question we examine the nature are included and fairly aggregated by the most appro- of EBSE and the extent of its adoption by the empirical priate forms. research community over the past eight years. We describe • Repeatable by other researchers (although we should the research method underpinning our paper; present the note that the changes that occur in digital libraries outcomes; and then interpret these outcomes in the light from time to time might mean that searches are not of our own experiences as teachers. Finally, we return to completely repeatable). consider our original research question. Of course, one consequence of taking such a thoroughly II. EVIDENCE-BASED SOFTWARE ENGINEERING systematic approach is that conducting an SLR may be time- In interpreting the evidence-based paradigm for the soft- consuming. However, evidence-based researchers generally ware engineering domain, Kitchenham et al. defined the goal consider this to be a worth-while price to pay in order to of evidence-based software engineering (EBSE) as being: ensure much greater rigour for their research procedures. To provide the means by which current best evi- B. Some Limitations of EBSE dence from research can be integrated with prac- While conceptually at least, EBSE has a sound scientific tical experience and human values in the decision basis and the potential to deliver rigorous, unbiased appraisal making process regarding the development and of software engineering processes and products, we need maintenance of software. [3] to recognise that there are several factors that limit this This section examines the procedures for doing this, identi- potential. These include: fies some limitations upon their effectiveness, and describes • The role of the participant in empirical studies (and current progress with realising EBSE. especially experiments). Since this role usually involves A. The EBSE Process performing skill-based tasks, it is impossible to apply techniques such as double-blinding (of participants and Drawing upon the experiences from other domains that experimenters), considered essential for medical trials. have adopted the evidence-based paradigm for research, in- • The extensive use of laboratory experiments and quasi- cluding clinical medicine, education, and various healthcare experiments in SE, rather than field trials.