Going with the Grain
Going with the grain? Issues in the evaluation of educational innovations.
Charles Anderson, Kate Day, Jeff Haywood, Ray Land and Hamish Macleod
Department of Higher and Further Education, University of Edinburgh.
Paper presented at 8th European Conference for research on Learning and Instruction, Gothenberg, 25-29 August 1999.
Abstract
Whilst considerable attention has been paid in both the evaluation and innovation literatures to the value of setting in context the object of enquiry and viewing it from differing perspectives, relatively little account has been taken of the related and important issue of what constitutes an appropriate grain of description. This article takes up the challenge of raising and exploring a number of distinct issues concerning the 'granularity' of an evaluation, and in the process sets out a framework to guide reflection on these issues. General concerns and points of principle are illustrated with reference to a recent large scale evaluation, conducted by the authors, of a programme promoting the use of ICT for learning and teaching within higher education.
Background
A central theme of the literature on educational evaluation in the last few decades has been the need to ensure that innovations are not assessed solely on the basis of a disembeded set of outcome measures; but studied within the wider contexts that frame them, with an eye to identifying the factors that constrain or enable their immediate uptake and longer-term existence within a school or a whole educational system. This stance concerning evaluation has been argued for from a number of different theoretical perspectives; but it can also be justified on the more pragmatic grounds of pointing to overwhelming evidence from existing studies of the ways in which innovations are powerfully shaped, or indeed 'rejected' by individual contexts. For instance, a recent study reported by Cole (1995, 1996) of the 'life history' of the "Fifth Dimension", a programme of computer-mediated after-school activities for children implemented in different sites, revealed very distinct differences across the sites studied in the sustainability of this particular programme – differences that were clearly linked to the pre-existing purposes, ethos and patterns of interaction in the individual sites.
Agreement on the value of examining the wider context within which innovations are embedded can, however, mask important conceptual difficulties surrounding the definition of context and its identification[1]. Attention to context also confronts an evaluation team with difficult operational decisions in designing and implementing a particular study. How in practice does one separate out the 'figure' of an innovation from the 'ground' of its context and then view the relationship between them? To what extent does one focus on illuminating the immediate context of application of an innovation, or also attempt to consider the effects of more distal factors?
Grain of description
This article presents the ways in which, as a research group carrying out evaluations of the use of ICT in higher education, we have attempted to take account of the immediate and wider institutional contexts in which the innovations we have been studying were framed; and our reflections on how our attempt to view innovations from different perspectives affected the detail and type of description that it was possible and/or desirable to achieve. It will be argued, (largely by means of illustrative material from a recent evaluation project), that it may help evaluators to clarify their purposes and communicative intentions, if they direct attention explicitly to the question of the grain of description that is appropriate or feasible when situating innovations in context or presenting the perspectives of different actors on the processes and outcomes of an innovation. On this central theme, it is necessary first to consider the crucial matter of audience requirements.
Audience requirements
The question of providing a well-contextualised assessment of an innovation is complicated by the fact that evaluators are hardly ever, and some would argue should never be, able to aspire to the role of mythical neutral observers independently determining how they will depict the terrain that they are surveying. They must take into account the requirements of the evaluation’s audiences. Professional researchers, policy makers and practitioners have, in Bell and Raffe's useful phrase, different "normative world views of research" ( Bell and Raffe, 1991, p.133). In particular, judgements are likely to vary between these three constituencies as to what makes for a good evaluation report. Researchers will judge a report according to the ‘canons’ of sound professional practice, including the degree to which observations and claims are well supported by a full, clear presentation of evidence. Policy-makers are likely to want a succinct, synoptic report which makes appropriate recommendations and is guided by a realistic appreciation of the constraints within which policies have to be pursued. On the other hand, practitioners will see merit in reporting which focuses closely on the day-to-day challenges involved in implementing an innovation and provides insights and suggestions that are of clear practical relevance.
Viewing the needs of practitioners from a different angle, educational psychologists are recognising once again that if the goal of improving practice is to be achieved serious attention needs to be given to the matter of the nature and level of description provided within studies of teaching and learning. For instance, Entwistle, McCune and Walker (2000) recognise that, with an audience of practitioners in mind judgments concerning the level of explanation and the choice of conceptual frameworks need to be guided both by "realism (to what extent is the explanation recognised by participants) and what has been called pedagogical fertility", i.e. the capacity to generate innovations within, or new perspectives on, learning and teaching (Entwistle and Hounsell, 1987).
Very often evaluators will wish, or be obliged, to respond to certain at least of these expectations of policy-makers and practitioners in writing up their findings, sometimes by producing separate versions of a report targeted at different audiences. Being alert to the needs of different audiences clearly has implications not only for report writing but for all stages of an evaluation. Figures 1 and 2 present some of the key points concerning nature and level of description that would seem to flow from an attempt to respond to the separate expectations of practitioners and policy makers.
The level and nature of description and explanation identified as desirable in Figure 1 could be provided by following one of the illuminative approaches to evaluation that have held such sway since Parlett and Hamilton's (1972) groundbreaking efforts in the early seventies. Similarly the desiderata in terms of type and detail of account highlighted in Figure 2 can be seen to be compatible, to some degree at least, with a systems approach to evaluation. However, there is a clear contrast between the implications for the 'grain' of evaluation identified in Figure 1 and those identified in Figure 2 – a contrast which poses a challenge to evaluators who wish to bear in mind the needs of the different audiences of practitioners, policy makers and peers within the research community and still conduct their study in a principled manner. At the stage of report writing there is also the need to avoid the problems that may arise from employing a “blurred genre” (Geertz, 1980). These challenges are made particularly acute by the fact that evaluators will customarily be working to a contract which imposes, (possibly considerable), constraints on their freedom of manoeuvre.
Figure 1: Practitioners' Expectations and Implications for the 'Grain' of an Evaluation
Figure 2: Policy Makers' Expectations and Implications for the 'Grain' of an Evaluation
The preceding paragraphs have then established what are likely desiderata held by policy-makers and practitioners concerning the grain of description provided by evaluators; and the general challenges faced in providing information for more than one audience. A later section of the paper will detail the attempts made to meet these challenges in a recent evaluation carried out by our research team. The following section will briefly describe the ICT project whose products were being researched and the key tasks of the evaluation. Having set the scene, attention will then turn to highlighting some of the central questions which arose concerning detail and type of description within this evaluation. Examining these questions will allow us to illustrate general issues that need to be addressed in making principled decisions about what constitutes an appropriate grain of investigation, analysis and reporting in evaluative research.
Detail, Level and Type of Description
The Teaching and Learning Technology Programme
In February 1992, the Universities Funding Council of the United Kingdom launched the first phase of a national Teaching and Learning Technology Programme (TLTP) with the intention of making "teaching and learning [in higher education] more productive and efficient by harnessing modern technology" and of helping "institutions to respond effectively to the current substantial growth in student numbers and to promote and maintain the quality of their provision" (UFC Circular 8/92). In this first phase of the programme, £7.5 million funding was provided each year for three years and academics were invited to bid for monies for projects to develop new methods of learning and teaching using ICT. Of the 43 projects which were funded during this first phase, 11 were concerned with implementing the use of learning technology within mainstream teaching and learning in single universities. The remaining 32 projects involved academics from different universities working as consortia and focused on courseware development. These consortia ranged in size from 2 to 44 member institutions and the projects covered a wide range of disciplines. A second phase of the programme was later announced, under which a further 33 projects were funded. These two first phases spanned the period 1992-1996, with over £11 million being invested in the programme by the UK Funding Councils, plus investments made by universities themselves.
Range of materials created
The 76 TLTP projects supported under Phase 1 and 2, (some of which received additional continuation funding after the end of the second phase), created a wide range of different products. The majority of the projects produced computer-based materials, with some supporting ‘teacher user guides’. There were, however, a few projects in this group which used video as a major or minor part of their output. Products of the subject-based projects were most likely to be used in academic departments or supplied by central services to students via networks. For institutional and some subject-based projects their aim was mainly to influence the way in which teaching and learning took place rather than to produce courseware, computer-based or otherwise. The majority of projects set out to create tangible materials which could be used within courses (albeit with the same broad aim of affecting teaching practice, but this was not centre stage in their activities).
The First TLTP evaluation
An independent evaluation of these first two phases of TLTP was commissioned from a research team drawn from Coopers & Lybrand, The Institute of Education and the Tavistock Institute; and the report of this evaluation was published in June 1996 (Coopers & Lybrand et al., 1996). This report made recommendations for greater central co-ordination of the programme, progress monitoring of projects and both formative and summative project evaluation. These recommendations became conditions of the TLTP Phase 3 funding announced in May 1998. Although the evaluation that reported in 1996 was able to give a clear view of the management and general direction taken by the TLTP projects, many of the learning materials they had produced had just been released, or were still in the pipeline. It was, therefore, deemed at that point to be premature to take on the task of assessing the extent to which TLTP products, broadly defined, had been disseminated throughout the UK higher education sector and the degree of integration that these products had achieved within day-to-day teaching and learning. This task fell to our own research group.
Evaluating the use of TLTP materials in UK higher education
In January 1998 we were commissioned by the Higher Education Funding Council of England to conduct a study of the use within UK higher education of the learning materials produced by TLTP Phases 1 and 2. The study itself was carried out to a tight schedule between February and August 1998 (Haywood, et al., 1999).
Key tasks in this evaluation were:
· to find out which TLTP products were being used, where and how
· to examine the pattern of usage of TLTP courseware in relation to other uses of ICT within learning and teaching in UK higher education
· to explore how ‘contextual’ factors might have influenced the uptake of TLTP products
· to assess the overall impact of TLTP
· to conduct a bibliographic search to track existing studies of TLTP use.
These tasks were pursued with surveys of:
· all TLTP projects (76),
· all teaching departments in all Higher Education institutions in the UK (3854),
· courses/modules using TLTP in those departments and
· key informants in 102 medium and large Higher Education institutions.
(The surveys included a considerable number of open questions that required a qualitative analysis.)
Data were also gathered from the TLTP Co-ordinator and the TLTP central collection of materials. Case studies of the implementation and use of TLTP materials were constructed, largely through interviews.
A considerable proportion of higher education courses are now delivered through franchising arrangements with further education colleges. For the sake of completeness, it was appropriate to gain information from the further education colleges thought to deliver higher education courses on the extent of their TLTP usage.
In carrying out these tasks the research team was faced with a number of difficult, general, methodological challenges which will be the subject of a forthcoming paper. In this present article attention will remain focused on methodological problems and decisions which had very direct implications for the grain of description that could be achieved within this evaluation, starting with the matter of the timescale over which the TLTP learning materials had been produced.
Timescale of the Teaching and Learning Technology Programme
The TLTP materials which were the object of our study were created by seventy-six projects over the six year period 1992 to 1998. Thus the timescale over which materials were produced and brought into use was extensive, with some projects delivering their early, or at least prototype, products within a few months of start up and others only just reaching delivery by mid 1998. Accordingly an evaluation taking place in 1998 required some members of staff in universities to think back several years to when TLTP began, whereas others had only recently begun to see relevant products appearing. We thus faced the difficulties common to all retrospective surveys of how to achieve reliable recall as memories fade and to deal with the tendency to view the past through the screen of hindsight and rationalisations. An innovation implemented several years ago that has taken root may be enfolded within ongoing practices, and therefore be harder to separate out as a distinct object in itself. An added complication in our evaluation was the considerable variability within the population in the time period (extent and recency) on which they could comment, and correspondingly in the detail of description that it was reasonable to seek. Technological changes had also occurred over Phases 1 and 2 of TLTP that needed to be taken into account.