Building a Formative Assessment System That is Easy to Adopt Yet Supports Long-term Improvement: A Review of the Literature and Design Recommendations Ellis Solaiman, Joshua Barney and Martin Smith School of Computing Science, Newcastle University, Newcastle upon Tyne, U.K. Keywords: Formative Assessment, Assessment Design, Review, Feedback Automation, Education Technology. Abstract: Formative assessment has been shown to be an effective teaching tool, yet is infrequently used in practice. With the intent of building a formative e-assessment platform, we examine research on formative practices and supporting computer-based systems with a focus on: institutional barriers to adoption of previous systems; senses in which students and teachers can improve their practices across varying timescales; and collectible data (self-reported or otherwise) necessary or advantageous in supporting these processes. From this research we identify the minimal set of data which adequately supports these processes of improvement, arrive at a set of requirements and recommendations for an innovative system which collects, processes, and presents this data appropriately, and from these requirements design the architecture of an extensible electronic formative assessment system which balances the need for complex long-term analytics with that of accessibility. 1 INTRODUCTION tion with systems in use within institutions, and edu- cational data-mining efforts fail due to the absence Formative assessment refers to the use of testing to of tools which are easy for non-experts to use (Ro- allow both students to reflect on their own learning mero and Ventura, 2010). A key factor inhibiting the and teachers to determine the needs of individual stu- adoption of formative assessment technologies is staff dents, or groups of students. This can also be des- time. Teacher time is an especially scarce resource cribed as assessment for learning. Formative as- – those who see the value of educational technology sessment is often contrasted with summative asses- may still avoid full utilization for this reason; ”These sment, described as assessment of learning, which are all great initiatives, but I’m running out of hours uses testing as a means for summarizing the perfor- in a day ... I’m in overload!” (Hall and Hall, 2010). mance of students and their level of knowledge at the Research by (Macfadyen and Dawson, 2012) highlig- end of a course (Looney, 2011) (Black and Wiliam, hts that in order to overcome individual and group re- 2009). Despite the clear benefits of formative asses- sistance to innovation and change, planning processes sment as an effective teaching tool, it is infrequently must create conditions that allow participants to both used in practice. Also whilst VLEs (Virtual Learning think and feel positively about change; ”conditions Environments) are widespread in higher education, that appeal to both the heart and the head”. Without their formative abilities are often severely underuti- investing resources for effective automated data ana- lized (Blin and Munro, 2008). Even when change is lysis and visualization, large quantities of data will not resisted by faculty staff, adoption of educational not compel social or cultural change (Kotter and Co- technology tends to proceed at a slow pace, and often hen, 2002); ”increasing observability of change is an innovation is driven by lone individuals deciding to effective strategy for reducing resistance” (Rogers, use technologies without backing from above (Rus- 1995). sell, 2009). Although perception of student expec- This paper builds upon and complements previous tation can provide positive incentive, technical bar- work and literature reviews such as (Gikandi et al., riers along with a lack of institutional support are key 2011), by taking a more integrative approach. We suppressing factors (King and Boyatt, 2015). Cur- attempt through the development a holistic theoretic rently available Education technologies are hampe- framework,to arrive at the design of a system suitable red by technical problems such as lack of integra- for active use both as a tool for student and teacher 442 Solaiman, E., Barney, J. and Smith, M. Building a Formative Assessment System That is Easy to Adopt Yet Supports Long-term Improvement: A Review of the Literature and Design Recommendations. DOI: 10.5220/0006339404420449 In Proceedings of the 9th International Conference on Computer Supported Education (CSEDU 2017) - Volume 1, pages 442-449 ISBN: 978-989-758-239-4 Copyright © 2017 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved Building a Formative Assessment System That is Easy to Adopt Yet Supports Long-term Improvement: A Review of the Literature and Design Recommendations improvement, and as a platform for further empirical fers from previous formulations like that of (Wiliam research about the impact of various formative stra- and Thompson, 2007) in that we conceive of teachers tegies. Initially, multiple high impact journals in the not just in terms of the role they can play in providing field of educational technology were identified, and feedback to students but also as individuals who can the titles and abstract of all publications from the the benefit from feedback themselves. last 5 years were manually examined. A number of potentially relevant papers were identified and scruti- nized in further detail, and common subjects and re- ferences were noted. Keywords related to these sub- 3 FEEDBACK PROCESSES AND jects were used to find further papers using academic THEIR DATA COLLECTION search engines. The process of reading papers, disco- REQUIREMENTS vered through references or search, continued recur- sively until a clear picture of a range of formative and We examine the specifics of feedback processes be- learning practices emerged. A conceptual framework low, considering issues related by their similar data- was developed to organize these practices by time and requirements together. This view allows us to move individual, and the body of collected papers was re- closer to the design of a real system. examined in the context of this framework to explore the data-collection requirements of each of the iden- tified practices. Recommendations are made for im- 3.1 Formative Feedback Principles plementation, and from these recommendations, the design of an extensible system to collect this data is Synthesizing research literature, (Nicol and presented. Macfarlane-Dick, 2006) present seven princi- ples, reordered for clarity here, arguing effective formative feedback practice: 1. delivers high quality information to students 2 CLASSIFICATION OF about their learning. 2. helps clarify what good FEEDBACK BY USER AND performance is (goals, criteria, expected standards). 3. provides opportunities to close the gap between DURATION current and desired performance. 4. facilitates the development of self-assessment (reflection) in lear- Multiple researchers distinguish types of formative ning. 5. encourages positive motivational beliefs and assessment based on their duration. Allal and self-esteem. 6. encourages teacher and peer dialogue Schwartz (Allal and Schwartz, 1996) use the termi- around learning. 7. provides information to teachers nology of “Level 1” and “Level 2’ formative asses- that can be used to help shape teaching. sments, which directly benefit students who are asses- The first three principles mirror (Wiliam and sed as or use data gathered to benefit future instructi- Thompson, 2007)’s theory of formative assessment, onal activities or new groups of students respecti- which emphasizes establishing where the learners are vely. Alternatively, (Wiliam, 2006) distinguishes be- in their learning, where they are going, and what tween short-, medium-, and long-cycle formative as- needs to be done to get them there. Black and Wil- sessment, which operate within and between lessons, liam, (Black and Wiliam, 1998) warn that classroom within and between teaching units, and across mar- cultures focused on rewards or grades encourage stu- king periods, semesters or even years. In addition to dents to game the system, avoid difficult tasks, and acting as a helpful framework within which to clas- where difficulties or poor results are encountered, sify and organize research, this viewpoint is useful ”retire hurt”. By 2004, this had been strengthened as a design tool – collecting data required for longer- in (Black et al., 2004) into an explicit recommenda- term formative processes requires action over a sus- tion to adopt comment-only marking practices; ”Stu- tained period of time, and this affects the short-term dents given marks are likely to see it as a way to com- experience of new users. As discussed below, a sy- pare themselves with others; those given only com- stem must show immediate value in order to have ments see it as helping them to improve. The latter the best chance at being widely adopted, and balan- group outperforms the former.” A literature review by cing the conflict between supporting longer-term go- (Gikandi et al., 2011) provides supporting studies and als and maintaining accessibility is a design challenge explicitly reaffirms the above seven principles as ”an that deserves thinking about explicitly. In addition to essential condition for effective formative feedback”. distinguishing processes by time we also do so by the Whilst the above principles may be sufficient to agents involved (See Figure 1). Our framework dif- guide the behavior of teachers in conversation with 443 CSEDU