Educational and Serious Game Design: Case Study in Collaboration

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design”

EDUCATIONAL AND SERIOUS GAME DESIGN: CASE STUDY IN COLLABORATION

Jon A. Preston, Southern Polytechnic State University

INTRODUCTION

When we first created the Bachelor of Science in Computer Game Design and Development (CGDD), we consulted the International Game Developers Association (IGDA) Curriculum Framework (Gold, 2008). This document and the suggestions the IGDA provide cover a broad range of topics within game development curricula. Serious games are a major topic among the game design section of the IGDA Curriculum Framework, with ten different enumerated topics in the serious games section. These include: medical and non-entertainment industries, education, training, therapeutic uses, simulations, the use of games for political statement, the use of games as an artistic medium, working with content experts, instructional design, and evaluation of games as an educational/training tool. We were motivated by these guiding principles and the importance of serious game design topics when constructing our four-year Bachelor degree program, and as a result, we require a full semester-long course on serious and educational game design as part of our core Computer Game Design and Development curriculum. All CGDD majors are required to take the Educational and Serious Game Design course in their Junior or Senior year of study.

When first designing and teaching the Educational and Serious Game Design course, it became apparent that a hands-on environment in which the theory discussed in the textbook and classroom could be applied to real-world projects would enhance students’ experience and learning in the course. In the year prior to offering the Educational and Serious Game Design course for the first time, a group of four students volunteered with a local elementary school to develop an underwater exploration game. This prototype, developed in collaboration with third, fourth, and fifth grade students and teachers, focused on learning about fish species, pairing nicely with the local school’s emphasis on marine biology for the year. This prototype project also emphasized Engineering principles that the elementary school wanted students to gain exposure. After a year of collaboration during this prototype game development, our volunteers and the lead professor on the project were invited to be featured at a county-wide exposition of student projects and industry-collaborative projects. This event was hosted at the Georgia Aquarium where, throughout the night, a crowd of students gathered around the demonstration table with keen interest in the game: Parents and other educators were intrigued by the idea of collaborating between a local university and elementary school to generate a game that supported Engineering and science learning outcomes for the students. As you can see from the figure below, showcasing the game at this important event was a good

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design” way to catalyze interest in the project and position us to expand this collaboration into a formal partnership with the local elementary school.

Spring-boarding from this prototype collaboration and the external interest it garnered, we proposed a formal partnership with the elementary school. The elementary teachers would define which learning standards on which we would focus for the year and provide design guidance and access to their students, and we would provide the game development expertise.

Figure 1: Demonstrating the Prototype Educational Game

BACKGROUND & RELATED WORK

The idea of motivating student learning through collaboration and game design and play is a well-defined area of pedagogical research. We note that Tang defines computer games as “a form of real-time interactive software wrapped in creatively crafted media that offers game- players engaging, goal-directed play” (Tang, 2011). Since games by definition are interactive, creative, engaging, and goal-directed, games are an excellent means to achieve student-centric learning. But designing a game for commercial success is certainly not trivial and designing a successful educational/serious game is even harder. Kaprolos (2012) identifies the importance of not only meeting industry-standard expectations of engaging gameplay experiences but also delivering the needed educational content and ensuring effective teaching and learning occurs as a result of the educational/serious game. Clearly there is much promise in utilizing games to

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design” achieve learning, but the success of this learning as a result of the game depends upon the quality of the educational/serious game design.

Our approach is novel in that we don’t merely have student learners play a game; rather, we have them participate in the design and development of the game. This meta approach is central to the success of our project. We believe this collaboration motivates students more than merely getting to “play games” in the classroom. Beyond involving students in designing video games, it is essential to foster a sense of collaboration to enhance learning. As done by Bououd (2012), we require students to collaborate in their game play and design to ensure deeper, collaborative learning occurs throughout the design and development processes. Whereas Bououd emphasizes the idea of collaborative gameplay as enhancing learning, we believe this approach is also applicable at a meta level. Students learn how to design by “playing” the course as well.

When initially designing the collaboration with the local K-12 schools, we understood that the technology resources available in the school might be a limitation. Our university game development lab is equipped with the latest computing hardware and software, but this is certainly not the case of most K-12 schools, which often struggle to obtain resources. While this seemed a daunting problem to overcome, we found by examining the literature that the limitation of technology resources could actually benefit our proposed collaboration and actually benefit our university students’ learning. With respect to constraints, Stephen Jacobs makes the case that constraining the design and development of games for learning to utilize less powerful computing (and in the case of Jacobs’ work—using the One Laptop Per Child (OLPC) with fairly minimal computational power relative to game systems) can be beneficial for the university student design team (Jacobs, 2010). In addition to having to focus on core game development concepts, students who work in technology-constrained environments are better positioned to work in mobile and web technologies relative to their counterparts who work on and develop for high-end gaming systems.

OUR PROCESS

Similar to a software requirements specification (SRS) document, we utilize a Game Design Document (GDD) for each project (Salazar et al., 2012). This helps the student teams clearly and completely define the scope and intent of the game project without having to worry about implementation details. The GDD and SRS act to define the what of the software system; later, the student teams can define a technical specification document that defines the how—i.e., how the system will be implemented, which specifies the implementation programming language, technical details about algorithms, and specific approaches to enabling the requirements specified in the SRS/GDD.

Since the collaboration between the student teams and the partner K-12 schools is so vital to the success of this course, we require teams to collaborate in defining the GDD with the partner teachers. This requires the students to communicate with an authentic, external customer on

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design” the project, and it also enables the partner teachers to have a strong voice in the design process.

We also strongly encourage that the games that are designed feature collaboration rather than competition. Whereas most traditional U.S. games are designed with a zero-sum approach (where if one player advances, another must decline), our game projects emphasize collaboration and minimize the competitive nature of game play. This creates a more open, friendly game play space for players, borrowing from a successful approach of such German game design giants as Klaus Teuber who designed the popular The Settlers of Catan (Kosmos, 1995) (Curry, 2009).

Our process involves the following steps each year. We have followed and refined this process over the past four years, repeating this course and the collaboration with local schools to develop the games in the course.

September-October Connect with local teachers and discuss possible learning standards that could be supported by games for learning.

November-December Finalize learning standards and acquire all existing worksheets and activities that the teachers utilize in the classroom.

January Educational and Serious Game Design course begins and students are able to review the possible learning outcomes and general topics to be supported. Teams are formed (with three to five students per team) and a preliminary design is drafted.

February Design is presented to and refined with the K-12 teachers.

March-April University student teams develop the game as defined in the design document, and partner K-12 students are asked to generate content (text descriptions, 2D art/drawings, audio/music) to be included in the game. Demos are presented to the K-12 teachers and students at least once every two weeks as part of an “iterative refinement” development process.

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design”

May Final demonstrations and “handoff” to the K-12 teachers is made. Our university hosts a game demo night to showcase the student projects and have parents, teachers, and students playtest the final games.

At the end of the course, students perform a formal analysis of the design and quality of their game using the Design-Play-Experience framework proposed by Winn (2009), and students must justify how their games enable Gagne’s Nine Events of Instruction (1985).

This process has worked well over the past four years, with impressive results that I will discuss later in this paper.

The week-by-week schedule for the semester in which the course is run appears below:

Week Topic Reading Assignment Due

NPR 12/20/2010 Video Games...

Play SPENT (McKinney, 1 Course Introduction Intro and Capter 1 - Foundations 2011). of Serious and Educational Game Design

Constructivism Create a college version of

The Utility of Games in SPENT focusing on "A Journey into Constructivism" Education trials/problems faced by 2 by Dougiamas students. Paper version only

Constructivism at this point. Focus on "Moving Learning Forward" by design and message. Klopfer, Osterwiel, and Salen

Create a digital version of MLK Holiday - No Digital Games for Learning your game playable through Class 3 the browser.

Global Game Jam.

No Class - Recover from OSTP: Games that Can Change the Establish team and pick 4 Game Jam Weekend World project.

Using Computer Games Chapter 2 - Player Motivation and Traditional Project web site (Google

Education Models Sites recommended) with 5 NYTimes Article on iPads and iPads in the Classroom invitation sent to another post in contrast Discussion collaborators.

Introduction to

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design”

Simulations Winn's DPE Paper Design, Play, & Experience (DPE)

Examine the Games for Change web site and review Chapter 3 - Game Mechanics Part some of the games 6 Actions and Results 1 (informally); prepare a 2- How do these games play? page (single spaced, 11 point font) discussion of your experience with 3+ games.

Chapter 4 - Game Mechanics Part Interactive 7 2 Environments Engagement

Interactive Chapter 5 - Feedback and 8 Environments Reinforcement

9 Midterm Game Demos Midterm Game.

Gamification Case 10 Chapter 6 - Case Studies Studies

The Need for Final Game. 11 Frustration & Chapter 7 - Mechanics Tutorial Demo/Prototype. Resolution

Chapter 8 - Designing a Rewards Program

Proof of Learning: Assessment in 12 Analysis of Learning Serious Games

Designing and Evaluating Player Learning in Human Computation Games

Game Beta with 13 Project Progress Demos Intro/Credits/Analysis.

14 Project Work

15 Project Work

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design”

16 Final Project Demos Final Project.

Table 1: Course Schedule

ASSESSMENT

Anecdotally, we have heard many positive remarks from the partner teachers and students in the K-12 schools, and the university students have also reported positive experiences in the Educational and Serious Game Design course. Beyond the subjective feedback we have received, it is important to analyze the course’s effectiveness empirically. There are a few means by which we can judge the course’s success. Herein, we analyze the course’s success through pass/progression rates among students and feedback on the Student Instructional Report (SIR).

The table below demonstrates a comparison of five courses taught by the CGDD4303 instructor over the past four academic years. This represents seventeen sections of the course serving a total of over 400 students.

Number Average Student Weighted Average Course Term of Earned GPA in Student Earned GPA in Students Course Course

2011 Fall 44 2.31

Fundamentals 2012 Spring 30 2.50 of Game 2.57 Design 2012 Fall 70 2.57 2013 Fall 37 2.57

2011 Spring 32 2.53 Application Extension and 2011 Fall 32 2.31 2.48 Scripting 2012 Fall 29 2.62

2011 Fall 6 2.67 Mobile and Casual Game 2012 Fall 21 3.14 2.93 Development 2013 Fall 28 2.82

2011 Spring 4 3.00 Educational 3.33 and Serious 2012 Spring 18 3.39

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design”

Game Design 2013 Spring 22 3.36

2014 Spring 20 3.30

2011 Spring 7 2.00 Modeling and 2013 Spring 10 2.90 2.75 Simulation 2014 Spring 11 3.09

Table 2: Course Comparison

As the above table shows, the weighted average earned grade (A=4, F=0) for each course ranges from a low of 2.48 (Application Extension and Scripting) to a high of 3.33 (Educational and Serious Game Design). Students in the Application Extension and Scripting course earned on average a 2.48, representing a C-B average; students in the Educational and Serious Game Design course earned on average a 3.33, representing a B-A average. The Fundamentals of Game Design course is a first-year course and is included in the comparison to show that the difference between upper and lower division courses is statistically insignificant. Further, all sections of the courses listed in the table above were taught by the same instructor, and all courses are required of the major except Modeling and Simulation. Students take the Mobile- Casual Game Development and Educational-Serious Game Design in the same year—typically in the Junior year of studies. Consequently, it is all the more significant that there is nearly a half letter grade difference on average between the courses. Clearly, there is a performance difference among the students, and since the student population and instructor remains consistent among the courses, something else must impact the students’ performance.

Certainly, the course content varies from course to course, but the sections within a course remain stable with little change internally to the courses during the period of these results. We believe the Educational and Serious Game Design course shows markedly higher student performance given the motivation and collaboration within the course. The content is relatively (based upon placement within the curriculum) no less or no more rigorous than in other courses within this comparison.

Moving beyond student performance, let us examine the students’ evaluation of the course using the Student Instructional Report (SIR). This end-of-term survey queries students as to their opinions on communication, faculty/student interaction, assessment, outcomes, and student effort within the course. As you can see in the table below, which shows the average SIR scores per category for four courses, the Educational and Serious Game Design course rated the lowest in overall evaluation; so while the students feel the course was organized and planned well and there was ample communication and interaction (relative to other courses), they felt the course outcomes were not met as well as they were in other courses. Additionally, assessment was not rated as high as in other courses.

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design”

Foundations App Mobile Ed Scripting Casual Serious

COURSE ORGANIZATION AND 4.57 4.47 4.25 4.47 PLANNING

COMMUNICATION 4.69 4.56 4.42 4.52

FACULTY/STUDENT INTERACTION 4.62 4.56 4.49 4.59

ASSIGNMENTS, EXAMS, AND 4.52 4.40 * 4.28 GRADING

COURSE OUTCOMES 4.30 4.17 * 3.97

STUDENT EFFORT AND INVOLVEMENT 3.96 4.14 3.91 3.95

OVERALL EVALUATION 4.33 4.07 4.10 4.01

Table 3: SIR II Course Comparison on a 5-Point Scale (* no results available)

These SIR results are intriguing and motivate us to more clearly articulate how the Educational and Serious Game Design course’s assessment and outcomes are supported by the well- received organization, communication, and interaction. This may make the assessment within the course more explicitly understandable and better received by the students.

Finally, we can judge the success of the Educational and Serious Game Design course by the number of games created and the continued collaborative partnership with the K-12 schools. In 2013, our university was invited to formally become a “Partners In Education” with the local elementary school with whom we’d partnered over the past four years. This achievement validated our work and helped demonstrate the value of the work we have been doing. The collaboration through this course has resulted in over two-dozen games created. A sample of these games can be seen in the images on the following page.

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design”

Figure 2: Sample Games Developed

LESSONS LEARNED

As discussed earlier, we require all teams to define a Game Design Document (GDD). We have found that teams that more clearly and completely articulate a complete GDD then go on to be more successful in implementing a complete game as they intended. Teams that do not get

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design” detailed and leave their GDD vague or incomplete typically go on to have incomplete games and are less successful.

When we first offered this course four years ago, we assumed we could install the resultant games onto the partner elementary school’s computers. While this was possible and did happen eventually, it was laborious to gain permission and work with local school administrators to get the software installed. After this first year, we changed tactics and required that all games developed by our university students must run within the Internet browser. As a result, as long as our university’s host server was “white listed” (i.e., allowed to be accessed from the K-12 school computers), then our games would work remotely via the Web browser; no installation to the local computers was required. This follows a modern trend in allowing software to be hosted remotely and run through a browser, so this solution enabled our university students to explore this emerging trend in software delivery, while at the same time, this approach solved the access problem at the local K-12 schools.

Further, we have learned that some teams will not complete their projects within the term. In other courses, the students could earn a grade and progress in their academic studies, but since we have ‘real customers’ waiting for these games to be delivered, we do not want to progress students with an incomplete educational game. The solution we devised was to allow students to progress with a less-than-perfect grade (and game) but also allow the possibility of an “incomplete” so students could finish the game over the summer immediately following the course term. The incomplete grade would then be replaced with an ‘A’ in the course and the completed game would be delivered. This is a win-win opportunity to maximize the chance that the game will be completed (and delivered to the waiting customer) without imposing the artificial deadline of the end of the term. We tried this approach in 2013 and had seven (of nine) students leveraging this option, so we believe this is a viable approach to affording flexibility without requiring perfection.

Finally, our most important lesson learned is that this collaboration is possible because of the support and flexibility of the partner K-12 schools’ administration and teachers. We have been fortunate to find such supportive personnel involved at our partner school, but this collaboration would not be successful without their interest and participation. We encourage readers to form such partnerships and develop relationships with local businesses and schools to empower service-based learning at your universities. It takes time to develop, but once in place, the rewards are immeasurable. In the past year, we have expanded our collaboration into two additional schools, and we continue to develop partners at these three local schools in our service area.

CONCLUSION

We believe this course offers a unique opportunity for students to apply the skills they have developed in previous courses and use the new, focused material within this course centered on serious and educational game development. The partnership we have formed with local schools has been validated and deemed a success by the local school district’s superintendent,

Syllabus 4/1 (2015) J. Preston, “Educational & Serious Game Design” the principal at one of our partner schools, the teachers and partner students, and by the students within the university course. We believe such service-based learning is motivating to the university students and have evidence to suggest learning improves as a result of this “real customer” environment. We hope that interacting with the K-12 students encourages them to consider attending college if they hadn’t otherwise thought of this possibility, and we also believe the process of jointly developing games focused on STEM learning outcomes is fruitful in promoting STEM fields as well as critical thinking and learning in general.

REFERENCES

Bououd, I., Boughzala, I., The design of a collaboration-oriented serious game, 2012 International Conference on Communications and Information Technology (ICCIT), IEEE, Hammamet, pp. 62-67, 2012.

Curry, A., Monopoly Killer: Perfect German Board Game Redefines Genre, Wired Magazine, 17, 4, March, 2009.

Gagne, The Conditions of Learning. New York: Holt, Rinehart & Winston, 1985.

Gold, S., IGDA Curriculum Framework: The Study of Games and Game Development, version 3.2 beta, IGDA, 2008, http://wiki.igda.org/images/e/ee/Igda2008cf.pdf, accessed March 2014.

Jacobs, S., Building an education ecology on serious game design and development for the One Laptop Per Child and Sugar platforms: A service learning course builds a base for peer mentoring, Cooperative Education internships and sponsored research, 2010 International IEEE Consumer Electronics Society's Games Innovations Conference (ICE- GIC), IEEE, Hong Kong, pp 1-6, 2010.

Kapralos, B., A course on the design and development of serious games and virtual simulations, 2012 IEEE International Games Innovation Conference (IGIC), IEEE, Rochester, NY, pp. 1- 4, 2012.

Salazar, M.G., Mitre, H.A., Olalde, C.L., Sanchez, J.L.G., Proposal of Game Design Document from software engineering requirements perspective, 2012 17th International Conference on Computer Games (CGAMES), IEEE, Louisville, KY, pp. 81-85, 2012.

Tang, S., Game Content Model: An Ontology for Documenting Serious Game Design, Developments in E-systems Engineering (DeSE), IEEE, Dubai, pp. 431-436, 2011.

Winn, B., The Design, Play, and Experience Framework, Handbook of Research on Effective Electronic Gaming in Education, Michigan State University, 2009.