Audience Response Software As a Learning Tool in University Courses

education sciences

Article Audience Response Software as a Learning Tool in University Courses

Ana María Díez-Pascual 1,* and María Pilar García Díaz 2 1 Department of Analytical Chemistry, Physical Chemistry and Chemical Engineering, Faculty of Sciences, Alcalá University, 28871 Madrid, Spain 2 Department of Signal Theory and Communications, Polytechnic High School, Alcalá University, 28871 Madrid, Spain; [email protected] * Correspondence: [email protected]; Tel.: +34-918-856-430

Received: 28 October 2020; Accepted: 23 November 2020; Published: 25 November 2020

Abstract: The use of information and communication technologies (ICTs) has become a fundamental tool in all areas of today’s society, including higher education. Lessons cannot be envisaged without the use of tools such as computers, tablets or mobile devices. Many lecturers use audience response systems (ARS) to keep their classes engaged. ARS software allows teachers to interact with students via polls, text responses, or multiple-choice questions displayed via their mobile devices. A new example of the use of this type of devices in education is gamification, a technique that uses a set of activities with ludic character as a learning methodology in order to facilitate the acquisition of knowledge and competences. One of the most used gamification tools is Kahoot!, a free learning application based on a mixture of game and creativity, which encourages attention and participation of students through questions and answers formulated by the teacher and designed in a way that students respond via their mobile phones. This paper examines the use of Kahoot! in a subject belonging to the chemistry area. In order to assess the benefits of this tool, it was tested in a group of students to review the knowledge and skills acquired during the theoretical lessons prior to the exams, and the academic results were compared with those of a control group of students who did not use the tool. The results demonstrate that the use of Kahoot! led to an improvement in the teaching–learning process of the students and a noteworthy rise in their marks, and that its positive effects rise with increasing the frequency of use of this didactic tool.

Keywords: gamiﬁcation; higher education; competences; Kahoot!; information and communication technologies; university students

1. Introduction Taking into account the relevance of information and communication technologies (ICTs) in all areas of our current society, lessons cannot be envisaged without the use of technological tools [1]. Therefore, one of the leading focuses in the educational area is to define how ICTs can contribute to the current requirements of education. Several researchers have reported that ICTs can improve the quality and effectiveness of education by increasing student motivation, via making the achievement of basic skills simpler and through improving the teacher training [2]. On the other hand, the question that arises is how to promote student interest and participation in the classroom in order to develop an active role in the learning process, mainly because students do not recognize the application of knowledge to their field of study. Previous studies [3] demonstrated an effective learning process when the students played an active role. Collaboration, cooperation and student engagement as well as problem-based learning are essential for knowledge acquisition. Although it is not feasible to address all the problems listed above, it is possible to try to solve some of

Educ. Sci. 2020, 10, 350; doi:10.3390/educsci10120350 www.mdpi.com/journal/education Educ. Sci. 2020, 10, 350x FOR PEER REVIEW 22 of of 19 solve some of them in order to minimize their influence on the student’s academic results. In them in order to minimize their influence on the student’s academic results. In particular, the didactic particular, the didactic contents can be reviewed with tools that are attractive to the student. contents can be reviewed with tools that are attractive to the student. Lately, most university students attend the lectures with tablets, smartphones or laptops. In Lately, most university students attend the lectures with tablets, smartphones or laptops. particular, mobile devices are used to encourage interaction between students and promote self‐ In particular, mobile devices are used to encourage interaction between students and promote learning (M‐learning). Mobiles are proposed as a complementary tool for the assimilation of concepts self-learning (M-learning). Mobiles are proposed as a complementary tool for the assimilation by students. Nonetheless, it is essential to show scholars how to the use mobile devices for of concepts by students. Nonetheless, it is essential to show scholars how to the use mobile educational aims [4]. A recent example of the usage of mobiles in education is gamification, which devices for educational aims [4]. A recent example of the usage of mobiles in education is attracts student encouraging action, promoting learning and solving problems [5]. Clearly, current gamification, which attracts student encouraging action, promoting learning and solving problems [5]. technologies have collaborated to the development of gamification: games can be performed in a Clearly, current technologies have collaborated to the development of gamification: games can be dynamic way and enable the incorporation of typical devices such as smartphones, thus opening up performed in a dynamic way and enable the incorporation of typical devices such as smartphones, novel possibilities for attracting students’ attention [6]. thus opening up novel possibilities for attracting students’ attention [6]. Gamified activities based on the use of audience response systems (ARS) are a common practice Gamified activities based on the use of audience response systems (ARS) are a common practice nowadays. They incorporate different game elements such as goals, rules, competitions, timing, nowadays. They incorporate different game elements such as goals, rules, competitions, timing, points, points, or feedback [5]. They use clickers, small movable devices that transmit and record student´s or feedback [5]. They use clickers, small movable devices that transmit and record student´s answers answers to questions displayed in the classroom. Once the educator asks a question, students press to questions displayed in the classroom. Once the educator asks a question, students press buttons buttons on their clickers to reply. Owed to the real‐time feedback offered by this tool, students can on their clickers to reply. Owed to the real-time feedback offered by this tool, students can evaluate evaluate their level of comprehension of the material taught [7]. This type of technology promotes their level of comprehension of the material taught [7]. This type of technology promotes students’ students’ active collaborative learning, encourages interaction and engagement, enhances classroom active collaborative learning, encourages interaction and engagement, enhances classroom attendance attendance and attention [8]. However, clickers can be stressful due to technological issues, such as and attention [8]. However, clickers can be stressful due to technological issues, such as when remote when remote devices do not function properly, and can be costly for most educational institutions. devices do not function properly, and can be costly for most educational institutions. Amongst the most commonly used online game‐based learning platforms are Kahoot!, Quizziz, Amongst the most commonly used online game-based learning platforms are Kahoot!, Quizziz, Quizlet Live and TalentLMS. All are useful, interactive, and fun ways to learn and/or review Quizlet Live and TalentLMS. All are useful, interactive, and fun ways to learn and/or review concepts, concepts, each with its own advantages as well as drawbacks. Kahoot! was one of the original online each with its own advantages as well as drawbacks. Kahoot! was one of the original online games games launched in August of 2013. It is a free application based on a mixture of entertainment and launched in August of 2013. It is a free application based on a mixture of entertainment and creativity creativity that has increased attention among educators owed to its easiness and its capacity to that has increased attention among educators owed to its easiness and its capacity to develop dynamic develop dynamic activities. Questionnaires can be launched at the beginning of a course or a thematic activities. Questionnaires can be launched at the beginning of a course or a thematic unit, to check the unit, to check the former knowledge of the students on the topic, during the class as an activity to former knowledge of the students on the topic, during the class as an activity to keep the attention of keep the attention of the students or at the end of a lesson or even the whole subject to assess the the students or at the end of a lesson or even the whole subject to assess the learning acquired [6]. learning acquired [6]. The teacher chooses the questions and defines the right answer as well as the response time. The teacher chooses the questions and defines the right answer as well as the response time. Typically, the queries are projected in the class and the scholars respond to them by means of their Typically, the queries are projected in the class and the scholars respond to them by means of their mobiles (Figure1). Points are provided for the fastest right answers, while incorrect answers get no mobiles (Figure 1). Points are provided for the fastest right answers, while incorrect answers get no points. Applicants can enter nicknames, and their points are matched after every question. The design points. Applicants can enter nicknames, and their points are matched after every question. The design style is attractive and very simple. style is attractive and very simple.

Figure Figure1. Students 1. Students answering answering a question a question with Kahoot! with Kahoot! (Source: (Source: https:/getkahoot.com/). https:/getkahoot.com /). Educ. Sci. 2020, 10, 350 3 of 19

The major benefits arising from this approach are: (a) Students feel it to be an entertaining activity. (b) It makes them actively participate in the learning process. (c) It is a shared experience (everyone responds at the same time). (d) Teachers can get immediate and fair information about how students have assimilated the concepts described in the lessons, can identify their weak and strong points, detect specific problems in the learning process with a certain topic and take the corresponding actions. (e) Students obtain instantaneous constructive feedback on their learning without risk of shame [9]. (f) It can be used for the initial, formative and final evaluation of the scholars. This tool is more active than Quizziz and Quizlet Live, owed to its intense colors onscreen and the animated music, which boosts students to reply each question quickly. There is a leaderboard of the players depending on speed and accuracy. It allows the preparation of online questionnaires, discussions or exams [10]. Moreover, it only requires an internet connection, and does not need former software or knowledge, since questions can be created quickly and in a very simple way. It also enables one to record the answers of all the students in an Excel file, and to create multiple choice question images, videos and diagrams [11]. However, it also shows some drawbacks: students are persuaded to answer quickly, hence speed tends to prevail over correctness. Further, it is easy to copy the responses of other students. Previous studies [12,13] have compared the results of the same questionnaire answered using three different methodologies: printed form, a remote control system and Kahoot!, and increased motivation and participation in the classroom were found for the latter methodology. Former researchers on Kahoot! agree that the application of this tool enhances both the learning process and the students’ participation, as well as the relationship amongst classmates. Considering this tool as a game enables one to share the comments among the students in a relaxed manner, and also promotes the participation of the whole class, facilitating the development of social skills [14]. Further, in view of the relevance that the formative and continuous assessment has nowadays in our educational system, this strategy is very appropriate, since it can also allow the design of test exams about specific concepts. Quizizz is very similar to Kahoot!, except that the questions and possible answers are displayed individually on each student device. The game is not as lively and musical as Kahoot!, though it creates a well set starting line for students to see their upgrading. The major advantages of this platform are that it is student-paced, hence no one gets disappointed because their device did not load the game fast enough to compete. Besides, educators can exhibit a student progress dashboard on the projector and immediately realize how many queries the class replied right/wrong. It is not necessary to complete a game live in-person. Homework mode can be used to assign it to be completed by a deadline. Nonetheless, most of the excitement is lost, since everyone answers the different questions at different times. Quizlet Live is more focused on vocabulary compared to the other gaming platforms. Students are randomly divided into small groups and are encouraged to move next to their partners, where they have the opportunity to interact and communicate with other students to win the game. This mixes up their environment and encourages physical movement, which boosts cognitive function. It is a true team play: each team player is provided with a set of words to match to the definition, and all teams are excited trying to be the first to answer all the given words. However, Quizlet Live is not as useful for grammar or other types of questions that are not vocabulary-based. In addition, at least six students are needed to play a game (at least two teams of three students) and at least six cards in a flashcard set. TalentLMS is an award-winning, cloud-based learning management solution designed specifically for usability and easy course creation: the system enables you to create courses by re-using any material you already possess or by directly integrating content from the web. The main benefits of this platform are its ease of use, great integration, and numerous add-ons. It has a minimalistic interface that allows users to focus on the content and it is easily customizable. It offers embedded content tools for video, audio, SCORM, xAPI and flash content, along with search functionality to find and insert YouTube videos, SlideShare presentations, Wikipedia articles, etc. It can be used on multiple devices including desktop, smartphone, tablet, and phablet. Further, the platform is easy to navigate and does not require Educ. Sci. 2020, 10, 350 4 of 19 much training. Additionally, it provides rewards to the winners such as discounts, special offers and so forth. Levels can be assumed as a hierarchy that the user can move up on (such as receiving a promotion at work or completing a course at school). By default, all users start on Level 1. Moving up a level enables users to unlock courses, and TalentLMS administrators have the choice to arrange the minimum level required for unlocking any particular course. On the other hand, it is very complex to make any changes to the interface or to add items to the portal, and the tool can be expensive for the institutions. The following is a case study, carried out by a group of students representing about 37% of the total enrolled scholars in a subject of chemistry at the University of Alcalá (Spain) during the academic years 2018–2019 and 2019–2020. In order to be able to assess the possible benefits of the application of ARS software, a game-based learning platform such as Kahoot!, on the students’ academic progress, results were compared with a control group of students who did not use the tool, and also with those obtained for previous years in the same subject. Although a large number of studies dealing with Kahoot! have been published in recent years, very few of them have accomplished this comparative approach [15]. In particular, to the best of our knowledge, it is the first time that this type of comparison is applied to a subject in the field of chemistry. Further, the methodology has been implemented in two consecutive academic courses, in order to check for reproducibility, and the influence of Kahoot! questionnaires frequency on the scholars’ academic performance has also been evaluated.

2. Gamification as a Didactic Tool Different methods related to education and gamification have been developed, such as game-based learning (GBL), and two main education gamification categories have been proposed [16]: structural and content. According to former investigations, the success of gamification in the educational field depends on several factors. Firstly, the adaptation of game design elements is very complex. Secondly, it is conditioned by the learning styles. Thirdly, it is highly influenced by the motivational response of the students to the game elements. Finally, the learning environment is also of significant importance.

2.1. Levels of Gamification Modification in Education The adaptation of gamification in education can take place in several levels: In the first one, a single application is employed to enhance the student´s activity; nonetheless, the course conforms to the conventional teaching way. Such applications are commonly interactive response systems (IRS), such as Kahoot!, Socrative, Mentimeter, Quizlet Live, Poll Everywhere or Plickers [17]. These tools offer numerous question–answer choices, quizzes and short answers to select from, and frequently use points and scoreboards. Students using IRS have been reported to be more involved, interested and concentrated compared with paper forms or non-game-based student responses [18]. The second level consists of the incorporation of game design elements to the course to recompense certain actions. A broad variety of elements can be selected to boost a change in the student’s behavior including points, badges and leaderboards [19]. Points are the simplest way to reward a competitor for completing an action. This technique is useful to motivate people who like to feel progress. The gathered points are frequently displayed on a leaderboard, which makes the performance of every participant accessible to the others. Badges are a visual plot of some achievement and are used to boost and distinguish certain behaviors. In addition, they can be used to incentivize reaching a milestone. Two types of badges can be set: invisible and visible. Participants typically know what kind of behavior or action they have to take to attain the second ones, while the former would arise as a surprise; hence only those who spontaneously have a certain behavior are rewarded. In the third level, game design elements are incorporated to attain improved results when using gamification. They offer processes and factors such as the Hook Model [20], which considers the four phases of a typical habit-forming product. Other models are the Fogg’s Behavior that identifies three factors that should converge so that a behavior takes place [21]: motivation, ability and trigger. Besides, Educ. Sci. 2020, 10, x FOR PEER REVIEW 5 of 19 Educ. Sci. 2020, 10, 350 5 of 19 factors that should converge so that a behavior takes place [21]: motivation, ability and trigger. Besides, the self‐determination theory (SDT) [22] outlines things people find motivating, based on threethe self-determination main ideas: autonomy, theory competence, (SDT) [22] outlines and relatedness. things people find motivating, based on three main ideas:The autonomy, final and competence, most complex and level relatedness. takes into account the different learning styles and the diverse scholarThe types. final To and prevent most complex masking level the possible takes into positive account consequences the different learningof gamification, styles and proper the diversedesign ofscholar the learning types. Toenvironments prevent masking is required, the possible considering positive students’ consequences differences of gamification, and motivations proper [23]. design of the learning environments is required, considering students’ differences and motivations [23]. 2.2. Students’ Types, Learning Styles and Environments 2.2. Students’ Types, Learning Styles and Environments GBL has been applied in all educational levels, from elementary education [24] to university GBL has been applied in all educational levels, from elementary education [24] to university courses [25]. However, it is important to notice different learning styles when GBL is used: an Index courses [25]. However, it is important to notice different learning styles when GBL is used: an Index of Learning Styles (ILS) has been proposed for classifying learners, and four main types have been of Learning Styles (ILS) has been proposed for classifying learners, and four main types have been identified: active/reflective, sensitive/intuitive, visual/verbal and sequential/global. Further models identified: active/reflective, sensitive/intuitive, visual/verbal and sequential/global. Further models have been developed in which learning styles are based on a four‐stage cycle [26]. Other researchers have been developed in which learning styles are based on a four-stage cycle [26]. Other researchers recognized typologies of game competitors and their features such as competitiveness, openness and recognized typologies of game competitors and their features such as competitiveness, openness and investigative skills [27]. The Bartle taxonomy of player types identifies four characters: achievers, investigative skills [27]. The Bartle taxonomy of player types identifies four characters: achievers, explorers, socializers and killers. Several studies have also been devoted to exploring the student explorers, socializers and killers. Several studies have also been devoted to exploring the student classification; though, there is an absence of works analyzing the influence of gamified elements on classification; though, there is an absence of works analyzing the influence of gamified elements on the the learning environment. It appears that player types can be correlated with the students’ activities learning environment. It appears that player types can be correlated with the students’ activities and and their participation patterns. their participation patterns. In recent years, gamified learning environments have grown substantially to achieve the needs In recent years, gamified learning environments have grown substantially to achieve the needs of of both teachers and students. Applications and platforms with different forms have been developed both teachers and students. Applications and platforms with different forms have been developed to to fulfill the needs from primary school (e.g., ClassDojo, Socrative, Schoooools.com) to higher fulfill the needs from primary school (e.g., ClassDojo, Socrative, Schoooools.com) to higher education education (e.g., ClassFlow, Google School) [25]. In particular, you can push lessons out to a (e.g., ClassFlow, Google School) [25]. In particular, you can push lessons out to a smartphone if your smartphone if your students have one and collect assessment data through the ClassFlow lesson students have one and collect assessment data through the ClassFlow lesson (Figure2). (Figure 2).

Figure 2. 2. DifferentDifferent types types of activities of activities created created in the inClassFlow the ClassFlow Activity ActivityBuilder (Source: Builder https://classflow.com/teachers(Source: https://classflow.com/‐teachers-use-classflow-activitiesuse‐classflow‐activities/). /).

GamificationGamification in in higher education education is is effective effective if a digital learning management system such as Moodle or or Blackboard Blackboard is is applied. applied. Students Students gain gain access access to various to various materials, materials, communicate communicate with each with othereach otherand andteachers teachers and andcan canindependently independently test test their their knowledge knowledge online. online. Teachers Teachers can can provide materials and setup reviews, as well as evaluate and supervise students’ activities. The importance of gamification gamification has been corroborated by by the EU project calls in the Horizon 2020 funding scheme (ICT(ICT-21-2014‐21‐2014 and ICT-24-2016)ICT‐24‐2016) [[28].28]. Educ. Sci. 2020, 10, 350 6 of 19

3. Purpose, Research Questions, and Methodology

3.1. Aims of the Study and Research Design The general purpose of this article is to assess the extent to which students’ knowledge can be improved via using a game-based learning platform, in particular Kahoot! as a learning tool, its possible benefits on the students’ academic performance and how the efficiency of this methodology depends on its frequency of application. In order to evaluate its beneficial effects, it was implemented only in a group representing around 37% of the total enrolled scholars, both during the academic courses of 2018–2019 and 2019–2020, and the outcomes were compared with a control group of scholars who did not use the tool, and also with those attained for previous academic courses. The frequency of use of the tool during the period 2019–2020 was twice that of the academic course 2018–2019. The questionnaires did not imply extra work, and the two groups received the same teaching hours. Further, there was no correlation between the different groups and the student’s abilities. The idea was to find out how well they recalled the information during the exam, considering that they had already answered some of the questions during sessions on Kahoot!. The use of the tool was optional, since its compulsory character would have turned it into a duty, whereas activities in higher education should not be mandatory. Its voluntary nature promotes the students’ autonomy and management of the learning process, which is crucial from an educational perspective. Therefore, the questionnaires made with the tool were not included within the assessment criteria in the course guide, although they had a positive effect on the final mark of the students. The specific aims of the current study are listed below:

To assess the potential of gamification within the framework of higher education. • Implement the use of novel ICTs within an educational context. • To determine whether there are differences and whether improved learning results are attained • when using an ARS software compared to conventional teaching methods. To encourage the practice of new technologies and digital games in teaching–learning processes • via ARS software. Increase the motivation of the students for learning the course contents through an attractive • environment, which will result in a better academic performance. To improve the understanding of the gamification tool in order to use it as a means of evaluation • in subjects belonging to the field of chemistry. Boost students’ capacity for discussion, critical logic, choice, and making decisions. • Facilitate a game-based learning platform as an educational technology. • Considering the differences between GBL and the learning models most commonly used in higher education, the following questions arose: (1) Can gamification tools like Kahoot! be used to promote student learning and engagement, that is, to support the active participation of the students in the matter? (2) Can this tool aid in memorizing concepts, thus facilitating the students’ learning process and consequently improving their academic progress? (3) Can it expand students’ skills by reviewing concepts in an amusing and motivating way? (4) Can it help to identify the weak and strong points of the students, detect specific problems in the learning process and take the corresponding actions? (5) Does the effect of this methodology on the students’ learning process depend on its frequency of use?

3.2. Research Methodology The research methodology chosen was action research, given that it uses a cyclic or spiral process which alternates between action and critical reflection suitable for an educational context [29]. This methodology includes five consecutive steps: problem identification, action plan, data acquisition, data analysis and interpretation, and plan for future action. Educ. Sci. 2020, 10, 350 7 of 19

Following the abovementioned steps, firstly, different questionnaires were designed using the web Educ. Sci. 2020, 10, x FOR PEER REVIEW 7 of 19 https://kahoot.com, which is free and very easy to deal with. Afterwards, the teachers explained to the studentsimplemented, the aims the of students using the voluntarily tool and the participated work dynamics. in the different Once the launched ARS software questionnaires. was implemented, All the thestudents students to whom voluntarily it was participated proposed agreed in the to di ffparticipateerent launched in the questionnaires. activity, although All it the did students not have to whomweight it in was the proposedfinal grade. agreed to participate in the activity, although it did not have weight in the finalEach grade. questionnaire consisted of 15–20 questions and provided four different answers with a singleEach correct questionnaire option; depending consisted on of the 15–20 difficulty, questions the andstudents provided had between four diff 30erent and answers 60 s to answer with a singleeach question. correct option; Some were depending short onquestions the diffi referringculty, the to students theoretical had concepts between while 30 and others 60 s to required answer eachsmall question. numerical Some calculations. were short Additionally, questions referringmany of them to theoretical included concepts an image while or short others explanatory required smallvideo. numerical As the test calculations. progressed, Additionally,and the questions many were of them answered, included the anteacher image promoted or short explanatorya process of video.collaborative As the reflection test progressed, that aided and reviewing the questions concepts. were answered,Thus, a debate the teacherwas generated promoted that a had process a very of collaborativepositive impact reflection on the students’ that aided learning reviewing process. concepts. Thus, a debate was generated that had a very positiveThe impact teacher on provided the students’ “feedback” learning after process. each questionnaire, encouraging students to think about theirThe issues. teacher Students’ provided participation “feedback” and after scores each were questionnaire, monitored encouragingvia “reports” students option, which to think enables about theirone to issues. identify Students’ the most participation difficult questions and scores that need were reteaching, monitored identify via “reports” if some option, learners which need enables follow oneup or to identifystruggled the with most completing difficult questions the challenge. that need Further, reteaching, the “Questions identify if some tab” learnersallowed needteachers follow to upobtain or struggled a detailed with breakdown completing of correct the challenge. and incorrect Further, answers, the “Questions assess class tab” allowedperformance teachers and to identify obtain agaps detailed in knowledge. breakdown Reflection of correct and incorrectanalysis answers,on the gamification assess class activity performance results and were identify explored gaps in knowledge.successive seminar Reflection discussions. and analysis Finally, on the the gamification teachers reflected activity on results the results were explored and proposed in successive future seminaractions. discussions. Finally, the teachers reflected on the results and proposed future actions. To verifyverify whether whether the the ARS ARS software software contributed contributed positively positively to learning to learning outcomes, outcomes, questions questions similar tosimilar those to used those in used the questionnaires in the questionnaires were included were included in the final in the exam final of exam both theof both control the groupcontrol and group the groupand the using group it asusing a learning it as a learning tool. Thus, tool. it wasThus, possible it was possible to corroborate to corroborate if the ratio if the of positive ratio of answerspositive increasedanswers increased for students for students that have that reviewed have reviewed the concepts the concepts with this with gamification this gamification means. means.

4. Design and Implementation of the ARS Software The easiesteasiest way way to to create create the gamiﬁcationthe gamification tests istests using is ausing computer. a computer. The address The of address the application of the is:application https://kahoot.com is: https://kahoot.com.. Once inside, Once you have inside, to identifyyou have yourself to identify as a useryourself or create as a anuser account or create if it an is theaccount ﬁrst timeif it is you the use first the time tool you (Figure use the3). tool (Figure 3).

Figure 3. Screenshot of the home page of Kahoot! Figure 3. Screenshot of the home page of Kahoot! There are two options in each user’s space: create a new Kahoot! or modify a previous one. The creator tool allows designing four different types of tests: (1) questionnaires (quiz) to select the correct option from multiple answers, (2) puzzles (jumble) to order a sequence of stages, (3) debate (discussion), where open questions are formulated, and (4) survey to make an inquiry about students’ opinions on a specific topic. The tool allows you to add details such as a title, a depiction, the idiom, Educ. Sci. 2020, 10, 350 8 of 19

There are two options in each user’s space: create a new Kahoot! or modify a previous one. The creator tool allows designing four diﬀerent types of tests: (1) questionnaires (quiz) to select the correct option from multiple answers, (2) puzzles (jumble) to order a sequence of stages, (3) debate (discussion), where open questions are formulated, and (4) survey to make an inquiry about students’ Educ. Sci. 2020, 10, x FOR PEER REVIEW 8 of 19 opinions on a speciﬁc topic. The tool allows you to add details such as a title, a depiction, the idiom, the intended audience and a cover picture. In addition, it allows you to include a link to a video to be the intended audience and a cover picture. In addition, it allows you to include a link to a video to be played in the background when the scholars are joining the activity. played in the background when the scholars are joining the activity. Once the type of exercise has been selected, the questions and up to four possible answers for each Once the type of exercise has been selected, the questions and up to four possible answers for one are written (Figure4). The teacher marks the correct option for each question and sets the time the each one are written (Figure 4). The teacher marks the correct option for each question and sets the students will have to answer the questions. This time can vary from one to another, being typically time the students will have to answer the questions. This time can vary from one to another, being between 30 and 60 s. In the case of questions such as debate or survey, up to four possible answers can typically between 30 and 60 s. In the case of questions such as debate or survey, up to four possible also be formulated, but the correct one cannot be established. An interesting option of this tool is the answers can also be formulated, but the correct one cannot be established. An interesting option of preview, which allows you to view the test as the students will see it in the classroom. In this option, this tool is the preview, which allows you to view the test as the students will see it in the classroom. the computer screen is divided into two parts: On the left is what the student will see projected in the In this option, the computer screen is divided into two parts: On the left is what the student will see classroom and on the right what he will see on his mobile phone. projected in the classroom and on the right what he will see on his mobile phone.

Figure 4. Example of the multiple‐choice response in a Kahoot! questionnaire. Figure 4. Example of the multiple-choice response in a Kahoot! questionnaire.

BeforeBefore starting the activity, activity, the teacher imparted a seminar in order to explain how to handle handle the the tooltool to to the the students. Thus, Thus, all all of of them them understood understood how how the the application application works works as as well well as as the the purpose andand dynamics of the game. TheThe stages stages to develop the gamification gamiﬁcation activities are: (a) The teacher launches a questionnaire, and the students join the game through their mobile (a) The teacher launches a questionnaire, and the students join the game through their mobile phones. phones. They need to enter the game PIN and a name (or nickname). The use of a pseudonym They need to enter the game PIN and a name (or nickname). The use of a pseudonym gave gave them more freedom when answering the questions. These data are recorded on the them more freedom when answering the questions. These data are recorded on the teacher’s teacher’s computer and will be displayed on the classroom screen along with the updated total computer and will be displayed on the classroom screen along with the updated total number number of participants. This allows students to verify if they have successfully registered. In of participants. This allows students to verify if they have successfully registered. In case the case the teacher recognizes any inappropriate nicknames, the participant can be kicked out. A teacher recognizes any inappropriate nicknames, the participant can be kicked out. A tune can be tune can be played during the game, to provide a suitable environment. played during the game, to provide a suitable environment. (b) The game begins when the professor presses on the “Start” button. The game session moves (b) The game begins when the professor presses on the “Start” button. The game session moves over over three stages. First, only the question is shown on the screen, then students can read the three stages. First, only the question is shown on the screen, then students can read the question question and the answers. Each response is framed in squares of different colors. They have to choose the correct option among four possible answers by pressing on their mobile phones the color that contains the response they consider correct (Figure 5). Educ. Sci. 2020, 10, 350 9 of 19

and the answers. Each response is framed in squares of different colors. They have to choose the correct option among four possible answers by pressing on their mobile phones the color that contains the response they consider correct (Figure5). (c) Once the response time has expired or all students have answered the question, students will be able to check on their mobile whether they have succeeded or failed. During this second stage, the screen will indicate the correct option and a bar chart showing the number of students who have answered each option. (d) The third stage is the scoreboard, showing the top five participants (Figure6). During this phase, each individual screen will display the scholars’ score for the last question, the total score and hisEduc./her Sci. position 2020, 10, x FOR in the PEER student’s REVIEW list. 9 of 19 (e) All questionnaires will go through these three stages until all questions have been displayed. At the end, the scholars can rate the activity with one to five stars, indicate whether they learned anything and rate their satisfaction level. Educ. Sci. 2020, 10, x FOR PEER REVIEW 9 of 19

Figure 5. Example of the multiple‐choice options the student can select with the mobile in a questionnaire (stage one).

(c) Once the response time has expired or all students have answered the question, students will be able to check on their mobile whether they have succeeded or failed. During this second stage, the screen will indicate the correct option and a bar chart showing the number of students who have answered each option. (d) The third stage is the scoreboard, showing the top five participants (Figure 6). During this phase, FigureFigure 5. each5.Example Example individual of theof screen multiple-choicethe multiple will display‐choice options the scholars’options the student scorethe student can for selectthe lastcan with question, select the mobile with the total inthe a questionnaire scoremobile and in a (stagequestionnaire one).his/her position(stage one). in the student’s list.

(c) Once the response time has expired or all students have answered the question, students will be able to check on their mobile whether they have succeeded or failed. During this second stage, the screen will indicate the correct option and a bar chart showing the number of students who have answered each option. (d) The third stage is the scoreboard, showing the top five participants (Figure 6). During this phase, each individual screen will display the scholars’ score for the last question, the total score and his/her position in the student’s list.

Figure 6.FigureDistribution 6. Distribution of answers of answers during during the the secondsecond stage stage of of a questionnaire a questionnaire along along with the with correct the correct answeranswer (left). ( Thirdleft). Third stage stage of the of the questionnaire questionnaire showingshowing the the scoreboard, scoreboard, with with the top the five top participant ﬁve participant (right). (right).

Once(e) theAll testquestionnaires is ﬁnished, will students go through receive these three on theirstages mobiles until all questions the number have been of correct displayed. and At incorrect the end, the scholars can rate the activity with one to five stars, indicate whether they learned answers and the position they occupy compared to their peers in a scoreboard. This information is very anything and rate their satisfaction level.

(e) All questionnaires will go through these three stages until all questions have been displayed. At the end, the scholars can rate the activity with one to five stars, indicate whether they learned anything and rate their satisfaction level.

Once the test is finished, students receive on their mobiles the number of correct and incorrect answers and the position they occupy compared to their peers in a scoreboard. This information is very valuable, since it allows students to improve self‐knowledge and individual learning. Starting from the idea of error as part of the learning process, and following the game, students can evaluate Educ. Sci. 2020, 10, 350 10 of 19 valuable, since it allows students to improve self-knowledge and individual learning. Starting from the idea of error as part of the learning process, and following the game, students can evaluate themselves and, therefore, identify the aspects in which they should focus as well as raise their doubts about the subject.Educ. Sci. 2020, 10, x FOR PEER REVIEW 10 of 19 At the end of the game, results can be downloaded into an Excel table with the statistics of the themselves and, therefore, identify the aspects in which they should focus as well as raise their doubts successes and failures by questions, which provide an overview of the level of knowledge of the group. about the subject. From the teacher’sAt the end perspective, of the game, this results tool can enables be downloaded them to compareinto an Excel the table progress with the of individualstatistics of the and group work throughsuccesses checking and failures the by contributions questions, which of each provide student, an overview paying of attention the level of to knowledge the class diversityof the and the differentgroup. learning From the rates.teacher’s This perspective, facilitates this the tool formative enables them evaluation to compare of thethe groupprogress and of individual a more in-depth and constructiveand group work feedback. through In checking addition, the knowing contributions the answersof each student, with the paying highest attention failure to the rate class allows the teacherdiversity to reflect and on the the different aspects learning with less rates. understanding This facilitates the and formative make decisions evaluation about of the group the best and strategies a more in‐depth and constructive feedback. In addition, knowing the answers with the highest failure to addressrate them.allows the teacher to reflect on the aspects with less understanding and make decisions about the best strategies to address them. 5. Results and Discussion 5. Results and Discussion The statistics of the students who participated in this gamification activity during the academic course 2018–2019The statistics and of their the students relationship who participated with the finalin this grade gamification obtained activity in during the course the academic are shown in Figure7course. 2018–2019 and their relationship with the final grade obtained in the course are shown in Figure 7.

Figure 7.Figure(a) statistics 7. (a) statistics of student of student participation participation in thein the gamification gamification activities activities duringduring the academic course course 2018–2019; (b) students who reviewed with the questionnaires and passed the course in the 2018–2019; (b) students who reviewed with the questionnaires and passed the course in the first call; first call; (c) students who passed the course and reviewed with the tool; (d) students who took and (c) studentspassed who the final passed exam the and course participated and reviewed in the gamification with the activities. tool; (d ) students who took and passed the final exam and participated in the gamification activities. As shown in Figure 7a, the gamification activity was undertaken by 37% of the total enrolled Asscholars. shown inMore Figure importantly,7a, the gamification 85% of the students activity waswho undertakenreviewed their by knowledge 37% of the with total the enrolled scholars.questionnaires More importantly, passed the 85% course of the in the students first call who (Figure reviewed 7b). In total, their 31.5% knowledge of the scholars with thewho questionnaires passed passedthe course course correspond in the first to students call (Figure who reviewed7b). In total, the concepts 31.5% before of the the scholars evaluation who exams passed with thethis course gamification activity (Figure 7c). Likewise, 36% of the students who presented and passed the final correspond to students who reviewed the concepts before the evaluation exams with this gamification exam for the course correspond to students who used this gamification tool (Figure 7d). Therefore, activitystatistics (Figure show7c). Likewise,that this methodology 36% of the is studentsvery beneficial who to presented improve students’ and passed knowledge the final acquisition. exam for the course correspondA similar trend to was students found whofor the used academic this gamificationyear 2019–2020 tool (Figure (Figure 8), in7 whichd). Therefore, the questionnaires statistics show that thiswere methodology carried out by is a very group beneficial representing to improve36% of the students’ total enrolled knowledge students. acquisition. A similar trend was found for the academic year 2019–2020 (Figure8), in which the questionnaires were carried out by a group representing 36% of the total enrolled students. Educ. Sci. 2020, 10, 350 11 of 19

Educ. Sci. 2020, 10, x FOR PEER REVIEW 11 of 19

Figure 8. (a) Statistics of student participation in the gamification activities during the academic Figure 8. (a) Statistics of student participation in the gamification activities during the academic course course 2019–2020. (b) Students who reviewed with the questionnaires and passed the course in the 2019–2020.first call; (b) ( Studentsc) Students who who reviewedpassed the course with the and questionnaires reviewed with the and tool; passed (d) Students the course who took in and the first call; (c) Studentspassed who the final passed exam the and course participated and reviewed in the gamification with the activities. tool; (d ) Students who took and passed the final exam and participated in the gamification activities. The comparison of Figures 7 and 8 indicates that upon increasing the frequency of use of the Thetool, comparison the percentage of Figures of students7 and that8 indicates passed the that course upon in increasing the first call the increased. frequency Further, of use ofthe the tool, the percentagepercentage of of students scholars using that passed the tool the who course attended in and the firstpassed call the increased. final exam Further,also improved. the percentage This of can be explained considering that this tool creates a good learning environment, increases student scholars using the tool who attended and passed the final exam also improved. This can be explained motivation and reduces their anxiety (as corroborated by the educators). Further, it results in greater consideringstudent that confidence this tool in their creates knowledge a good and learning skills. Thus, environment, the more frequent increases the questionnaires, student motivation the and reducesbetter their the anxiety learning (as outcomes. corroborated These byfindings the educators). also suggest Further,that students it results take formative in greater assessment student via confidence in theirthis knowledge ARS software and as skills.an amusing Thus, learning the more activity. frequent Students the are questionnaires, prone to be more the interested better in the the learning outcomes.course These if it findingsis entertaining, also suggest attractive, that and students enjoyable. take Consequently, formative assessment the higher via the this number ARS software of as an amusingquestionnaires, learning activity.the greater Students the students’ are prone interest to bein the more subject. interested It is worth in the noting course that if itstudents is entertaining, respond positively to learning activities that allow them to interact with their teachers and receive attractive, and enjoyable. Consequently, the higher the number of questionnaires, the greater the immediate feedback. students’ interestTo obtain in themore subject. information It is worth about the noting beneficial that students effect of respondthe ARS software positively on tothe learning students’ activities that allowlearning them process, to interact the statistics with their of the teachers students and that receive passed immediate the final exam feedback. on the first call were Tocompared obtain more to those information of scholars about of control the beneficialgroups who e ffdidect not of theuse ARSthe gamification software on tool, the as students’ well as to learning process,those the of statistics previous of years the studentsof the same that subject, passed and thethe results final exam are shown on the in firstFigure call 9. It were should compared be noted to those of scholarsthat in of all control cases the groups exam structure who did was not very use similar, the gamification with the only tool, exception as well that as in to the those last oftwo previous academic courses a few questions were similar to those previously formulated in the questionnaires, years ofin the order same to ascertain subject, the and contribution the results made are by shown this gamification in Figure9 .tool. It should be noted that in all cases the exam structureBased was on the very gathered similar, data, with it is the clear only that exception the percentage that of in students the last who two participated academic coursesin the a few questionsquestionnaires were similar and to passed those the previously exam is considerably formulated higher in the compared questionnaires, to those of the in ordercontrol to groups ascertain the contributionof students made who by had this not gamification undergone the tool. activity. This could arise from several reasons. Firstly, it was Basedeasier on for thestudents gathered that took data, questionnaires it is clear to that recall the the percentage information of since students they had who reviewed participated most of in the it before. Furthermore, students who participated in the gamification tool tended to mark less questionnaires and passed the exam is considerably higher compared to those of the control groups of incorrect answers in the exam, even if they did not answer correctly during the activity. These studentsconclusions who had can not be undergone drawn considering the activity. both the This control could groups arise and from students several from reasons. previous Firstly,years. it was easier for students that took questionnaires to recall the information since they had reviewed most of it before. Furthermore, students who participated in the gamification tool tended to mark less incorrect answers in the exam, even if they did not answer correctly during the activity. These conclusions can be drawn considering both the control groups and students from previous years. Therefore, participating in the gamification tool raised the efficiency of the students’ learning process, the improvement being more significant upon raising the game frequency. Educ. Sci. 2020, 10, x FOR PEER REVIEW 12 of 19

Therefore, participating in the gamification tool raised the efficiency of the students’ learning process, the improvement being more significant upon raising the game frequency.

Educ. Sci. 2020, 10, x FOR PEER REVIEW 12 of 19 Educ. Sci. 2020, 10, 350 12 of 19 Therefore, participating in the gamification tool raised the efficiency of the students’ learning process, the improvement being more significant upon raising the game frequency.

Figure 9. Percentage of students that passed the final exam on the first call: (a) control group (2018– Figure 9. Percentage of students that passed the final exam on the first call: (a) control group (2018– 2019);Figure (b 9.) groupPercentage2019); that ( bused of) group students the that gamification used that the passedgamification tool the tool final(2018–2019); (2018–2019); exam on (c) group( thec) group first that did call: that not ( usea did) controlthe nottool (2017–use group the (2018–2019);tool (2017– 2018);(b) group (d) group that used2018); that ( the dused) group gamification the that toolused the(2019–2020). tool tool (2019–2020). (2018–2019); (c) group that did not use the tool (2017–2018);

(d) group that usedFurther the information tool (2019–2020). about the effect of the ARS software on the students’ academic progress was Further informationobtained by comparing about the the statisticseffect ofof thethe final ARS subject software grading for on the the control students’ group and academic those who progress was obtainedFurther by comparing informationused the gamification the about statistics thetool e(Figures ffofect the of 10 final the and ARS subject11 for software academic grading courses on for the 2018–2019 the students’ control and academic2019–2020, group and progress those who was respectively). usedobtained the bygamification comparing thetool statistics (Figures of the10 finaland subject11 for gradingacademic for thecourses control 2018–2019 group and and those 2019–2020, who used respectively).the gamification tool (Figures 10 and 11 for academic courses 2018–2019 and 2019–2020, respectively).

Figure 10. Statistics of the ﬁnal marks of the subject during the academic course 2018–2019 for the group that used the gamiﬁcation tool (top) and the control group (bottom). Educ. Sci. 2020, 10, x FOR PEER REVIEW 13 of 19

Figure 10. Statistics of the final marks of the subject during the academic course 2018–2019 for the group that used the gamification tool (top) and the control group (bottom).

Clearly, the marks of the group participating in the activity are significantly better: for the academic course 2018–2019, only 2% of the students that participated in the questionnaires got a grade lower than 3, while for the control group this percentage increased to 9%. Thus, just 15% of the scholars that underwent the gamification activity failed the course, whereas for the control group it increased to 32%. On the other hand, 29% of the students that used the ARS software got a final score higher than 9, while for the control group this percentage was only 1%. Even better results were obtained for the academic course 2019–2020 (Figure 11), in which the differences between the control group and the students using the didactic tool were larger. In this case, 32% of the students that used the tool attained a final score higher than 9, while none of the students of the control group got this grading. For the group that participated in the gamification activities, just 1% of the scholars got a mark lower than 3 and 5.5% of them failed, whereas for the control group the corresponding percentages were 8% and 35%, respectively. Overall, results demonstrate that the use of the ARS tool led to a considerable rise in the final grade and in the number of students passing the subject. The application of a simple gamification tool has been proven to be very helpful for improving the students’ academic performance; the beneficial effects are more pronounced the greater the frequency of questionnaires. More questionnaires worked on likely imply more cognitive learning, since they involve a more participatory and applied learning than that of a conventional theoretical class. Further, it promotes knowledge retention, since it aids with memorizing key concepts. It was also noted that participants were more likely to remember the Educ. Sci.questions2020, 10, 350 they answered wrong, given that they had to think about the reason for the wrong answer 13 of 19 and seek the correct one. In addition, it triggers positive attention and focus, which would be reflected in reduced study time.

Figure 11. Comparison of the ﬁnal marks of the subject during the academic course 2019–2020 for the group that used the gamiﬁcation tool (top) and the control group (bottom).

Clearly, the marks of the group participating in the activity are significantly better: for the academic course 2018–2019, only 2% of the students that participated in the questionnaires got a grade lower than 3, while for the control group this percentage increased to 9%. Thus, just 15% of the scholars that underwent the gamification activity failed the course, whereas for the control group it increased to 32%. On the other hand, 29% of the students that used the ARS software got a final score higher than 9, while for the control group this percentage was only 1%. Even better results were obtained for the academic course 2019–2020 (Figure 11), in which the differences between the control group and the students using the didactic tool were larger. In this case, 32% of the students that used the tool attained a final score higher than 9, while none of the students of the control group got this grading. For the group that participated in the gamification activities, just 1% of the scholars got a mark lower than 3 and 5.5% of them failed, whereas for the control group the corresponding percentages were 8% and 35%, respectively. Overall, results demonstrate that the use of the ARS tool led to a considerable rise in the final grade and in the number of students passing the subject. The application of a simple gamification tool has been proven to be very helpful for improving the students’ academic performance; the beneficial effects are more pronounced the greater the frequency of questionnaires. More questionnaires worked on likely imply more cognitive learning, since they involve a more participatory and applied learning than that of a conventional theoretical class. Further, it promotes knowledge retention, since it aids with memorizing key concepts. It was also noted that participants were more likely to remember the questions they answered wrong, given that they had to think about the reason for the wrong answer and seek the correct one. In addition, it triggers positive attention and focus, which would be reflected in reduced study time. Regarding the academic course 2018–2019, it was found that the marks for the control group follow a normal distribution (Figure 10) with mean 5.62 and standard deviation 1.81 (Table1). The marks Educ. Sci. 2020, 10, 350 14 of 19 obtained for the following academic course display a very similar statistical distribution as shown in Figure 11, with mean 5.21 and standard deviation 1.49. In both cases the distribution is approximately symmetrical around the mean, which is within the interval 5 to 6, hence the mean, median and mode are almost coincident. However, the mentioned figures reveal that the marks for the group that used the didactic tool do not follow a statistical normal distribution, but a right skewed distribution; consequently, the mode is higher than both the median and the mean, which take values of 7.59 and 7.95 for the two aforementioned academic courses. All these statistical parameters have been included in Table1. Thus, the percentage of students with an average mark higher than 5–6 is 82% and 88% for 2018–2019 and 2019–2020 courses, respectively, and the median (7.59 and 7.95) is more representative of the obtained data than the mean. A potential cause of this skewness could be start-up effects, since a tool that has a lot of success during a long start-up period is known to create a positive skew on the data [30]. Nonetheless, results obtained for both academic years are quite similar, which accounts for the reproducibility of the data. On the other hand, for both courses, the relative standard deviation (RSD) of the mean final marks obtained applying the ARS software are smaller than those of the control group, indicating better precision and repeatability.

Table 1. Statistical comparison of the ﬁnal marks of the group that used the gamiﬁcation tool (1) and the control group (2).

Group Academic Course Mean Mark SD RSD (%) Median Mode 1 2018–2019 7.59 1.96 25.82 8.00 8.25 2 2018–2019 5.62 1.81 32.21 5.50 5.50 1 2019–2020 7.95 1.38 17.35 8.50 8.50 2 2019–2020 5.21 1.49 28.59 5.00 5.00 SD: standard deviation of the mean; RSD: relative standard deviation.

To further assess the effect of the questionnaires on the students’ knowledge and exam preparation, scholars were provided with a self-assessment rubric (see Table2). The students had to evaluate the gamification tool according to the following criteria: (a) questions’ content (to what extent they contain relevant information related to the main concepts learned throughout the course); (b) formulation and correctness (correct use of the specific vocabulary of the course, accuracy of the contents, spelling and syntactic correction, easiness to understand the meaning and so forth); (c) answer options (response choices reflect a logical flow, degree of difficulty, etc.); (d) presentation and creativity (degree of creativeness, number of pictures or videos included/music). An anonymous survey was conducted so that the students were able to value the new methodology with total freedom, indicating its weaknesses and strengths, and to express their opinions. Most of the students graded the gamification tool as “effective”, meaning that they believed it had a positive effect on their learning process, although there were still some aspects that could be improved. Some of them claimed that the number of questionnaires should be increased, may be one after each thematic lesson. Consequently, it is essential for teachers to carefully structure lectures in order that Kahoot! time is properly allocated. Others indicated that more numerical questions (including problems) should be included in the questionnaires, and that more time should be provided for each question. In addition, some indicated that it would be highly desirable to offer students with more and timely feedback. Overall, the students believe that they have managed to self-assess their learning process, which has become more active and practical, and that they had the opportunity to demonstrate what they have learnt, which is highly remarkable from the viewpoint of the metacognitive process. Additionally, other adjustments of the use of the tool were mentioned, for instance to allow scholars to make questions that can be replied by the other colleagues, as a review. Educ. Sci. 2020, 10, x FOR PEER REVIEW 15 of 19

Additionally, other adjustments of the use of the tool were mentioned, for instance to allow scholars to make questions that can be replied by the other colleagues, as a review.

Table 2. Rubric used to evaluate the effectiveness of the questionnaires.

Exemplary Effective Minimal Unsatisfactory Questions contain Questions contain Questions contain relevant information some information information not related to the main related to the directly related to the Questions do not Questions’ concepts learned in concepts learned in concepts learned in relate to the content the course. Critical the course. Do not the course. Do not concepts studied Educ. Sci. 2020, 10, 350thinking is necessary require too much require specific during the course.15 of 19 to answer the knowledge to be knowledge to be questions. answered. answered. Table 2. Rubric used to evaluate the effectiveness of the questionnaires. Seldom uses the Never makes use Usually uses the Always usesExemplary the Effectivespecific Minimalvocabulary. of the Unsatisfactory specific precise vocabulary. specific vocabulary. Poor grammar, vocabulary. Formulation Questions contain CorrectnessQuestions contain in some Questions contain Perfectrelevant spelling information and information related to mistakesinformation in not spelling directly Incorrectness in and related to the main spellingthe concepts and syntax. learned in related to the concepts Questions do not relate syntax. Clearly and syntax. Difficult spelling and CorrectnessQuestions’ content concepts learned in the Generallythe course. expresses Do not learned in the course. Do to the concepts studied expressescourse. what Critical the thinking require too much tonot understand require specific what duringsyntax. the course. clearly what the questionis necessary is asking. to answer knowledge to theknowledge question to is Questions are the questions. questionbe answered. is asking. be answered. asking. confusing. Seldom uses the specific Usually uses the precise Each questionAlways uses has the specific vocabulary. Poor Never makes use of the vocabulary. vocabulary. Perfect grammar, mistakes in specific vocabulary. Formulation and four thoughtful Correctness in spelling spelling and syntax. Correctspelling answer and syntax. is Incorrectness in spelling Correctness and syntax. Generally answersClearly connected expresses to what Each question has Difficult to understand and syntax. expresses clearly what included but does The correct Answer the contentsthe question that is asking. four answers what the question Questions are confusing. the question is asking. not fit the question answer is not Options promote critical generally connected is asking. perfectly. Answers included. thinking.Each Best question has four to the contents. thoughtful answers Each question has four seemCorrect random. answer is possibleconnected incorrect to the contents answers generally included but does not fit The correct answer is Answer Options answers.that promote critical connected to the question perfectly. not included. Quiz isthinking. very well Best possible the contents. Answers seem random. incorrect answers. Quiz is very well constructed and Quiz has missing Quiz is very well Presentation constructedQuiz is very and well Quiz was not well creative.constructed It has and creative. Quiz was not well Quizinformation. has missing andPresentation and creative.constructed It has and a creative. few constructed. It has It has pictures and music constructed. It has very information. There are Creativity pictures and music to It has a few pictures There are no Creativity to promote student´s pictures and/or veryfew pictures,few pictures, no pictures or music. promote student´s and/or music. pictures or music. participation. music. participation.

ToTo get aa deeper deeper insight insight on theon etheffect effect of the of gamification the gamification tool on the tool students’ on the academic students’ performance, academic performance,the average percentage the average of correct percentage answers of obtained correct inanswers the questionnaires obtained in were the correlated questionnaires with the were final correlatedgrading in with the subject,the final as grading depicted in inthe Figures subject, 12 as and depicted 13. in Figures 12 and 13.

Figure 12. Correlation between the ﬁnal mark in the subject during the academic course 2018–2019 and the average percentage of correct answers in the questionnaires. Educ. Sci. 2020, 10, x FOR PEER REVIEW 16 of 19

Figure 12. Correlation between the final mark in the subject during the academic course 2018–2019 and the average percentage of correct answers in the questionnaires.

A direct correlation can be observed between the number of correct answers in the questionnaires and the final score in the subject exam. Thus, during the academic year 2018–2019, students with a final mark lower than 3 only got an average of 5% correct questions in the questionnaires, while those with a score higher than 9 reached about 87%. Those with a grade between 5 and 6 had about 50% right responses. A very similar trend was found for the academic year 2019–2020 (Figure 13). Scholars with a final grade inferior than 3 just attained 5% correct questions, those with a mark in the range of 5–6 got around 55% of the answers right whereas scholars with a score higher than 9 got almost 90% correct responses. Thus, those students who obtained the highest number of hits in the questionnaires didEduc. better Sci. 2020 on, 10 the, 350 full exam. This result cannot be fully attributed to the gamification tool, since 16it ofhas 19 been found that those students were more hardworking.

FigureFigure 13. 13. RelationshipRelationship between between the the final ﬁnal mark mark during during the the academic academic course course 2019–2020 2019–2020 and and the the average average percentagepercentage of of correct correct answers in the questionnaires.

Overall,A direct correlationfive different can patterns be observed could between be distinguished the number among of correct the answers students in according the questionnaires to their averageand the percentage final score inof thecorrect subject answers, exam. which Thus, were during in agreement the academic with year their 2018–2019, final score: students with a final mark lower than 3 only got an average of 5% correct questions in the questionnaires, while those with Type a score 1: Students higher than who 9 systematically reached about reviewed 87%. Those the withconcepts a grade after between each lesson 5 and and 6 had had developed about 50% righta responses. critical thinking skill, who replied correctly to most of the questions, hence successfully passed theA very course similar in the trend first call was with found a final for thegrading academic higher year than 2019–2020 9. (Figure 13). Scholars with a final Type grade 2: inferior Students than that 3 just frequently attained 5%reviewed correct questions,the learned those concepts with a and mark have in the gained range a of general 5–6 got aroundknowledge 55% of the of answersthe subject, right who whereas answered scholars correctly with amore score than higher 60% than of the 9 got questions almost 90% and correct got a responses.final grading Thus, thosehigher students than 6. who obtained the highest number of hits in the questionnaires did better Type on the 3: Students full exam. who This randomly result cannot reviewed be fully the attributed concepts to and the gamificationworked just tool,enough since to it pass has been the foundsubject, that those who studentsonly replied were correctly more hardworking. to about 50% of the questions and attained a score between 5 andOverall, 6. five different patterns could be distinguished among the students according to their average Type percentage 4: Students of who correct seldom answers, reviewed which the were concepts in agreement and had with not their developed final score: a critical thinking skill, who only replied properly to 15–35% of the questions and got a final grading between 3 Type 1: Students who systematically reviewed the concepts after each lesson and had developed a • and 5. critical thinking skill, who replied correctly to most of the questions, hence successfully passed  Type 5: Students who never reviewed the concepts and got less than 10% correct questions, who the course in the first call with a final grading higher than 9. are not interested in the subject and their final mark was lower than 3. Type 2: Students that frequently reviewed the learned concepts and have gained a general • 6. Limitationsknowledge on of the the Application subject, who of answered Gamification correctly Activities more than Based 60% on of an the ARS questions Software and got a final grading higher than 6. Type 3: Students who randomly reviewed the concepts and worked just enough to pass the subject, • who only replied correctly to about 50% of the questions and attained a score between 5 and 6. Type 4: Students who seldom reviewed the concepts and had not developed a critical thinking • skill, who only replied properly to 15–35% of the questions and got a final grading between 3 and 5. Type 5: Students who never reviewed the concepts and got less than 10% correct questions, • who are not interested in the subject and their final mark was lower than 3.

6. Limitations on the Application of Gamification Activities Based on an ARS Software Despite the huge potential of gamification activities as a learning tool, a number of limitations are also envisaged, which promote further investigation in the field in order to solve certain issues. An odd Educ. Sci. 2020, 10, 350 17 of 19 concept characteristic of this type of activities is that to gain more points, one should be the first in answering the question, or faster than the other participants. This game factor communicates that speediness overcomes precise knowledge of the course material and comprehension of the question. The questionnaires are also frequently a base for extra credit in lessons, hence students became more concerned on participating in the game for extra credit and do not retain the responses to the different questions. When a novel question is raised, the student has just a short time to read it before the answers are revealed along with it. Besides, due to the time constraints and the tool format, it is difficult to ask challenging and complex queries that will measure the students’ knowledge on the course material more accurately. Although questionnaires are not used to systematically teach material, the approach is strongly dependent on the time provided to answer the question and the teacher debate regarding erroneous answers. For instance, if several students respond to a question incorrectly, depending on whether or not the educator is willing to explain the reason, or just continue with the activity, the scholars will not attain a clarification to why they answered the question in the wrong way, nor will the educator comprehend why the students selected a certain response over the correct one. For such purpose, teachers could diminish the length of game-based sessions albeit devote more time to the post-test discussion of the answers and the problem-solving strategies taken to achieve the correct answers. Thus, the productivity and outcomes of the tool are profoundly dependent on the teacher who is leading the activity; hence it might be better not to use it as the foremost didactic tool. Teachers should also reach a balance between testing students on novel versus recently acquired content to preserve their attraction and maximize the efficiency of the gamification activity as a learning tool. Another disadvantage is that it is hard to track student progress. Students have nicknames they can create and during the game points are tallied based on how fast and how accurate the given answers are. In order to monitor student progress, the teacher should match the nicknames to the students’ names and then record the number of answers each student got right in every questionnaire that was performed, and this is a very tedious task. Another limitation is that there is no function to measure open response questions. The software is limited to multiple choice or true and false. It will be more useful if it could include more answering formats such as text answers. On the other hand, because of multiple participants connected to the same platform, this tool requires a strong WiFi connection. Further, sometimes the background music can be distracting and stressful, which ultimately leads to the deviation of the student from its goal.

7. Conclusions and Future Perspectives The use of gamification activities based on ARS software, in particular Kahoot! tool, has been found to be helpful for improving the students’ academic performance in a subject of chemistry. Noticeably, all the students that got a high percentage of correct answers in the questionnaires attained a good mark in the subject exam and passed the course in the first call. The gamification tool increases the motivation of the students for learning the course contents via an attractive environment, which results in better academic progress, and promotes memorizing concepts, thus facilitating the students’ learning process. Further, it encourages self-learning and self-evaluation, and allows the scholars to better understand those points on which they should focus and deepen their study. It has been an amusing, appealing, dynamic and collaborative activity that has encouraged the participation of all the students. It has been effective in reducing the frustration level of students and the stress about the fear of formative assessments in the traditional method. Likewise, the opinion of the students towards the developed tool has been very positive, and during the activity, the students showed considerable attention and interest. Further, improved learning results were attained when using the gamification tool compared to the control group that used conventional teaching methods, and its positive effects were found to be more pronounced with increasing the frequency of use of this didactic tool. Overall, it is demonstrated that the integration of mobile devices in the educational system favors the teaching–learning process. Educ. Sci. 2020, 10, 350 18 of 19

The applied tool provides teachers the facility to carry out a formative assessment of the whole class at one time without putting scholars on the spot. Future work will include a larger number of students from other institutions in order to check for data reproducibility. In addition, a comparative study between different subjects would be interesting, considering more cognitive aspects, as well as to apply the tool to more students, matters and courses (even from different departments) to better assess its effectiveness and to generalize the outcomes of this research to other matters, especially in the fields of Science. On the other hand, a comparison between the user experience of Kahoot! and other tools such as Socrative, Edpuzzle, Slido, Gimkit, Quizziz, Quizlet and TalentMS could also be highly valuable. Despite there still being some hindrances in the path of Kahoot! to become a successful leader in the educational world, it can be considered a very efficient and interactive learning tool for the students. Nonetheless, a balance between testing students on new versus recently acquired tools should be attained, in order to keep their attention, and maximize the efficiency of the gamification activity as a didactic means.

Author Contributions: Conceptualization, methodology and resources, A.M.D.-P. and M.P.G.D.; formal analysis and data curation A.M.D.-P. and M.P.G.D.; writing—original draft preparation, A.M.D.-P.; writing—review and editing, A.M.D.-P. and M.P.G.D. All authors have read and agreed to the published version of the manuscript. Funding: This research received no external funding. Conﬂicts of Interest: The authors declare no conﬂict of interest.

References

1. Raja, R.; Nagasubramani, P.C. Impact of modern technology in education. J. Appl. Adv. Res. 2018, 3, 33–35. [CrossRef] 2. Amutha, D. The Role and Impact of ICT in Improving the Quality of Education. SSRN 2020, 25, 6. [CrossRef] 3. Hartikainen, S.; Rintala, H.; Pylväs, L.; Nokelainen, P. The Concept of Active Learning and the Measurement of Learning Outcomes: A Review of Research in Engineering Higher Education. Educ. Sci. 2019, 9, 276. [CrossRef] 4. Ng, A.P.W.; Nicholas, H. A framework for sustainable mobile learning in schools. Br. J. Educ. Technol. 2013, 44, 695–715. [CrossRef] 5. Kapp, K.M. The Gamification of Learning and Instruction: Game-Based Methods and Strategies for Training and Education; Pfeiffer (Willey): New York, NY, USA, 2012. 6. Young, S.; Nichols, H. A reflexive evaluation of technology-enhanced learning. Res. Learn. Technol. 2017, 25, 1998. [CrossRef] 7. Sun, J.C.-Y. Influence of polling technologies on student engagement: An analysis of student motivation, academic performance, and brainwave data. Comput. Educ. 2014, 72, 80–89. [CrossRef] 8. Blasco-Arcas, L.; Buil, I.; Hernández, B.; Sesé, F.J. Using clickers in class. The role of interactivity, active collaborative learning and engagement in learning performance. Comput. Educ. 2013, 62, 102–110. [CrossRef] 9. Hussein, B. A Blended Learning Approach to Teaching Project Management: A Model for Active Participation and Involvement: Insights from Norway. Educ. Sci. 2015, 5, 104–125. [CrossRef] 10. Bicen, H.; Kocakoyun, S. Perceptions of Students for Gamification Approach: Kahoot as a Case Study. Int. J. Emerg. Technol. Learn. 2018, 13, 72–93. [CrossRef] 11. Bryant, S.G.; Correll, J.M.; Clarke, B.M. Fun with Pharmacology: Winning Students Over with Kahoot! Game-Based Learning. J. Nurs. Educ. 2018, 57, 320. [CrossRef][PubMed] 12. Wang, A.I. The wear out effect of a game-based student response system. Comput. Educ. 2015, 82, 217–227. [CrossRef] 13. Tan, P.; Saucerman, J. Enhancing Learning and Engagement through Gamification of Student Response Systems. In 2017 ASEE Annual Conference & Exposition; Paper ID #18943; American Society for Engineering Education: Columbus, OH, USA, 2018. 14. Zarzycka, E. Kahoot it or not? Can games be motivating in learning grammar? Teach. Engl. Technol. 2014, 16, 17–36. Educ. Sci. 2020, 10, 350 19 of 19

15. Pertegal-Felices, M.L.; Jimeno-Morenilla, A.; Sanchez-Romero, J.-L.; Mora, H. Comparison of the Effects of the Kahoot Tool on Teacher Training and Computer Engineering Students for Sustainable Education. Sustainability 2020, 12, 4778. [CrossRef] 16. Oliver, E. Gamification as transformative assessment in higher education. HTS Teol. Stud. 2017, 73, 15. [CrossRef] 17. Solmaz, E.; Cetin, E. Ask-Response-Play-Learn: Students’ views on gamification based interactive response systems. J. Educ. Instruct. Stud. 2017, 7, 28–40. 18. Wang, A.I.; Zhu, M.; Saetre, R. The Effect of Digitizing and Gamifying Quizzing in Classrooms. In Proceedings of the 10th European Conference on Games Based Learning, Paisley, UK, 6–7 October 2016; pp. 729–737. 19. González, C.S.; Toledo, P.; Muñoz, V. Enhancing the engagement of intelligent tutorial systems through personalization of gamification. Int. J. Eng. Educ. 2016, 32, 532–541. 20. Eyal, N. Hooked: How to Build Habit-Forming Products; Hoover, R., Ed.; Portfolio: New York, NY, USA, 2014; ISBN 978-1591847786. 21. Challco, G.C.; Mizoguchi, R.; Bittencourt, I.I.; Isotani, S. Steps towards the Gamification of Collaborative Learning Scenarios Supported by Ontologies. In Artificial Intelligence in Education; Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M., Eds.; AIED 2015, Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2015; Volume 9112. 22. Legault, L. Self-Determination Theory. In Encyclopedia of Personality and Individual Differences; Zeigler-Hill, V., Shackelford, T., Eds.; Springer: Cham, Switzerland, 2017. 23. Kocadere, S.A.; Ça˘glar, ¸S.Gamification from Player Type Perspective: A Case Study. J. Educ. Technol. Soc. 2018, 21, 12–22. 24. Simões, J.; Redondo, R.P.D.; Vilas, A.F. A social gamification framework for a K-6 learning platform. Comput. Hum. Behav. 2013, 29, 345–353. [CrossRef] 25. Hamari, J. Do badges increase user activity? A field experiment on the effects of gamification. Comput. Hum. Behav. 2017, 71, 469–478. [CrossRef] 26. Lindberg, R.S.N.; Laine, T.H. Approaches to Detecting and Utilizing Play and Learning Styles in Adaptive Educational Games. In Computers Supported Education. CSEDU 2016. Communications in Computer and Information Science; Costagliola, G., Uhomoibhi, J., Zvacek, S., McLaren, B., Eds.; Springer: Cham, Switzerland, 2017; Volume 739, pp. 336–358. 27. Ferro, L.S.; Walz, S.P.; Greuter, S. Towards Personalised, Gamified Systems: An Investigation into Game Design, Personality and Player Typologies. In Proceedings of the 9th Australasian Conference on Interactive Entertainment: Matters of Life and Death, Melbourne, Australia, 30 September–1 October 2013. Article No. 7. 28. ICT 2016—Information and Communications Technologies, Sub Call of: H2020-ICT-2016. Available online: https://ec.europa.eu/research/participants/portal/desktop/en/opportunities/h2020/topics/ 5088-ict-24-2016.html (accessed on 1 February 2019). 29. Nicodemus, B.; Swabey, L. Action Research. In Researching Translation and Interpreting; Angelelli, C.V., Baer, B.J., Eds.; Routledge: New York, NY, USA, 2015; pp. 157–167. 30. Dávila, V.H.L.; Cabral, C.R.B.; Zeller, C.B. Finite Mixture of Skewed Distributions; Springer: Berlin, Germany, 2018.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional aﬃliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).