sustainability

Article Effect of Flipped Teaching on Cognitive Load Level with Mobile Devices: The Case of a Graphic Design Course

Yi-Chieh Chen 1,*, Kuo-Kuang Fan 2 and Kwo-Ting Fang 3

1 Doctoral Program, Graduate School of Design, National Yunlin University of Science and Technology, 123, Sec. 3, University Road, Douliu 64002, Yunlin, Taiwan 2 Graduate School of Design, National Yunlin University of Science and Technology, 123, Sec. 3, University Road, Douliu 64002, Yunlin, Taiwan; [email protected] 3 Department of Information Management, National Yunlin University of Science and Technology, 123, Sec. 3, University Road, Douliu 64002, Yunlin, Taiwan; [email protected] * Correspondence: [email protected] or [email protected]; Tel.: +886-963-733-756

Abstract: Due to the emergence of computer education, AI education, the Internet of Things, big data, and technological wisdom, it is easy for students to be distracted when engaged in traditional education. Flipped teaching is a teaching strategy frequently used in colleges and universities. The focus of this research was conducted by a comparative analysis of the cognitive load between the experimental group and the control group through a quasi-experimental design for research with different learning methods and different classes. More specifically, flipped teaching was carried out



 with an experimental group, and traditional teaching a control group; they were observed at the same time, and 213 private university students participated in the experiment. The research proposes Citation: Chen, Y.-C.; Fan, K.-K.; a practice of mixed teaching, carried out in a group communication behavior system, and enhancing Fang, K.-T. Effect of Flipped Teaching the spirit of group interaction and learning through mobile devices. The core value of the research on Cognitive Load Level with Mobile lies in (1) online learning, (2) group interaction, and (3) the learning load of the conceptual model. Devices: The Case of a Graphic Design Course. Sustainability 2021, 13, In addition, focus group interviews were used to provide feedback on participants’ cognition and 7092. https://doi.org/10.3390/ emotions. The results indicate that there were differences in cognitive load between the two classes. su13137092 Keywords: flipped teaching; online learning; AI education; mobile devices; cognitive load; learn- Academic Editors: Haoran Xie, ing load Gary Cheng, Gwo-Jen Hwang, Morris JONG Siu-yung and Adela García-Aracil 1. Introduction Received: 1 April 2021 Online learning has been widely used in recent years. It has gradually become a Accepted: 2 June 2021 common teaching method in formal and informal education environments [1], such as Published: 24 June 2021 Massive Open Online Courses (MOOC), Small Private Online Courses (SPOC), TronClass, etc. [2]. These learning platforms not only support multiple application scenarios, such as Publisher’s Note: MDPI stays neutral “learning management”, a “flipped classroom”, and “smart classrooms”, but also provide with regard to jurisdictional claims in an interactive teaching experience [3]. The TronClass Online Learning Platform (TOLP) published maps and institutional affil- uses wireless mobile communication network technology and operates through evaluation, iations. analysis, execution, and synthesis. It provides learners with new opportunities for active learning and uses active and autonomous pre-class video learning for learners to get a preview before class. Since the invention of the smartphone, technology has allowed people to rely on Internet information and computerization more and more. However, based on Copyright: © 2021 by the authors. the global reliance on online learning, TOLP has also been used on campuses for network Licensee MDPI, Basel, Switzerland. teaching. TOLP has the advantages of no time limit and no regional learning limit. It helps This article is an open access article learners use fragmented time to promote independent learning. Nevertheless, before the distributed under the terms and introduction of the full potential of TOLP, mobile learning needs to be integrated into the conditions of the Creative Commons Attribution (CC BY) license (https:// most suitable situation to fit the formal education environment. However, video-based creativecommons.org/licenses/by/ learning with flipped teaching can help improve students’ academic performance [4]. 4.0/).

Sustainability 2021, 13, 7092. https://doi.org/10.3390/su13137092 https://www.mdpi.com/journal/sustainability Sustainability 2021, 13, 7092 2 of 26

In reviewing past studies, we see that Goal 4 of the UN Agenda 2030 sustainable development goals (SDG) is aimed at working towards quality education. Colleges and universities have an important role to play in teaching sustainability principles [5]. Al- though most experts believe that the existence of teachers is irreplaceable, there will still be many changes in the work of teachers and the best practices of AI smart education and interdisciplinary partnerships [6]. In this research, TOLP was carried out on an experimen- tal group in September 2019. Currently, this learning platform is rarely used, according to the surveyed literature. We have used a graphic design course (GDC) as an example. It is assumed that there is no significant difference between the practice of flipped teaching and traditional teaching methods. That explains why video teaching can be used to acquire knowledge, through data verification. Learners can learn autonomously and construct a domain of knowledge. This is, thus, a better learning method than traditional teaching tools (textbooks, oral delivery of knowledge, whiteboards, chalkboards, and briefings). This research focuses on learning cognition with the flipped teaching method and the interaction between cognitive loads.

2. Literature Review 2.1. What Is a Flipped Classroom? The current environment shows a trend towards emphasizing multitasking and social learning. The enhancement of digital experience has become a daily part of the learning habits of a new generation of students [7]. There is evidence that millennial students (born between 1981 and 1996) are inclined to use technology as a tool to obtain new knowledge [8]. However, “flipped teaching” was promoted as early as 1999, by MIT’s “Internet teaching course”. At that time, teachers had the tendency to transfer their knowledge material online and launched Open Course Ware (OCW) free open online courses the following year. It is used by institutions, students, and self-learners in various parts of the world [9]. “Flipped classroom” is a term for letting learners preview the contents of the classroom teaching when they are outside the classroom. It is a teaching model that involves pre-class preparation and interactive participation in the classroom. In recent years, many educational researchers and educators have tried to use the “flipped classroom” concept to solve problems that learners experience in the classroom. First of all, the teacher solves the problem of absence among secondary students by record- ing a short video of 4 to 6 min. When students miss a lecture, they can view the video lecture afterwards. It can also be used to review the main points of the class at home. However, teachers realized that some students would be to give up doing their homework, and let them try to find the answers in the video [10]. New teaching methods mean that stu- dents who do not like homework can increase their motivation to learn independently and can make fuller use of resources and time. The flipped classroom approach has, therefore, become one of the most well-known instructional approaches in recent years. If appropri- ate educational tools and methods are not provided to support students, their learning performance may not be as good as expected. Therefore, this highlights the importance of flipped classrooms [11].

2.2. Benefits of Flipped Teaching In an era of technological overload, educators have observed that it is increasingly difficult for learners to pay attention to traditional lectures. Therefore, educators must be to shift away from traditional teacher-centered classroom lectures to student-centered teaching methods. These will incorporate more diverse, cross-domain materials. Recently, many educators have begun to question traditional teaching, although the lecture format is still the main teaching method for adult learners [12]. If the goal of teaching is to improve students’ understanding, we can see that the past method of enforcing the memorizing of knowledge encourages a kind of “surface learning” [13]. Students spend time in class to study problems, have discussions, put forward ideas, and participate in collaborative learning (CL) [10]. To achieve this goal, educators try to use online video lectures to Sustainability 2021, 13, 7092 3 of 26

pre-record teaching materials before class and assign homework. Thereafter, students can discuss and undertake interactive learning activities during class time. Cooperative learning and discussion in the “flipped classroom” are regarded as important learning activities. The principle of peer-assisted learning and the cooperative learning theory are new concepts in science and technology that can assist with education in the, so-called, “flipped classroom” [14,15]. A list of subjects suited to computer-assisted instruction (CAI) [16] has been devel- oped, including textiles, graphic design, clothing construction, and the visualization of three-dimensional designs from two-dimensional patterns. CAI modules can be used by teachers to guide students in limited classroom time, and can be a learning method that suits the individual’s skill level or project needs. For example, the implementation of a flipped classroom in a clothing construction context allows students and teachers to focus their classroom time on skill development, problem-solving, and active learning while performing homework. It can also be inferred that the flipped classroom has advantages over classroom explanation. Similarly, experimental research was conducted using CAI modules combined with traditional classrooms in university courses. The results showed that students like to use CAI and have improved knowledge acquisition, so they [17–19] get a mixed-method education and show improved academic performance. Some studies [11] have indicated that students involve in collective problem-solving in a promotion-based flipped classroom not only have better learning performance and collective efficacy, but also a higher level of knowledge construction and deeper interactions. These studies can be a reference for orientation of future teaching and research. The “flipped classroom” is an innovative teaching method that relies on digital video and practical problems given as homework. The group’s active learning and use of advanced information technology will produce a new direction for educational research. This represents a major difference from the traditional approach of delivering lectures first, and assigning practice and problem-solving after class. Flipped teaching focuses on collective interaction and problem solving in the classroom. Video lectures, quizzes, and exercises are used as the course model before class [20]. In “The flipped classroom: A survey of the research” [21], college students viewed online video lectures, which were considered better than live lectures, and interactive online videos were considered even better (effect size = 0.5). Students [10] think video lectures are the better choice. The research adopts formal or informal online teaching. Although the research reveals some differences, student perceptions are relatively consistent and often positive. Only a small number of students do not like the change.

2.3. Defining of the Flipped Classroom What is the impact of pre-class video teaching on flipped classrooms of learners? Many videos created by teachers are of questionable quality and require a lot of time and resources. Nowadays, college students in higher education participate in flipped classrooms. They use field notes, interviews, and focus groups in an introductory statistics course, compared with a traditional introductory statistics course at the same university. The results show that the students in the flipped classroom are not satisfied with the adaptation of the teaching structure and their learning tasks, but the students are still willing to accept cooperative learning and innovative teaching methods [22]. “Considerations about Flip Education in the Teaching of Advanced Mathematics” indicated that, parallel to the social transformation, there is a transformation of information and communication technologies (ICT). ICT has already undergone important developments, accompanied by a clear reduction in costs and increased ease of use of different tools to create and edit quality videos, with the possibility of disseminating them on a network [23]. Some scholars have proposed that technology and innovation empower higher educational institutions (HEI) to execute different teaching and learning systems. These online interactions are useful for understanding the effectiveness of this kind of learning [24]. Sustainability 2021, 13, 7092 4 of 26

In the past, it was emphasized that educators are always experimenting, using in- novative teaching methods, and developing effective learning materials and resources. Good teaching should arouse the learning motivation of the student, and active partici- pation is necessary. Success at arousing student aptitude and interest is determined by the inclusion of moving and interesting stories [20]. Teaching in a flipped classroom can involve case studies, active learning in the classroom, experiments, games, simulations, and discussion. It also covers automatic tutorials, team learning, peer guidance, inquiry learning, instant teaching, mixed classrooms, mixed courses, and process-oriented guided inquiry learning (POGIL) [25]. In [26], the classroom response system (CRS) was mixed with the flipped classroom (FC) and team–based learning (TBL) to provide a super-mixed teaching model. These strategies can help learners to think critically and strengthen their abilities. Unfortunately, they require a high pay-off. Some researchers [21] have shown that the flipped classroom is meant to attract the interest of students through video teaching, and proposed that student-centered active learning be combined with lectures to solve problems. Therefore, researchers have explored the distinctions between “flipped teaching” and “traditional teaching”, as shown in Table1.

Table 1. Definition of the flipped teaching and traditional teaching (induction and finishing by Yi-Chieh Chen).

Inside Classroom Outside Classroom Teaching Orientation → Student-Centered Learning Interactive lecture, Video lecture, Diversity Theories Flipped teaching Peer-assisted learning, course handouts, Practice * Emphasize active learning and Solve problem exercises (Homework) interactive lectures (Solve problem Contextual learning) Practice exercises → Teacher-Centered Learning Traditional Oral lecture, Text, (Homework), Solve Theories teaching Presentation problem * Emphasize clear guidance Note: * Principles of teaching of the flipped teaching and traditional teaching.

2.4. Role of Cognitive Load in Learning Human memory can be divided into short-term memory (STM) and long-term mem- ory (LTM). It is used to log, store, and retrieve memories. Short-term memory refers to the conversion of information into memory after about 20 s, and most of it uses audi- tory and verbal triggers to enhance the learner’s memory [27]. The storage capacity for information is quite limited. However, the interaction of elements in storage requires working memory space. Long-term memory is the result of continuous practice before the short-term memory disappears. It may involve some small, fragmented, complex, and serialized human cognitive activities. Learners can transform certain perceptual features into concepts or meaningful coding forms, practice retelling, and, thereby, turn them into long-term memory. Long-term memory is the basis of learning and requires learning in a lasting sense [28]. Well-organized information can reduce the load of learning on working memory, and the tolerance of each learner to the learning load may also be different. There are three types of cognitive load (CLT): (1) intrinsic cognitive load, (2) extrane- ous cognitive load, and (3) germane cognitive load [29]. Under the interaction of elements, the demand for working memory and learning materials is the first type of load. The pre- sentation of different materials or the complexity of the task may also cause differences in load. Of course, simple learning materials and tasks, which pose a smaller relative burden, are important. However, materials with highly complex elements are unavoidable [30]. In the second category, extraneous cognitive load, the presentation of teaching materials and the information organization will affect the learner’s process of absorbing knowledge. In addition to the normal teaching process, an additional cognitive load may be generated. Moreover, the difference in level of students in the class may also lead to different adapt- ability when absorbing knowledge. If extra irrelevant or ineffective materials and resources Sustainability 2021, 13, 7092 5 of 26

are added in the process, it may cause problems with the learning and understanding of learners [31]. Therefore, it is important to design usable instructional design and procedures to reduce unnecessary cognitive load [32]. Germane cognitive load is the final type of instruc- tional design and attracts learners to focus on the cognitive process of learning content. When learning motivation is enhanced, the tasks of cognitive resources are increased simultaneously [31]. The effective cognitive load is also improved. The total cognitive load cannot exceed human mental resources. Therefore, in experimental research, a more effective teaching design is usually used to reduce irrelevant cognitive load and improve learning [33].

2.5. Cognitive Load Measurement Methods and Changes The measurement of the cognitive load was undertaken in the early stage of the evaluation of experimental design. We conducted the experimental research through two groups to understand their differences [34], and, particularly, to evaluate the problem- solving ability of the two groups. The variables to use to judge cognitive load were differences in test scores, teaching hours, and error rate. Moreover, it was recognized that earlier “classifications of cognitive load” were carried out through NASA’s Task Load [35]. The development and research of NASA-TLX obtained a subjective evaluation of 10 workload-related factors from 16 different experiments (experimental tasks include: cognition, manual control, supervisory control tasks, aircraft simulation, etc.). This task divides cognitive load into 10 categories, including overall workload, work difficulty, time pressure, performance, mental/perceptual effort, physical effort, frustration, stress, fatigue, and behavior types. The research by Chen [36] divides NASA’s task load into mental demand, physical demand, temporal demand, effort, frustration level, and performance categories (see Table2).

Table 2. Contents of NASA classification of Cognitive Load: Yi-kang Chen summarized.

Project Content How much mental effort do you need to put in to Mental Demand perform the activity? (For example: imagine, look, search, etc.) How much physical demand do you need for activity? Physical Demand (For example: push, pull, turn, etc.) How much time do you need to process the task Temporal Demand content? (fast/slow) How much effort do you need to achieve the level of Effort performance you have performed? (easy/difficult) When solving a task, do you feel relaxed, happy, Frustration Level satisfied, or does the task make you upset, frustrated, or troubled? Do you think that completing the task is successful for Performance you? (How does it perform?)

2.6. Cognitive Load in Online Learning The purpose of teaching is “meaningful learning”. If the effectiveness of teaching is not good, then the teaching will be meaningless [37]. In the past, many educators have made great contributions to the research into cognitive load. Researchers have also put forward hypothetical cases for reducing the load on working memory, successfully reducing extraneous cognitive load, and releasing memory capacity to increase the load. Afterward, dynamic methods began to appear in instructional design [33]. These provide researchers with a new cognitive load development opportunity by using instructional control to Sustainability 2021, 13, 7092 6 of 26

manipulate the way cognitive load interacts with objects. As the professional level of learners improves, these technologies will change their cognitive load to create a kind of knowledge reversal effect. This shows that cognitive load has a reasonable theoretical explanation [30]. Most learners [36] in a multimedia environment will encounter cognitive load in learning when they deal with the complexity of text and picture presentations. In the past, scholars have proposed that evaluating cognitive load also helps us to adjust learning tasks in an online environment. The authors tested five overloaded multimedia cognitive situations. However, multimedia learning is generally effective for promoting the learning of individual tasks. At the psychological level, teaching demonstration materials have also become an important part of the cognitive structure. In this way, researchers use problem-solving transfer tests to measure learning results [38]. At the same time, educators are aware that a major challenge that multimedia instructional design must face is the possibility of cognitive overload [36]. Researchers [38] have tried to solve this difficulty in their research by employing an unfamiliar material environment the assumption being that, when students do not have enough time to participate in the module, this high-load learning mode may make them unable to digest and move on to the next stage. They were used complicated teaching materials and unknown training as a guide for teaching through the construction of psychological modules. They also deleted the redundant external load material in the teaching demonstration. In the laboratory, students learn from synchronized teaching demonstrations and perform better than in the problem-solving transfer test. Research has shown that synchronous teaching can reduce cognitive load. The researcher calls this result the effect of time continuity. Therefore, [39] how to reduce cognitive load has become an important topic for educational researchers and educators. Meaningful teaching [37] entails many cognitive processing, but the cognitive resources of learners in information processing are greatly restricted. Therefore, in the application of classroom teaching design, unnecessary cognitive load needs to be avoided as much as possible to achieve an effective reduction in cognitive load.

2.7. AI Used in Education and Sustainable Learning Nowadays, educators, psychologists, and parents are most concerned about the time that students or children spend using screen. This kind of emerging technology appears in the form of artificial intelligence education and machine learning, and it will change the environment of future learning and education. The “Artificial Intelligence Market in the U.S. Education Field” reports that the U.S. predicts and analyzes the growth of 49.22% in artificial intelligence in 2021–2025 and the global effects of the COVID-19 pandemic [40]. When various factors make it impossible to conduct regular education, online learning in AI intelligence can fill the gap in the needs of teaching and learning [41]. For many years, especially with class management in private schools, teachers have had to face the educational management of the individual special needs of 60–70 students in each class. The degree of differentiation of the learners themselves makes it difficult for teachers to operate. Therefore, it is relatively important to provide learners in smart education and digital platforms. For artificial intelligence, it can help break down silos between schools and between traditional grades [42]. However, for most parents, supporting AI education is very excited about their children’s counseling and remedial plans. Moreover, it shows the importance of AI instructor’s communication method and social existence [43]. In the future, AI education [42] can help open classrooms around the world to ev- eryone. With the development and innovation of machinery and equipment, for other special functions, such as: visual or hearing impaired of functional facilities, students who cannot go to school due to illness, etc. Even in the future, the learner may have facial expressions through machines, which can immediately return the teacher’s attention. The recent development of artificial intelligence and big data technology will promote the rapid transformation of e-learning systems and applications into a new era, e-learning 4.0 [44]. Sustainability 2021, 13, 7092 7 of 26

Artificial intelligence (AI) has considerable potential in the future education vision, and is regarded as a sustainable educational tool.

3. Materials and Methods Considering the popularization of AI education and the use of the Internet in modern society, it is clear that the constraints of time and space have already been broken. Recently, some universities have begun to implement TOLP as a learning method. In the past, some scholars [45] (2013) have indicated that learning in the beginning and bottleneck stages of learning can be guided by extrinsic motivation. Once students become autonomous learners, extrinsic incentives become unnecessary. Intrinsic and extrinsic motivations complement each another. Learning motivation and external motivation are affected by (1) learning performance, (2) teacher rewards, and (3) the professional skills acquired. These become the motivation for learners to learn independently, and they will be obtained through some incentive measures. Therefore, learning motivation serves as an intermediary between stimulation and learning. It is more and more common, especially among private university students, for a lack of learning motivation to be seen. Students who repeatedly fail in learning will, in many cases, drop out. In order to address this situation, teachers, schools, and parents use TOLP learning as a source of after-school remedial materials and tools. It is hoped that students’ interest and attitude can be improved, and, thereby, their learning performance can be improved as well. Scholars [46] proved in the past that cognitive variables are related to specific professional abilities and expectations. In addition, learning performance and learning effectiveness have an interactive influence. Hence, TOLP is also regarded as a learning system for remedial teaching in universities. The remedial teaching-learning system can be used to enhance students’ interest and performance. The researcher uses TOLP as a pre-class preview method and Kahoot, Quizlet Live, and XMind mobile device apps in the classroom as a mixed teaching method. For the group discussion, the learner uses the app device to increase motivation and memory through game-manner interaction. The literature were reviewed above indicates that TOLP is a potential avenue to design effective remedial GDC classrooms and peer interaction can promote students’ learning and development. We sorted it out (1) TOLP, (2) a classroom game of Kahoot and Quizlet Live’s group interaction model, and (3) XMind (mind mapping app) to engage students in remedial GDC classrooms and promote intrinsic motivation and self-learning, as well as to measure whether the teaching methods employed for the two classes have a significant impact.

3.1. Framing and Hypothesis Model The influence of the learner in the course of study is often due to many factors. This research takes teaching method as the independent variable (IV) with divides it into two groups to compare “flipped teaching” and “traditional teaching”, taking cognitive load as the dependent variable (DV). The students from two different classes will be observed to determine whether differences in teaching methods have a significant effect on cognitive load (Figure1). Based on the above research framework, we propose the following four hypotheses:

Hypothesis 1 (H1). There is no significant difference in mental load between flipped teaching and traditional teaching.

Hypothesis 2 (H2). There is no significant difference in physical load between flipped teaching and traditional teaching.

Hypothesis 3 (H3). There is no significant difference in time load between flipped teaching and traditional teaching. Sustainability 2021, 13, 7092 8 of 26

Sustainability 2021, 13, x FOR PEER REVIEW 8 of 29 Hypothesis 4 (H4). There is no significant difference in classroom interaction between flipped teaching and traditional teaching.

Cognitive load

Mental load Teaching method H1

H2 Physical load Flipped Traditional teaching teaching H3 Time load

H4 Classroom interaction

Figure 1. Hypothesis Model (the(the following four hypothesis).hypothesis).

3.2.Hypothesis Research Design1 (H1): and There Structure is no significant difference in mental load between flipped teaching and traditional teaching. We used a quasi-experimental design to observe, measure, and record the relation- ships between variables. Our research adopts the methods of “The Flipped Classroom: Hypothesis 2 (H2): There is no significant difference in physical load between flipped instructional efficiency and impact of achievement and cognitive load levels”, published teaching and traditional teaching. by Turan and Goktas in 2016 [47]. Their study concerned a 10-week course in the spring semester of 2013–2014. Participants included experimental groups and a sample size of Hypothesis 3 (H3): There is no significant difference in time load between flipped teaching 116 in the control group. The results showed that, compared with the traditional teaching and traditional teaching. model, the students in the flipped classroom teaching model had higher academic perfor- mance and lower cognitive load, and so, when designed effectively, the flipped classroom Hypothesis 4 (H4): There is no significant difference in classroom interaction between method can be considered a useful approach in higher education settings. Therefore, this flipped teaching and traditional teaching. teaching experiment was conducted during an 18-week GDC course from September 2019 to January 2020 (one semester). 3.2. Research Design and Structure This research used TOLP in experimental group. The teacher recorded a short teaching videoWe before used class. a Studentsquasi-experimental were allowed design to enter to the observe, digital platform measure, for and learning record before the class.relationships In the flipped between classroom, variables. with Our six hours research of classes adopts per week, the m weethods used ofgroup “The interactive Flipped gamesClassroom: to encourage instructional creative efficiency thinking, and assigning impact of Kahoot achievement and Quizlet and Livecognitive on mobile load devices.levels”, Inpublished the process by Turan of participating and Goktas in in the 2016 discussion, [47]. Their students study learnedconcerned (1) a team 10-week spirit, course (2) peer in mutualthe spring assistance, semester (3) ofindependent 2013–2014. exploration, Participants and included (4) thinking experimental skills. Creative groups thinking and a andsample design size keywords of 116 in the after control the group’s group. discussion The results were showed made that, into compared a mind map, with and the XMindtraditional (amind teaching mapping model, app) the was students used onin the mobile flipped devices. classroom Finally, teaching we continued model had the designhigher academic thinking inperformance graphic design and lower and let cognitive students load, participate and so, in when the TISDCdesigned international effectively, studentthe flipped design classroom competition. method In can order be toconsidered enhance students’ a useful interestapproach and in motivationhigher education in the course,settings. we Therefore, allowed studentsthis teaching to review experiment the learning was conducted materials duringand learn an independently 18-week GDC atcourse home from through September TOLP’s 2019 mobile to January devices. 2020 Moreover, (one semester we used). a mixed teaching method and theThis group research interactive used TOLP game inmechanism experimental in the group. flipped The classroom. teacher recorded In the classroom a short implementation,teaching video before a variety class. of learning Students platforms were allowed and apps to enter were the used digital on mobile platform devices for (Figurelearning2). before However, class. in In the the controlflipped group, classroom, the “traditional with six hours teaching” of classes method per week, was we used used to explaingroup interactive the stepsof games computer to enc graphicourage design creative during thinking, classroom assigning time, Kahoot supplementing and Quizlet the classroomLive on mobile explanations devices. withIn the a process PowerPoint of participating (PPT) presentation, in the discussion, and assigning students homework learned exercises(1) team spirit, for after (2) class. peer mutual assistance, (3) independent exploration, and (4) thinking skills. Creative thinking and design keywords after the group’s discussion were made into a mind map, and XMind (a mind mapping app) was used on mobile devices. Finally, we continued the design thinking in graphic design and let students participate in the TISDC international student design competition. In order to enhance students’ interest and motivation in the course, we allowed students to review the learning materials and learn independently at home through TOLP’s mobile devices. Moreover, we used a mixed teaching method and the group interactive game mechanism in the flipped classroom. In the classroom implementation, a variety of learning platforms and apps were used on mobile devices (Figure 2). However, in the control group, the “traditional teaching” Sustainability 2021, 13, x FOR PEER REVIEW 9 of 29

method was used to explain the steps of computer graphic design during classroom time, Sustainability 2021, 13, 7092 9 of 26 supplementing the classroom explanations with a PowerPoint (PPT) presentation, and assigning homework exercises for after class.

FigureFigure 2 2.. FlippedFlipped classroom classroom of of execute execute program program ( (byby Yi Yi-Chieh-Chieh Chen Chen).).

TheThe contents contents of of flipped flipped teaching and traditional teaching were designed with two forms:forms: ( (1)1) Flipped teaching: accessing accessing text, text, images, images, online online videos, and diversified diversified visual learninglearning materials materials before before class, class, with with interac interactivetive lectures, lectures, collaborative collaborative design design and and creation creation ((group),group), peer-assistedpeer-assisted learning, learning, case case participation, participation, and and solving solving homework homework problems problem in class;s in class;will be will carried be carriedout by means out by of means after-class of practiceafter-class exercises practice and exercises homework and assignments. homework assignments.(2) Traditional (2 teaching:) Traditional practice teaching: exercises practice and assignments exercises mustand assignments be completed must at home be completedbefore class; at therehome are before oral lecturesclass; there and are writing oral lectures on a whiteboard and writing or chalkboard;on a whiteboard and theor chalkboard;teacher uses and spoken the teacher language uses and spoken PPT presentationslanguage and toPPT explain presentations basic knowledge to explain inbasic the knowledgeclassroom, asin shownthe classroom, in Table 3as. shown in Table 3.

Table 3. Experimental design program:program: course.course.

Experimental Group/Flipped Teaching Control GroupControl/Traditional Experimental Group/Flipped Teaching Group/Traditional Teaching Teaching video teaching materials, diversified visual video teaching materials, diversified visual learning practice exercises homework Before class learning materials, using TOLP in the Mobile practice exercises homework Before class materials, using TOLP in the Mobile Device or computer, assignments Device or computer, etc. assignments ExperimentalExperimental designdesign program: etc. Course interactive lecture, collaborative design and oral lecture, writing teaching program: Course interactive lecture,creation collaborative (group), Peer-assisted design and creation learning, case oralon lecture, whiteboard writing and teaching chalk, on In(group class), Peer-participation,assisted learning, solve case homework participation, problems, solve whiteboardspoken language and chalk, and spoken PPT In class homework problemsusing Kahoot,, using QuizletKahoot, Live,Quizlet and Live, XMind and of languagepresentations and PPT for presentations knowledge app in the Mobile Device explanation XMind of app in the Mobile Device for knowledge explanation After classAfter practice class exercises practice homework exercises assignments homework assignmentssolve solve homework homework problems problems

3.3.3.3. Research Research ObjectObject and and Survey Survey Sample Sample ThisThis experiment experiment took took two two different classes of the universities ( (seniorsenior undergraduate studentsstudents)) as as the the ex experimentalperimental objects objects at at the the private universities in our country country.. GDC was the teaching subject, including 123 students from experimental group and 90 students from control group, with a total of 213 students. Sustainability 2021, 13, 7092 10 of 26

3.4. Variable Measurement and Design In terms of the “face validity” of this research, we used a 21 question questionnaire designed according to an article by Chen from 2010 [36]. He mentioned that the items were divided into mental demand, physical demand, temporal demand, effort, frustration level, and performance; these six perspectives are discussed in Section 2.5. This research adopted Chen’s [36] classification of cognitive load, which was originally divided into mental demand, physical demand, and temporal demand. In the past, scholars had pro- posed various learning methods, emphasizing the importance of classroom interaction (between students and between students and the teacher) [14,16,21,25]. Classroom interac- tion generally receives positive feedback [10,17]. In this research, therefore, the concept of “classroom interaction” has been added as an aspect of cognitive load. Therefore, we propose the broad category of “cognitive load”, which includes four aspects: “mental load”, “physical load”, “time load”, and “classroom interaction”. Table4 presents the variables in the questionnaire to give a better sense of the degree of impact on students’ cognitive load. Three experts were solicited to undertake the “expert validity” testing, and the in- fluence variables of this research were assessed. They gave their professional opinions and suggested amendments, and the questionnaire design was examined. The original questionnaire was designed with a total of 23 influence variables. Experts suggested that the open-ended feedback for question X23 should be deleted. The original question X7 was removed because it does not assess mental load, but performance in terms of “be- havioral motivation”. The experts instead recommended the use of a “gatekeeper (the role of the teacher)”. The questionnaire is based on the 5-point Likert scale, ranging from “strongly disagree” to “strongly agree”. Closed questions are used to provide feedback on the cognitive load of learning throughout the semester.

3.5. Research Ethics and Limitations This research adopted quantitative and qualitative research methods, comparing them with focus group interviews to set specific research goals. Quantitative data were measured by anonymous questionnaire recovery for regional private universities. The aim was to understand whether learners encounter obstacles in the learning process that teacher does not see in class, and to decide how to evaluate learning results. The results are only applicable to regional private university education in this area of Taiwan. Due to the different cultures and customs of other regions, it is necessary to base the strategies on the opinions of students from different backgrounds. It will be necessary to choose reasonable and scientific evaluation methods for the educational design that suit the particular context. Sustainability 2021, 13, 7092 11 of 26

Table 4. Cognitive load of this research: definition collation.

Facet Question Number Content Definition Question (Variable Content) Data Source X1. I can understand the learning content of the course. X2. I can remember the professional skills of unit learning How much mental effort do you need to put in after being taught by the teacher. to perform tasks? (For example: search for X3. I can practice the content of the class repeatedly after Q1–Q6 Mental load information before class, practice repeatedly the class. after class, listen carefully, etc.) X4. I think I am studying and listening seriously. X5. I feel pressured to study. X6. I have studied hard, but I still can’t keep up. X7. My whole body is very tired. How much physical stamina do individuals X8. My eyes are very tired. need to perform the learning activities? (For X9. I feel sore in my shoulders and neck. Yi-kang Chen (2010) Physical Q7–Q13 example, assess the degree of eye fatigue, X10. I have a sore back and waist. load shoulder and neck pain, wrist pain, back and X11. I have sore hands and wrists. waist soreness, etc.) X12. I have a severe headache. X13. I have experienced shortness of breath. How much time does the individual need to X14. I think the context of the teacher’s explanation is clear. use to process tasks? (Fast/slow learning, time X15. I think the teaching schedule is appropriate. Time load Q14–Q17 control of teaching context, length of time X16. I can hand in the homework on time in class. required for homework, etc.) X17. I think the degree of difficulty of the homework is appropriate. X18. I participate in activities in class and interact well with Foot et al. (1998), How is the individual’s learning interaction teacher and classmates. Slocum et al. (2005), Classroom Q18–Q21 X19. When participating in class group assignments, I interaction and participation in the classroom? Bishop et al. (2013), discussed and interacted well with my classmates. Herreid et al. (2013), X20. I think my classmates can help me in time. Byrd-Bredbenner et al. (1991), X21. I think the teacher will assist me as much as possible. Tucker (2012) Sustainability 2021, 13, 7092 12 of 26

4. Results 4.1. Research Object and Academic Performance: Basic Information In terms of gender, females accounted for 74% of the class in flipped teaching and males 26%. The class with traditional teaching had the opposite situation in terms of gender, with males and females accounting for 87% and 13%, respectively. However, about 60% of the students in the flipped teaching class came from a relevant design background (i.e., high school learning). In the traditional teaching class, only about 40% of the students had a relevant design background (see Table5). The average grades for the two classes for the semester were 89.40 score and 80.95 score, respectively. It is clear that student performance after flipped teaching was higher than that of traditional teaching.

Table 5. Number of subject samples: background variables and information.

Experimental Group/Flipped Teaching Control Group/Traditional Teaching Teaching Method Category Number of Percentage Category Number of Percentage People People Male 32 26% Male 78 87% Gender Female 91 74% Female 12 13% Related 79 64% Related 48 53% Department department department Nonrelated 44 36% Nonrelated 42 47% department department Academic perfor- Semester grades Score 89.40 SD 9.85 Semester grades Score 80.95 SD 7.84 mance/score Sample size 123 57% 90 43%

4.2. Sample Size Determination 4.2.1. Non-Parametric Testing We used the Chi-squared test to determine the representativeness of the recovered samples. First, we used non-parametric tests to measure the ratio of male to female students. H0 = The gender ratio of students taught by flipped teaching is the same as that for traditional teaching. The Chi-squared statistic was 0.011a, and the P-value = 0.917, which is greater than 0.05, so the null hypothesis (H0) was accepted.

4.2.2. Test of Homogeneity We used the test of homogeneity to measure the distribution of two samples for the same variable. In analyzing samples from a private university in the central region of the country and a private university in the south, we hoped to determine whether the two counties and cities had similar distribution patterns and the same characteristics, to avoid external interference. H0 = The sample size of flipped teaching is the same as that of traditional teaching. The Chi-squared statistic was 213.000a, and the P-value = 0.468, which is greater than 0.05, so the null hypothesis (H0) was accepted. H0 = The related department of flipped teaching is the same as that of traditional teaching. The Chi-squared statistic was 2.562a, and the P-value = 0.109, which is greater than 0.05, so the null hypothesis (H0) was accepted. Based on the above observations, the gender ratio of the sample population is con- sistent. It is also consistent with the homogeneity in related departments. Therefore, we used the sample size of the two classes as a follow-up measure. In addition, the sample size corresponds to 5 times the smallest variable.

4.3. Variables’ Mean and Standard Deviation In terms of the average value of “mental load” for flipped teaching and traditional teaching, it can be seen that for X1 “I can understand the learning content of the course”, the average values are 4.56 and 4.37, respectively. This means that, under their current Sustainability 2021, 13, 7092 13 of 26

mental load, the students can understand the classroom teaching content. Variable with the lowest average value is: X6 “I have studied hard, but I still can’t keep up;” the average values are 2.20 and 2.03, respectively. This indicates that the students are keeping up with the course. The average value for “physical load” can be seen in the flipped teaching: X8 “My eyes are very tired”. This variable item is the highest, with an average value of 3.24. The average value for traditional teaching is 2.53, indicating that the degree of eye fatigue is close to the acceptable level. For X13, “I have experienced shortness of breath”, the average values for the two teaching methods were low, at 2.04 and 1.87, respectively, which both equate to “disagree” answers. As for the average values of “time load”, different results were seen for the flipped teaching method. The average value for X14, “I think the context of the teacher’s expla- nation is clear”, was 4.55. The highest average value for traditional teaching was for X15 “I think the teaching schedule is appropriate”, at 4.14. The teaching methods of the two classes were different considering the average value of X16 “I can hand in the homework on time in class”, which indicated a large gap with values of 4.46 and 3.54, respectively. Classroom learning materials are previewed before class, which means that students can be more punctual about handing in “homework” during class. In traditional teaching, students who do not use video teaching and Supplementary Materials to preview the work at home in advance are required to hand in “homework” in class on time, and their performance is not as good as with flipped teaching. Finally, the average values of flipped teaching and traditional teaching for the “classroom interaction” section indicated that the teacher will assist students as much as possible with learning. The average values for the two were 4.67 and 4.05, respectively, for question X21, “I think the teacher will assist me as much as possible”. The lowest average was for X19, “When participating in class group assignments, I discussed and interacted well with my classmates”. The averages were 4.47 and 3.67, respectively, as shown in Table6.

Table 6. Variable of problem: mean and standard deviation.

Flipped Teaching Traditional Teaching Factor Code Name Variable (Problem) Mean Standard Mean Standard Deviation Deviation X1 I can understand the learning content of the course. 4.56 0.54 4.37 0.81 I can remember the professional skills of unit learning after X2 4.45 0.64 4.00 0.88 being taught by the teacher. I can practice the content of the class repeatedly after Mental load X3 3.52 1.00 3.13 1.06 the class. X4 I think I am studying and listening seriously. 3.97 0.81 3.44 1.01 X5 I feel pressured to study. 2.33 0.98 2.05 0.92 X6 I have studied hard, but I still can’t keep up. 2.20 1.03 2.03 0.90 X7 My whole body is very tired. 2.80 0.96 2.47 1.14 X8 My eyes are very tired. 3.24 1.05 2.53 1.21 X9 I feel sore in my shoulders and neck. 2.86 1.07 2.32 1.02 Physical load X10 I have a sore back and waist. 2.71 0.97 2.33 1.04 X11 I have sore hands and wrists. 2.69 0.96 2.14 0.94 X12 I have a severe headache. 2.34 0.96 2.06 1.06 X13 I have experienced shortness of breath. 2.04 0.91 1.87 0.93 X14 I think the context of the teacher’s explanation is clear. 4.55 0.60 4.13 0.78 X15 I think the teaching schedule is appropriate. 4.52 0.57 4.14 0.72 Time load X16 I can hand in the homework on time in class. 4.46 0.84 3.54 1.21 X17 I think the degree of difficulty of the homework is 4.48 0.66 3.93 0.88 appropriate. I participate in activities in class and interact well with X18 4.49 0.70 3.83 0.87 teacher and classmates. Classroom When participating in class group assignments, I discussed interaction X19 4.47 0.71 3.67 0.84 and interacted well with my classmates. X20 I think my classmates can help me in time. 4.51 0.64 3.74 0.90 X21 I think the teacher will assist me as much as possible. 4.67 0.47 4.05 0.90 Sustainability 2021, 13, 7092 14 of 26

4.4. Factor Analysis: Reliability and Validity A reliability analysis measures the consistency and stability of the results. In this research, the Cronbach’s α value, suggested by Cronbach [48], was adopted. If the Cron- bach’s α value is less than 0.5, it is unacceptable. Kaiser [49] proposed that the retention items of factor loadings should be at least 0.4. In the research of the Varimax rotation matrix, we can see that the factor loadings of variables X5, X6, and X16 are 0.341, 0.312, and 0.369. Therefore, we removed these three variables and then performed the reliability analysis. After deleting items with factor loadings <0.4, the reliability values were all greater than 0.5, which meets the acceptable criteria level (Table7).

Table 7. Factor analysis: “Factor loading and Cronbach’s α of Cognitive Load”.

Factor Variable Item Factor Loading Cronbach’s α X3 0.793 X2 0.741 0.817 X4 0.671 Mental load X1 0.629 X5 0.341 <0.4 (Unacceptable) X6 0.312 X9 0.906 X11 0.877 X10 0.874 Physical load X12 0.856 0.935 X8 0.830 X7 0.830 X13 0.757 X14 0.885 X15 0.856 0.877 Time load X17 0.666 X16 0.369 <0.4 (Unacceptable) X19 0.899 Classroom X20 0.897 0.910 interaction X18 0.835 X21 0.665

Fornell and Larcker [50] proposed that the square root of the average variance ex- tracted (AVE) of each factor must be greater than the correlation coefficient between the two factors in order to have good validity. It can be seen from Table8 that the square root of AVE of mental load, physical load, time load, and classroom interaction was 0.710, 0.847, 0.808, and 0.829, respectively. All of these are acceptable levels.

Table 8. Validity analysis.

Classroom Factor AVE Mental Load Physical Load Time Load Interaction Mental load 0.505 (0.710) Physical load 0.719 −0.007 (0.847) Time load 0.653 0.615 0.114 (0.808) Classroom 0.688 0.566 0.138 0.540 (0.829) interaction Note: The elements on the diagonal values in parentheses represent the square root of the average variance extracted (AVE) and the values outside the diagonal show the correlations between the factors. Sustainability 2021, 13, 7092 15 of 26

4.5. Hypothesis Test The experiment involved anonymous data on graphic design students from two classes. One of them was an experimental group and the other was considered a control group to compare the differences in flipped teaching. We aimed to determine whether there are significant differences between the flipped teaching and traditional teaching methods in terms of the cognitive load of students. A t-test (independent samples) was used to measure the two groups of classes. The results showed that the P-values of the four factors in the cognitive load <0.05 for flipped teaching and traditional teaching all had significant differences (as shown in Tables 9, 11, 13, and 15). The following is the series of hypothesis testing. There is no significant difference in mental load between flipped teaching and tradi- tional teaching (H1; Table9).

Table 9. Mean, Standard deviation, T-value, P-value, and results of mental load.

Flipped Teaching Traditional Teaching Factor Standard Standard T-Value P-Value Result Mean Mean Deviation Deviation H1- mental Reject 4.13 0.58 3.73 0.78 4.194 0.000 *** load hypothesis Note: *** P < 0.001 (extremely significant); statistics data.

Based on the above result, a post hoc analysis method was used to find the detailed analysis of the mental load (Table 10).

Table 10. The relationship between the variables, P-values and significant results in mental load.

Flipped Teaching Traditional Teaching Variable Standard Standard T-Value P-Value Result Mean Mean Deviation Deviation Accept X1 4.56 0.54 4.37 0.81 2.052 0.055 hypothesis Reject X2 4.45 0.64 4.00 0.88 4.343 0.000 *** hypothesis Reject X3 3.52 1.00 3.13 1.06 2.770 0.006 ** hypothesis Reject X4 3.97 0.81 3.44 1.01 4.227 0.000 *** hypothesis Note: ** P < 0.01 (highly significant), *** P < 0.001 (extremely significant); statistics data.

A number of studies have suggested that flipped teaching can help students to: (a) think effectively and critically; (b) solve problems; (c) understand more easily; and (d) memorize knowledge content, thereby achieving a good learning performance [51]. Among the variables comprising “mental load”, only X1, “I can understand the learning content of the course”, did not reach significance (P = 0.055); the other variables reached significant difference, indicating that, with both flipped teaching and traditional teaching, students can understand the learning content of the course. X2, “I can remember the professional skills of unit learning after being taught by the teacher”, had an average value of 4.45 for flipped teaching and an average of 4.0 for traditional teaching, which means that students are learning actively and independently. Experiments have proven that flipped teaching can promote the active and autonomous learning of students and enhance their achievements (Table 10). There is no significant difference in physical load between flipped teaching and traditional teaching (H2; Table 11). Sustainability 2021, 13, 7092 16 of 26

Table 11. Mean, Standard deviation, T-value, P-value, and results of physical load.

Flipped Teaching Traditional Teaching Factor Standard Standard T-Value P-Value Result Mean Mean Deviation Deviation Reject hy- H2- physical load 2.67 0.81 2.25 0.92 3.543 0.001 ** pothesis Note: ** P < 0.01 (highly significant); statistics data.

Based on the above result, a post hoc analysis method was used to find the detailed analysis of the physical load (Table 12).

Table 12. The relationship between the variables, P-values and significant results in physical load.

Flipped Teaching Traditional Teaching Variable Standard Standard T-Value P-Value Result Mean Mean Deviation Deviation Reject X7 2.80 0.96 2.47 1.14 2.260 0.025 * hypothesis Reject X8 3.24 1.05 2.53 1.21 4.553 0.000 *** hypothesis Reject X9 2.86 1.07 2.32 1.02 3.753 0.000 *** hypothesis Reject X10 2.71 0.97 2.33 1.04 2.741 0.007 ** hypothesis Reject X11 2.69 0.96 2.14 0.94 4.116 0.000 *** hypothesis Accept X12 2.34 0.96 2.06 1.06 1.961 0.051 hypothesis Accept X13 2.04 0.91 1.87 0.93 1.337 0.183 hypothesis Note: * P < 0.05 (significant), ** P < 0.01 (highly significant), *** P < 0.001 (extremely significant); statistics data.

Meaningful teaching and learning require a lot of cognitive processing. In information processing and teaching design, unnecessary cognitive loads should be avoided. The only variables of “physical load” whose P-values have no significant difference are X12 and X13, “I have a severe headache” and “I have experienced shortness of breath”. The average values of X12 are 2.34 and 2.06. The average values of X13 are 2.04 and 1.87, indicating that the students do not have these physical symptoms. However, students had significant P-values for “My whole body is very tired”, “My eyes are very tired”, and “I have sore shoulders and neck/back and waist/hands and wrist”. These two different teaching methods led to significant differences in these variables (Table 12). There is no significant difference in time load between flipped teaching and traditional teaching (H3; Table 13).

Table 13. Mean, Standard deviation, T-value, P-value, and results of time load.

Flipped Teaching Traditional Teaching Factor Standard Standard T-Value P-Value Result Mean Mean Deviation Deviation H3- time Reject 4.52 0.57 4.07 0.67 5.259 0.000 *** load hypothesis Note: *** P < 0.001 (extremely significant); statistics data. Sustainability 2021, 13, 7092 17 of 26

Based on the above result, a post hoc analysis method was used to find the detailed analysis of the time load (Table 14).

Table 14. The relationship between the variables, P-values and significant results in time load.

Flipped Teaching Traditional Teaching Variable Standard Standard T-Value P-Value Result Mean Mean Deviation Deviation Reject X14 4.55 0.60 4.13 0.78 4.420 0.000 *** hypothesis Reject X15 4.52 0.57 4.14 0.72 4.292 0.000 *** hypothesis Reject X17 4.48 0.66 3.93 0.88 5.207 0.000 *** hypothesis Note: *** P < 0.001 (extremely significant); statistics data.

In the past, a lot of research has proposed the benefits of online learning. The main purpose is to solve the problem of student absence so that students who do not attend class can review the key points at home. However, it was unexpectedly discovered that the same was true for students who were not absent [10]. In flipped teaching, the teacher records short instructional videos before class, which allows students to preview the lecture content before the class, and then solve related problems in the classroom. The results show that the P-values all have significant differences. The students who were taught in this way agreed with the statements “I think the context of the teacher’s explanation is clear”, “I think the teaching schedule is appropriate”, and “I think the degree of difficulty of the homework is appropriate”. These values were higher than for traditional teaching. The new teaching methods improve the motivation of students to learn independently and make better use of resources and time (Table 14). There is no significant difference in classroom interaction between flipped teaching and traditional teaching (H4; Table 15).

Table 15. Mean, Standard deviation, T-value, P-value, and results of classroom interaction.

Flipped Teaching Traditional Teaching Factor Standard Standard T-Value P-Value Result Mean Mean Deviation Deviation H4- Reject classroom 4.53 0.55 3.82 0.75 7.898 0.000 *** hypothesis interaction Note: *** P < 0.001 (extremely significant); statistics data.

Based on the above result, a post hoc analysis method was used to find the detailed analysis of the classroom interaction (Table 16). In recent years, scientific and technological information has become essential to peo- ple’s daily lives. Educators have also begun to perceive that traditional teaching methods and content (a teacher giving classroom lectures) cannot ensure that learners fully concen- trate during classroom time, which may affect their learning interest and achievement. At present, many colleges and universities are using online learning platforms. The teacher uploads digital videos and teaching materials to provide learners with pre-class learning. Flipped teaching not only involves pre-class teaching videos but also allows learners to participate in classroom interactions during lectures. In the classroom, learners can discuss and solve problems with their classmates. Sustainability 2021, 13, 7092 18 of 26

Table 16. The relationship between the variables, P-values and significant results in classroom interaction.

Flipped Teaching Traditional teaching Variable Standard Standard T-Value P-Value Result Mean Mean Deviation Deviation Reject X18 4.49 0.70 3.83 0.87 6.103 0.000 *** hypothesis Reject X19 4.47 0.71 3.67 0.84 7.396 0.000 *** hypothesis Reject X20 4.51 0.64 3.74 0.90 7.224 0.000 *** hypothesis Reject X21 4.67 0.47 4.05 0.90 6.494 0.000 *** hypothesis Note: *** P < 0.001 (extremely significant); statistics data.

Some scholars [5,21,25,51–53] have proposed that flipped teaching is different from traditional teaching in that the traditional lectures are emphasized by “lecture teaching” first, with practice and problem solving as the teaching strategies after class. Flipped teaching focuses on collective interaction and solving problems in the classroom. It involves video lectures, quizzes, and exercises that are used as the course model before class. Results of the experiments show that the P-values of variables X18 to X21 indicated significant differences (0.000 ***). This means that the learners have implemented classroom interaction and pre-class videos and materials. The average value of flipped teaching is higher than that of traditional teaching (Table 16). Based on the above quantitative data analysis and results, Table 17 indicates that we should reject the hypotheses.

Table 17. Hypothesis test results of the research.

Hypothesis Result There is no significant difference in mental load between H1: Reject hypothesis flipped teaching and traditional teaching. There is no significant difference in physical load between H2: Reject hypothesis flipped teaching and traditional teaching. There is no significant difference in time load between H3: Reject hypothesis flipped teaching and traditional teaching. There is no significant difference in classroom interaction H4: Reject hypothesis between flipped teaching and traditional teaching.

4.6. Transcripts of Focus Group Interviews This research used focus group interviews to assess the flipped teaching, and axial coding was performed. We aimed to understand the learners’ feelings and experiences in relation to the use of innovative mixed materials. We compile the interview transcripts below (at the GDC classroom of by Yi-Chieh Chen teacher): Flipped teaching method (AC01) Question: How do you think the learning method of flipped teaching is different from your past experience (AC01)? • Student (A-05): In the past, after the teacher entered the classroom, the teacher opened the textbook or handout, explained the contents according to the book, and then ended the class. But in the flipped teaching method, the teacher gave us mobile phones to use in class. Then after opening the mobile phone to download the app, the teacher allowed us to play some games and engage in creative thinking. I think it is very fun. • Student (D-03): The teacher is really cool. We all know that Yi-Chieh teacher often takes students to participate in design competitions and win prizes. Previous students obtained a prize of 100,000 NT dollars in the TISDC International Student Design Sustainability 2021, 13, 7092 19 of 26

Competition. This was the Brno Biennale Association Special Award and was on the news—so amazing. The teacher is very flexible and lively in class, and also encourages students to assist and discuss with each other, to arouse interest and motivation in learning. I really want to win the prize, and I am envious. • Student (F-02): I think TOLP is really easy to use. I’m not familiar with my classmates. Sometimes I can use the function of contacting my teacher and classmates on TOLP. You can also ask questions in the discussion board. If the teacher sees it, she will reply to me. I find it very convenient and practical. Online learning with TOLP (AC02) Question: What do you think of the online learning (TOLP) for pre-class tutoring and after-class remedial of ideas (AC02)? • Student (B-07): Hmm ... I think I often can’t keep up with the class, but the teacher will put a lot of teaching materials and videos in TOLP for us to use to learn at home. Sometimes I am accidentally late, but I will check it again when I get home. If there are still problems, I will ask my classmates, and I should be able to keep up with the coursework. • Student (J-06): I think that although it takes some time to watch the videos and read through the other teaching materials at home, I can coordinate my time. At first, I found it a little troublesome, but after I became familiar with the software applications, I found them very convenient. Sometimes you are late for class or miss a class, but you can look at the information in TOLP at home to catch up on the relevant content. • Student (K-01): Every time, I read what the teacher is going to teach that day before the class starts. The previous teachers only provided a course outline. The teacher will record a GDC video in advance and put it in TOLP. This is very convenient: you know in advance what the teacher wants to teach, and you can learn it quickly. Learning by classroom interaction (AC03) Question: Do you think the current method of classroom interaction is different for your relationships with classmates (AC03)? • Student (G-04): I really like using Kahoot and Quizlet Live. The teacher uses these two apps to arrange games in groups for us. During the game, classmates will also help you. Sometimes the examination will use app software to give us tests, which is very exciting and I like it very much. • Student (C-05): The teacher uses the XMind app to draw a mind map for let us during class. This app is very easy to use, and the mind map can be made easily with connecting lines. During the class, we discussed with our classmates, played games, and thought about creativity. In this process, the teacher will check everyone’s progress and status. If you don’t understand, you can ask questions right away; it’s great. • Student (M-03): A lot of the homework the teacher will solve and discuss in class. For example, the teacher allows time for us to make sketches, and write down the keywords and images in our minds through the process of creative thinking and brainstorming. Then, you can think about your own artwork, or you can discuss it with your classmates. I think it’s better than trying to do it on my own at home. Mental and physical load (AC04) Question: Do you think that, now that the teacher is using flipped teaching methods, there will be an additional physical or mental load to cope with (AC04)? • Student (E-02): Originally, there was a lot of homework for students in the Visual Communication Department, but now the homework will be reserved for us to do in class a lot of the time. The teacher hopes that the problems can be solved and discussed in class, so I think it’s okay there is no trouble in this regard. But occasionally I feel that my shoulders are tight, it may be a frozen shoulder? I do not know ... Then, when I am home, I will practice. Furiously practice my techniques, ah ah ah... Sustainability 2021, 13, 7092 20 of 26

• Student (I-06): I think the teacher will demonstrate the problem-solving method in class, but other teachers may not be doing this. I think this has solved a lot of problems; it’s great, there is no issue. And after I go home, I may be more interested; because I want to participate in the competition, I will spend time searching for information. Because I do regular exercises, I won’t be troubled by soreness. • Student (H-03): I think the students of the design department often draw pictures. Although sometimes I feel sore when I keep painting, I can accept it once I get used to it... Anyway, so long as the homework is complete and handed in on time, I will not flunk. Just fine, ha ha ha. I should add that I am painting at home and at school. Painting is an interest, and what I care about is painting. Based on the content of the above focus group interviews, we can see that some students may be late or absent, but after using online videos and teaching materials, they can catch up to get good learning results. The benefit of flipped teaching is that one can control one’s own learning time. The learner can review the material before and after class according to his or her own time arrangements. With the development of the course content from simple to complex, it is essential to appreciate the importance of integrated course design. The TISDC international student design competition also enhances students’ interest and motivation. To summarize the above interviews, due to the advent of smart phones, students are happy to use their mobile phones to gain knowledge during class. A mobile device is used as a test tool for using an app and to strengthen students’ interest in learning. TOLP online learning is regarded as a bridge between teacher and students. At the same time, app software such as Kahoot, Quizlet Live, and XMind leads to discussion and interaction between classmates and the peer relationship is strengthened. It also helps students to solve homework problems they cannot solve alone at home. As for the mental and physical loads, the teacher has set aside a lot of classroom time for solving problems. Even if students have some physical troubles caused by a long drawing time, these can generally be solved through appropriate exercise and rest. Learning is not only to gain knowledge but also to help them know how to practice and make up for any lost learning adaptability.

5. Discussion 5.1. Assisting Students with Solving Practical Problems Among the popular flipped classroom methods, experimental evidence from most researchers has shown that active autonomous learning can effectively improve learning performance [54]. When conducting courses in the form of flipped classrooms, instructional videos can help learners to acquire knowledge and skills. On the other hand, learning interaction skills from instructional videos enhance the construction and potential of knowledge through interpersonal interaction [51]. The “Review on teaching and learning of computational thinking through programming: What is next for K-12?” adopts a flipped teaching model of sustainable teaching. First, programming courses involve multiple skills and learning processes, which leads to the need for teachers to carefully design the curriculum [55,56]. In flipped teaching, the teaching method allows all students to employ integrated thinking, and students have a greater flow experience. Students were able to use their mobile phones in class and felt quite happy. In addition to allowing students to acquire knowledge; the teacher also hopes that they can learn happily. In the focus group interviews, students stated that they are happy to use smartphones and applications in the classroom. Students regard modules where teacher uses smartphones as an interesting exercise in creative thinking. Through the method of game design, students are interested in learning.

5.2. How Effective Is Online Learning for Flipped Teaching? The methodology of the flipped classroom is a possible alternative for pedagogic renovation. Online learning with TOLP has the classroom function of a discussion area: Sustainability 2021, 13, 7092 21 of 26

students can also ask questions online, and the teacher or classmates can reply. Students think TOLP are really easy to use. They can contact their teacher and classmates, and find it very convenient to use. Flipped classrooms are a good way to learn actively. Students have addressed smartphones as a crucial tool for learning [57–59]. From the many relevant studies [52,57–59], it can be seen that flipped learning has taken a step forward into a new age, using innovative strategies and technologies to facilitate flipped learning results. This research demonstrates that students gave the following feedback: Before every class, I read what the teacher is going to teach that day. The teacher will record a GDC video in advance and put it in TOLP. This is very convenient: you know in advance what the teacher wants to teach, and you can learn it quickly. In addition, we wanted to determine how online learning aids pre-class tutoring and after-class remedial work. In the interview, students did feel apprehensive when they first came into contact with unfamiliar software. However, after getting used to the online learning environment, they found it convenient. At the same time, they found that they could manage their own time and make their learning more autonomous. Hmm ... I think I often can’t keep up with the class, and sometimes I am accidentally late, I will check it again when I get home. So, I should be able to keep up with the coursework. (AC02, B-07) At first, I found it a little troublesome, but after I became familiar with the software applications, I found them very convenient. (AC02, J-06) Students’ desire to learn actively and autonomously in the classroom has become a challenge that educators need to face [26]. At present, many colleges have online learning platforms. The digital videos and teaching materials uploaded by teacher cans provide students with activities to complete at their own pace. Through the digital platform, teacher and students can communicate and connect. We found that flipped teaching and traditional teaching both have a significant P-value for time load. This means that, although flipped teaching requires additional time to learn the content of the course, it allows students to be in control of the learning progress much more than traditional teaching does. The average value of the time load for flipped teaching is 4.52, higher than traditional teaching at 4.07. Consequently, students can keep up with the course through online learning.

5.3. Mixed Modes for Group Interaction in AI Education Computer education, AI education, the Internet of Things, big data, and technological wisdom have changed the learning habits of students [60]. Mobile devices are regarded as important the learning tools in group interactive games. Peers use smartphones to download apps and use Kahoot and Quizlet Live to have real-time discussions, and exchange ideas and creative thinking in group interactions. The teacher uses these two apps to arrange games in groups for students. During the game, classmates will also help each other, and it can achieve the effect of mutual assistance. Sometimes the examination will use app software to give students tests, which is very exciting and students like it very much. In addition, TOLP online learning is regarded as a bridge between teacher and stu- dents. Flipped teaching uses the time at school instead of assigning homework. In this way, the help of peers and the ability to learn from each other are strengthened and can increase interest and motivation to learn. The teacher uses the XMind app to draw a mind map for students during class. During the class, students discussed with their classmates, played games, and thought about creativity. So that students would not be idle while studying independently. At the same time, the teacher set aside a lot of classroom time for solving problems. The teacher believes that, in the process of teaching and learning, independent thinking and discussion among students are very important. In terms of classroom interaction, the average value for flipped teaching is 4.53, which is higher than the 3.82 of traditional Sustainability 2021, 13, 7092 22 of 26

teaching, so there were significant differences and we rejected the hypothesis. This also means that, in a flipped classroom, the interaction between students is better than that in traditional lecture settings. In the end, students learn to help each other and something to struggle through alone.

5.4. Cognitive Load has Different Effects on Learning The processes of learning and memorization involve different brain structures and information conversion processes. Effectively employing these is very important for learning. There are many different structures in the human brain involved in learning and memory [61]. The human cognitive framework meets this requirement by combining working memory and long-term memory [62]. Therefore, the excessive load will make the learner unable to absorb the knowledge perfectly. In recent years, many educators have also incorporated cognitive load into their classroom, and made the classroom more interesting with lectures on cooperative learning. A cognitive load test is also useful to understand whether a learner’s learning load in the learning process is acceptable. Learners who are different, in terms of information processing, may also have different outcomes due to the interaction of different elements (e.g., the teaching stage, teaching plan design, teaching strategy, etc.). Among the factors of mental load, the average value of flipped teaching was 4.13, and the average value of traditional teaching was 3.73. Both of the P-values reached a significant level. In sharing the feelings of students, they think there was a lot of homework for students in the Visual Communication Department, but now the homework is reserved for them to do in class a lot of the time. When at home, Students can do exercises to try to loosen up their muscles. The teacher will demonstrate the problem- solving method in class, but other teachers may not be doing this. Furthermore, students are interested in participating in the competition and they will spend time searching for information. Meaningful learning involves cognitive processing, which may include a connection between graphic and verbal expression [63]. So, in the learning process, do learners also have a physical load to deal with? This is something we must consider. In the transcripts of the interviews, we see some realistic responses. Some students believe that the teacher sets homework problems in the classroom and sets aside a lot of time for students to work on their drawing homework at school. However, some students indicated that students in the design department have a relatively large amount of homework, plus practice techniques, etc., which leads to a long time spent on drawing homework and can lead to some physical strain. Some students also said that, through exercise and rest, these physical problems can be improved. The results showed a significant difference between the practice of flipped teaching and the traditional method at 2.67 versus 2.25. This indicates that flipped teaching does take more time, causing students to suffer more physical strain than during traditional teaching.

5.5. Group Communication in Teaching and Learning In the process of teaching and learning, some learners rely on videos but forget the essence of the classroom lectures. Learners who are new to video teaching may have an automatic rejection reaction that requires them to work from home in advance. Considering course management and the online learning design, the teacher must develop a training program to promote scientific flipped learning, including the use of technology as a teaching and learning support tool, and learning activities employing different strategies for different stages of flipped learning [53]. Based on the above analysis, the group communication behavior system divides the flipped teaching method into three stages: (1) online learning (before class), (2) group interaction (in class), and (3) learning load (after class). Teaching and learning in GDC are developed through the mode of group communication and interaction, and it is a multi- party relationship established between the teacher, students, and classmates. Learning is no longer via the one-way communication that prevails in traditional teaching methods. Sustainability 2021, 13, x FOR PEER REVIEW 25 of 29

technology as a teaching and learning support tool, and learning activities employing different strategies for different stages of flipped learning [53]. Based on the above analysis, the group communication behavior system divides the flipped teaching method into three stages: (1) online learning (before class), (2) group interaction (in class), and (3) learning load (after class). Teaching and learning in GDC are developed through the mode of group communication and interaction, and it is a multi- party relationship established between the teacher, students, and classmates. Learning is no longer via the one-way communication that prevails in traditional teaching methods. Instead, it is two-way or multi-party communication. The teacher uses the time in the classroom to solve many problems, and students develop their skills through independent learning and pursuing their own motivation and interests. Through such a mode, the mental, physical, and time loads can be addressed at the same time, and the classroom interaction and peer-assisted learning can be enhanced. Teaching and learning are inherently complementary. The flipped classroom can also be regarded as a kind of pre-class tutoring, and at the same time as a mixed teaching mode for after-class remedial. There are many factors influencing cognitive load, in addition to the four factors mentioned in this research. Among them is external interference by Sustainability 2021, 13, 7092 teacher in the teaching design, teaching strategy, and teaching stages. Consequently,23 with of 26 different backgrounds and variables, the results may be different (Figure 3).

The teacher allows time for us to make sketches, and write down the keywords Instead, it is two-way or multi-party communication. The teacher uses the time in the and images in our minds through the process of creative thinking and classroom to solve many problems, and students develop their skills through independent brainstorming. I think it’s better than trying to do it on my own at home (AC03, learning and pursuing their own motivation and interests. M-03). Through such a mode, the mental, physical, and time loads can be addressed at the sameAlthough time, and sometimes the classroom I feel interactionsore when I and keep peer-assisted painting, I can learning accept canit once be enhanced.I get Teachingused and to it... learning Anyway, are so inherently long as the complementary. homework is complete The flipped and handed classroom in on can time, also be regardedI will as not a kind flunk of pre-class(AC04, H- tutoring,03). and at the same time as a mixed teaching mode for after-class remedial. There are many factors influencing cognitive load, in addition to the four factorsThe teacher mentioned is very in flexible this research. and lively Among in class, them and is external also encourages interference students by teacher to in the teachingassist and design, discuss teaching with each strategy, other, and to arouse teaching interest stages. and Consequently, motivation in withlearning different backgrounds(AC01, D and-03) variables,. the results may be different (Figure3).

Group Communication Behaviour System

Online learning (Before class) Self-learning Pre class TOLP interface, Video tutoring Time load After class Motivation remedial Interest Two-way Communication Instructional Group Communication design

Independent Group interaction Instructional (In class) Smartphone strategy App Cooperation Kahoot, Quizlet Live, XMind Classroom interaction

Instructional stage Two-way Communication Group Communication Cognitive Motivation load Interest Learning load (After class) Sport Rest Mental load Physical load

FigureFigure3. 3.Group Group communicationcommunication behavior system ( (byby Yi Yi-Chieh-Chieh Chen Chen).).

The teacher allows time for us to make sketches, and write down the keywords and images in our minds through the process of creative thinking and brainstorming. I think it’s better than trying to do it on my own at home. (AC03, M-03) Although sometimes I feel sore when I keep painting, I can accept it once I get used to it... Anyway, so long as the homework is complete and handed in on time, I will not flunk. (AC04, H-03) The teacher is very flexible and lively in class, and also encourages students to assist and discuss with each other, to arouse interest and motivation in learning. (AC01, D-03)

6. Conclusions According to research survey and interviews, the mixed learning method of multi- party communication is a multi-faceted process. It needs to involve discussion and inter- action at home and in the classroom, which is a way to improve learning achievements. The purpose of education is to allow learners to gain something in the process of educa- tion. In the generation of smartphones, how to apply innovative technology to teaching systems is a topic that is valued. Educational auxiliary tools provide students with an easier-to-understand operating procedure and communication method. An innovative, Sustainability 2021, 13, 7092 24 of 26

mixed learning model requires a spirit of teamwork. However, too many unnecessary learning methods and materials may lead to the poor adaptation of learners. This research has evaluated whether students involved in a group communication behavior system can represents their opinions on GDC. The results indicated that most learners find that this mixed learning method improves their creative thinking and knowl- edge, encourages them to participate in group discussions in the classroom, and enhances their learning motivation and interest. At the same time, AI education and smart learning are regarded as sustainable education.

Author Contributions: Y.-C.C.: (a) Implement sample’s lectures teaching; (b) Data investigation, validation, and analysis; (c) Original text, draft writing, and full text manuscript, etc.; (d) Overall editing and visualization; (e) Methodology and final approval. K.-K.F.: (a) Supervision. K.-T.F.: (a) Guidance: Statistics and Methodology. All authors have read and agreed to the published version of the manuscript. Funding: This research did not receive any external funding, nor was it funded by the foundation. Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: Not applicable. Acknowledgments: Thanks to Yi-Chieh Chen’s parents Hai-Chin Chen and Shu-Fen Kuo as well as husband Yi-Hsien Chuang for co-sponsoring the funding cost in the article publication fee. Conflicts of Interest: The authors declare no conflict of interest.

References 1. Kinshuk; Chen, N.S.; Cheng, I.L.; Chew, S.W. Evolution Is not enough: Revolutionizing Current Learning Environments to Smart Learning Environments. Int. J. Artif. Intell. Educ. 2016, 26, 561–581. [CrossRef] 2. Ruiz-Palmero, J.; Fernández-Lacorte, J.-M.; Sánchez-Rivas, E.; Colomo-Magaña, E. The implementation of Small Private Online Courses (SPOC) as a new approach to education. Int. J. Educ. Technol. High. Educ. 2020, 17, 27. 3. Bognar, B.; Sablic, M.; Škugor, A. Flipped Learning and Online Discussion in Higher Education Teaching: Smart Pedagogy for Technology Enhanced Learning. In Didactics of Smart Pedagogy; Springer: Cham, Switzerland, 2019; pp. 371–392. 4. Wei, X.; Cheng, I.L.; Chen, N.S.; Yang, X.; Liu, Y.; Dong, Y.; Zhai, X.; Kinshuk, D. Effect of the flipped classroom on the mathematics performance of middle school students. Educ. Technol. Res. Dev. 2020, 68, 1–24. [CrossRef] 5. Buil-Fabregá, M.; Casanovas, M.M.; Munzón, N.R.; Filho, W.L. Flipped Classroom as an Active Learning Methodology in Sustainable Development Curricula. Sustainability 2019, 11, 4577. [CrossRef] 6. Luckin, R.; Cukurova, M. Designing educational technologies in the age of AI: A learning sciences-driven approach. Br. J. Educ. Technol. 2019, 50, 2824–2838. [CrossRef] 7. Raja, R.; Nagasubramani, P.C. Impact of modern technology in education. J. Appl. Adv. Res. 2018, 3, 33–35. [CrossRef] 8. Roehl, A.; Reddy, S.L.; Shannon, G.J. The flipped classroom: An opportunity to engage millennial students through active learning strategies. J. Fam. Consum. Sci. 2013, 105, 44–49. [CrossRef] 9. MIT. MIT OpenCourseWare (2014–Our History). Available online: https://ocw.mit.edu/about/our-history/ (accessed on 23 October 2020). 10. Tucker, B. The flipped classroom: Online instruction at home frees class time for learning. Educ. Next 2012, 12, 82–83. 11. Hwang, G.J.; Chen, P.Y. Effects of a collective problem-solving promotion-based flipped classroom on students’ learning performances and interactive patterns. Interact. Learn. Environ. 2019, 1–16. [CrossRef] 12. Bligh, D.A. What’s the Use of Lectures? Intellect Books: Bristol, UK, 1998. 13. Ritchhart, R.; Church, M.; Morrison, K. Making Thinking Visible: How to Promote Engagement, Understanding and Independence for all Learners; Jossey-Bass: San Francisco, CA, USA, 2011. 14. Foot, H.; Howe, C. The psychoeducational basis of peer-assisted learning. Peer Assist. Learn. 1998, 27, 27–43. 15. Vygotsky, L.S. Mind and Society: The Development of Higher Mental Processes; Harvard University Press: Cambridge, MA, USA, 1978; Volume 38. 16. Slocum, A.C.; Beard, C.A. Development of a CAI module and comparison of its effectiveness with traditional classroom instruction. Cloth. Text. Res. J. 2005, 23, 298–306. [CrossRef] 17. Byrd-Bredbenner, C.; Bauer, K. The development and evaluation of computer assisted instruction modules for an introductory. College-level nutrition course. J. Nutr. Educ. 1991, 23, 275–283. [CrossRef] 18. Carew, L.B.; Chamberlain, V.M.; Alster, E.A. The evaluation of a computer-assisted instructional component in a college-level nutrition course. J. Nutr. Educ. 1997, 29, 327–334. [CrossRef] Sustainability 2021, 13, 7092 25 of 26

19. Zubas, P.; Heiss, C.; Pedersen, M. Comparing the effectiveness of a supplemental online tutorial to traditional instruction with nutritional science students. J. Interact. Online Learn. 2006, 5, 75–81. 20. Prabaharan, I. The Flipped Classroom. In Proceedings of the Conference on Building Innovations for Creative Society and Generating Employability Beyond Digital Age, Chennai, India, 19–21 May 2014. 21. Bishop, J.L.; Verleger, M.A. The Flipped Classroom: A Survey of the Research. In Proceedings of the ASEE National Conference, Atlanta, GA, USA, 23 June 2013; pp. 1–18. 22. Strayer, J.F. How learning in an inverted classroom influences cooperation, innovation and task orientation. Learn. Environ. Res. 2012, 15, 171–193. [CrossRef] 23. Jordán, C.; Magreñán, Á.A.; Orcos, L. Considerations about Flip Education in the Teaching of Advanced Mathematics. Educ. Sci. 2019, 9, 227. [CrossRef] 24. Hasan, R.; Palaniappan, S.; Mahmood, S.; Abbas, A.; Sarker, K.U.; Sattar, M.U. Predicting Student Performance in Higher Educational Institutions Using Video Learning Analytics and Data Mining Techniques. J. Appl. Sci. 2020, 10, 3894. [CrossRef] 25. Herreid, C.F.; Schiller, N.A. Case studies and the flipped classroom. J. Coll. Sci. Teach. Eric 2013, 42, 62–67. 26. Malekigorji, M.; Hatahet, T. Classroom Response System in a Super-Blended Learning and Teaching Model: Individual or Team-Based Learning? Pharmacy 2020, 8, 197. [CrossRef] 27. Norris, D. Short-Term Memory and Long-Term Memory are Still Different. Psychol. Bull. 2017, 143, 992–1009. [CrossRef] 28. Gagn’e, R.M. The Conditions of Learning and Theory of Instruction, 4th ed.; Holt, Rinehart and Wniston: New York, NY, USA, 1985. 29. Sweller, J. Recent Theoretical Advances. In Cognitive Load Theory; Plass, J.L., Moreno, R., Brünken, R., Eds.; Cambridge University Press: Cambridge, UK, 2010; pp. 29–47. 30. Seufert, T.; Paas, G.W.C. Measuring Cognitive Load; Brünken, R., Ed.; Cambridge University Press: Cambridg, UK, 2010; pp. 181–202. 31. Kirschner, P.A. Cognitive Load Theory: Implications of cognitive load theory on the design of learning. Learn. Instr. 2002, 12, 1–10. [CrossRef] 32. Merrienboer, J.J.G.V.; Sweller, J. Cognitive Load Theory and Complex Learning: Recent Developments and Future Directions. Educ. Psychol. Rev. 2005, 17, 147–177. [CrossRef] 33. Paas, F.; Renkl, A.; Sweller, J. Cognitive load Theory and Instructional Design: Recent developments. Educ. Psychol. 2003, 38, 1–4. [CrossRef] 34. Chandler, P.; Sweller, J. Cognitive load Theory and the Format of Instruction. J. Cogn. Instr. 1991, 8, 293–332. [CrossRef] 35. Hart, S.G.; Staveland, L.E. Development of NASA-TLX. Results of empirical and theoretical research. In Task Load Index; Hancock, P.A., Meshkati, N., Eds.; North-Holland: Amsterdam, The Netherlands, 1988; Volume 52, pp. 139–183. 36. Chen, Y.K. A Study on The Effects of Multimedia Instruction Context to The Students’ Cognitive Load and Learning Performance. Master’s Thesis, University of Macau, Macau, China, June 2010. 37. Thomas, J.S. Phases of Meaningful Learning. Rev. Educ. Res. 1990, 60, 531–547. 38. Mayer, R.E.; Wittrock, M.C. Problem-Solving Transfer. In Handbook of Educational Psycholog; Berliner, D., Calfee, R., Eds.; Macmillan: New York, NY, USA, 1996; pp. 45–61. 39. Sweller, J. Instructional Design in Technical Areas; ACER Press: Camberwell, Australia, 1999. 40. Technavio. Artificial Intelligence Market in the Education Sector in US by End-user and Education Model–Forecast and Analysis 2021–2025. 2021. Available online: https://newsroom.technavio.com/news/artificial-intelligencemarket (accessed on 25 May 2021). 41. Sun, Y.; Wang, T.H.; Wang, L.F. Implementation of Web-Based Dynamic Assessments as Sustainable Educational Technique for Enhancing Reading Strategies in English Class during the COVID-19 Pandemic. Sustainability 2021, 13, 5842. [CrossRef] 42. Marr, B. How Is AI Used in Education–Real World Examples of Today and a Peek into the Future. Forbes Magazine, 25 July 2018. 43. Kim, J.; Merrill, K.J.; Xu, K.; Sellnow, D.D. I Like My Relational Machine Teacher: An AI Instructor’s Communication Styles and Social Presence in Online Education. Int. J. Hum. Comput. Interact. 2021, 1–11. [CrossRef] 44. Zou, D.; Xie, H. Vocabulary Learning Based on Learner-Generated Pictorial Annotations: Using Big Data as Learning Resources. Sustainability 2021, 13, 5767. [CrossRef] 45. Block, L.; Jesness, R.; Schools, M.P. One-to-One Learning with iPads: Planning & Evaluation of Teacher Professional Development; College of Education, Leadership and Counseling, University of ST.Thomas Minnesota: Minneapolis, MN, USA, 2013. 46. Lindahl, E.W.; Winship, C. A longit model with interactions for predicting major gift donors. Res. High. Educ. 1994, 35, 729–743. [CrossRef] 47. Turan, Z.; Goktas, Y. The Flipped Classroom: Instructional efficency and impact of achievement and cognitive load levels. J. eLearn. Knowl. Soc. 2016, 12, 51–62. 48. Cronbach, L.J. Coefficient alpha and the internal structure of tests. Psychometrika 1951, 16, 297–334. [CrossRef] 49. Kaiser, H.F. The varimax criterion for analytic rotation in factor analysis. Psychometrika 1958, 23, 187–200. [CrossRef] 50. Fornell, C.; Larcker, D.F. Evaluating structural equation models with unobservable variables and measurement error. J. Mark. Res. 1981, 18, 39–50. [CrossRef] 51. DeLozier, S.J.; Rhodes, M.G. Flipped classrooms: A review of key ideas and recommendations for practice. Educ. Psychol. Rev. 2017, 29, 141–151. [CrossRef] Sustainability 2021, 13, 7092 26 of 26

52. Hwang, G.J.; Yin, C.; Chu, H.C. The era of flipped learning: Promoting active learning and higher order thinking with innovative flipped learning strategies and supporting systems. Interact. Learn. Environ. 2019, 27, 991–994. [CrossRef] 53. Chen, C.K.; Huang, N.T.N.; Hwang, G.J. Findings and implications of flipped science learning research: A review of journal publications. Interact. Learn. Environ. 2019, 22, 1–18. [CrossRef] 54. Marks, J.; Ketchman, K.J.; Riley, D.R.; Riley, L.; Brown, M.; Bilec, M. Understanding the benefits of the flipped classroom in the context of sustainable engineering. Age 2014, 24, 1. 55. Lye, S.Y.; Koh, J.H.L. Review on teaching and learning of computational thinking through programming: What is next for K-12? Comput. Hum. Behav. 2014, 41, 51–61. [CrossRef] 56. Zhong, H.X.; Chiu, P.S.; Lai, C.F. Effects of the Use of CDIO Engineering Design in a Flipped Programming Course on Flow Experience, Cognitive Load. Sustainability 2021, 13, 1381. [CrossRef] 57. Ramamuruthy, V.; Rao, S. Smartphones promote autonomous learning in ESL classrooms. J. Educ. Technol. 2015, 3, 23–35. 58. Godwin-Jones, R. Emerging technologies, looking back and ahead: 20 years of technologies for language learning. Lang. Learn. Technol. 2016, 20, 5–12. 59. Wu, Q. Designing a smartphone app to teach English (L2) vocabulary. Comput. Educ. 2015, 85, 170–179. [CrossRef] 60. Chen, X.; Zou, D.; Xie, H.; Cheng, G. A Topic-Based Bibliometric Review of Computers in Human Behavior: Contributors, Collaborations, and Research Topics. Sustainability 2021, 13, 4859. 61. Greeno, J.G.; Bjork, R.A. Mathematical learning theory and the new “Mental Forestry”. Ann. Rev. Psychol. 1973, 24, 81–116. [CrossRef] 62. Mestre, J.P.; Brian Ross, H. The Psychology of Learning and Motivation: Cognition in Education; Academic Press: Cambridge, MA, USA, 2011; Volume 55, pp. 1–313. [CrossRef] 63. Mayer, R.E.; Moreno, R. Nine ways to reduce Cognitive load in Multimedia Learning. Educ. Psychol. 2003, 38, 43–52. [CrossRef]